Published on 09.07.15 in Vol 2, No 2 (2015): Jul-Dec
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/4361, first published Feb 20, 2015.
Participatory Research as One Piece of the Puzzle: A Systematic Review of Consumer Involvement in Design of Technology-Based Youth Mental Health and Well-Being Interventions
Background: Despite the potential of technology-based mental health interventions for young people, limited uptake and/or adherence is a significant challenge. It is thought that involving young people in the development and delivery of services designed for them leads to better engagement. Further research is required to understand the role of participatory approaches in design of technology-based mental health and well-being interventions for youth.
Objective: To investigate consumer involvement processes and associated outcomes from studies using participatory methods in development of technology-based mental health and well-being interventions for youth.
Methods: Fifteen electronic databases, using both resource-specific subject headings and text words, were searched describing 2 broad concepts-participatory research and mental health/illness. Grey literature was accessed via Google Advanced search, and relevant conference Web sites and reference lists were also searched. A first screening of titles/abstracts eliminated irrelevant citations and documents. The remaining citations were screened by a second reviewer. Full text articles were double screened. All projects employing participatory research processes in development and/or design of (ICT/digital) technology-based youth mental health and well-being interventions were included. No date restrictions were applied; English language only. Data on consumer involvement, research and design process, and outcomes were extracted via framework analysis.
Results: A total of 6210 studies were reviewed, 38 full articles retrieved, and 17 included in this study. It was found that consumer participation was predominantly consultative and consumerist in nature and involved design specification and intervention development, and usability/pilot testing. Sustainable participation was difficult to achieve. Projects reported clear dichotomies around designer/researcher and consumer assumptions of effective and acceptable interventions. It was not possible to determine the impact of participatory research on intervention effectiveness due to lack of outcome data. Planning for or having pre-existing implementation sites assisted implementation. The review also revealed a lack of theory-based design and process evaluation.
Conclusions: Consumer consultations helped shape intervention design. However, with little evidence of outcomes and a lack of implementation following piloting, the value of participatory research remains unclear.
JMIR Human Factors 2015;2(2):e12
Technology and Youth Mental Health
More than a quarter of young Australians aged 16-24 years old will experience a mental illness in a 12-month period, with anxiety, substance abuse, and mood disorders the most common . Alarmingly, 3 quarters of first episode mental illness occurs before the age of 25 years [ ], and it has been reported that only 30% of these younger people are accessing the professional help that would benefit them [ , ]. With that in mind, technology-based mental health resources and interventions, part of Australia’s e-mental health strategy [ ], may offer an opportunity to engage the other 70%. The potential of technology, therefore, to increase youth engagement with formal mental health services, particularly in rural and remote contexts where service options can be limited, is yet to be fully realized.
Technology-based mental health care interventions are often cited as methods for providing greater access to and engagement with services [- ]. A recent review, however, identified only 2 studies that investigated the use of technology to increase engagement with clinical youth mental health services, and a further 3 explored the role of technology as an adjunct to face-to-face therapy [ ]. This review detailed promising results and possibilities for the role of technology in creating and augmenting developmentally appropriate and responsive youth mental health services. However, the research included lacked rigor and the dearth of studies highlight the need for more research and development in the field that is guided by an evidence base [ ].
Technology-based health interventions commonly suffer from limited uptake and/or adherence [- ], which may be dependent on methodological issues such as design, particularly how human factors are incorporated [ , , ]. For example, failing to obtain an in-depth insight into intended consumer behavior and their environments, which is crucial for good design [ ]. Guidelines for technology-based mental health design increasingly emphasize the need for formal incorporation of consumer participation into intervention design [ , - ]. Therefore, engaging young people and their support communities at all stages of development is likely to be crucial in enhancing uptake and adherence of technology-based interventions, particularly those from rural, remote, and disadvantaged communities [ , ].
There is a rich history of participatory research with children and young people in the social sciences [- ]. Participatory research is conducted in partnership with the individuals or community of interest and not on them, and in this way differs from traditional research. It purports to increase research relevance and usability through improved context appreciation. Other reported benefits of participatory research include greater stakeholder buy-in and improved efficacy and sustainability of research products (or outcomes) [ - ]. When considering the reported average 17-year gap between publication and translation of findings in health care, it is not surprising that participatory methodologies have gained prominence in the field over the last 20 years [ , , , ].
Within mental health design research, common participatory methodologies include community-based participatory research (CBPR), participatory action research (PAR), participatory design (PD), and user-centered design (UCD). PAR aims to develop an egalitarian partnership with a chosen community or group to generate positive, self-identified individual-, group-, and community-level change. While the research goals and associated theories of change may vary, PAR and CBPR are different terms for 1 research methodology underpinned by the same core principles. As such, the terms are used interchangeably in the literature depending on the country of origin [, ]. PD—borne out of British, North American, and Scandinavian traditions—employs iterative design cycles in which knowledge production and research output(s) are shared by researchers and end-users [ ]. Unlike PD, UCD is controlled by the design and research professionals, and participation takes on a strictly consultative role; the project is led, and decisions are made, by “experts” [ ]. At the other end of the participatory continuum sits consumer-led research (ie, research initiated and/or controlled by consumers), which has recently taken on new life in the context of social media.
Most research has focused on consumer participation in service delivery, with the literature around participation in intervention design via research projects still developing . It is also less common for the intervention development process to be reported [ ]. Boote, Telford, and Cooper [ ] argue that consumer involvement in research can be rationalized in 2 ways: (1) empowerment—defined as consumer involvement linked to greater autonomy in decision-making for disempowered/marginalized groups; and (2) consumerism—defined as consumer involvement linked to creating outcomes (eg, products, services or interventions) that generate satisfaction and value-for-money, with consumer input directed at improving efficiency, economy, and effectiveness. Each has different implications for the chosen methodology and role of the consumer.
The Current Review
Given the potential for technology to increase engagement with mental health services, the current review explored the question: “How have participatory methodologies been employed to develop technology-based youth mental health and well-being interventions?”
Youth participation in the development and delivery of mental health services designed to benefit them has received attention and resourcing for some time . On- and offline service-wide youth participation models are well documented and demonstrate a recognition that young people are best placed to judge what works for them given their developmental-specific experience of mental illness [ ]. Online services such as Eheadspace [ ], beyondblue [ ], and ReachOut.com [ ] provide examples of youth participation best practice. This review, however, focuses on participatory development of technology-based interventions by research groups, which may include collaboration with services or other health organizations, as compared to youth participation in an existing service. Project teams involved in production and design of technology-based mental health interventions are interdisciplinary and diverse, and their outputs and findings are distributed across multiple channels and fields depending on the discipline focus of the authors. These factors make a review of this kind a complex undertaking. This review has chosen to focus on work titled, indexed, and stored in databases with a mental health focus and, as such, will not have accessed the body of literature that exists in humanities and social sciences databases (particularly around child, youth and consumer rights and youth participation) that are reflective of multiple stakeholder contributions.
Projects that involved consumers in the design and development of interventions spanning the breadth of the mental health intervention spectrum were included to maximize learning opportunities and to gain a broad understanding of participatory processes in this emerging field of research. The aim was to synthesize previous literature and make practical recommendations for mental health technology designers who wish to employ participatory research methods in a youth context. The major concepts under investigation were: (1) the nature of consumer involvement and the participatory process in intervention development; (2) the nature and outcomes of the design process; and (3) the relationship between participatory research and the implementation of research.
By “technology-based” we refer to information and communications technology-based (ICT-based) digital interventions such as health promotion/prevention Web sites, community-focused health promotion/prevention technologies, treatment-focused Web sites/programs/therapies, and other mental health apps, games, and products. The interventions may act as standalone entities or as an adjunct to existing face-to-face treatment or programs. For inclusion in this review, developers need to have adequately defined and documented (ie, via a project report, journal article, conference paper, or thesis) a participatory development/design project.
A systematic search strategy was used to identify published and unpublished studies that described participatory research mental health projects. Database search strategies employed both resource-specific subject headings (where available) and keywords describing 2 broad concepts—participatory research and mental health/illness (the emphasis on illness terms reflected the focus on treatment-focused interventions). Keywords were often combined using proximity operators in order to increase search sensitivity (generated by SO, RD, SL, and NB). Comprehensive literature searches were undertaken in the following 15 databases: OvidSP Medline (1946-), PubMed, PsycINFO (1806-), CINAHL, Scopus, Web of Science, Informit (health, social sciences, and science and engineering subsets), arXiv.org, ACM Digital Library, and IEEE Xplore Digital Library. Database searches were limited to studies published in English. The time period for searches was database inception to June 2014. Full search strategies for the OvidSP Medline and PsycINFO databases are provided as.
To identify unpublished studies, 3 simplified versions of the search strategy were used in the Google Advanced search engine and results were restricted to PDF documents. Only the first 100 results for each search variant were reviewed for relevance (ie, total n=300). Web sites of relevant conferences were also checked for additional unpublished papers, including: Participatory Design Conference; Special Interest Group on Computer-Human Interaction; and the Computer-Human Interaction Special Interest Group of the Human Factors and Ergonomics Society of Australia. Reference lists of relevant citations were checked and email contact was made with authors to source additional relevant documentation and current information on the intervention. All searches were conducted in June 2014. EndNote X6 (Thomson Reuters) was used to manage all database citations. A first screening of titles/abstracts by a research assistant (MW) eliminated clearly irrelevant citations/documents based on research method and age group. The remaining citations were screened by a second reviewer (SO). Full text articles were sourced when a decision on relevance could not be made by title or abstract alone.
Inclusion and Exclusion Criteria
All research papers that involved projects judged as having a primary focus on youth mental health and well-being were included in the review, irrespective of whether the mental health focus was related to an existing physical condition. This decision ensured that learnings from the development of interventions spanning the breadth of the health intervention spectrum would inform development of treatment-focused interventions. Specific criteria are outlined below.
- Mental health or well-being focus (defined in consultation with a multidisciplinary team comprised of clinical mental health, technology and consumer perspectives, and informed by the DSM-V definition of mental disorder) [ ]
- English language
- Development and/or design of ICT- or digital technology-based intervention
- Youth-based intervention (or include a youth element)
- Inclusion of participatory research processes or elements thereof
- Commentaries, opinion pieces, or editorials
- Photovoice studies (judged as a distinct research methodology that does not involve design or development of a technology-based intervention)
Data Collection and Analyses
A multidimensional framework analysis, adapted from research conducted by Oliver et al  and Lorenc et al [ ], was employed to categorize research. This involved an iterative approach of familiarization with the literature and gradual development of the conceptual framework based on the broad research question. Concepts were drawn from the literature around participatory research and technology-based health intervention design. The outcome criteria were populated by criteria drawn from previous participatory research evaluation and the information needs of the study [ , , , ]. Due to the exploratory nature of the review, all levels of evidence were considered. Refer to for definitions of concepts used and their relationship to the areas of investigation. Each study was evaluated by 2 members of the research team using the definitions in . Discrepancies were discussed and consensus reached. A third member of the team was consulted if required.
In total, 14,021 citations and Web documents were identified through database searches and open Web searching. Once duplicate citations were removed, 6210 items remained for preliminary assessment of relevance. After title, abstract, and full paper screening, 17 studies were chosen for inclusion in this systematic review (and ). Of these, 1 study reached proposal stage [ ], and 1 was designed but not developed [ ].
|Project authors (publication year)||Participatory methodology||Project context||Nature of intervention and intended consumers||Fate of intervention|
|Carroll, Burge, Robertson, and Rosson (2010) ||PAR||Proposed intervention design developed by researchers at Pennsylvania State University.||Preventive Intervention: an on- and offline community network health intervention for university students and families with children with autism.||Not designed or developed (project reached proposal stage).|
|Coyle and Doherty (2009) ||UCD/ collaborative design||Project driven by human computer interaction researchers at Trinity College, Dublin.||Treatment Intervention: 3D computer game (Personal Investigator) to support therapists working with adolescents in public clinical mental health services.||Personal Investigator has undergone initial clinical evaluation over 6 months at multiple sites (n=8 mental health clinicians; and n=22 youth, aged 10-16, gender not reported).|
Indicated that more formal evaluations of the game were under way, no further information beyond time of publication.
|Ekberg, Timpka, and Angbratt, et al (2013) ||CPBR with PD process for intervention design||Collaboration between university- and government service-based researchers in Sweden. Grant funded by the Research Council for South-East Sweden.||Preventive Intervention: Online health-promoting community (OHPC) aimed at addressing factors that prevent obesity, including mental health, targeting young people aged 15-20.||Email correspondence with first author indicated a pilot of the OHPC was carried out; however, no formal evaluation was written up.|
The lead author wished to obtain sustainable funding before launching the OHPC and this is yet to be secured.
|Elf, Rystedt, Lundin, and Krevers (2012) ||PD||PhD project of first author, in Sweden. Funded by The Swedish Institute for Health Science, the University of Gothenburg, and Vinnvård.||Preventive Intervention: Web-based support system (WBSS) for young caregivers (aged 16-25) living close to someone with mental illness.||During Web site development phase, after previous attempts to pass the Web site on, the original Web site (Molnhopp.nu) was partially redesigned and rebuilt on a different platform (Livlinan.org, Lifeline) run by SPIV (a suicide prevention organization) and a volunteer-run local mental health service for ongoing management.|
The first author published on the relationship between intended (Molnhopp.nu) and real (Livlinan.org) use of the Web site. Intended and real use were weakly related and dependent on context and the needs/interests of users.
The original Web site Molnhopp.nu progressed to a randomized controlled trial (RCT) carried out over 8 months (N=241, aged 16-25 years); WBSS (Molnhopp.nu) n=120 (73% female); folder support (containing information on 24 different kinds of available support services in the community or society) n=121. The intention to treat for the primary outcome (stress) showed no significant differences between the Web group and the folder support group. Stress decreased significantly in the folder group.
|Hallett, Brown, Maycock, and Langdon (2007) ||PAR||Project driven by a multi-stakeholder participatory action research committee, led by a project officer of the West Australian Aids Council (WAAC) and funded by Healthway (West Australian Health Promotion Foundation).||Preventive Intervention: online, peer-based sexual and mental health promotion (CyberReach) for adult men who have sex with men (MSM) and same sex attracted young people (SSAY). No exact age groups stated, likely to be 14-25 for SSAY and 25+ for MSM.||Stated project objectives met (ie, developing sustainable, transferrable protocols and training, and development of transferrable protocols for peer-based Internet outreach).|
Paper reports that the piloted intervention became 2 separate services offered by the WAAC: (1) Expanded the existing SSAY to include online outreach and chat; and (2) After a more extensive trial, the MSM service eventually became a national program called “Netreach” offered by the AIDS Councils in Queensland, Victoria, Western Australia, and Tasmania. Netreach primarily provides online chat and support for MSM. Program supported by the Australian Federation of AIDS Organisations and by Gaydar.com.au.
No health promotion outcome data available.
|Løventoft, Nørregaard, and Frøkjær (2012) ||PD with modified form of classic contextual inquiry||University-based research project in Denmark.|
Project supported by Lundbeck A/S, DIKU, Telenor A/S, HTC Denmark A/S, and PROSA.
|Treatment Intervention: mobile phone app aimed at supporting people with depression by assisting with their daily lives. No target age explicitly stated. Youth consumers participating in the study aged 17-24.||Small scale 4-week evaluation of the intervention with participants who assisted with the design process—no further information available on intervention after publication.|
|Madsen, el Kaliouby, Eckhardt, Hoque, Goodwin, and Picard (2009) ||UCD with PD iterative design sessions||Project carried out by MIT Media Lab. Close links with Groden Center and Things That Think Consortium. Funded by National Science Foundation grant (hardware and software prototypes provided by Google and Samsung).||Treatment Intervention: prototype interactive socio-emotional toolkit (iSET) to assist adolescents with autism to improve social interactions (recognition, understanding, and expression of both the user’s and others’ facial expressions via software and hardware).||At time of publication, the iSET intervention was still under development, no further information is available beyond this date.|
|Matthews and Doherty (2011) ||UCD||Project driven by Human Computer Interaction researchers at Trinity College, Dublin (funding source and trial partners not stated).||Treatment Intervention: a mobile phone and online symptom tracking tool (Mobile Mood Diary) to assist adolescents with depression.||Clinical pilot (n=3 therapist, n=9 clients, mean age = 13.78, SD= 2.63, n=3 males and females, respectively) and n=1 parent, across a range of issues, including depression, mood disorders, self-harm, and anger management.|
No further information available on intervention after time of publication.
|Mazzone, Read, and Beale (2008) ||UCD with PD||PhD study of first author who was the design researcher in a multidisciplinary research team. UK university-based project led by researchers in developmental psychology and computing.|
Overall project, joint collaboration between a team of psychologists, interaction designers, and developers. Funded by the HEFCE’s Strategic Development Urban Regeneration Fund, devoted to a consortium of universities in the UK, with additional funding from Esmee Fairburn Foundation.
|Treatment Intervention: e-learning product to improve teenagers’ emotional intelligence for pupils (aged 12-15 years old) taken out of mainstream schooling due to behavioral issues (participating consumers were recruited from Pupil Referral Units).||Intervention (Uthink) implemented in Flash by a graphic designer.|
Uthink evaluation: N=84 (youth aged 14-16, n=72 males, n=12 females), no control group. Significant changes in a number of socio-emotional skills, including stress management, adaptability, and the ability to appreciate relationships between environmental cues and emotions. Participants demonstrated experiencing more care and guidance within friendships and less conflict. Reduced delinquent behavior and a desire to be increasingly challenged in school was also demonstrated.
Correspondence with project leads indicated that the game is freely available at the Uthink Web site and is currently being used by schools in Lancashire, England, and is recommended by the Lancashire County Council for use in high schools.
|Moen and Smørdal (2012) ||Action research with PD workshops||University-hospital collaboration in Norway. Funded by Centre for Rare Disorders and the IT department at Oslo University Hospital. Exploratory study.||Preventive Intervention: wiki-like site offering information, strategies, and support for people (and their families) living with anorectal anomaly focused on “living well.” Indicated all ages were being targeted, but email correspondence with first author indicated a significant youth component.||Email correspondence with the first author indicates there is no outcome paper for the intervention due to employment changes for key contributors.|
Piloting was undertaken but was challenging due to technical and interoperability problems and lack of professional and organizational support.
|Monshat, Vella-Brodrick, Burns, and Herrman (2012) ||Participatory research||Researcher-led via Orygen Youth Health Research Centre. Funding: K.M. Australian National Health and Medical Research Council (NHMRC) Public Health Postgraduate Scholarship, J.B. Victorian Health Promotion Foundation (VicHealth) Fellowship, and H.H. NHMRC Practitioner Fellowship.||Preventive Intervention: online mindfulness therapy program (mindfulness awareness training and education (MATE)) targeted at young people aged 14-25.||Pilot testing: (n=11 young people, aged 16-24, gender not reported) evaluated the 6-week MATE program. Focus group (n=7) and interview (n=5) data.|
No further information available.
|Lakey (2014) ||Participatory research||Project driven and funded by the National Health Service Greater Glasgow and Clyde as part of their strategic direction for Child and Youth Mental Health. The Greater Glasgow & Clyde NHS, Mental Health Foundation, Snook, and Young Scot were commissioned to carry out project in partnership. Outcome of project is to provide a basis for discussion with stakeholders in the board area to translate findings.||Preventive Intervention: Aimed at exploring the potential of the Internet, social media, and mobile technologies in promoting better mental health and well-being for young people.|
Multiple planned outputs. Produced digital postcards that act as a guide to staying safe and well online for young people aged 15-21.
|Project supported the development of youth-generated ideas for digital interventions to promote youth mental health and well-being. Animated GIFs (youth guide) developed but not available to the public yet. The project also developed other health service/resource design briefs. Work officially launched by Health Board on March 28, 2014.|
Project opened up connections with innovators across the UK who are willing to collaborate and develop it further.
Email correspondence with project lead: project is close to gaining confirmation of funding that will allow development and delivery of recommendations from the project’s first phase.
|Owens, Farrand, Darvill, Emmens, Hewis, and Aitken (2011) ||Participatory research||Collaboration between university and government service researchers and representatives in the UK. Funded by the National Institute for Health Research.||Treatment Intervention: text-messaging intervention to reduce self-harm for all ages.||Exploratory trial in progress at time of publication. No further information available.|
|Schmidt (2009) ||PAR||Source document was author’s master’s thesis. Youth Voices for Change (YVC) project was a subset of a larger research project (Healthy Youth/Healthy Region) that investigated connections between youth well-being and regional prosperity in the Sacramento, California, region in the US. Participating agencies: The Center for Regional Change at the University of California Davis (UC Davis) in collaboration with other project centers in UC Davis and the West Sacramento Youth Resource Coalition (WSYRC), which led the project. Funding from Sierra Health Foundation and The California Endowment.||Preventive Intervention: Google map (containing youth-produced videos and photos relating the built environment and well-being—eg, favorite, challenge, and adjust places in the community) and project Web page (the project produced other outputs but they were not technology-based). The overall aim was to investigate links between the built environment and youth well-being.||Media products presented at the planned youth community event. Qualitative data (interviews and surveys) indicated that the media products created for the event were perceived as successful by both the youth and the attendees (in terms of overall satisfaction, learning about the community, inspiring discussion, understanding people in the community and its diversity).|
At time of writing, the thesis indicates that the videos (and other project outputs) were being used by youth groups involved in the project, the Sactown Heroes, to promote their ideas and profile within the community (no clear idea how).
The current utilization status of the Google Map is unknown as it was transferred from the project Web page (which was discontinued) and placed on a community Web site. The WSYRC is using the output and connections made as a result of the YVC project to develop a sustainability plan for the Sactown Heroes group as other funding comes to an end.
|Stewart, Riecken, Scott, Tanaka, and Riecken (2008) ||PAR, youth participation model||Collaboration between university-based researchers and Canadian indigenous youth.||Preventive Intervention: Canadian indigenous youth developed artistic educational videos to address self-identified health concerns. For use in the local and other communities (aimed at high school and university students). Key research question: how can creating videos contribute to expanding health literacy?||Student videos presented at planned showcase event at the end of the school term to an audience of peers, friends, family, and community members.|
No information as to whether the videos have been used in other communities/contexts as planned.
|Valaitis, O’Mara, and Bezaire (2007) ||PD||Campus-community partnership between researchers at McMaster University and the local government health unit in Ontario, Canada (rural context). Funded by Health Canada’s Drug Strategy Community Initiatives Fund.||Preventive Intervention: rural youth (aged 14-24) developed a Web site aimed at meeting their specific health promotion needs (with moderated peer support) with a broad aim to address problematic alcohol use. The project also aimed to provide an opportunity and skills for local youth at-risk to develop and implement the health promotion Web site.||No peer reviewed papers published for this study.|
Project report: the Web site was evaluated over 8 months (2006-2007). No outcome data available on ability of Web site to meet identified health promotion needs.
The Youth Spark Web site was functional and updated until late 2014, when it was converted to a Facebook page.
|Wadley, Lederman, Gleeson, and Alvarez-Jimenez (2013) ||PD||Research project that involved collaboration between universities (from human-computer interaction and clinical backgrounds) and a research supportive youth mental health clinic in Australia. Supported by Victorian Government, University of Melbourne, Telstra Foundation, IBES, the Telematics Trust, and the Helen Macpherson Trust.||Treatment Intervention: online therapy involving psycho-education, peer-to-peer social interaction, advice, and moderation from mental health practitioners for young people with psychosis aged 15-25.||Completed a 4-week safety and acceptability trial (n=20 clients, n=3 clinicians, age and gender not reported).|
Results of pilot testing results secured funding for a 4-year RCT.
Email correspondence with first author indicates that the intervention is in the first year of a RCT—no final outcomes available.
Characteristics of the Included Studies
Of the 17 projects included in the review, included treatment-focused interventions [, , - , ]. The remaining 10 were preventive interventions [ - , - ]. UCD [ , - ], PD [ , , , ], and PAR [ , , , ] were the most common methodologies used (4 projects each). PD provided the sub-framework for an iterative design process in a further 4 projects [ , , , ]. UCD or PD methodologies tended to scaffold development of treatment-focused interventions. Three projects were based in the US and Australia, respectively, and 2 each in Ireland, Sweden, England, and Canada. The final 3 were based in Denmark, Norway, and Scotland. The age range of youth involved was 10-26 years old; 5 studies did not report age, 9 did not report gender. Besides age, no other socio-demographic variables were reported.
Nature of Consumer Involvement and the Participatory Process
Most projects (11 of the 17) involved young people (and other relevant stakeholders) for principally consumerist purposes [, , , - , , , ]; that is, to create usable, effective, and efficient interventions. A further 2 reported elements of both empowerment and consumerism [ , ]. No projects actively involved youth consumers in the project planning stage, with project aims and goals unreflective of their input.
Overall, consumers were involved in a combination of 3 main stages of research: (1) Needs analysis/design specification; (2) Intervention design/prototyping and development; and (3) Usability and pilot testing. Two projects involved consumers in all 3 stages [, ]. Projects commonly included consumers, who were most often youth and mental health clinicians (rarely family or caregivers), in the needs analysis/design specification stage [ , , , , , , , , ]. Some projects entered this stage with a predetermined intervention in mind [ , , , , ], while others operated with a looser set of intended outcomes [ , , , ]. Four projects involved consumers in the intervention design/prototyping and development stage [ , , , ]. In other projects, consumer involvement involved consulting to refine an existing intervention [ , ] or solely usability and pilot testing [ , ]. The community-based projects of Schmidt [ ] and Stewart et al [ ] developed community health education tools. They involved consumers at all stages of the project besides initial project planning.
Youth participation was variable, both across and within projects. Overall, 70% of projects reported predominantly consultative consumer involvement [, - , ] and the remaining projects were collaborative in nature [ , - ]. The projects, therefore, sat in the middle of Biggs’ modes of participation [ ]. Youth involvement was consultative in 6 of 7 treatment-focused projects [ , - , ], and 4 of these projects involved mental health clinicians as part of the research team [ , , , ]. Projects that developed treatment-focused interventions generally involved the most limited forms of consumer input. The highest level of youth participation was evident in the prevention-intervention projects, particularly Lakey, Stewart et al, Valaitis et al, and Schmidt [ - ].
Families, caregivers, and intended implementation-site representatives were under-represented in the projects. Of the 16 carried out, 7 projects clearly identified the intended implementation site and included representatives in the design phase [, , , , , , ]. The Stewart et al [ ] and Schmidt [ ] projects developed community-education focused interventions with local community representatives; however, it was unclear how widely their products were intended for distribution and thus the specific implementation site(s).
Overall, it was difficult to gain insight into consumers’ views on their participation in the projects (process evaluation) and their outputs (evaluation of the intervention). Three projects involved consumer evaluation of their experience of research [- ]. These evaluations suggested a general trend toward perceived legitimacy and accountability of the research process and its outputs, but they also served to highlight the different expectations regarding process and outcomes between project/research leads and consumers. Other projects reported informal and anecdotal consumer support for the research process [ , - ]. In some cases, pilot and small-scale clinical evaluation data were reported [ , , , , , , ].
In line with the consumerist rationale for most projects, deliberate capacity building and learning for consumers was limited; only 5 projects involved significant opportunities for this [, - ]. These involved development of preventive interventions.
Consumer involvement was seen as crucial to intervention design and development in most projects; emergent knowledge was evident in all project outputs and each made explicit reference to value of consumer involvement in intervention development. Projects reported clear dichotomies around designer/researcher assumptions of effective and acceptable interventions and those of the intended consumer. These differences were present in intervention premise and content , and mode of delivery and characteristics/components [ , , ]. Projects reported compromises between the perspectives, which were evident in the designs. Consumer consultations in the needs analysis/design specification stage were used to underpin and inform intervention design [ , , , , , , , , ]. Consumers also played a role in tailoring and contextualizing interventions [ , - ].
Eleven of the 15 completed projects reported challenges with consumer recruitment, capacity, commitment, and reliability [, , - , , , , , , ]. Cited reasons included lack of access to the target consumer group, consumer personal circumstances and/or condition-related factors, and the busy lives of youth. All projects aiming to develop treatment-focused mental health interventions found recruitment and ongoing participation of intended youth consumers difficult to achieve; however, youth consumer attrition during intervention design and development was not specific to development of treatment-focused interventions [ , , ].
Nature and Outcomes of the Design Process
Three projects used heuristic guidelines to support intervention design [, , ]. Monshat et al [ ] was guided by constructs of the Technology Acceptance Model (TAM) [ ]. Overall, 4 projects reported use of technology frameworks or theory to guide intervention development [ , , , ]. Valaitis et al [ ] used logic models to support major project decisions, including those specifically related to intervention design, such as the prototyping process, as well as techniques from scenario-based design [ , ]. Ekberg et al [ ] employed design rationales and design space analysis, which detail reasons for and justification of design decisions, to guide development of their intervention [ ]. Eight of 17 studies utilized PD methodology or principles to guide intervention development [ , , , , , , , ]. Nine projects mentioned the broad theories (including psychological, health, education, group, empowerment, and cultural) on which the intervention or project were based [ , , , , - , , ] (the details of 2 were found in project reports provided by the authors, not in the published articles [ , ]).
A structured design process, with activities able to scaffold consumer input through the design stages, was seen to be effective in a third of completed projects [, , , , ]. Use of scenario-based design—which included techniques such as storyboarding, personas [ , ], think-aloud techniques [ , ], and varied methods for capturing user experience and knowledge—was seen to assist the design process. Inspiration/idea progression and prototyping was facilitated by appropriate planning and resourcing with respect to design activities and the space in which they were conducted.
Project flexibility and responsiveness, including the ability to adapt to changing resources, priorities, work styles/preferences, output standards, and deadlines, was often built into design and was a common thread throughout projects that reported high levels of consumer involvement and influence [, , , ]. Projects led by nontechnical researchers also reported the need for integration of technical expertise at all stages of intervention design and development [ , , ]. A professional appearance of the final intervention product was also seen as important by youth consumers in a number of projects [ , , ].
In addition, balancing consumer requirements with what was possible technically, ethically, and practically (ie, time and resource, both financial and human, restrictions) was highlighted in 3 projects [, , ]. Of particular concern were social and consumer self-authoring components of interventions, privacy, confidentiality, clinical risk, and authenticity of information. Formal outcome data was available for 2 projects [ , ].
Relationship Between Participatory Research and Implementation
While leadership was not always clearly defined, most projects were researcher-led. Interdisciplinary project teams were common, including researchers or professionals with various combinations of mental health and technology domain expertise. Often, however, 1 discipline had overall responsibility for the project.
Five projects [, , , , ] reported existing relationships with outside champions who were linked to implementation sites or organizations capable of progressing the project beyond the intervention development stage. In 2 projects, Hallett et al [ ] and Valaitis et al [ ], project and governance plans were designed such that implementation of the intervention was integrated and a further 4 studies reported established links with intended intervention sites [ , , , ]. Stewart et al [ ] and Schmidt [ ] integrated community-based dissemination of outputs into their project plans. Many projects were, however, exploratory and involved development of technology-based interventions with a limited evidence base.
With the information available at the time of writing, 5 projects had extended beyond the intervention design, development, and pilot stage [, , , , ]. It is unclear the extent to which outputs from the 2 community-based projects [ , ] were used in a health promotion or prevention capacity beyond the life of the project.
Eleven projects utilized existing relationships and networks to assist with recruitment of target consumers [, - , , - , ]. The benefits of accessing consumers through existing networks was often noted; in particular, this made a significant difference in recruiting consumers with lived experience of mental illness for studies developing treatment-focused mental health interventions [ , , , ].
Nature of Consumer Involvement and the Participatory Process
A strong history of youth participation in mental health research and service development exists, rooted in the empowerment of young people to address service quality and access issues . In contrast, the projects included in this review generally involved consumers for consumerist intentions and in a consultative capacity. This represents a departure from the traditional empowerment and emancipatory rationales for participatory research demonstrated in a minority of projects in this review [ , , , - ]. These increasingly consumerist underpinnings have implications for why and how consumers are asked to participate in research and the degree of mutual benefit that is possible, desired, and ethical.
Eight of the 17 projects explicitly reported using PD methodology or methods to guide intervention development, and others used PD-related design techniques such as user journeys, personas, and workshops. PD originated in the 1970s from a Scandinavian tradition of empowering workers to exercise control over the role of technology in their workplace . Increasingly, however, the application of PD as a methodology or collection of techniques/methods has moved into design underpinned by consumerist principles that emphasize usability, effectiveness, and acceptability of the product [ , ]. This shift was embodied in several projects in this review [ , , , , , ]. Participatory methodologies with consumerist underpinnings tend to seek information and understanding through consultation and, thus, support a more passive role of the consumer in the research.
In attempting to assess perceived accountability and the legitimacy of the research process and outputs in the studies reviewed, it became clear that researchers are not in the practice of evaluating and reporting on the consumers’ participation experience. This is not only a missed opportunity for consumers to collect data in order to reflect on and learn from their experience of research, but it represents an invaluable source of data from which other projects wishing to conduct participatory intervention design and development could benefit. Email correspondence with 1 author of the studies reviewed revealed that the intervention did not progress any further from the design stage due to possible consumer dissatisfaction with the design, despite the intervention being designed and developed in collaboration with them. This highlights the need for formal assessment of consumer perceptions of accountability and legitimacy of the intervention. Existing literature notes the value derived by researchers and consumers in building in evaluation/reflection cycles, particularly for promoting the dialogue, critical reflection, and trust that are crucial components of high-quality participatory research .
While it has been reported that participatory research can enhance recruitment rates [, ], this review highlights the consumer access, recruitment, and participation challenges faced by projects aiming to develop mental health and well-being interventions, particularly those with a treatment focus that target involvement of consumers with lived experience of mental illness. Those individuals who identify as struggling with mental illness still face stigma and privacy concerns, which restrict use of common recruitment methods such as advertising [ ]. Even projects that reported collaboration with mental health services or access to those with lived experience of mental illness noted ongoing participation difficulties with maintaining consumer participation throughout the intervention design and development process [ , ].
Collaborating with existing groups of young people such as schools and youth groups [, , , - ] or organizations with a strong track record of engagement and outreach with the target consumers [ , , ] represented a recruitment starting point for multiple projects. However, they too still reported struggling with ongoing participation difficulties. These recruitment concerns are not surprising considering the move into more consumerist-based projects that tend to be less integrated into communities than traditional participatory research.
Personal capacity, reliability, and attrition of consumers, particularly in the treatment-focused intervention development projects, must also be considered [, , , , , ]. Todays’ young people contend with a myriad of demands on their time, and projects included in this review experienced this in the form of participant nonattendance, unreliability, and dropout. This effect may be amplified when the youth consumer is currently living with a mental illness. Consumers may also face financial or transport [ ] barriers in attending planned project activities that may be related to their age and/or health status. Broadly speaking, participatory research that involves consumers, particularly those who are members of minority or vulnerable populations, carries with it particular ethical considerations that require careful and sensitive negotiation and practical restrictions [ - ]. This is best exemplified in the Løventoft et al project [ ], which reported moving from egalitarian principles of PD to a designer-led user-centered approach due to challenges with consumer engagement, retention, and capacity.
The projects with the most extensive youth consumer participation were those in which young people were involved in design and development of health prevention interventions, as exemplified in Stewart et al , Valaitis et al [ ], Lakey [ ], and Schmidt [ ]. This nonclinical consumer group is far easier to access and does not have the same privacy, stigma, and personal capacity concerns facing the clinical youth consumers.
Despite this, many studies reported successful participatory research with youth consumers from a range of backgrounds. Participation is greatly assisted by links to existing consumer groups. Integration into the community of interest, via sustained partnerships between academic and nonacademic partners, is a hallmark of participatory research and has previously been shown to enhance recruitment capacity [, ]. Beyond this, future research projects would be well advised to plan for attrition; both with respect to an ongoing recruitment source and development of materials that can be provided to consumers for seamless integration into the project whenever they choose to engage or reengage. As borne out in this review, participation can and will fluctuate throughout the project and must be planned for and communicated to consumers [ ].
Flexibility and open-mindedness, embodied by a willingness to work with a non-static group of consumers and to renegotiate the time, length, style, and content of planned interactions, was repeatedly noted by the projects included in this review [, , , ]. Owens et al [ ] in particular highlights the flexibility required by a project when working in an egalitarian manner with consumers. Their intervention became more complex than planned and required extra time and resources to create. Increased cost in terms of necessary resources, time and expertise associated with participatory research [ ], along with the need for flexibility in terms of role division, project structure(s), timeframes, and even communication methods have been noted elsewhere [ ].
In working with adolescents with behavioral problems, Mazzone et al  recommend small groups and many short activities with simple tasks and objectives. They also endorse building in praise and a sense of ownership when working with all youth consumers (see also Dold et al [ ]). A structured design process that scaffolds consumers throughout was also found to be effective [ , , , , ]. Given the probable lack of technical and design knowledge of the average consumer (via techniques like storyboarding, think-aloud techniques, and scenario-based design), scaffolding the design process appears to be an important consideration for researchers.
Planning for and understanding consumer expectations of participation in research, along with their self-perceptions as mental health consumers, matters . Given the limited data available regarding consumer experience of research, building reflection and evaluation into research plans should be a focus for future research projects. Ideally, projects wishing to collaborate with youth mental health consumers require committed, youth-supportive research leadership and a process that is well-resourced and supported. Previous research suggests that projects that are age and developmentally appropriate and incorporate meaningful, individualized, empowering, and capacity-building elements improve consumer output and buy-in [ , , ], which has obvious implications for improving the current recruitment and participation issues.
Recognizing that issues of power and agency are embedded in participatory research with young people, it is important to achieve best practice . When researchers adopt the mind-set that “young people are creative agents who bring about change” [ ], participatory research can represent an important opportunity for young people to be recognized and contribute meaningfully.
Nature and Outcomes of the Design Process
Most studies indicated that consumer participation was integral to good intervention design and development [, , - , - , , - ]. Accessing consumers’ implicit domain knowledge was the cornerstone of producing relevant, accessible, and usable interventions and output, which is consistent with prior reviews of participatory research [ , ].
Consumer involvement was associated with flexibility, responsiveness, human-centeredness, and adaptability in design. For example, in their online adaptation of peer-based health promotion for adult men who have sex with men and same sex attracted young people, Hallett et al  engaged peer volunteers to develop and pilot the intervention. This allowed the project to be responsive and to adapt the intervention and its evaluation as needed. The peer volunteers provided important information regarding online etiquette and technical proficiency, and during piloting facilitated access to clients and development of rapport and credibility through use of shared language and cultural understandings.
Consumer collaboration significantly altered Owen et al’s  text-based self-harm prevention intervention from the original design brief. Researchers originally planned for a replication study in which generic texts were sent at predetermined, high-risk times; the co-design process resulted in a more flexible and human-centered design involving client self-authored texts accessible on demand. Authors noted that the final form and function of the intervention would not have been possible without consumer input.
Successful outcomes require researchers to balance consumer requirements against those of other stakeholders, such as funders and implementation sites, while managing time, resourcing, and ethical considerations. This difficult task requires careful negotiation along with clear and ongoing communication [, , , , ].
This is best exemplified by analysis of (1) an exit focus group with youth consumers; and (2) youth consumer-designer/researcher email conversations throughout the Elf et al  project. Analysis revealed that, as the project progressed, the mind-set of the researcher/designers changed from exploration of ideas with consumers to concrete production of output. This shift in priorities was attributed to increasing pressure around resources (eg, human, financial, time), and delivering technical components on time became the priority over implementing consumer ideas/suggestions.
Theory to Support Intervention Design
Consistent with prior literature, limited application of theory to guide technology development was evident . As a result, researchers are not maximizing the potential uptake, efficacy, and impact of their interventions. Three projects [ , , ] used heuristic guidelines to support technology-based intervention design and development. The guidelines emphasize design for outcomes, with mental health professionals, within a UCD framework [ , ]. Consideration of clinical validity, therapist and client usability, along with intervention acceptability, access, engagement, adaptability, and sustainability are also highlighted. Monshat et al [ ] was guided by constructs from TAM [ ]. Beyond this, theory or models with the ability to explain consumer interaction with the technology were absent.
While the literature is still developing, the behavioral intervention technology model  is an example of a model to guide the conceptual and technical architecture of behavior-changing eHealth and mHealth interventions—where eHealth is defined as “internet or other electronic media to disseminate health related information or services” [ ] and mHealth as “medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, PDAs, and other wireless devices” [ ]. The model guides researchers through development of clinical and usage aims, choice of technical elements and characteristics, and development of the intended workflow associated with the intervention. It assists in translating intervention aims into intervention elements and characteristics [ ].
eHealth participatory design best practice advocates for intended users as co-designers and partners in all phases of research, along with intervention evaluation criteria that balances youth relevance, meaning, and engagement with existing evidence . This type of theoretical integration is sorely needed in a field constrained by issues with uptake, adherence, and engagement [ - , ]. Furthermore, persuasive features that “reinforce, change, or shape attitudes or behaviors or both without using coercion or deception” [ ] and consumer motivation have had limited application in participatory technology-based mental health intervention design and, therefore, represents a focus of inquiry for future projects [ , , , ].
Planning for uptake and established connections with intervention sites were common to projects that successfully implemented their interventions or secured future funding [, , , , ]. Few projects reported evidence of inclusion of representatives from intended implementation sites in design and development of their interventions, even when accounting for the exploratory nature of some of the projects. A narrow definition of consumer may have led to limited representation of intervention site stakeholders in the intervention design phase.
Researchers need to be designing with an implementation site in mind and integrating influential system and organization level representatives into the process. In the case of treatment-focused interventions, mental health teams exist within larger systems that play an important role in acceptance and adoption of new interventions. Intimate knowledge of, and a strong working relationship with, the implementation sites of interest must be a priority of designer-researchers. Wolbling et al  argues that “ground-breaking ideas that arise within an existing organization that are not consistent with their values, routines, and overall strategy will be more difficult, if not impossible, to implement.” This assertion has clear implications for a research team wishing to implement new interventions from the outside. Organizational factors such as workplace ICT culture and policy and availability of resources have shown to be facilitators of uptake of ICT in health care [ ]. Whilst Coyle et al [ ] and Doherty et al [ ] account for individual therapist considerations in their heuristic guidelines, they fail to account for organizational and system level factors that can impact on intervention uptake and impact.
Designing with target consumers is crucial. The most commonly reported barriers to uptake of ICT in health care are design and technology concerns including lack of clinical relevance or impracticality; in addition lack of clinician time and perceived ICT skills are frequently reported barriers. On the flip side, facilitators of ICT uptake include system usefulness and functionality, clinical relevance and ease of use [, ]. This research indicates a clear role for application of theory to guide design and systematic consideration of human factors.
A limitation of this review was the broad inclusion criteria. This is particularly evident with respect to the Schmidt  project, which developed community health education outputs to explore youth conceptions of the relationship between the built environment and well-being. Whether these outputs can be categorized as interventions is debatable given the limited detail reported on the project. Despite the fact that youth participation was identifiable in the Owens et al [ ] paper, it did not have an exclusive youth focus. It was chosen for inclusion due to the nature of the project and its value in contributing to the aims of the review. In addition, the screening process may have benefited from involvement of a second reviewer to double screen. Evaluation of consumer representation was deemed too complex and broad to explore fully within this review beyond the description provided in the results table ( ). Finally, while every reasonable effort was made to find all relevant citations, the broad terminology used to describe the research in question may have resulted in some studies being overlooked, particularly where participatory processes may have been described in the methods sections of papers and not noted in the keywords, title, or abstract. Furthermore, the broad research field means the publication of some studies may not have been amenable to the titles, search terms, and databases that were used to construct this study and answer the research question. Moreover, participatory approaches are used in service settings but not always evaluated with the findings published and as such this work was not represented in the review. This review highlights the need for more research, evaluation, and publication on the use and outcomes of participatory approaches in the design and delivery of technology-based youth mental health services and interventions. The Young and Well Cooperative Research Centre (CRC) [ ] is an initiative that prioritizes this connection and creates the required space for the corresponding evidence base to be built.
Given the nascent stage of this field of research and the corresponding exploratory aims of this review, the broad nature of the search terms and studies included facilitated a wide-ranging description and analysis of participatory design and development of technology-based youth mental health and well-being interventions. This ensured that insights and learnings from the breadth of the mental health intervention spectrum were incorporated. The heterogeneous nature of the projects included, however, prevented the number of specific comparisons that could be made between similar projects and intervention types. We also wish to acknowledge that analysis and results of this review attempted to define and summarize a diverse and often ill-defined research field, and in doing so may have inadvertently oversimplified the practical application of participatory intervention design. Finally, in a rapidly evolving field, the search cutoff date meant that highly relevant recent projects found in conference abstracts were not included in the review.
The current review found limited evidence that consumer consultations lead to routine uptake of interventions in practice; that is, consumer participation does not act as a default implementation or uptake strategy. Overall, strategies aimed at increasing uptake of technology in health care practice are not well understood or reported. A consumerist rationale, which prioritizes acceptability and usability of the intervention, has characterized most projects in this field. It was clear that consumer involvement shaped intervention design in ways that were reported as beneficial by the designers/researchers. While consumer consultations were associated with flexibility, responsiveness, human-centeredness, and adaptability in design, it was not possible to determine the impact of this on intervention effectiveness due to lack of outcome data. The implications for why and how consumers are asked to participate in this field of research and the degree of mutual benefit that is possible, desired, and ethical requires rigorous examination. Participatory intervention design projects are advised to develop flexible and well-resourced project plans, which integrate theory and implementation within the design and make space for reflection, evaluation, and publication of consumer experience of research.
This research is part of a collaborative project supported by the Young and Well CRC and in partnership with Flinders University and Country Health SA. The Young and Well CRC is established under the Australian Government’s Cooperative Research Centres Program.
Conflicts of Interest
Multimedia Appendix 1
Systematic review search strategy.PDF File (Adobe PDF File), 501KB
- Slade T, Johnston A, Oakley Browne Mark A, Andrews G, Whiteford H. 2007 National Survey of Mental Health and Wellbeing: methods and key findings. Aust N Z J Psychiatry 2009 Jul;43(7):594-605. [CrossRef] [Medline]
- Kessler RC, Amminger GP, Aguilar-Gaxiola S, Alonso J, Lee S, Ustün TB. Age of onset of mental disorders: a review of recent literature. Curr Opin Psychiatry 2007 Jul;20(4):359-364 [FREE Full text] [CrossRef] [Medline]
- Burns JM, Davenport TA, Durkin LA, Luscombe GM, Hickie IB. The internet as a setting for mental health service utilisation by young people. Med J Aust 2010 Jun 7;192(11 Suppl):S22-S26. [Medline]
- Health D. mindhealthconnect. 2015. e-Mental Health Strategy URL: http://www.mindhealthconnect.org.au/emental-health-strategy [accessed 2015-01-08] [WebCite Cache]
- Clemensen J, Larsen SB, Kyng M, Kirkevold M. Participatory design in health sciences: Using cooperative experimental methods in developing health services and computer technology. Qual Health Res 2007 Jan;17(1):122-130. [CrossRef] [Medline]
- Coyle D, Doherty G, Matthews M, Sharry J. Computers in talk-based mental health interventions. Interacting with Computers 2007 Jul;19(4):545-562. [CrossRef]
- Coyle D, Doherty G. Clinical evaluations and collaborative design: developing new technologies for mental healthcare interventions. New York, USA: ACM; 2009 Presented at: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; April 4-9, 2009; Boston, USA p. 2051-2060. [CrossRef]
- Montague A, Varcin K, Parker A. Melbourne: Young and Well Cooperative Research Centre. 2014. Putting Technology into Practice: Evidence and opinions on integrating technology with youth health services URL: http://www.youngandwellcrc.org.au/wp-content/uploads/2014/10/Putting-Technology-into-Practice_FINAL.pdf [accessed 2015-01-08] [WebCite Cache]
- Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [CrossRef] [Medline]
- Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen Julia E W C. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012;14(6):e152 [FREE Full text] [CrossRef] [Medline]
- Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res 2009;11(2):e13 [FREE Full text] [CrossRef] [Medline]
- Christensen H, Mackinnon A. The law of attrition revisited. J Med Internet Res 2006;8(3):e20; author reply e21 [FREE Full text] [CrossRef] [Medline]
- Lillevoll KR, Vangberg Hans Christian B, Griffiths KM, Waterloo K, Eisemann MR. Uptake and adherence of a self-directed internet-based mental health intervention with tailored e-mail reminders in senior high schools in Norway. BMC Psychiatry 2014;14:14 [FREE Full text] [CrossRef] [Medline]
- Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res 2011;13(1):e30 [FREE Full text] [CrossRef] [Medline]
- Plattner H. Design School Stanford. Palo Alto Bootcamp Bootleg URL: http://dschool.stanford.edu/wp-content/uploads/2011/03/BootcampBootleg2010v2SLIM.pdf [accessed 2015-01-25] [WebCite Cache]
- Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry 2013 Aug;35(4):332-338 [FREE Full text] [CrossRef] [Medline]
- Mohr DC, Schueller SM, Montague E, Burns MN, Rashidi P. The behavioral intervention technology model: an integrated conceptual and technological framework for eHealth and mHealth interventions. J Med Internet Res 2014;16(6):e146 [FREE Full text] [CrossRef] [Medline]
- Doherty G, Coyle D, Matthews M. Design and evaluation guidelines for mental health technologies. Interacting with Computers 2010 Jul;22(4):243-252. [CrossRef]
- Hagen P, Collin P, Metcalf A, Nicholas M, Rahilly K, Swainston N. Young and Well Cooperative Research Centre. Melbourne; 2012. Participatory Design of evidence-based online youth mental health promotion, intervention and treatment URL: http://www.youngandwellcrc.org.au/wp-content/uploads/2014/03/Young_and_Well_CRC_IM_PD_Guide.pdf [accessed 2014-01-08] [WebCite Cache]
- Howe D, Batchelor S, Coates D, Cashman E. Nine key principles to guide youth mental health: development of service models in New South Wales. Early Interv Psychiatry 2014 May;8(2):190-197. [CrossRef] [Medline]
- Anderson JE, Lowen CA. Connecting youth with health services: Systematic review. Can Fam Physician 2010 Aug;56(8):778-784 [FREE Full text] [Medline]
- Ollner A. York University. Toronto, Ontario; 2010. Involving Youth in Research: A guide to the literature on participatory research with youth URL: http://www.yorku.ca/act/reports/InvolvingYouthInResearch.pdf [accessed 2015-01-26] [WebCite Cache]
- Boeck T, Collin P. Youth Adult Researcher Reflections on Participatory Research in Australia and the United Kingdom. In: Fleming J, Boeck T, editors. Involving Children and Young People in Health and Social Care Research. Oxford, UK: Routledge; 2012:197-208.
- Powers JL, Tiffany JS. Engaging youth in participatory research and evaluation. J Public Health Manag Pract 2006 Nov;Suppl:S79-S87. [Medline]
- Chen S, Poland B, Skinner HA. Youth voices: Evaluation of participatory action research. Canadian Journal of Program Evaluation 2007;22(1):125-150.
- Allen ML, Culhane-Pera KA, Pergament S, Call KT. A capacity building program to promote CBPR partnerships between academic researchers and community members. Clin Transl Sci 2011 Dec;4(6):428-433 [FREE Full text] [CrossRef] [Medline]
- Chambers DA, Pringle B, Juliano-Bult D. Connecting science and practice in child and adolescent mental health services research. Adm Policy Ment Health 2012 Jul;39(4):321-326. [CrossRef] [Medline]
- Cornwall A, Jewkes R. What is participatory research? Soc Sci Med 1995 Dec;41(12):1667-1676. [Medline]
- Gibbs L, Gold L, Kulkens M, Riggs E, Van GC, Waters E. Are the potential benefits of a community-based participatory approach to public health research worth the potential costs? Just policy: A journal of Australian social policy 2008;47:54-59 [FREE Full text]
- Alegría M, Wong Y, Mulvaney-Day N, Nillni A, Proctor E, Nickel M, et al. Community-based partnered research: new directions in mental health services research. Ethn Dis 2011;21(3 Suppl 1):S1-S8 [FREE Full text] [Medline]
- Szebeko D. Co-designing for Society. AMJ 2010 Sep 30;3(9):580-590. [CrossRef]
- Liamputtong P. Qualitative research methods. 4th ed. South Melbourne, Victoria: Oxford University Press; 2012.
- Wallerstein N, Duran B. The theoretical, historical, practice roots of CBPR. In: Community-based participatory research for health: From process to outcomes. 2nd ed. San Francisco: Jossey-Bass; 2008:25-46.
- Muller MJ, Kuhn S. Participatory Design. Communications of the ACM 1993;36(6):24-28 [FREE Full text]
- Sanders EB, Stappers PJ. Co-creation and the new landscapes of design. CoDesign 2008 Mar;4(1):5-18. [CrossRef]
- Owens C, Farrand P, Darvill R, Emmens T, Hewis E, Aitken P. Involving service users in intervention design: a participatory approach to developing a text-messaging intervention to reduce repetition of self-harm. Health Expect 2011 Sep;14(3):285-295. [CrossRef] [Medline]
- Boote J, Telford R, Cooper C. Consumer involvement in health research: a review and research agenda. Health Policy 2002 Aug;61(2):213-236. [Medline]
- James AM. Principles of youth participation in mental health services. Med J Aust 2007 Oct 1;187(7 Suppl):S57-S60. [Medline]
- eheadspace. 2015. URL: http://www.eheadspace.org.au/ [accessed 2015-02-01] [WebCite Cache]
- beyondblue. URL: http://www.beyondblue.org.au [accessed 2015-02-01] [WebCite Cache]
- ReachOut.com. 2015. URL: http://au.reachout.com/ [accessed 2015-02-01] [WebCite Cache]
- American Psychiatric Association. Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: American Psychiatric Association; 2000.
- Oliver SR, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect 2008 Mar;11(1):72-84. [CrossRef] [Medline]
- Lorenc T, Brunton G, Oliver S, Oliver K, Oakley A. Attitudes to walking and cycling among children, young people and parents: a systematic review. J Epidemiol Community Health 2008 Oct;62(10):852-857. [CrossRef] [Medline]
- Blackstock K, Kelly G, Horsey B. Developing and applying a framework to evaluate participatory research for sustainability. Ecological Economics 2007 Feb;60(4):726-742. [CrossRef]
- Biggs S. Resource-poor farmer participation in research: a synthesis of experiences from nine national agricultural research systems. In: OFCOR-comparative study paper; no. 3. The Hague, Netherlands: ISNAR; 1989.
- Arnstein S. JAIP; 35(4). 1969. A ladder of citizen participation URL: https://www.planning.org/pas/memo/2007/mar/pdf/JAPA35No4.pdf [accessed 2015-01-26] [WebCite Cache]
- Carroll J, Burge J, Robertson S, Rosson M. Participatory design of an autism community network to enhance community participation, health, and well-being. New York, USA: ACM; 2010 Presented at: Proceedings of the 1st ACM International Health Informatics Symposium; ACM; 2010; Washington, DC p. 415-419. [CrossRef]
- Ekberg J, Timpka T, Angbratt M, Frank L, Norén A, Hedin L, et al. Design of an online health-promoting community: negotiating user community needs with public health goals and service capabilities. BMC Health Serv Res 2013;13:258 [FREE Full text] [CrossRef] [Medline]
- Elf M, Rystedt H, Lundin J, Krevers B. Young carers as co-designers of a web-based support system: the views of two publics. Inform Health Soc Care 2012 Dec;37(4):203-216. [CrossRef] [Medline]
- Hallett J, Brown G, Maycock B, Langdon P. Changing communities, changing spaces: the challenges of health promotion outreach in cyberspace. Promot Educ 2007;14(3):150-154. [Medline]
- Løventoft P, Nørregaard L, Frøkjær E. Designing daybuilder: an experimental app to support people with depression. New York, USA: ACM; 2012 Presented at: Proceedings of the 12th Participatory Design Conference: Exploratory Papers, Workshop Descriptions, Industry Cases - Volume 2; August 2012; Roskilde, Denmark p. 1-4. [CrossRef]
- Madsen M, el Kaliouby R, Eckhardt M, Hoque M, Goodwin M, Picard R. Lessons from Participatory Design with Adolescents on the Autism Spectrum. New York, USA: ACM; 2009 Presented at: CHI â€™09 Extended Abstracts on Human Factors in Computing Systems; April 2009; Boston, USA p. 3835-3840. [CrossRef]
- Matthews M, Doherty G. In the mood: engaging teenagers in psychotherapy using mobile phones. New York, USA: ACM Press; 2011 Presented at: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; May 2011; Vancouver, BC, Canada p. 2011. [CrossRef]
- Mazzone E, Read JC, Beale R. Design with and for Disaffected Teenagers. New York, USA: ACM; 2008 Presented at: Proceedings of the 5th Nordic Conference on Human-Computer Interaction: Building Bridges; October, 2008; Lund, Sweden p. 290-297. [CrossRef]
- Moen A, Smørdal O. RareICT: a web-based resource to augment self-care and independence with a rare medical condition. Work 2012;41(3):329-337. [CrossRef] [Medline]
- Monshat K, Vella-Brodrick D, Burns J, Herrman H. Mental health promotion in the Internet age: a consultation with Australian young people to inform the design of an online mindfulness training programme. Health Promot Int 2012 Jun;27(2):177-186 [FREE Full text] [CrossRef] [Medline]
- Lakey T. Project 99: Exploring internet based approaches to support youth mental health in Greater Glasgow & Clyde. UK: Greater Glasgow and Clyde NHS; 2014. URL: http://www.wegot99.com/wp-content/uploads/2013/12/p_wegot99_codesignoutcomes1.pdf [accessed 2015-01-08] [WebCite Cache]
- Schmidt EE. University of California, Davis. 2009. Engaging youth in community change: A reflection of the youth participatory project, Youth Voices for Change URL: http://search.proquest.com/barcode?accountid=10910&groupid=1007856 [accessed 2015-06-21] [WebCite Cache]
- Stewart S, Riecken T, Scott T, Tanaka M, Riecken J. Expanding health literacy: indigenous youth creating videos. J Health Psychol 2008 Mar;13(2):180-189. [CrossRef] [Medline]
- Valaitis R, O’Mara L, Bezaire S. Produced by Huron County Health Unit and McMaster University, School of Nursing. 2007. Tailoring online health promotion: a toolkit for communities URL: http://fhs.mcmaster.ca/OHPToolkit/Content/Toolkit_Printer_Friendly.pdf [accessed 2015-01-08] [WebCite Cache]
- Wadley G, Lederman R, Gleeson J, Alvarez-Jimenez M. Participatory design of an online therapy for youth mental health. In: Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration. New York, USA: ACM; 2013 Presented at: 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration; Nov 25-Nov 29, 2013; Adelaide, Australia p. 517-526. [CrossRef]
- Blythe MA, Wright PC. Pastiche scenarios: Fiction as a resource for user centred design. Interacting with Computers 2006 Sep;18(5):1139-1164. [CrossRef]
- Davis FD. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly 1989 Sep;13(3):319. [CrossRef]
- Bødker S. Scenarios in user-centred design—setting the stage for reflection and action. Interacting with Computers 2000 Sep;13(1):61-75. [CrossRef]
- MacLean A, Young RM, Moran TP. Design rationale: the argument behind the artifact. ACM SIGCHI Bulletin 1989 Mar 01;20(SI):247-252. [CrossRef]
- Birru MS, Monaco VM, Charles L, Drew H, Njie V, Bierria T, et al. Internet usage by low-literacy adults seeking health information: an observational analysis. J Med Internet Res 2004 Sep 3;6(3):e25 [FREE Full text] [CrossRef] [Medline]
- Hansen DL, Derry HA, Resnick PJ, Richardson CR. Adolescents searching for health information on the Internet: an observational study. J Med Internet Res 2003 Oct 17;5(4):e25 [FREE Full text] [CrossRef] [Medline]
- Spinuzzi C. The methodology of participatory design. Tech Commun 2005;52(2):163-174 [FREE Full text]
- Cargo M, Mercer SL. The value and challenges of participatory research: strengthening its practice. Annu Rev Public Health 2008;29:325-350. [CrossRef] [Medline]
- Jagosh J, Macaulay AC, Pluye P, Salsberg J, Bush PL, Henderson J, et al. Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q 2012 Jun;90(2):311-346 [FREE Full text] [CrossRef] [Medline]
- Bengtsson-Tops A, Svensson B. Mental health users' experiences of being interviewed by another user in a research project. A qualitative study. J Ment Health 2010 Jun;19(3):237-242. [CrossRef] [Medline]
- Dold CJ, Chapman RA. Hearing a Voice: Results of a Participatory Action Research Study. J Child Fam Stud 2011 May 20;21(3):512-519. [CrossRef]
- Kraemer Diaz Anne E, Spears Johnson Chaya R, Arcury TA. Variation in the interpretation of scientific integrity in community-based participatory health research. Soc Sci Med 2013 Nov;97:134-142 [FREE Full text] [CrossRef] [Medline]
- Moltu C, Stefansen J, Svisdahl M, Veseth M. Negotiating the coresearcher mandate - service users' experiences of doing collaborative research on mental health. Disabil Rehabil 2012;34(19):1608-1616. [CrossRef] [Medline]
- Delman J. Participatory action research and young adults with psychiatric disabilities. Psychiatr Rehabil J 2012;35(3):231-234. [CrossRef] [Medline]
- O'Fallon LR, Dearry A. Community-based participatory research as a tool to advance environmental health sciences. Environ Health Perspect 2002 Apr;110 Suppl 2:155-159 [FREE Full text] [Medline]
- Gustafson DH, Wyatt JC. Evaluation of ehealth systems and services. BMJ 2004 May 15;328(7449):1150 [FREE Full text] [CrossRef] [Medline]
- Kay M. Geneva: World Health Organization. 2011. mHealth: New horizons for health through mobile technologies URL: http://www.who.int/goe/publications/goe_mhealth_web.pdf [accessed 2015-01-26] [WebCite Cache]
- Oinas-Kukkonen H, Harjumaa M. Towards deeper understanding of persuasion in software and information systems. Washington, DC, USA: IEEE Computer Society; 2008 Presented at: Advances in Computer-Human Interaction, 2008 First International Conference; February 2008; Sainte Luce, Martinique p. 200-205. [CrossRef]
- Fogg B. A behavior model for persuasive design. New York, USA: ACM; 2009 Presented at: Proceedings of the 4th international Conference on Persuasive Technology; April 2009; Claremont, CA, USA p. 40 URL: http://bjfogg.com/fbm_files/page4_1.pdf
- Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol 2000 Jan;55(1):68-78. [Medline]
- Wölbling A, Krämer K, Buss C, Dribbisch K, LoBue P, Taherivand A. Design Thinking: An Innovative Concept for Developing User-Centered Software. In: Maedche A, Botzenhardt A. Neer L editors. Software for People: Fundamentals, Trends and Best Practices. New York: Springer Berlin Heidelberg; 2012:978-973.
- Gagnon M, Desmartis M, Labrecque M, Car J, Pagliari C, Pluye P, et al. Systematic review of factors influencing the adoption of information and communication technologies by healthcare professionals. J Med Syst 2012 Feb;36(1):241-277 [FREE Full text] [CrossRef] [Medline]
- Young and Well Cooperative Research Centre. URL: http://www.youngandwellcrc.org.au/ [accessed 2015-06-20] [WebCite Cache]
|ICT: information and communications technology|
|iSET: interactive socio-emotional toolkit|
|CBPR: community based participatory research|
|CRC: Cooperative Research Centre|
|MSM: men who have sex with men|
|NHMRC: National Health and Medical Research Council|
|OHPC: online health-promoting community|
|PAR: participatory action research|
|PD: participatory design|
|RCT: randomized controlled trial|
|SSAY: same sex attracted young people|
|TAM: technology acceptance model|
|UCD: user-centered design|
|WAAC: West Australian Aids Council|
|WBSS: Web-based support system|
|YVC: Youth Voices for Change|
Edited by G Eysenbach; submitted 20.02.15; peer-reviewed by O Zelenko; comments to author 29.04.15; revised version received 25.05.15; accepted 27.05.15; published 09.07.15
©Simone Kate Orlowski, Sharon Lawn, Anthony Venning, Megan Winsall, Gabrielle M Jones, Kaisha Wyld, Raechel A Damarell, Gaston Antezana, Geoffrey Schrader, David Smith, Philippa Collin, Niranjan Bidargaddi. Originally published in JMIR Human Factors (http://humanfactors.jmir.org), 09.07.2015.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on http://humanfactors.jmir.org, as well as this copyright and license information must be included.