Published on in Vol 9, No 3 (2022): Jul-Sep

Preprints (earlier versions) of this paper are available at, first published .
Implementation of Smartphone Systems to Improve Quality of Life for People With Substance Use Disorder: Interim Report on a Randomized Controlled Trial

Implementation of Smartphone Systems to Improve Quality of Life for People With Substance Use Disorder: Interim Report on a Randomized Controlled Trial

Implementation of Smartphone Systems to Improve Quality of Life for People With Substance Use Disorder: Interim Report on a Randomized Controlled Trial

Original Paper

1Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI, United States

2Institute for Clinical and Translational Research, University of Wisconsin-Madison, Madison, WI, United States

3School of Nursing, University of Wisconsin-Madison, Madison, WI, United States

Corresponding Author:

David Gustafson Sr, PhD

Department of Industrial and Systems Engineering

University of Wisconsin-Madison

1513 University Ave

Madison, WI, 53706

United States

Phone: 1 608 239 5535


Background: Researchers have conducted numerous studies seeking to understand how to improve the implementation of changes in health care organizations, but less focus has been given to applying lessons already learned from implementation science. Finding innovative ways to apply these findings efficiently and consistently will improve current research on implementation strategies and allow organizations utilizing these techniques to make changes more effectively.

Objective: This research aims to compare a practical implementation approach that uses principles from prior implementation studies to more traditional ways of implementing change.

Methods: A total of 43 addiction treatment sites in Iowa were randomly assigned to 2 different implementation strategies in a randomized comparative effectiveness trial studying the implementation of an eHealth substance use disorder treatment technology. One strategy used an adaptation of the Network for the Improvement of Addiction Treatment (NIATx) improvement approach, while the other used a traditional product training model. This paper discusses lessons learned about implementation.

Results: This midterm report indicates that use of the NIATx approach appears to be leading to improved outcomes on several measures, including initial and sustained use of new technology by both counselors and patients. Additionally, this research indicates that seamlessly integrating organizational changes into existing workflows and using coaching to overcome hurdles and assess progress are important to improve implementation projects.

Conclusions: At this interim point in the study, it appears that the use of the NIATx improvement process leads to better outcomes in implementation of changes within health care organizations. Moreover, some strategies used in this improvement process are particularly useful and should be drawn on more heavily in future implementation efforts.

Trial Registration: NCT03954184;

JMIR Hum Factors 2022;9(3):e35125



This is a midterm report on a randomized trial comparing 2 different strategies for implementing innovations. Implementation and dissemination are ongoing challenges for innovations. While efforts to enhance theories in implementation science theory may help, we need to shift focus more to applying what we already know with efficiency and fidelity. We developed a simple and easily replicable implementation approach and are comparing it to a more traditional approach.

The challenges of implementation have been studied for hundreds (some would say thousands) of years by innovators such as Jethro of Midian [1], Heraclitus [2], Taylor [3], Box [4], Ishikawa [5], Batalden [6], Maslow [7], Phillips [8], Gantt [9], Deming [10], Delbecq [11], Mayo [12], Ohno [13], Maidque [14], Conner [15], Shewhart [16], Utterback [17], Fayol [18], Rogers [19], Van de Ven [20], Barnard [21], Barnett [22], Kanter [23], Gawande [24], Lewin [25], Cooper [26], Berwick [27], Argyris [28], Simon [29], Taguchi [30], Kotter [31], Hage [32], Kilbourne [33], Ackerman [34], and Damschroder [35].

A personal example may help elucidate why we cannot wait for more theory: Tim was 31 years old when he died alone in his room with a half-used syringe on his nightstand. He had fought opioid use disorder for almost half of his life. Ultimately, an act of kindness led to his demise. The last years of his life were among Tim’s best. He was in successful recovery. He had been clean and sober for nearly 5 years, reunited with his family, and gotten a job. He was 2 months away from getting his bachelor’s degree in brain and behavior studies and was planning to pursue a master’s degree. Both Tim’s dad (his favorite golfing partner) and his mom (the rock of the family) were hesitantly breathing a sigh of relief. Things were going well. But there was also a warning sign. Tim had stopped using Suboxone (a medication-assisted treatment designed to reduce the desire for opioids) because of side effects (terrible constipation and plummeting libido).

Then a friend of Tim’s called him from the hospital—her husband had survived an overdose and would soon be discharged. She knew he had heroin in their apartment. Would Tim search for it and clear the apartment before her husband’s return? Tim’s kindness drove him; he agreed to help. He sped to the hospital, got their key, returned to their apartment, and removed what he could find. Within hours of having heroin in his possession, for the first time in almost 5 years, Tim relapsed. He died from what was ultimately determined to be a “speedball” (a mixture of heroin and cocaine). Tim’s death devastated his family and friends. One of the authors of this paper attended the funeral. He sat next to a young person who was crying; we all were. The young person said, “Tim was my hope. If he couldn’t make it, how can I?” Tim had been trained in evidence-based cognitive behavioral therapy (CBT). One of CBT’s key principles relates to “seemingly irrelevant decisions” where one puts themselves in harm’s way without realizing it. Tim forgot that principle, and it killed him.

Computer-based technologies that could have helped Tim have been in place since the early 1980s. Examples include the Body Awareness Resources Network (BARN) [36], Computer Based Training for Cognitive Behavioral Therapy (CBT4CBT) [37], Treatment Evaluation Services (TES) [38], and Addiction Comprehensive Health Enhancement Support System (ACHESS) [39]. For instance, ACHESS contains brief reminders of CBT principles that might have called Tim’s seemingly irrelevant decision—to help his friend—to his attention. Alternatively, the weekly patient surveys in ACHESS might have called his attention to his increased anxiety and led him to think twice about his decision. However, few people with or without a substance use disorder [40] remember the CBT skills they have learned in treatment or have access to tools that could help them remember in the moments when those skills are most needed.

ACHESS, the original smartphone innovation that we used to compare implementation strategies, was designed to help counselors and patients. It offers a variety of services: communicate anonymously with peer support groups, help assess a patient’s relapse risk and link to interventions, use reminders to encourage adherence to therapeutic goals, privately communicate with the patient's counselor, provide addiction-related educational materials and tools, and send alerts if a patient visits a high-risk location such as a favorite bar.

Several studies found that ACHESS reduced heavy drinking [41] and doubled retention in treatment [42]. However, only a few thousand of the millions of people facing substance use disorder (SUD) are using ACHESS or other technologically based systems. Hence, such a system provides an ideal target to compare implementation strategies. For this study, ACHESS was renamed RISE Iowa (Recovering Iowans Supporting Each Other) to make the app more appealing to treatment agencies in Iowa (Figure 1).

This paper seeks to answer the question, “What have we learned so far in this study about how to implement evidence-based practices?” Our ongoing randomized trial compares 2 strategies for widescale implementation, using RISE Iowa as the object of those implementation strategies.

Figure 1. RISE Iowa (Recovering Iowans Supporting Each Other) app.
View this figure


The first strategy is a typical product training approach where a sales representative introduces a product to key personnel in the adopting organization, trains key players on how to use the product, and offers a source for further support, such as training manual and computer accessible responses to frequently asked questions (FAQs). The second strategy adds a quality improvement (QI) methodology (the NIATx model [43]) that assigns an external coach who calls the agency once a month to monitor, support, and encourage the organization and uses a set of QI tools (eg, a checklist of steps to implement RISE Iowa at their site, flowchart to see how to integrate RISE Iowa into the organization’s workflow, and tools to predict and explain an organization’s readiness for change or to examine the potential of embedding the innovation). The NIATx model has been implemented by over 3500 addiction treatment organizations and was tested in a randomized trial involving over 200 addiction treatment agencies [44]. NIATx is designed to add structure and increase fidelity to the implementation process.

In the NIATx model, quality improvement coaches made 1 trip to each agency to get to know and train staff in RISE Iowa and to set implementation goals, followed by monthly Zoom calls to an organization’s change leader and change team. Coaches’ calls served as methods to assess progress and roadblocks, train and remind staff, celebrate successes, give feedback on progress, set follow-up goals, and identify and provide answers to questions and concerns. In addition, 4 times during the 18-month implementation period, organizations in the NIATx model were invited to convene via Zoom for more training and to share successes and challenges. Table 1 compares NIATx to the product training approach.

Table 1. Comparison of implementation strategies.
NIATxaProduct training
Coach and trainer conduct Zoom meeting with leadership of organization to review study and provide overview of RISEb Iowa app.Trainer conducts Zoom meeting with leadership of organization to review study and provide overview of RISE Iowa app.
Staff at organizational sites take survey regarding organization’s approach to change.Staff at organizational sites take survey regarding organization’s approach to change.
Coach and trainer conduct Zoom meeting with change leader(s) and teams to review survey results and preview app.N/Ac
Trainer provides 2-hour staff training on the RISE Iowa app.Trainer provides 2-hour staff training on the RISE Iowa app.
Coach provides 2-hour NIATx training.N/A
Tech support is available via email.Tech support is available via email.
Coach and trainer hold monthly coaching Zoom meeting with change leaders at study sites, in which coach shares ideas gained from working with other organizations.N/A
Study staff emails staff and patients with RISE Iowa accounts regarding updates to the app.Study staff emails staff and patients with RISE Iowa accounts regarding updates to the app.
Study staff send weekly and monthly emails providing data on new RISE Iowa accounts and RISE Iowa usage by staff and patients to change teams.N/A
Additional resources are added to the app approximately monthly.Additional resources are added to the app approximately monthly.
One additional cross-agency Zoom meeting is held with executive sponsors as well as 1 with change leaders. Additionally, 2 additional cross-agency training opportunities are offeredN/A

aNIATx: Network for the Improvement of Addiction Treatment.

bRISE: Recovering Iowans Supporting Each Other.

cN/A: Not applicable.

Coaches are individuals with experience in leading NIATx projects in their own organizations and who have received additional NIATx coaching training. Coaches have had at least 10 years of experience leading NIATx quality improvement projects. To assure fidelity and consistency among coaches, the coaches convened monthly with key study team members to review progress with the sites and discuss challenges and potential approaches to those challenges. Due to COVID-19 restrictions, coaches were unable to make in-person visits to 3 of the 11 agencies. Those agencies were trained via Zoom. A resource provided in the NIATx approach is a set of organizational surveys used to predict and explain the likelihood of successful RISE Iowa implementation [45] and assess chances that RISE Iowa will be sustained by the organization in the long run [46]. These analyses are designed to help coaches determine the strengths and challenges they will face in promoting adoption and sustainment at the agency and offer advice on how to overcome the challenges.

This paper presents interim results of our ongoing multicenter randomized trial in Iowa SUD treatment agencies but focuses mostly on lessons learned so far about implementation. No attempt was made to evaluate the RISE Iowa app itself. We are interested in implementation progress with NIATx compared to the product training approach. We expected the NIATx coaching calls alone would be superior, but we also wanted to explore what other aspects of the NIATx approach make a difference.

To compare the 2 implementation strategies, 11 Iowa-based addiction treatment organizations with 43 addiction treatment sites were randomly assigned to receive the product training or the NIATx approach. The sites’ progress is being tracked for an 18-month period during which aspects of the 2 implementation strategies are still active (eg, FAQs for product training, monthly coach calls for NIATx), followed by 10 months with no support to examine sustainability. The first cohort began the study intervention in 2019. The final cohort will complete the intervention by mid-2022.

Counselors and peer recovery coaches use RISE Iowa in a variety of ways, including by assigning material within it (eg, reading personal stories of others successfully struggling with SUD), monitoring patient progress by examining the weekly patient surveys, and participating in discussion groups. RISE Iowa automatically collects and stores data on how many patients sign up to use RISE Iowa and the amount of use by both counselors and patients. Counselors can track the data on patient use of RISE Iowa to better understand how well patients are following through on their recovery efforts.

In this report, we compare patient and counselor use of RISE Iowa between sites assigned to the NIATx approach versus product training only. RISE Iowa utilization is summarized as the average number of days logging into RISE Iowa per month and the percentage of patients or counselors who logged in per month.

To better understand the features of the NIATx approach that influenced implementation in this study, 3 coaches and 8 research staff conducted a combination of semistructured interviews and nominal group technique meetings [11,47] to identify and assess factors that played a role in the implementation and ones they would concentrate on if they were to implement RISE Iowa in another setting (or, in other words, what worked). These interviews identified 24 factors that were then evaluated by inviting coaches and research staff to select the 10 factors they considered to be most important in guiding adoption and sustainment. We prioritized these factors by counting the votes (from 11 possible voters) of each of the 24 factors received. We present the 10 factors that received the most votes alongside explanations provided by voters and summarize the number of votes received for the remaining 14 factors. We did not conduct a formal statistical analysis as this is preliminary data.

Ethics Approval

This study received approval from Advarra Institutional Review Board (# 2018-0997). Interview participants provided oral informed consent.

Use of RISE Iowa

Use of the RISE Iowa app was tracked over time for all participants. Here, we present the data collected so far for the first 5 months of each participants’ app usage after activating their account. The average number of days per months of app usage was calculated by tallying the number of days per month each participant opened the app and then calculating the mean of all tallies. The percent of active participants per month was calculated by dividing the number of participants who opened the app that month by the number of participants who had access to the app at that time point.

Figure 2 compares counselor retention rates between the implementation approaches. By retention, we mean that counselors continue to log on to RISE Iowa after they are first introduced to it. For example, 24 of 68 (35%) of counselors and peer recovery coaches trained in RISE Iowa with the NIATx approach were still logging in 5 months later versus 2 of 51 (4%) who received product training. Furthermore, those 35% used RISE Iowa an average of about 4 days per week with NIATx versus an average of about 1.5 days per week in the product training arm. For the patient data, an account was removed from the analysis if it had been created on the day the analysis was run. For example, if person A created an account at 10 AM on day X and study staff downloaded the data at 4 PM on Day X, person A would have created an account the day that study staff downloaded the data, so person A would be removed from the analysis.

Figure 2. Counselors' average days of use of RISE Iowa (Recovering Iowans Supporting Each Other) app. NIATx: Network for the Improvement of Addiction Treatment.
View this figure

Figure 3 displays differences in patients’ use of RISE Iowa between the product training and NIATx approaches. The average days of RISE Iowa use per patient per month was about 6.5 at the NIATx sites versus about 5.5 in product training sites. Further, the number of patients using RISE Iowa is much smaller in the product training approach (11/81,14%) versus in the NIATx approach (150/722, 21%) at 5 months after each patient first logged on. Finally, it should be noted that the product training sites in this study treat 17% more patients than do the NIATx sites; however, there was a much smaller RISE Iowa enrollment in the product training locations. Table 2 shows the descriptive statistics for use of the RISE Iowa app by participants.

Figure 3. Patients' average days of use of RISE Iowa (Recovering Iowans Supporting Each Other) app. NIATx: Network for the Improvement of Addiction Treatment.
View this figure
Table 2. Descriptive statistics for use of the RISE Iowa app by participants.
Type of RISEa Iowa userArmStudy monthNumber of days per month, mean (SD)People, nPeople possible, nPeople active, %
PatientsProduct training16.05 (5.53)8282100
PatientsProduct training24.53 (5.31)388147
PatientsProduct training34.11 (3.61)188122
PatientsProduct training45 (8.27)158119
PatientsProduct training55.36 (8.58)118114
PatientsNIATxb17.02 (6.25)79279799
PatientsNIATx25.83 (6.74)40278951
PatientsNIATx35.24 (6.34)27676336
PatientsNIATx45.79 (7.55)19374926
PatientsNIATx55.92 (7.54)15072221
CounselorsProduct training11.81 (1.45)475192
CounselorsProduct training22 (1.41)4518
CounselorsProduct training31.86 (1.46)75114
CounselorsProduct training41 (0)3516
CounselorsProduct training51 (0)2514
CounselorsNIATx13.67 (2.94)697099
CounselorsNIATx23.87 (3.42)307043
CounselorsNIATx34.62 (5.28)267037
CounselorsNIATx43.83 (5.77)237033
CounselorsNIATx53.92 (4.80)246835

aRISE: Recovering Iowans Supporting Each Other.

bNIATx: Network for the Improvement of Addiction Treatment.

Factors Influencing Implementation

We explored what it is about the NIATx approach that is thus far leading to better results. Ordered by total number of votes received, the top factors are (including the number of votes each received) are as follows.

First, identify patient and counselor pain points and show how RISE Iowa helps to relieve that pain: 10 votes out of a possible 11. This factor creates a value proposition on why the change should be adopted. As a counselor put it, “From the beginning, I’ve really tried to make [RISE Iowa] something that will feel useful. I am trying to have our IT [person] put as many relevant things on RISE Iowa as possible [to keep it fresh].” Counselor pain points may include busyness, demands of their job, and having to accomplish several required activities with each patient. Patient pain points include factors that inhibit recovery, such as cravings, triggers, and isolation.

Second, find ways to integrate RISE Iowa into the organization’s standard workflow: 9 votes. Another counselor said, “I've been more intentional about going through my case load prior to the monthly coaching calls asking: ‘Who are the eligible folks and why aren't they signed up with RISE Iowa yet?’. That's a trigger for me to say, ‘Here are the folks that I need to target.’“ As such, the counselor has made RISE Iowa enrollments part of their standard workflow.

Third, use coaches to motivate, help overcome hurdles, assess progress, and share learnings across the sites: 8 votes. A coach said, “Clinicians are asked to manage many competing demands as they assist patients in their recovery. Adding a tool [like RISE Iowa] to their workflow takes mental energy and focus. Coaches trained agency staff in NIATx process improvement and the associated tools. With coaches available, this means that providers do not need to have all the answers as they pilot test improvements in their workflows.”

Fourth, get senior executives to encourage agency staff to try RISE Iowa: 8 votes. Another coach said, “Leaders can encourage staff to make RISE Iowa part of their regular client intake and treatment process. Leaders can also free staff time to use RISE Iowa themselves and become familiar with its features.”

Fifth, get bugs out of the RISE Iowa app and make it user friendly: 7 votes. A researcher said. “I just know that nothing is more of a deterrent than trying to use something that doesn’t work.”

Sixth, provide data and administrative support: 6 votes. A coach said, “Administrative staff offer real-time technical support and training to agencies, as well as data on RISE Iowa adoption and use. It was easier for organizations to justify the allocation of limited resources to support sustainment after staff provided data demonstrating measurable progress.”

Seventh, use testimonials to reinforce the value of RISE Iowa: 5 votes. A coach said, “In my 20 years of facilitating organizational change, stories/testimonials are among the most impactful resources. To hear a story from someone you identify with can provide a path for change and a belief that a change might be worth the effort.”

Eighth, have cross-agency calls to share learnings and concerns: 5 votes. A research staff member noted that “cross agency calls provide a way for counselors and agency staff to learn from each other. Nothing works better than to hear good things from a colleague.”

Ninth, use financial incentives: 5 votes. A researcher noted that “incentives (such as lotteries) can motivate people toward chosen behaviors. Incentives can tip the scales away from competing priorities.”

Tenth, remind and retrain both counselors and patients to use RISE Iowa: 5 votes. One counselor said, “The 'coping with cravings’ stuff; when I first heard about that, I thought, ‘this is beautiful!’ But that had not been [an] area that I really tapped into. Its presence makes me think of a lot of new things that I could be doing with RISE Iowa.”

Other factors and the number of votes they received (indicated in parenthesis) are listed as follows:

  1. Assess organizational readiness for change to learn what areas they need to work on to have the best chance of successful implementation (4).
  2. Get many patients on RISE Iowa right away to reach critical mass for discussion groups because there needs to be enough people to have an active discussion (4).
  3. Find ways to address the digital divide (4).
  4. Give agencies tools they can use to improve quality (4).
  5. Protect privacy (3).
  6. Celebrate successes (2).
  7. Make access to RISE Iowa a privilege. Making people invest to participate makes a system more appealing (1).
  8. Do not prejudge who will use RISE Iowa; clinicians may pick and choose which patients are best suited for a recovery app and may misjudge (1).
  9. Use lessons from the COVID-19 pandemic. For instance, virtual visits were found to be effective (1).
  10. Coaches should make 1 in-person visit if possible (1).
  11. Refresh RISE Iowa often to keep it interesting and up to date (1).
  12. Make RISE Iowa free of charge (1).
  13. Ignore skeptics; however, ignoring skeptics would probably be a mistake because they likely have important insights to offer (0).
  14. Do not use inflammatory words, such as “track.” The “recovery tracker” on the app could be seen as an inflammatory term because people do not like to feel like they are being watched (0).

Interim Findings

This study’s interim results suggest that the NIATx process is especially useful for integrating RISE Iowa within existing workflows and helping clinicians address the challenges they already face at work. It is further notable that in an earlier randomized trial [48] involving 201 addiction treatment agencies across 4 states, we explored whether processes improved more in agencies that received the product training approach versus a full improvement collaborative (ie, periodic face-to-face meetings, coaches, joint calls with all agencies at once) versus joint improvement calls only versus monthly coaching calls with individual agencies, with the latter method used in this study. We found that monthly coaching calls (the NIATx approach) worked at least as well as the full collaborative and better on both cost and effectiveness outcomes than the other arms.

The results of this study point to the importance of identifying and responding to the pain points faced by the implementing organization, along with finding a way to integrate the new technology easily into the organization’s workflow. The more that staff are asked to change their workflow, the less likely the implementation will be to succeed.

Other research finds that the average commercial app loses 77% of its users within the first 3 days after installation. Within 30 days, it has lost 90% of users. Within 90 days, it has lost over 99% [49]. In contrast, in earlier tests of ACHESS in Federally Qualified Health Centers, over 60% of actively enrolled people continued using ACHESS 4 months after enrolling [44].

Therefore, we were surprised at this study’s retention rate of only 22% at 5 months. Multiple reasons may have influenced this higher-than-expected dropout rate. The first is that in prior studies, we typically delayed offering ACHESS (system on which RISE Iowa is based) until the patient demonstrated their commitment to recovery by attending at least 3 clinic visits. In this study, many clinics offered RISE Iowa at the first visit. As it turns out, the dropout rate from treatment after the first visit is high, typically about 30% to 50% in the first month [50,51]. Hence, by the time we offered ACHESS in prior studies, only 50% to 70% of the initial patients remained in treatment, and the retention rate among that 50% to 70% of the original population was closer to 28% to 38%. Second, 25% of Iowa residents live in rural areas with no or inadequate access to the internet [52], moving retention rates of potential patients closer to 50% [53]. This study was also conducted during the COVID-19 pandemic, which could have had varied and unpredictable influences on patient retention and on counselors’ ability to introduce patients to the app. Considering these factors, a lower retention rate in this interim analysis is not surprising compared to previous studies of ACHESS.

As the voting results demonstrate, we value the use of financial incentives [54] to implement and sustain innovations. However, our counselors and clinic staff felt that incentives would not be their first choice, if given a choice. When we raised with agencies the option to incentivize counselors and patients with our own grant money, they rejected the option in most cases. Moreover, substantial literature supports the importance of deeply understanding our customers (patients, agencies, and staff) above all [55]. When designing RISE Iowa, we employed critical incident interviews [56] and conducted walk-throughs [57] to understand what it is like to be a patient, a counselor, and an agency administrator. This effort helped us create a value proposition that makes it in the agency’s best interest to adopt and use the technology. These efforts resulted in our team simplifying RISE Iowa adoption and ways to use it as much as possible and understanding where RISE Iowa fits in the workflow to make it as easy to adopt as possible. As one counselor said, “I am so busy that if I have to so much as lift a finger to use RISE Iowa, I won’t do it.”

NIATX coaches provided strategic support to the agencies’ implementation efforts. During their monthly phone sessions, coaches attempted to build motivation to use RISE Iowa by stressing how RISE Iowa can help address “pain” points. Coaches encouraged executives to support RISE Iowa, addressed resistance, and removed barriers while stressing the need to build care delivery systems that integrate RISE Iowa into the clinical and administrative workflows. Coaches also made their organizations aware of evidence-based practices, such as contingency management and motivational interviewing, which might facilitate RISE Iowa use. Overall, with just 1 phone call per month, coaches helped organizations over the rough patches and provided persistent motivational support to expand RISE Iowa use.


This report has several limitations. First, these are preliminary findings during the intervention phase of the trial. Therefore, our results do not include the sustainability of postintervention RISE Iowa utilization. However, the primary purpose of the paper is to delve more deeply into reasons why a process like NIATx would be superior, not a definitive response on the superiority of the intervention. Second, the qualitative results do not include patient perspectives. This study focused on organization implementation [58] aspects of RISE Iowa app use. Accordingly, these results are from clinician, organizational leader, and coach perspectives. Future studies should also include patient perspectives. The people we did include bring an important perspective that needs to be understood. Third, this study was not designed to address the effectiveness of the RISE Iowa app. Past research addressed this issue [44,59]. Fourth, while there were 43 different sites in this study, there were only 11 different organizations. While most of the sites operated independently of their senior leadership, there were times when corporate policy limited independence. In that sense, the number of truly independent units is smaller than might initially appear. Finally, there was a site dropout rate of 7%, and the study design had a planned dropout rate of 20%. We find this to be a reasonable rate due to the impact that personnel changes may had have on needed participation in the research trial. In addition, the organization that dropped out was replaced with an organization with similar characteristics used for randomization.


As mentioned earlier, this interim report describes what we have found so far. The results imply that the NIATx approach, along with the use of coaching, led to higher adoption by both patients and counselors, but we are awaiting final results, which are approximately 6 months away. After receiving our final results, we will seek to understand more than we do today, particularly about how well RISE Iowa was sustained in this project.

We hope readers will find this interim report valuable. Finally, we want to reinforce our belief that finding easy-to-use tools to reliably implement evidence-based practice is more important today than adding to theory. People need help now.


This research was supported by the National Institute on Drug Abuse (NIDA) of the National Institutes of Health (NIH) (award # R01DA044159). The contents of this paper are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health. We gratefully acknowledge Maureen Fitzgerald for her editing services and Judith Ganch for her assistance with the references for this manuscript.

Data Availability

The data sets generated and analyzed in this study are not publicly available due to the complexity of app use data and personal information embedded in the data set, but they are available from the corresponding author on reasonable request.

Authors' Contributions

DG is the main preparer of this manuscript. TM and DG are the principal investigators; they secured funding, developed the initial project framework, and provided input for the project implementation. JH, KF, and OC assisted with data collection and contributed to data analysis. DB, MR, and JR provided coaching to the organizations in this study and helped write the paper. NJ, RG, and TS also helped write paper. All authors reviewed the paper before submission.

Conflicts of Interest

DG is a part-owner of CHESS Health, which is devoted to marketing information technologies to agencies that deliver addiction treatment. He is also on the board of directors of the nonprofit NIATx Foundation, as well as a small consulting company. These relationships do not carry any restrictions on publication, and any associated intellectual property will be disclosed and processed according to his institution’s policies. TM is a faculty member at CHESS and has a less than .1% ownership with CHESS Health, the organization responsible for making the ACHESS addiction recovery app commercially available to the public. TM has worked extensively with his institution to manage any conflicts of interest. An external advisory board approved all survey instruments applied, and the individuals who will conduct the data collection and interpretation for this study will have no affiliation with CHESS Health. Additionally, parts of the NIATx organizational change model used in part of this trial were developed by the Center for Health Enhancement System Studies (CHESS) at the University of Wisconsin-Madison, where TM is a faculty member. He is also affiliated with the NIATx Foundation, the organization responsible for making the NIATx organizational change model available to the public. For this scenario, DG and TM have an institutionally approved plan to manage potential conflicts of interest. The individuals who will conduct the data collection and interpretation for this study manuscript will have no affiliation with the NIATx Foundation or CHESS Health.

Multimedia Appendix 1

CONSORT-eHEALTH checklist (V 1.6.1).

PDF File (Adobe PDF File), 346 KB

  1. Jethro of Midian. The Bible. Authorized King James Version, Exodus Chapter 18. London, UK: Oxford UP; 1998:12-23.
  2. Kahn CH, editor. The Art and Thought of Heraclitus: An Edition of the Fragments With Translation and Commentary. Cambridge, UK: Cambridge University Press; 1981:1-23.
  3. Taylor F. The Principles of Scientific Management. Mineola, NY: Dover Publications; 1998.
  4. Box GEP. Science and statistics. Journal of the American Statistical Association 1976 Dec;71(356):791-799. [CrossRef]
  5. Ishikawa K. What Is Total Quality Control? The Japanese Way. Upper Saddle River, NJ: Prentice Hall; 1985:0139524339.
  6. Batalden PB, Davidoff F. What is "quality improvement" and how can it transform healthcare? Qual Saf Health Care 2007 Feb;16(1):2-3 [FREE Full text] [CrossRef] [Medline]
  7. Maslow AH. A theory of human motivation. Psychol Rev 1943 Jul;50(4):370-396. [CrossRef]
  8. Phillips JR. Enhancing the effectiveness of organizational change management. Hum Resour Manage 1983;22(1-2):183-199. [CrossRef]
  9. Gantt HL. Organizing For Work. New York, NY: Harcourt, Brace and Howe; 1919.
  10. Deming WE. Improvement of quality and productivity through action by management. Natl Prod Rev 1981;1(1):12-22. [CrossRef]
  11. Delbecq A, Van de Ven D, Gustafson D. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Glenview, IL: Scott Foresman; 1975.
  12. Mayo E. The Human Problems of an Industrial Civilization. New York, NY: The Macmillan Company; 1933:0405101848.
  13. Ohno T. Toyota Production System: Beyond Large-Scale Production. Boca Raton, FL: CRC Press; 1988:0915299143.
  14. Maidique MA, Zirger BJ. A study of success and failure in product innovation: The case of the U.S. electronics industry. IEEE Trans Eng Manage 1984 Nov;EM-31(4):192-203. [CrossRef]
  15. Conner D. Managing At The Speed Of Change: How Resilient Managers Succeed And Prosper Where Others Fail. 1st Edition. New York, NY: Villard Books; 1993:0679406840.
  16. Shewhart W. Economic Control of Quality of Manufactured Product. Milwaukee, WI: ASQ Quality Press; 2002.
  17. Utterback J. Mastering the Dynamics of Innovation: How Companies Can Seize Opportunities in the Face of Technological Change. University of Illinois at Urbana-Champaign, IL: University of Illinois at Urbana-Champaign's Academy for Entrepreneurial Leadership, Historical Research Reference in Entrepreneurship; 1994.
  18. Fayol H, editor. Administration Industrielle et Générale: Prévoyance, Organisation, Commandement, Coordination, Contrôle. Paris, France: H Dunod et E Pinat; 1918.
  19. Rogers E. Diffusion of Innovations. 3rd edition. New York, NY: Free Press; 1983:0029266505.
  20. Van de Ven AH. Problem solving, planning, and innovation. Part I. Test of the Program Planning Model. Hum Relat 2016 Apr 22;33(10):711-740. [CrossRef]
  21. Barnard C. The economy of incentives. In: Shafritz J, Ott J, Jang Y, editors. Classics of Organization Theory. Belmont, CA: Wadsworth Publishing Company; 2005:93-102.
  22. Barnett WP, Carroll GR. Modeling internal organizational change. Annu Rev Sociol 1995 Aug;21(1):217-236. [CrossRef]
  23. Kanter R. When a thousand flowers bloom: structural, collective, social conditions for innovation in organizations. In: Staw B, Sutton R, editors. Research in Organizational Behavior. New York, NY: Elsevier Science Inc; 2000.
  24. Gawande A. The checklist Manifesto: How to Get Things Right. New York, NY: Metropolitan Books; 2010.
  25. Lewin M. The impact of Kurt Lewin's life on the place of social issues in his work. J Soc Issues 1992;48(2):15-29. [CrossRef]
  26. Cooper R. The dimensions of industrial new product success and failure. J Mark 1979;43(3):93-103. [CrossRef]
  27. Berwick DM. Disseminating innovations in health care. JAMA 2003 Apr 16;289(15):1969-1975. [CrossRef] [Medline]
  28. Argyris C. Intervention Theory and Method: A Behavioral Science View. Reading, MA: Addison-Wesley; 1970.
  29. Simon HA. Decision-making and administrative organization. Public Adm Rev 1944;4(1):16. [CrossRef]
  30. Taguchi G. Introduction to Quality Engineering: Designing Quality Into Products and Processes. Tokyo, Japan: Asian Productivity Organization; 1986.
  31. Kotter J, Schlesinger L. Choosing strategies for change. Harv Bus Rev 1979;57(2):106-114. [CrossRef]
  32. Hage J, Aiken M. Social Change in Complex Organizations. New York, NY: Random House; 1970.
  33. Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R. Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci 2007 Dec 09;2:42 [FREE Full text] [CrossRef] [Medline]
  34. Ackerman LS. Transition management: An in-depth look at managing complex change. Organ Dyn 1982 Jun;11(1):46-66. [CrossRef]
  35. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50 [FREE Full text] [CrossRef] [Medline]
  36. Gustafson DH, Bosworth K, Chewning B, Hawkins RP. Computer-based health promotion: combining technological advances with problem-solving techniques to effect successful health behavior changes. Annu Rev Public Health 1987;8:387-415. [CrossRef] [Medline]
  37. Carroll KM, Ball SA, Martino S, Nich C, Babuscio TA, Nuro KF, et al. Computer-assisted delivery of cognitive-behavioral therapy for addiction: a randomized trial of CBT4CBT. Am J Psychiatry 2008 Jul;165(7):881-888 [FREE Full text] [CrossRef] [Medline]
  38. Marsch LA, Dallery J. Advances in the psychosocial treatment of addiction: the role of technology in the delivery of evidence-based psychosocial treatment. Psychiatr Clin North Am 2012 Jun;35(2):481-493 [FREE Full text] [CrossRef] [Medline]
  39. White VM, Molfenter T, Gustafson DH, Horst J, Greller R, Gustafson DH, et al. NIATx-TI versus typical product training on e-health technology implementation: a clustered randomized controlled trial study protocol. Implement Sci 2020 Oct 23;15(1):94 [FREE Full text] [CrossRef] [Medline]
  40. Dong L, Zhao X, Ong SL, Harvey AG. Patient recall of specific cognitive therapy contents predicts adherence and outcome in adults with major depressive disorder. Behav Res Ther 2017 Oct;97:189-199 [FREE Full text] [CrossRef] [Medline]
  41. Gustafson DH, McTavish FM, Chih M, Atwood AK, Johnson RA, Boyle MG, et al. A smartphone application to support recovery from alcoholism: a randomized clinical trial. JAMA Psychiatry 2014 May;71(5):566-572 [FREE Full text] [CrossRef] [Medline]
  42. Johnston DC, Mathews WD, Maus A, Gustafson DH. Using smartphones to improve treatment retention among impoverished substance-using Appalachian women: a naturalistic study. Subst Abuse 2019;13:1178221819861377 [FREE Full text] [CrossRef] [Medline]
  43. Gustafson D, Johnson K, Capoccia V, Cotter F, Ford II J, Holloway D. The NIATx Model: Process Improvement in Behavioral Health. Madison, WI: University of Wisconsin-Madison; 2011.
  44. Quanbeck AR, Gustafson DH, Marsch LA, McTavish F, Brown RT, Mares M, et al. Integrating addiction treatment into primary care using mobile health technology: protocol for an implementation research study. Implement Sci 2014 May 29;9:65 [FREE Full text] [CrossRef] [Medline]
  45. Gustafson DH, Sainfort F, Eichler M, Adams L, Bisognano M, Steudel H. Developing and testing a model to predict outcomes of organizational change. Health Serv Res 2003 Apr;38(2):751-776 [FREE Full text] [CrossRef] [Medline]
  46. Maher L, Gustafson D, Evans A. Sustainability, Model and Guide. Leicester, UK: British National Health Service (Modernisation Agency), NHS Institute for Innovation and Improvement; 2004.   URL: https:/​/webarchive.​​ukgwa/​20160805122935/​http:/​/www.​​media/​2757778/​nhs_sustainability_model_-_february_2010_1_.​pdf [accessed 2021-11-16]
  47. Fleddermann K, Molfenter T, Jacobson N, Horst J, Roosa MR, Boss D, et al. Clinician perspectives on barriers and facilitators to implementing e-health technology in substance use disorder (SUD) treatment facilities. Subst Abuse 2021;15:11782218211053360 [FREE Full text] [CrossRef] [Medline]
  48. Gustafson DH, Quanbeck AR, Robinson JM, Ford II JH, Pulvermacher A, French MT, et al. Which elements of improvement collaboratives are most effective? A cluster-randomized trial. Addiction 2013 Jun;108(6):1145-1157 [FREE Full text] [CrossRef] [Medline]
  49. Chen A. New data shows losing 80% of mobile users is normal, and why the best apps do better. @andrewchen. 2021.   URL: https:/​/andrewchen.​com/​new-data-shows-why-losing-80-of-your-mobile-users-is-normal-and-that-the-best-apps-do-much-better/​ [accessed 2021-11-15]
  50. Simpson DD. Treatment for drug abuse. Follow-up outcomes and length of time spent. Arch Gen Psychiatry 1981 Aug;38(8):875-880. [CrossRef] [Medline]
  51. Mahon D. Improving retention in opioid treatment. J Addict Addictv Disord 2020;7:47. [CrossRef]
  52. Gonzalez L. Des Moines Register runs op ed on satellite, FCC stats, and risks to Iowans.: The Institute for Local Self-Reliance (ILSR); 2018 Nov 26.   URL: [accessed 2021-11-16]
  53. Dornauer M, Bryce R. Too many rural Americans are living in the digital dark: the problem demands a new deal solution. Health Affairs. 2020 Oct 28.   URL: [accessed 2020-11-16]
  54. Petry NM, Alessi SM, Olmstead TA, Rash CJ, Zajac K. Contingency management treatment for substance use disorders: How far has it come, and where does it need to go? Psychol Addict Behav 2017 Dec;31(8):897-906 [FREE Full text] [CrossRef] [Medline]
  55. Gustafson DH, Hundt AS. Findings of innovation research applied to quality management principles for health care. Health Care Manage Rev 1995;20(2):16-33. [Medline]
  56. Rosala M. The critical incident technique in UX. Fremont, CA: Nielsen Norman Group; 2020 Jan 02.   URL: [accessed 2021-11-16]
  57. Walkthrough.: Wikipedia, The Free Encyclopedia; 2021 Jul 18.   URL: [accessed 2020-11-16]
  58. Gillam J, Davies N, Aworinde J, Yorganci E, Anderson JE, Evans C. Implementation of ehealth to support assessment and decision-making for residents with dementia in long-term care: systematic review. J Med Internet Res 2022 Feb 03;24(2):e29837 [FREE Full text] [CrossRef] [Medline]
  59. Gustafson DH, McTavish FM, Chih M, Atwood AK, Johnson RA, Boyle MG, et al. A smartphone application to support recovery from alcoholism: a randomized clinical trial. JAMA Psychiatry 2014 May;71(5):566-572 [FREE Full text] [CrossRef] [Medline]

ACHESS: Addiction Comprehensive Health Enhancement Support System
BARN: Body Awareness Resources Network
CBT: cognitive behavioral therapy
CBT4CBT: Computer Based Training for Cognitive Behavioral Therapy
FAQ: frequently asked question
NIATx: Network for the Improvement of Addiction Treatment
NIDA: National Institute on Drug Abuse
NIH: National Institutes of Health
QI: quality improvement
RISE Iowa: Recovering Iowans Supporting Each Other
SUD: substance use disorder
TES: Treatment Evaluation Services

Edited by A Kushniruk; submitted 22.11.21; peer-reviewed by K Johnson, S Mukherjee, T Fazzino; comments to author 03.02.22; revised version received 19.04.22; accepted 31.05.22; published 14.07.22


©David Gustafson Sr, Julie Horst, Deanne Boss, Kathryn Fleddermann, Nora Jacobson, Mathew Roosa, J Charles Ross, Rachel Gicquelais, Olivia Vjorn, Tracy Siegler, Todd Molfenter. Originally published in JMIR Human Factors (, 14.07.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.