This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on http://humanfactors.jmir.org, as well as this copyright and license information must be included.
Clinical pathways (CPs) can improve patient outcomes but can be complex to implement. Technologies, such as clinical decision support (CDS) tools, can facilitate their use, but require end-user testing in clinical settings.
This study applied the Technology Acceptance Model to evaluate the individual, organizational, and technological contexts impacting application of a portal to facilitate a CP for anxiety and depression (the ADAPT Portal) in a metropolitan cancer service. The ADAPT Portal triggers patient screening on patient reported outcomes, alerts staff to high scores, recommends evidence-based management, and triggers review and rescreening at set intervals.
Quantitative and qualitative data on portal activity, data accuracy, and health service staff perspectives were collected. Quantitative data were analyzed descriptively, and thematic analysis was applied to qualitative data.
Overall, 15 (100% of those invited) health service staff agreed to be interviewed. During the pilot, 73 users (36 health service staff members and 37 patients) were registered on the ADAPT Portal. Of the 37 patients registered, 16 (43%) completed screening at least once, with seven screening positive and triaged appropriately. In total, 34 support requests were lodged, resulting in 17 portal enhancements (technical issues). Health service staff considered the ADAPT Portal easy to use and useful; however, some deemed it unnecessary or burdensome (individual issues), particularly in a busy cancer service (organizational issues).
User testing of a CDS to facilitate screening and assessment of anxiety and depression in cancer patients highlighted some technological issues in implementing the ADAPT CDS, resulting in 17 enhancements. Our results highlight the importance of obtaining health service staff feedback when piloting specialized CDS tools and addressing contextual factors when implementing them.
In the last 25 years, health care has focused on improving the quality and value of care delivery through standardization of the management of specific conditions with guidelines and clinical pathways (CPs) [
The
However, studies across numerous health conditions confirm that guidelines and CPs are not enough to change patient care within complex health systems owing to knowledge gaps, poor communication, and insufficient implementation efforts [
Several theoretical models have been proposed to explain uptake and guide assessment of CDS tools, and the most widely used is the Technology Acceptance Model [
This study sought to apply the Technology Acceptance Model in a pilot of the ADAPT Portal with target end users to rigorously evaluate its utility prior to a large-scale evaluation of the ADAPT CP overall. Our aim was to refine the system to best meet users’ needs prior to a large-scale implementation of the ADAPT Portal. More specifically, the study aimed to evaluate the individual, organizational, and technological contexts impacting the ADAPT Portal’s perceived usability, usefulness, and appropriateness within a clinical cancer service.
The study was conducted in a cancer service within a large Australian metropolitan hospital. The cancer service elected to include patients receiving chemotherapy as part of their care in the study.
A triangulation mixed methods design [
After senior management confirmed participation in the study and a research participation agreement was established with the cancer service, a subset of health service staff at the oncology service (purposively selected to ensure diversity in professional backgrounds and ADAPT CP roles) was invited to participate in the study. Staff received an email from the study team inviting them to participate and provide written informed consent. Participating staff were interviewed after the implementation period to capture their experience of planning for and using the ADAPT Portal within their service.
All patients commencing treatment during the study period at the site were invited to participate in the study. Interested patients provided written consent to participate in ADAPT screening and allow the research team to access their medical records.
A lead team comprising management staff, nursing staff, social work staff, psychology staff, clinical system specialists, medical oncology specialists, and service improvement staff worked with the research team to tailor the ADAPT Portal to their local needs, resources, and preferences. The lead team mapped the CP and cancer service operations and compiled these into a workflow that operationalized how the ADAPT Portal would be used at the center. User training on the tailored ADAPT CP and Portal was provided to medical oncology, nursing, and allied health staff, with key ADAPT Portal users attending individual training sessions according to their roles and responsibilities in the ADAPT CP and associated tasks within the ADAPT Portal.
The ADAPT CP was then implemented for 5 months among several tumor streams within the medical oncology service. During implementation, users (health service staff and patients) had access to online, phone, and email support from the research team. After implementation, staff interviews were carried out, and portal usage and contacts with the research team were collated.
The study was approved by the Human Research Ethics Committee of the participating health care institution.
The ADAPT Portal was developed by a multidisciplinary working group (comprising psycho-oncologists, oncologists, researchers, patient representatives, and information technology [IT] web designers and programmers) tasked with defining the ADAPT Portal’s scope and functionality via agile design [
The web-based ADAPT Portal ultimately consisted of two parts. The first part was a patient-directed portal where patients verify their registration and create a password to activate their portal account, and are directed to the home page where information and resources are available. At scheduled time points, patients receive an email alert with a direct link to complete anxiety and depression screening measures and can access self-management and information resources. The second part was a health service staff portal where health service staff log in using a password, register patients who have agreed to participate in the CP with their contact details, receive alerts of patients scoring above clinical cutoffs, and are prompted to complete evidence-based actions according to CP recommendations. Clinical staff can visually track patients’ longitudinal screening data and CP progression, as well as generate reports at an individual or service level. Links to education and training resources are accessible to staff via the portal along with portal user guides and a support messaging service.
ADAPT Portal user activity was reviewed to identify system functionality and uptake. A random selection of registered consenting patients’ medical records was reviewed to assess the quality of data captured and discrepancies between CP documentation in the ADAPT Portal and patients’ electronic medical records.
During the 5-month implementation, user support contacts were tracked, capturing the reason for contact and duration of support required. This information was reviewed and coded according to the ADAPT Portal functional domains (ie, registration, screening, referral, review, rescreening, user error, and system error) for analysis. Additionally, potential design improvements identified during lead team meetings, training sessions, and user support contacts to improve system performance and user satisfaction of the ADAPT Portal were logged throughout the study. These were reviewed by the study team and classified as
Data were obtained via health service staff user interviews, review of user support contacts, and field observations by the ADAPT research team. Using purposive sampling, 15 health service staff members participated in semistructured interviews with an interviewer independent of the core ADAPT research team. Interviews explored perceived acceptability and utility of the ADAPT Portal, problems and challenges encountered with the system, and recommendations for improvement. Interviews were transcribed for analysis. Additional data from the staff interviews focusing on staff and organizational barriers to utilizing the ADAPT CP are published elsewhere [
Quantitative data were entered into the Statistical Package for the Social Sciences (SPSS) database. Descriptive statistics (means and medians for continuous data and percentages for categorical data) were generated.
Interview transcripts were thematically analyzed by two researchers using the platform NVIVO. The two researchers independently performed initial coding to group information according to the modified Technology Acceptance Model themes [
A total of 73 ADAPT Portal users (36 health service staff and 37 patients) were registered on the ADAPT Portal during the pilot, of whom 67 (92%) accessed the Portal.
Registered health service staff included one administrator, two data managers, eight medical oncologists, 13 registered nurses, three cancer care coordinators, one clinical nurse specialist, one clinical nurse educator, four clinical psychologists, and three social workers. Of these, 15 were purposively selected (to ensure diversity of background and ADAPT CP roles) to participate in the postimplementation interview (all agreed). The interview sample included both full-time and part-time staff, who had been in their current role for an average of 3 years (
Interviewee demographic profile.
Demographic | Total (n=15) | |
|
|
|
|
26-50 years | 12 (80%) |
|
51-75 years | 3 (20%) |
|
|
|
|
Female | 15 |
|
|
|
|
Oncologist | 1 |
|
Nurse-RNa | 2 |
|
Nurse-CNSb, CNCc, coordinator | 3 |
|
NUMd/clinical managers | 3 |
|
Clinical psychologist | 3 |
|
Social worker | 1 |
|
Clinical trial manager | 1 |
|
Data manager | 1 |
Duration in the current role, mean (range) | 3.4 years (5 months to 10 years) | |
|
|
|
|
Full time | 9 (60%) |
|
Part time | 6 (40%) |
aRN: registered nurse.
bCNS: clinical nurse specialist.
cCNC: clinical nurse consultant.
dNUM: nursing unit manager.
Of the 37 patients registered, 16 (43%) completed screening once, with seven screening positive. In response to system alerts sent to nominated clinical staff, staff triaged all seven patients. Following triage, the step allocation for two patients was downgraded and documented in the ADAPT Portal, two patients declined additional support, and three patients were referred via the ADAPT Portal to psychosocial services (
Portal user activity.
Portal activity | Patients, n | Health service staff, n |
Number registered | 37 | 36 |
Number accessed the portal | 35 | 32 |
Number screened | 16 | N/Aa |
Total number of screening events | 17 | N/A |
Total number of positive screens | 7 | N/A |
Number of patients triaged | 7 | N/A |
Number of referrals | 3 | N/A |
aN/A: not applicable.
A total of 34 research support requests were lodged during the 5-month implementation period, with the majority lodged by health service staff (n=32, 94%) and 2 (6%) by patients.
Summary of unplanned support contact.
Support contact domain | Total (n=34), n |
Workflow | 13 |
User error | 7 |
System & network error | 6 |
Set-up & configuration | 3 |
Training | 3 |
Usability | 2 |
Regular review of support contacts and researcher observations led to 17 suggestions for improvements in the system, and of these, five were classified as critical and four were classified as serious (
Summary of system improvements.
Functionality domain | Severity, n | |||
Critical | Serious | Minor | Total (n=17) | |
Reporting | N/Aa | 1 | 3 | 4 |
Screening | 2 | 2 | 1 | 5 |
Patient registration | N/A | 1 | 4 | 4 |
Triage functionality | 1 | N/A | 1 | 2 |
Configuration | 1 | N/A | N/A | 1 |
Referral | 1 | N/A | N/A | 1 |
aN/A: not applicable.
Ten patients’ electronic medical records (EMRs) were compared with ADAPT Portal extracts to evaluate data capture and accuracy. These highlighted frequent missing or incorrect data on cancer diagnosis date and cancer staging in the ADAPT Portal, which occurred when these data were not available in the EMR system at the time of patient registration and were not subsequently updated in the ADAPT Portal when the information became available. CP activity recorded in the ADAPT Portal was consistent with actual psychosocial care documented in the EMR, except in two cases where the patients refused treatment. In these cases, users did not document this via the ADAPT Portal referral functionality, but rather as a free text note similar to current EMR documentation practice.
Interview length ranged from 16 to 50 minutes (average, 25 minutes), and the themes identified focused on usability and views of ADAPT Portal processes. Staff reported that the system was easy to use and navigate as follows:
I’m not very tech savvy, but it was fine, it was very easy.
However, some staff reported difficulty logging into the ADAPT Portal owing to forgetting their passwords or poor Wi-Fi connectivity, while others reported that the time lag between training and actually using the system was too long, impacting their ADAPT Portal use confidence. Nevertheless, these challenges were quickly overcome as shown in the following comment:
By the time we got a referral we thought, oh how do we do this? How do we log in? What do we do? But, it was fine – you know, we figured it out and we could email [the support team] and she helped us.
Staff also commented positively on system support, preferring this to user guides. One staff user made the following comment:
Contact was good – if staff asked team for resources or help, response was prompt.
Feedback on the usefulness of the ADAPT Portal for patient care was polarized. Some staff believed the ADAPT Portal did not improve on existing service processes that were well established, demonstrated in this comment:
So I think it [the ADAPT Portal] has a very good role but we’re already covering those areas.
Others reported that the ADAPT Portal was a useful mechanism to formally document psychosocial processes and remind staff that psychosocial assistance was part of standard patient care. One participant clarified their view:
I think we need to probably formalize what processes we've already got in place…I think it's important we're doing it with all patients, it’s part of the ongoing assessment of them.
Staff endorsed the patient resources containing local and national support information, as patients could access relevant information in one location at their own convenience. An example of a comment made by participants was:
It's useful to have and it's good for the patient.
Staff reported varied responses from patients, with most patients open to and positive about using the ADAPT Portal, but others rejected routine screening as unnecessary or too complex. One staff user observed an elderly patient having trouble screening via a tablet and decided to abandon screening.
Staff reported the need for the ADAPT Portal to be linked with the existing EMR as staff already log into multiple systems for patient care and other patient screening assessments are integrated into the EMR. Participants noted that the service has undergone major technological change in the last 2 years and were therefore reluctant to undertake further technological change. This was highlighted in the following comment:
We’ve only had that I think, just two years or, so we’ve just had a massive change with that, when everybody made electronic referrals and things, and I guess maybe this was just another thing that was put onto people
Staff reported that their work habits changed during the implementation period because they had to access an additional system, and their workload increased. For one user, the role expanded. Regarding the ADAPT Portal, a health service staff member made the following comment:
...was an extra thing that you’re being asked to do.
The service found it necessary to nominate one nurse to remind staff when their patients were due for screening, despite the ADAPT Portal automatically alerting staff, to ensure screening was completed, as summarized in the comment below:
Even though there’s a reminder we still forget sometimes. So, I think that one person [overseeing] is good.
This study is the first to review an online clinical decision support system for a CP addressing anxiety and depression screening and management (the ADAPT Portal) in an Australian cancer service. We assessed the individual, organizational, and technological contexts impacting the ADAPT Portal’s perceived usability, usefulness, and appropriateness, and adjusted the system where possible to facilitate uptake in a larger implementation study. This is a critical step in the development of new systems for use in clinical care, and is rarely evaluated qualitatively and quantitatively.
Testing the system, responding to staff support contacts, making changes to the CDS, and providing training in altered processes and components took some time and delayed patient registrations for some weeks. Ultimately, 37 patients were successfully registered, and their progress through the system was tracked.
Our study highlighted a number of usability issues, technical barriers, and training requirements that resulted in 17 improvements to the ADAPT Portal. Improvements to the ADAPT Portal allowed better recording of the rationale behind decisions and adjustment for real-world variations in patient flow through the system. These findings highlight the importance of addressing perceived usability to ensure the smooth delivery of CDS tools, such as the ADAPT CP, and mirror findings from other studies on diverse CDS tools (such as a movie recommendation system [
Nevertheless, while a number of usability issues were revealed and rectified during the study, staff on the whole had positive perceptions regarding the usefulness of the ADAPT Portal to their patients and the oncology service, which proved to be a strong motivator for ongoing use of the portal. This finding further supports the validity of the Technology Acceptance Model and reflects findings from previous studies [
Not all staff perceived the ADAPT CP to be useful in their practice. Some believed that their existing internal processes were already effective in identifying patients requiring psychosocial support, thus rendering the ADAPT Portal unnecessary in their eyes. In contrast, 7 of 16 patients screened on the ADAPT Portal scored in the range requiring triage and referral, and may have been missed without the system in place. The PARiHS implementation framework, commonly applied to health service change efforts, suggests that staff require evidence of intervention efficacy from not only randomized controlled trials, but also their own and patient experiences, and local evidence of needs and benefits [
While ADAPT Portal usability was addressed in this study and staff were positive about the system on the whole, some contextual issues remained as barriers. These included our inability to integrate the portal into the established electronic record management system, which increased staff burden in learning and accessing an additional system. Furthermore, staff had only recently experienced a sharp learning curve in adapting to a new EMR, reducing their capacity to learn another. James Tcheng from the US National Academy of Medicine [
This study had a number of strengths, including a mixed methods design that produced a rich and complementary data set and the use of a recognized model for evaluating technology acceptance. A number of study limitations must also be considered. This was a small pilot in one urban site and may not reflect findings in other oncology services, including those in small rural areas. Implementation was for 5 months, and some issues related to technology usability may not have arisen in that time. Evaluation over a longer implementation period is required.
As a clinical decision support system, the ADAPT Portal achieved its goal in aligning patient care at a metropolitan cancer service with the recommendations of the ADAPT CP [
Australian clinical pathway for the screening, assessment, and management of anxiety and depression in adults with cancer
clinical decision support
clinical pathway
electronic medical record
information technology
This study and the ADAPT Portal was developed as part of the Anxiety and Depression Pathway (ADAPT) Program led by the Psycho-oncology Cooperative Research Group (PoCoG). Members of the ADAPT Program group have contributed to ADAPT activities and resources. The authors acknowledge the ADAPT Program group members for their contribution to the design and delivery of the ADAPT Program.
ADAPT Program group members include Prof Gavin Andrews, Kate Baychek, A/Prof Philip Beale, A/Prof Josephine Clayton, Dr Joseph Coll, Prof Afaf Girgis, Dr Peter Grimison, Prof Tom Hack, Prof Brian Kelly, Dr Laura Kirsten, Dr Toni Lindsay, A/Prof Melanie Lovell, Dr Tim Luckett, Dr Michael Murphy, Dr Jill Newby, Dr Frances Orr, Don Piro, Prof Tim Shaw, John Stubbs, and Prof Rosalie Viney.
The authors would also like to acknowledge the commitment and contribution of Dr Melanie Price (1965-2018) to this study. Dr Price was a respected member of the psycho-oncology and palliative care community in Australia for over 22 years. She was a tireless advocate for people affected by cancer, their families, and psycho-oncology as a discipline.
The authors acknowledge and thank the cancer service, the staff, and the patients for their participation in the study. Finally, the authors thank Dr Subo Thavaneswaran, Medical Oncologist, who reviewed the paper from a clinical perspective.
The ADAPT Program is funded by a Translational Program Grant from the Cancer Institute NSW. Liesbeth Geerligs is funded by a scholarship from the Australian Post-Graduate Awards Scheme (Australian Government), and additional top-up funding from Sydney Catalyst and the ADAPT Program. The funding bodies had no role in study design, data collection, analysis, or writing of the manuscript.
None declared.