This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on https://humanfactors.jmir.org, as well as this copyright and license information must be included.
e-Consultations between primary care physicians and specialists are a valuable means of improving access to specialty care. Adolescents and young adults (AYAs) face unique challenges in accessing limited adolescent medicine specialty care resources, which contributes to delayed or forgone care. e-Consultations between general pediatricians and adolescent medicine specialists may alleviate these barriers to care. However, the optimal application of this model in adolescent medicine requires careful attention to the nuances of AYA care.
This study aims to qualitatively analyze feedback obtained during the iterative development of an e-consultation system for communication between general pediatricians and adolescent medicine specialists tailored to the specific health care needs of AYAs.
We conducted an iterative user-centered design and evaluation process in two phases. In the first phase, we created a static e-consultation prototype and storyboards and evaluated them with target users (general pediatricians and adolescent medicine specialists). In the second phase, we incorporated feedback to develop a functional prototype within the electronic health record and again evaluated this with general pediatricians and adolescent medicine specialists. In each phase, general pediatricians and adolescent medicine specialists provided
Both general pediatricians (n=12) and adolescent medicine specialists (n=12) perceived the usefulness of e-consultations for AYA patients, with more varied perceptions of potential usefulness for generalist and adolescent medicine clinicians. General pediatricians and adolescent medicine specialists discussed ways to maximize the usability of e-consultations for AYAs, primarily by improving efficiency (eg, reducing documentation, emphasizing critical information, using autopopulated data fields, and balancing specificity and efficiency through text prompts) and reducing the potential for errors (eg, prompting a review of autopopulated data fields, requiring physician contact information, and prompting explicit discussion of patient communication and confidentiality expectations). Through iterative design, patient history documentation was streamlined, whereas documentation of communication and confidentiality expectations were enhanced.
Through an iterative user-centered design process, we identified user perspectives to guide the refinement of an e-consultation system based on general pediatrician and adolescent medicine specialist feedback on usefulness and usability related to the care of AYAs. Qualitative analysis of this feedback revealed both opportunities and risks related to confidentiality, communication, and the use of tailored documentation prompts that should be considered in the development and use of e-consultations with AYAs.
Although general pediatricians manage a variety of adolescent health concerns, referrals to adolescent medicine specialists are common. Common referrals to adolescent medicine specialists include management of menstrual disorders, sexual and reproductive health care, gender-affirming care, and behavioral health care [
For AYAs referred to adolescent medicine specialists for confidential health concerns, the barriers to attending these visits without the knowledge or assistance of their families may be increased or insurmountable. Confidentiality concerns have been associated with forgone care [
e-Consultations are an alternative strategy for subspecialists to offer clinical guidance to PCPs. In this asynchronous model of telehealth care, a PCP submits patient-specific information to a subspecialist, who later reviews this information and provides recommendations back to the PCP to guide their care [
Although e-consultations have the potential to increase access to adolescent medicine specialists, the application of this model within adolescent medicine requires careful attention to the nuances of adolescent care. Issues of confidentiality, consent, and communication among PCPs, adolescent medicine specialists, AYAs, and caregivers of AYAs warrant specific focus. In addition, consideration of the optimal information to transfer between PCPs and adolescent medicine specialists [
In this study, we seek to examine feedback during the iterative development of e-consultations for communication between PCPs and adolescent medicine specialists. Informed by prior studies examining e-consultation features and design [
We conducted an iterative user-centered design and evaluation process in two phases. In phase 1, we developed a prototype e-consultation template and workflow based on a literature review and conversations with individuals using e-consultations in other settings, including the Veterans Health Administration and University of Pittsburgh Medical Center. We evaluated the static prototype and workflow storyboards with target users, including general pediatrician PCPs and adolescent medicine specialists. We incorporated feedback from this process to develop a functional prototype within the electronic health record (EHR) and again evaluated this with PCPs and adolescent medicine specialists in phase 2. We performed a qualitative analysis of the feedback obtained through
We approached target users via email from an existing research network of primary care practices (Pediatric PittNet) and the Division of Adolescent and Young Adult Medicine of the University of Pittsburgh Medical Center Children’s Hospital of Pittsburgh. The Division of Adolescent and Young Adult Medicine is the primary group of adolescent medicine specialists caring for AYAs across 26 counties in Western Pennsylvania as well as portions of Ohio and West Virginia. The general pediatrician PCPs across more than 40 practices affiliated with the Children’s Hospital of Pittsburgh span both academic and community practices across 13 counties in Western Pennsylvania. Approximately one-quarter of these practices are in rural counties, and all use a shared EHR, which is also used by the Division of Adolescent and Young Adult Medicine. At the time of this study, no payers in our region paid clinicians for their time conducting store-and-forward provider-to-provider consultations. General pediatrician PCPs and adolescent medicine specialists were eligible for participation, including both academic and community general pediatricians, advanced practice providers, and trainees in fellowship. Clinicians who were not engaged in patient care were excluded from the study. The participants received US $25 gift cards. Participants were purposefully sampled to ensure a balance between PCPs and adolescent medicine specialists and diversity of years of experience and clinical effort. The University of Pittsburgh institutional review board provided ethical approval. We obtained written documentation of informed consent from participants.
Phase 1 was conducted from July to August 2018. A study team member (DY or KNR) met with each participant in a private office, obtained informed consent, and talked through a series of 22 slides. These slides reviewed the goals of e-consultations, the research process, prototype note templates for both PCPs and adolescent medicine specialists, and storyboards that provided a visual representation of planned workflow and prototype screenshots illustrating the planned process as a clinician worked through the system. We started with these static prototypes because they could be developed with minimal time investment and rapidly adapted in response to feedback. During this phase, data were collected through think-alouds, exit interviews, and web-based surveys. Changes were made to the prototype after every 3 participants to continuously incorporate feedback.
After incorporating feedback from phase 1, we developed a functional electronic prototype within the play environment of our EHR (EpicCare). Phase 2 was conducted from July to September 2019. In this phase, participants were asked to complete specific tasks depending on their role (PCP vs adolescent medicine specialist).
During PCP sessions, participants were given 3 vignettes where adolescent medicine specialist advice might be sought related to polycystic ovary syndrome, gender-affirming care, and eating disorders. Participants were asked to prepare mock e-consultations for each vignette within a practice EHR environment. During the first 2 vignettes, PCPs provided think-aloud commentary while completing the task. For the third vignette, participants completed the notes at their usual working pace while their efforts were timed. For the final 3 PCP participants, mock responses created by prior adolescent medicine specialist participants were shared after the prior session components, and the PCPs were asked to
During adolescent medicine specialist sessions, participants were given 3 mock e-consultations (1 for each vignette) generated by PCP participants during prior PCP sessions without identifiers of the participating PCP. During the first 2 vignettes, adolescent medicine specialists provided think-aloud commentary while preparing mock responses. For the third vignette, participants completed the mock response at their usual working pace while their efforts were timed.
After completing these tasks, all participants completed semistructured exit interviews and the same web-based survey used in phase 1. Throughout this phase, changes were again continuously made to the EHR prototype in response to feedback.
Throughout the iterative user-centered design and evaluation process, feedback was collected from PCPs and adolescent medicine specialists through the processes described above: think-aloud commentary during the use of the prototypes and semistructured exit interviews. During think-alouds, participants were asked to talk through their thoughts as they reviewed the prototypes and were encouraged to comment on possible limitations or undesirable components of the proposed system. The semistructured exit interviews included questions on the perceived usefulness and usability of the e-consultation system.
A study team member (DY or KNR) took written notes during think-alouds and exit interviews, which were deidentified and securely stored. Next, 2 investigators (DY and KNR) analyzed think-aloud and exit interview data using content analysis, informed by elements of the Technology Acceptance Model and usability theory [
After reviewing the prototypes, participants also completed a web-based survey with 31 questions, including demographic questions and items adapted from the Technology Acceptance Model survey [
Participants included 12 general pediatrician PCPs and 12 adolescent medicine specialists (
Participant characteristics of phase 1 and phase 2.
Characteristics | Phase 1: static prototype (n=12), n (%) | Phase 2: EHRa prototype (n=12), n (%) | |||
|
|||||
|
PCPb | 6 (50) | 6 (50) | ||
|
Adolescent medicine specialist | 6 (50) | 6 (50) | ||
|
|||||
|
Currently in training | 2 (17) | 4 (33) | ||
|
0-5 years | 2 (17) | 4 (33) | ||
|
6-10 years | 1 (8) | 0 (0) | ||
|
11-20 years | 0 (0) | 2 (17) | ||
|
>21 years | 7 (58) | 2 (17) | ||
|
|||||
|
0-2 | 3 (25) | 4 (33) | ||
|
3-4 | 4 (33) | 3 (25) | ||
|
5-6 | 3 (25) | 2 (17) | ||
|
>7 | 2 (17) | 3 (25) | ||
|
|||||
|
Male | 3 (25) | 2 (17) | ||
|
Female | 8 (67) | 10 (83) | ||
|
Other or prefer not to answer | 1 (8) | 0 (0) |
aEHR: electronic health record.
bPCP: primary care physician.
Regarding the usefulness of e-consultations, PCPs and adolescent medicine specialists perceived potential advantages for patients, including themes of improving access to care, increasing timeliness and convenience of care, reducing travel burden, and enhancing the role of the patient-centered medical home (
Participant perspectives on the usefulness of e-consultations.
Domain | Primary care themes | Adolescent medicine themes |
Relative advantage for patients |
Reduces visits (+)a Increases timeliness of care (+) Increases access for patients with barriers (+) Increases communication (+) Improves health care for patients (+) |
Reduces visits (+) Increases timeliness of care (+) Increases convenience (+) Decreases transportation barriers (+) Keeps care within the medical home (+) May lead to increased PCPb visits in lieu of specialty visits (+/−)c |
Relative advantage for clinicians |
Facilitates communication with a specialist in a more structured way (+) Value in PCP getting answers for patient and family (+) Takes a lot of PCP time (−)d Adoption will require adequate payment for time (+/−) May be better for some clinical scenarios than other (+/−) |
Consult will provide structure and documentation to activities specialists already do (+) Could use to connect with other specialists or subspecialists (+) May help to ensure PCP provides information needed for a consult (+) |
Value for generalists versus |
Makes the process easier for specialists but increases work for PCPs (−) |
PCPs may not be comfortable with carrying out adolescent medicine plan (−) |
Complexity |
Difficult to anticipate what specialist needs to know (−) |
Difficult to know what PCPs should be expected to include in a consult (−) Difficult to know what management PCPs are comfortable initiating (−) |
Compatibility |
Slightly repetitive of documentation for the visit itself (−) |
Anticipate fitting easily into everyday workflow (+) Fitting into the workday will depend on the volume (−) Interoperability issues with other EHRse (−) Would be ideal if integrated with the scheduling process (+/−) |
Learning incentive |
Will lead to less reliance on specialists in the future (+) |
Will help PCPs learn for future patients (+) May lead to fewer consults in the future (+) |
Expected frequency |
Would use on a regular basis (+) |
Definitely anticipate using (+) |
aTheme perceived as a positive effect of e-consultations.
bPCP: primary care physician.
cTheme perceived as a positive or negative effect of e-consultations.
dTheme perceived as a negative effect of e-consultations.
eEHR: electronic health record.
PCP and adolescent medicine specialist perceptions of the advantages of e-consultations for clinicians were more variable. Although adolescent medicine specialists felt that e-consultations would provide structure and documentation to activities that specialists already do, PCPs identified a burden on their time and desired adequate reimbursement in a fee-for-service environment. Similarly, regarding the value for generalists versus specialists, PCPs felt that e-consultations would make the referral process easier for specialists while increasing the workload of PCPs. Regarding the complexity of the process, adolescent medicine specialists expressed concern that PCPs may not be comfortable carrying out their management plans, and both groups noted the complexity of anticipating the knowledge that the other clinician would need. Regarding the compatibility of e-consultations, PCPs raised concerns about duplicating their work, and adolescent medicine specialists expressed varying degrees of concern about incorporating e-consultations into their daily workflow and interoperability with other EHRs. Both PCPs and adolescent medicine specialists noted learning incentives related to increasing PCP knowledge of AYA health concerns and decreasing reliance on specialists for this care in the future. Both groups anticipated the frequent use of e-consultations.
Efficiency was a common focus for both PCPs and adolescent medicine specialists, with comments falling into four themes. First, participants recognized the importance of reducing documentation time by avoiding redundancy, minimizing free-text entry, and using drop-down menus (
Comments related to usability also frequently addressed reducing potential errors within the four themes. First, recommendations to use autopopulated fields to enhance efficiency were tempered with the acknowledgment that autopopulated fields can contain inaccurate or outdated information, leading to recommendations to follow autopopulated data fields with prompts to encourage PCP review and annotation of autopopulated data. Second, the importance of ensuring accurate information for interprofessional communication was emphasized. Third, both PCPs and adolescent medicine specialists advocated for prompts for specific patient and clinical information. These recommendations included prompts for PCPs relevant to the care of AYAs (eg, affirmed name and pronouns), prompts for adolescent medicine specialists to enhance the specificity of their recommendations (eg, exact laboratory test order number), and simple text changes to enhance clarity overall (eg, replacing
Comments related to the affective experience of using e-consultations were mixed. Although some users found the amount of data entry to be frustrating and burdensome, others commented that the process was straightforward and clear. Both PCPs and adolescent medicine specialists felt that the e-consultation process was generally easy to learn with minimal training.
Participant perspectives on the usability of e-consultations.
Domain and theme | Subtheme | |
|
||
|
Reduce documentation time |
Avoid redundant documentation (PCPa and AMb) Minimize free text (PCP and AM) Use drop-down menus when possible (PCP and AM) Allow the ability to incorporate images (AM) |
|
Organize to prioritize relevant data and questions |
Place question to a specialist upfront to frame the consult (PCP and AM) |
|
Maximize autopopulated data field use |
Pull in existing data fields (eg, family history) to avoid redundant data entry (PCP) Place free text after relevant autopopulated data to avoid duplicating information (PCP) Allow objective data to be pulled in and interpreted (eg, BMI percentiles; AM) |
|
Balance efficiency of documentation with the efficiency of the process |
Develop diagnosis-specific templates to enhance efficient data sharing (AM) Template prompts may be interpreted as a mandate for PCP to obtain information and may increase the burden of the documentation process (PCP) |
|
||
|
Minimize potential errors in autopopulated data in EHRc |
Autopopulating data fields contain errors and need PCP review (AM and PCP) Data with a high error rate should be entered rather than pulled in (PCP) |
|
Ensure accurate information for interprofessional communication |
Specialist needs information on how to get in touch with PCP if needed (PCP and AM) Communicating with PCP by phone is sometimes better than electronically (AM) PCP would like clarity about which specialist is receiving the consult (PCP) |
|
Maximize clarity of prompts for specific patient and clinical information |
Maximize clarity of the desired outcome of consult (PCP) Provide specific prompts for key information (eg, preferred language, pronouns; AM) Maximize clarity of language (AM) Templates tailored to specific complaints may reduce missing key information (PCP) Specialist reply templates should use prompts for specific recommendations (eg, lab test order numbers, medication dosing, follow-up interval; PCP and AM) |
|
Maximize clarity of prompts about confidentiality and communication |
Specify parent involvement (or lack thereof) in consultation (AM) Specify the degree of confidentiality to be maintained (AM and PCP) Specify whether adolescent, parent, or PCP could be contacted (AM) |
|
||
|
Potential frustration versus straightforward |
Amount of data entry could be frustrating and increase cognitive burden (PCP) Use blank space and text formatting to increase readability (PCP) System is straightforward and clear (AM) |
|
||
|
Learnable |
System is easy to learn and use; building on familiar design is helpful (PCP) Recommend focused training in the use of e-consultations (<20 minutes; AM) |
aPCP: primary care physician.
bAM: adolescent medicine.
cEHR: electronic health record.
During phase 1, PCPs’ responses trended lower for perceived usefulness and usability compared with adolescent medicine specialists (
Perceived usefulness, usability, intention to use, and system satisfaction. AM: adolescent medicine; PCP: primary care physician.
During the timed tasks during phase 2, the median time for PCPs to enter the mock e-consultation was 6 minutes 9 seconds (range 4 minutes 31 seconds-7 minutes 26 seconds). The median time for subspecialists to review and respond to the mock e-consultation was 7 minutes 49 seconds (range 3 minutes 56 seconds-11 minutes 8 seconds).
During the first phase of development, we made changes based on feedback as participants reviewed the static prototype. For the PCP note template requesting the e-consultation, we moved the consult question for the adolescent medicine specialist to the beginning of the note template, changed the order of items to enhance accuracy and efficiency when autopopulated text was pulled in, and reduced the number of unique prompts across the review of systems, physical exam, family history, and social history fields. Although these portions were streamlined, other specific prompts relevant to the care of AYAs (eg, gender identity, confidentiality, and contact information for patients and families, and PCP) were enhanced. For the adolescent medicine specialist note template responding to the e-consultation, substantially more details were added regarding follow-up recommendations (eg, location, interval, and the specific adolescent medicine specialist to be seen if relevant).
During the second phase of development, we made further changes based on feedback as participants interacted with an EHR prototype (
Changes in primary care physician e-consultation request note template through the iterative design process.
Changes in adolescent medicine specialist e-consultation response note template through the iterative design process.
We used a replicable user-centered design process to develop and refine e-consultation templates and processes for general pediatrician consultation with adolescent medicine specialists. We qualitatively analyzed user feedback during two phases of development to guide the revision of our e-consultations and produced a final prototype with similar perceived usefulness and usability among PCPs and adolescent medicine specialists. Our iterative design process and feedback themes may inform the development of similar e-consultation systems in other settings. In particular, by engaging experts in pediatric and adolescent health in our user-centered design process, we identified specific considerations for the use of e-consultations with AYAs to promote safe, equitable, and high-quality care for this population.
Participants identified both opportunities and risks for the use of e-consultations with AYAs related to confidentiality. A major perceived advantage was the potential to enhance access to confidential care for AYAs. AYAs may delay or forgo care because of a lack of knowledge of how or where to obtain services, transportation challenges, and concerns about maintaining confidentiality [
Participants further identified opportunities for the use of e-consultations with AYAs to facilitate communication. Overall, participants perceived e-consultations as a beneficial way to enhance communication between clinicians and AYAs and between PCPs and specialists. Both PCPs and adolescent medicine specialists suggested that e-consultations could increase centralized communication from the PCP to their patients, which was seen as promoting high-quality and coordinated care within the patient-centered medical home. e-Consultations were further seen as benefiting AYA health care by facilitating immediate and ongoing communication between PCPs and adolescent medicine specialists in a way not achieved through traditional referral methods [
AYA-specific e-consultation templates were also seen as a means of enhancing the quality of care for AYAs. Perceived opportunities to improve AYA care delivered by PCPs included placing prompts in PCP templates for patients’ names and pronouns—information integration to the sensitive and respectful care of AYAs that may otherwise not be elicited, may not align with EHR documentation, and which AYAs may be reticent to disclose without such signals from providers that they are in a safe space [
The ability to refine the minimum required documentation to balance efficiency and precision in e-consultations shows the value of iterative user-centered design in this context. Using a multiphase development process including think-alouds, exit interviews, and surveys with representative users, we elicited increasingly specific feedback that facilitated the first broad updates to the content and layout of the e-consultation prototypes and later finer changes to the template language. By engaging users who were both generalists and specialists, we were able to increase the alignment of PCP and adolescent medicine specialist perceptions throughout time through iterative modification with input from both sides. Such user-centered design has the potential to increase the uptake and acceptability of new health information technologies [
The user-centered design process outlined here is both a strength and a limitation of this study. Our study included a small number of participants from a single geographic area and practice within a specific clinical context. This allowed the creation of a tailored e-consultation system that may be readily implemented within our local health system but which may lack generalizability or acceptability for other health systems broadly, including those without compatible EHR systems across general pediatric and adolescent medicine specialty practices (although the lessons learned about needed content in these notes could apply to other systems as well). As a result, the themes identified from this development process should be viewed as starting places for conversations in systems other than final recommendations. We also note that the years in practice for PCPs and adolescent medicine specialists varied from phase 1 to phase 2. As this analysis focused on design rather than the implementation of e-consultations, it does not include the evaluation of quality or outcomes of use. Although e-consultations were perceived to have high usefulness and usability during the design process, new concerns may arise during implementation [
In conclusion, we used an iterative user-centered design process to develop and refine e-consultations for use by general pediatric PCPs and adolescent medicine specialists. We used qualitative analysis of user feedback elicited during the design process to identify themes relevant to the development and implementation of similar e-consultation systems for use in other health systems or among other patient populations. By engaging experts in the care of children and adolescents in the design process, we identified both opportunities and risks related to confidentiality, communication, and the use of tailored documentation prompts that should be considered in the development and use of e-consultations with AYAs.
Final templates for adolescent medicine e-consultation requests and responses.
adolescent and young adult
electronic health record
primary care physician
This work was supported in part by a career development award from the National Institute of Child Health and Human Development (grant K23HD088642, KNR). This project was also supported by the National Institutes of Health (grants UL1 TR001857 and T32HD087162-05). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.
None declared.