Evaluating the Feasibility of a Software Prototype Supporting the Management of Multimorbid Seniors: Mixed Methods Study in General Practices

Introduction

Background

Health care management of seniors (persons aged ≥65 years) is complex as more than 55% are multimorbid [1]. Electronic health records (EHRs) have the potential to support care management, but were shown to insufficiently support longitudinal patient-centered management as they (1) require high user-system interaction and work slow, (2) lack user-friendliness and orientation to typical health care processes, (3) provide insufficient interoperability, (4) offer insufficient service if help is needed, and (5) inadequately distinguish between information relevant and irrelevant for patient care [2-7]. Surveys evaluating the use and functional capacity of EHR in German primary care revealed a high implementation rate of EHR systems but little multifunctional capacity, especially with regard to reminder and recall systems [2,6,8,9].

Considering that the demographic change will lead to an increasing number of seniors and complex, multimorbid patients, health information technologies (IT), which adequately support health care management, are needed. EHR add-ons such as clinical decision support systems (CDSS), including reminder systems, can provide targeted support as they were shown to effectively support management of, for example, diabetes or hypertension [10-14]. Limiting the benefit of such disease-specific CDSS, a survey among primary care physicians revealed that physicians are more willing to accept new IT solutions, when supporting the management of elderly patients with multiple conditions or polypharmacy [15]. In addition, many systems do not adequately meet physicians’ needs, even though an early involvement of the target group into software development is described as key facilitator for software acceptance [16-18] and is therefore highly recommended [19].

Objectives

To overcome deficits of the current EHR solutions, we developed a new software add-on for German EHR systems. The add-on named eCare*Seniors aims to provide a quick and intuitive overview of each patient’s care needs to adequately support comprehensive and longitudinal patient-centered care management not only for multimorbid seniors in general practices, but for other patient groups as well [20,21]. To provide a quick overview of each patient’s care needs, relevant information is extracted from a patient’s EHR and reedited using so-called flags, which are a combination of colored fields and short keywords (Figure 1) [20,21]. The theoretical concept is based on the chronic care model [22-24], whereas from a functional perspective, eCare*Seniors is classified as a CDSS, integrating the functions of a reminder, advisor, and critic [25-27]. Further details on the new add-on and its functions are published elsewhere [20].

‎View this figure

Following the recommendation for the development of health IT [19], eCare*Seniors was developed using a user-oriented, stepwise approach. First, an assessment of the status quo showed that German general practitioners (GPs) and practice assistants (PAs) are not satisfied with current EHR solutions and had started to self-design reminder systems within the EHR to maintain an overview of each patient’s care [5,28]. In a second step, we presented the concept for the new software add-on to GPs and PAs who generally welcomed the approach but expressed a desire for configuration options to adapt the software to their individual practice needs. On the basis of this target group information, we then implemented the software prototype in a third step. In this paper, we present the results of the feasibility study of the new software add-on, which was the fourth step of the development: the software was tested in GP practices to assess the feasibility from both a technical and users’ perspective.

Methods

Study Design

The feasibility study was conducted using a simultaneous mixed methods design involving qualitative and quantitative approaches [29]. According to recommendations on how to design feasibility studies, we focused on the 5 key criteria, integration, implementation, acceptability, demand, and practicability [30], which we subdivided into 2 categories:

1. Technical feasibility: Integration of the new add-on into existing IT infrastructure and implementation of the new add-on with regard to its success or failure of execution during the work process.
2. Feasibility from the users’ perspective: Demand for and acceptance of the new add-on within the target group, and assessment of the usability from the users’ perspective. Usability as one of the most important indicators for feasibility was measured using the validated System Usability Scale (SUS) [31,32].

Setting and Recruitment

As recommendations on how to perform usability studies recommend sample sizes of 3 to 20 participants and the SUS was even shown to provide consistent results in study groups of 12 persons [33,34], we targeted a sample size ≥12 participants. The study was performed in a convenience sample of GP practices of the practice networks of the Institute for General Medicine, University of Duisburg-Essen, Germany, and the Institute for Family Medicine and General Practice of the University of Bonn, Germany (both >100 practices), with 1 GP and 1 PA each. Owing to the lack of a standardized, open interface of existing EHRs, a second software was used to extract the data needed for our new software add-on. To assure this connectivity to existing EHRs, only practices that apply one of 13 specific EHR solutions, which are used by about 85% of all German GP practices [35], were eligible to participate. As the new software is meant to be usable and feasible for every practice, no further inclusion and exclusion criteria were defined. Considering a participation rate of at least 40%, 30 participants from 15 practices who met the inclusion criteria and were located in the Greater Essen region, Germany, or those who already participated in an earlier step of the study were contacted and invited to participate.

Data Collection and Data Management

All participating GP practices were visited twice by a research team member of the institute. During the first visit, the researcher installed the prototype of the newly developed add-on eCare*Seniors. During the second visit, the new add-on was introduced to the practice and tested by 1 GP and 1 PA per practice with patient data. To evaluate the technical feasibility, all problems that occurred during the installation process and when executing the new add-on during the test were documented. To evaluate the feasibility from the users’ perspective, data were collected using a prestructured interview guide for documentation of the software test and a self-administered questionnaire, which was completed by each participant after the software test:

(1) Documentation of the software test: First, the GP was asked to configure the software according to the practice’s needs. The preferences were documented using a checklist. Afterward, the software was tested independently by the GP and the PA. Both accessed 4 complex patients (defined as seniors aged 65 years or above with 2 or more chronic conditions) within the prototype of the new add-on and within the practice’s EHR and compared both systems based on the following questions:

• Is there any information on this patient you would have forgotten had you not used the new add-on? (yes/no)
• Does the new add-on display any information at first glance which you do not see instantly in the patient’s EHR? (yes/no)
  • What information? (open answer)
  • Is this information important? (yes/no)
• Is there any information available at first glance in the patient’s EHR which you do not see instantly in the new add-on? (yes/no)
  • What information? (open answer)
  • Is this information important? (yes/no)

To complete the test, GPs and PAs clicked through the remaining menu of the software add-on and applied the method of thinking aloud to help the researcher identify problems regarding comprehensibility and handling.

(2) Written questionnaire including the following aspects:

• German version of the validated SUS to evaluate the usability of software solutions based on 10 questions on a 5-point Likert scale [31,32]
• Two open-ended questions to request which aspects of the software were liked most and which aspects needed to be optimized
• One close-ended question to assess whether participants would choose to use the new add-on long-term
• Five questions comparing the new software add-on with the current practice-specific care management strategies to assess the aspects of quick overview, reminder functionality, support of individual, guideline-oriented care planning, support of time-efficient processes, and financial benefits on a 5-point Likert scale
• One open-ended question to assess how much money (in Euros) the potential users would be willing to pay for such software per month.

In addition, sociodemographic characteristics of the participants were assessed.

All data were entered manually in an access-restricted database in IBM SPSS Statistics for Windows, version 24 (IBM Corp).

Data Analysis

Following an approach of Creswell and Plano Clark for integrating quantitative and qualitative data during the analysis process, we merged the data by transforming qualitative into quantitative data using categorization [29]. After that, all data were analyzed using descriptive statistics in SPSS. The usability was analyzed by determining the mean SUS score. The score can assume values between 0 and 40. Results were interpreted in percentage, that is, each score was multiplied by the factor 2.5 [31,32]. A SUS score ≥70% denotes good usability, whereas a score ≤50% indicates a considerable need for improvement [32].

Ethical Considerations

Ethical approval was obtained from the Ethics Committee of the Medical Faculty of the University of Duisburg-Essen (reference number: 14-5980-BO; date of approval: June 1, 2015). All participants received written information and signed the informed consent forms, which are stored at the institute. All members of the practice network had provided written informed consent for the pseudonymized analysis of data on practice characteristics within the scope of scientific research of the institute.

Results

Study Characteristics

Of the 30 participants from 15 practices invited, 6 participants from 3 practices refused to participate, yielding an initial response rate of 80% (24/30). As detailed below, 3 practices with 6 participants willing to participate used an operating system, which was not supported by the new software (32-bit instead of the required 64-bit). Finally, 18 participants (9 GPs, 9 PAs) from 9 practices took part. The practices used 4 different EHR software solutions covering about 26% of all EHR solutions installed in German GP practices [35]. All practices participated with 1 GP and 1 PA each (N=18). Furthermore, 5 of these practices were group practices. The GPs were aged between 40 and 56 years (mean 47.4, SD 5.2); 8 of them were male. All PAs were female and aged between 22 and 60 years (mean 36.7, SD 14.3). All participants tested the software add-on with 4 patient charts, yielding a total of 72 patient tests.

Technical Feasibility

The following technical problems were observed during the installation processes:

• As mentioned above, the new software could not be installed in 3 practices as they used a 32-bit operating system rather than the required 64-bit system.
• Installation was delayed in 4 practices as their Microsoft .NET program, which is required to run the desktop application of our software, needed to be updated.
• One of the practices did not have sufficient administration rights to install the software. To continue the installation process, access data had to be requested from the responsible software publisher.

Once these installation problems were resolved, the software prototype worked without problems. No technical problems occurred when executing the software during the practice tests. Nevertheless, in all 9 practices, few flags were not displayed in eCare*Seniors. In all 9 practices, the present year was named as due date in all flags on upcoming vaccinations. Compared with the information available within the patient charts, flags on chronic medication were incomplete in 4 practices, those on chronic diseases in 3 practices, and those on laboratory values in 1 practice. As all these flags were completely and successfully implemented in tests conducted during the software development, these problems are likely because of connectivity problems of the second software used to extract EHR data.

Feasibility From the Users’ Perspective

During the practice test, each participant compared the information displayed at first glance in the practice’s EHR with the flags presented by eCare*Seniors. For 69 of the 72 (96%) patient charts selected for the test, participants stated that information which they consider important for comprehensive care management was available at first glance only in eCare*Seniors but not in their EHR (Table 1).

During the interviews alongside the chart tests, for 54 of the 72 (75%) patient cases tested, participants named some additional information which they would like to be presented in eCare*Seniors. The researchers judged the following information as reasonable to be evaluated for integration into a future version of eCare*Seniors:

• More detailed information on chronic diseases, for example, renal impairment in diabetic patients (mentioned in 14 of the 54 patient charts, 26%)
• More space for freely formulated notes and risks (12/54, 22%)
• More detailed information on allergies (11/54, 20%)
• More detailed information on chronic medication (9/54, 17%)
• Residential care/home visit patient (8/54, 15%)
• More detailed information on physical parameters such as height and weight (5/54, 9%)

Other wishes mentioned need be evaluated with regard to technical implementability (eg, the availability of an electronic medication plan or the date of the next disease management program examination).

Furthermore, 8 GPs and 8 PAs each (16/18, 89%) specified one or more aspect of the new software they liked most (Figure 2) and 9 GPs and 7 PAs (16/18, 89%) specified one or more aspect which needed to be optimized (Figure 3).

‎View this figure

‎View this figure

The results of the questionnaire survey showed that 14 of the 18 participants (78%; 7 GPs and 7 PAs) wished to use the new software long-term. In addition, 3 GPs even stated that they would implement and use the software immediately, if it was linked directly to the EHR software and did not require a second software for bridging the interface. Despite its design to support care management of seniors, the configuration options of eCare*Seniors allowed for applying the software for other patient groups as well. This option was selected by 7 of the 9 practices; 1 practice extended the use to patients aged ≥35 years, 6 practices to all patients. eCare*Seniors was rated positively in comparison with existing patient management strategies of the individual practices (Table 2). The 2 GPs and 2 PAs who did not express a desire to use the new software in the long term said they were already using a well-functioning system which offered the same functionality. All GPs except one were willing to pay a monthly license fee for the software, ranging from 5 to 25 Euros (mean 14.75, SD 5.93).

The average SUS score was 75% (SD 14%), varying from 43% to 95%. A total of 14 of the 18 participants (78%) rated the usability as ≥70%, which denotes good usability, although 1 participant (5%) rated the usability as <50%. Stratified for GPs and PAs, the respective SUS score was 78% (SD 16%; range 43%-95%) and 73% (SD 12%; range 53%-93%). Evaluation results on item level are illustrated in Table 3.

Discussion

Principal Findings

From the users’ perspective, the new EHR add-on meets its intended purpose; GPs and PAs positively emphasize the intuitive, quick, and comprehensive overview of relevant, patient-centered information provided by eCare*Seniors. Applying the feasibility criteria defined by Bowen et al [30], the new add-on is generally feasible; it meets the users’ demands, is accepted within the target group, has a good usability rating, and no technical problems are encountered during the implementation, as long as practices use a 64-bit operating system. Currently, the biggest challenge is the connectivity between the new add-on and the existing IT infrastructure. In perspective, this will be facilitated by standardized and open interfaces to EHR solutions, which are legally consolidated in Germany since 2017 [36].

In the standardized and validated SUS scale, users rated the usability of eCare*Seniors as 75%, which denotes good usability [32]. As no similar IT approaches were studied in German general practices, the usability cannot be compared with that of other IT solutions. In the literature, only few studies report on user-oriented usability evaluations of newly developed electronic tools supporting patient-centered care management. To a minor extent, comparable approaches are described in 4 studies from the United States and Canada. Furthermore, 3 of these studies assessed the usability of newly developed, EHR-integrated CDSS supporting the standardized, guideline-oriented therapy of patients with chronic pain [37,38] or arterial fibrillation [39] in samples of 4 to 12 GPs and PAs in real primary care scenarios; other than eCare*Seniors these systems only support disease-specific patient management rather than comprehensive, patient-centered management [37-39]. Similar to eCare*Seniors, an approach from the United States aims at priority-oriented restructuring of patient-centered information within the EHR to provide a quick overview of patient’s care needs. On the basis of a previous study on information needs, 4 options for information presentation were developed and tested by 16 physicians using fictive patient cases. However, to our knowledge, this concept was only realized as prototype and has not been transferred to EHR solutions thus far [40]. Despite the limited comparability, these studies report similar usability ratings based on the SUS scores or comparable items [37-40].

Although eCare*Seniors predominantly aims to support health care management of multimorbid seniors, most practices wanted to use it for all their patients. This shows that the demand is not limited to complex patients only, which might be explained by the fact that the current EHR solutions were described as inconvenient, requiring high interaction, and lacking reminders relevant for care [5]. Interestingly, this result contradicts a study by Sittig et al indicating that physicians prefer IT solutions supporting the management of elderly patients with multiple conditions or polypharmacy [15]. Nevertheless, it has to be considered that our approach aimed to overcome deficits of German EHR systems. This also makes it difficult to compare our approach with other studies on CDSS as most studies evaluated the effect of newly developed, often imposed systems instead of further developing existing IT solutions with regard to deficits and users’ needs [10-14].

Limitations

The key strength of the development and feasibility testing of eCare*Seniors is the user-oriented, bottom-up approach, which facilitated adequate consideration and realization of the users’ demands. In addition, the mixed methods design involved qualitative and quantitative elements allowing for a more detailed assessment of the users’ opinions than a strict quantitative approach. For feasibility testing, the study sample included 18 GPs/PAs and 4 patient cases per participant. Although this is within the range of sample sizes recommended for usability studies [33], the sample analyzed is too small to generate generalizable and reproducible results. Another aspect which limits the results’ generalizability and reproducibility is the fact that the SUS provides self-reported, subjective measures of usability and the study lacks objective usability measures. In addition, the results might be limited because of a selection and response bias. First, it cannot be excluded that the participating practices had a greater interest in new health IT or had a higher affinity for technology than those who refused to participate. Second, the responses of the participants might be positively influenced by the presence of a member of the research team. As GPs and PAs expressed their criticism ad hoc, this bias is considered negligible.

Conclusions

This feasibility study shows that the newly developed EHR add-on is well accepted and usable; however, the technical integration into present IT infrastructures of general practices could be facilitated. To determine whether the utilization of the software positively influences practice organization, patient care, and patients’ health outcomes, randomized controlled interventional studies are needed.