This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on https://humanfactors.jmir.org, as well as this copyright and license information must be included.
The availability of mobile clinical decision support (CDS) tools has grown substantially with the increased prevalence of smartphone devices and apps. Although health care providers express interest in integrating mobile health (mHealth) technologies into their clinical settings, concerns have been raised, including perceived disagreements between information provided by mobile CDS tools and standard guidelines. Despite their potential to transform health care delivery, there remains limited literature on the provider’s perspective on the clinical utility of mobile CDS tools for improving patient outcomes, especially in low- and middle-income countries.
This study aims to describe providers’ perceptions about the utility of a mobile CDS tool accessed via a smartphone app for diarrhea management in Bangladesh. In addition, feedback was collected on the preliminary components of the mobile CDS tool to address clinicians’ concerns and incorporate their preferences.
From November to December 2020, qualitative data were gathered through 8 web-based focus group discussions with physicians and nurses from 3 Bangladeshi hospitals. Each discussion was conducted in the local language—Bangla—and audio recorded for transcription and translation by the local research team. Transcripts and codes were entered into NVivo (version 12; QSR International), and applied thematic analysis was used to identify themes that explore the clinical utility of an mHealth app for assessing dehydration severity in patients with acute diarrhea. Summaries of concepts and themes were generated from reviews of the aggregated coded data; thematic memos were written and used for the final analysis.
Of the 27 focus group participants, 14 (52%) were nurses and 13 (48%) were physicians; 15 (56%) worked at a diarrhea specialty hospital and 12 (44%) worked in government district or subdistrict hospitals. Participants’ experience in their current position ranged from 2 to 14 years, with an average of 10.3 (SD 9.0) years. Key themes from the qualitative data analysis included current experience with CDS, overall perception of the app’s utility and its potential role in clinical care, barriers to and facilitators of app use, considerations of overtreatment and undertreatment, and guidelines for the app’s clinical recommendations. Participants felt that the tool would initially take time to use, but once learned, it could be useful during epidemic cholera. Some felt that clinical experience remains an important part of treatment that can be supplemented, but not replaced, by a CDS tool. In addition, diagnostic information, including mid-upper arm circumference and blood pressure, might not be available to directly inform programming decisions.
Participants were positive about the mHealth app and its potential to inform diarrhea management. They provided detailed feedback, which developers used to revise the mobile CDS tool. These formative qualitative data provided timely and relevant feedback to improve the utility of a CDS tool for diarrhea treatment in Bangladesh.
Mobile technology has had a major impact on the rapid access and transfer of information globally. Today, it is estimated that >5 billion people have mobile devices, with over half of these being smartphones [
Although existing CDS systems enable health care professionals to leverage the benefits of technology and information for their clinical practice, many individual, institutional, and technological barriers affect the engagement of clinicians with these new technologies. For instance, in a study conducted in the United Kingdom, physicians found it difficult to integrate mobile CDS into their pattern of work, prompting them to seek alternative sources of CDS [
Such concerns are not limited to high-income countries (HICs). A cross-sectional study aimed at assessing smartphone medical app use among physicians in Ethiopia found that the perceived usefulness of the app was one of the most notable factors associated with medical app use by physicians along with attitude, internet access, technical skills, and information technology support staff [
Despite the potential to transform health care delivery, much still remains unknown about the clinical usability of mobile CDS from a provider perspective, especially in LMICs. As such, the aim of this study is to describe providers’ perceptions about the utility of a mobile CDS tool that integrates predictive models for dehydration assessment in patients with acute diarrhea in Bangladesh and to seek their feedback on the preliminary components of this CDS tool. Qualitative data were gathered through focus groups, with physicians and nurses working in diverse clinical settings, including specialty research and general public hospitals. In consulting with clinicians, we hope to better adapt, and increase user confidence in, the predictive models and treatment recommendations provided by this tool. By seeking feedback on the tool’s layout and design to ensure that it fits appropriately into different clinical contexts, this formative qualitative research better enables the designers to build a CDS tool that anticipates and addresses the aforementioned barriers.
Qualitative data were collected in a series of focus group discussions (FGDs) from November to December 2020 among clinicians working at 3 distinct hospitals in Bangladesh as part of the
Owing to travel restrictions imposed by the COVID-19 pandemic, all data were gathered using a web-based platform (Zoom; Zoom Video Communications, Inc). Data were collected from clinicians working at three hospitals in Bangladesh: (1) the International Centre for Diarrhoeal Disease Research, Bangladesh’s (icddr,b) Dhaka Hospital; (2) Narayanganj General Victoria District Hospital; and (3) Shaheed Ahsan Ullah Master General Hospital (also known as Tongi Upazilla or Subdistrict Hospital). icddr,b’s Dhaka Hospital is a 350-bed, not-for-profit international research hospital specializing in the treatment of diarrheal illnesses and providing clinical services at no charge to over 100,000 patients with acute diarrhea a year from a catchment area of over 17 million people from the city of Dhaka and its nearby rural districts [
Appropriate rehydration with oral and intravenous fluids is the most important treatment for acute diarrhea and requires an accurate assessment of dehydration level [
The NIRUDAK app was developed to incorporate several clinical diagnostic models, derived using logistical regression, for assessment of dehydration severity in patients with acute diarrhea aged >5 years (full and simplified NIRUDAK models) and in children aged <5 years (DHAKA [Dehydration: Assessing Kids Accurately] score) [
Screenshots of the NIRUDAK app’s input and output screens shown to the participants during focus group discussions. (A) The input screen illustrates where clinicians enter relevant data based on a clinical assessment. (B) The output screen displays the patient’s dehydration severity level and fluid deficit as well as targeted recommendations for rehydration. NIRUDAK: Novel, Innovative Research for Understanding Dehydration in Adults and Kids.
A total of 8 focus groups were conducted. Of the 8 focus groups, 2 (25%) were conducted with clinicians from each of the district and subdistrict hospitals, one with nurses and the other with physicians. Moreover, of the 8 focus groups, 4 (50%) were conducted with icddr,b providers, 2 (25%) with physicians and 2 (25%) with nurses. Each focus group ranged from 2 to 4 participants, with most focus groups being conducted with 4 participants. The number of participants per focus group was deliberately kept low, in keeping with best practices for remote focus groups [
All FGDs were conducted using a web-based platform (Zoom) in Bangla and facilitated by a member of the Bangladesh-based research team. Facilitators used a written focus group agenda, which ensured that all groups were facilitated similarly, and all participants were asked the same series of questions. The agenda asked about the current use of mHealth tools and then presented a standardized case of a patient with diarrhea. Clinicians were asked to identify the patient data essential to determining diarrheal treatment, including rehydration. Facilitators then showed a short video of a prototype app, demonstrating its key features, input screens, and components. Still images of each screenshot were then shown; participants were asked for feedback on input and output screens and to choose between different possible layouts and models for the final mHealth app. Most of the focus groups were approximately 1-hour long, and typed transcripts ranged from 26 to 29 pages. Each discussion was audio recorded for transcription and translation by the local research team, which included the facilitators.
Transcription and translation were conducted in multiple steps and took approximately 8 weeks to complete. First, audio recordings were transcribed into Bangla. Next, another team member reviewed the audio and Bangla transcript to ensure accuracy and that all data had been deidentified. The Bangla transcripts were then translated into English by a research team member proficient in written and spoken English. The English transcripts were reviewed in Bangladesh by a third team member for accuracy. Finally, a US-based research team member read each English transcript to determine if any further clarification was needed. The Bangladesh-based research team reviewed and resolved all translation clarification requests. Once these were addressed, the English transcripts were considered finalized and used in the coding and data analysis process.
The research team used applied thematic analysis, a rigorous yet inductive approach designed to identify and examine themes from textual data [
In stage 1, coding structures were derived inductively as themes and repetitions emerged from reading the first 3 transcripts line by line and systematically categorizing emergent codes. A codebook was created to index and define each emergent code. An audit trial was used to document the iterative process of consolidating and creating emergent codes. In stage 2, the 8 transcripts were independently coded by 2 analysts who then met to compare codes and resolve discrepancies to establish intercoder agreement. Transcripts and codes were entered into NVivo (version 12; QSR International) for analysis [
Next, summaries of concepts and themes were generated from reviews of the key aggregated coded data. Coding summaries report participant comments for relevant codes, tracking the number of comments and the distribution of the data among participants. Data in the code summaries were organized by clinician category (nurse or physician) and hospital type (specialty, district, or subdistrict) for easy comparison of similarities and differences among the participants in those categories. Thematic memos, which gathered data from several summaries into key topic areas, were then written and used for the final analysis.
Five qualitative team members participated in the analysis: three coders (the study project coordinator [MG]; a master’s level analyst [RL]; and one of the coinvestigators, a PhD anthropologist [RKR]), who were supported by two coinvestigators with experience in treating diarrhea in Bangladesh (SCG and SN). All data and memos were interpreted in collaboration with both US- and Bangladesh-based research team members and with the principal investigator (ACL) and app designer (EJN; both doctors of medicine with extensive global health and diarrheal disease expertise) for the purpose of identifying themes that explore the clinical utility of an mHealth app to assess dehydration severity in patients aged >5 years with acute diarrhea.
Ethical approval for the formative phase of the NIRUDAK study was obtained from the icddr,b (PR-20048) and the Lifespan-Rhode Island Hospital (1624612) institutional review boards.
In total, 8 focus groups were attended by 27 participants. Of these 27 participants, 14 (52%) were nurses and 13 (48%) were physicians; in addition, of the 27 participants, 15 (56%) worked at icddr,b Dhaka Hospital and 12 (44%) worked in government district and subdistrict hospitals. The participants’ experience in their current position ranged from 2 to 14 years, with an average of 10.3 (SD 9.0) years. Additional demographic information is shown in
Several key themes emerged from the qualitative data analysis: current experience with CDS, overall perception of the app’s utility and its potential role in clinical care, feedback on specific app details, barriers to and facilitators of app use, considerations of overtreatment and undertreatment, and guidelines for the app’s clinical recommendations. We present these themes in our results and consider the implications of each for the CDS tool development in the discussion that follows.
Baseline demographics (N=27).
Characteristics | Values | |||
|
||||
|
25-34 | 6 (22) | ||
|
35-44 | 16 (59) | ||
|
45-54 | 4 (14) | ||
|
55-64 | 1 (3) | ||
|
||||
|
Men | 9 (33) | ||
|
Women | 18 (66) | ||
|
||||
|
|
14 (51) | ||
|
|
Diploma | 5 (35) | |
|
|
Bachelor’s degree | 4 (28) | |
|
|
Master’s degree | 5 (35) | |
|
|
13 (48) | ||
|
|
MBBSa | 8 (61) | |
|
|
Master’s degree | 5 (38) | |
|
||||
|
10,001-50,000 BDT (US $116-580) | 7 (25) | ||
|
50,001-100,000 BDT (US $581-1160) | 6 (22) | ||
|
>100,000 BDT (>US $1160) | 14 (51) | ||
Experience in current position (years), mean (SD) | 10.3 (9.0) | |||
|
||||
|
icddr,bc | 15 (55) | ||
|
Tongi Upazilla Subdistrict Hospital | 7 (25) | ||
|
Narayanganj General Victoria District Hospital | 5 (18) |
aMBBS: Bachelor of Medicine, Bachelor of Surgery (a degree for physicians in Bangladesh).
bAt the time of the focus group discussion, the US dollar to Bangladesh taka exchange rate was US $1=83.3 BDT.
cicddr,b: International Centre for Diarrhoeal Disease Research, Bangladesh.
Many, if not all, icddr,b participants (both physicians and nurses) indicated that they had previous experience with CDS and web-based tools. icddr,b clinicians had used SHEBA, an integrated, computerized, and paperless hospital information management system that has been in use since 2009 [
Most participants were enthusiastic about the NIRUDAK app. Participants felt that it would take some time to learn to use the app but that once learned, the app would be easy to use and have the potential to ease or decrease their workload over time. A participant noted that when they started practicing medicine, patient assessment was tracked on paper and is now being tracked by computer. They noted the following about using this app:
[Using the app] is a matter of time, also learning. We want to assess [dehydration] with less things. By this I mean...having less buttons or features so our work will be easier. If that can happen, that will be good for us.
To start using a new thing, initially some problems will occur, which is normal. But it’s an excellent app. If anyone uses [this app], it definitely will be beneficial, and it’s easy. Not only doctors but also nurses can use it comfortably and, in many cases, our [work] load will decrease. Yes, [the app] is excellent, I think, it can be used.
Confidence in the app was explicitly discussed in 2 FGDs with nurses. In these FGDs, the nurses reported enthusiasm for the app and felt confidence in use of the app: “You can send this app to any end of the world without hesitation” [specialty hospital nurse].
Several participants noted that the app and its recommendations should not exist in isolation from the clinician and could not replace clinician experience. For instance, participants noted that clinician decision-making for a patient with diarrhea may be more complex than solely determining the level of dehydration. Other comorbidities, such as diabetes and electrolyte imbalances, are important to recognize and may impact the management of diarrhea and dehydration versus the standardized output from the app. Several participants felt that clinical experience could be relevant to decide when the app use was appropriate; for example, clinicians must first make a determination that patients have dehydration versus sepsis in which fluid management strategies may differ substantially: “[If] I give the total [amount of] fluid [recommended by the app] for a patient with severe dehydration and [the patient also has] sepsis, in that case it will be detrimental for the patient
During the focus group, participants reviewed screenshots of the NIRUDAK app prototype, including images of the input and output screens. Here, we present comments from our participants about the following three specific app elements: age, danger signs, and the fluid deficit bar used in treatment recommendations. Each element is described in the next sections, with associated participant comments.
Patient age is one of the first characteristics on the app input screen (
The input screen also includes several specific danger signs. Temperature, entered in either degrees Celsius or degrees Fahrenheit, is recorded by choosing one of three radial buttons: <35, normal, and >37.9. A total of two
The danger sign output screen provides algorithm-based recommendations using the input data. For example, when high fever is present, the app recommends “Check for sepsis or other causes.” If the patient is vomiting and unable to drink, the app recommends “Space ORS sips or use IV fluids” (see the example in
The NIRUDAK app assesses dehydration according to what is entered in the dehydration assessment section of the input page (
...
Using the information provided in the dehydration assessment section of the input screen, the output screen indicates whether the patient has
Yes, definitely. Yes, obviously [it] will be helpful because I am getting it absolutely readymade. I do not have to think that much…This is excellent, isn’t it? Excellent, nothing else, absolutely first class.
We asked participants what they thought it would be like to use the NIRUDAK app in clinical practice, who should use the app tool, and what would support app use. Participants identified a variety of potential barriers to and facilitators of app use.
Barriers to use included the time it could take to train clinicians to use the app and the requirement for MUAC and systolic blood pressure (SBP) measurements. MUAC and SBP are used in calculating treatment recommendations in the full NIRUDAK model; however, participants noted that some clinical environments do not have MUAC measurement tape and blood pressure cuffs available:
...many times...digital blood pressure for children are not available. In that case, getting these two things [MUAC, SBP] accurately will be a bit difficult if I want to use it for mass population...I think the things [MUAC, BP] are good, but to use in mass population is a bit difficult.
Similarly, it is not always possible to know if the patient has any medication allergies or another required field, particularly if they are nonresponsive.
Factors that ease or facilitate app use included clinicians’ current experience with and use of other web-based tools, including local electronic health record systems, and familiarity with touch screens and other clinical apps:
As we are used to using the SHEBA app, in that case for us, it will not take so much time [to learn how to use app]. But, at the community level, [there may be patient] rush over there or [limited] manpower. In that case, for them, [app use] might face some problems.
Recognizing that the app could be used in a variety of contexts, we asked what it would be like to use the app via telemedicine during a cholera outbreak and if community health workers (CHWs) could use it. There was a difference of opinion about whether the app could be incorporated into telemedicine. Some participants felt telemedicine use was possible, whereas others cautioned that important symptoms, such as sunken eyes, cannot be properly assessed via telemedicine. Many participants saw the app as useful for quick assessment and diagnosis, relatively easy to learn, and not requiring too much time to use, all of which would make it particularly useful during outbreaks. In contrast, some participants noted that time is further limited by the high numbers of patients requiring treatment during a cholera epidemic. In addition in that context, intravenous rehydration is usually started immediately upon arrival. As such, once rehydration has begun, there could be less need and time for assessment via the app, especially if MUAC and SBP measurements were required. The ability to calculate the recommended treatment without MUAC and blood pressure fields was considered a useful option for this context.
Opinion was divided on whether CHWs could use the app as a CDS tool. Although several participants, of which many were nurses, felt that CHWs would be able to use the app, others—often physicians—cautioned that clinical experience was essential and could be a limiting factor to nonphysician use of the app. Those who felt CHWs could use the app noted that it would make treatment decisions easy:
...to input information will be very easy. It may take 1 minute, or 2 minutes, or 3 minutes. Therefore, if health workers are trained, I think that they all can use this app...to make treatment decision.
This could be a benefit in areas where physicians are less available. Caveats to CHW use included a concern that they would need to be trained to avoid mistreatment and to specifically observe the danger signs that indicate when hospital-based treatment is necessary.
Participants emphasized the relevance and importance of considering the end user’s clinical expertise and judgment in two main ways: first, clinical experience is needed to support the use of the app through the accurate assessment of clinical signs, and second, the importance of avoiding overreliance on the app for clinical decision-making. Participants recognized that clinicians with varying levels of experience may interact differently with the app and that those with less experience may not recognize situations in which the app is less accurate or when there may be a degree of subjectivity (eg, assessment of clinical signs, such as sunken eyes). Similar comments were made by participants about the use of other formal clinical guidelines, which were felt to be primarily used by less experienced clinicians, whereas more experienced clinicians do not rely on guidelines as heavily: “We really do not treat people by [only using] guidelines in front of us, we [also] use our clinical judgement” [government hospital physician]
Physicians were shown the WHO’s Integrated Management of Adolescent and Adult Illness (IMAI) guidelines for the classification of dehydration [
Physicians were also provided with data about the likelihood of correct treatment, overtreatment, and undertreatment of dehydration using the WHO’s IMAI algorithm, and with 3 possible prediction cut points for classifying patients with severe dehydration in the NIRUDAK models (
Overtreatment or undertreatment diagram presented to participants during focus group discussions. IMAI: Integrated Management of Adolescent and Adult Illness; NIRUDAK: Novel, Innovative Research for Understanding Dehydration in Adults and Kids; WHO: World Health Organization.
In general, participants indicated that it is important to avoid both undertreating and overtreating patients. In discussing the importance of not missing cases of severe dehydration, the physicians listed the subjective nature of some of the signs and symptoms used by the app and noted that these differed among very young, middle-aged, and older participants.
Physician preference was generally split between options 1 and 2. There were also no differences between physicians from the two settings regarding option choice; 3 doctors from icddr,b and 4 from government hospitals preferred option 1, whereas 2 physicians from each setting preferred option 2. In addition, 2 participants declined to make a selection or argued that although missing a patient with severe dehydration is problematic, because the patient may die, overtreatment can cause loss and damage as well: “over treatment is as perilous as [being] dehydrated
Participants in the 8 focus groups were mostly enthusiastic about the NIRUDAK app, a novel CDS tool for diarrheal management in low-resource settings. They highlighted the potential for time saving and the utility of the product during high-volume patient periods, such as during a cholera outbreak. Participants’ opinions about key components and the barriers to and facilitators of app use were shared with the coinvestigators and the app design team and informed the next stage of the app development.
Factors that support the use of the NIRUDAK app include clinicians’ familiarity with, and current use of, other mHealth tools, including a touch screen or haptic electronic medical record and other phone-based clinical apps. Notably, discussions about the anticipated utility of the app often occurred during FGDs with nurses, which could reflect a greater role that CDS tools may have for nurses than for physicians. This is an important consideration for the scale-up of the use of the app, given that a large number of patients with acute diarrhea and other common illnesses in LMICs are attended to by nurses or nonphysician health workers working in health centers rather than by physicians in hospital settings [
However, several physicians, cautioned against the use of the app by those with no or little formal clinical training, such as CHWs, versus those with formal clinical training, such as nurses and physicians. Such concerns were due to the possible misuse of the app or that CHWs may not be able to adapt the app’s recommendations to unique cases. Further guidance and training in the use of the app and assessment of clinical signs may be important for the implementation of the app among CHWs and is in line with other recommendations that improving knowledge and skills is essential to improving the quality of care provided in LMICs [
The NIRUDAK app uses patient data to provide rehydration and other treatment recommendations based on whether the entered information indicates that the patient has no, some, or severe dehydration. The focus group questions about participants’ use of existing treatment models, specifically the WHO’s IMAI and the DHAKA method, were designed to help the researchers understand clinicians’ opinions and treatment practices and to guide the models used in the NIRUDAK app itself. Of note, the overtreatment and undertreatment models shown in
The app developers made several other changes based on focus group participants’ feedback (
Participant comments about the varying contexts of use and the varying needs of users indicate that this decision support tool for diarrhea treatment will be used differently depending on the clinician’s role and the clinical context. For nurses and CHWs, the treatment recommendations provided through the app may be directive or proscriptive. For physicians and other advanced practitioners with significant diarrheal treatment experience, the app will support their clinical experience and judgment. Allowing clinicians to adjust the app settings, choosing between a more sensitive, more specific setting and the default setting both preserves the physician autonomy and allows for flexibility, making the app more generalizable to different clinical contexts, including when cholera is epidemic. In addition to the design choice allowing users to flexibly adapt the sensitivity and specificity of treatment recommendations, information tabs, which provide details on the model used, have been added to the app.
Screenshots of the NIRUDAK app’s (A) input screen and (B) output screen after participants provided detailed feedback on the app. NIRUDAK: Novel, Innovative Research for Understanding Dehydration in Adults and Kids.
These qualitative data about the clinical utility of the NIRUDAK app come from focus groups in which participants were shown still images of the app prototype. Consequently, feedback is limited to opinions about the appearance and content of the app; it is not based on actual use. Although conducting FGDs virtually using the Zoom platform allowed for the completion of data collection during the COVID-19 pandemic, poor internet connectivity prevented 19% (5/27) of the participants (4/5, 80%, were from government hospitals) from attending the focus groups, which may have influenced the findings. Meanings may have been lost in the translation process or interpretation of the data during the analytic process and could have introduced biases. However, the involvement of >1 researcher during the transcription, translation, and coding processes minimizes the likelihood of misinterpreting research findings. In addition, this study was conducted in urban or semiurban hospital settings. Future work should focus on evaluating an mHealth app’s clinical utility in rural or outpatient or ambulatory settings, as well as in other countries, and would be especially valuable.
The NIRUDAK app has been revised based upon formative qualitative data, which have contributed to the app’s development and programming. The current iteration has been programmed with several features that were influenced by focus group participant feedback, including an option for clinicians to change between 2 different dehydration treatment models. The NIRUDAK app will be field-tested at icddr,b in 2022 to validate those models. Additional qualitative data will also be collected via individual interviews with nurses and physicians who field-test the app to further evaluate NIRUDAK’s usability and understand the clinical users’ experiences.
Predictors included in full and simplified Novel, Innovative Research for Understanding Dehydration in Adults and Kids models [
breaths per minute
clinical decision support
community health worker
Dehydration: Assessing Kids Accurately
focus group discussion
high-income country
International Centre for Diarrhoeal Disease Research, Bangladesh
Integrated Management of Adolescent and Adult Illness
low- and middle-income country
mobile health
mid-upper arm circumference
Novel, Innovative Research for Understanding Dehydration in Adults and Kids
systolic blood pressure
World Health Organization
The authors would like to thank all the participants and study staff at the International Centre for Diarrhoeal Disease Research, Bangladesh, Dhaka Hospital, for their help and support. Funding was provided through grants from the National Institutes of Health, National Institute for Diabetes and Diarrheal and Kidney Diseases (principal investigator: ACL; grant 1R01DK116163-01A1). The funders had no role in the study design, data collection, or reporting processes. The content of this manuscript is solely the responsibility of the authors and does not necessarily represent the views of the National Institutes of Health or any governmental bodies or academic organizations.
RKR, EJN, and ACL contributed to the study concept. RKR, ACL, SCG, MG, SN, and EJN contributed to the study design. SS, TH, SN, MG, RKR, NE, and SCG contributed to the data collection. SS, TH, and SN contributed to focus group discussion transcription and translations. SN, MG, and RL contributed to transcript editing. RKR, SCG, MG, and RL contributed to data analysis and the drafting of the manuscript. All authors edited and approved the final draft of the manuscript.
None declared.