The study adhered to good practices for pragmatic usability assessment in clinical settings. The study is reported in accordance with the Good Reporting of A Mixed Methods Study (GRAMMS) recommendations [22], and the completed GRAMMS checklist is provided as Multimedia Appendix 1.

This study used a postimplementation, cross-sectional usability, and UX evaluation embedded within a longitudinal mixed-methods project. The use of mixed methods triangulation aligns with established approaches for evaluating complex health care interventions, combining quantitative usability metrics with qualitative contextual insight to enhance interpretive depth [23-25]. The evaluation combined standardized usability instruments with a project-specific Post-Study Neonatal Utility Questionnaire (PSNUQ) and semistructured key informant interviews (KIIs) to triangulate frontline users’ perceptions with contextual implementation data. The quantitative usability component reported here was administered after a 4-month implementation period of the paper-based neonatal risk predictor tool. The study followed good practice for pragmatic usability assessment in clinical settings [14].

This study represents the completed user-centered usability evaluation described in the previously published study protocol [21]. This paper reports the implementation and postimplementation usability findings from that protocol-defined study. Preintervention key informant interviews were conducted in August 2025, followed by implementation of the paper-based neonatal risk predictor and newborn data collection from August through November 2025. The postimplementation usability survey was administered in December 2025. The overall study design and mixed-methods usability evaluation framework are illustrated in Figure 1.

The ML–derived neonatal risk predictor tool was implemented as a paper-based documentation instrument across 3 Kenyan neonatal units over a 4-month period. Following implementation, usability and user experience were evaluated using a mixed methods approach that combined standardized usability questionnaires (System Usability Scale [SUS], Post-Study System Usability Questionnaire [PSSUQ], Questionnaire for User Interaction Satisfaction [QUIS], and PSNUQ) completed by frontline neonatal staff (n=10) with qualitative KIIs with neonatal unit leaders (n=3). Quantitative and qualitative findings were triangulated to assess usability, learnability, clinical utility, workflow integration, and implementation context.

The study was conducted in 3 Kenyan health facilities representing distinct service contexts: an urban informal settlement facility in Kibra (Nairobi County), a periurban public hospital in Matuu (Machakos County), and a faith-based referral hospital in Nazareth (Kiambu County). Facilities were located in Nairobi County (Kibra), Machakos County (Matuu), and Kiambu County (Nazareth). Contextual details and facility characteristics are described in the project report and site summaries.

The published study protocol specified a purposive sample of 10 health care workers across the 3 participating facilities, with a planned allocation of 4 participants from Kibera, 4 from Matuu, and 2 from Nazareth [21]. In the completed study, however, recruitment was based on the availability of eligible frontline neonatal health care workers during the implementation period. All eligible and available staff involved in the implementation were invited to participate, and 10 frontline staff completed the postimplementation survey.

The protocol’s prespecified allocation of 4 participants from Kibra, 4 from Matuu, and 2 from Nazareth was intended to ensure representation across the 3 participating facilities. The allocation was a planned recruitment target rather than a statistically derived proportional allocation. In the completed study, the final numbers of participants from each facility corresponded to these planned numbers, although recruitment was based on the availability of eligible frontline neonatal health care workers during the implementation period.

A total of 2 participant groups contributed to the usability evaluation:

All participants provided information and consent for the use of anonymized aggregate responses in research outputs.

The paper-based neonatal risk predictor evaluated in this study was derived from a previously developed and validated ML model for neonatal mortality prediction reported by Shukla et al [4]. The original model was developed using routinely collected maternal, obstetric, delivery, neonatal, and early postnatal variables and demonstrated good predictive performance in LMIC settings (area under the receiver operating characteristic curve >0.80). Rather than evaluating the predictive performance of the ML algorithm itself, this study focused on the feasibility, usability, user experience, and workflow integration of a paper-based implementation of the predictor set within routine neonatal care across 3 Kenyan health facilities.

The complete list of predictor variables and the paper-based neonatal risk predictor form used during implementation are provided in Multimedia Appendix 2.

The object of evaluation was a paper-based neonatal risk predictor tool (the “predictors set”) that had been integrated into routine documentation for newborns at delivery (Day 1) and at a postdelivery check (Day 2). The tool captured 21 candidate predictors, organized into 5 domains: maternal characteristics, pregnancy-related factors, delivery characteristics, neonatal clinical indicators, and early postdelivery observations. The complete list of predictors and the paper-based neonatal risk predictor form are provided in Multimedia Appendix 2 to enhance transparency and reproducibility. During the 4-month implementation period, the tool was used prospectively within routine neonatal care, resulting in 400 newborn cases recorded across the 3 participating sites: 100 at Kibra, 100 at Nazareth, and 200 at Matuu. These site-specific totals represent the cases captured during the implementation period and were not determined through a statistically derived per-site sampling allocation.

Ethical approval (SU-ISERC2949/25) was obtained from the Strathmore University Institutional Scientific Ethics Review Committee (SU-ISERC), Nairobi, Kenya, on 28 July 2025, prior to commencement of the study. All participants provided informed consent before participation after receiving information about the study aims, voluntary participation, confidentiality protections, and data management procedures. Interview transcripts were anonymized before analysis to protect participant confidentiality and institutional privacy. The study adhered to established ethical principles for qualitative health systems research throughout data collection, analysis, and reporting.

A total of 10 frontline neonatal health care workers completed the postintervention usability evaluation across the 3 study sites. The majority of respondents were mid-career nurses with diploma-level qualifications, reflecting the cadre primarily responsible for routine neonatal documentation and triage. The characteristics and study-site distribution of the participating frontline neonatal health care workers are summarized in Table 1.

Among participating frontline health care workers, overall perceptions of usability were generally favorable. Most respondents reported positive overall perceptions of the neonatal risk predictor tool's usability. Specifically, 75% (6/8) reported a willingness to use the tool frequently in routine practice, and 55% (5/9) indicated confidence in using it independently. Regarding perceived complexity, around half (5/10) disagreed that the tool was complex, while 22% (2/9) agreed. Together, these findings suggest the tool's overall acceptability, with moderate variability in users' perceptions of complexity.

PSSUQ subscales were computed using canonical scoring procedures (lower scores indicate better usability). Median scores across all subscales were below 3, indicating favorable perceptions of usability.

Information quality and interface quality showed slightly greater dispersion, reflecting variability in respondents’ perceptions of clarity and layout. Median PSSUQ subscale scores are presented in Table 2. All median scores were below 3, indicating generally favorable perceptions of system usability.

Qualitative insights from KIIs provided contextual interpretation of the usability findings. All 3 informants (n=3, 100%) identified staffing shortages as a structural factor influencing perceived documentation burden. A total of 2 informants (n=3, 67%) reported that parallel documentation requirements contributed to concerns about duplication, particularly in facilities where existing paper records must be maintained alongside new tools. Additionally, 2 informants (n=3, 67%) emphasized that visible administrative endorsement influenced staff motivation to adopt and consistently use the tool.

Triangulation of quantitative survey findings with qualitative themes indicates that perceived usability was shaped not only by interface clarity and learnability but also by broader structural workflow conditions within the neonatal units.

In Summary, the neonatal risk predictor tool demonstrated favorable usability performance across multiple domains. Survey responses indicated generally positive global usability ratings, acceptable learnability following initial exposure, and strong perceived clinical relevance for the early identification and prioritization of high-risk neonates. At the same time, item-level distribution revealed moderate variability in perceptions of complexity, with a subset of respondents reporting increased burden under high-workload conditions. These patterns collectively reflect broad usability across participants, alongside measurable differences in experience related to workflow intensity and staffing context.

This pilot, multisite usability evaluation suggests that the neonatal risk predictor tool was generally perceived as usable, learnable, and clinically relevant by participating frontline neonatal health care workers across the 3 Kenyan facilities. Most respondents reported willingness to use the tool regularly and increasing confidence with continued exposure; median composite usability scores were in the favorable range. At the same time, item-level responses revealed measurable polarization on perceived complexity and workload sensitivity, with a minority of users reporting initial difficulty and persistent concerns about documentation burden under high patient loads. These findings suggest that while the tool is broadly acceptable to users, its successful integration into routine neonatal workflows will depend on workload-aware implementation strategies and structured onboarding.

Our results align with broader evidence emphasizing the importance of usability and contextual fit for clinical decision-support tools in LMICs. The neonatal period remains the most vulnerable window for child survival globally, and interventions that improve early identification of high-risk neonates must be operationally feasible at the point of care to affect outcomes [1,3]. The finding that frontline health care workers value clinically relevant predictors (eg, Appearance, Pulse, Grimace, Activity, and Respiration [APGAR], gestational age, and vital signs) is consistent with prior ML prediction reviews that identify these variables as strong, widely used predictors of neonatal outcomes [5]. However, systematic reviews and expert commentaries have repeatedly highlighted that technical model performance alone is insufficient for impact: external validation, predictor availability, and deployment logistics are essential for translation [5]. Our triangulated usability and KII data confirm this; technical validity must be matched by usability and health-system readiness for effective adoption [3,5].

Our findings are also consistent with previous usability evaluations of digital health interventions in maternal and neonatal care. For example, Balderas-Díaz et al [32] demonstrated that user-centered design, iterative proof-of-concept evaluation, and usability testing are essential for promoting user acceptance and successful implementation of adaptive mHealth systems in perinatal care. Consistent with their findings, our study highlights that, beyond technical performance, successful adoption depends on usability, workflow integration, and alignment with users' clinical needs.

The observed variability in perceived complexity and information-quality dispersion highlights the need to follow established usability standards and HCD principles when refining and scaling the tool. ISO 9241-11 defines usability as effectiveness, efficiency, and satisfaction in a specified context of use; our findings show acceptable satisfaction and improved efficiency with use, but situational constraints reduce effectiveness in high-workload settings [14]. Similarly, reviews of usability instruments in mHealth underline the importance of combining standardized psychometric measures (eg, PSSUQ, QUIS, and SUS) with context-specific items that capture workflow and documentation interactions [15]. Our approach, combining standardized scales with a project-specific PSNUQ and KIIs, therefore, aligns with recommended practice for pragmatic usability evaluation in health settings [14,15].

The observed polarization in perceived complexity is not merely noise—it is a signal that the intervention interacts with heterogeneous user contexts, capacities, and expectations. Some frontline staff found the tool intuitive and readily incorporated it into routine care, while others experienced friction that increased perceived burden. This pattern commonly appears in early-stage usability evaluations and often reflects differences in prior exposure to structured documentation, individual triage strategies, and local workflow realities (eg, staffing and shift patterns). Interpreting polarization, therefore, requires moving beyond averages to examine subgroups (by facility, cadre, or prior digital familiarity) and to identify threshold conditions: small increases in caseload or complexity may flip a user from “comfortable” to “overloaded,” producing bimodal response distributions. Empirically, such subgroup heterogeneity is a cue for targeted refinements (eg, simplified pathways for high-workload contexts) rather than wholesale redesign [33].

From a theoretical perspective, polarization can also reflect differences in cognitive load and schema availability: users with established mental models for structured documentation experience lower intrinsic and extraneous cognitive load and therefore report lower perceived complexity; those lacking such schemas are more affected by extraneous load and report higher complexity (cognitive load theory). Targeted training that promotes rapid schema formation (eg, worked examples and supervised practice) is therefore likely to reduce polarization by elevating baseline competence across the team [34].

Finally, polarization carries implementation implications: rather than treating the tool as binary (adopt vs reject), implementation plans should include adaptive pathways, for example, a “light” checklist for high-pressure shifts and a richer version for less busy times, to maintain fidelity while accounting for context. This approach preserves core clinical value while lowering the activation energy for use under strain (implementation outcomes and feasibility guidance) [35].

The literature on ML for neonatal prediction is clear: many models demonstrate high retrospective performance but fail to translate into routine care due to gaps in predictor availability, data quality, and workflow fit (external validation, calibration, and deployment are often missing). In other words, algorithmic accuracy is necessary but not sufficient for clinical impact; the “last mile” is practical integration into health care workers’ daily tasks [9,10]. This study therefore complements previous model-development research by evaluating the implementation of an ML-derived predictor set under routine clinical conditions rather than reassessing algorithmic performance. Consistent with broader analyses of health information technology incidents, system-level fragilities often emerge not from algorithmic logic itself but from workflow integration failures, configuration issues, and documentation burdens.

Our findings reinforce this translation gap: frontline staff valued clinically relevant predictors (APGAR, gestational age [GA], and vitals), but even clinically sensible predictors must be routinely and rapidly available in usable form for an ML-based tool to function in practice. This reinforces calls in the literature for deployment-focused research that explicitly couples model development with operational capture and usability testing [5,36].

Operational requirements for ML deployment, therefore, extend beyond model metrics to include measures of data completeness, real-time availability, and process fit. Our mixed results, reasonable data capture for many predictors but workload-sensitive usability, illustrate why ML systems often falter post deployment: data pipelines may look good in retrospective datasets but break when frontline realities impose delays, missing fields, or inconsistent recording habits. Addressing this requires co-designed data workflows, automated data extraction when possible, and continuous monitoring of data quality during deployment [5,37].

Although the neonatal risk predictor evaluated in this study was derived from a validated ML model, participants were not asked to evaluate or compare the underlying algorithm. Instead, the usability evaluation focused on frontline users' experiences with the paper-based implementation of the predictor set within routine clinical workflows. Consequently, the findings should be interpreted as evidence regarding implementation and usability rather than perceptions of algorithmic transparency or trust in specific ML approaches.

Clinical tasks in neonatal units are time-pressured and cognitively intensive. Cognitive load theory highlights that working memory is limited and that extraneous load (poorly designed interfaces, unnecessary duplication) reduces the capacity available for core clinical reasoning. Our results, which showed that some users reported perceived complexity and workload-related friction, map precisely onto this theory. In practice, design choices that reduce extraneous cognitive load (prefilled defaults, binary checkboxes rather than free-text, and minimized decision branches) and scaffold intrinsic load (clear prompts and quick-reference thresholds) are most likely to improve consistent uptake. Empirical work on electronic health records (EHRs) and clinical decision support system (CDSS) shows that poor interface design increases cognitive load and error risk; conversely, simplified, context-aware interfaces reduce time-on-task and decision fatigue [34,38].

From a workflow standpoint, the tool must respect “attention budgets.” During high-volume shifts, health care workers have very limited time to document; even an additional 30 seconds per baby can be prohibitive. Therefore, interventions should be evaluated in terms of marginal time cost under realistic caseloads (time-motion studies), and design should aim to minimize per-case time through automation, task reallocation, or condensed item sets for critical screening. This attention to microeconomics of time turns usability into an outcome (time cost and adherence), not just a perception metric [38,39].

Our findings can also be interpreted through the Systems Engineering Initiative for Patient Safety (SEIPS) framework, which conceptualizes health care quality and patient safety as emerging from interactions among people, tasks, tools, and technologies, the physical environment, and organizational conditions [40,41]. From a SEIPS perspective, the documentation burden reported during periods of high workload reflects a work-system issue rather than a limitation of the neonatal risk predictor itself. Similarly, the qualitative findings regarding staffing shortages, parallel documentation systems, and organizational support indicate that successful implementation depends not only on the tool's usability but also on optimizing the broader clinical work system and integrating workflows. In low-resource settings, these considerations extend to the availability and reliability of the infrastructure supporting digital health implementation; evidence from Kenyan health facilities has highlighted the importance of addressing electricity and connectivity constraints and considering context-appropriate approaches to maintain continuity of digital health services [42,43].

Implementation science provides conceptual tools to interpret our findings and plan next steps. The Implementation Outcomes Framework (Proctor et al [35]) identifies acceptability, feasibility, appropriateness, and adoption as proximal outcomes that must precede penetration and sustainability. Our data show reasonable acceptability and perceived appropriateness, but feasibility and adoption are conditional on staffing and documentation realities, precisely the intermediate implementation outcomes that predict later penetration and sustainability. Framing results in this taxonomy clarifies operational priorities (eg, improving feasibility via workflow changes) and defines measurable implementation end points for subsequent trials.

Normalization process theory is also useful: it articulates how new practices become embedded through coherence (sense-making), cognitive participation (engagement), collective action (operational work), and reflexive monitoring [44]. Our KIIs and survey data indicate that coherence exists (staff see the value), but cognitive participation and collective action are hindered by workload and governance barriers, meaning that embedding will require active facilitation, local champions, and adaptive workflows to transform individual acceptance into routinized practice. These theoretical lenses suggest concrete implementation strategies (champion identification, cocreation, iterative feedback loops, and fidelity monitoring) that align with our empirical patterns [45].

This study contributes, both practically and conceptually, to the emerging LMIC digital health literature in several ways. First, it presents real-world usability evidence from neonatal units, a high-impact but under-studied clinical domain in LMICs, and ties usability to operational capture metrics, bridging a common gap between model-focused and deployment-focused work [39]. Second, by triangulating standardized usability instruments with context-specific qualitative insights, the work demonstrates a pragmatic mixed methods evaluation blueprint that other LMIC implementers can replicate. Third, it underscores that pilot usability samples (n≈5-15) are methodologically defensible for early user-centered design validation while pointing toward necessary next steps (larger trials and psychometric validation), thereby setting realistic methodological expectations for the field. Finally, it emphasizes implementation equity: without attention to staffing, governance, and data burden, digital tools risk widening care quality gaps across facilities [46]. Digital inequities and differential adoption across populations have been documented in other contexts, where infrastructure, governance, and sociocultural factors shape digital uptake [47,48]. These contributions respond directly to calls in recent mHealth and implementation reviews for context-sensitive, deployment-oriented evidence.

Based on our findings, implementers and facility managers should consider the following practical steps before scaling:

These pragmatic steps align with implementation science principles and can increase the likelihood that technically valid predictor models translate into practice improvements.

Several features increase the study’s utility to researchers and implementers: (1) real-world exposure, that is, the usability assessment followed a 4-month implementation period rather than a short laboratory test, increasing ecological validity); (2) multisite design, that is, 3 diverse facility contexts (urban informal, periurban public, and faith-based referral) enhance transferability of implementation insights across LMIC delivery settings; (3) triangulated methods, that is, combining standardized usability instruments (SUS, QUIS and PSSUQ-derived items) with KIIs provided depth and helped explain quantitative polarization; and (4) operational realism, that is, the usability evaluation was explicitly tied to routine documentation and staffing realities, yielding actionable recommendations for onboarding and workflow adaptation.

We must acknowledge several limitations that constrain the generalizability and strength of conclusions. First, the postsurvey sample size was small (n=10), limiting statistical power and precluding robust inferential testing; this study should therefore be understood as a pilot usability evaluation rather than a definitive validation. Accordingly, the findings should be interpreted as preliminary evidence intended to inform future larger-scale usability and implementation studies rather than to support broad generalization. Second, although multi-site, qualitative key informant data were limited to 3 senior health care workers (n=3), which constrains claims about saturation for broad thematic generalization; however, those informants were purposively selected leaders with facility-level oversight, increasing the relevance of governance and workflow insights.

Third, the usability measures relied on self-report; objective measures of time on task, error rates, or direct observation could provide complementary evidence of efficiency and effectiveness. Fourth, the PSNUQ instrument is study-adapted; while combining it with validated scales strengthens construct coverage, formal psychometric validation (eg, reliability and factor structure) was not possible with this sample. Finally, the study used a paper-based implementation of the predictor tool; findings may differ with a fully digital interface and therefore should not be assumed to directly generalize to other delivery modalities. These limitations should guide the interpretation and design of future studies.

To build on these pilot findings, we recommend:

This pilot usability evaluation suggests that participating frontline neonatal health care workers generally perceived the neonatal risk predictor tool as usable, learnable, and clinically relevant, while also identifying workflow-sensitive barriers, particularly staffing pressures and documentation duplication, that may influence implementation. These findings support proceeding to larger, methodologically rigorous evaluations while simultaneously investing in pragmatic onboarding, workflow integration, and infrastructure planning to maximize the potential for real-world impact. Given the global priority of reducing neonatal mortality, pragmatic usability evidence such as ours is a necessary step toward responsibly translating ML into safer, more timely neonatal care.