Hemophilia is an inherited X-linked bleeding disorder. Hemophilia A is characterized by a deficiency in the clotting factor VIII, while Hemophilia B is a deficiency in factor IX. Given the reduced ability to form clots, people living with hemophilia experience an increased frequency and duration of bleeding events, which tend to occur mostly within the joints or muscles [1]. The standard of care for hemophilia treatment involves infusing factor replacement to increase factor concentrations in blood [1]. Another option for prophylaxis in people living with hemophilia A is subcutaneous infusions of emicizumab, a bispecific antibody that mimics the function of factor VIII [2].

Limiting the effects of bleeding episodes is important for maintaining patient quality of life, as bleeds cause deterioration of the joints and can result in pain and disability [3]. The quality of life of people living with hemophilia and the burden of the disease are also affected by other aspects, for example, the burden of adhering to a regular prophylactic regimen and comorbidities, particularly bleeding-related arthropathy and transfusion-transmitted infections with hepatitis B virus, hepatitis C virus, or HIV [4]. Outcome assessment in hemophilia often comprises clinical measures, such as number of target joints, number of emergency room visits, and hospital length of stay [5]. In recent years, patient-reported outcomes (PROs) have been increasingly used to measure additional metrics that capture a patient’s perspective of their own health [6,7]. The assessment of PROs involves asking an individual to assess a variety of domains, such as pain, disability, function, and satisfaction with treatment. This, when combined with clinical outcomes, provides a more comprehensive understanding of the disease and available treatments [8]. Multiple PROs and quality of life measuring instruments exist in hemophilia, although they have not involved patients throughout the development of the questionnaire, despite clear guidance to do so [9]. The Patient-Reported Outcomes, Burdens, and Experiences (PROBE) questionnaire was developed for patients by patients, with the support of an epidemiologist, an expert in outcome research, an expert in management science, and economics-engineering systems [10]. The PROBE questionnaire has undergone assessment for feasibility, validity [10], and test-retest reliability (including cross-validation of paper and online versions) [11], and evaluation of other psychometric properties [12]. The finalized version of the PROBE questionnaire is composed of 29 questions with additional subquestions depending on responses, which have been divided into 4 sections, focusing on personal demographics, general health problems, disease-specific health problems, and the EuroQol 5 dimensions 5 levels (EQ-5D-5L) and Visual Analog Scale (VAS). Before this project, PROBE had been tested across 21 countries, and was available in 11 languages (with 20 localized versions worldwide). It has been shown that PROBE is a valid questionnaire for the evaluation of PROs in people living with hemophilia and a control population [12]. Its discriminative properties allow its use in clinical trials, longitudinal studies, health technology assessment studies, routine clinical care, or disease registries. Before this project, the questionnaire could be filled on the PROBE website anonymously by people living with hemophilia and controls (individuals without a bleeding disorder) alike. This system did not allow for longitudinal data collection. If the same individual returned at a later date to take the questionnaire again, the new questionnaire was not linked to the previous one. The PROBE investigators assessed that the capacity to collect longitudinal data and to link responses to other databases would provide benefits to physicians, people living with hemophilia, and researchers. By doing so, if agreed by the participant, individual responses could be tracked, and progress could be monitored and evaluated.

The acceptability and feasibility of digital health interventions in chronic conditions (including rare diseases) have already been proven, while evidence about the efficacy of such interventions on patient-oriented outcomes is not definitive [13-16]. In our case, we aimed at eliciting the measurement of PROs. This is not an intervention directly aimed at affecting outcomes, even though this can be an indirect effect. Longitudinal PRO data could support decisions surrounding novel treatments or treatment schedule changes, provide important insights on the changes in quality of life following certain events (eg, bleeds, surgery, or a change in treatment regimen), or provide a tool for physicians or health systems to track patient outcomes over time. Different solutions have been proposed for the digital collection of PRO measures, including modifying the architecture of the electronic health record to integrate data collection from different mobile apps [17]. However, once implemented, the usability of these systems needs to be tested, as changes might be needed if the results are unsatisfactory [18,19].

The backbone of this project was making prospective longitudinal data collection of the PROBE questionnaire possible, by developing a stand-alone individual longitudinal PROBE modality and integrating the data collection of PROBE with other databases. The Canadian Bleeding Disorders Registry (CBDR) was used to demonstrate the feasibility of registry integration, and the usability of the app was tested.

To plan for the development of the software infrastructure and design of the PROBE project, we conducted a series of semistructured interviews with a convenience sample of front-end and back-end users, including all relevant stakeholders. In detail, back-end users were the PROBE investigators, representatives of the sponsor, and researchers, and front-end users were people with hemophilia. The interviewees were based in Canada, Ireland, Italy, Switzerland, and the United States of America. In preparation for the interview, we asked the interviewees to complete the PROBE questionnaire on the website, unless they had recently taken it. The semistructured interview guide is provided in Textbox 1. Characteristics of the stakeholders are provided in Multimedia Appendix 1.

We put together a technical group involving (1) 3 programmers on staff at the Health Information Research Unit at McMaster University, (2) an external consultant (Design2Code [D2C]) based in Waterloo and skilled in mobile app development, and (3) stakeholder representatives. The needs identified in the previous phase were used to guide the creation of 2 alternative plans for software development: updating the website and creating a mobile app or only updating the website to meet those needs. The 2 plans were discussed within the technical group using the Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis process to compare the different alternatives, including their costs and the time needed for development. The product of this phase was a justification of the choice and a detailed description of the technical specifications for realizing the PROBE suite. The main areas of work were defining if an app was needed and which characteristics were required, and understanding how the existing PROBE website, database, and application programming interface (API) needed to be modified.

All members of the technical team and selected PROBE investigators were invited to test the software under development and provide feedback. Testing included 3 sessions of group testing and independent individual testing. The first round of testing and feedback was based on a mock-up of the product. The second round was based on the beta version of the new website and the app, released on TestFlight for iOS and on the testing environment of Google Play. These products were working with a fake development environment (API and database). The third round was based on the first release of the website and mobile app, using a test modality on the production database.

Once the software was stable in its first version, which took 3 releases, formal usability testing was performed. A user group composed of a convenience sample of patients, representatives of the sponsor from the same countries specified above, and graduate students from McMaster University evaluated the final product with the System Usability Scale (SUS) [20] and an open feedback framework. The SUS is composed of 10 questions (reported in Multimedia Appendix 2) to be answered on a 5-point Likert scale, from “strongly agree” to “strongly disagree.”

Based on the results of the needs assessment and following the structured approach described as module 2 of the study, the PROBE team met with D2C to decide if an app was needed or if a device-responsive website was sufficient to achieve the objectives. The SWAT analysis results between the 3 plans were translated in pros and cons of having an app on top of the website, which have been summarized in Table 1. The team determined that an app was needed, primarily to facilitate the accessibility of the content, to send notifications to users when it is time to repeat the questionnaire (eg, after 6 months or when a bleed occurs), and to allow, in the future, leveraging of smartphone features like physical activity tracking or access to the camera. Furthermore, off-line questionnaire completion would not be possible on a website, while this was perceived as important, especially from the perspective of users from less developed countries where internet access or bandwidth is limited. Therefore, to meet these needs, it was agreed that an app for Android and iOS environments was required.

To allow for the longitudinal repetition of the PROBE questionnaire, log-in options were required. Retaining the anonymous completion option was deemed important, as was adding an email-based login and a single sign-on (SSO) option for people living with hemophilia to participate through their national registry patient app (eg, myCBDR). The CBDR option was only made available to users selecting “Canada” as their home country.

The data flow from completion to data analysis and study reports is described in Figure 1. Individual data needed to flow from the app and the website to the PROBE database. For security reasons, personal information (email address or CBDR identifier) had to be stored separately from the PROBE questionnaire’s data. The capability to allow part of the data to flow bidirectionally from the PROBE database to others, like hemophilia registries, was needed.

Before this project, the PROBE website was communicating directly with the PROBE database. To implement the new functionalities, an API was required. Having an API in place allows (1) sending data to the databases from both the app and the website, (2) authenticating users with a dedicated email and password or with a token (eg, obtained using CBDR credentials or coming back to the PROBE website after the first log-in), and (3) supporting different versions of the questionnaire and other data (eg, notifications or calculation and report of PROBE and EQ-5D scores) from the database to the app and website. The technical specifications for the PROBE suite are reported in Multimedia Appendix 3 and Multimedia Appendix 4.

In collaboration with the PROBE investigators, the team at McMaster University and D2C developed an online survey using Microsoft.Net technologies and an app for iOS and Android using react-native. For the duration of the development phase, the McMaster University team and D2C met monthly to discuss progress and to find solutions to unanticipated problems. The PROBE investigators and the sponsor were involved as needed. Textbox 2 reports the solutions implemented to realize the main system functionalities. A sign-in and log-in interface was created. The SSO with MyCBDR credentials was implemented using OAuth 2.0 technologies. A survey manager and a template builder were implemented to allow creating new versions of the PROBE questionnaire by adding or removing sections and questions.

The 3 cycles of testing of the app and website allowed bugs and required fixes to be identified. Feedback from beta-testers led to improvements in the user experience, for example, adding autoscroll for pages displaying more than one question, changing the app buttons and graphics when they were perceived as not being clear, and adding descriptive text (eg, feet, inches, and pounds to the height and weight question for countries not using the metric system).