Perceptions of a Digital Mental Health Platform Among Participants With Depressive Disorder, Anxiety Disorder, and Other Clinically Diagnosed Mental Disorders in Singapore: Usability and Acceptability Study

Introduction

According to the United Nations News, mental disorders affect nearly 1 billion people worldwide [1]. There is also a large treatment gap in mental disorders, which is defined as the difference between the numbers of patients needing and receiving mental health treatment. It was estimated that 76%-85% of the people with severe mental disorders receive no medical treatment in low- and middle-income countries, and that number is around 35%-50% in high-income countries [2]. In Singapore, which is considered a high-income country, 78.6% of individuals met the criteria of needing mental health care but did not receive any treatment or help [3].

The low cost and high accessibility of digital therapeutic tools for mental health have the potential to bridge some of the treatment gaps in Singapore. Indeed, some studies have found evidence to support the use of internet-based mental health self-help tools [4-6].

In June 2020, the Ministry of Health Office for Healthcare Transformation, a subsidiary of the Ministry of Health, launched mindline.sg [7], an anonymous, digital mental health platform. This platform was developed to empower users in Singapore with knowledge, tools, and pathways to self-care as well as resources to help individuals seek out professional help when needed [8]. The website has since rapidly expanded to now include more than 500 curated resources, a self-assessment tool, and an emotionally intelligent artificial intelligence chatbot from Wysa that deploys a suite of interactive digital therapeutic exercises based on cognitive behavioral therapy. In the 2 years following its launch, mindline.sg received over 485,000 unique visitors.

The platform mindline.sg was not designed to serve individuals with moderate to severe anxiety or depression or those with clinically diagnosed mental disorders. However, it could eventually be expanded to aid health care professionals as an adjunct to therapy for these individuals. To successfully expand mindline.sg to users with diagnosed mental disorders, its usability and acceptability must first be evaluated among this population. Additionally, studies have found that higher acceptability improves uptake and adherence to digital intervention programs [9,10]. This study could also generate insights into product improvement and expansion.

In this study, we aimed to determine the perceived usability and acceptability of mindline.sg among patients with diagnosed mental disorders within a clinical setting. The primary objective of the study was to determine the perceived usability of mindline.sg through the Post-Study System Usability Questionnaire (PSSUQ). The secondary objective was to determine the perceived usefulness and acceptability of mindline.sg through a custom survey.

Methods

Study Design and Recruitment

A cross-sectional study with 173 participants was conducted from April 2021 to January 2022 in the waiting room of a psychiatrist’s office at the Department of Psychological Medicine, National University Hospital, Singapore. Participants waiting for an appointment were given 30 minutes and a simple set of instructions to use three prominent features of mindline.sg: (1) a novel self-assessment and “wellness triaging” questionnaire consisting of a dynamically evolving set of questions from among the Patient Health Questionaire-9 (PHQ-9) and Generalized Anxiety Disorder-7 (GAD-7) surveys, which aims to help users understand their current levels of anxiety or depression and to direct them to appropriate content; (2) an emotionally intelligent chatbot from Wysa that conducts a range of cognitive behavioral therapy (CBT)–inspired digital therapeutic tools and can also converse with the user in a free form; and (3) a collection of resources provided by health care ecosystem partners on topics such as mental health literacy, employment support, caregiver support, financial support, fitness tips, and domestic abuse support. Following the 30-minute usage period, the participants answered a set of web-based survey questions, including a reduced version of the PSSUQ with 16 questions designed to measure the perceived usability and satisfaction with the platform and 5 yes/no questions designed to understand the acceptability and perceived usefulness of the website.

The participants were recruited by the Department of Psychological Medicine, National University Hospital in Singapore. Participation was optional, and no remuneration was given. Participants between the ages of 21 and 65 years who were waiting for their psychiatrist appointment at the hospital were invited to the study. For safety reasons, we excluded participants with any form of cancer or major neurological disorder (eg, epilepsy and stroke), heart disease (eg, ischemic heart disease), lung disease (eg, chronic obstructive pulmonary disease), liver disease (eg, liver failure), or kidney disease (eg, kidney failure). All participants were required to provide physical informed consent before the commencement of the study.

Ethics Approval

Ethics approval was granted by the National Healthcare Group Domain Specific Review Board (NHG DSRB 2020/01326) for a study period from January 2021 to January 2022.

Web-Based Survey Design

The web-based survey consisted of 3 sections. In the first section, participant demographic data were collected. This included age, gender, marital status, education level, annual income, employment status, race, and medical history, as shown in Table 1.

The second section was modelled after the reduced version of the PSSUQ, which is a widely deployed usability quantification survey [11]. The PSSUQ consists of 16 questions that are scored on a 7-point scale (from 1 as “strongly agree” to 7 as “strongly disagree”; Multimedia Appendix 1). The scores determine an overall satisfaction scale, computed as the average score across all 16 items (and so takes a value between 16 and 112) and 3 subscales of system usefulness, information quality, and interface quality, taking values in the ranges of 6-42, 6-42, and 4-24, respectively, each computed as the average score across various subsets of the items. For all scales, lower scores indicate better usability. The PSSUQ questionnaire has shown a satisfactory level of reliability, sensitivity, and validity [12,13].

The third section of the survey consisted of the following 5 yes/no questions constructed by the study team to measure the acceptability and perceived usefulness of the major features of mindline.sg:

1. Did you like taking the “I need help to manage my emotions” questionnaires?
2. Did you find the resources that are listed useful?
3. Did you like talking with the Wysa chatbot (the penguin)?
4. Did you find any of the exercises recommended by mindline.sg or Wysa (the emotionally intelligent chatbot) useful?
5. Would you recommend mindline.sg to a friend?

These questions have not been tested for reliability, sensitivity, and validity.

Statistical Analysis

We used multiple linear regression analyses to discover factors that are associated with the overall PSSUQ scale, such as age, marital status, annual income, education level, and medical history. The empirical distribution of the overall PSSUQ scale was not well modelled by a Gaussian distribution (Shapiro-Wilks test: P<.001), so a power transformation was applied to the PSSUQ scales before training.

To investigate any relationship between the self-reported psychiatric conditions and the responses to the 5 yes/no survey questions measuring the acceptability and perceived usefulness of the platform, we performed chi-square tests based on self-reported psychiatric conditions. Three population comparisons were performed: (1) between participants with and without depressive disorder, (2) between participants with and without anxiety disorder, and (3) between participants diagnosed with any other psychiatric conditions and those diagnosed with either depressive or anxiety disorders. We noted that 10 participants indicated they either did not have any psychiatric disorder or did not know their diagnosis; the responses of these participants were excluded from this analysis.

All statistical tests were 2-tailed. We reported factors at P<.05 as significant, and corrections for multiple comparisons were not used.

Results

Demographic Data

The demographic data collected, summarized in Table 1, revealed the majority of the participants were female (107/173, 61.9%), between the ages of 21 and 30 years (83/173, 48%), had an education level of “undergraduate degree and above” (79/173, 45.7%), and were single (115/173, 66.5%).

Medical History

The medical history data are summarized in Table 2. The majority of the participants (88/173, 50.8%) self-reported being diagnosed with depressive disorder, were most likely to be on either medication (84/173, 48.6%) or both medication and psychotherapy or counselling (71/173, 41%), and had not been hospitalized due to their psychiatric condition (107/173, 61.9%).

PSSUQ Results

In Table 3, we report the mean and SD values of the PSSUQ overall score and subscores. A meta-analysis of 5 years of usability studies (which were predominantly on speech recognition systems, though the meta-analysis showed a good ability to generalize) provided the means and 99% CIs of analyzed PSSUQ scores (Table 3) [11]. We will henceforth refer to the literature-derived mean and 99% CIs as PSSUQ norms. A lower score indicates better usability.

In Table 4, we report the parameter estimates in a multiple linear regression model (a Box-Cox transformation) of the PSSUQ overall satisfaction scale onto the participant demographic and medical history data.

Acceptability and Perceived Usefulness of “mindline.sg”

The last section of the survey consists of five yes/no questions designed to measure the acceptability and perceived usefulness of mindline.sg. The overall responses were largely positive with all the 5 questions receiving more than a 60% positive response. The question “Did you find the resources that are listed useful?” received the highest percentage of positive responses (86.7% of the users). The question “Did you find any of the exercises recommended by mindline.sg or Wysa useful?” received the lowest percentage of positive response (60%). Table 5 shows the full results of the survey. We found no significant differences between the responses of any subgroups.

Discussion

Usability Findings

Comparing the PSSUQ overall score (describing the perceived usability of mindline.sg among the survey respondents) to the literature-derived norms (Table 3), we found that the system usefulness (mean 2.74, SD 1.46), information quality (mean 2.98, SD 1.33), and the platform overall score (mean 2.86, SD 1.46) were perceived as “good” and were comparable to most other digital apps (within a 99% CI of the literature-derived norm). Although the interface quality score (mean 2.98, SD 1.33) is lower in this regard than most other digital apps, it is also perceived as “good” because it is above the neutral score of 4 on the PSSUQ scale.

Based on the results of the multiple linear regression with the overall PSSUQ scores, we found that education level is the factor with the highest association with the PSSUQ score (the largest magnitude coefficient is reported in Table 4). In particular, participants with an education lower than an O-Level or N-Level General Certificate of Education tend to give mindline.sg a lower usability score (β=0.49; P=.02). Because the majority of the resources on mindline.sg are text-based and require a certain level of English literacy to use, we find the results in line with our expectations. However, these findings are in contrast with earlier studies, which found a weak to no correlation between education level and usability of connected medical devices and internet-based CBT (iCBT) platforms [10,13].

In addition, participants who have not been previously hospitalized due to their psychiatric condition are likely to give a poorer overall PSSUQ score compared to participants who have been hospitalized (β=0.177; P=.03). At the point of writing, we could not find any prior study to explain this finding.

Acceptability and Perceived Usefulness Findings

We found that mindline.sg is generally acceptable to participants with self-reported mental disorders, with all 5 questions having more than 60% positive response. These results are consistent with the findings from a meta-analysis that found iCBT platforms were acceptable and effective for patients with depression and anxiety disorders [14].

When comparing patients’ responses on mindline.sg, acceptability and perceived usefulness between the three different mental condition subgroups (as illustrated in Table 5), we found no significant differences between responses by type of psychiatric disorder.

Limitations

The participants were given only 30 minutes to use the three features of mindline.sg before they were asked to complete the survey, but some of the therapeutic exercises provided by the Wysa chatbot on the website could take around 20-30 minutes to complete. This could explain why the questions on “Did you find any of the exercises recommended by mindline.sg or Wysa useful?” received the least amount of positive response (60%) compared to the other acceptability questions, as the participants may not have had enough time to fully explore these exercises. Other resources are usually completed in a shorter amount of time, as they take the form of articles and videos that generally take less than 10 minutes to consume. The short usage period of mindline.sg in this study may affect the generalizability of this study.

Additionally, the medical history collected in this survey is self-reported and has not been independently verified with clinical records (this was not put forward to the ethics committee to protect patient confidentiality and preserve the anonymity feature of the mindline.sg platform). The data reported in our study are also from participants who had an appointment in the Department of Psychological Medicine in National University Hospital and were not randomly selected, which could result in some form of selection bias. Given the limitations mentioned above, any generalization from this study should be evaluated with caution.

Although the PSSUQ norms were used as a basis for comparison with our collected results, it is important to note that the norms were established by products from a variety of sources (which were predominantly speech recognition systems) and at different stages of development [12].

Although we compare our results of acceptability to a meta-analysis, it is also important to note that many of the studies in the meta-analysis use adherence and patient satisfaction in a longer-term treatment program as a proxy for acceptability [13]. Since our participants only use mindline.sg for around 30 minutes, the answer to the 5 questions is instead used as a proxy to acceptability.

Lastly, as the evaluation of usability in mobile health varies substantially [15], it presents a challenge for us to compare our findings to previously published usability results. Although the PSSUQ questionnaire has shown a satisfactory level of reliability, sensitivity, and validity [11,12], the 5 yes/no questions that were constructed by the study team as a proxy for acceptability have not been tested for validity and reliability. The nature of the yes/no questions could also limit the range of responses as compared to a Likert-type scale.

Conclusions

Despite the limitations mentioned above, this study shows that mindline.sg could be a viable self-help tool for individuals with diagnosed mental health conditions due to its well-rated usability and acceptability. Furthermore, the accessibility of a free, anonymous, and web-based tool like mindline.sg allows people with diagnosed mental conditions to access these services at any time and from the comfort and privacy of their homes. However, the clinical effectiveness of mindline.sg as a mental health resource for people diagnosed with mental conditions has not yet been validated and might be an important focus for future studies.