MINDSET is a theoretically and empirically based digital ESM program, originally accessible on desktop or tablet computers in the clinic [11,19]. Patients choose their preferred language (English or Spanish) and input data (“My Epilepsy”) about their epilepsy condition (overall subjective health rating, seizure frequency in the last 30 days, seizure symptoms, antiseizure drug [ASD] prescription, and days of missed doses in the last 2 weeks) and their adherence to ESM behaviors within each of the 3 ESM domains (seizure, medication, and lifestyle management) using the Epilepsy Self-management Survey (ESMS) [20-22]. They complete the Neurological Disorders Depression Inventory in Epilepsy (NDDI-E) to assess depression [23] and the Liverpool Adverse Events Scale to assess ASD side effects [24-26] (Table 1). Patients select up to 3 ESM goals (one each for seizure, medication, and lifestyle management) to improve low adherent ESM behaviors (“My Goals”), choose strategies to accomplish these goals, rate their self-efficacy to achieve their selected goals, and list barriers to meet these goals [20,22]. The goals are designed to increase patient adherence to low adherent behaviors. A tailored goal-based action plan (AP; “My Action Plan”) is dynamically generated and is printed to review and discuss with the HCP during the clinic encounter (Figures 1 and 2). Page 1 of the AP comprises a summary of the patient’s epilepsy history and current condition, assessment of ESM behaviors and depression (Figure 2). Pages 2 to 4 summarize the selected goal, strategies, self-efficacy, and barriers for seizure, medication, and lifestyle management, respectively.

MINDSET has been enhanced to MINDSET 2.0 with added features (Table 1) that include (1) accessibility on any device with an internet connection, (2) assessment of depression using the NDDI-E where a score of ≥15 triggers a summary message in the AP to discuss mood and a recommendation that the patient may benefit from the Using Practice and Learning to Increase Favorable Thoughts (UPLIFT) program designed to train patients with epilepsy to manage comorbid depression [38], and (3) assessment of quality of life using the Quality of Life in Epilepsy Inventory-10 (QOLIE-10) [33] and cognitive (memory) difficulties using the Quality of Life in Epilepsy Inventory-31 (QOLIE-31) cognitive subscale [34]. A QOLIE-31 cognitive subscale score of ≤70 and/or the QOLIE-10 memory item can trigger a recommendation that the patient may benefit from the Home-Based Self-Management and Cognitive Training Changes Lives (HOBSCOTCH) program designed to train patients with epilepsy to manage comorbid memory problems (Figure 2, AP page 5) [39]. Additional features included (4) a default recommendation for the Program for Active Consumer Engagement in Self-Management (PACES) as a foundational ESM training program given its suitability for all patients with epilepsy. UPLIFT, HOBSCOTCH, and PACES are available in English and Spanish. Lastly, MINDSET 2.0 added an (5) assessment of social determinants using the Health Leads social determinant inventory [35] to cue discussion about possible social service needs (Figure 3). The HCP can link the patient to recommended epilepsy programs and connect them with community resources as needed.

In-house alpha testing by the research team was conducted to ensure that the content was accurate and consistent with current clinic practice, that spelling and grammar were error-free, that inputted data corresponded to the backend database, and that the Spanish translation was correct. User experience testing occurred in January 2022. A convenience sample of 7 patients with epilepsy was recruited from the Epilepsy Foundation Central & South Texas (EFCST) clinic in San Antonio, Texas (n=5) and the Banner Clinic in Tucson, Arizona (n=2). Inclusion criteria were that patients be 18 years or older; have a diagnosis of epilepsy; and have access to a computer, phone, or tablet with a working internet connection. MINDSET was designed for patients with epilepsy who are able to self-manage their epilepsy, so neurologists determined exclusion of patients with neurological, medical, motor disorders, or learning problems as well as psychiatric and/or behavioral problems that would excessively inhibit their ability to use MINDSET 2.0, complete surveys, or practice ESM activities.

Clinic staff contacted eligible patients via phone and introduced those who verbally consented to the research team coordinator who arranged a testing session. Sessions were conducted on Zoom (Zoom Communications, Inc), a familiar platform to most participants, due to clinical mandates for COVID-19–related social distancing. In each session, participants completed a consent form, a 26-item sociodemographic survey, accessed all components of MINDSET (eg, the screening tool, patient profile, recommendations, and AP), and then completed an adapted 36-item usability rating scale (Multimedia Appendix 1) and a 30-minute structured exit interview (Table 1). The research team coordinators guided participants through the session and noted any issues in a problem log. The coordinators comprised a doctoral candidate in Behavioral Sciences with 6 years of experience in qualitative research in Texas and an assistant research professor at a state college of public health with an adjunct appointment at a department of neurology in Arizona. A bilingual master’s in public health (MPH) student took notes and served as a Spanish translator for patients electing to complete MINDSET 2.0 and study surveys in Spanish. Participants were allowed to complete the study with assistance from caregivers if needed. Participants were compensated for their time with Amazon e-gift cards valued at US $60 at the end of the session. User experience data were assessed with descriptive statistics using Stata 18 (StataCorp LLC) [40]. A priori criteria user agreement of at least 70% was set as a usability benchmark. Features that fell below this were identified for modification. Exit interviews were recorded, and transcripts were assessed and collated to qualify each usability parameter.

The study was approved by the institutional review boards of the University of Texas Health Science Center at Houston (HSC-SPH-19‐0893) and the University of Arizona (2005674587). Informed consent was obtained from all participants who were given the option to withdraw from the study at any time. All data collected in the study were deidentified to maintain patient privacy and confidentiality.

The functionality assessment comprised (1) noting “bugs” or suboptimal performance in a program log and (2) reviewing algorithm logic fidelity from the user experience testing data, whereby inputted participant data were compared to AP output and assessed against an a priori benchmark for success of 100% correspondence. Staff manually calculated participant scores (eg, NDDI-E depression scores and QOLIE-31 cognitive scores) to predict expected recommendations and compared these to the actual recommendations generated on the AP to test functionality of the algorithms. The assessment also included (3) challenging the MINDSET 2.0 algorithm function with an array of user profile test cases ranging from a “worst case” ESM patient profile to a “best case” and variations between these. A “worst case” patient answered all questions negatively and a “best case” positively. User test cases were assigned unique ESM behavior profiles for seizure, medication, and lifestyle management and varied depression, Quality of Life (QoL), and social determinant indices. Each tester was assigned a predetermined set of responses to ensure that all item answers were inputted at least once. Testers selected the corresponding goal for each assigned at-risk item in the profile to ensure all goals were selected at least once. APs were then reviewed to determine that messages and goals were logically consistent with the data input. Similar to the user experience algorithm review, an a priori benchmark for success was set at 100% correspondence between data input and AP output.

Six participants completed MINDSET. One participant withdrew without completing the AP citing epilepsy cognitive impairment. Participants had a mean age of 37.3 (SD 15) years, were equally male and female (n=3, 50%), Hispanic or Latino (n=6, 100%), with a mean US residency of 23 (SD 13.2) years, mainly with high school (n=3, 50%) or 1‐3 years of college (n=2, 33.3%) education, single (n=4, 66.7%), with income below US $50K (n=5, 83.3%), and were either health insured (n=3, 50%) or not health insured (n=3, 50%; Table 2). Most reported having access to a smartphone and/or a computer (n=5, ≥83.3%) and preferred English or mainly English for reading, speaking at home, and thinking (n=5, 83.3%).

Participants rated their QoL as mostly “very good” (n=3, 50%; Table 3). Their first ever seizure occurred between ages 2 and 15 years. They had active epilepsy, reporting seizures in the last 12 months (n=6), and in the last 30 days (n=3). They mostly reported 5 to 6 seizures in the last month and 4 seizures since their last clinic visit. They reported adherence to their ASD dosing schedule, on average missing a dose on one day in the last 2 weeks. They reported 3 to 8 side effects. The sample mean NDDI-E score was 12.8 (SD 5.6) with 2 participants scoring ≥15. The mean QOLIE-31 item score was 85 (SD 15.2) and the mean QOLIE-31 cognitive subscale score was 51.6 (SD 33.3), indicating poor QoL. However, the mean QOLIE-10 score was 2.5 (SD 1.02), indicating good QoL in relation to patients’ epilepsy. The greatest adherence to ESM was for medicine management behaviors (mean 4.5, SD 0.70), and the lowest adherence was for lifestyle management behaviors (mean 3.9, SD 0.41) and information management behaviors (mean 3.5, SD 0.63; Table 3).

Participants took approximately 30‐60 minutes to complete MINDSET, up to 30 minutes to complete the survey and ratings scale, and up to 30 minutes to complete the exit interview. Participants mostly selected the English language version (n=5, 83.3%). More than 80% of participants agreed that MINDSET 2.0 was acceptable, likable, and motivationally appealing (that they would use it again during future visits and recommend it to others) and indicated it to be “exciting,” “colorful,” and “relevant” (Table 4). They rated it as credible (n=6, 100%) and of appropriate duration (n=5, 83.3%), and that, in its entirety, it was “easy to use” and “simple” (n=6, 100%), and that the words used were understandable (n=6, 100%), requiring no help to answer survey items (n=5, 83.3%). They commented favorably on its comprehensiveness, stating that it was “thorough” and “in-depth,” and appreciated the degree of detail in the tailored AP. They rated it as providing needed (n=5, 83.3%) and helpful (n=6, 100%) information, recommending ESM programs appropriate to their needs (n=5, 83.3%), and that it helped them think about and manage their epilepsy more carefully, and that it would improve decision-making with their HCPs (n=6, 100%). They rated it as more useful than other programs they had used in the clinic (n=5, 83.3%) and described it as an effective and “innovative” evaluation tool that introduced them to new things and assisted them to understand broader lifestyle factors associated with seizure reduction such as sleep, stress, diet, and substance use. Participant recommendations for enhancement of MINDSET content were confined to suggestions that MINDSET “assess family dynamics” due to the reliance of many patients with epilepsy on family members to assist with seizure and medication management, and provide support to mitigate stigma and financial strain [7,13]. Requests were also made that MINDSET 2.0 flag suicidal ideation based on participant experience.

Patients reported lower agreement with the ease of selecting goals and strategies (n=3, 50%) and understanding and using the AP (n=2, ≤33%). The MINDSET 2.0 AP occasionally provided feedback and goal options that seemed to the participants to be misaligned or incongruous to their daily ESM. Participants expressed some frustration and confusion as to why certain behaviors seemed incorrectly flagged as low adherent. The interviews revealed that many patients had difficulty providing responses to the questions due to a restricted 4-point Likert scale. For example, to the item “I use safety precautions in the kitchen,” a patient in Texas responded “never,” because she is “never in the kitchen.” Given this rationale, the more accurate response would have been “not applicable” rather than a response indicative of nonadherence. A patient in Arizona responded, “all the time” to the statement “I go swimming alone,” their rationale being that they do not need to “hold on to anyone or anything” when going to swim rather than being unaccompanied. Patients’ misinterpretation of the meaning and purpose of the questions suggested problems with understandability at the item level. Despite published findings of empirically sound psychometrics for the ESM survey, this feedback indicated the need to incorporate “not applicable” response options into items to increase response validity. In interviews, most reported difficulties were in the development and understanding of the AP. One participant reported that it seemed “repetitive” and “arduous,” while another expressed difficulty in understanding how to use it, inclusive of selecting goals and strategies, interpreting the output, and AP download.

MINDSET 2.0 exhibited no connectivity or bandwidth lags or data dropout, and back-end data collection was error-free. ESM was positively skewed and highest for medicine, seizure, and safety management subdomains (Table 3). The number of low adherence behaviors per participant ranged from 1 or 2 for seizure management, 2 to 6 for medication management, and 2 to 15 for lifestyle management (Table 5). Most patients (n=4, 66.7%) reported high adherence to medicine management behaviors, which was compatible with the low frequency of missed ASDs. Selected goals were “always keeping enough medicine” to avoid running out and “using a pill container or pill box” for ASDs (Table 5, column 2). All patients were flagged as having at least one low adherent lifestyle behavior. Selected goals focused on stress and relaxation techniques, getting an average of 6‐8 hours of sleep (n=2), carrying an epilepsy ID, forming social connections, and eating a healthy diet. The goals were unique to each participant apart from “getting enough sleep,” which was selected by 2 patients. For any chosen goal, patients selected between 0 and 7 strategies (Table 5, Column 3). The majority selected one strategy from the list of strategies provided. Two patients listed potential barriers to meeting their goals, both for seizure management, that related to the need to get a primary care doctor or neurologist, and health insurance (Table 5, column 3). No participant self-efficacy ratings were downloaded in the AP. This was traced to an error in coding logic failure that was converting categorical Likert scale scores into binary scores.

All participants reported having no past use or experience with any of the Managing Epilepsy Well (MEW) Network programs (UPLIFT, HOBSCOTCH, or PACES). In accordance with the MINDSET 2.0 design, all patients were recommended to attend the PACES epilepsy training program, which was the default recommendation. The HOBSCOTCH cognitive skills training program recommendations for patients flagged as having cognitive difficulties (n=5) were also consistent with the MINDSET logic model (Table 6). However, correspondence between predicted and actual recommendations for the UPLIFT program was less than 100% for 2 participants (Table 6, participants #3 and #6). These 2 participants were flagged as having depressive symptoms (NDDI-E score >15) but were not recommended by MINDSET 2.0 to attend the UPLIFT depression program despite no prior history with this program. This represented a failure of the algorithm to translate NDDI-E depression scores into a recommendation to remediate for depression and was traced to incorrect algorithm decision logic.

Four patients reported at least one social determinant that may have hindered their epilepsy management, mainly food insecurity (n=3) and financial resource strain (n=3). Other listed determinants were housing instability (n=2), low health literacy (n=2), transportation challenges (n=1), utility needs (n=1), and social isolation (n=1).

The purpose of this study was to assess the usability of MINDSET 2.0 prior to deployment for feasibility testing in neurology clinics that serve patients with epilepsy. The 6 testers, who were people with a doctor’s diagnosis of epilepsy, were experienced in epilepsy management and forthcoming about recommending improvements to MINDSET 2.0 to enhance its comprehensiveness and relevance. Consistent with the health disparities often cited for patients with epilepsy at the national level, the participants reported limited insurance coverage, education, and employment, and the most reported social determinants were food insecurity and financial strain that can impede their ability to manage their epilepsy. The usability findings were largely consistent with previous studies of MINDSET in that the patients agreed that MINDSET 2.0 was acceptable, likable, credible, easy to use, and useful for decision-making and ESM, meeting a priori criteria of 70% agreement [12,37]. Patients reported less overall adherence to, and competency in, information management (monitoring and recording seizures and symptoms, and communicating about epilepsy) and lifestyle management (eg, ensuring adequate exercise, sleep, and strategies to relieve stress) compared to other more medically oriented ESM behaviors of managing medication, safety, and seizures. This was consistent with prior studies on ESM among patients with epilepsy [12,37].

MINDSET 2.0 has added features to screen for comorbidities of depression symptoms and subjective cognitive difficulties and to recommend training programs of UPLIFT and/or HOBSCOTCH, respectively. Patients with epilepsy have a greater prevalence of psychiatric comorbidities than the general population and are at greater risk of being undertreated for depression, which is consistently demonstrated to be inversely associated with successful ESM [41,42]. Consistent with previous studies, more than 33% (n=2) of MINDSET participants reported symptoms of depression. A question item regarding suicidal ideation is also embedded in the NDDI-E survey. Adding an alert in the AP specific to suicidality was suggested that will further highlight this issue. Specific reference to this item has since been included in the neurologist MINDSET 2.0 training.

People with epilepsy have an increased risk for impaired memory, executive function, attention, and social skills in comparison to people without epilepsy [43,44]. Cognitive function can be further compromised by early onset of epilepsy, frequent recurrent seizures, and polytherapy [43]. Low QOLIE-31 cognitive subscale scores (≤70%) were consistent with the prevalence of cognitive impairment of 60% and 70% for patients with epilepsy in general [43,45]. Patients with epilepsy often have cognitive impairments that can compromise their use of MINDSET 2.0. The positive ratings for many usability parameters are encouraging and consistent with previous studies of ESM interventions that report outcomes of increased awareness, feelings of autonomy, ESM adherence, and a sense of hope [13,17] and significant improvement in cognitive health and depression. Previous studies have also reported that between 30% and 50% of patients with epilepsy will underestimate or overestimate their cognitive performance, so objective assessments, in conjunction with caregiver reports as well as mood and anxiety measures, are required to establish truly comprehensive and accurate cognitive assessments [46,47]. MINDSET assists HCPs in identifying and referring those patients who require further objective neuropsychological assessment but does not provide a direct referral pathway for neuropsychological testing nor does it replace neuropsychological testing.

The words used in MINDSET were understandable, suggesting an acceptable reading level. Participants appreciated the program’s comprehensiveness while also indicating it was of appropriate duration, appraisals that are typically inversely associated. However, participants had difficulty when responding to and interpreting items that they perceived as incongruous to their experience. The “forced” selection resulted in inaccurate responses. The addition of a “not applicable” response option to previously published survey items and associated adjustment of the decision algorithm logic will help to increase response validity and increase the salience of AP recommendations. While not a majority opinion, the difficulties in AP development reinforced the advantage of involving caregivers in using MINDSET and generating the AP, and of ensuring HCPs reinforce it as well as instruct on its use.

A large sample size for usability testing is not necessarily required since statistical significance is not a determining factor to establish usability problems, but a limited sample does reduce generalizability to the broader epilepsy population [48]. MINDSET 2.0 offers the potential for increased accessibility and reach. Paradoxically, there is a “digital divide” that may naturally limit reach to some patients. The study’s requirement of access to internet-accessible devices introduces a selection bias toward higher socioeconomic status and greater digital literacy. Further, the system supports Spanish, but testing in Spanish was limited (n=1), and thus, no conclusions about user experiences with the Spanish translation can be drawn, but additional testing of the Spanish version is indicated. Patients with more severe forms of epilepsy that impair their ability to use MINDSET were also excluded from this study, further limiting generalizability to the broader population of people with epilepsy. Therefore, future testing of MINDSET among more representative clinical groups is needed.

MINDSET elicits self-reported responses from patients with epilepsy and is therefore subject to associated recall bias. Consistent with their long-term experience with epilepsy, participants were adherent to prescribed ASD dosing and medication management behaviors. However, self-reported medication adherence has low sensitivity for unintentional nonadherence because people with epilepsy tend to overestimate their self-reported medication adherence [49,50] and unintentionally fail to notice suboptimal ASD adherence [51].

MINDSET 2.0 was developed to be used independently, and patients indicated that they could complete it unaided, but the study supported the benefits of enabling assistance from caregivers and/or clinic staff, especially accounting for patients who have comorbid cognitive challenges. Caregiver and family support is an important associated factor, so content addressing this was suggested for inclusion since the concept of “familias” is particularly salient to the Hispanic culture [7]. Inclusion of caregivers, as well as clinic staff, can improve the validity of data collection, goal selection, and AP management. Studies on the application of MINDSET 2.0 in conjunction with caregivers and community health workers are planned.

This study indicates that MINDSET 2.0 may be a useful program for addressing low adherent ESM behaviors, providing tailored management intervention based on patient-mediated goals and strategy selection, screening for depression and memory function, identification of salient social determinants, and cuing linkage of patients to local resources designed to alleviate social needs and to evidence-based ESM epilepsy programs. Findings from this formative usability study replicated those from earlier versions of MINDSET but also highlighted the need for flexible protocols to enable caregiver support and lead to the initiation of enhancements to mitigate inaccurate algorithm decision logic and item response formatting prior to feasibility testing in neurology clinics [12,37].