A cross-sectional study was designed using an online self-administered questionnaire (the survey is available upon request). The survey was written in English and then translated into Dutch and verified by a person proficient in Dutch. The questionnaire was distributed throughout several Facebook groups aimed at (peer) support and providing information for patients who are chronically ill in May 2020.

After displaying the introduction of the questionnaire, informed consent was obtained electronically, before actual enrollment. It was explicitly stated that participation was voluntary, and participants could withdraw at any time without any consequences. Moreover, complete anonymity of the response was ensured.

During the period of data collection, the number of Dutch people who tested positive for COVID-19 exceeded 40,000, over 11,000 people were hospitalized, and almost 6000 deaths related to COVID-19 had been reported in a population of 17 million inhabitants [26]. When distributing the questionnaire, the national intelligent lockdown had been active for approximately 1.5 months. As compensation for the time spent on the survey, online gift codes were distributed through a raffle. To ensure good quality responses, some survey items were programmed to be restricted in range so that incorrect inputs were not allowed.

This study followed the Institutional Review Board (IRB) for the Protection of Human Subjects Guidelines. All procedures in this study were approved by the IRB (2020/04/24-61392qko) prior to its initiation.

Since the study focused on patient portals as implemented by Dutch hospitals, the targeted population for our study was patients who are chronically ill and residing in the Netherlands. Inclusion criteria were as follows: patients aged between 18 and 65 years, having at least one chronic illness, and having spent more than 2 minutes completing the questionnaire. Questionnaires that were not completed were removed from the final data set.

Descriptive analysis was performed to gain insight into the patient population, portal users, and nonusers regarding their demographics, physical health status, hospital visits, and mental health status. Additionally, chi-square tests (for categorical variables) and Welch t tests (for numerical variables) were performed to compare the characteristics between portal users and nonusers. Next, univariate analyses were performed for the dependent variable (portal use) to detect its possible predictors. Variables with P<.20 in the univariate regression were consequently included in the multiple regression analysis after considering the correlation between variables (using statistical analysis and expert opinion). A stepwise backward elimination was then applied to reduce the number of independent variables and obtain the final multiple regression model. This approach allowed thorough exploration and testing of possible predictors to arrive at a final model [35]. Similar procedures were applied to the other two dependent variables (willingness to recommend among portal users and likelihood of use among portal nonusers) to get the final multiple regression models. All analyses were performed using RStudio (version 4.0.2; RStudio, PBC).

A total of 652 respondents started the questionnaire, whereby 461 respondents completed it successfully. Only completed questionnaires were used in the final data set for analysis. Of all the participants, 307 (66.6%) reported to have used a patient portal, and 154 (33.4%) reported that they had not used a patient portal until the moment the survey was conducted.

Demographics of all participants, portal users, and nonusers are displayed in Table 1. From our sample (N=461), 94 (20.4%) were male, 365 (79.2%) were female, and 2 (0.4%) individuals identified as other. The mean age of the sample was 42.9 (SD 13.0) years. The number of participants that reported having a single chronic disease was 302 (65.5%), and 159 (34.5%) reported having multiple chronic diseases. Significant differences were noted in the mean age (P=.008) and main occupations (P=.03) between portal users and nonusers.

Table 2 displays the physical health status, hospital visits, and mental health status of all participants, portal users, and nonusers. The majority of the 461 patients reported to have their illness “a little bit” (n=113, 24.5%) to “moderately in control” (n=229, 49.7%), 66 (14.3%) reported to have total control, and 46 (10.0%) reported to have no control over their illness at all. Furthermore, few patients (n=30, 6.5%) reported that their perceived health was very good or excellent, while the majority reported good (n=127, 27.6%), fair (n=177, 38.4%), or poor (n=127, 27.6%) perceived health. Moreover, the majority of respondents reported spending 0.5 hours to 1 hour (n=151, 32.8%) and 1 hour to 2 hours (n=166, 36.0%) each time they visit a hospital (including travel time); 70 (15.2%) respondents spent less than half an hour, while only 19 (4.1%) spent more than 3 hours. Furthermore, only 18 (3.9%) participants reported having or having had COVID-19, 130 (28.2%) were uncertain, and 313 (67.9%) reported that they never had COVID-19. About mental health, most participants reported having no (n=172, 37.3%) to mild forms of (n=158, 34.3%) depression, while only 9.8% (n=45) reported having moderately severe or severe depression. Relative to life satisfaction, 146 (31.7%) and 117 (25.4%) of the participants were satisfied and slightly satisfied with their lives, respectively. Moreover, 27 (5.9%) and 28 (6.1%) were extremely satisfied and extremely dissatisfied with their lives, respectively. Among the measured characteristics, level of control (P=.005), average time of hospital visits (P=.04), depression (P=.02), and life satisfaction (P=.005) were significantly different between portal users and nonusers.

Table 3 reports the frequency of portal use before and after the lockdown. An increase in the frequency of portal use has been observed after the lockdown as compared to before, whereby the relative difference was 500%, 221.1%, and 8.3% in daily, weekly, and monthly use, respectively. After the lockdown, 67 (21.8%) reported daily to weekly use, and 106 (34.5%) have used the patient portal monthly. Among all the portal users, the most common use times were “5 minutes or less” (n=121, 39.4%) and “5-10 minute” (n=124, 40.4%), while 62 (20.2%) of them reported using the portals for more than 10 minutes.

In relation to the maximum acceptable waiting time, nonusers reported a lower maximum acceptable waiting time than users. Among the users, 78 (28.4%) reported a longer actual waiting time than they deem acceptable. Finally, among portal users, 257 (83.7%) would likely recommend portals to others, and among nonusers, the average likelihood of future use (ranging from 0% to 100%) was 53.6% (SD 33.3%).

To investigate which combinations of the different predictors could best explain the variance in portal use versus nonuse and portal users’ willingness to recommend and portal nonusers’ likelihood of future use, three separate regression models were constructed after performing univariate regression analysis and considering possible correlations. In the first analysis (model 1), a logistic regression was performed to investigate the association between portal use and the included variables after the first steps, which were age, hospital visit time, level of control, depression, and life satisfaction. In the second analysis (model 2), a logistic regression was performed to study the relationship between portal users’ willingness to recommend and the variables average number of hours spent on the internet daily, the frequency of portal use after the COVID-19 lockdown in March 2020, waiting time for portal response, and maximum acceptable time to wait. In the third analysis (model 3), a multiple regression analysis was conducted between portal nonusers’ likelihood of use and age, income, maximum acceptable waiting time, and their awareness of patient portals’ existence as candidate variables. The results of the regression analysis are displayed in Table 4.

Regression results of model 1 showed that shorter hospital visit times (“less than half an hour”) predict less portal use (β=–.725; P=.03) compared to longer visit times. Compared to “totally under control,” moderate level of control predicts a higher chance (β=.629; P=.04) of portal use. Two mental health conditions were shown to significantly affect participants’ portal use. Participants with severe depression (β=–1.652; P=.03) and life dissatisfaction or extreme life dissatisfaction (β=–.844; P=.02) were found to be less likely to use patient portals. Furthermore, age demonstrates a small yet nonsignificant impact on portal use, whereby older age negatively affects portal use (β=–.015; P=.08).

Among portal users, the logistic regression results from model 2 showed that actual waiting time and maximum acceptable waiting time were the strongest predictors of users’ willingness to recommend. Participants whose average waiting time was between 1 to 2 days (β=–2.081; P<.001) or greater than 2 days (β=–1.784; P<.001) were less likely to recommend the portal system to others, compared to those who received responses via portal systems within 24 hours. Participants who reported a moderate maximum waiting time (1-2 days) were more likely to recommend portal systems (β=2.292; P<.001).

For portal nonusers (model 3), awareness of the portal existence was the strongest predictor besides maximum acceptable waiting time. Among nonusers, 85 (55.2%) reported being unaware of the existence of a patient portal at their hospital. Participants that were unaware of the existence of portal systems were 25.9% (P<.001) more likely to use portal systems, compared to those that already knew of their existence before the time of the survey. Participants who had a moderate maximum acceptable waiting time (12-24 hours) were 21.2% (P<.001) more likely to use portal systems in the future. Furthermore, middle income class participants (€30,001 [US $35,440.20] to €40,000 [US $47,252.00]) were 15.3% (P=.01) more likely to use portal systems compared to low income class participants (<€20,001 [US $23,627.20]), and older-aged participants also showed a slightly lower likelihood (β=–.003; P=.10) of use.

Although the societal and health impacts of the COVID-19 pandemic have been present for nearly a year, there is no evidence on factors that affect patient portal adoption among patients who are chronically ill during the COVID-19 pandemic. Moreover, little research has been done on what influences users’ willingness to recommend and nonusers’ likelihood of using patient portals during the COVID-19 pandemic. Our findings portray some interesting insights for portal service providers and health care professionals.

In the participant population under study, we found that almost 67% (307/461) of participants were portal users, which is much higher than for general patient populations reported [36-38]. For example, Griffin et al [36] found in their study that 83.4% of patients were nonusers of the UNC Chart patient portal among a general patient population. It could be attributed to the difference in the study population and the impact of the COVID-19 pandemic. Ancker et al [39] found that patients with chronic illness were more likely to use a patient portal. Table 3 shows that both the number of portal users and frequency of use have increased significantly after the lockdown initiation in spring 2020. We found that participants whose level of control was moderate had a higher likelihood of using portal systems than participants with total control. This may be attributed to participants’ perception whereby they deem a portal as unnecessary when their health is well managed and under control.

Besides, we found that participants with shorter visit times to a hospital have a reduced likelihood of portal use compared to those with longer visit times. As reported in many other studies [40-42], savings on travel time and cost are among the major benefits of eHealth. This result suggests that the convenience of physical visits most likely reduces remote visits using patient portals. Furthermore, participants with severe depression and lower life satisfaction tend to use patient portals less. Mental health problems likely deter patients from using portal systems. This result coincides with the observation that patients with chronic anxiety and depression are less likely to be intense eHealth users [37]. Future studies should focus on determinates of portal use and nonuse in this specific population.

Our results show that older age may negatively affect portal use. It is in line with a recent study in the older population on the intention to use medical applications. Feelings of having control, service availability, perceived ease of use and usefulness, and attitude toward the medical application affect the intention to use in older adults, which may be attributed to anxiety triggered by technology use, lack of privacy, or trust [43]. Another study also argues that this is probably because older people often lack the infrastructure, knowledge, and skills needed to use eHealth programs [44]. Future studies are required to investigate determinants of portal use and nonuse in the older adult population.

Willingness to recommend patient portal systems was also investigated. No less than 83.7% reported willingness to recommend the portal to family and friends, which suggests that most users were satisfied and loyal with their hospital’s patient portal system [34]. The average waiting time to receive a response was a strong predictor for users’ positive experience using portal systems. Approximately 29% of patients reported receiving responses within 2 hours of a request, which is considered rapid. Numerous studies in the appointment scheduling area have shown the importance of managing waiting time in health care management [45-47]. Marketing research has shown that waiting time is a crucial determinant of customer satisfaction and loyalty [48]. Nonusers seem to expect faster response rates from patient portals than users. Palawatta [49] demonstrated that if nonusers perceive the response rate is longer than their perceived acceptable waiting time, they will feel less satisfied and, therefore, less inclined to try the portal system. Users, if they experience disconfirmation in waiting time and maximum acceptable waiting time, are likely to be less satisfied and therefore less likely to remain committed to using the portal system. These are essential insights for health care practitioners and managers to leverage operational efficiencies such as appointment scheduling and resource allocation.

Among the nonuser group, the majority (85/154, 55%) reported not being aware of a patient portal system at their hospital. Awareness of portal systems was found to be the largest predictor for future use in our study. It seems that many patients do not use portals partly due to unawareness of their existence. This result is in line with Griffin et al [36], who found that patients often did not use patient portals simply because they were unaware of their existence. This result suggests that enhancing the awareness of portal systems is the first step for health care organizations to take to increase portal use.

There are some limitations bound to this study. First, the survey is cross-sectional, making it impossible to make causal claims, limiting the study to predictions only. Furthermore, the study relies on self-reported data on portal use. This is because when the study was implemented, we did not have access to the actual use data, such as log data of portal users. Besides, this survey focused on both portal users and nonusers to study factors that influence portal use and future use of nonusers. This, for example, cannot be replicated by merely approaching the actual users. However, it would be more insightful to use real use data (eg, log data retrieved from the portal) to establish the length and frequency of use. We suggest this as a future study when access to portal data is possible.

Second, the study invited participants via social media (Facebook peer support group) to complete the self-administered questionnaire. On one hand, sampling from Facebook support groups has apparent benefits, such as convenience and its focus on the targeted population. On the other hand, it also has a few known biases [50,51]. For example, Facebook excludes people who have a lower eHealth literacy, one important predictor of portal use among adult patients [22]. Besides, not everybody uses Facebook, especially older people. Although this problem is partly compensated by focusing on the age group 18 to 65 years, our results might overestimate the proportion of portal users among the total population. This partly explains why the ratio of portal users is higher than reported in many other studies. Little is known about the characteristics of people who do not use technology and why they do not use the portal. We suggest that future studies should focus on older people and people with less eHealth literacy.

Moreover, more females than males participated in this study. According to Smith [52], females are more likely to respond to (online) surveys than males. The authors proposed different reasons that could be grounded to this observation, including behavioral differences between males and females in relation to the internet or inherent internal feelings. Another study [53] found similar results (70% female response).

Finally, it is important to see which functionalities users use and the respective frequency to understand the perceived value of these functionalities to patients. This will potentially improve the frequency of use and tailoring portal systems according to the needs of patient. Future research could build on our results, aimed at further investigation of the use dimension of patient portals.

Individuals that have spent less time on physical hospital visits, whose health is moderately under control or with severe depression or lower life satisfaction are less likely to use patient portal systems. Among users, short waiting time was the most important predictor for satisfaction of portal use, and among nonusers, awareness was the most important predictor of future portal system use. These findings provide insights for health care providers on how to promote patient portal use and improve user satisfaction.