Technologically-enabled health care is important at both the patient and service level, given the increasing resource and timing pressures on the health care system [1], the digital transformation of health-related industries [2], and changing consumer expectations about their role in care [3]. Electronic health (eHealth) refers to the organization and delivery of health services and information using the internet and related technologies [4]. eHealth holds potential as a scalable form of service delivery that is accessible, low-cost, promotes patient empowerment, and enhances patient-provider information exchange [5]. However, to reap the possible benefits, patients must be eHealth literate [6]. eHealth literacy refers to an individual’s ability to seek, find, understand, and appraise health information from electronic sources, and apply the knowledge gained to addressing or solving a health problem [6]. Limited ability to seek, find, understand, and appraise electronic health information has been recognized as a key self-reported barrier to the utilization of the internet for health purposes [7]. The first step in identifying individuals who may benefit from improved eHealth literacy is the development of valid and reliable tools assessing this construct.

The eHealth literacy scale (eHEALS) was among the first and continues to be one of the most commonly used self-reported measures of eHealth literacy [8,9]. The scale comprises 8 items, which assess consumers’ combined knowledge, comfort, and perceived skills at finding, evaluating, and applying electronic health information to health problems [8]. Consistent with the current definition of eHealth [4], all eHEALS items are specific to health information access via the Internet, as opposed to other electronic forms of information provision (eg, Compact Disc Read-Only Memory [CD-ROM], computer games). The scale was developed to address the need for an easily self-administrable eHealth literacy measure that could be applied across a wide range of populations and contexts [8]. Widespread adoption of the scale has been demonstrated, with the measure translated into multiple languages [10-17] and used across participants with diverse sociodemographic [10,15,16,18], ethnic [11,14,19], and disease profiles [13,20,21]. Items were originally developed and validated among Canadian youths more than a decade ago [8], and subsequent studies have demonstrated test-retest reliability across younger [14] and older age cohorts [10], internal consistency across populations of varying age and ethnicity [10,11,14,15,19,22], and measurement invariance across English-speaking countries [23]. However, inconsistent findings exist regarding the convergent and predictive validity of the scale [10,11,24], and debate continues about its factor structure [10-17,22,23,25-28]. We sought to contribute to this knowledge by assessing the factorial validity and internal consistency of a three-factor structure of the eHEALS.

Norman and Skinner’s original factorial validation of the eHEALS found that the scale assesses a single dimension [8]. Numerous studies with the general public have supported this finding [10,11,14-16,22,25,26], including those specific to children [15], university students [14,16], and older adults [10,22]. However, the strength of these conclusions is limited by the common use of exploratory factor analysis (EFA) [8,10,11,14,15,22,25,26]. EFA originates from classical test theory and holds value in the early stages of scale development when factor structure is unknown and latent variable structures need to be identified [29]. EFA does, however, have some limitations. For example, it often involves subjective decision-making processes and does not account for the theory which may inform latent variable structures [30].

Confirmatory factor analysis (CFA) is an alternative analysis technique, also derived from classical test theory, which allows models to be tested via theoretically or empirically-driven hypotheses [31]. However, studies assessing a unidimensional eHEALS structure using CFA commonly report poor fit indices [13,23,27,28]. This may be because a single factor structure does not account for the multifaceted nature of the concept of eHealth literacy, such as its inherent literacy types (ie, traditional, health, information, scientific, media, and computer) or the multiple components of information retrieval and use (ie, finding, applying and evaluating electronic health information) [6]. Paige and colleagues [13] completed one of the only studies of the construct validity of the eHEALS using CFA with chronically ill patients and found evidence for a three-factor structure. Despite this, multidimensionality of the eHEALS was refuted on the basis that a large proportion of variance loaded on one factor only. The authors applied the partial credit model, which is a unidimensional item response theory technique, to conclude that a single structure exists, despite CFA values indicating a poor unidimensional fit [13]. A two-factor model based on the concepts of information-seeking and appraisal has also been tested [12,27,28]. Although this model has a strong theoretical basis, 2 of the 3 studies testing this structure reported inadequate fit indices [12,27]. Furthermore, all were based on translated versions of the scale, which can result in varied item meaning and interpretation [32].

Sudbury-Riley and colleagues [23] used CFA to test a three-factor structure of the English-language version of the eHEALS with a multinational sample of adult internet users from the United Kingdom (n=407), New Zealand (n=276), and the United States (n=313). A hypothesis-driven approach was adopted, whereby 2 eHEALS items were mapped to an “awareness” factor, 3 items to a “skills” factor, and 3 items to an “evaluate” factor. These factors were derived from the self-efficacy and social-cognitive theoretical constructs underpinning eHealth literacy [8,23]. Self-efficacy theory is based on the premise that goal achievement is mediated by self-belief and confidence, and social cognitive theory states that social context influences goal achievement [33]. Sudbury-Riley and colleagues [23] therefore proposed that an individual’s awareness is shaped by their environment (eg, exposure to Web-based health information), their skills are influenced by social factors (eg, modeling, instruction, and social persuasion), and their ability to evaluate eHealth resources is mediated by their confidence and persistence. CFA fit indices supported the hypothesized three-factor eHEALS structure across all 3 countries [23].

Sudbury-Riley and colleagues’ [23] study contributes to our understanding of the underlying structures of the eHEALS, however, it has some limitations. In particular, a modified version of the scale was used, based on feedback from the authors’ family, friends, and colleagues, in which “and information” was added to items to address the increasing interactivity of eHealth materials. It is therefore unclear whether the three-factor structure also applies to the original version of the scale. The study was also conducted with middle-aged members of the general population, restricting the generalizability of findings across medical populations and age cohorts. This adds to the common underrepresentation of chronically ill patients in the eHEALS measurement literature, despite the potential benefits of eHealth to this population [13].

Given that evidence about the properties of a measure is accumulated over a number of studies, the appropriate next step it is to determine whether Sudbury-Riley and colleagues’ [23] findings can be replicated in a different population. To address this need, and also overcome some of the limitations of Sudbury-Riley and colleagues’ work [23], this factorial validation study was conducted with patients, using the standardized eHEALS. Magnetic resonance imaging (MRI) and computed tomography (CT) medical imaging outpatients represent a high volume of patients with diverse demographic characteristics and medical diagnoses [34,35], and as such, research completed with these patients may have high generalizability. Furthermore, MRI and CT medical imaging outpatients require substantial preparatory information that could potentially be delivered online [36]. Hence, this study aimed to test the factorial validity and internal consistency of the three-factor structure of the eHEALS, identified by Sudbury-Riley and colleagues [23], among MRI and CT medical imaging outpatients.

A cross-sectional survey of CT and MRI medical imaging outpatients was conducted in a medical imaging clinic at a tertiary referral hospital located in regional New South Wales, Australia.

Eligible participants were attending for an outpatient CT or MRI appointment at the tertiary referral hospital, were 18 years or older, and had access to the internet for personal use. Participants were excluded from the study if they had a cognitive or physical impairment that precluded them from providing informed consent or participating in the study, or if they were unable to complete the questionnaire because of poor English proficiency. These criteria mean that a diversity of participants in terms of frequency, confidence, and reasons for personal use of the internet were eligible to participate. Consistent with the original eHEALS validation study [8], use of the internet for health was not an eligibility requirement.

Patients who were potentially eligible for the study were identified by medical imaging reception staff when they presented for their outpatient appointment. These patients were informed about the research and invited to speak with a trained research assistant. Interested patients were provided with a written information sheet and introduced to the research assistant, who gave an overview of the study and obtained patients’ verbal consent to participate. The age, gender, and scan type of noninterested and nonconsenting patients were recorded. Consenting participants were provided with a tablet computer and asked to complete a Web-based questionnaire before their scan. A paper version of the questionnaire was provided to participants who requested it. Ethics approval was obtained from the Hunter New England Human Research Ethics Committee (16/10/19/5.11) and University of Newcastle (H-2016-0386).

Participants’ eHealth literacy was assessed using the 8-item English-language version of the eHEALS [8]. Respondents indicated their level of agreement with each statement on a 5-point Likert scale, which was scored from 1 “strongly disagree” to 5 “strongly agree.”

Sociodemographic, scan, and information preference characteristics were examined using standard items. These items assessed participant age, gender, marital status, highest level of education completed, postcode, and scan type. Postcodes were mapped to the Accessibility/Remoteness Index of Australia Plus 2011 classification to examine remoteness [37] and categorized as metropolitan (major cities of Australia) or nonmetropolitan (inner regional, outer regional, remote, or very remote Australia). One item, adapted from an existing health information wants questionnaire [38], assessed how much information participants liked to have about their health. Response options were “no information,” “some information,” and “a lot of information.”

Internet characteristics were assessed by 2 items. Use of the internet for scan preparation was assessed by an author-developed item: Have you searched the internet for information to help you prepare for your scan? with response options “no,” “yes,” and “don’t know.” Frequency of internet use was measured with a single item used in existing informatics literature [39], in which participants respond on a 6-point scale ranging from “less than once a month” to “several times a day.”

Rules of thumb for CFA recommend a sample size of at least 200 participants [40,41] or 10 participants per parameter estimated [42]. Wolf and colleagues [43] found that a sample size of at least 150 is required for three-factor models with fewer than 4 indicator variables per factor and assuming strong factor loadings of 0.80. To accommodate deviation from these assumptions, and given that 19 parameters were estimated for the eHEALS CFA, the more conservative estimate of at least 200 participants was applied to this study.

Participant characteristics and eHEALS responses were summarized as frequencies and percentages, or means and standard deviations. Consent bias was assessed for gender, scan type, and age group using chi-square tests. CFA was undertaken using the CALIS procedure of SAS software v9.4 (SAS Institute, Cary, NC, USA). We chose CFA as it is the same theoretically-sound technique used by Sudbury-Riley and colleagues [23] and therefore allowed for a direct comparison of results. Given the high completion rate (98.1% [256/261] of participants who started the eHEALS completed all items), this analysis was restricted to participants with complete eHEALS data. The relationship between latent variables (ie, awareness, skills, evaluate) and manifest variables (eHEALS items 1-8), as proposed by Sudbury-Riley and colleagues [23], was tested using structural equation modeling (Figure 1). All loadings were standardized, with variances fixed at 1. The model was estimated using the full information maximum likelihood method. Standardized factor loadings and covariances were calculated with 95% CIs.

Reliability measures included indicator reliability to determine the percentage of variation in the item explained by each factor, composite reliability to assess internal consistency (>.70 ideal) [29], and variance extracted estimates (VEEs) to determine the amount of variance captured by factors with regard to variance attributable to measurement error (>.50 ideal) [44]. Discriminant validity was assessed following the method proposed by Anderson and Girbing [45].

Model goodness of fit was assessed using a range of metrics. Absolute indices included the chi-square statistic, the chi-square to degrees of freedom ratio (<2 ideal) [46], and the standardized root mean square residual (SRMR; <.055 ideal) [29]. The incremental index was reported as the comparative fit index (CFI; >.95 good fit) [47]. The parsimony index used was the root mean square error of approximation (RMSEA; <.05 close approximate fit, .05-.08 acceptable fit, >.10 poor fit) [29,47]. Lagrange multiplier (LM) estimates of items on different factors were assessed to identify complex items and possible ways to improve the model.

A total of 405 potentially eligible patients were invited to discuss the study with a research assistant during the 7-week recruitment period. Of the invited patients, 87.4% (354/405) were interested in participating in the study, and of these, 75.7% (268/354) were eligible. Of these eligible participants, 97.4% (261/268) started the eHEALS, and 95.5% (256/268) completed all eHEALS items. There were no significant differences between patients who were and were not interested in participating in the study based on gender, scan type, or age group. Table 1 provides a summary of the sociodemographic, scan, and internet characteristics of eligible participants. Multimedia Appendix 1 provides a summary of participant responses to eHEALS items.

Convergence between the implied and observed variance covariance matrices was achieved within 10 iterations. As shown in Table 2, all factor loadings were at or above 0.80 and were statistically significant (P<.001). All CRs exceeded .70, indicating good reliability, and all VEEs exceeded the cutoff of .50, indicating convergent validity. Discriminant validity of the model was demonstrated, with statistically significant chi-square difference-tests (P<.001) for each pair of factors. The absolute index SRMR was .038, indicating adequate fit to the hypothesized model. The incremental index CFI was .944 and therefore close to the .95 threshold of acceptability (Table 3). However, the chi-square statistic (χ²₁₇=124.2) was highly significant and suggestive of poor fit, and the chi-square statistic to degrees of freedom ratio of 7.3 exceeded the acceptability cutoff of 2 [46]. The parsimony index RMSEA was .16, indicating poor fit.

When investigating the possible reasons for less than ideal fit, LM estimates provided strong evidence for a path between item 3 “I know how to find helpful health resources on the Internet” and the awareness factor (LM estimate 107.66; P<.001). There was also strong evidence for a path between item 2 “I know where to find helpful health resources on the Internet” and item 3 “I know how to find helpful health resources on the Internet” (LM estimate 91.11; P<.001). Given apparent overlap between items 2 and 3, a 7-item model which excluded item 3 was tested, which indicated good model fit (Table 3). See Multimedia Appendix 2 for factor loading and residual error estimates for this altered model.

This study was the first to examine the theoretically-derived three-factor structure of the eHEALS, as proposed by Sudbury-Riley and colleagues [23], among a sample of MRI and CT medical imaging outpatients. This three-factor structure was supported, with 2 out of 3 goodness-of-fit indices indicating adequate fit to the hypothesized model. Although these findings oppose accumulated evidence for a unidimensional structure of the eHEALS [8,10,11,14-16,22,25,26], they are consistent with the social cognitive and self-efficacy theory underpinning eHealth literacy [8,23,33]. As a result, it may be timely for researchers to examine patients’ eHealth literacy across eHEALS factors to inform targeted eHealth literacy improvement interventions. This study contributes important knowledge about the structure of the eHEALS, yet further factorial analyses, including multidimensional item response theory analyses, are required across populations to increase the reliability of these findings.

CFA was selected as it represents an understudied yet rigorous aspect of classical test theory and logically extends on the existing body of EFA and CFA measurement literature. The recent emergence of item response theory analyses of the eHEALS [12,13,16] has advantages over classical test theory approaches, including the capacity to establish increased item level psychometric information (eg, item difficulty). The application of multidimensional item response theory techniques to validate the three-factor eHEALS structure should be explored further. Furthermore, this study assessed one psychometric property (ie, factorial validity), and more research is needed to investigate other understudied measurement properties of the eHEALS, such as its predictive validity.

It is possible that findings may not be generalizable beyond the medical imaging context. Similarly, as most participants reported using the internet at least daily (75.3%, 201/267), study findings may not be generalizable to those who use the internet less frequently. As we did not ask participants about the activities they undertook online, it is unclear whether the results are applicable to those who do or do not use the internet for health. Future research is consequently needed to validate study findings across patients with diverse demographics, medical diagnoses, and internet use patterns. Additionally, our study was based on the standardized version of the eHEALS. As recognized in prior research [12,23], this version may not sufficiently capture competency in using Web 2.0 (eg, social networking) for health. Further research is needed to determine whether scale modifications are needed to reflect the evolving nature of eHealth interventions.

Although potential item redundancy impacted fit indices, the three-factor structure of the eHEALS was broadly supported. On the basis of these findings, the eHEALS could be used to inform the development of tailored eHealth literacy enhancement strategies, which may in turn increase engagement with Web-based health resources. Further research is needed to confirm the three-factor structure across other medical settings and populations to support the generalizability of these findings.