Original Paper
Abstract
Background: The internet is a key source of health information, but the quality of content from popular search engines varies, posing challenges for users—especially those with low health or digital health literacy. To address this, the “tala-med” search engine was developed in 2020 to provide access to high-quality, evidence-based content. It prioritizes German health websites based on trustworthiness, recency, user-friendliness, and comprehensibility, offering category-based filters while ensuring privacy by avoiding data collection and advertisements.
Objective: This study aims to evaluate the acceptance and usability of this independent, noncommercial search engine from the users’ perspectives and their actual use of the search engine.
Methods: For the questionnaire study, a cross-sectional study design was used. In total, 802 participants were recruited through a web-based panel and were asked to interact with the new search engine before completing a web-based questionnaire. Descriptive statistics and multiple regression analyses were used to assess participants’ acceptance and usability ratings, as well as predictors of acceptance. Furthermore, from October 2020 to June 2021, we used the open-source web analytics platform Matomo to collect behavior-tracking data from consenting users of the search engine.
Results: The study indicated positive findings on the acceptance and usability of the search engine, with more than half of the participants willing to reuse (465/802, 58%) and recommend it (507/802, 63.2%). Of the 802 users, 747 (93.1%) valued the absence of advertising. Furthermore, 92.3% (518/561), 93.9% (553/589), 94.7% (567/599), and 96.5% (600/622) of those users who used the filters agreed at least partially that the filter functions were helpful in finding trustworthy, recent, user-friendly, or comprehensible results. Participants criticized some of the search results regarding the selection of domains and shared ideas for potential improvements (eg, for a clearer design). Regression analyses showed that the search engine was especially well accepted among older users, frequent internet users, and those with lower educational levels, indicating an effective targeting of segments of the population with lower health literacy and digital health literacy. Tracking data analysis revealed 1631 sessions, comprising 3090 searches across 1984 unique terms. Users performed 1.64 (SD 1.31) searches per visit on average. They prioritized the search terms “corona,” “back pain,” and “cough.” Filter changes were common, especially for recency and trustworthiness, reflecting the importance that users placed on these criteria.
Conclusions: User questionnaires and behavior tracking showed the platform was well received, particularly by older and less educated users, especially for its advertisement-free design and filtering system. While feedback highlighted areas for improvement in design and filter functionality, the search engine’s focus on transparency, evidence-based content, and user privacy shows promise in addressing health literacy and navigational needs. Future updates and research will further refine its effectiveness and impact on promoting access to quality health information.
doi:10.2196/56941
Keywords
Introduction
Background
The internet has become a crucial resource for accessing medical information, making it a valuable tool for individuals seeking knowledge about health. In Germany, medical professionals, notably physicians, have long been the primary source of health care information for the general population [
, ]. However, the landscape is evolving, and the internet now stands as the second most significant channel for health-related information [ ], indicating a new trend in how people seek medical knowledge. A study conducted by the Bertelsmann Foundation showed a strong and increasing demand for health-related information online, including a rapidly increasing demand by older generations [ ]. In 2020, approximately 70% of the German population used the internet to actively engage in searching for health-related content [ ]. While these numbers have varied in recent years, there is an overall trend of an increased use of the internet [ ].A major complaint that users have about health-related information online is the lack of clarity regarding the trustworthiness and seriousness of existing websites. The information online has varying levels of quality [
, ]. Frequently used and popular websites are regarded as reliable by users, irrespective of the actual quality of their content, while independent public websites are not as well known and do not seem to be more reliable to users [ ]. The reliance on digital platforms for health information also poses a challenge due to the profound influence of the order of the results generated by popular search engines. Users often initiate their search for medical information through well-known search engines, predominantly relying on the first few search results displayed on the search engine results page (SERP), which are often advertising [ - ]. Unfortunately, these top hits are not only clicked on more frequently but are also perceived as more trustworthy, despite the variation in information quality [ , ]. This highlights the ambiguity of users feeling incompetent in finding reliable information online, while at the same time trusting the top results and being satisfied with the information they find. A more recent study evaluated the quality of online health and nutrition information related to cancer supplements through a Google search, using the Health Information Quality Index [ ]. The 160 relevant search results yielded median and mean Health Information Quality Index scores of 8, with one-quarter of the results scoring high (10-12). No correlation was found between high quality scores and an early appearance of these results, indicating potential limitations in using Google for obtaining accurate information on dietary supplements and cancer, particularly given the prevalence of advertisements outnumbering search results [ ].The problem with the reliance on search engines becomes even clearer when considering the low health literacy and digital health literacy of the German population [
- ]. The first Health Literacy Survey Germany study in 2014 [ ] underscored a significant health literacy challenge in Germany, with 54.3% of the population reporting health literacy problems. This result pointed to challenges in navigating the health care system and understanding health-related information, with identified associations with factors such as age, migrant background, self-assessed social status, and functional literacy. The second Health Literacy Survey Germany study in 2020 [ ] reinforced previous concerns with a further increased rate of individuals (58.8%) struggling considerably with health information. Even more so than in 2014, evaluating the trustworthiness of health-related information presented difficulties [ ]. Up to 45% of the German population exhibit “problematic” or even “inadequate” internet skills [ ]. Many individuals face so-called navigational needs. They depend on support from others to efficiently search and evaluate health information online [ ]. As the amount of digital health information grows each year, the challenge of effectively navigating through this vast array of content becomes increasingly daunting, raising concerns about misinformation and health-related misconceptions.During the work on the comprehensive GAP (Gut informierte Kommunikation zwischen Arzt und Patient, meaning “Well-informed communication between physician and patient” in English) project, which addressed evidence-based information [
- ], we studied the needs and requirements of internet users for health information online [ ]. We developed criteria to assess web domains according to these needs. Assessing the quality of websites is nothing new. There are existing seals of approval, such as the afgis (Aktionsforum Gesundheitsinformationssystem eV, meaning “Action Forum Health Information System eV” in English) and Health On the Net Foundation Code of Conduct seals, which check whether the website operator meets certain transparency and quality standards. An afgis certificate is valid for 1 year. However, with Health On the Net Foundation Code of Conduct, there were several issues [ ], and on December 15, 2022, the Health On the Net Foundation discontinued its services. Different seals potentially signify different standards of certification. Another issue with these seals is that the pages that have a seal cannot easily be found via a central search. To address these issues by centrally and consistently rating domains inside a search platform, we developed a search engine in 2020. This search engine was originally called “GAP search” and later renamed as “tala-med Suche” in German and “tala-med search” in English.The topic of data retrieval in medicine is generally relevant and widely researched. Efforts ranging from decentralized search systems for patient data in registries [
] to tools such as Informatics for Integrating Biology and the Bedside (i2b2) [ ] that allow the creation of cohorts, for which data security plays an important role [ ], have been investigated. In addition, for public data, such as the Medical Subject Headings (MeSH), search capabilities play an important role [ ], including the requirement for an intuitive user interface design [ ]. In terms of scraping web content, which could be used to create a search platform, the Sampled German Health Web [ , ] created an index of German-language health-related web pages using a special focused crawler. The Sampled German Health Web index was restricted to pages from the top-level domains .de, .at, and .ch, and the domains were automatically filtered for health-related content while crawling using a support vector machine. By contrast, the new tala-med search engine, which was used in this study, relied on hand-picked, high-quality, German-language health websites. These domains—more than 50 of them—underwent a rigorous evaluation process. This evaluation was based on the categories trustworthiness (with the subcategories authority, independence, and evidence based), recency, user-friendliness, and comprehensibility. The ratings of these categories influence the order of the results on the SERP. The search engine also values user privacy by avoiding data collection and advertising. The detailed design of the search engine is described in the Methods section and in -3.By assessing the acceptance and usability of this new technology, we aimed to improve the implementation of the search tool. Numerous studies in the field emphasize the significance of outcome measures as crucial indicators of a technology’s effectiveness, user-friendliness, and potential for widespread adoption. User acceptance, often determined by perceived usefulness, ease of use, and individual attitudes, is critical for a technology’s long-term viability [
, ]. Moreover, the usability of a technology, including learnability, efficiency, and user satisfaction, significantly affects user experience and reduces possible adoption barriers [ ]. Evaluating these dimensions can provide insight into how well technology meets users’ expectations, addresses their needs, and contributes to successful implementation. In the context of health care technology, this scrutiny is particularly relevant because effective tools for accessing medical information can significantly impact health care outcomes [ , ].In this study, the acceptance and usability of our newly developed search engine was evaluated by means of a questionnaire study and actual use by tracking user behavior during the first months of operation.
Related Works
The usability and accessibility of digital platforms are essential for ensuring that all users, including those with disabilities, can effectively engage with websites and mobile apps. Mateus et al [
] conducted a systematic mapping of accessibility issues, revealing that automated tests covered <40% of accessibility problems on websites and even fewer on mobile apps. The study stressed the importance of including users with disabilities in evaluations because user testing uncovered many issues missed by automated tools and expert inspections. This underscores the need for a comprehensive evaluation approach that combines expert and real-world user input to improve accessibility.Petrie and Bevan [
] expanded on this by exploring the interplay between usability, accessibility, and user experience. The authors defined usability in terms of effectiveness, efficiency, and satisfaction, focusing on aspects such as learnability, flexibility, and safety. Importantly, they emphasized that accessibility is an integral component of usability and that digital systems should cater to the widest possible range of users, including those with disabilities. They also introduced the concept of user experience, which goes beyond usability to include users’ emotional responses and satisfaction with a system. This holistic approach is essential for creating digital platforms that are both functional and engaging for all users.Belinda et al [
] provided a deeper look into the internal and external usability factors that affect website performance. The authors used automated tools such as GTmetrix and Website Grader to measure internal attributes such as performance, load time, and page size, while external attributes such as ease of navigation and user satisfaction were assessed through surveys. Their findings revealed that some websites performed well from a user perspective but were found lacking in technical performance, particularly in terms of load times and page requests. This highlights the need to address both internal and external usability factors to create well-rounded digital platforms.Kritz et al [
] explored the online resources and tools used by European physicians to gather medical information. The authors found that physicians frequently relied on general search engines and faced significant barriers in accessing high-quality, trustworthy medical content. Medical specialists were more likely to use medical research databases, while general practitioners often faced barriers such as lack of time and language restrictions. The study highlighted the need for improved medical search tools tailored to the specific needs of different physician subgroups. Kritz et al [ ] concluded that user-centered medical search tools could significantly improve accessibility and the quality of online medical information.Strecker et al [
] also contributed to the usability of medical search tools, focusing on the MeSH Browser. The authors evaluated a newly developed multilingual MeSH Browser, which introduced improvements in user interface design to enhance the accessibility and usability of medical literature searches. The results showed that contemporary web design principles led to significant improvements in navigation and overall user satisfaction, further underscoring the importance of continual evaluation and enhancement of information systems in the medical realm.Eysenbach and Köhler [
] studied how consumers search for and appraise health information on the internet. The qualitative study revealed that users, despite using suboptimal search techniques, were able to retrieve health information quickly. However, users rarely checked critical indicators of credibility, such as the “About us” sections or disclaimers, relying instead on superficial factors such as professional design and ease of use. This reliance on superficial credibility indicators poses risks, particularly in the health care field, where the accuracy of information is paramount. Eysenbach and Köhler [ ] suggested that further research is needed to develop educational and technological tools that guide users toward high-quality health information.Zhang [
] expanded the understanding of how consumers select sources for health information by identifying 5 categories of factors that influence source selection: source-related factors, user-related factors, user-source relationships, characteristics of the problematic situation, and social influences. The study also identified a range of criteria that mediate the influence of these factors on source-selection decisions, including accessibility, quality, usability, interactivity, relevance, usefulness, and familiarity. Zhang [ ] concluded that a personalized approach to health information systems is necessary to provide effective access to health information because different consumers prioritize different factors when selecting sources. This insight strongly indicates the need for more personalized information services that cater to individual user preferences and needs.In conclusion, the literature emphasizes the critical role of accessibility, usability, and user experience in the design of digital platforms, especially health care websites. While Saad et al [
] have highlighted general usability problems in health care websites, the studies by Kritz et al [ ], Eysenbach and Köhler [ ], and Zhang [ ] focus on the challenges that users face in retrieving and assessing the credibility of health information online. A personalized and user-centered approach to the design of health care information systems would improve the accessibility and quality of health information, meeting the diverse needs of consumers and health care professionals alike [ , ].Methods
Search Engine Development
Requirements
The functional requirements for the tala-med search engine included the ability to crawl and index health information websites, implement quality assessment filters, and provide search term suggestions and synonym handling. The nonfunctional requirements focused on maintaining user privacy through self-hosting, ensuring a user-friendly interface, and optimizing performance to handle large sets of synonym mappings efficiently.
Implementation
Design
To implement tala-med search, we used software with a self-hosting capability because of privacy considerations. Furthermore, modern technology with single-page application design using a web application programming interface was required to compete with modern search engines. Therefore, the selection available was limited. As no single product met all our requirements, we built a search engine stack ourselves, consisting of crawler, middleware, and front end, using existing components and established technologies.
Back End
We created the web index using the open-source crawler software Fess [
]. For boilerplate removal, we used Mozilla’s Readability tool to strip HTML tags and display core content, cleaning up crawled websites for our search index [ ]. As middleware, we used Elastic App Search [ ] (now part of Elastic Enterprise Search [ ]), which relies on Elasticsearch. This software allows for setting up weighting mechanisms for the search equation ( ) and configuring filters. In our case, a set of criteria for quality assessment were developed, which we integrated as filters. These quality assessment criteria were adapted from a systematic review by Eysenbach et al [ ] that identified criteria used from 1969 to 2001 for evaluating the quality of health information online. We identified 74 criteria [ ] to assess the content quality of different health-related websites and rigorously evaluated >50 German-language health information providers ( ) using these criteria, assessing 1 main page and 5 randomly selected subpages per provider. To compile the list of relevant providers, we started with a list from a Bertelsmann study [ ] and adapted it with the help of domain experts. The 74 quality criteria were later condensed into 4 categories: trustworthiness, recency, user-friendliness, and comprehensibility. The scores of these categories were added to the search index after boilerplate removal and divided into 4 value ranges reflecting the quality of each category, using 4 quantiles for differentiation. These scores served as quality indicators and filter categories, influencing the order of the results on the SERP. More details about the internal mechanisms of the search ranking can be found in .App Search supports the consideration of synonyms, which is crucial in medical language due to varied etymology and numerous abbreviations. Our goal was to enable the search engine to handle synonyms effectively. In App Search, sets of up to 32 synonyms can be created, allowing synonyms of a matched word to be considered in searches. We generated synonym sets by collecting word types and pairs from the document text, filtering against a German stop word list, resulting in approximately 6 million unique entries. These entries were matched against an experimental German interface terminology—SCT-GIT—linked to Systematized Nomenclature of Medicine–Clinical Terms (SNOMED-CT) [
], yielding approximately 40,000 mappings to SNOMED-CT codes. Synonym links in SCT-GIT were then used to add synonyms for each matching string, finding at least 1 synonym for approximately 30,000 SCT-GIT terms. We limited the number of synonym sets to 500 to maintain performance, using only sets with at least 4 synonyms, resulting in a median set size of 4.Front End
The front end was custom developed with a slim design inspired by Strecker et al [
], featuring search term suggestions and a user-friendly filter selection ( ). After entering a search term, the interface displayed a SERP with evaluated German-language health information providers, showing their scores in the 4 filter categories through graphical indicators ( ). Hovering over these graphics revealed explanations, and the bottom of the page included pagination and a footer with logos and links to pages displaying the imprint and data protection information.

Pilot Testing
The search engine was pilot-tested between October 2020 and the end of June 2021. Adapting to the challenges posed by the COVID-19 pandemic, the pilot testing format shifted from a face-to-face setting to a web-based setting with a small group of academic staff from the University of Freiburg. The evaluation, involving 12 participants, consisted of screen recordings of search tasks and a qualitative survey regarding user-friendliness. After the pilot tests, technical enhancements were implemented, guided by the qualitative insights. These findings formed the basis for the subsequent questionnaire study.
Availability
The current version of the search engine can be accessed on the web [
]. Please note that there have been some significant changes to the website structure since this study was conducted.Evaluation
Design
We used 2 data collection methods to evaluate user behavior. First, we asked users to fill out a questionnaire (self-perception). Second, we recorded user behavior via web tracking independently and unreferenced to the questionnaire (external perception). The study protocol was published in 2019 [
].Questionnaire Study
The study used a cross-sectional design with data collected through a web-based questionnaire. The questionnaire was developed after pilot testing of the search engine and underwent an internal pretesting round before recruitment began.
Recruitment
The study used a recruitment strategy aimed at enrolling 200 participants. The first rounds of recruitment, starting in October 2020, which involved sending email invitations to local health initiatives, professional networks for physicians, and personal contacts, were unsuccessful in obtaining the desired number of participants. Participants were then recruited from the WisoPanel, an established online panel of German-speaking individuals [
]. The panel provided a pool of 14,900 potential respondents who had previously expressed an interest in participating in research studies. The survey was open for anyone accessing the link, but only participants from the panel (>70% of the data) were included in the analysis of this study to maintain a credible sample. Recruitment via the panel took place in May 2021. The survey ended at the end of May 2021. Panel members were invited to participate in the study via email. The email informed participants about the project and the aspects distinguishing the new search engine from other platforms.To ensure that participants had firsthand experience with the platform before proceeding to the survey, they were asked to test the search engine first. We recommended that participants carry out 1 or 2 searches and familiarize themselves with the SERP and the features and functions of the website.
Survey Administration
The hyperlink that led participants to the search engine was sent to the panel members by email. Subsequently, the link to the survey was displayed as a pop-up element after 10 to 20 seconds of interaction with the search engine. Throughout the assessment phase, a button to access the survey was strategically positioned in the top right corner of the website to allow a direct link to the survey interface. No registration was required to complete the survey, ensuring seamless and voluntary participation. Participants had the option to cancel their responses at any time, and there were no time restrictions regarding the completion of the questionnaire. Back buttons allowed participants to review their answers before submitting the survey. Conventional technical methods, such as cookies, were used to prevent visitors from submitting their survey more than once.
The web-based survey was conducted using the Unipark tool [
], with the survey page seamlessly embedded into the search page via a modal window. After inviting panel participants, the survey remained accessible for 11 days and closed at the end of May 2021.Before filling out the survey, participants were provided with an introductory text outlining the anticipated time commitment, project objectives, details about anonymity, the option to terminate the survey at any point, data protection measures, and contact information for queries. To proceed, participants were required to confirm their understanding by checking a box and consenting to the specified use of their data (informed consent). No personal data were collected in the survey, and participants were instructed not to provide any identifying information in their open responses. Access to the data was restricted to members of the institutions participating in the study.
Survey Characteristics
On the basis of established scales (the German version of the self-assessment eHealth Literacy Scale [G-eHEALS] [
] and the System Usability Scale [ ] adapted by Quirmbach [ ] and Magin et al [ ]), we developed a 25-item questionnaire across five dimensions: (1) sociodemographic data; (2) internet use; (3) digital health literacy; (4) usability, acceptance, and innovative aspects of the search engine; and (5) search filters. The final questionnaire contained 24 single-choice questions and 1 open-ended question (Tables S1 and S2 in ). The items in the questionnaire were not randomized. No form of adaptive questioning was used. All questions, except for the open-ended question, were mandatory; nonresponse options were not provided. Thus, there was no additional consistency or completeness check before submission. The survey took approximately 5 minutes to complete.Statistical Analysis
Descriptive statistics were used to summarize the characteristics of the study participants, including age, gender, occupation, educational level, and internet use for health information, as well as self-perceived digital health literacy. Only completed questionnaires were analyzed. No statistical correction was applied to adjust the sample. To assess the acceptance and usability of the search engine, Likert-scale items related to these dimensions were combined to create scale values. The acceptance scale measured participants’ willingness to use the search engine again and to recommend it to others. The usability scale evaluated participants’ agreement with statements related to comprehensibility, clarity, the effectiveness of the search, confidentiality, advertising, the ease of learning, and using the search engine.
One reviewer analyzed the content of the open-ended question and assigned the comments to different categories inductively.
Multiple regression analyses were conducted to explore predictive factors influencing the acceptance of the search engine. The factors tested in the regression analyses included age, gender, occupation, educational level, the frequency of internet use for health information, and self-perceived digital health literacy.
Reporting
This study is reported based on the guidelines of the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) [
].User Behavior Tracking
Behavior tracking data were collected from all consenting users accessing the search engine between October 2020 and June 2021 using the web analytics software Matomo (Matomo.org) [
]. Matomo is a powerful open-source web analytics platform designed to help website owners and organizations gain insights into their online presence while respecting user privacy. Its primary purpose is to track website traffic by tracking website visitor behavior. What sets Matomo apart from services such as Google Analytics is its commitment to data privacy and General Data Protection Regulation (GDPR) compliance. Matomo allows users to maintain control over their data by hosting the data on their own servers, ensuring that sensitive information is not shared with third parties. Matomo is easy to integrate into web pages and can also be customized to track specific custom data. Ripp and Falke [ ] successfully used Matomo to track user behavior in their research project, where they analyzed search behavior for certain keywords on the online information system Grammis.We installed Matomo on our own server and integrated the tracking into our search engine’s front end. We enabled the tracking to record the filter settings and capture the search term beyond the standard visit data, such as how long users spent on the website, and which of the results they clicked on. Structurally, Matomo records several actions for each visit, containing the duration among other data (Figure S1 in
). However, for the last action of a visit, the duration is not recorded. We have filled the empty value with the median of all actions from the particular visit to obtain a more realistic value for the entire visit duration.Only data from consenting users was recorded and subsequently descriptively analyzed for the 9-month time period. Therefore, this sample differs from the sample of the questionnaire study but may also include users who completed the questionnaire. The data processing for the data analysis was realized with a Python script directly accessing Matomo’s database using specific SQL queries.
Ethical Considerations
The study was approved by the institutional review board at the University of Freiburg (Ethikkomission-Freiburg 559/17) as an extension to the GAP study [
], and a document on data protection was agreed upon. As an incentive, the questionnaire study participants could take part in a prize draw for 1 of 25 vouchers to a bookstore worth €20 (US $24.33) after completing the survey.Results
Questionnaire Study
Participants
During the time the survey was open (from October 2020 to the end of May 2021), there were 1577 unique site visitors to the search engine website, with 1426 (90.4%) visitors to the first survey page. Of these 1426 visitors, 1250 (87.7%) agreed to participate in the survey. Of these 1250 participants, 1123 (89.8%) completed the survey.
Of the 1123 survey respondents, 802 (71.4%) were from the panel described in the Recruitment subsection. The age distribution of these participants was approximately normal, with a little more than a quarter of the participants (215/802, 26.8%) falling into the age category of 50-59 years. The gender distribution was also fairly balanced between male and female, with a little more than half of the respondents (432/802, 53.9%) self-identifying as female. Regarding occupational background, the majority of the participants (726/802, 90.5%) did not work in a medical profession.
Educational levels were notably higher than in the general population due to recruitment through the scientific panel, with 69.3% (556/802) of the participants having completed upper secondary education (at least level 3 in the International Standard Classification of Education [ISCED]-2011) and 46.3% (372/802) holding a bachelor’s or master’s degree or higher (ISCED-2011 levels 5-8).
The study participants recruited from the web-based panel reported searching for health information online more frequently than the general population, with 65.4% (524/802) searching several times a month at the minimum and 27.7% (222/802) searching at least several times a week. Only 1.5% (12/802) of the participants indicated that they “never” searched the internet for health information.
Participants’ self-perceived digital health literacy was measured by the G-eHEALS items. Of the 802 participants, 291 (36.2%) felt confident in making health-related decisions based on internet information, while 461 (57.5%) felt capable of distinguishing reliable from questionable information online.
More information about the participants is available in Table S3 in
.Acceptance and Usability
Scale values were created by combining items relating to the acceptability and usability of the search engine (
).
The results show agreement regarding the acceptance and usability measures of the new search engine (
and ). In particular, more than half of the participants expressed their willingness to use the search engine again (465/802, 58%) and to recommend it to others (507/802, 63.2%). In addition, 79.3% (636/802) of the respondents agreed or strongly agreed that using the search engine was quick to learn, and 77.3% (620/802) to 98.6% (791/802) agreed at least partially with other statements related to usability aspects.Content | Scores, mean (SD) | |
Acceptance | ||
Item 1: potential reuse | 3.58 (1.02) | |
Item 2: recommendation to others | 3.68 (1.03) | |
Usability | ||
Item 1: quick to learn | 4.03 (0.86) | |
Item 2: efficient scannability of the SERPa | 3.63 (1.00) | |
Item 3: clarity of the SERP | 3.75 (1.00) | |
Item 4: functions not comprehensibleb | 2.37 (1.29) | |
Item 5: appreciation for the absence of advertising | 4.66 (0.66) | |
Item 6: functionality meets expectations | 3.45 (1.03) | |
Item 7: careful handling of personal data | 3.59 (0.88) | |
Item 8: no commercial bias | 3.55 (0.95) | |
Item 9: fast access to relevant information | 3.77 (0.99) | |
Item 10: innovative approach | 3.58 (1.03) | |
Innovative aspects of the search engine (anonymous searches) | 4.50 (0.76) |
aSERP: search engine results page.
bItem 4 (comprehensibility of functions) was a negated question.
Users particularly appreciated the absence of advertising on the platform, with 93.1% (747/802) agreeing or strongly agreeing that the absence of advertising was a significant benefit. The search engine’s user-friendly interface and the ability to adjust search results through the use of filters were also well received by participants. Furthermore, the unique features of the search engine, notably the anonymous searches without the creation of user profiles and independence from sponsors, received high importance ratings from 91.3% (733/802) of those surveyed.
More detailed results are available in Tables S4-S6 in
.Of the 802 participants, 561 (70%) to 622 (77.6%) indicated that they used the filter functions that could be enabled to change what was displayed on the SERP (
; ). Furthermore, 92.3% (518/561), 93.9% (553/589), 94.7% (567/599), and 96.5% (600/622) of those users who used the filters agreed at least partially that the filter functions were helpful in finding trustworthy, recent, user-friendly, and comprehensible results, respectively. The recency filter was particularly well received, with 46% (286/622) of the users strongly agreeing that it was helpful. More detailed results are available in Table S7 in .Content | Participants, n (%)a | Values, mean (SD) | |||
Helpfulness of filters for each filter | |||||
Item 1: trustworthiness | 599 (74.7) | 4.11 (0.92) | |||
Item 2: recency | 622 (77.6) | 4.25 (0.84) | |||
Item 3: comprehensibility | 589 (73.4) | 4.05 (0.93) | |||
Item 4: user-friendliness | 561 (70) | 3.91 (0.93) |
aAll other participants answered, “I did not use this function, I cannot evaluate this question.”
In the open-ended question, which aimed to supplement the data with qualitative results, participants were asked to share their comments and suggestions for improvements. Of the 802 participants, 263 (32.8%) replied to the question. Multiple aspects were mentioned. In an inductive approach, 10 main themes were identified (
). Many of the participants commented on the usability aspects of the website. Of the 236 comments, 76 (32.2%) included constructive criticism and specific suggestions for improving the website in the future. Many commenters made suggestions for improving the design because they found some aspects unclear or overwhelming. Others did not fully understand the methodology behind the evaluation of the selected providers or the reason why only specific providers, including commercial web pages, were shown on the results page and asked for more clarification. Some commenters were happy with the site as it stood and did not ask for any changes, while some conveyed disappointment with the results they discovered on the SERP.
Predictors of Acceptance
We conducted multiple regression analyses to examine the predictive factors influencing responses to the acceptance scale. Age, gender, occupation, educational level, frequency of internet use for health information, and self-perceived digital health literacy were tested as influencing factors. All factors were first tested individually and subsequently tested simultaneously due to the potential error introduced by multiple testing. Overall, only minor discrepancies were identified. Significance levels were set at P<.05.
presents more details.Predictive factors for acceptance | Individual testing of each predictor | Simultaneous testing of all predictorsa | |||||||||||||||
B (SE) | F test (df) | P value | R2 (%) | B (SE) | F test (df) | P value | ηp2 (%) | ||||||||||
Age | 0.07b (0.02) | 7.97 (1) | .005 | 0.99 | 0.07 (0.03) | 7.27 (1) | .007 | 0.91 | |||||||||
Gender | 0.04 (2) | .96 | 0.01 | 0.20 (2) | .82 | 0 | |||||||||||
Diverse | 0.01 (0.49) | 0.00 (1) | .99 | −0.07 (0.49) | 0.02 (1) | .89 | |||||||||||
Female | 0.02 (0.07) | 0.09 (1) | .77 | 0.04 (0.07) | 0.35 (1) | .55 | |||||||||||
Male | Reference | Reference | Reference | Reference | Reference | Reference | Reference | Reference | |||||||||
Occupation | 1.82 (2) | .16 | 0.45 | 1.14 (2) | .32 | 0.29 | |||||||||||
Medical doctor | 0.12 (0.12) | 0.92 (1) | .34 | 0.07 (0.12) | 0.28 (1) | .59 | |||||||||||
Other medical profession | −0.56 (0.35) | 2.62 (1) | .11 | −0.49 (0.35) | 1.93 (1) | .17 | |||||||||||
Not working in a medical profession | Reference | Reference | Reference | Reference | Reference | Reference | Reference | Reference | |||||||||
Highest level of educational attainment | 2.57 (4) | .04 | 1.27 | 1.55 (4) | .19 | 0.78 | |||||||||||
No formal secondary education (ISCEDc levels 0-1) or lower secondary education (Hauptschulabschluss, ISCED-2011 level 2) | 0.51 (0.19) | 6.81 (1) | .009 | 0.41 (0.20) | 4.22 (1) | .04 | |||||||||||
Lower secondary education (Realschulabschluss, ISCED-2011 level 2) | 0.33 (0.17) | 3.85 (1) | .05 | 0.26 (0.17) | 2.33 (1) | .13 | |||||||||||
Upper secondary education (Fachabitur oder Abitur, ISCED-2011 levels 3-4) | 0.16 (0.17) | 0.94 (1) | .33 | 0.13 (0.17) | 0.58 (1) | .45 | |||||||||||
Tertiary education: bachelor’s or master’s degree or equivalent (ISCED-2011 levels 6-7) | 0.21 (0.16) | 1.67 (1) | .20 | 0.17 (0.16) | 1.14 (1) | .29 | |||||||||||
Doctorate degree (ISCED-2011 level 8) | Reference | Reference | Reference | Reference | Reference | Reference | Reference | ||||||||||
Frequency of internet use for health information | 0.15 (0.04) | 17.16 (1) | < .001 | 2.1 | 0.13 (0.04) | 11.98 (1) | <.001 | 1.5 | |||||||||
Self-perceived digital health literacy: information search | 0.12 (0.04) | 9.69 (1) | .002 | 1.2 | 0.08 (0.05) | 3.03 (1) | .08 | 0.38 | |||||||||
Self-perceived digital health literacy: information assessment | 0.05 (0.04) | 1.79 (1) | .18 | 0.22 | −0.01 (0.05) | 0.04 (1) | .85 | 0 |
aVariance resolution of simultaneous testing: R2=5.02%.
bFactors with significant predictions (P<.05) are shown in italics.
cISCED: International Standard Classification of Education.
There were 2 significant results in predicting the acceptance of the search engine. First, the acceptance rating increased with age, rising by 0.07 scale points (B=0.07) for each additional 10-year age group. Second, the ratings for acceptance were higher among participants with more frequent internet use for health information.
The factors “highest level of educational attainment” and “self-perceived digital health literacy” only showed significant correlations with the acceptance ratings in the individual tests and slightly lower correlations in the simultaneous tests. Participants with lower levels of education showed stronger acceptance ratings than those with higher levels of education. For this predictor, the reference category was set as “doctorate degree” (ISCED-2011 level 8). Acceptance ratings among persons without any formal secondary education (ISCED-2011 levels 0-1) or with a lower secondary education (Hauptschulabschluss, ISCED-2011 level 2) were 0.41 scale points higher than among persons with a doctorate degree (B=0.41). However, if we examine the effect strength using partial eta–squared (ηp2), only small effects were shown overall for all factors (1%=small effect, 6%=medium effect). No significant results were found for the factors gender or occupation.
User Behavior Tracking
For a period of 9 months (from October 2020 to the end of June 2021), user behavior of consenting users was tracked using Matomo. The main aspects recorded were the search queries and the use of the filter functions. The data indicated that 1631 visitor sessions had taken place, with 3090 searches using 1984 different search terms across 1924 visits. The search terms had an average lexical token count of 1.74 (SD 1.14). Of the 1984 search terms, 1096 (55.24%) had a lexical token count of 1, while 888 (44.76%) had a lexical token count of >1. The maximum lexical token count was 13. We observed that 28.9% (893/3090) of the search queries were conducted through mobile devices such as smartphones and tablets. On average, visitors conducted 1.64 (SD 1.31) searches per visit, with an average duration of 137.45 (SD 278) seconds and a median duration of 49 (IQR 24-119) seconds. The top 3 search terms were “corona,” “rückenschmerzen” (back pain), and “husten” (cough; Table S1 in
). Each search term was queried 1.57 (SD 3.46) times on average; of the 1984 search terms, 1621 (81.7%) were entered just once. Users could personalize the order of the displayed search results, depending on the prioritization of the different filter categories (trustworthiness, recency, user-friendliness, and comprehensibility), all of which were initially set to “unimportant” by default. Within the search interactions, overall, 47,532 actions were performed, among which were 1358 (2.86%) outlink actions (ie, external domains clicked by website visitors) and 34,490 (72.56%) filter changes in total. The high number of filter changes was due to the fact that the filters were often switched back and forth in a search, which led to multiple occurrences of the same filter settings. This behavior was probably observable due to the fact that, given the limited corpus, the result set could be empty for certain search terms in combination with a narrow filter setup; therefore, it was necessary to change back the filter to obtain results. Excluding these repeated changes, of the 34,490 filter changes, 5534 (16.05%) were unique. In addition, after removing the initial filter setup where all categories were set to the default value, we ended up with 2784 nondefault unique filter changes used in 15.37% (475/3090) of the searches. Recency was used the most, followed by trustworthiness, user-friendliness, and comprehensibility ( ). Consistent with the results on the usefulness of the filter function from the questionnaire study, the user behavior data showed that the filters for recency and trustworthiness were adjusted most frequently.
Discussion
Principal Findings
In this study, we evaluated the independent, noncommercial tala-med search for German-language medical information, placing a primary focus on transparency, independence, and evidence-based information. Our questionnaire study and user behavior tracking provide insights into the acceptance and usability of this innovative platform.
The survey resulted in a mean acceptance score of 3.63 (SD 0.98) and a mean usability score of 3.76 (SD 0.61) on a scale of 1 to 5 (1 signifying strong disagreement and 5 signifying strong agreement with acceptance and usability statements). Given that our user interface is similar to that of popular search engines such as Google and Bing and is advertisement-free, it is unsurprising that it was well received. However, there is still a need for optimization to achieve a higher reuse potential and a greater willingness to recommend the platform. Within the usability measures, item 1, which reflected the learnability of the search, achieved a high mean value of 4.03 (SD 0.86;
). In terms of the effectiveness and clarity of our design (refer to items 2, 3, and 9 in ), we still see room for improvement but are satisfied with the results for the initial release. Our findings align with those of Petrie and Bevan [ ], who emphasized that usability, particularly learnability, is crucial for ensuring the platform’s effectiveness and user satisfaction. Similar to the study by Petrie and Bevan [ ], our study highlights the need to continually evaluate and refine digital systems to cater to user needs and improve overall usability. The survey also revealed that the filters were used by >70% of the 802 users. Among these users, the filters were highly accepted on average, which matches with the tracking statistics revealing the high use of the filter functionality. The most used filter was recency, followed by trustworthiness, matching the survey outcome, which asked about the helpfulness of the filters. This was not the case for the 2 remaining filters. Although the filter for user-friendliness was used more often in our tracking data than the one for comprehensibility, it was perceived as less helpful than comprehensibility. In the future, we might inspect these filter categories in more detail and potentially improve their estimation. The insights support our decisions, including basing the design of our search engine on the insights of Strecker et al [ ], which demonstrated how improvements in user interface design could lead to higher user satisfaction. The high acceptance of the filters for recency and trustworthiness in our study also echoes the findings of Strecker et al [ ], who highlighted that content navigation and relevance contribute to overall user satisfaction. Nevertheless, the insights encourage us to improve and possibly expand the filters and the design. The fact that our search engine does not share any data with partners or advertisers and does not record the visitor’s behavior to create profiles was rated as very relevant, with a mean score of 4.5 (SD 0.76). This result confirms the importance of our general endeavor driven by data privacy and noncommerciality as our fundamental motivations for creating the search engine. This finding is consistent with the work of Eysenbach and Köhler [ ], who found that while internet users claimed to prioritize credibility when assessing health information online, they often relied on superficial factors such as website design and ease of use, rather than more reliable indicators such as the “About us” sections or disclaimers. Our platform addresses these shortcomings by offering more explicit filters, such as trustworthiness, which help users find credible information without having to rely solely on superficial design elements. In our calculations on the predictors of acceptance, it can be concluded that while the effects may be small, the search engine was particularly well accepted among older users, users with a high frequency of internet use for health information, and those with lower levels of education. This finding suggests that the new search engine may effectively cater to segments of the population with lower health literacy and digital health literacy and greater requirements for navigational assistance, bridging the gap in accessing reliable medical information online. As discussed by Kritz et al [ ], general practitioners and medical specialists often face barriers in accessing high-quality medical content due to limited time and the overwhelming amount of information online. Our search engine’s focus on evidence-based information and user-friendly design helps mitigate these issues, making it a suitable tool for both health care professionals and the general public.This reinforces the necessity to consider navigational needs in health information platforms. Digital health literacy has improved in recent years, particularly among persons with low levels of education [
]. The ability to assess health information has also improved over time and over the course of the COVID-19 pandemic. Digital health literacy improved for younger but not for older people over the course of the pandemic [ ]. Even with some of these improvements, health literacy is still low within the German population and particularly among people with low levels of education. This observation aligns with the findings by Zhang [ ], who identified that source selection for health information is influenced by a wide range of factors, including accessibility, quality, usability, and personal relevance. Our platform’s appeal to older users and those with lower health literacy supports the conclusion drawn by Zhang [ ] that a personalized approach, considering different user-related factors, is crucial for effectively meeting the diverse needs of consumers when they access health information. The correlation between low educational levels, low social status, higher age, and lower health literacy rates has become even stronger [ ]. This emphasizes the need to address low health literacy to improve social inequities. The new search engine attempts to address this need by assessing health information online and by making high-quality information easily searchable.Many of the comments highlight the fact that at the time of the survey, participants used the first version of the search engine, which still had some technical and appearance-related weaknesses. Since the survey was conducted, a number of revisions have been made, some of which correspond to suggestions made in the comments, such as adding more information about the principles of the quality assessment and the functions of the websites and removing certain commercially funded information sources.
The platform’s emphasis on anonymous searches and independence from advertising and sponsors resonated especially well with the participants, underscoring the importance of developing alternatives to commercial search platforms. The usability of our platform, free from external advertising influence, aligns with the findings by Belinda et al [
], who evaluated both internal factors (such as page load time and performance) and external factors (such as ease of navigation and organization of information) as critical to the overall user satisfaction with websites. Our platform similarly emphasizes ease of use and efficient performance, which were key contributors to its positive reception. The general results align with previous findings on the importance of user-friendly interfaces [ - ] and the impact of advertising on users’ perceptions [ ].Limitations
Limitations of the Technology
Some participants expressed dissatisfaction with the displayed results. Several respondents found the design of the SERP and individual results confusing due to an overload of information in one place. Potential design changes have been suggested to improve clarity. Currently, some search results are duplicated on the results page. To enhance the appearance of the SERP, it is desirable to deduplicate the search index. This would mean displaying more relevant information and fewer distracting results. To address the fact that sometimes no results were displayed due to strict filter settings, we could display results with nonexact matches for the filter category score below the exact matches and visually indicate them.
Participants encountered difficulties in understanding how the evaluation process worked. Efforts should be made to make the evaluation process easier to understand when using the website. Some improvements have been made over time, such as providing brief explanations of the filter categories when hovering over the graphic showing the score for each domain.
In addition, further clarification is necessary regarding the domains chosen, which comprise both independent and commercial providers. Over time, certain websites were excluded from being displayed on the site because of the conflicts of interest of their publishers (such as the site “Zentrum der Gesundheit” and a site funded by a pharmaceutical company). Currently, the website no longer displays commercial providers.
Limitations of the Questionnaire Study
While the questionnaire study provides valuable insights into the acceptance and usability of the new search engine, several limitations should be acknowledged. First, the recruitment methodology, relying on web-based panel recruitment, unintentionally led to an overrepresentation of individuals with higher educational levels than the general population and potentially more frequent internet use due to the online recruitment process. Among our participants, 46.3% (371/802) had completed tertiary education compared to only 18.5% of the general German population with a university degree or equivalent [
]. Similarly, 65.4% (524/802) of our participants reported searching for health information only at least once a month, while a Bertelsmann Foundation 2018 study indicated that only approximately 50% of the general German population seek such information at least once a month [ ]. This potentially influences acceptance and usability ratings. Participants’ self-perceived digital health literacy, as measured by the G-eHEALS items, approximately aligned with that of the sample of the validation study of the German version of the questionnaire by Söllner et al [ ]. However, this information should be viewed with caution because Kim et al [ ] discovered a discrepancy between self-assessments and actual ability. Actual ability was not assessed in our study.Second, as some of the participants only interacted with the search engine and its features for a brief duration, this short testing time proved to be a limiting factor, hindering the formation of comprehensive first impressions. The responses to the open-ended question suggest that some participants tested the features of the search engine for only a few seconds. This was caused by the pop-up window linking to the questionnaire opening after interacting with the website for 10 to 20 seconds. Such a brief interaction period was likely insufficient for users to thoroughly explore and understand the functionalities and benefits of the search engine. Consequently, the feedback provided by users on the acceptance and usability of the platform may not accurately reflect its true potential. The limited duration of testing may lead to superficial evaluations, whereby participants may base their judgments on initial impressions rather than informed use. This can result in an inaccurate assessment and therefore impact the results presented for the acceptance and usability measures. More extended testing periods could offer a more accurate assessment of the platform’s acceptance and usability over time. Furthermore, the implementation of controls for the length of the testing period could provide more reliable data, thereby ensuring that all participants have sufficient time to engage with the platform before completing the questionnaire.
Limitations of the User Behavior Tracking
As Matomo does not record the last action, we decided to substitute this missing value with the median of all actions from the particular visit. However, a more optimal solution would be to record of the last action, which could be triggered when closing the browser, but Matomo does not support this feature.
Unfortunately, due to data protection regulations, we were unable to link the search behavior with the survey data. It would have been interesting to see to what extent the search behavior and the questionnaire would have provided more information.
Future Work
Further Evaluation of the New Search Engine
As these results are based on the first impressions of a group of users, data on long-term use and satisfaction have yet to be compiled. Future work should therefore address the need to evaluate long-term acceptance and usability measures of the site using a longitudinal study design. This could provide valuable insight into whether users actually use the search engine when they have an acute need to seek out medical information and whether they return to the site over time.
Future research should also consider comparing the search engine with other existing and established platforms to gain a deeper understanding of its unique advantages. Such investigations might involve participants executing identical search queries on both the novel search engine and other platforms and sharing their impressions of each experience, similar to the study by Strecker et al [
]. Particular attention should be given to participants’ abilities to assess the quality and reliability of the information found. Furthermore, a comparison of the specific results retrieved from both search engines could be undertaken, examining both the congruence and discrepancies in the results.Further Improvements to the Search Engine
Ongoing revisions and updates to the search engine, including integration on partner websites, hold promise in expanding its reach and use, ensuring that it remains relevant and effective in addressing users’ health information needs. Since the data collection for this study was completed, there have been several revisions to the website.
As the qualitative survey showed, some users found the SERP complicated. To address this issue, a potential solution could be a clearly linked help page with a video explanation. Another solution could be to provide an animated assistant that guides users through the page.
One major disadvantage of the search engine in its current state stems from the fact that the evaluation of the domains—more than 50 of them—according to the quality criteria was performed in 2020, and there has not been any update to the evaluations thus far. The domains’ quality may have improved or declined in the time since the evaluation. A continual process with constant reevaluations of the domains is necessary to ensure an up-to-date rating of the domains accessible through the search engine. As we currently do not have the capacity to perform another evaluation or to establish a continual evaluation process for the domains, we have temporarily disabled the display of the filter category scores and the option to adjust the results based on these scores because the scores do not reflect the actual quality of the domains in their current state.
Thus, in future, inspired by Zowalla et al [
], we intend to evaluate various criteria automatically. In the study by Zowalla et al [ ], the readability of the language was analyzed supported by a support vector machine. We plan to use more sophisticated approaches that also model contextual context. Therefore, we plan to create classifiers based on Bidirectional Encoder Representations from Transformers (BERT) [ ]. We have in mind the domain-specific models GerMedBert [ ] and BioGottBERT [ ], the latter being a model of the German monolingual Robustly Optimized BERT Pretraining Approach (RoBERTa) [ ] model GottBERT [ ] specialized on medical language. A further improvement might be the enrichment of the texts with MeSH terms, similar to PubMed. Such approaches would enable us to process and rate documents individually, rather than evaluating domains using random samples.From a technological standpoint, there are still notable areas for enhancement of the middleware, which could be implemented in the future. As App Search is proprietary and monolithic, to make it customizable and more extensible, we are currently exploring potential improvements. In particular, we are considering alternative technologies, as used by Scheible et al [
]. This would enable us to add features based on modern technologies. Specifically, we are contemplating a different approach to computing synonyms. Instead of using discrete structures, we plan to train and use a FastText model [ ].Conclusions
The development of the independent, noncommercial tala-med search marks a step toward improving access to reliable and evidence-based health information online. By prioritizing transparency, independence, and high-quality content, the platform has the potential to bridge the gap in health information accessibility, especially for older and less-educated individuals. This study achieved its objective of evaluating the acceptance and usability of the platform, using user questionnaires and behavior tracking, and the results have been encouraging. The search engine was well received, with users appreciating its advertising-free design and filtering system, which addresses the navigational needs of individuals with lower health and digital health literacy.
User feedback also highlighted areas for improvement, particularly in the clarity of the design and filter functionality. Nevertheless, the platform’s emphasis on anonymous searches and independence from advertising resonated strongly with participants, reinforcing the importance of alternatives to commercial search engines. These findings are based on initial impressions, with long-term use and satisfaction data still to be collected. Future research should compare the search engine with existing platforms to further explore its unique advantages, while ongoing updates and potential integration on partner websites will enhance its reach and relevance.
In conclusion, the tala-med search demonstrates a promising step toward enhancing health literacy and empowering individuals to make well-informed health decisions. By addressing users’ navigational needs and promoting equitable access to high-quality, evidence-based medical information, the search engine has the potential to positively impact health literacy, reduce health disparities, and promote patient empowerment in the digital age.
Acknowledgments
The study was funded by the Innovationsausschuss of the Gemeinsamer Bundesausschuss (01NVF17010) as part of the GAP (Gut informierte Kommunikation zwischen Arzt und Patient, meaning “Well-informed communication between physician and patient” in English) project. Furthermore, this work was supported by the German Ministry for Education and Research (01ZZ1801B, 01ZZ1804A, 01KX2121, and 01ZZ2304A). Finally, we acknowledge support by the Open Access Publication Fund of the University of Freiburg.
Klaus Koch of the Institute for Quality and Efficiency in Health Care supported the domain selection for the search index. Julia Serong and Jakob Henke of the TU Dortmund University contributed to the evaluation concept.
Authors' Contributions
RS, NK, and CW planned and carried out the implementation of the search engine supported by FT under the supervision of MB and SV-R. RS carried out the mathematical framework for the search engine and filter estimation with the help of LS. KW and AS-L worked out the criteria with which the domains were evaluated. MS created the procedure for the weighting of the filter categories and evaluated the pages. MS developed the questionnaire under the supervision of AM. CS prepared the data analyses for the predictors of acceptance under the supervision of EF-G. CS and RS prepared the data analyses of the web tracking. RS and LS wrote the manuscript with the help of all authors and under the supervision of AM. The manuscript has been read and approved by all named authors.
Conflicts of Interest
None declared.
Multimedia Appendix 1
Search engine development: search equation and filter estimation.
PDF File (Adobe PDF File), 830 KBMultimedia Appendix 3
Search engine development: criteria evaluated.
XLSX File (Microsoft Excel File), 19 KBMultimedia Appendix 4
Questionnaire study: questionnaire and detailed results.
PDF File (Adobe PDF File), 313 KBReferences
- Baumann E, Czerwinski F, Rosset M, Seelig M, Suhr R. [How do people in Germany seek health information? Insights from the first wave of HINTS Germany]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. Sep 2020;63(9):1151-1160. [CrossRef] [Medline]
- Haschke C, Westrick MG, Schwenk U. SPOTLIGHT gesundheit: gesundheitsinfos. Bertelsmann Stiftung. URL: https://www.bertelsmann-stiftung.de/de/publikationen/publikation/did/spotlight-gesundheit-gesundheitsinfos [accessed 2022-05-16]
- Marstedt PG. Das internet: auch ihr ratgeber für gesundheitsfragen? Bertelsmann Stiftung. Jan 26, 2018. URL: https://www.bertelsmann-stiftung.de/de/publikationen/publikation/did/das-internet-auch-ihr-ratgeber-fuer-gesundheitsfragen [accessed 2022-05-18]
- Individuals - internet activities. Eurostat. URL: https://ec.europa.eu/eurostat/databrowser/view/isoc_ci_ac_i/default/table?lang=en [accessed 2024-11-26]
- Zhang Y, Sun Y, Xie B. Quality of health information for consumers on the web: a systematic review of indicators, criteria, tools, and evaluation results. J Assoc Inf Sci Technol. Oct 2015;66(10):2071-2084. [CrossRef]
- Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA. 2002;287(20):2691-2700. [CrossRef] [Medline]
- Hambrock U. Die suche nach gesundheitsinformationen. Bertelsmann Stiftung. Jan 26, 2018. URL: https://www.bertelsmann-stiftung.de/de/publikationen/publikation/did/die-suche-nach-gesundheitsinformationen [accessed 2022-05-18]
- Baumann E, Czerwinski F. Erst mal doktor Google fragen? Nutzung neuer medien zur information und zum austausch über gesundheitsthemen. In: Böcken J, Braun B, Meierjürgen R, editors. Gesundheitsmonitor 2015: Bürgerorientierung im Gesundheitswesen. Gütersloh, Germany. Bertelsmann Stiftung; 2015:57-79.
- Kammerer Y, Gerjets P. Effects of search interface and internet-specific epistemic beliefs on source evaluations during web search for medical information: an eye-tracking study. Behav Inf Technol. Jan 2012;31(1):83-97. [CrossRef]
- Haas A, Unkel J. Ranking versus reputation: perception and effects of search result credibility. Behav Inf Technol. Sep 26, 2017;36(12):1285-1298. [CrossRef]
- Cai HC, King LE, Dwyer JT. Using the Google™ search engine for health information: is there a problem? Case study: supplements for cancer. Curr Dev Nutr. Feb 2021;5(2):nzab002. [FREE Full text] [CrossRef] [Medline]
- Schaeffer D, Vogt D, Berens EM, Hurrelmann K. Gesundheitskompetenz der bevölkerung in Deutschland: ergebnisbericht. Universität Bielefeld, Fakultät für Gesundheitswissenschaften. 2017. URL: https://pub.uni-bielefeld.de/record/2908845#fileDetails [accessed 2024-12-20]
- Schaeffer D, Berens EM, Vogt D. Health literacy in the German population. Dtsch Arztebl Int. Jan 27, 2017;114(4):53-60. [FREE Full text] [CrossRef] [Medline]
- Schaeffer D, Berens EM, Vogt D, Gille S, Griese L, Klinger J, et al. Health literacy in Germany - findings of a representative follow-up survey. Dtsch Arztebl Int. Oct 29, 2021;118(43):723-728. [FREE Full text] [CrossRef] [Medline]
- Hurrelmann K, Klinger J, Schaeffer D. Gesundheitskompetenz der Bevölkerung in Deutschland: Vergleich der Erhebungen 2014 und 2020. Universität Bielefeld, Interdisziplinäres Zentrum für Gesundheitskompetenzforschung. 2020. URL: https://pub.uni-bielefeld.de/record/2950303 [accessed 2024-12-20]
- Jordan S, Hoebel J. [Health literacy of adults in Germany: findings from the German Health Update (GEDA) study]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. Sep 31, 2015;58(9):942-950. [CrossRef] [Medline]
- Lee K, Hoti K, Hughes JD, Emmerton LM. Consumer use of "Dr Google": a survey on health information-seeking behaviors and navigational needs. J Med Internet Res. Dec 29, 2015;17(12):e288. [FREE Full text] [CrossRef] [Medline]
- Wollmann K, van der Keylen P, Tomandl J, Meerpohl JJ, Sofroniou M, Maun A, et al. The information needs of internet users and their requirements for online health information-a scoping review of qualitative and quantitative studies. Patient Educ Couns. Aug 2021;104(8):1904-1932. [CrossRef] [Medline]
- Röttele N, Schlett C, Körner M, Farin-Glattacker E, Schöpf-Lazzarino AC, Voigt-Radloff S, et al. Variance components of ratings of physician-patient communication: a generalizability theory analysis. PLoS One. 2021;16(6):e0252968. [FREE Full text] [CrossRef] [Medline]
- Radloff S, Wollmann K, Voigt-Radloff S, Farin-Glattacker E, Schlett C. Online-interventionen zur förderung der informiertheit. Impu!se für Gesundheitsförderung. 2021;113(4):18.
- van der Keylen P, Tomandl J, Wollmann K, Möhler R, Sofroniou M, Maun A, et al. The online health information needs of family physicians: systematic review of qualitative and quantitative studies. J Med Internet Res. Dec 30, 2020;22(12):e18816. [FREE Full text] [CrossRef] [Medline]
- Voigt-Radloff S, Schöpf AC, Boeker M, Frank L, Farin E, Kaier K, et al. Well informed physician-patient communication in consultations on back pain - study protocol of the cluster randomized GAP trial. BMC Fam Pract. Feb 25, 2019;20(1):33. [FREE Full text] [CrossRef] [Medline]
- van der Keylen P, Maun A, Hoyer S, Roos M, Beier M, Voigt-Radloff S, et al. "Mind the gap" — verzahnung von wissenschaft und hausärztlicher Praxis. Z Allg Med. 2019;95:179-184. [FREE Full text] [CrossRef]
- Braun C, Voigt-Radloff S. [Treatment of non-specific low back pain: evidence map of systematic reviews from 2015 to 2019]. Z Evid Fortbild Qual Gesundhwes. Dec 2019;149:12-21. [CrossRef] [Medline]
- Lang B, Kunzweiler K, Voigt-Radloff S. Gut informierte kommunikation zwischen arzt und patient im digitalen zeitalter – Das GAP-Projekt. In: Knieps F, Pfaff H, editors. Digitale Arbeit - Digitale Gesundheit. Berlin, Germany. Medizinisch Wissenschaftliche Verlagsgesellschaft; 2017.
- Schlett C, Röttele N, van der Keylen P, Schöpf-Lazzarino AC, Klimmek M, Körner M, et al. The acceptance, usability, and utility of a web portal for back pain as recommended by primary care physicians: qualitative interview study with patients. JMIR Form Res. Dec 29, 2022;6(12):e38748. [FREE Full text] [CrossRef] [Medline]
- Nater T, Boyer C. Debate about evaluation and monitoring of sites carrying the HON-Logo. J Med Internet Res. Jun 30, 2000;2(2):E13. [FREE Full text] [CrossRef] [Medline]
- Scheible R, Kadioglu D, Ehl S, Blum M, Boeker M, Folz M, et al. Enabling external inquiries to an existing patient registry by using the open source registry system for rare diseases: demonstration of the system using the European Society for Immunodeficiencies Registry. JMIR Med Inform. Oct 07, 2020;8(10):e17420. [FREE Full text] [CrossRef] [Medline]
- Murphy SN, Weber G, Mendis M, Gainer V, Chueh HC, Churchill S, et al. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Inform Assoc. 2010;17(2):124-130. [FREE Full text] [CrossRef] [Medline]
- Scheible R, Thomczyk F, Blum M, Rautenberg M, Prunotto A, Yazijy S, et al. Integrating row level security in i2b2: segregation of medical records into data marts without data replication and synchronization. JAMIA Open. Oct 2023;6(3):ooad068. [FREE Full text] [CrossRef] [Medline]
- Scheible R, Strecker P, Yazijy S, Thomczyk F, Talpa R, Puhl A, et al. A multilingual browser platform for medical subject headings. Stud Health Technol Inform. Jan 14, 2022;289:384-387. [CrossRef] [Medline]
- Strecker P, Boeker M, Buechner S, Scheible R. Usability evaluation of a modern multilingual MeSH browser. Stud Health Technol Inform. Jun 29, 2022;295:37-40. [CrossRef] [Medline]
- Zowalla R, Wetter T, Pfeifer D. Crawling the German Health Web: exploratory study and graph analysis. J Med Internet Res. Jul 24, 2020;22(7):e17853. [FREE Full text] [CrossRef] [Medline]
- Zowalla R, Pfeifer D, Wetter T. Readability and topics of the German Health Web: exploratory study and text analysis. PLoS One. Feb 10, 2023;18(2):e0281582. [FREE Full text] [CrossRef] [Medline]
- Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. Sep 1989;13(3):319-340. [CrossRef]
- Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. MIS Q. 2003;27(3):425-478. [CrossRef]
- Nielsen J. Usability Engineering. Burlington, MA. Morgan Kaufmann Publishers; 1994.
- Baumel A, Yom-Tov E. Predicting user adherence to behavioral eHealth interventions in the real world: examining which aspects of intervention design matter most. Transl Behav Med. Sep 08, 2018;8(5):793-798. [CrossRef] [Medline]
- Mateus DA, Silva CA, de Oliveira AF, Costa H, Freire AP. A systematic mapping of accessibility problems encountered on websites and mobile apps: a comparison between automated tests, manual inspections and user evaluations. J Interact Syst. Nov 08, 2021;12(1):145-171. [CrossRef]
- Petrie H, Bevan N. The evaluation of accessibility, usability and user experience. University of York. 2009. URL: https://pure.york.ac.uk/portal/en/publications/the-evaluation-of-accessibility-usability-and-user-experience [accessed 2024-12-20]
- Belinda BI, Kayode AB, Solomon N, Bethy TA. Analysis of internal and external website usability factors. Commun Appl Electron. Apr 2021;7(35):9-18. [CrossRef]
- Kritz M, Gschwandtner M, Stefanov V, Hanbury A, Samwald M. Utilization and perceived problems of online medical resources and search tools among different groups of European physicians. J Med Internet Res. Jun 26, 2013;15(6):e122. [FREE Full text] [CrossRef] [Medline]
- Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ. Mar 09, 2002;324(7337):573-577. [FREE Full text] [CrossRef] [Medline]
- Zhang Y. Beyond quality and accessibility: source selection in consumer health information searching. J Assoc Inf Sci Technol. Jan 07, 2014;65(5):911-927. [CrossRef]
- Saad M, Zia A, Raza M, Kundi M, Haleem M. A comprehensive analysis of healthcare websites usability features, testing techniques and issues. IEEE Access. 2022;10:97701-97718. [CrossRef]
- Fess - enterprise search server. Fess. URL: https://fess.codelibs.org/index.html [accessed 2021-04-07]
- mozilla / readability. GitHub. URL: https://github.com/mozilla/readability [accessed 2021-04-07]
- Tetrault D. Elastic app search: now available as a self-managed download. Elastic. Jun 25, 2019. URL: https://www.elastic.co/blog/elastic-app-search-7-2-0-released [accessed 2021-04-07]
- Elasticsearch homepage. Elastic. URL: https://www.elastic.co/de/enterprise-search [accessed 2023-07-17]
- Hashemian Nik D, Kasáč Z, Goda Z, Semlitsch A, Schulz S. Building an experimental German user interface terminology linked to SNOMED CT. Stud Health Technol Inform. Aug 21, 2019;264:153-157. [CrossRef] [Medline]
- tala-med Suche. tala-med Suche Evidenzbasierte Gesundheitsinformationen. URL: https://suche.tala-med.info/ [accessed 2024-12-23]
- WisoPanel homepage. WisoPanel. URL: https://www.wisopanel.net/ [accessed 2023-12-21]
- Umfragesoftware. Unipark. URL: https://www.unipark.com/umfragesoftware/ [accessed 2024-12-23]
- Soellner R, Huber S, Reder M. The concept of eHealth literacy and its measurement. J Media Psychol. Jan 2014;26(1):29-38. [CrossRef]
- Brooke J. SUS: a 'quick and dirty' usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B, editors. Usability Evaluation in Industry. Boca Raton, FL. CRC Press; 1996.
- Quirmbach SM. User experience und usability in suchmaschinen. In: Suchmaschinen: User Experience, Usability und Nutzerzentrierte Website-Gestaltung. Berlin, Heidelberg. Springer; 2012:57-65.
- Magin M, Steiner M, Heinbach D, Bosold S, Pieper A, Felka EM, et al. Suchmaschinen auf dem Prüfstand – eine Vergleichende Inhaltsanalyse der Qualität von Trefferlisten. M&K. 2015;63(4):495-516. [CrossRef]
- Eysenbach G. Improving the quality of web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. Sep 29, 2004;6(3):e34. [FREE Full text] [CrossRef] [Medline]
- Matomo homepage. Matomo. URL: https://matomo.org/ [accessed 2023-10-26]
- Ripp S, Falke S. Analyzing user behavior with Matomo in the online information system Grammis. In: Proceedings of the XVIII EURALEX International Congress Lexicography in Global Contexts. 2018. Presented at: XVIII EURALEX; July 17-21, 2018; Ljubljana, Slovenia. URL: https://ids-pub.bsz-bw.de/frontdoor/index/index/docId/7706
- Shamat NA, Sulaiman S, Sinpang JS. A systematic literature review on user interface design for web applications. J Telecommun Electron Comput Eng. 2017;9(3-4):57-61.
- Mazumder FK, Das UK. Usability guidelines for usable user interface. Int J Res Eng Technol. Sep 2014;03(09):79-82. [CrossRef]
- Nichols DM, McKay D, Twidale MB. Participatory usability: supporting proactive users. In: Proceedings of the 4th Annual Conference of the ACM Special Interest Group on Computer-Human Interaction. 2003. Presented at: CHINZ '03; July 3-4, 2003; Dunedin, New Zealand. [CrossRef]
- Davranova MD. Internet advertising: perceptions of the users. Int J Mark Bus Commun. 2019;8(2 & 3):25-36. [FREE Full text]
- Bildung in Deutschland (Bildungsstand). Destatis Statistisches Bundesamt. URL: https://www.destatis.de/DE/Themen/Gesellschaft-Umwelt/Bildung-Forschung-Kultur/Bildungsstand/_inhalt.html [accessed 2023-12-08]
- Kim S, Park K, Jo HS. Gap between perceived eHealth literacy and ability to use online cancer-related information. J Korean Med Sci. Jun 22, 2020;35(24):e187. [FREE Full text] [CrossRef] [Medline]
- Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019. Presented at: NAACL-HLT 2019; June 2-7, 2019; Minneapolis, MN. [CrossRef]
- Bressem KK, Papaioannou JM, Grundmann P, Borchert F, Adams LC, Liu L, et al. medBERT.de: a comprehensive German BERT model for the medical domain. Expert Syst Appl. Mar 2024;237:121598. [CrossRef]
- Lentzen M, Madan S, Lage-Rupprecht V, Kühnel L, Fluck J, Jacobs M, et al. Critical assessment of transformer-based AI models for German clinical notes. JAMIA Open. Dec 2022;5(4):ooac087. [FREE Full text] [CrossRef] [Medline]
- Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, et al. RoBERTa: a robustly optimized BERT pretraining approach. arXiv. Preprint posted online on July 26, 2019. [FREE Full text]
- Scheible R, Frei J, Thomczyk F, He H, Tippmann P, Knaus J, et al. GottBERT: a pure German language model. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. Presented at: EMNLP; November 12, 2024; Miami, FL. URL: https://aclanthology.org/2024.emnlp-main.1183 [CrossRef]
- Bojanowski P, Grave E, Joulin A, Mikolov T. Enriching word vectors with subword information. Transact Assoc Comput Linguist. Jun 1, 2017;5:135-146. [CrossRef]
Abbreviations
afgis: Aktionsforum Gesundheitsinformationssystem eV (Action Forum Health Information System eV) |
BERT: Bidirectional Encoder Representations from Transformers |
CHERRIES: Checklist for Reporting Results of Internet E-Surveys |
GAP: Gut informierte Kommunikation zwischen Arzt und Patient (Well-informed communication between physician and patient) |
GDPR: General Data Protection Regulation |
G-eHEALS: eHealth Literacy Scale, German version |
i2b2: Integrating Biology and the Bedside |
ISCED: International Standard Classification of Education |
MeSH: Medical Subject Headings |
RoBERTa: Robustly Optimized Bidirectional Encoder Representations from Transformers Pretraining Approach |
SERP: search engine results page |
SNOMED-CT: Systematized Nomenclature of Medicine–Clinical Terms |
Edited by E Borycki, Y Senathirajah; submitted 31.01.24; peer-reviewed by F Jahn, S Markham, L Hasan; comments to author 02.06.24; revised version received 12.07.24; accepted 23.10.24; published 23.01.25.
Copyright©Lisa Specht, Raphael Scheible, Martin Boeker, Erik Farin-Glattacker, Nikolas Kampel, Marina Schmölz, Andrea Schöpf-Lazzarino, Stefan Schulz, Christian Schlett, Fabian Thomczyk, Sebastian Voigt-Radloff, Constanze Wegner, Katharina Wollmann, Andy Maun. Originally published in JMIR Human Factors (https://humanfactors.jmir.org), 23.01.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on https://humanfactors.jmir.org, as well as this copyright and license information must be included.