During the last 8 months of 2020, the average number of visits to the website per month was just above 2500, and the number of unique visitors was 870. Most visitors were from Sweden, with some users from other countries in northern Europe, especially Scandinavia. Visitors from Sweden’s 3 largest cities (Stockholm, Gothenburg, and Malmoe) accounted for more than 50% of all visits. Most visits took place on Mondays to Fridays, but there were 5-10 visits per day on weekends.

From the survey with 292 answers, it became clear that the visitors were mainly physicians (49%, 144/292) and nurses (36%, 104/292), and, to some extent, pharmacists (7%, 21/292). The rest were from the pharmaceutical industry and others. Almost 80% (224/292) stated that they searched for regimens and 90% (264/292) stated that they found what they were looking for. Most of the users were satisfied with their visit. The less satisfied visitors (8%, 22/292) lacked the regimens or diagnostic areas that they were looking for. There were 32 comments, 18 of which asked for more regimens and information sheets for patients. The rest of the comments concerned the design of the survey or were positive reviews.

The heuristic evaluation was performed in accordance with the Nielsen and Molich’s [20] framework, which implies that a product or system is evaluated in a structured way following ten predefined usability heuristics: (1) visibility of system status; (2) match between systems and the real world; (3) user control and freedom; (4) consistency and standards; (5) error prevention; (6) recognition rather than recall; (7) flexibility and efficiency of use; (8) aesthetic and minimalist design; (9) recognize, diagnose, and recover from errors; and (10) help and documentation. The results show that the e-library follows many of the existing design principles; therefore. it could be concluded to be usable in its current form. Nevertheless, some improvement suggestions were identified, such as moving patient information sheets to a separate tab in the menu, improving the search function by providing suggestions, and improving the start page by moving newsletters to a separate tab.

A total of 86 emails with spontaneous user feedback were received from nurses (50%, 43/86), pharmacists (20%, 17/86), physicians (17%, 15/86), the pharmaceutical industry (8%, 7/86), and others (5%, 4/86). Of the emails, 78% (67/86) concerned questions such as “When will regimens arrive in a certain diagnostic group?” “How long should the drug temozolomide be given before starting radiation therapy?” In 19 of 86 (22%) emails, the sender had discovered errors in the e-library, which mainly concern the regimens, such as patient information sheets where there is a lack of a certain side effect or that the incorrect “treatment intention” had been chosen. All emails were answered, and in cases where there was incorrect or unclear information in the e-library’s documents, it was rectified.

Interview results showed that most hospitals used a CPOE system for their prescription and administration (3 systems exist). One hospital has no system; instead, they use the administration schedule from the e-library. Those with a CPOE system use the library in various ways, including importing the XML files, reading the information in the library and then transferring the information to their CPOE system, or using the library as a reference to check their information. The newsletter is published regularly on the e-library’s website, with information on new regimens or basic facts, and detailed information about changes in the regimens was appreciated by most of the interviewees, but not all of them had noticed them. The basic facts are used in various ways depending on profession; the pharmacists focus on preparation, shelf-life, and references to stability studies, the nurses focus on ADRs, and the physicians focus on new drugs. The nurses, especially, were interested in the patient information sheets that are available for most of the regimens. Most hospitals already had their patient information sheets but will switch to the ones in the e-library. The available supporting documents for health care professionals, and especially “Management of side effects associated with immunotherapy with checkpoint inhibitors,” are indicated by the nurses and physicians as very useful. Among the expectations for further development was a harmonization of regimens used in different diagnostic areas; for example, the same amount of infusion fluid and infusion times for the same drug.

The results from the user evaluation indicate that the e-library can be concluded to be used in the intended manner, and the users do not have any problems interacting with the knowledge source. From the web surveys and interviews, it becomes clear that it is the content that the users focus on. The usability of the website is not addressed by the users, which may be interpreted as subjective satisfaction in the interaction with the system. The heuristic evaluation showed that there were minor usability issues that should be addressed to improve the overall usability of the website. The web survey showed that the users substantially are satisfied with their visit. Most of the users found what they were looking for, and their main feedback was a desire for more regimens, more diagnostic areas to be covered, and more patient information sheets, which are continuously added and updated. The expressed needs from the users are useful to understand which areas and regimens should be prioritized in the development work. The number of visits indicates that the resource is used extensively, and the geographical span shows that the e-library has emerged as the national resource it is intended to be. Users from almost all Swedish regions exist, although the 3 largest cities stand for the main usage. However, the introduction of the library has proceeded fairly quickly, and it is believed that it is only a matter of time before all regions have adopted the use of the e-library.

The spontaneous user feedback shows clearly that the e-library is used. Contact with the users is vital, and websites generally facilitate a rapid and comprehensive means of knowledge dissemination [7]. Emails from users reporting incorrectness are gratefully received, and they show that errors can slip through despite several checks. The interviews revealed various ways to use the information in the regimens. Some use the XML files to import the regimen to their CPOE system, which is a way to ensure that the transmitted information is accurate. Those transferring the information explained that they had to adjust information in their CPOE system; therefore, is it easier to copy one of their regimens and implement the necessary changes to match the new regimen. Some clinics do not have any CPOE system yet, still relying on handwritten orders and documentation. To transfer information manually is always a risk for introducing errors. Users have different needs depending on whether they are doctors, nurses, or pharmacists and depending on whether they work at a university hospital or a smaller hospital. The e-library can support these different needs from the users.

Automated downloading of the regimens from the e-library to the CPOE systems could improve the process, ensuring updated information without manual handling. However, the CPOE systems do not support this yet, and the process and approval of the updated information at each local clinic must be adapted to the change.

This study uses a mix of methods to address the usage and usability of a web-based resource for chemotherapy regimens. One can always discuss which mix of methods is optimal, and the combination used here could certainly be extended to include additional ones. Cognitive walkthrough is one such method, which could have been adopted—it is an explicit and detailed procedure to simulate a uses’ problem-solving process at each step through the dialogue with the system having potentially provided additional input in evaluating its usability [21].

The qualitative interviews revealed that nurses have started to replace their old patient information sheets with the ones in the e-library. In a follow-up study, it would be relevant to include patients in the evaluation process.

The study is performed in a geographically delimited system (Sweden), but the article contributes with working methods for how user-driven development contributes to a more standardized working method, which could also provide safer care. That knowledge is useful beyond the Swedish healthcare context. Finally, a longer-term evaluation is required to gain good insight into how the system is used and to evaluate how it contributes to increased safety in cancer care.

The comprehensive user evaluation conducted is an important part of continuing the user-centered process that started already during the development of the e-library. Multiple evaluation methods complement one another by providing input from several perspectives (ie, expert or user, subjective or objective) that may be triangulated and hence identify critical design aspects and user needs [19]. The combination of methods applied in the evaluation presented in this paper included both objective usage statistics, expert methods for usability, and the users’ subjective feedback through a web survey and qualitative interviews. This ensures that the design and content of the e-library comply with the users’ needs and works as feedback for continuous development and learning [22]. Thus, the evaluation is a vital part assuring that the e-library act as a safety barrier, is well designed and that general design flaw is avoided, a design flaw that otherwise could create new risks in the chemotherapy process. The evaluation will contribute to a deeper understanding of users’ judgements about the library content and help to develop strategies for increasing the national usage of the library. With a broad national usage, the e-library can become a source for organizational and national learning and a source for continuous improvement of cancer care in Sweden.