Background: Accurate information about chemotherapy drugs and regimens is needed to reduce chemotherapy errors. A national e-library, as a common knowledge source with standardized chemotherapy nomenclature and content, was developed. Since the information in the library is both complex and extensive, it is central that the users can use the resource as intended.
Objective: The aim of this study was to evaluate the usage and usability of an extensive e-library for chemotherapy regimens developed to reduce medication errors, support the health care staff in their work, and increase patient safety.
Methods: To obtain a comprehensive evaluation, a mixed methods study was performed for a broad view of the usage, including a compilation of subjective views of the users (web survey, spontaneous user feedback, and qualitative interviews), analysis of statistics from the website, and an expert evaluation of the usability of the webpage.
Results: Statistics from the website show an average of just over 2500 visits and 870 unique visitors per month. Most visits took place Mondays to Fridays, but there were 5-10 visits per day on weekends. The web survey, with 292 answers, shows that the visitors were mainly physicians and nurses. Almost 80% (224/292) of respondents searched for regimens and 90% (264/292) found what they were looking for and were satisfied with their visit. The expert evaluation shows that the e-library follows many existing design principles, thus providing some useful improvement suggestions. A total of 86 emails were received in 2020 with user feedback, most of which were from nurses. The main part (78%, 67/86) contained a question, and the rest had discovered errors mainly in some regimen. The interviews reveal that most hospitals use a computerized physician order entry system, and they use the e-library in various ways, import XML files, transfer information, or use it as a reference. One hospital without a system uses the administration schedules from the library.
Conclusions: The user evaluation indicates that the e-library is used in the intended manner and that the users can interact without problems. Users have different needs depending on their profession and their workplace, and these can be supported. The combination of methods applied ensures that the design and content comply with the users’ needs and serves as feedback for continuous design and learning. With a broad national usage, the e-library can become a source for organizational and national learning and a source for continuous improvement of cancer care in Sweden.
Chemotherapy treatments are highly complex, and errors may cause serious harm among patients with cancer, which is a particularly sensitive group owing to impaired tolerance [, ]. Errors in the chemotherapy process occur at all stages of medication use [ ]. However, the prescribing stage plays a key role in the creation of chemotherapy errors [ , ]. Therefore, the use of computerized physician order entry (CPOE) or chemotherapy prescription clinical decision support systems (CDSSs) is recommended [ , ]. Multiple stakeholders, physicians, nurses, and pharmacists, but also patients and their relatives, are involved in the process. This adds to the risk picture, which emphasizes the importance of accurate and timely information about dosages, sequence of therapies, supportive medications, and duration of treatment [ ]. Therefore, a common source providing this information was needed. This paper presents a newly developed e-library for chemotherapy regimens with a focus on results from a usage and usability study.
In Sweden, descriptions of chemotherapy regimens were earlier developed and compiled locally within health care organizations. This was mostly because Sweden, with its 10 million inhabitants, is divided into 21 county councils, representing different geographical regions, all having far-reaching autonomy regarding the planning, financing, and operation of the region’s health care. In total, there are 17 oncology clinics in Sweden, which manage chemotherapy regimens, of which 7 are located at university hospitals. Additionally, there are several smaller units at other health care clinics, which also manage regimens. The many autonomous clinics have resulted in the same chemotherapy treatments occurring under different names and with different dosages causing uncertainty and risks for mix-ups. To overcome these uncertainties and risks, a national knowledge source for regimens (an e-library) was developed, with standardized nomenclature and content in chemotherapy regimens, also facilitating the exchange of information between hospitals, CPOE systems, and patients.
A standardized national source for chemotherapy regimens can constitute a preventive safety barrier function in the chemotherapy process . Standardization of workflow processes, prescribing, preparation, dispensing, and administration are recommended as safeguards against medication errors [ , ]. Standardized order sets [ ], standardized design and architecture [ ], and standardized protocols and dosing [ ] are beneficial when implementing CPOE systems. In 2020, the HemOnc group (working with a collaborative web-based knowledge platform for oncology professionals in the United States) published a proposal for a standardized nomenclature for chemotherapy regimens that were also compared to the thesaurus of the US National Cancer Institute [ ]. In a recently published article, a review of attempts to standardize chemotherapy nomenclature is presented together with recommendations from a European expert panel of oncology pharmacists [ ]. In this case, standardization is one way to ensure that all involved health care units have access to a standardized nomenclature and the latest evidence on chemotherapy treatments, which also allows for increased patient safety.
For the national e-library to serve as the intended safety barrier function, it needs to be designed and developed in accordance with user and usability criteria. Shulman et al  discusses principles for oncology eHealth records and claims that such systems should be designed to perform logically and straightforwardly, be user-friendly, and always be available to users. The same is applicable for a web-based resource. Based on such an approach the e-library was developed in a user-centered process and has been available since 2015 [ ]. Representatives from different user groups were involved throughout the development process, including oncology nurses, physicians, hospital pharmacists, and patients with cancer.
The national e-library  contains the following parts: (1) basic facts containing important medical and pharmaceutical information on drugs; (2) regimens presented per diagnosis with an overview (including instructions, precautions, and recommendation for dose reduction), adverse drug reactions (ADRs), and a detailed administration schedule; (3) information sheets for patients per regimen, providing a short description of the treatment and the most common or important ADRs with advice for self-treatment and when to contact the hospital; (4) support documents for health care professionals; and (5) newsletters published after updates of the e-library. Part of a regimen is shown in . The primary users are physicians, nurses, and pharmacists. They can access the information in the e-library for reading, printing, or downloading XML files for the CPOE systems used in Sweden. The patients gain access to the information sheets per regimen through their nurse.
Since the information in the library is both complex and extensive, it is central that the users can utilize the resource as intended. Otherwise, it might not be the source for improved performance and quality in cancer care that is envisioned. The aim of this study was to evaluate the usage and usability of the e-library for chemotherapy regimens developed to reduce medication errors and increase patient safety. This paper presents the results from the evaluation, which will also serve as input for continuous development and improvement.
A combination of methods was chosen to obtain a comprehensive view of the usage, including a compilation of subjective views of the users, analysis of statistics from the website, and an expert evaluation of the usability of the e-library . The results will help understand where further development is needed or will have the most impact ( ). The evaluation consisted of five parts:
- Website statistics collected from May to December 2020 show the number of users in total, unique users, when they visit the e-library, and from where the visitors were.
- A web survey for people visiting the e-library was conducted during January 2020. The survey consisted of questions about the visitor’s role and function, what information they were looking for, how pleased they were with the visit, and if they had suggestions of improvements.
- A heuristic evaluation of the user interface [ ] was performed in June 2020 to identify details in the user interface design that could be improved to increase usability.
- Spontaneous user feedback was collected in the form of emails sent to the development team or project leader during 2020, which was obtained from users presenting their role and question or suggestions.
- Qualitative interviews with 4 nurses, 3 physicians, and 3 pharmacists from various regions in Sweden were performed at the end of 2020 and at the beginning of 2021. As the intention is that the patients should get regimen-related information through their nurse and not through the e-library, no patient interviews were conducted. Interview questions concerned what parts of the e-library they used, how they used it, their experiences of using the e-library in their work compared to how they worked before, and their expectations for further development. The interviews were transcribed verbatim, and a process of transcript analysis was applied to identify themes.
During the last 8 months of 2020, the average number of visits to the website per month was just above 2500, and the number of unique visitors was 870. Most visitors were from Sweden, with some users from other countries in northern Europe, especially Scandinavia. Visitors from Sweden’s 3 largest cities (Stockholm, Gothenburg, and Malmoe) accounted for more than 50% of all visits. Most visits took place on Mondays to Fridays, but there were 5-10 visits per day on weekends.
From the survey with 292 answers, it became clear that the visitors were mainly physicians (49%, 144/292) and nurses (36%, 104/292), and, to some extent, pharmacists (7%, 21/292). The rest were from the pharmaceutical industry and others. Almost 80% (224/292) stated that they searched for regimens and 90% (264/292) stated that they found what they were looking for. Most of the users were satisfied with their visit. The less satisfied visitors (8%, 22/292) lacked the regimens or diagnostic areas that they were looking for. There were 32 comments, 18 of which asked for more regimens and information sheets for patients. The rest of the comments concerned the design of the survey or were positive reviews.
The heuristic evaluation was performed in accordance with the Nielsen and Molich’s  framework, which implies that a product or system is evaluated in a structured way following ten predefined usability heuristics: (1) visibility of system status; (2) match between systems and the real world; (3) user control and freedom; (4) consistency and standards; (5) error prevention; (6) recognition rather than recall; (7) flexibility and efficiency of use; (8) aesthetic and minimalist design; (9) recognize, diagnose, and recover from errors; and (10) help and documentation. The results show that the e-library follows many of the existing design principles; therefore. it could be concluded to be usable in its current form. Nevertheless, some improvement suggestions were identified, such as moving patient information sheets to a separate tab in the menu, improving the search function by providing suggestions, and improving the start page by moving newsletters to a separate tab.
Spontaneous User Feedback
A total of 86 emails with spontaneous user feedback were received from nurses (50%, 43/86), pharmacists (20%, 17/86), physicians (17%, 15/86), the pharmaceutical industry (8%, 7/86), and others (5%, 4/86). Of the emails, 78% (67/86) concerned questions such as “When will regimens arrive in a certain diagnostic group?” “How long should the drug temozolomide be given before starting radiation therapy?” In 19 of 86 (22%) emails, the sender had discovered errors in the e-library, which mainly concern the regimens, such as patient information sheets where there is a lack of a certain side effect or that the incorrect “treatment intention” had been chosen. All emails were answered, and in cases where there was incorrect or unclear information in the e-library’s documents, it was rectified.
Interview results showed that most hospitals used a CPOE system for their prescription and administration (3 systems exist). One hospital has no system; instead, they use the administration schedule from the e-library. Those with a CPOE system use the library in various ways, including importing the XML files, reading the information in the library and then transferring the information to their CPOE system, or using the library as a reference to check their information. The newsletter is published regularly on the e-library’s website, with information on new regimens or basic facts, and detailed information about changes in the regimens was appreciated by most of the interviewees, but not all of them had noticed them. The basic facts are used in various ways depending on profession; the pharmacists focus on preparation, shelf-life, and references to stability studies, the nurses focus on ADRs, and the physicians focus on new drugs. The nurses, especially, were interested in the patient information sheets that are available for most of the regimens. Most hospitals already had their patient information sheets but will switch to the ones in the e-library. The available supporting documents for health care professionals, and especially “Management of side effects associated with immunotherapy with checkpoint inhibitors,” are indicated by the nurses and physicians as very useful. Among the expectations for further development was a harmonization of regimens used in different diagnostic areas; for example, the same amount of infusion fluid and infusion times for the same drug.
The results from the user evaluation indicate that the e-library can be concluded to be used in the intended manner, and the users do not have any problems interacting with the knowledge source. From the web surveys and interviews, it becomes clear that it is the content that the users focus on. The usability of the website is not addressed by the users, which may be interpreted as subjective satisfaction in the interaction with the system. The heuristic evaluation showed that there were minor usability issues that should be addressed to improve the overall usability of the website. The web survey showed that the users substantially are satisfied with their visit. Most of the users found what they were looking for, and their main feedback was a desire for more regimens, more diagnostic areas to be covered, and more patient information sheets, which are continuously added and updated. The expressed needs from the users are useful to understand which areas and regimens should be prioritized in the development work. The number of visits indicates that the resource is used extensively, and the geographical span shows that the e-library has emerged as the national resource it is intended to be. Users from almost all Swedish regions exist, although the 3 largest cities stand for the main usage. However, the introduction of the library has proceeded fairly quickly, and it is believed that it is only a matter of time before all regions have adopted the use of the e-library.
The spontaneous user feedback shows clearly that the e-library is used. Contact with the users is vital, and websites generally facilitate a rapid and comprehensive means of knowledge dissemination . Emails from users reporting incorrectness are gratefully received, and they show that errors can slip through despite several checks. The interviews revealed various ways to use the information in the regimens. Some use the XML files to import the regimen to their CPOE system, which is a way to ensure that the transmitted information is accurate. Those transferring the information explained that they had to adjust information in their CPOE system; therefore, is it easier to copy one of their regimens and implement the necessary changes to match the new regimen. Some clinics do not have any CPOE system yet, still relying on handwritten orders and documentation. To transfer information manually is always a risk for introducing errors. Users have different needs depending on whether they are doctors, nurses, or pharmacists and depending on whether they work at a university hospital or a smaller hospital. The e-library can support these different needs from the users.
Automated downloading of the regimens from the e-library to the CPOE systems could improve the process, ensuring updated information without manual handling. However, the CPOE systems do not support this yet, and the process and approval of the updated information at each local clinic must be adapted to the change.
This study uses a mix of methods to address the usage and usability of a web-based resource for chemotherapy regimens. One can always discuss which mix of methods is optimal, and the combination used here could certainly be extended to include additional ones. Cognitive walkthrough is one such method, which could have been adopted—it is an explicit and detailed procedure to simulate a uses’ problem-solving process at each step through the dialogue with the system having potentially provided additional input in evaluating its usability .
The qualitative interviews revealed that nurses have started to replace their old patient information sheets with the ones in the e-library. In a follow-up study, it would be relevant to include patients in the evaluation process.
The study is performed in a geographically delimited system (Sweden), but the article contributes with working methods for how user-driven development contributes to a more standardized working method, which could also provide safer care. That knowledge is useful beyond the Swedish healthcare context. Finally, a longer-term evaluation is required to gain good insight into how the system is used and to evaluate how it contributes to increased safety in cancer care.
The comprehensive user evaluation conducted is an important part of continuing the user-centered process that started already during the development of the e-library. Multiple evaluation methods complement one another by providing input from several perspectives (ie, expert or user, subjective or objective) that may be triangulated and hence identify critical design aspects and user needs . The combination of methods applied in the evaluation presented in this paper included both objective usage statistics, expert methods for usability, and the users’ subjective feedback through a web survey and qualitative interviews. This ensures that the design and content of the e-library comply with the users’ needs and works as feedback for continuous development and learning [ ]. Thus, the evaluation is a vital part assuring that the e-library act as a safety barrier, is well designed and that general design flaw is avoided, a design flaw that otherwise could create new risks in the chemotherapy process. The evaluation will contribute to a deeper understanding of users’ judgements about the library content and help to develop strategies for increasing the national usage of the library. With a broad national usage, the e-library can become a source for organizational and national learning and a source for continuous improvement of cancer care in Sweden.
We would like to thank the health care personnel who provided feedback on the usage and usability of the e-library. This research received no grant from any funding agency in the public, commercial, or not-for-profit sectors.
The study was conceived and designed by all authors. AF performed the data collection, as well as most of the data analysis. All authors took part in drafting the manuscript; however, AF performed the main drafting.
Conflicts of Interest
- Müller T. Typical medication errors in oncology: analysis and prevention strategies. Onkologie 2003 Dec;26(6):539-544. [CrossRef] [Medline]
- Weingart SN, Zhang L, Sweeney M, Hassett M. Chemotherapy medication errors. Lancet Oncol 2018 Apr;19(4):e191-e199. [CrossRef] [Medline]
- Serrano-Fabiá A, Albert-Marí A, Almenar-Cubells D, Jiménez-Torres NV. Multidisciplinary system for detecting medication errors in antineoplastic chemotherapy. J Oncol Pharm Pract 2010 Jun;16(2):105-112. [CrossRef] [Medline]
- Fyhr A, Akselsson R. Characteristics of medication errors with parenteral cytotoxic drugs. Eur J Cancer Care (Engl) 2012 Sep;21(5):606-613 [FREE Full text] [CrossRef] [Medline]
- Kullberg A, Larsen J, Sharp L. 'Why is there another person's name on my infusion bag?' Patient safety in chemotherapy care - a review of the literature. Eur J Oncol Nurs 2013 Apr;17(2):228-235. [CrossRef] [Medline]
- Rahimi R, Moghaddasi H, Rafsanjani KA, Bahoush G, Kazemi A. Effects of chemotherapy prescription clinical decision-support systems on the chemotherapy process: A systematic review. Int J Med Inform 2019 Feb;122:20-26. [CrossRef] [Medline]
- Warner JL, Cowan AJ, Hall AC, Yang PC. HemOnc.org: A Collaborative Online Knowledge Platform for Oncology Professionals. J Oncol Pract 2015 May;11(3):e336-e350 [FREE Full text] [CrossRef] [Medline]
- Hollnagel E. Barriers and Accident Prevention. Farnham: Ashgate; 2004:A.
- Goldspiel B, Hoffman JM, Griffith NL, Goodin S, DeChristoforo R, Montello CM, et al. ASHP guidelines on preventing medication errors with chemotherapy and biotherapy. Am J Health Syst Pharm 2015 Apr 15;72(8):e6-e35. [CrossRef] [Medline]
- Huertas-Fernández MJ, Martínez-Bautista MJ, Rodríguez-Mateos ME, Zarzuela-Ramírez M, Muñoz-Lucero T, Baena-Cañada JM. Implementation of safeguards to improve patient safety in chemotherapy. Clin Transl Oncol 2017 Sep;19(9):1099-1106. [CrossRef] [Medline]
- Classen D, Bates DW, Denham CR. Meaningful use of computerized prescriber order entry. J Patient Saf 2010 Mar;6(1):15-23. [CrossRef] [Medline]
- Kukreti V, Cosby R, Cheung A, Lankshear S, ST Computerized Prescriber Order Entry Guideline Development Group. Computerized prescriber order entry in the outpatient oncology setting: from evidence to meaningful use. Curr Oncol 2014 Aug;21(4):e604-e612 [FREE Full text] [CrossRef] [Medline]
- Dabliz R, Poon SK, Ritchie A, Burke R, Penm J. Usability evaluation of an integrated electronic medication management system implemented in an oncology setting using the unified theory of the acceptance and use of technology. BMC Med Inform Decis Mak 2021 Jan 06;21(1):4 [FREE Full text] [CrossRef] [Medline]
- Rubinstein SM, Yang PC, Cowan AJ, Warner JL. Standardizing Chemotherapy Regimen Nomenclature: A Proposal and Evaluation of the HemOnc and National Cancer Institute Thesaurus Regimen Content. JCO Clin Cancer Inform 2020 Jan;4:60-70 [FREE Full text] [CrossRef] [Medline]
- Terkola R, Bardin C, Lizeaga Cundin G, Zeinab N, Crul M. Identifying options for oncology therapy regimen codification to improve standardization-combined results of an expert panel and a review. J Clin Pharm Ther 2021 Oct;46(5):1238-1244 [FREE Full text] [CrossRef] [Medline]
- Shulman LN, Miller RS, Ambinder EP, Yu PP, Cox JV. Principles of Safe Practice Using an Oncology EHR System for Chemotherapy Ordering, Preparation, and Administration, Part 1 of 2. J Oncol Pract 2008 Jul;4(4):203-206 [FREE Full text] [CrossRef] [Medline]
- Fyhr A, Borell J, Jerkeman M, Ek Å. National e-library for standardized chemotherapy regimens. Acta Oncol 2020 Sep;59(9):1079-1083. [CrossRef] [Medline]
- Regionala Cancercentrum i Samverkan. URL: https://kunskapsbanken.cancercentrum.se/ [accessed 2022-02-10]
- Price M, Weber J, Bellwood P, Diemert S, Habibi R. Evaluation of eHealth System Usability and Safety. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach. Victoria, BC: University of Victoria; Feb 27, 2017.
- Nielsen J, Molich R. Heuristic evaluation of user interfaces. 1990 Presented at: CHI90: Conference on Human Factors in Computing; April 1-5, 1990; Seattle, WA p. 249-256. [CrossRef]
- Nielsen J, Mack RL. Usability Inspection Methods. New York, NY: Wiley; May 09, 1994.
- Carayon P, Salwei ME. Moving toward a sociotechnical systems approach to continuous health information technology design: the path forward for improving electronic health record usability and reducing clinician burnout. J Am Med Inform Assoc 2021 Apr 23;28(5):1026-1028 [FREE Full text] [CrossRef] [Medline]
|ADR: adverse drug reaction|
|CDSS: clinical decision support system|
|CPOE: computerized physician order entry|
Edited by A Kushniruk; submitted 17.09.21; peer-reviewed by H Moghaddasi, M Hellaby; comments to author 28.11.21; revised version received 09.12.21; accepted 19.12.21; published 17.02.22Copyright
©AnnSofie Fyhr, Johanna Persson, Åsa Ek. Originally published in JMIR Human Factors (https://humanfactors.jmir.org), 17.02.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on https://humanfactors.jmir.org, as well as this copyright and license information must be included.