Published on in Vol 9 , No 4 (2022) :Oct-Dec

Preprints (earlier versions) of this paper are available at, first published .
A Mobile App Leveraging Citizenship Engagement to Perform Anonymized Longitudinal Studies in the Context of COVID-19 Adverse Drug Reaction Monitoring: Development and Usability Study

A Mobile App Leveraging Citizenship Engagement to Perform Anonymized Longitudinal Studies in the Context of COVID-19 Adverse Drug Reaction Monitoring: Development and Usability Study

A Mobile App Leveraging Citizenship Engagement to Perform Anonymized Longitudinal Studies in the Context of COVID-19 Adverse Drug Reaction Monitoring: Development and Usability Study

Original Paper

1Department of Statistics and Quantitative Methods, University of Milano-Bicocca, Milan, Italy

2School of Medicine and Surgery, University of Milano-Bicocca, Monza, Italy

Corresponding Author:

Dario Pescini, PhD

Department of Statistics and Quantitative Methods

University of Milano-Bicocca

Via Bicocca degli Arcimboldi 8

Milan, 20126


Phone: 39 0264485835


Background: Over the past few years, studies have increasingly focused on the development of mobile apps as complementary tools to existing traditional pharmacovigilance surveillance systems for improving and facilitating adverse drug reaction (ADR) reporting.

Objective: In this research, we evaluated the potentiality of a new mobile app (vaxEffect@UniMiB) to perform longitudinal studies, while preserving the anonymity of the respondents. We applied the app to monitor the ADRs during the COVID-19 vaccination campaign in a sample of the Italian population.

Methods: We administered vaxEffect@UniMiB to a convenience sample of academic subjects vaccinated at the Milano-Bicocca University hub for COVID-19 during the Italian national vaccination campaign. vaxEffect@UniMiB was developed for both Android and iOS devices. The mobile app asks users to send their medical history and, upon every vaccine administration, their vaccination data and the ADRs that occurred within 7 days postvaccination, making it possible to follow the ADR dynamics for each respondent. The app sends data over the web to an application server. The server, along with receiving all user data, saves the data in a SQL database server and reminds patients to submit vaccine and ADR data by push notifications sent to the mobile app through Firebase Cloud Messaging (FCM). On initial startup of the app, a unique user identifier (UUID) was generated for each respondent, so its anonymity was completely ensured, while enabling longitudinal studies.

Results: A total of 3712 people were vaccinated during the first vaccination wave. A total of 2733 (73.6%) respondents between the ages of 19 and 80 years, coming from the University of Milano-Bicocca (UniMiB) and the Politecnico of Milan (PoliMi), participated in the survey. Overall, we collected information about vaccination and ADRs to the first vaccine dose for 2226 subjects (60.0% of the first dose vaccinated), to the second dose for 1610 subjects (43.4% of the second dose vaccinated), and, in a nonsponsored fashion, to the third dose for 169 individuals (4.6%).

Conclusions: vaxEffect@UniMiB was revealed to be the first attempt in performing longitudinal studies to monitor the same subject over time in terms of the reported ADRs after each vaccine administration, while guaranteeing complete anonymity of the subject. A series of aspects contributed to the positive involvement from people in using this app to report their ADRs to vaccination: ease of use, availability from multiple platforms, anonymity of all survey participants and protection of the submitted data, and the health care workers’ support.

JMIR Hum Factors 2022;9(4):e38701



Pharmacovigilance is defined by the World Health Organization (WHO) as “the science and activities relating to the detection, assessment, understanding, and prevention of adverse effects or any other drug-related problem” [1]. Monitoring the safety of medicines is fundamental because many previously undetected adverse drug reactions (ADRs) during clinical trials may occur when patients are exposed to drugs. ADRs refer to any undesired effect emerged during treatment with a given pharmaceutical product [2].

We performed a literature analysis to review the currently adopted methods for ADR reporting. In particular, we searched PubMed for papers published in the past 10 years using the following query: “adverse drug reactions” OR “adverse drug reaction reporting” OR “adverse drug reaction reporting systems” (query performed on June 22, 2022). The considered papers included clinical trials, meta-analyses, randomized controlled trials, and reviews. This search identified 131 studies. By excluding papers that, after a full text or abstract screening, seemed too descriptive and generic or focused on just presenting the incidence or prevalence of ADRs, the literature survey performed was limited to 28 eligible papers. We found 9 other papers that met the considered criteria. We grouped the studies based on the system used to conduct ADR surveillance.

Of the total 39 papers found, 7 (17.9%) [3-9] are based on traditional safety surveillance. This approach relied on data collected through passive reports of ADRs from consumers and health providers, medical literature, and observational databases [10]. This way, any active measure is taken to search for ADRs, except for voluntary and spontaneous reports by health care providers or vaccinees [11,12]. Although passive reporting is still used, thanks to the possibility of identifying rare and unexpected ADRs, its ease of implementation, and relatively low cost, it can be limited by underreporting, lack of information, and difficulty in determining ADR rates [12,13]. Moreover, traditional surveillance of ADRs after drug exposure is slow and patchy. Indeed, once reported by patients or health care professionals, ADRs are first assessed by drug experts and pharmaceutical companies, and then the result of the assessment is passed on to government agencies. This path may entail underreporting issues, with considerable data loss.

Substantial delays may occur between occurrence and discovery of ADRs, because reports received by government agencies are often released with a delay of months or even years due to the period required for proper assessment of ADR data. Further limitations of traditional ADR surveillance relate to the fact that ADR reporting is voluntary, and several studies have shown that as many as 90% of serious ADRs end up being unreported [2,14]. Moreover, several studies have reported low awareness of the national public ADR reporting systems within the general population, with a portion of the public not even being aware of them. This adds up to a problem of declining reporting by physicians that has been linked to the belief that it is only necessary to report serious or unexpected ADRs or that a single observed ADR could not contribute to medical knowledge or that only ADRs derived from a certain causal relationship with the use of a particular drug should be reported [10,15].

In response to these limitations, ADR surveillance is also conducted by applying a participant-centered active approach that, contrary to passive reporting, involves an intentional searching for ADRs via continuous and organized contact with health care providers or other relevant reporting sources, such as hospitals, laboratories, and patients [11,12].

At the same time, active surveillance requires more resources for a positive outcome. Of the 39 papers, in 16 (41%), active systems falling under this category used digital strategies to facilitate and encourage ADR surveillance, which means the adoption of electronic technology for capturing and processing data.

Common participant-centered active surveillance methods are conducted using health diary reporting cards [11], SMS text messages (eg, SmartVax [16], VaxTracker [17], stimulated telephone-assisted rapid safety surveillance [STARSS] [18], FASTMum [19], FAST-Health [20]), and email (eg, VaxTracker [17], Lareb Intensive Monitoring [21], and integrated vaccine surveillance system [IVSS] [22]) to interact with individuals and prompt them to report their ADRs by sending a web link to complete an online questionnaire, interviews collected over the phone (eg, FASTMum [19], IVSS [22], and a project developed in Brazil [13,23,24]), during medical visits [25], or mailing questionnaires [26-29].

Thanks to the spread of internet use, our literature survey suggested increasing usage of web-based platforms to promote ADR reporting. A large proportion of the papers found (16/39, 41%) adopted interactive web-based ADR-reporting tools [30-33] also accessible from mobile devices [34] that linked to the receipt of spontaneous ADR reports [30], to the use of electronic health records to directly report ADRs [35-37], and to spontaneous reports sent through multiple forms of notification (eg, via an ADR portal, telephone, and email [38]) and sent on the online Yellow Card scheme [39-42]. The same explored ways are also used individually or in combination to report ADRs through a paper-based form sent by email or fax, an electronic form sent on a website, or telephone interviews [43,44]. This category of web-based platforms to report ADRs also includes the CANVAS program [45], which relies on reports received through a web-based survey, with an additional telephone follow-up when severe ADRs need to be reported.

The results of the European Union's Innovative Medicines Initiative Web–Recognising Adverse Drug Reactions (WEB-RADR) project, reported in a recent paper [46], suggest that the patient’s increased interest in improving knowledge about drug safety, combined with recent technological advances in information communication, make it possible to improve ADR reporting and patient safety communication using mobile apps.

Over the past few years, studies have increasingly focused on the development and usage of mobile apps as additional tools to pharmacovigilance for improving and facilitating reporting of ADRs, as reported by Defer et al [47], Ahn et al [48], Prakash et al [49], Montastruc et al [50], and de Vries et al [51]. Indeed, in contrast to classical surveillance systems, mobile devices offer a platform for improving the accessibility of data and increasing the speed at which they are transmitted between institutions. These tools lead to real-time systems of pharmacovigilance with the potential of enabling a near-instantaneous transmission of patient safety information through user-friendly and interactive graphics within the app [10]. The extensive usage of smartphones and other mobile devices with internet access leads to the expectation of a greater involvement from the public in using mobile health apps to report suspected ADRs, complementing traditional surveillance systems.

In view of these findings, we searched for available mobile apps with the ability to allow ADR reporting for medical devices, drugs, or vaccines, without searching for apps that are specific for a particular case of study.

Among the tools we found, the MedWatcher app [52] is a web-based and mobile app available to the US public, developed in partnership with the Food and Drug Administration (FDA) Center for Devices and Radiologic Health (CDRH) and available on Apple App Store and Google Play Store for iOS and Android devices, respectively. Through MedWatcher, reports are processed in a secure cloud computing environment, manually reviewed, and then transmitted electronically to the Manufacturer and User Facility Device Experience (MAUDE) database. VigiBIP [50], designed by the pharmacovigilance network in France and available for Android and iOS devices, has the notable functionality to allow reporting of not only data but also photographs and images [50,53]. Similarly, the ADR PvPi app [49], which was developed by the National Coordination Centre-Pharmacovigilance Programme of India, facilitates ADR reporting, enabling document and image attachment by health care professionals as well as consumers. However, it was available just on Android devices from the Google Play Store.

Another available tool is presented by Ahn et al [48], where the system design is accurately presented as consisting of a mobile app, a cloud server, and a dashboard. Patients’ data on symptoms and vaccination are captured using a mobile device or a web interface and then sent to the cloud server. Medical personnel then use the web dashboard to receive and analyze the submitted data. This mobile app was developed for both Android and iOS devices.

My eReport France [47] is a free mobile app available for both Android and iOS devices. When individuals use this app, they receive a participant card containing quick response (QR) codes to download the app and submit their ADR report.

The WEB-RADR project developed 3 mobile apps suitable for patients, caregivers, and health care professionals not only to report their ADRs but also to receive up-to-date information and news alerts on selected medicines, check the number of received reports for a particular drug, and view previous ADR reports submitted through the app. These 3 apps, which are free to use for everyone on iOS and Android devices, are the free Yellow Card smartphone app [42] developed in 2015, the Halmed app [54] launched in 2016 to submit ADRs to drugs directly to the Agency for Medicinal Products and Medical Devices of Croatia, and the Lareb app [55] developed in 2016 by the Netherlands Pharmacovigilance Centre Lareb but no longer maintained or available for download.

In the context of the WEB-RADR project, the international WEB-RADR app has been launched under the name Med Safety [56]. Similar to the other WEB-RADR apps, Med Safety is a smartphone app freely available on both iOS and Android devices for reporting ADRs to national competent authorities, keeping track of previously reported information and receiving news about drugs of interest. With the ability to function without an internet connection, individuals can partially create reports and save them for completion later or create and save reports without sending them immediately but subsequently once connectivity is re-established.

In the following cases, no specific details about the app design and features, other than the chosen platform and the declared ability to allow ADR reporting, are available: the CANVAS app [45] and the ADR online app [57], which are available only for iOS devices; the ADR Reporter app [58] and the TMDA Adverse Reactions Reporting Tool app [59], which are available only for Android devices; and the Easypharm app [60] and the UAE RADR app [61], which are available for both Android and iOS devices.

Although the development of mobile apps to report ADRs is continuing to progress, providing valuable knowledge about drug safety, one feature that is not present in any of the reviewed apps in our literature analysis is the ability to perform longitudinal analysis in an anonymous way, a key point to foster citizenship engagement.

Although having valuable features, all the currently available mobile apps lack the ability to collect individual subjects’ ADR time courses anonymously. Monitoring the same subjects over time, while guaranteeing complete privacy and anonymity of the submitted information, can be a valuable addition to pharmacovigilance because it can be used to gather longitudinal safety data of a drug. Consequently, the temporal investigation of the same patient can help detect, within a specific clinical context, any temporal patterns or ADR combinations.

On these grounds, we developed, for Android and iOS devices, a new, free app called vaxEffect@UniMiB, tested its capability to enable longitudinal studies by analyzing per subject time series, and evaluated its potential in fostering spontaneous citizenship participation in an ADR data collection campaign. In this work, we specifically presented app features by evaluating the reporting rate of ADRs in a sample of Italian academic subjects vaccinated for COVID-19.

Study Design

During the COVID-19 health emergency, the University of Milano-Bicocca (UniMiB) participated in a vaccination campaign deploying a hub. The hub administered the first dose between March 5 and 29, 2021, and the second dose between May 31 and June 11, 2021. A third dose was administered at the national level, but the authorities decided to not involve the hub. The campaign involved professors, researchers, adjunct professors, postdocs, PhD students, and the technical administrative staff of UniMiB or of the Politecnico of Milan (PoliMi). Due to the deputed authority emergency plan, the UniMiB hub invited all UniMiB employees and only a third of PoliMi ones, the others being distributed to different hubs.

Ethical Considerations

Ethical approval was obtained from the Ethical Committee of UniMiB (protocol no. 0041302/21).

Data Collection

After receiving ethical approval, vaxEffect@UniMiB was made available on March 24, 2021, from Google Play Store and on March 29, 2021, from Apple App Store. It was advertised to subjects invited to the vaccination campaign at the UniMiB hub.

We started to collect reports sent to vaxEffect@UniMiB, starting from January 1, 2021. Given the health emergency, it was not possible to plan the different phases of the study. To highlight the uncertainties of the situation, we report that the UniMiB hub was not aware at the end of the first dose administration whether the second dose would have been administered by the hub itself. When the third dose was administered at the national level, the hub was not involved, and we decided to test the flexibility of the app by keeping it active to allow respondents to use it on a spontaneous basis. The app was deactivated on February 9, 2022.

At each vaccination wave, the physicians involved in the vaccination campaign explained to attendees the importance of reporting ADRs and delivered a brochure explaining the aim of the research, as well as the guarantee of anonymization of the digital tool. Moreover, corresponding to the beginning of the first 2 vaccine doses’ administration campaign, an informative mail was sent to UniMiB employees.

System Overview

Our system consists of a mobile app, a web server, and a Firebase Cloud Messaging (FCM) server (Google Inc), as shown in Figure 1. At initial startup, the mobile app inquiries about the user’s health condition, focusing only on aspects relevant to the vaccination campaign, and sends the answers to the application server. Upon every vaccine administration, vaccine data and ADRs occurring within 7 days postvaccination are sent by the patient using the mobile app.

The application server is responsible for receiving all the user data, to store them in a SQL database server, and to remind the patients to submit vaccine and ADR data. The reminders are sent to the patients via push notifications sent to the mobile app by the application server through FCM.

The complete data exchange between the mobile app and the application server is described in Figure 2.

Figure 1. Our system. FCM: Firebase Cloud Messaging.
View this figure
Figure 2. Our system’s data exchange. UID: user identifier.
View this figure

Security by Design

The framework for recording the responses provided by the participants is a one-way client/server architecture. We decided to opt for this type of structure to ensure the safety of the respondents. This way, it was not necessary to store the answers to the questionnaires on the terminal used by the respondents, minimizing the possibility of illegitimate acquisition of information in the event of loss, theft, or fraudulent use of the device.

Furthermore, to ensure the anonymity of the respondents, while preserving the possibility of analyzing the time series relating to the same individual, we decided to generate a unique user identifier (UUID) using an FCM [62] token so that the UUID could not be used to trace the user of the mobile device (the FCM token identifies a single app on a single device and is not associated with any sensitive user data).

Mobile App

The mobile app was developed for both Android and iOS devices using the FCM software development kit (SDK). We specifically used Android Studio (Java language) and XCode (Swift language) for, respectively, the Android and iOS versions. Our app sends data over the web to an application server using an application programming interface (API) gateway. The data are sent using the https protocol and are validated by the application server.

On initial startup of our mobile app, a splash screen is displayed (Figure 3A), and at the same time, the FCM SDK generates a registration token for the client app instance. This token uniquely identifies an app installation and cannot be linked in any way to the device owner, and it is used to receive a push notification from the application server. After the splash screen, a short description of the mobile app (Figure 3B), the privacy agreement (Figure 3C), and a medical history questionnaire (Figure 3D) are displayed. Medical history data and the registration token are sent to the application server. The application server stores the data received in the database and returns a UUID, which is encrypted and saved in the local mobile app file system.

On subsequent startup or after completing the medical history questionnaire, the patient is asked for vaccination data (Figure 4A) and, 7 days later, for ADRs (Figure 4B).

Figure 3. Mobile app first startup. (A) Splash screen. (B) Mobile app description. In this panel, the app is presented, specifying that the user will be asked for vaccination data after the first and, if any, every subsequent vaccination. Seven days after each vaccination, the user will be then asked to submit ADRs. Alerts will be sent to remind the user about the data submission after each moment. In addition, this panel specifies that data are collected on a voluntary basis, ensuring total anonymity of the submitted data. (C) Privacy. In this panel, it is specified that the architecture underlying vaxEffect@UniMiB ensures the privacy of the submitted data, which can be viewed by only the responsible physician and authorized subjects. Moreover, the user needs to confirm to be of legal age, to authorize the processing of the data that will be submitted, and to have read the provided privacy statements. (D) Medical history. At this stage, the user has to complete the medical history questionnaire by inserting demographic data and indicating whether 1 or more chronic diseases among those specified in the “Data collected by the app” section have been ever diagnosed. In this panel, an example of the data that the user needs to submit at this stage is provided by asking whether the user suffers from diabetes, bronchitis, or asthma. ADR: adverse drug reaction.
View this figure
Figure 4. Mobile app at subsequent startup. (A) Vaccine data. After each vaccination, the app asks the user to submit vaccination data. In this panel, an example of the data that the user needs to submit at this stage is provided by asking the name of the administered vaccine (choosing among Comirnaty, moderna-mRNA1273, or Vaxzevria) and the hospital or institution where the vaccine has been administered. (B) ADRs. Seven days after each vaccination, the user will be asked to submit specific ADRs among those reported in the “Data collected by the app” section. In this panel, an example of the data that the user needs to submit at this stage is provided by asking whether fever, headache, or tiredness has occurred after vaccination. ADR: adverse drug reaction.
View this figure

Application Server

The application server was developed in Python. Its main purpose is to collect data from the mobile app and store them in the database. The application server is also responsible for generating the UUID, to pair it with the FCM token, and to send push notification requests to the FCM server. Indeed, according to the data provided by the patient, after 7 days from the date of vaccination, the application server sends a push notification request to the FCM server in order to inform the patient that the ADR data need to be collected. The notification messages delivered to the mobile app from the FCM server open when the patient taps on the notification.

Data Collected by the App

In addition to demographic data, the medical history questionnaire that respondents had to complete required them to indicate whether they had been diagnosed with one or more chronic diseases among those suggested by the app: dyslipidemia, hypertension, autoimmune diseases, asthma, cardiovascular diseases, tumor, liver diseases, diabetes, bronchitis, or kidney disease.

In addition, vaxEffect@UniMiB asks users to register the occurrence of specific ADRs among those reported as a COVID-19 vaccine side effect by the Centers for Disease Control and Prevention (CDC) and WHO: fever, headache, injection site pain, tiredness, muscular pain, swollen lymph nodes, joint pain, paresthesia, dizziness, sleepiness, nausea, and abdominal pain. At the same stage, vaxEffect@UniMiB collects information about the vaccine administrated to the subject, such as its kind: Vaxzevria (AstraZeneca), Comirnaty (Pfizer/BioNTech), moderna-mRNA1273 (Moderna), or Janssen (Johnson & Johnson).

Data Analysis

To avoid cases of intentional or unintentional false reports by users, we checked the consistency of the submitted data at different time points, discarding nonconsistent data. Due the free access to the app and the data themselves, it was not possible to automatically detect fraudulent data reporting, but during a manual inspection, no fraudulent report was found.

For each user, we labeled the registered data, defining whether they belonged to the first, second, or third vaccination dose. By keeping separate the 3 sets of records, we analyzed in each the number of ADRs registered by each user by separating into 2 groups subjects who reported more than 3 postvaccination symptoms and those who reported fewer than 3 symptoms. These 2 groups of subjects were compared in terms of sex and age by using the chi-square test [63].

Through radar plots, we analyzed the kinds of ADRs reported by each user after every vaccine dose by keeping subjects separated in terms of gender and age.

Using a Sankey diagram [64], we followed the ADRs reported by users who submitted their data after all 3 vaccine doses, classifying who registered 0, 1, 2, 3, or more than 3 postvaccination symptoms.

Characterization of vaxEffect@UniMiB Users

The total number of people vaccinated during the first wave of vaccine administration (March 2021) conducted at the Bicocca hub was 3712 (a third of the entire personnel of UniMiB and PoliMi). This partial coverage of the personnel can be ascribed to the fact that health care workers received the vaccine before the launching of the app, with the effect that not everyone may have necessarily participated in the survey presented in this work.

During this wave, vaxEffect@UniMiB received a total of 2733 medical history questionnaires from individuals between the ages of 19 and 80 years. In particular, as shown in the Medical History column of Tables 1-3, a greater involvement of UniMiB users compared to PoliMi users resulted in more usage of the app (n=1676, 61.3%, vs n=1057, 38.7%) at UniMiB due to how vaccinations at the UniMiB hub were organized. Indeed, all the UniMiB employees were allowed to get vaccinated at the hub. On the contrary, only part of PoliMi employees were allowed to get vaccinated at this hub.

vaxEffect@UniMiB allows characterizing the vaccinated population by the kind of administered vaccine. It is possible to observe that the administered vaccines were Vaxzevria (n=2711, 62.8%), Comirnaty (n=1253, 29.0%), moderna-mRNA1273 (n=349, 8.1%), and Janssen (n=5, 0.1%). The collected data made it possible to follow the proportion of each vaccine kind through the 3 doses (Figure 5).

Table 1. Characterization of vaxEffect@UniMiB users for the institution stratification.a
InstitutionMedical history (N=2733), n (%)Dose 1Dose 2Dose 3

Admin.b (N=2360), n (%)ADRc (N=2226), n (%)Admin. (N=1779), n (%)ADR (N=1610), n (%)Admin. (N=179), n (%)ADR (N=169), n (%)
UniMiBd1676 (61.3)1383 (58.6)1332 (59.8)1252 (70.4)1138 (70.7)116 (64.8)111 (65.7)
PoliMie1057 (38.7)977 (41.1)894 (40.2)527 (29.6)472 (29.3)63 (35.2)58 (34.3)

aEach cell reports the absolute and the relative frequency computed for the sample size N reported in the column headings.

bAdmin.: individuals who submitted their vaccination data.

cADR: adverse drug reaction (individuals who registered their ADRs 7 days postvaccination).

dUniMiB: University of Milano-Bicocca.

ePoliMi: Politecnico of Milan.

Table 2. Characterization of vaxEffect@UniMiB users for the gender stratification.a
GenderMedical history (N=2733), n (%)Dose 1Dose 2Dose 3

Admin.b (N=2360), n (%)ADRc (N=2226), n (%)Admin. (N=1779), n (%)ADR (N=1610), n (%)Admin. (N=179), n (%)ADR (N=169), n (%)
Male1224 (44.8)1075 (45.6)998 (44.8)713 (40.1)645 (40.1)68 (38.0)65 (38.5)
Female1509 (55.2)1285 (54.4)1228 (55.2)1066 (59.9)965 (59.9)111 (62.0)104 (61.5)

aEach cell reports the absolute and the relative frequency computed for the sample size N reported in the column headings.

bAdmin.: individuals who submitted their vaccination data.

cADR: adverse drug reaction (individuals who registered their ADRs 7 days postvaccination).

Table 3. Characterization of vaxEffect@UniMiB users for the age stratification.a
Age (years)Medical history (N=2733), n (%)Dose 1Dose 2Dose 3

Admin.b (N=2360), n (%)ADRc (N=2226), n (%)Admin. (N=1779), n (%)ADR (N=1610), n (%)Admin. (N=179), n (%)ADR (N=169), n (%)
<301301 (47.6)1024 (43.4)962 (43.2)955 (53.7)861 (53.5)64 (35.8)59 (34.9)
30-551109 (40.6)1043 (44.2)983 (44.2)634 (35.6)575 (35.7)80 (44.7)76 (45.0)
>55323 (11.8)293 (12.4)281 (12.6)190 (10.7)174 (10.8)35 (19.5)34 (20.1)

aEach cell reports the absolute and the relative frequency computed for the sample size N reported in the column headings.

bAdmin.: individuals who submitted their vaccination data.

cADR: adverse drug reaction (individuals who registered their ADRs 7 days postvaccination).

Figure 5. Kinds of vaccines administered to the same subject through the 3 vaccination campaigns.
View this figure

Stratification of subjects by gender revealed a slightly greater proportion of female (n=1509, 55.2%) compared to male (n=1224, 44.8%) subjects. When stratified by age, younger (<30 years) and middle-aged (30-55 years) subjects represented the most active users, together constituting 88.2% (n=2410) of the total participants. On the contrary, older users (>55 years) represented just 11.8% (n=323) of the total set of survey respondents. Age and sex distributions of users employed in UniMiB were in line with those in PoliMi.

Once the medical history questionnaire was completed, 2360 (86.3%) and 1779 (65.1%) of all survey participants continued to use vaxEffect@UniMiB, reporting their vaccination data of, respectively, the first and the second dose campaign. Subsequently, 2226 (94.3%) and 1610 (90.5%) users who submitted their vaccination data after, respectively, the first and the second doses also reported their ADRs to the vaccination.

In response to the first vaccine dose, we collected information about vaccination and ADRs for 2360 and 2226 subjects (see the Dose 1 column in Tables 1-3), which represented, respectively, 63.6% and 59.9% of the 3712 individuals who were vaccinated during the first dose campaign. After the second vaccine dose, we collected information about vaccination and ADRs for 1779 and 1610 subjects (see the Dose 2 column in Tables 1-3), which represented, respectively, 47.8% and 43.3% of the 3718 individuals who were vaccinated during the second dose campaign.

The Dose 3 column in Tables 1-3 reports the same information relative to the administration of the third vaccine dose. Users who went further in reporting their data were considerably lower in number compared to the first two dose campaigns, since 179 users spontaneously continued to use vaxEffect@UniMiB to report their vaccination data and 169 of them also submitted their postvaccination ADRs. In this vaccination campaign, differently from the previous 2 ones, UniMiB was no longer the vaccination hub.

Stratifying the subjects who participated in the survey by gender (Figure 6) the prevalence of chronic diseases was overall fairly low since a maximum value of 0.078 was registered. Exploring the inserted data, the prevalence of the registered chronic diseases was higher in male than in female users, except for asthma, liver disorders, and autoimmune diseases. Moreover, focusing on cases where the gap between males and females was wider, we found that male subjects suffer more from hypertension, whereas females are more prone to autoimmune diseases.

Figure 6. Prevalence expressed in percentage of chronic diseases among male (M) and female (F) users of vaxEffect@UniMiB. The blue and pink lines depict the percentage prevalence of each single chronic disease between, respectively, male and female subjects. Below each label, the corresponding absolute count in male and female users is reported colored according to the color scale on the right.
View this figure

Characterization of ADRs After Each Vaccination Dose

As shown in Figure 7, independently of the type of vaccine and the dose administered, the response of subjects in terms of the number of reported ADRs to vaccination depended on gender and age. Indeed, females were more prone to a worse response to the vaccine, given the constant higher proportion of females compared to males who registered more than 3 symptoms/week postvaccination. In terms of age, the prevalence of ADRs to vaccination turned out to be worse in younger and middle-aged respondents compared to older users.

Moving on to a more in-depth analysis of the kinds of symptoms reported after each vaccination dose, the most common vaccine ADRs were fever, headache, injection site pain, tiredness, muscular pain, joint pain, and sleepiness (Figure 8).

The types of ADRs to vaccine administration were dependent on both the gender and age of the participants. When analyzing the response of males and females separately, a progressive reduction in ADR prevalence through the 3 vaccine doses emerged for all the 3 age groups. However, we noted an exacerbation after the third vaccination dose for some of them. Specifically, among female subjects, more cases of fever, headache, injection site pain, and muscular pain were reported by young users; more cases of injection site pain, muscular pain, joint pain, and sleepiness by middle-aged users; and more cases of headache and injection site pain by older users. Instead, among male subjects, more cases of headache, injection site pain, tiredness, muscular pain, and joint pain were reported by young users; more cases of fever, headache, injection site pain, muscular pain, and joint pain by middle-aged users; and more cases of injection site pain by older users. Comparing males and females within the same vaccination dose, the same behavior as seen in Figure 7 emerged since females complained about more ADRs after each vaccine administration compared to male participants.

Figure 7. Number of ADRs registered by users after each vaccination dose. Subjects separately stratified by gender (A) and age (B) were grouped according to the number of postvaccination symptoms (>=3 Sympt. and <3 Sympt.). On the top of each plot, the result of the chi-square test and the corresponding P value are reported. ADR: adverse drug reaction.
View this figure
Figure 8. Kinds of ADRs registered by users after each vaccination dose. (A) Light-, middle-, and dark-pink lines denote the percentage prevalence of ADRs reported, respectively, by younger (labeled as <30 years), middle-aged (labeled as 30-55 years), and older (labeled as >55 years) female users. (B) Light-, middle-, and dark-blue lines denote the percentage prevalence of ADRs reported, respectively, by younger (labeled as <30 years), middle-aged (labeled as 30-55 years), and older (labeled as >55 years) male users. Below each label, the corresponding absolute count in younger, middle-aged, and older users is reported for females and males, colored according to the color scale on the right. ADR: adverse drug reaction.
View this figure

Longitudinal Analysis of ADRs Through the 3 Vaccine Doses

The innovative feature that distinguishes vaxEffect@UniMiB from the existing mobile apps aimed at reporting ADRs is the ability to perform a longitudinal analysis to monitor the same subject over time in terms of the reported ADRs after each vaccine administration, while guaranteeing anonymity of the submitted data.

In the overall cohort of the survey presented in this work, 167 users regularly submitted their ADRs after each of the 3 doses, enabling us to detect how the response of individuals changes depending on the number of ADRs registered after each dose (Figure 9).

Figure 9. Longitudinal analysis of ADRs through the 3 vaccine doses. Each level identifies each of the 3 vaccine doses, respectively, labeled as Dose 1, Dose 2, and Dose 3. Each node identifies the 5 classes of symptom abundance, respectively, labeled as 0 Sympt., 1 Sympt., 2 Sympt., 3 Sympt., and >3 Sympt. and colored according to the legend on the right. The width of the links is proportional to the amount of subjects who experienced a given number of ADRs between Dose 1 and Dose 2 and between Dose 2 and Dose 3. ADR: adverse drug reaction.
View this figure

Principal Results

We proposed a new mobile app called vaxEffect@UniMiB, able to perform anonymized longitudinal studies, that, combined with the health care workers’ sponsorships, fostered the spontaneous citizenship participation in COVID-19 vaccination ADR data collection. In this work, over 3712 subjects were vaccinated during the first dose at the Bicocca hub, and we received a total of 2733 (73.6%) reports.

It should be noted that health care workers, receiving the vaccine before the launching of the app, may not have necessarily participated in the survey presented in this work. Moreover, due to deputed authority emergency plan, the hub invited all UniMiB employees and only part of PoliMi ones, the others being distributed to different hubs. This aspect further affected the number of reports received over the total number of subjects invited to the vaccination campaign.

Thanks to vaxEffect@UniMiB, we collected vaccination and ADR data relative to the first vaccine dose for 2226 (60.0%) of the 3712 individuals vaccinated during the first dose campaign, to the second dose for 1610 (43.3%) of the 3718 individuals vaccinated during the second dose campaign, and, in a nonsponsored fashion, to the third dose for 169 (4.6%) individuals.

Although a direct comparison of our reporting rate against traditional ones is not possible, a reporting rate of 120 questionnaires submitted to the National Pharmacovigilance Network for any 100,000 doses administered was reported, according to the Annual Report of Aifa (Agenzia italiana del farmaco) related to the surveillance of COVID-19 vaccine safety during December 27, 2020-December 26, 2021.

Following a review of the currently available mobile apps for monitoring ADR reporting for medical devices, drugs, or vaccines, our app revealed to be one step ahead. Specifically, vaxEffect@UniMiB is the first successful attempt in tracking the same user over time and identifying putative temporal patterns in an anonymous fashion. With this ability, we were able to analyze in this study the trend that emerged through the 3 vaccine doses administered to the same user. The first vaccination resulted in the strongest response among individuals, given the high prevalence of subjects with 3 or more than 3 symptoms. As already evidenced by Haas et al [65], part of this high response to the vaccine could be due to a misattribution of commonly experienced and nonspecific symptoms, such as headache and tiredness, as specific reactions due to the vaccine administration, instead of a condition of anxiety and worry making people hyperalert to the occurrence of any possible adverse events. Among the 3 administered doses, the second one was the least symptomatic since a general decreased trend in the adverse events from the first to the second dose emerged. Finally, between the second and the third dose, there was a predominantly increased trend in the number of reported symptoms. This behavior might be due to a change in the vaccine type, since after the second dose, AstraZeneca has not been administered since. However, given the low number of subjects who regularly submitted their data after all the 3 vaccine doses, we are not in a position to determine whether this increased trend may for sure be due to the change in the vaccine type that occurred between the second and the third vaccination campaign.

A series of aspects contributed to a greater involvement from people in using this app to report their ADRs to vaccination.

First, the user-friendly and interactive graphics of vaxEffect@UniMiB allowed ease of use of the app, resulting in a positive response rate after both the first and the second vaccination campaign. Although the response after the third campaign was lower compared to the previous 2 ones, it could be still considered a positive result. Indeed, during the third vaccination campaign, any reminder notification was sent to users, denoting a completely spontaneous participation of subjects in that phase. This result indicates the effectiveness of vaxEffect@UniMiB as an ADR-reporting system, although the active contribution of an institutional sponsorship is necessary.

Another aspect that contributed to the increase in the usage of vaxEffect@UniMiB was its availability for both Android and iOS devices, which made it reachable to as many users as possible.

Even more importantly, the protection of the submitted information assured users that high standards of confidentiality were maintained during data reporting. The architecture underlying vaxEffect@UniMiB ensured the anonymity of all the survey participants and also the privacy of their data, minimizing the illegitimate acquisition of information in the case of loss, theft, or fraudulent use of the device. These actions allowed guaranteeing proper use of the outcomes of the research, while preserving the possibility of tracking the same subject over time.


Although these features allowed for a positive response from the public, we are also aware that following an even stronger sponsorship of the vaccination campaign, the 373 users who just compiled the medical questionnaire without then reporting any other data would continue using the app. Likewise, increased support in the dissemination of the app could encourage who partially submitted data for only 1 or 2 vaccine doses to complete the entire survey.

We intend to also clarify that since the study presented in this work was a pilot study conducted on a selected voluntary population, we cannot exclude that our statistical measures could be affected by volunteer bias. This bias can occur at all stages of the study, and differences between volunteers and the target population are not restricted to sociodemographic factors but can include attitudes toward the study and institutions involved. However, the likelihood of volunteer bias increases as the refusal rate to volunteer increases. We observed a high rate of participation and a low rate of attraction until the third vaccination wave (when the Bicocca hub was not more in place and the in-person stimulus ended), probably due to the endorsement of participants’ anonymity and confidentiality.

The impact of this bias is unknown, but the observed sample should be quite younger and have a higher level of education and income than the target population, showing a higher rate of participation. These characteristics have been associated with greater extension and severity, so we cannot exclude that we overestimated the prevalence of symptoms [66].

This work being a pilot study, we chose to restrict the analyses to the respondents explicitly imputable to the hub itself. Consequently, data entry was strictly controlled since only ADRs to the administered vaccination were asked to be submitted. In this way, we were sure that all the reported information was not false-true ADRs due to other medical conditions of the user. However, as a suggestion for the future, if vaxEffect@UniMiB will be used in a different and more open context, it will be necessary to integrate a preprocessing phase to tame the possible distortions induced by the noncontrolled environment in which respondents report information.


The results of this study showed that it is possible to leverage active citizenship engagement to increase the ADR-reporting rate using an agile and pervasive tool, such as a mobile app.

The combination between the synergy of the physician who proposed the app and the flexibility of vaxEffect@UniMiB seems to be an excellent tool to ensure high response rates in situations in which a priori information is scarce and decisions need to be made in a short time.

In addition to the promising results so far obtained, this work highlighted some practices that need to be carried out to foster approach efficacity, such as conspicuous support in the dissemination of the app to encourage more subjects to report their ADRs and steady user engagement to sustain data submission through the entire vaccination treatment when only partial information is reported.


AA, AZ, and DP performed conceptualization; AA and DP, data curation; MD, AA, and DP, formal analysis; AZ and DP, funding acquisition; MB, MEP, and MAR, investigation; AA, AZ, and DP, methodology; AZ and DP, project administration; MB, MEP, MAR, and DP, resources; AA and DP, software; AZ and DP, supervision; MD and DP, visualization; MD, AA, AZ, and DP, writing—original draft; and MD, AA, AZ, DP, and MAR, writing—review and editing.

Conflicts of Interest

None declared.

  1. World Health Organization. The Importance of Pharmacovigilance. Geneva: World Health Organization; 2002.
  2. Pappa D, Stergioulas LK. Harnessing social media data for pharmacovigilance: a review of current state of the art, challenges and future directions. Int J Data Sci Anal 2019 Feb 12;8(2):113-135 [FREE Full text] [CrossRef]
  3. Fernandez S, Lenoir C, Samer C, Rollason V. Drug interactions with apixaban: a systematic review of the literature and an analysis of VigiBase, the World Health Organization database of spontaneous safety reports. Pharmacol Res Perspect 2020 Oct 02;8(5):e00647 [FREE Full text] [CrossRef] [Medline]
  4. Moro PL, Woo EJ, Marquez P, Cano M. Monitoring the safety of high-dose, trivalent inactivated influenza vaccine in the vaccine adverse event reporting system (VAERS), 2011 - 2019. Vaccine 2020 Aug 18;38(37):5923-5926. [CrossRef] [Medline]
  5. Logan JK, Wickramaratne Senarath Yapa S, Harinstein L, Saluja B, Muñoz M, Sahajwalla C, et al. Drug interaction between febuxostat and thiopurine antimetabolites: a review of the FDA adverse event reporting system and medical literature. Pharmacotherapy 2020 Feb 20;40(2):125-132. [CrossRef] [Medline]
  6. Shimabukuro TT, Nguyen M, Martin D, DeStefano F. Safety monitoring in the Vaccine Adverse Event Reporting System (VAERS). Vaccine 2015 Aug 26;33(36):4398-4405 [FREE Full text] [CrossRef] [Medline]
  7. Fang H, Lin X, Zhang J, Hong Z, Sugiyama K, Nozaki T, et al. Multifaceted interventions for improving spontaneous reporting of adverse drug reactions in a general hospital in China. BMC Pharmacol Toxicol 2017 Jun 26;18(1):49 [FREE Full text] [CrossRef] [Medline]
  8. Qing-ping S, Xiao-dong J, Feng D, Yan L, Mei-ling Y, Jin-xiu Z, et al. Consequences, measurement, and evaluation of the costs associated with adverse drug reactions among hospitalized patients in China. BMC Health Serv Res 2014 Feb 17;14(1):73 [FREE Full text] [CrossRef] [Medline]
  9. Ng WL, Lim KS, Hariraj V, Lee SC, Wo WK, Ramli A, et al. Incidence of allopurinol-induced severe cutaneous adverse drug reaction in Malaysia. Br J Clin Pharmacol 2022 Aug;88(8):3782-3788 [FREE Full text] [CrossRef] [Medline]
  10. Salathé M. Digital pharmacovigilance and disease surveillance: combining traditional and big-data systems for better public health. J Infect Dis 2016 Dec 01;214(suppl_4):S399-S403 [FREE Full text] [CrossRef] [Medline]
  11. Psihogios A, Bota AB, Mithani SS, Greyson D, Zhu DT, Fung SG, et al. A scoping review of active, participant-centred, digital adverse events following immunization (AEFI) surveillance: a Canadian immunization research network study. Vaccine 2022 Jul 29;40(31):4065-4080 [FREE Full text] [CrossRef] [Medline]
  12. Heininger U, Holm K, Caplanusi I, Bailey SR, Abdoellah SA, Arellano F. Guide to active vaccine safety surveillance: report of CIOMS working group on vaccine safety-executive summary. Vaccine 2017;35(32):3917-3921 [FREE Full text] [CrossRef]
  13. Cashman P, Macartney K, Khandaker G, King C, Gold M, Durrheim DN. Participant-centred active surveillance of adverse events following immunisation: a narrative review. Int Health 2017 May 01;9(3):164-176 [FREE Full text] [CrossRef] [Medline]
  14. Liu F, Jagannatha A, Yu H. Towards drug safety surveillance and pharmacovigilance: current progress in detecting medication and adverse drug events from electronic health records. Drug Saf 2019 Jan 16;42(1):95-97 [FREE Full text] [CrossRef] [Medline]
  15. Palleria C, Leporini C, Chimirri S, Marrazzo G, Sacchetta S, Bruno L, et al. Limitations and obstacles of the spontaneous adverse drugs reactions reporting: two “challenging” case reports. Journal of Pharmacology and Pharmacotherapeutics 2013 Dec 01;4(1_suppl):S66-S72. [CrossRef]
  16. Salter S, Singh G, Nissen L, Trentino K, Murray K, Lee K, et al. Active vaccine safety surveillance of seasonal influenza vaccination via a scalable, integrated system in Western Australian pharmacies: a prospective cohort study. BMJ Open 2021 Jun 08;11(6):e048109 [FREE Full text] [CrossRef] [Medline]
  17. Cashman P, Moberley S, Dalton C, Stephenson J, Elvidge E, Butler M, et al. Vaxtracker: active on-line surveillance for adverse events following inactivated influenza vaccine in children. Vaccine 2014 Sep 22;32(42):5503-5508 [FREE Full text] [CrossRef] [Medline]
  18. Gold MS, Lincoln G, Cashman P, Braunack-Mayer A, Stocks N. Efficacy of m-Health for the detection of adverse events following immunization: the stimulated telephone assisted rapid safety surveillance (STARSS) randomised control trial. Vaccine 2021 Jan 08;39(2):332-342. [CrossRef] [Medline]
  19. Regan AK, Blyth CC, Tracey L, Mak DB, Richmond PC, Effler PV. Comparison of text-messaging to voice telephone interviews for active surveillance of adverse events following immunisation. Vaccine 2015 Jul 17;33(31):3689-3694. [CrossRef] [Medline]
  20. Tracey LE, Regan AK, Mak DB, Effler PV. Adverse events following influenza immunization reported by healthcare personnel using active surveillance based on text messages. Infect Control Hosp Epidemiol 2015 May 05;36(5):608-610. [CrossRef] [Medline]
  21. van Balveren-Slingerland L, Kant A, Härmark L. Web-based intensive monitoring of adverse events following influenza vaccination in general practice. Vaccine 2015 May 05;33(19):2283-2288. [CrossRef] [Medline]
  22. Newes-Adeyi G, Greece J, Bozeman S, Walker DK, Lewis F, Gidudu J. Active surveillance for influenza vaccine adverse events: the integrated vaccine surveillance system. Vaccine 2012 Feb 01;30(6):1050-1055. [CrossRef] [Medline]
  23. O'Connor MN, O'Sullivan D, Gallagher PF, Eustace J, Byrne S, O'Mahony D. Prevention of hospital-acquired adverse drug reactions in older people using screening tool of older persons' prescriptions and screening tool to alert to right treatment criteria: a cluster randomized controlled trial. J Am Geriatr Soc 2016 Aug;64(8):1558-1566 [FREE Full text] [CrossRef] [Medline]
  24. Herdeiro MT, Ribeiro-Vaz I, Ferreira M, Polónia J, Falcão A, Figueiras A. Workshop- and telephone-based interventions to improve adverse drug reaction reporting. Drug Saf 2012 Dec 13;35(8):655-665. [CrossRef]
  25. Egberts KM, Gerlach M, Correll CU, Plener PL, Malzahn U, Heuschmann P, et al. Serious adverse drug reactions in children and adolescents treated on- and off-label with antidepressants and antipsychotics in clinical practice. Pharmacopsychiatry 2022 Sep 07;55(5):255-265 [FREE Full text] [CrossRef] [Medline]
  26. Potlog Shchory M, Goldstein LH, Arcavi L, Shihmanter R, Berkovitch M, Levy A. Increasing adverse drug reaction reporting: how can we do better? PLoS One 2020 Aug 13;15(8):e0235591 [FREE Full text] [CrossRef] [Medline]
  27. Nolte H, Bernstein DI, Sussman GL, Svanholm Fogh B, Lu S, Husøy B, et al. Impact of adverse event solicitation on the safety profile of SQ house dust mite sublingual immunotherapy tablet. J Allergy Clin Immunol Pract 2018;6(6):2081-2086.e1 [FREE Full text] [CrossRef] [Medline]
  28. Coates M, Spanos M, Parmar P, Chandrasekhar T, Sikich L. A review of methods for monitoring adverse events in pediatric psychopharmacology clinical trials. Drug Saf 2018 May 9;41(5):465-471 [FREE Full text] [CrossRef] [Medline]
  29. Lindell-Osuagwu L, Sepponen K, Farooqui S, Kokki H, Hämeen-Anttila K, Vainio K. Parental reporting of adverse drug events and other drug-related problems in children in Finland. Eur J Clin Pharmacol 2013 Apr 24;69(4):985-994. [CrossRef] [Medline]
  30. Verden A, Dimbil M, Kyle R, Overstreet B, Hoffman KB. Analysis of spontaneous postmarket case reports submitted to the FDA regarding thromboembolic adverse events and JAK inhibitors. Drug Saf 2018 Apr;41(4):357-361 [FREE Full text] [CrossRef] [Medline]
  31. Abadie D, Chebane L, Bert M, Durrieu G, Montastruc JL. Addendum to: Online reporting of adverse drug reactions: a study from a French regional pharmacovigilance center. Thérapie 69(5): 395-400 (2014) DOI:10.2515/therapie/2014035. Therapie 2015 Oct 16:Online ahead of print. [CrossRef] [Medline]
  32. Huang CY, Hsieh YW, Lin WL, Wu HC. PCN1 the evaluation of ADR online reporting of radiopaque agents in a university hospital in TAIWAN. Value Health 2010 Nov;13(7):A510. [CrossRef]
  33. Ribeiro-Vaz I, Santos C, da Costa-Pereira A, Cruz-Correia R. Promoting spontaneous adverse drug reaction reporting in hospitals using a hyperlink to the online reporting form: an ecological study in Portugal. Drug Saf 2012 May 01;35(5):387-394. [CrossRef] [Medline]
  34. Jeon M, Kim J, Oh CE, Lee JY. Adverse events following immunization associated with the first and second doses of the ChAdOx1 nCoV-19 vaccine among healthcare workers in Korea. Vaccines (Basel) 2021 Sep 28;9(10):1096 [FREE Full text] [CrossRef] [Medline]
  35. Song H, Pei X, Liu Z, Shen C, Sun J, Liu Y, et al. Pharmacovigilance in China: evolution and future challenges. Br J Clin Pharmacol 2022 Feb 14:Online ahead of print. [CrossRef] [Medline]
  36. Lander AR, Blicher TM, Jimenez-Solem E, Jespersen M, Kampmann JP, Christensen HR. Introducing an adverse drug event manager. Eur J Hosp Pharm 2013 Feb 20;20(2):78-81. [CrossRef]
  37. Lee SB, Schepers GP, Goldberg KL. Electronic adverse-drug-reaction-reporting program. Am J Health Syst Pharm 2004 Jun 15;61(12):1230, 1232-1230, 1233. [CrossRef] [Medline]
  38. Salvador MR, Monteiro C, Pereira L, Duarte AP. Quality of spontaneous reports of adverse drug reactions sent to a regional pharmacovigilance unit. Int J Environ Res Public Health 2022 Mar 22;19(7):3754 [FREE Full text] [CrossRef] [Medline]
  39. Hawcutt DB, Mainie P, Riordan A, Smyth RL, Pirmohamed M. Reported paediatric adverse drug reactions in the UK 2000-2009. Br J Clin Pharmacol 2012 Mar;73(3):437-446 [FREE Full text] [CrossRef] [Medline]
  40. Ferner RE, Stevens RJ, Anton C, Aronson JK. Spontaneous reporting to regulatory authorities of suspected adverse drug reactions to COVID-19 vaccines over time: the effect of publicity. Drug Saf 2022 Feb 22;45(2):137-144 [FREE Full text] [CrossRef] [Medline]
  41. Hasan SS, Bal N, Baker I, Kow CS, Khan MU. Adverse drug reaction reporting and prescribing trends of drugs for attention deficit hyperactivity disorder in primary care England, 2010-2019. J Atten Disord 2022 Feb;26(3):467-475 [FREE Full text] [CrossRef] [Medline]
  42. Welcome to the Yellow Card Reporting Site.   URL: [accessed 2022-10-07]
  43. van Hunsel F, Härmark L, Pal S, Olsson S, van Grootheest K. Experiences with adverse drug reaction reporting by patients: an 11-country survey. Drug Saf 2012 Jan 01;35(1):45-60. [CrossRef] [Medline]
  44. Yousef NB, Yenugadhati N, Alqahtani N, Alshahrani A, Alshahrani M, Al Jeraisy M, et al. Patterns of adverse drug reactions (ADRs) in Saudi Arabia. Saudi Pharm J 2022 Jan;30(1):8-13 [FREE Full text] [CrossRef] [Medline]
  45. Wilson K, Atkinson KM, Westeinde J, Bell C, Marty K, Fergusson D, et al. An evaluation of the feasibility and usability of a proof of concept mobile app for adverse event reporting post influenza vaccination. Hum Vaccin Immunother 2016 Jul 02;12(7):1738-1748 [FREE Full text] [CrossRef] [Medline]
  46. Pierce CE, de Vries ST, Bodin-Parssinen S, Härmark L, Tregunno P, Lewis DJ, et al. Recommendations on the use of mobile applications for the collection and communication of pharmaceutical product safety information: lessons from IMI WEB-RADR. Drug Saf 2019 Apr 25;42(4):477-489 [FREE Full text] [CrossRef] [Medline]
  47. Defer G, Fedrizzi S, Chevanne D, Montastruc F, Briant AR, Parienti JJ, French VigipSEP Study Group, Société Francophone de la Sclérose en Plaques (SFSEP). Adverse drug reaction reporting using a mobile device application by persons with multiple sclerosis: a cluster randomized controlled trial. Drug Saf 2021 Feb 09;44(2):223-233 [FREE Full text] [CrossRef] [Medline]
  48. Ahn E, Liu N, Parekh T, Patel R, Baldacchino T, Mullavey T, et al. A mobile app and dashboard for early detection of infectious disease outbreaks: development study. JMIR Public Health Surveill 2021 Mar 09;7(3):e14837 [FREE Full text] [CrossRef] [Medline]
  49. Prakash J, Joshi K, Malik D, Mishra O, Sachan A, Kumar B, et al. “ADR PvPI” Android mobile app: report adverse drug reaction at any time anywhere in India. Indian J Pharmacol 2019;51(4):236. [CrossRef]
  50. Montastruc F, Bagheri H, Lacroix I, Damase-Michel C, Chebane L, Rousseau V, et al. Adverse drug reaction reports received through the mobile app, VigiBIP: a comparison with classical methods of reporting. Drug Saf 2018 May 21;41(5):511-514. [CrossRef] [Medline]
  51. de Vries ST, Denig P, Lasheras Ruiz C, Houÿez F, Wong L, Sutcliffe A, IMI Web-RADR Work Package 3b Consortium. Interest in a mobile app for two-way risk communication: a survey study among European healthcare professionals and patients. Drug Saf 2018 Jul 2;41(7):697-712 [FREE Full text] [CrossRef] [Medline]
  52. Bahk CY, Goshgarian M, Donahue K, Freifeld CC, Menone CM, Pierce CE, et al. Increasing patient engagement in pharmacovigilance through online community outreach and mobile reporting applications: an analysis of adverse event reporting for the Essure device in the US. Pharmaceut Med 2015;29(6):331-340 [FREE Full text] [CrossRef] [Medline]
  53. Unknown. Adverse drug reaction reports via VigiBIP app. Reactions Weekly 2018 Jan 13;1684(1):2-13. [CrossRef]
  54. WEB-RADR. Mobile Apps.   URL: [accessed 2022-10-07]
  55. Netherlands Pharmacovigilance Centre Lareb. New: Side Effect App.   URL: [accessed 2022-10-07]
  56. WEB-RADR. Med Safety App.   URL: [accessed 2022-10-07]
  57. ADR Online.   URL: [accessed 2022-10-07]
  58. ADR Reporter.   URL: [accessed 2022-10-07]
  59. TMDA Adverse Reactions Reporting Tool.   URL: [accessed 2022-10-07]
  60. Easypharm App.   URL: [accessed 2022-10-07]
  61. WEB-RADR. UAE RADR.   URL: [accessed 2022-10-07]
  62. Firebase Cloud Messaging.   URL: [accessed 2022-10-07]
  63. Pearson K. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philos Mag 2009 Apr 21;50(302):157-175. [CrossRef]
  64. Sankey Diagram in Python.   URL: [accessed 2022-10-07]
  65. Haas JW, Bender FL, Ballou S, Kelley JM, Wilhelm M, Miller FG, et al. Frequency of adverse events in the placebo arms of COVID-19 vaccine trials: a systematic review and meta-analysis. JAMA Netw Open 2022 Jan 04;5(1):e2143955 [FREE Full text] [CrossRef] [Medline]
  66. Camacho Moll ME, Salinas Martínez AM, Tovar Cisneros B, García Onofre JI, Navarrete Floriano G, Bermúdez de León M. Extension and severity of self-reported side effects of seven COVID-19 vaccines in Mexican population. Front Public Health 2022 Mar 14;10:834744 [FREE Full text] [CrossRef] [Medline]

ADR: adverse drug reaction
FCM: Firebase Cloud Messaging
IVSS: integrated vaccine surveillance system
PoliMi: Politecnico of Milan
SDK: software development kit
UID: user identifier
UniMiB: University of Milano-Bicocca
UUID: unique user identifier
WEB-RADR: Web–Recognising Adverse Drug Reactions
WHO: World Health Organization

Edited by A Kushniruk; submitted 20.04.22; peer-reviewed by S Rostam Niakan Kalhori, V Rajan, S Matsuda; comments to author 18.06.22; revised version received 08.07.22; accepted 02.08.22; published 04.11.22


©Marzia Di Filippo, Alessandro Avellone, Michael Belingheri, Maria Emilia Paladino, Michele Augusto Riva, Antonella Zambon, Dario Pescini. Originally published in JMIR Human Factors (, 04.11.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Human Factors, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.