This study did not require participants to be involved in any physical or mental intervention. This research also did not use personally identifiable information, thus exempted from institutional review board approval in accordance with the Ethical Guidelines for Medical and Health Research Involving Human Subjects stipulated by the Japanese national government.

The data in this study consists of geo-tagged tweets collected between January 1, 2020, and December 31, 2022, using Academic Research Access (discontinued in mid-2023). This dataset comprised 1,102,876 English tweets and 560,873 Japanese tweets. Note that we used the same dataset as our previous study on multilanguage data annotation [23]. Examples of paraphrased tweets showing their opinion about mask-wearing are shown in Textbox 1.

Demographic data were accessed from the Portal Site of Official Statistics of Japan [24] and Census Bureau data of the United States [25], obtained from the 2020 census. Previous studies have indicated that demographic characteristics such as sex, ethnicity, age, marital status, and employment status affect participation in preventive behaviors in the United States [26]. In this study, we focused on several demographics for population characteristics at the state/prefecture level: total population, population by age, education level, information on foreign members/race, and income (Table 1). For Japan, the education level was obtained from the number of employed persons aged 15 years and older.

The first preprocessing step involved validating each tweet’s location tag. Given the various types of location information in geo-tagged tweets, we focused on city-level information found in the “full-name” entity of the tweets. Instances where the geo-tagged location did not match the city lists in the United States and Japan were removed [27,28]. Next, we applied basic preprocessing steps, including changing usernames to common handles, addressing links, converting emojis to text, and removing duplicates and NAs.

Finally, we ensured that the tweets contained the keyword “mask” for English tweets and “マスク” (mask in Japanese) for Japanese tweets. In some cases, these keywords appeared as usernames or links, providing insufficient information or relevance to the topics and were thus eliminated in the process.

The annotation for our 1100 pairs of tweets was done by 4 annotators, 2 for each English and Japanese. We used Cohen kappa as the measure to evaluate interannotator agreement between annotators [29] and interpret the agreement results [30]. The complete detailed approach and evaluation are explained in Ferawati et al [23]. To combine the annotated label between the annotators, we assigned the agreed label as the final label and assigned the disagreed label as not relevant, not against, and unknown for each stage.

X studies often use human annotators with the aid of machine learning classifiers [31,32]. We designed a multilingual guideline to obtain a suitable data annotation for this study [23]. The results of the annotation show a case of imbalanced data, with one category having a much smaller number of tweets compared with another. However, due to the time and resources required for human annotation, it is not practical to annotate a large number of tweets. To address this issue, we propose a semisupervised approach by using the BERT (Bidirectional Encoder Representations from Transformers) score from BERT-based classification method as additional data for training. By including additional training sets from predicted tweets, we hope to improve the performance of the classifier to solve the cases, especially in public perception and attitudes for COVID-19 masking in the 2 countries.

An existing study explained that BERT surpassed traditional machine learning algorithms, such as Logistic Regression, Support Vector Machine, Random Forest, and Naive Bayes, where BERT was supported by its ability to understand contextual information appearing in social media [33]. BERT was also widely used to analyze and derive conclusions. Based on the results of these studies, we decided to use BERT-based models for our study. We trained the annotated datasets using BERT-based classifiers, RoBERTa base model [34] for English tweets and BERT base Japanese [35] for Japanese tweets. To evaluate the models, we set aside 15% of the annotated data as the test set and applied 5-fold cross-validation to the remaining data. The final model was selected from the fold with the best results on the test set, evaluated using F₁-macro to address data imbalance.

Due to the limited number of annotated data, we included all 1100 annotated tweets. For each language, tweets annotated with the same categories by both annotators were assigned to those categories. In cases of conflicting annotations—for example, where annotator 1 marked a tweet as not relevant while annotator 2 marked it as relevant—we categorized the tweet as not relevant. The same approach was applied for the stance stage (categorized as unclear, then assigned to not against) and mask-wearing (categorized as unknown).

We observed an imbalance in the categories of our annotated data, which likely contributed to the low performance of our models. Due to constraints in obtaining additional annotated data, we implemented a semisupervised approach for the stance and mask-wearing stages of the tweets. The semisupervised approach was a common measure taken to obtain more annotated data without a significant increase in annotation costs [36]. In this study, we stored the BERT outputs from the predictions and filtered them based on determined thresholds for each category. We considered 2 methods of adding pseudo-labeled data: adding an equal number of tweets to both categories and adding tweets only to the minority category. Using softmax-transformed values of the BERT output for each class, we selected a set of data to be added to the training. The objective for implementing the approach is to obtain labels with a higher confidence from the obtained softmax-transformer values as an additional sample for training. We maintained the chosen split of training and validation from the previous step in classifier training.

The selected tweets from this step were then included as additional training data using the same parameters and model as in the previous subsection. The data for validation and testing remained the same as the baseline. We experimented with several combinations of thresholds for each class in each stage (against and not for stance; no, yes, and unknown for mask-wearing), while endeavoring to keep the number of selected additional tweets balanced. When determining the threshold value for the selection, we selected a certain value to balance our objectives: retaining sufficient confident predictions for the reliability of our pseudo-label and having enough additional samples to meaningfully expand our dataset. The threshold was set by observing the transformed BERTScore from the prediction results, plotting them into a histogram with the aid of descriptive statistics of the score to determine the appropriate threshold for filtering. We then compared the results of each training.

We calculated the correlation between the predicted percentages and demographic data from the 2 countries, the United States and Japan, using Pearson correlation. In this study, we considered 3 levels of significance for the P values with a significance level of .05, .01, and .001.

For the annotation phase, we used only a subset of the entire dataset. Considering the reasonable workload of our annotators, we selected 1100 tweets from each language, as reported in another publication focusing on the annotation of a multicultural dataset [23]. Although the agreement levels for stance and mask-wearing stage are still less than <0.60—a score that was categorized as moderate by Cohen (1960) [29] but inadequate by McHugh (2012) [30]—they are deemed acceptable for our study (Table 2).

The number of tweets varies substantially across the country, with certain states or prefectures having far more tweets than others. The difference in number is primarily observed in densely populated areas, such as the capital or popular destinations. Among tweets identified as relevant to COVID-19, California leads the United States in mask-related discussions, while Tokyo tops the list in Japan. A list showing the top and bottom tweet counts for states (in the United States) and prefectures (in Japan) is provided (Table 3).

We fine-tuned roberta-base and bert-base-japanese-v3 with our annotated tweets to predict the rest of the data, a method commonly used for social media data [37,38]. We trained the classifier using this annotated data and then used it to predict the remaining tweets. The parameters for our final model used the AdamW optimizer, with a batch size of 32, 10 epochs with early stopping callback, learning rate of 0.00002, and weight decay of 0.01.

We used the entire sample of 1100 tweets for relevancy classification (Table 4). In the first stage of relevancy classification, we achieved the highest F₁-macro scores for both English and Japanese texts compared with the majority baseline. However, the stance and mask-wearing stage did not yield similar results, showing only a marginal increase from the majority baseline. This indicates that the models are still unable to correctly predict the minority categories. These results also mirror the annotation results, where annotators struggled to achieve higher interannotator agreement.

By adopting a semisupervised approach to train our classifier, we were able to improve second-stage F₁-scores that were nearly as high as those from the first stage—0.68 for English tweets and 0.72 for Japanese tweets.

Based on the results for the stance stage, the chosen threshold with a more balanced number of tweets in each category (eg, an additional 650 in the “against” category, bringing the total to 755 “against” and 775 “not against” for English data) achieved a better F₁-macro score than other alternatives.

When it comes to classifying mask-wearing behavior, maintaining a balanced sample for re-training is challenging due to the imbalance of labels in the annotated data. In English tweets, only 46 tweets were predicted as “not wearing mask” out of the entire dataset, so we included all the predicted tweets in the model. The best result was an F₁-score of 0.487, a substantial increase from the previous score of 0.301 with the original annotated data. For Japanese tweets, our best model only yielded an F₁-score of 0.48.

For the mask-wearing stage, the distribution of BERT score outputs was much more skewed than for stance in both languages. Because it was challenging to maintain balanced numbers for each category, we experimented with various threshold combinations to assess performance. Due to the difficulty in balancing the numbers, the improvement in the model performance seems rather limited. Therefore, it should be considered the best our classifiers could achieve and is acceptable to be used in our following tasks.

The final classifier model was trained on a combined dataset of the original annotated data and the semisupervised data. The remaining tweets were classified using the final model. We deemed the F₁ -score from the previous semisupervised results acceptable for our study and used it as a basis for generating a map of stance toward the mask mandate and actual mask-wearing behavior from related tweets in the United States and Japan, using both English and Japanese tweets.

The tweets were collected over a 3-year period, capturing the trend from early, peak, to post-peak periods of the COVID-19 pandemic. Throughout this time, various events and policies occurred, potentially influencing tweet volume and content (Figure 1). In the figure, no observable trend in the stance against COVID-19 masking is evident in the United States, suggesting little change in people’s opinion about mask-wearing from early 2020 to the end of 2022. Both countries show a decreasing trend in tweets against masking from January 2020 to April, followed by an upward trend after relevant departments released recommendations about mask-wearing.

Throughout the observed period, the CDC released several guidelines on the utilization of masks as a preventive measure. They initially mandated the use of masks in April 2020 and later revised the recommendation after the development of vaccines, such as lifting the recommendation to wear masks for fully vaccinated individuals in May 2021. In the month following CDC’s revision of their mask-wearing guidance to no need to wear a mask if vaccinated, tweets classified as “not wearing mask” reached their lowest point, while tweets classified as “wearing mask” peaked (Figure 1). Subsequently, the CDC released another recommendation to wear masks in high-transmission areas in July 2021, which is reflected in interesting changes observed in the time series plot of mask-wearing stages. The number of tweets indicating “not wearing mask” substantially increased from 0.03% in June 2021 to 0.21% in August 2021, while tweets about “wearing mask” saw a decrease from 24.41% in June 2021 to 16.76% in August 2021. In April 2022, as a result of a lawsuit regarding mask mandates deemed unlawful in Florida, nationwide mask mandates for airplanes and public transportation were no longer enforced [40]. As shown in Figure 1A, in the same month, a peak was observed in the stance against mask mandates in the United States, and tweets classified as “not wearing mask” peaked a month after the ruling, observed in May 2022.

In Japan, there are 3 noticeable fluctuations in the figure, closely related to COVID-19 policies. We observed some significant changes when the government announced a state of emergency (SoE) in April 2020 and subsequently lifted it in May 2020. After a dip in the percentage of tweets expressing a stance against masking after the first SoE in April 2020, the first peak appeared a month after the first SoE was lifted. These fluctuations are also consistent with findings from another study on responses to COVID-19 waves in Japan [41]. Another study discussed that there are indeed signs of disruptions due to COVID-19 in X, which shows top concerns during the SoE period and some impacts of the pandemic on societies in Japan [42]. The second peak appeared at the beginning of the Olympics. The Tokyo Olympics, held from July 23 to August 8, 2021, also influenced the observed changes in the figure. Before the Olympics began, in June 2021, there was a slight increase in tweets expressing a stance against mask-wearing, as shown in Figure 1B. The third peak appeared in June 2022, coinciding with the 7th surge in COVID-19 cases and an increase in “no mask” tweets, as observed by Suzuki et al [39].

We calculated the correlation between stance percentages and the demographic data from the country census for both the United States and Japan (Table 5). In the United States, we observed a negative correlation of −0.384 between household income and stance against masking percentages. Although this correlation is relatively low, it is still significant within the 1% level of significance threshold. This suggests that states with higher incomes tend to have lower percentages of tweets against masking. Additionally, education level also shows a significant negative correlation, showing that with an increase in the population completing higher studies, there was an observed decrease in tweets with a negative stance towards the mask mandate. In Japan, we found that the number of people older than 65 years showed a positive correlation with the percentage of tweets against the mask mandate at the prefecture level. Similar results were observed in the number of people who completed a bachelor or higher degree.

This study focuses solely on the English and Japanese languages for analyzing short and informal text, specifically tweet posts. In the second stage, we encountered numerous ambiguous and unclear tweets regarding attitude. Analyzing such stages proved challenging due to the short text and ambiguity of tweet content, which was often insufficient for accurately determining attitudes. Therefore, we chose to exclude a neutral option, limiting choices to either against or not.

Although X data might be subject to sampling bias due to its limited user base, the data were proven useful for infodemiology research, especially in identifying and monitoring public discussion topics during the pandemic [69,70]. Population-wise, our sample is restricted to individuals who voluntarily provide location information in their tweets. Despite the limitation of such data, it reflects the opinion of the users to a degree, as reported in an existing study that showed that geotagged tweets were able to display public opinion of people in the United States toward COVID-19 vaccinations [71]. Disparities in tweet counts across states/prefectures may influence the final percentage results displayed in the figures, which are based on the entire 3 years of data. We calculated the percentage based on the tweets observed in each state as one of the measures to address the potential problems due to the location data. The number of tweets in each state varies, so we avoid general bias caused by comparing the tweets based on their count, but based on the percentage of the classified tweets. Due to the data collection period and collective analysis after, the resulting figure, especially in the form of map visualization, may not sufficiently capture early COVID-19 period changes and post-pandemic trends shown in the time series plot.

A previous study identified around 8.4% of tweets as bots from around 200,000 accounts collected in the early period of the pandemic, March to June 2020 [72]. Another study mentioned that bots focused more on political content in the COVID-19 pandemic discourse compared with health content, such as actions taken by the public during the pandemic [73]. The bots mostly discussed information related to COVID-19 or pandemic statistics. In this study, we focused on mitigating bots’ influence, such as artificial amplification by removing duplicate tweets, one of which often appeared in posts by bot accounts. This action ensured that no single automated message could statistically dominate our social media dataset or skew our demographic correlations. To verify this approach, we conducted a manual audit in the annotated dataset, which was sampled randomly from the entire tweets data. We did not find apparent bot accounts during the check in our sample. Given this minimal presence, we conclude that the presence of bots does not significantly skew the overall stance analysis or correlations in our study.

The limited number of “not wearing” mask annotations constrained the number of predicted instances in the rest of the tweets. Consequently, the final predicted percentages for “not wearing” masks were also very small, only 0.07% for English tweets and 2.33% for Japanese tweets. This problem, caused by a small number of tweets annotated as “not wearing mask” was also further added by a surge in tweets mentioning “no-mask” after June 2022 [39]. This surge, as documented by Suzuki et al [39], may contribute to the disparity from previous studies and introduce bias into the correlation calculations, requiring people to interpret the results cautiously. To address these challenges, future improvements in classification and additional annotations are necessary for enhanced accuracy.

The initial score of the classification results obtained was not satisfactory. The F₁-score for stance against masking is 0.61 for English and 0.69 for Japanese. It was even smaller for the mask-wearing stage, with 0.31 for English and 0.40 for Japanese. While it is higher than the majority baseline, it is still not performing as well for the classification. There is a slight increase in the F₁-score after adding pseudo-labeled data, as observed in the stance and mask-wearing stage. However, the scores were still not as high, indicating that the topic to be classified is indeed challenging, as both human annotators and classifiers struggle to achieve a high score in classifying stance against masking and mask-wearing information of the users.

Regarding the mask-wearing stage, the number of tweets annotated as “not-wearing” masks is very small in the annotation phase (10 for English and 11 for Japanese). This small number affects the classification because when we split the data, we were working with an even smaller number of tweets for training, validation, and testing. This makes it even harder to classify the “not-wearing-mask” tweets, resulting in a low F₁-score in the testing phase. This issue is further highlighted in the classification of the remaining tweets, with only 0.07% predicted as “not-wearing” masks in English and 2.33% in Japanese out of all the predicted tweets. The low number of tweets predicted as “not-wearing” masks is also apparent in Figure 3, especially in the United States case. This limitation warrants caution in interpreting our results, especially since the annotation target was particularly challenging.

Due to the policy of X, we are unable to obtain the demographics of its users; thus, we considered using the demographics of the general population in the country. However, there are differences between the demographics of X users and the general population [61,62] which might affect the results of the correlation calculated in this study.

Our findings revealed a clear shift in public opinions at several key moments, often coinciding with major policy changes or public events, such as mask mandates and the Olympics, observed from 1,102,876 English tweets and 560,873 Japanese tweets from the United States and Japan between early 2020 and late 2022, covering the entire duration of the COVID-19 pandemic. Based on the correlation observed in the social media data, some variables are significantly correlated with the number of tweets, such as education levels, the population of the elderly, and household income. Some of our findings align with previous research across major social media platforms, suggesting that public opinions reflected in social media data are often interconnected. However, divisions based on education level, household income, and generational differences also indicate that social media interactions may be inherently biased, partly due to recommendation algorithms that encourage clicks and foster echo chambers.