The study was approved as “Exemption from full A*STAR IRB Review” (institutional review board reference number 2020 − 258) for using social media data obtained from approved Twitter APIs and existing anonymous public data approved by data providers to study research topics surrounding the COVID-19 outbreak.
To examine the research questions, we collected daily-level time-series data from multiple public sources concerning Singapore’s heightening and stabilising phases, covering 18 months from July 2020 to December 2021.
Data and indicators selection criteria
As the study’s purpose is to examine new data and tools that can be used as predictors to forecast downstream mental healthcare needs, the first and primary criterion is that the data and indicator extraction methods need to be accessible without any substantially high cost or barrier to collect.
Second, the data should be available continuously for time-series statistical analysis. This means that traditional survey data, collected every several months, are not usable for forecasting daily mental healthcare needs.
Third, the data and tools should have demonstrable validity, such as those shown by prior studies or their use in adjacent problem domains.
Situation data and indicators
For situation indicators, we used data sources published by the health authorities as they are routinely collected and reported to the public as primary indicators of the severity of the COVID-19 situation. The indicators include COVID-19 cases and COVID-19 deaths in daily new cases and accumulated case counts from WHO23. In addition, we collected the daily number of government announcements from MOH24 to serve as a proxy for the intensity of government actions to manage the COVID-19 situation. These indicators are used as comparative predictors for our study.
Social media data, data pre-processing and emotion indicators
For social media sources, we used Twitter as it provides public access to the post’s text content, user screen name, timestamp, and other relevant information for academic use via an application programming interface (API)25. We performed a keyword search and obtained tweets that contained at least one of the COVID-19-related keywords: “ncov”, “corona” and “covid”. For this study, we used the Singapore-based tweets dataset, which was selected based on the location disclosed by the tweet authors’ public profile. The details of the Twitter dataset can be found in Gupta et al.26.
To extract effective early indicators from the highly noisy social media data sources, we applied the following data processing and variable extraction steps: (i) clean the data, (ii) perform emotion classification and intensity measurement, (iii) prepare the final study data in daily aggregated forms for statistical analyses, and (iv) pre-process the aggregated data. We first cleaned the social media data by applying troll removal, that is, removing duplicated posts, posts considered as click baits or financial investment ads (e.g. comments with “bitcoin”, “click here”, “inbox me”), email addresses, and 1-character-only posts. We also removed tweets posted by influencers (news agencies, political leaders, etc.) by using the “followers: following” ratio27, where a ratio of more than 1 would be deemed as an influencer. By removing such tweets, we aim to obtain data that more accurately reflects general public emotions. The final Twitter data used for this study, consisting of 140,598 tweets, were obtained after removing 2,335 potential trolls and 234,830 tweets from possible influencers.
For emotion indicators, we used CrystalFeel28 as the API that allows the systematic processing of large-scale data for academic use. CrystalFeel is a multidimensional emotion analysis software package using Support Vector Machine (SVM)-trained algorithms. It can classify the emotion (joy, anger, fear, and sadness) and measure the intensity of the emotion in a given text, such as a tweet or Facebook comment28,29.
Most sentiment analysis algorithms consider sentiments and emotions in a categorical sense, which typically attempts to assign a tweet into positive, negative, or neutral classes or a happy vs. not happy, sad vs. no sad category. CrystalFeel’s key feature is the ability to quantify a tweet’s intensity level over four primary emotions – fear, anger, sadness, and joy – on a continuous scale of 0–1 (e.g., 0 indicates the absence of fear being expressed; 1 indicates an extremely high intensity of fear being expressed in the tweet). It also automatically classifies each tweet into one of the Fear, Anger, Sadness, Joy, or No Specific Emotion categories.
The CrystalFeel algorithms’ emotion intensity measurement accuracy has been validated in a SemEval-18 task30, where it achieved high Pearson correlation coefficients when evaluated against human-labelled emotion intensity scores: 0.816 (overall emotion or sentiment valence intensity), 0.708 (joy intensity), 0.740 (anger intensity), 0.700 (fear intensity) and 0.720 (sadness intensity)29. Some of CrystalFeel’s features have been found useful in other COVID-19 social media sentiment analysis studies31,32,33,34. Its predictive validity has been examined and tested to be useful in other natural language processing (NLP) tasks, including predicting the agency and social ingredients of happy moments35, predicting popular news on Facebook and Twitter36, detecting propaganda techniques in tweets37, predicting video-level multimodal emotions from YouTube videos38 and predicting user-level past vs. future temporal orientation39.
In this study, we used CrystalFeel’s full range of emotion analysis features, which cover the classification of the emotion class and the intensity value associated with the emotions of fear, anger, joy, and sadness, with examples of the emotion analysis results of tweets indicated (Table 4).
Table 4 Examples of social media data emotion classification and emotion intensity measurement (n_Twitter = 140,598).
Mental healthcare needs measures – IMH visits and Mindline Crisis
For mental health needs indicators, we used behavioural data that indicate how the public develops online and offline mental health needs over time. Data were taken from official governmental mental health needs services, from (1) psychiatric hospital emergency room visits and (2) online self-help portal.
The need for mental healthcare needs is reflected in the daily count of visitors to the emergency room from the Institute of Mental Health (IMH)40, the country’s primary psychiatric care hospital, which we used as the main proxy of the public need to approach psychiatric services for their mental health concerns.
Data from Mindline41 was used to indicate mental health needs expressed online. Mindline is an online mental health help portal created by the Ministry of Health Office for Healthcare Transformation (MOHT), a government-created entity. Mindline was set up in June 2020 amid COVID-19, when the population’s mental health could be adversely affected due to uncertainties, unemployment, and isolation. A primary feature of Mindline is to present the users with a mental health status questionnaire consisting of 16 items from standardised mental health screening instruments that are clinically validated to be suitable for self-administration, namely the 9-item Patient Health Questionnaire (PHQ-9)42 and the 7-item General Anxiety Disorder test (GAD-7)43.
In addition, respondents were asked to indicate the frequency at which they experienced each item (e.g., “Not being able to stop or control worrying”) on a 0–3 scale (0-Not at all, 3-Nearly every day) in response to the question “Over the last 2 weeks, how often have you been bothered by the following problems?”.
The scores of the responses were then summed up and categorised into four mental distress severity levels: ‘Well’, ‘Mild’, ‘Moderate’, or ‘Crisis’ based on a protocol term (Table 5). Different resources were recommended to the user according to the levels. For example, people with “Well” status were recommended to websites that can help them maintain their well-being, and people with “Crisis” status were asked to seek immediate help from a list of 24-hour hotlines. More details on Mindline’s development can be found in Weng et al.44.
Table 5 The protocol terms of Mindline severity levels and the correspondence with PHQ-9 and GAD-7.
For this study, we focus on analysing the user visit trends with self-assessment results in a “Crisis” situation, which reflects the most severe mental health status. Further, to focus on organic visits to Mindline, we removed the user visits led by marketing campaigns from the data.
Summary of all variables used in this study
A definition of all the study variables and the manner in which they were obtained is given below (Table 6).
Table 6 Data dictionary – summary of all variables used in this study. The variables or indicators are listed within each category from the most coarse-grained to the most fine-grained.
Statistical analysis – pre-processing aggregated data
Before carrying out the statistical analysis, we pre-processed all the daily aggregated data by normalising the data with a Z-score, using “first difference” to ensure stationarity, removing volatility by dividing by monthly standard deviation, and removing seasonality by subtracting monthly means as a prerequisite to analysing the time-series data. The Augmented Dickey-Fuller Test was used to verify that the variables were stationary.
Granger causality tests
The Granger causality test was used to investigate the dynamic relations between situational and Twitter indicators, as well as mental healthcare needs data. We used Python 3.6.10, including the following packages: Pandas 1.0.1, Numpy 1.19.5, Statsmodels 0.10.0rc2, and NLTK 3.4.5 on Anaconda 3-2019.10, for our analysis.
Granger causality estimates the causal effects of one time-series variable on another time-series variable after controlling for lagged values. It determines whether lagged values of and predicts better than lagged values of alone45,46. For instance, in investigating the relationship between the “Fear Count” (\(\:{x}_{t}\)), and “IMH visits” (\(\:{y}_{t}\)), is modelled as:
$$\:{y}_{t}=\:{\alpha\:}_{0}+\:{\sum\:}_{l\:=\:1}^{L}{\alpha\:}_{l}{y}_{t-l\:}+\:{\sum\:}_{l\:=\:1}^{L}{\beta\:}_{l\:}{x}_{t-l\:}+\:{\epsilon\:}_{t}$$
where L refers to the total number of lagged values, \(\:{\alpha\:}_{1}\) are the regression weights on \(\:{y}_{t-l}\), \(\:{\beta\:}_{l}\) is the regression weights on \(\:{x}_{t-l\:}\)and \(\:{\epsilon\:}_{t}\:\)is the time-variant residuals.
We used likelihood-ratio tests to determine the optimal lag length for each pair of variables. The series (xt) is considered Granger-cause series (yt) if the P-value is 0.05 or less. This study reports the earliest significant lag days. After a review of the P-value results, we found that increasing the number of lag days beyond five days did not improve the results. Other Granger causality studies have also tested lag days of up to 5 days47 or up to 7 days48,49. As such, we chose five days as the maximum lag days for the Granger causality tests.
However, it is useful to note that despite its name, the Granger-causality test does not test for true causality. One limitation of employing Granger Causality is that it is a bivariate analysis and does not factor in other predictors’ effects simultaneously. Further research should use a multivariate model to examine the combined effects of multiple predictors on the outcome variable.
ARIMA forecasting
We evaluated the performance of auto-regressive integrated moving average (ARIMA) models in forecasting based on the significant variables revealed from the Granger-causality results. We compared each ARIMA model with and without the additional lagged values of χ variable in predicting y variable. We performed a 95:5 split, where the first 95% of the data (n = 522) was used for training, whilst the remaining 5% (n = 27) was used for testing.
In an ARIMA model, the y variable is forecast using lagged values of the variable itself, as seen in Eq. (1) below. In our analysis, we evaluated the performance of this baseline model with other models incorporating additional x variables. The model would then comprise lagged values of the y variable and lagged values with the additional x variable, as seen in Eq. (2) below50.
(1)
Without additional x variable: \(\:{y}_{t}=\:{\alpha\:}_{0}+\:{\sum\:}_{l\:=\:1}^{L}{\alpha\:}_{l}{y}_{t-l\:}+\:{\epsilon\:}_{t}\)
(2)
With additional x variable: \(\:{y}_{t}=\:{\alpha\:}_{0}+\:{\sum\:}_{l\:=\:1}^{L}{\alpha\:}_{l}{y}_{t-l\:}+\:{\sum\:}_{l\:=\:1}^{L}{\beta\:}_{l\:}{x}_{t-l\:}+\:{\epsilon\:}_{t}\)
In the analysis, we used two error metrics: Root Mean Square Error (RMSE) and Mean Absolute Error (MAE)51. RMSE is a commonly used scale-dependent error measure that shows prediction error from the observed values using Euclidean distance52 (C3.ai, 2024). MAE is a commonly used forecast error measure which refers to the summation of absolute errors between each forecast value and real value, divided by the number of errors.
A lower error value for both metrics indicates that the model is performing better. We focus on RMSE as it helps show the observed value and illustrate the potential practical value of the new ARIMA forecasting model. The ARIMA models and the error rates were computed in R programming language using the fpp3 package53.
