Scientific Data Report | K-emophone: A Mobile and Wearable Dataset with On-site Emotion, Stress, and Attention Labels

Background

With the proliferation of low-cost mobile and wearable sensors, numerous studies have leveraged these devices to track and analyze human mental health, productivity, and behavioral patterns. However, despite the development of datasets collected in laboratory environments, there remains a lack of datasets labeled with emotions, stress, and attention captured in real-world contexts. This limits the progress of research in the fields of Affective Computing and Human-computer Interaction.

Research Source

This research was conducted by Soowon Kang, Woohyeok Choi, Cheul Young Park, Narae Cha, Auk Kim, Ahsan Habib Khandoker, Leontios Hadjileontiadis, Heepyung Kim, Yong Jeong, and Uichin Lee. They are from the Korea Advanced Institute of Science and Technology, Upstage AI Institute, Libl, Kangwon National University, Khalifa University, and Aristotle University respectively. The study was published in the 2023 issue of the journal Scientific Data.

Research Process

Data Collection Methods

The K-emophone dataset involved a seven-day real-world multimodal data collection among 77 student participants. The entire study included three main stages: pre-survey, actual data collection, and post-survey.

Pre-survey

Before data collection, a series of preliminary questionnaires were used to obtain participants’ basic demographic information and stable personality traits. For example, the Big Five Inventory (BFI) was used to measure participants’ openness, conscientiousness, neuroticism, extraversion, and agreeableness. To adapt to the Korean environment, a translated and shortened version of the BFI, the K-BFI 15 questionnaire, was used.

Actual Data Collection

The actual data collection was based on the Experience Sampling Method (ESM). A smartphone application named Paco was used to design the questionnaire and send notifications to participants. Participants received 16 randomized interval alerts between 10:00 and 22:00, asking them to report their real-time emotions, stress, attention, and other states.

In addition to ESM data, various sensor data were collected through Android smartphones and Microsoft (MS) Band 2 smartwatches. These included environmental brightness, UV exposure, heart rate, skin temperature, physiological signals, and behavioral signals like accelerometer data, step count, and walking distance.

The dataset not only collected sensor data but also included self-reported data from participants daily, introducing a more comprehensive dimension in monitoring emotions and stress.

Post-survey

After the data collection ended, the research team conducted a post-survey using tools like the Perceived Stress Scale (PSS), Patient Health Questionnaire (PHQ), and General Health Questionnaire (GHQ) to evaluate participants’ mental health status during the data collection period.

Data Processing and Privacy Protection

The research team performed comprehensive cleaning on the collected data. A total of 5,753 questionnaire responses and 12.7GB of multimodal sensor data were received. Due to significant issues in data from certain participants, the final dataset underwent screening and privacy measures such as encryption and noise addition, resulting in 5,582 valid questionnaire responses and 11.7GB of sensor data.

Main Results

The dataset’s technical validity was verified through machine learning methods. Models were developed to predict states like emotions, stress, and task interruption, further analyzing the most important features in these models. Models built using algorithms such as Random Forest and XGBoost exceeded benchmark models in some dimensions, showing overall good predictive performance despite existing challenges.

Data Features and Technical Validation

Feature extraction was conducted on multimodal sensor data, including current sensor readings, the time since the last sensor reading change, and the distribution of readings within a specific time window. Another important feature was the participants’ potential high emotional state, which played a key role in models predicting emotions, stress, etc.

In interaction detection, various machine learning algorithms were used for training and validation, and model performance was evaluated through cross-validation. Results indicated that models using the XGBoost algorithm performed well-balanced in valence, arousal, stress, and task disturbance, showing the dataset’s potential for emotion recognition and attention management.

Conclusion and Value

The K-emophone dataset demonstrates enormous potential for expanding research in the field of Affective Computing. By collecting multimodal data in real-world environments, the study provides a valuable data source ranging from mental health to productivity. It is anticipated that this dataset will play an important role in future research on mental health, Affective Computing, attention management, and more, providing new avenues for integrating emotion detection with daily behavior analysis.

Research Highlights

1. Multimodal Data: The study fully utilized various sensor data and ESM self-report data, providing a more comprehensive view of emotions, stress, and behaviors.
2. Real-world Data: Data collection was conducted in participants’ daily lives, closely reflecting real-world scenarios, thereby increasing the external validity of the data.
3. Cross-field Applications: The dataset has broad applications across multiple fields from Affective Computing to attention management, advancing research and application development in these areas.

Usage Instructions and Limitations

Code Availability

The research developed an Android application for data collection and released data exploration and machine learning code, providing comprehensive research resources.

Potential Applications

The research team hopes that the K-emophone dataset can assist other researchers in understanding emotional and cognitive states and developing new data-driven applications, such as emotion detection and stress recognition, advancing further research in mental health and productivity.

Limitations

Due to the unavailability of the Microsoft Band 2 smartwatch in the future, future research may need to combine other sensing devices to continue similar data collection. Additionally, participants may not fully follow guidelines during data collection, which could affect data quality in some results.

Despite certain limitations, the practical application and further validation of this dataset indicate good prospects for understanding and managing emotions and attention.