Computer Science
Leveraging Affect Transfer Learning for Behavior Prediction in an Intelligent Tutoring System
Document Type
Conference Paper
Abstract
In this work, we propose a video-based transfer learning approach for predicting problem outcomes of students working with an intelligent tutoring system (ITS). By analyzing a student's face and gestures, our method predicts the outcome of a student answering a problem in an ITS from a video feed. Our work is motivated by the reasoning that the ability to predict such outcomes enables tutoring systems to adjust interventions, such as hints and encouragement, and to ultimately yield improved student learning. We collected a large labeled dataset of student interactions with an intelligent online math tutor consisting of 68 sessions, where 54 individual students solved 2, 749 problems. We will release this dataset publicly upon publication of this paper. It will be available at https://www.cs.bu.edu/faculty/betke/research/learning/. Working with this dataset, our transfer-learning challenge was to design a representation in the source domain of pictures obtained 'in the wild' for the task of facial expression analysis, and transferring this learned representation to the task of human behavior prediction in the domain of webcam videos of students in a classroom environment. We developed a novel facial affect representation and a user-personalized training scheme that unlocks the potential of this representation. We designed several variants of a recurrent neural network that models the temporal structure of video sequences of students solving math problems. Our final model, named ATL-BP for Affect Transfer Learning for Behavior Prediction, achieves a relative increase in mean F -score of 50 % over the state-of-the-art method on this new dataset.
Publication Title
Proceedings - 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2021
Publication Date
2021
ISBN
9781665431767
DOI
10.1109/FG52635.2021.9667001
Repository Citation
Ruiz, Nataniel; Yu, Hao; Allessio, Danielle A.; Jalal, Mona; Joshi, Ajjen; Murray, Thomas; Magee, John J.; Whitehill, Jacob R.; Ablavsky, Vitaly; Arroyo, Ivon; Woolf, Beverly P.; Sclaroff, Stan; and Betke, Margrit, "Leveraging Affect Transfer Learning for Behavior Prediction in an Intelligent Tutoring System" (2021). Computer Science. 12.
https://commons.clarku.edu/faculty_computer_sciences/12
APA Citation
Ruiz, N., Yu, H., Allessio, D. A., Jalal, M., Joshi, A., Murray, T., ... & Betke, M. (2021, December). Leveraging affect transfer learning for behavior prediction in an intelligent tutoring system. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (pp. 1-8). IEEE.