Computer Science
Measuring and Integrating Facial Expressions and Head Pose as Indicators of Engagement and Affect in Tutoring Systems
Document Type
Conference Paper
Abstract
While using online learning software, students demonstrate many reactions, various levels of engagement, and emotions (e.g. confusion, boredom, excitement). Having such information automatically accessible to teachers (or digital tutors) can aid in understanding how students are progressing, and suggest who and when needs further assistance. As part of this work, we conducted two studies using computer vision techniques to measure students’ engagement and affective states from their head pose and facial expressions, as they use an online tutoring system, MathSpring.org, designed to aid students’ practice of mathematics problem-solving. We present a Head Pose Tutor, which estimates the real-time head direction of students and responds to potential disengagement, and a Facial Expression-Augmented Teacher Dashboard, that identifies students’ affective states and provides this information to teachers. We collected video data of undergraduate students interacting with MathSpring. Preliminary results on MathSpring videos were encouraging indicating accuracy in detecting head orientation. A usability study was conducted with actual teachers to start to evaluate the possible impact of the proposed Teacher Dashboard software.
Publication Title
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Publication Date
2021
Volume
12793 LNCS
First Page
219
Last Page
233
ISSN
0302-9743
ISBN
9783030778729
DOI
10.1007/978-3-030-77873-6_16
Keywords
computer vision, emotions, engagement, gaze, intelligent tutoring systems
Repository Citation
Yu, Hao; Gupta, Ankit; Lee, Will; Arroyo, Ivon; Betke, Margrit; Allesio, Danielle; Murray, Tom; Magee, John; and Woolf, Beverly P., "Measuring and Integrating Facial Expressions and Head Pose as Indicators of Engagement and Affect in Tutoring Systems" (2021). Computer Science. 13.
https://commons.clarku.edu/faculty_computer_sciences/13
APA Citation
Yu, H., Gupta, A., Lee, W., Arroyo, I., Betke, M., Allesio, D., ... & Woolf, B. P. (2021, July). Measuring and integrating facial expressions and head pose as indicators of engagement and affect in tutoring systems. In International Conference on Human-Computer Interaction (pp. 219-233). Cham: Springer International Publishing.