Computer Science

Measuring and Integrating Facial Expressions and Head Pose as Indicators of Engagement and Affect in Tutoring Systems

Document Type

Conference Paper


While using online learning software, students demonstrate many reactions, various levels of engagement, and emotions (e.g. confusion, boredom, excitement). Having such information automatically accessible to teachers (or digital tutors) can aid in understanding how students are progressing, and suggest who and when needs further assistance. As part of this work, we conducted two studies using computer vision techniques to measure students’ engagement and affective states from their head pose and facial expressions, as they use an online tutoring system,, designed to aid students’ practice of mathematics problem-solving. We present a Head Pose Tutor, which estimates the real-time head direction of students and responds to potential disengagement, and a Facial Expression-Augmented Teacher Dashboard, that identifies students’ affective states and provides this information to teachers. We collected video data of undergraduate students interacting with MathSpring. Preliminary results on MathSpring videos were encouraging indicating accuracy in detecting head orientation. A usability study was conducted with actual teachers to start to evaluate the possible impact of the proposed Teacher Dashboard software.

Publication Title

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Publication Date



12793 LNCS

First Page


Last Page









computer vision, emotions, engagement, gaze, intelligent tutoring systems

APA Citation

Yu, H., Gupta, A., Lee, W., Arroyo, I., Betke, M., Allesio, D., ... & Woolf, B. P. (2021, July). Measuring and integrating facial expressions and head pose as indicators of engagement and affect in tutoring systems. In International Conference on Human-Computer Interaction (pp. 219-233). Cham: Springer International Publishing.