Non-Intrusive Affective Assessment in the Circumplex Model from Pupil Diameter and Facial Expression Monitoring

Authors

Sudarat Tangnimitchok, Florida International UniversityFollow

Document Type

Dissertation

Degree

Doctor of Philosophy (PhD)

Major/Program

Electrical Engineering

First Advisor's Name

Armando Barreto

First Advisor's Committee Title

Committee chair

Second Advisor's Name

Malek Adjouadi

Second Advisor's Committee Title

Committee member

Third Advisor's Name

Jean H. Andrian

Third Advisor's Committee Title

Committee member

Fourth Advisor's Name

Ruogu Fang

Fourth Advisor's Committee Title

Committee member

Keywords

Digital Signal Processing, Machine Learning, Affective Computing, Human Computer Interaction, Artificial Intelligence

Date of Defense

6-26-2019

Abstract

Automatic methods for affective assessment seek to enable computer systems to recognize the affective state of their users. This dissertation proposes a system that uses non-intrusive measurements of the user’s pupil diameter and facial expression to characterize his /her affective state in the Circumplex Model of Affect. This affective characterization is achieved by estimating the affective arousal and valence of the user’s affective state.

In the proposed system the pupil diameter signal is obtained from a desktop eye gaze tracker, while the face expression components, called Facial Animation Parameters (FAPs) are obtained from a Microsoft Kinect module, which also captures the face surface as a cloud of points. Both types of data are recorded 10 times per second. This dissertation implemented pre-processing methods and fixture extraction approaches that yield a reduced number of features representative of discrete 10-second recordings, to estimate the level of affective arousal and the type of affective valence experienced by the user in those intervals.

The dissertation uses a machine learning approach, specifically Support Vector Machines (SVMs), to act as a model that will yield estimations of valence and arousal from the features derived from the data recorded.

Pupil diameter and facial expression recordings were collected from 50 subjects who volunteered to participate in an FIU IRB-approved experiment to capture their reactions to the presentation of 70 pictures from the International Affective Picture System (IAPS) database, which have been used in large calibration studies and therefore have associated arousal and valence mean values. Additionally, each of the 50 volunteers in the data collection experiment provided their own subjective assessment of the levels of arousal and valence elicited in him / her by each picture. This process resulted in a set of face and pupil data records, along with the expected reaction levels of arousal and valence, i.e., the “labels”, for the data used to train and test the SVM classifiers.

The trained SVM classifiers achieved 75% accuracy for valence estimation and 92% accuracy in arousal estimation, confirming the initial viability of non-intrusive affective assessment systems based on pupil diameter and face expression monitoring.

Identifier

FIDC007810

Recommended Citation

Tangnimitchok, Sudarat, "Non-Intrusive Affective Assessment in the Circumplex Model from Pupil Diameter and Facial Expression Monitoring" (2019). FIU Electronic Theses and Dissertations. 4224.
https://digitalcommons.fiu.edu/etd/4224