Document Type



Doctor of Philosophy (PhD)


Computer Science

First Advisor's Name

Christine Lisetti

First Advisor's Committee Title

Associate Professor

Second Advisor's Name

S. S. Iyengar

Second Advisor's Committee Title


Third Advisor's Name

Tao Li

Third Advisor's Committee Title


Fourth Advisor's Name

Armando Barreto

Fourth Advisor's Committee Title


Fifth Advisor's Name

Ubbo Visser

Fifth Advisor's Committee Title

Associate Professor

Sixth Advisor's Name

Jeffrey F. Cohn

Sixth Advisor's Committee Title



Rapport, Modeling Non-Verbal Behaviors, Intelligent Virtual Agents, Data-Driven Behavior Modeling, Modeling Rapport, Modeling Rapport Using Machine Learning, Modeling Rapport Using Hidden Markov Models, Modeling Non-Verbal Behaviors Using Machine Learning, Modeling Non-Verbal Behaviors Using Hidden Markov Models

Date of Defense



There is a growing societal need to address the increasing prevalence of behavioral health issues, such as obesity, alcohol or drug use, and general lack of treatment adherence for a variety of health problems. The statistics, worldwide and in the USA, are daunting. Excessive alcohol use is the third leading preventable cause of death in the United States (with 79,000 deaths annually), and is responsible for a wide range of health and social problems. On the positive side though, these behavioral health issues (and associated possible diseases) can often be prevented with relatively simple lifestyle changes, such as losing weight with a diet and/or physical exercise, or learning how to reduce alcohol consumption. Medicine has therefore started to move toward finding ways of preventively promoting wellness, rather than solely treating already established illness.

Evidence-based patient-centered Brief Motivational Interviewing (BMI) interven- tions have been found particularly effective in helping people find intrinsic motivation to change problem behaviors after short counseling sessions, and to maintain healthy lifestyles over the long-term. Lack of locally available personnel well-trained in BMI, however, often limits access to successful interventions for people in need. To fill this accessibility gap, Computer-Based Interventions (CBIs) have started to emerge.

Success of the CBIs, however, critically relies on insuring engagement and retention of CBI users so that they remain motivated to use these systems and come back to use them over the long term as necessary.

Because of their text-only interfaces, current CBIs can therefore only express limited empathy and rapport, which are the most important factors of health interventions. Fortunately, in the last decade, computer science research has progressed in the design of simulated human characters with anthropomorphic communicative abilities. Virtual characters interact using humans’ innate communication modalities, such as facial expressions, body language, speech, and natural language understanding. By advancing research in Artificial Intelligence (AI), we can improve the ability of artificial agents to help us solve CBI problems.

To facilitate successful communication and social interaction between artificial agents and human partners, it is essential that aspects of human social behavior, especially empathy and rapport, be considered when designing human-computer interfaces. Hence, the goal of the present dissertation is to provide a computational model of rapport to enhance an artificial agent’s social behavior, and to provide an experimental tool for the psychological theories shaping the model. Parts of this thesis were already published in [LYL+12, AYL12, AL13, ALYR13, LAYR13, YALR13, ALY14].



abstract.tex (2 kB)
chap1.tex (16 kB)
chap2.tex (163 kB)
chap3.tex (136 kB)
chap4.tex (82 kB)
chap5.tex (155 kB)
chap6.tex (7 kB)
Dissertation.tex (3 kB)
epilogue.tex (1 kB)
library-CLL.bib (322 kB)
library.bib (4548 kB)
macros.tex (1 kB)
NERlib.bib (1196 kB)
prologue.tex (1 kB)
R21-BY-Refs.bib (754 kB)
sentimentLib.bib (11 kB)
SHB-Refs.bib (732 kB)
vita.tex (3 kB)
Dissertation.bbl (71 kB)



Rights Statement

Rights Statement

In Copyright. URI:
This Item is protected by copyright and/or related rights. You are free to use this Item in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s).