Generative Facial Expressions and Eye Gaze Behavior from Prompts for Multi-Human-Robot Interaction
Nonverbal cues such as eye gaze and facial expressions play critical roles in conveying intent, regulating conversation, and fostering engagement. A robot's ability to effectively deploy these behaviors can significantly enhance human-robot collaboration. We describe a simple zero-shot learning approach to generate facial expression and gaze shifting behaviors to control a social robot conversing with an individual or group. An initial prompt provides instructions to a pre-trained large language model on how the model can control a robot's facial expression and eye gaze behaviors during a conversation. To demonstrate this, we describe a proof-of-concept implementation using the robot Furhat. This simple and easily customizable approach can be used to improve perception of a robot's social presence in multi-human-robot interactions.
- Year: 2023
- Category: Artificial Intelligence
- Tag: Social robotics, gesture synthesis, gaze alternation, multi-humanrobot interaction, pre-trained language models, nonverbal communication, prompt engineering
- Author: Safarty, Virgil O. Barnard, Joseph P. Salisbury
- Released: UIST '23 Adjunct: Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
Featured Riverside Research Author(s)
Joseph P. Salisbury
Dr. Joseph Salisbury is a neuroscientist (Ph.D., Brandeis University, 2013) and software developer whose current research focuses on human-computer interaction, human-robot interaction, and applications of large language models.
LinkedIN
Virgil O. Barnard
Virgil O. Barnard is a Senior Machine Learning Scientist in the Artificial Intelligence and Machine Learning group at Riverside Research. He received his bachelor’s in mathematics from University of Kentucky in 2012. He received his Ph.D. (ABD) in computer science from University of Kentucky in 2019. He has performed as a technical AI lead on many contracts for customers in the DoD & IC while at Riverside Research since 2019 spanning image, electro-optic, signals, and radar-based modalities. He is currently researching agent-based reasoning over knowledge graphs.
LinkedIN
The above listed authors are current or former employees of Riverside Research. Authors affiliated with other institutions are listed on the full paper. It is the responsibility of the author to list material disclosures in each paper, where applicable – they are not listed here. This academic papers directory is published in accordance with federal guidance to make public and available academic research funded by the federal government.