Academic Papers

Virtual Reality Framework for Multi-human Multi-agent Adaptive Teamwork

Nov 01, 2022

Individualized, adaptive intelligent technologies that continuously promote the emergence of team cohesion in novel groups of humans and intelligent agents are necessary to maximize collective adaptation to rapidly evolving environmental demands. Here, we describe an iterative three-phase human-centered design and evaluation framework to aid in the development of teamwork-promoting autonomous capabilities. Qualitative and quantitative feedback from each phase can be used to improve the fidelity of simulations for training, autonomy performance, and the design of user interfaces. In the initial phase, synthetic agents are used to approximate human behaviors and provide initial training of AI models. In the second phase, individual human operators are introduced to evaluate the capability in a simulated operational environment. In the third phase, networked virtual reality clients allow human teams to collaborate on tasks in an immersive, physics-based simulation environment to evaluate how the capability may complement and enhance teamwork performance. 

To demonstrate this framework, we developed a virtual reality combat simulation demonstrating a rifle-mounted fire control system with target detection and tracking algorithms. Test user and stakeholder feedback was collected and reviewed to establish requirements for an intelligent decision-making aid to fuse data from individual fire control systems and coordinate target allocation and threat prioritization. Miniature unmanned aerial vehicles were incorporated to assist target tracking. Combat simulations are used to train algorithms to detect and track threats, predict outcomes, and provide feedback to coordinate squad tactics based on individual and group factors. 

Integration of these artificial intelligence capabilities into the virtual reality environment enable human-in-the-loop evaluation of their impact on multi-human multi-agent teamwork. By providing a continuous environment for model training and human-in-the-loop evaluation in virtual reality, teamwork autonomies can be agilely developed centered around human factors to improve both performance and teammate acceptance.

  • Year: 2022
  • Category: Artificial Intelligence
  • Tag: Multi-human/multi-agent interaction, adaptive teamwork, reinforcement learning, simulation
  • Author: Joseph P. Salisbury, Ross. L. Bobb, Virgil O. Bernard, William D. Casebeer, Huberdau, David M.
  • Released: Conference: 2022 Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC)

Featured Riverside Research Author(s)

Joseph P. Salisbury

Dr. Joseph Salisbury is a neuroscientist (Ph.D., Brandeis University, 2013) and software developer whose current research focuses on human-computer interaction, human-robot interaction, and applications of large language models.

LinkedIN
Joseph P. Salisbury

Ross L. Bobb

Mr. Ross Bobb is a research scientist for Riverside Research in the Artificial Intelligence and Advanced Compute (AIAC) department. His main focus is on developing simulations to help with human factors research and synthetic data creation.

LinkedIN
Ross L. Bobb

Virgil O. Barnard

LinkedIN
Virgil O. Barnard

David M. Huberdeau

David M. Huberdeau is a research scientist at Riverside Research Institute in the Artificial Intelligence and Machine Learning Lab. He uses concepts from cognitive science and human neuroscience along with AI and ML approaches to conduct research and development in human-machine teaming. His current focus is in optimization, autonomous planning, human behavioral modeling, and reinforcement learning. Previously he completed a post-doc at Yale University in the Dept. of Psychology and a PhD in Biomedical Engineering at The Johns Hopkins University. He has experience in human behavior analysis and modeling, physiological recording and processing, and software development project management.

LinkedIN
David M. Huberdeau
Disclaimer

The above listed authors are current or former employees of Riverside Research. Authors affiliated with other institutions are listed on the full paper. It is the responsibility of the author to list material disclosures in each paper, where applicable – they are not listed here. This academic papers directory is published in accordance with federal guidance to make public and available academic research funded by the federal government.