Skip to content

USArmyResearchLab/ARL-SCOUT

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ARL SCOUT: The Situated Corpus of Understanding Transactions

The Situated Corpus Of Understanding Transactions (SCOUT) is a multi-modal collection of human-robot dialogue in the task domain of collaborative exploration. The corpus was constructed from multi-phased Wizard-of-Oz experiments where human participants gave verbal instructions to a remotely-located robot to move and gather information about its surroundings. Each dialogue involved a human Commander, a Dialogue Manager (DM), and a Robot Navigator (RN), and took place in physical or simulated environments.

SCOUT contains Transcripts of human-robot dialogues, aligned with the Images taken by the robot at the request of the Commander. The transcripts have been annotated with Dialogue Structure, including Transcaction Units (TUs), Antecedents, and Relations. A subset of utterances have been annotated with Dialogue Semantics including Standard-AMR and Dialogue-AMR. A subset of the Maps have been curated that depict the LIDAR Maps at the end of the dialogue and have been annotated as Exploration Maps.

Contents

Experimental Design

Our human-robot collaboration exercise is an exploration-and-search task in which human participants (who assume the role of 'Commander') work with their remotely-located robot teammate to explore an unknown area under low-bandwidth conditions--a situation representative of various inhospitable remote environments. This constraint prevents live-video streaming of the robot's perspective to the human or remote teleoperation of the robot.

In the first three experiments, robot control was conducted through “Wizard-of-Oz” (WoZ) methodology. One Wizard handled the robot's dialogue understanding and processing capabilities as the Dialogue Manager (DM) Wizard, and the other handled the robot's movement as the Robot Navigator (RN) Wizard.

The DM-Wizard listened to the participant's speech and generated text-based responses and resolved miscommunication in what is called the left conversational floor. Once the DM-Wizard deemed a participant's instruction was well-formed, the DM-Wizard translated the instruction into a more rigid natural language instruction and passed it to the RN-Wizard to execute. This took place in the right conversational floor, where only the DM-Wizard and RN-Wizard communicated. The RN-Wizard informed the DM-Wizard through spoken language to indicate completion or complication in executing an instruction; the DM-Wizard in turn informed the participant of this completion or complication via text.

The roles of dialogue manager and robot navigator changed over the course of the experimentation, from Wizard-of-Oz confederates to fully automated systems building upon the prior experimental findings.

Experiment 1 Experiment 2 Experiment 3 Experiment 4
Dialogue Processing wizard + keyboard wizard + button GUI wizard + button GUI ASR + auto-DM
Robot Behaviors wizard + joystick wizard + joystick wizard + joystick wizard + joystick
Robot & Environment physical physical virtual virtual

Commanders were given three resources: a robot, a map, and photographs. The robot, a Clearpath Jackal, was located in a remote environment, and it could understand spoken, natural language instructions from the participant (via the DM and RN). The verbal instructions were issued to the robot through a push-to-talk interface, and the robot responded through text messages. The map was a 2D, birds-eye view LIDAR map of the environment, which dynamically updated as the robot moved through the space. The photos were taken from the robot's front-facing RGB camera, showing the view directly in front of it. Below is a screenshot of the view seen by the human Commander.

SCOUT Contents

Unique Commanders (human participants) are named as p{exp}.{ID} where {exp} is the experiment number (1-4) and {ID} is the Commander's ID assigned within that experiment. For example, the Commander p1.01 is the first Commander in Experiment 1. Each Commander completed three trials {train, main1, main2}. The data/main_trial_distribution.xlsx file notes which counting task was completed in each main trial.

The data and annotation files are divided by type over five directories with the data/ directory of this repository:

Clicking on each of the links above will redirect you to a detailed overview of the data nomenclature, structure, and format.

Citation

Please cite the following paper to reference the SCOUT data release:

@inproceedings{lukin2024scout,
    title = {{SCOUT: A Situated and Multi-Modal Human-Robot Dialogue Corpus}},
    booktitle = {{The Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)}},
    author = {Lukin, Stephanie M., and Bonial, Claire N. and Marge, Matthew and Hudson, Taylor and Hayes, Cory J. and Pollard, Kimberly A. and Baker, Anthony and Foots, Ashley and Artstein, Ron and Gervits, Felix and Abrams, Mitchell and Henry, Cassidy and Donatelli, Lucia and Leuski, Anton and Hill, Susan G. and Traum, David and Voss, Clare R.},
    year = {2024}
}

Publications Using SCOUT

SCOUT has been used in a number of publications and research areas.

Dialogue Structure

Abstract Meaning Representation

Participant Analysis

Demonstrations

Experimental Design

License

ARL SCOUT is licensed under the Creative Commons Zero 1.0 Universal (CC0 1.0) license. Please see LICENSE for details.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published