ROB 340 | UMich

ROB 340: Human Evaluation of Robots

Instructors: Patricia Alves-Oliveira, Xiaoxiao Du
GSI: Miranda Mittleman
IA: Margaret Kempe
This course was designed and developed by Patricia Alves-Oliveira.
Robotics Department
University of Michigan Ann Arbor
Office: 2252 FRB

This course is running in Winter 2025

Lectures

Monday & Wednesday 1:30 - 3:00 PM GGBL 1025

Office Hours

Wednesday 10:30 - 11:30 AM 4150 FMCRB

Lab

Thursday 3:30 - 5:30 PM 4150 FMCRB

SCHEDULE

Lectures

01/08:

Class 1: Introduction

01/13:

Class 2: Evaluation Protocol

01/15:

Class 3: Evaluation Protocol cont.

01/22:

Class 4: Gaze behaviors

01/27:

Class 5: Emotions

01/29:

Class 6: Wizard of Oz & Peripheral interactions

02/03:

Class 7: Verbal interactions

02/05:

Class 8: Guest Lecture: Sarah

02/10:

Class 9: Guest Lecture: Mike

02/17:

Class 10: Ethics in HRI

02/19:

Class 11: Types of Studies

02/24:

Class 12: Guest Lecture: Sindhu

02/26:

Class 13: Guest Lecture: Michal

03/10:

Class 14: Guest Lecture: Hannah

03/12:

Class 15: Guest Lecture: Mafalda

03/17:

Class 16: Guest Lecture: Filipa

03/19:

Class 17: Case Study & Behavior observation

03/24:

Class 18: Project Feedback

03/26:

Class 19: Project Feedback

03/31:

Class 20: Crowdsourcing data in Machine Learning

04/02:

Class 21: Collaboration and Learning

04/07:

Class 22: Mixed methods studies

04/09:

Class 23: Guest Lecture: Yasmin

04/14:

Class 24: Project Work

04/16:

Class 25: Project Work & Surveys & Statistics

Labs

01/09:

Lab 1: Basics of Human-Robot Interaction

01/16:

Lab 2: Emotive Capabilities Using a Robotic Interface

01/23:

Lab 3: Group Dynamics

01/30:

Lab 4, Wk1: Conversational Gaze

02/06:

Lab 4, Wk2: Conversational Gaze

02/13:

Lab 5, Wk1: LLMs in Robotics

02/20:

Lab 5, Wk2: LLMs in Robotics

02/27:

Lab 6: Algorithm vs. User Experience

03/13:

Lab 7: Project Work

03/20:

Lab 8: Project Work

03/27:

Lab 9: Project Work

04/03:

Lab 10: Project Work

04/10:

Lab 11: Project Work

04/17:

Lab 12: Project Presentations

Exams

02/12:

Mid-Term

04/21:

Final

GRADING

Final Group Project - 30%

Students will engage in group projects aimed at implementing an interactive and creative Robot Design to gain hands-on practical experience.

Project teams will be graded based on consistent participation throughout the semester, deliverables, and technical content.

Lab Projects (Group/Individual) - 30%

Students will complete 6 total labs, intended to expand upon lecture content and allow for hands-on experience with a variety of robot designs.

Students will be graded both individually based on submissions of lab documents and as a group depending on the activity.

Exams - 25%

Students will be evaluated on their understanding of course content through a midterm and final exam.

Class Involvement - 15%

In-class activities & surveys

Reflections on guest lectures

Attentiveness and interest

STUDENTS

Undergraduates
Shanjida Akhtar
Audrey Armstrong
Cara Blashill
Donovan Cece
Eric Chen
Mason Cox
Lily Feldman
Matthew Greenbaum
Madel Humbert
Sarah Jamil
Saipranav Janyavula
Ishrat Khan
Skye Krzykwa
Alexis Lee
Brendan Ohr
Shirley O'Mara
Matthew Prince
Michael Robinson
Pranav Vaya
Carol Wang
Rachel Weldy
Jess Wu
Meiyi Yang
Jichao Zhang

INVITED TALKS

Sarah Sebo
Group Dynamics
University of Chicago

Mike Forst
Character Design
Freelancer Sounds and HRI Designer

Sindhu Kutty
Fairness & Bias in Machine Learning
University of Michigan

Michal Luria
Policies in AI Agents
Center for Democracy & Technology

Hannah Pelikan
Conversational Analysis
Linköping University

Mafalda Gamboa
Ethnography & autoethnography
Chalmers University of Technology

Filipa Correia
Multi-agents evaluation
Interactive Technologies Institute

Yasmin Keats
Inclusive Design
Open Style Lab

GROUP PROJECTS

A-iHop

Figuring It Out

Lantern

Rogue 4

Tennis Club

The Ford-150

Course Description:

This course covers the fundamentals of how robots can be evaluated by human users to better understand the success of robotic systems. Topics include metrics and analysis for human studies of robots, measuring human performance in the context of robotic systems, and exploring how robot behaviors such as gaze, movement, etc, are perceived by humans. By the end of this class, you will know the types of metrics, studies, and analyses you can conduct to understand the performance of a robot system informed by human feedback.

Learning Objectives:

Evaluate Human-Robot Interaction Systems.
Learn to work in teams.
Critique studies in user Human-Robot Interaction.
Select the appropriate study method for a human-robot interaction study.
Recognize differences between formative and summative studies.
Reflect on the difference between system metrics and user metrics (when metrics collide).
Create a research plan to evaluate robot systems
Compare and contrast method in user studies
Understand the connection between theory, design, implementation, and evaluation of a robot system

Group Project:

The group project is a hands-on research project whose research topic and protocol are defined by the students and culminates in a written report and an oral group presentation. Students are expected to choose from the below puzzle piece to create a research study. Students submit a 5-6 page report and give an oral presentation covering their project at the end of the term.

Labs

Lab 1: Basics of Human-Robot Interaction

The goal of this lab is to create an interaction using the given robotic arm embodiment and explore the role of human integration in robot task execution.

Lab 2: Emotive Capabilities using a Robotic Interface

The goal of this lab is to communicate an emotion using only an RC car by creating an emotive routine to understand aspects of expressive robotic behavior.