Course image
SCHEDULE
Lectures

01/08:

Class 1: Introduction

01/13:

Class 2: Evaluation Protocol

01/15:

Class 3: Evaluation Protocol cont.

01/22:

Class 4: Gaze behaviors

01/27:

Class 5: Emotions

01/29:

Class 6: Wizard of Oz & Peripheral interactions

02/03:

Class 7: Verbal interactions

02/05:

Class 8: Guest Lecture: Sarah

02/10:

Class 9: Guest Lecture: Mike

02/17:

Class 10: Ethics in HRI

02/19:

Class 11: Types of Studies

02/24:

Class 12: Guest Lecture: Sindhu

02/26:

Class 13: Guest Lecture: Michal

03/10:

Class 14: Guest Lecture: Hannah

03/12:

Class 15: Guest Lecture: Mafalda

03/17:

Class 16: Guest Lecture: Filipa

03/19:

Class 17: Case Study & Behavior observation

03/24:

Class 18: Project Feedback

03/26:

Class 19: Project Feedback

03/31:

Class 20: Crowdsourcing data in Machine Learning

04/02:

Class 21: Collaboration and Learning

04/07:

Class 22: Mixed methods studies

04/09:

Class 23: Guest Lecture: Yasmin

04/14:

Class 24: Project Work

04/16:

Class 25: Project Work & Surveys & Statistics

Labs

01/09:

Lab 1: Basics of Human-Robot Interaction

01/16:

Lab 2: Emotive Capabilities Using a Robotic Interface

01/23:

Lab 3: Group Dynamics

01/30:

Lab 4, Wk1: Conversational Gaze

02/06:

Lab 4, Wk2: Conversational Gaze

02/13:

Lab 5, Wk1: LLMs in Robotics

02/20:

Lab 5, Wk2: LLMs in Robotics

02/27:

Lab 6: Algorithm vs. User Experience

03/13:

Lab 7: Project Work

03/20:

Lab 8: Project Work

03/27:

Lab 9: Project Work

04/03:

Lab 10: Project Work

04/10:

Lab 11: Project Work

04/17:

Lab 12: Project Presentations

Exams

02/12:

Mid-Term

04/21:

Final

GRADING

Final Group Project - 30%

Students will engage in group projects aimed at implementing an interactive and creative Robot Design to gain hands-on practical experience.

Project teams will be graded based on consistent participation throughout the semester, deliverables, and technical content.

Lab Projects (Group/Individual) - 30%

Students will complete 6 total labs, intended to expand upon lecture content and allow for hands-on experience with a variety of robot designs.

Students will be graded both individually based on submissions of lab documents and as a group depending on the activity.

Exams - 25%

Students will be evaluated on their understanding of course content through a midterm and final exam.

Class Involvement - 15%

In-class activities & surveys

Reflections on guest lectures

Attentiveness and interest

STUDENTS

Undergraduates
Shanjida Akhtar
Audrey Armstrong
Cara Blashill
Donovan Cece
Eric Chen
Mason Cox
Lily Feldman
Matthew Greenbaum
Madel Humbert
Sarah Jamil
Saipranav Janyavula
Ishrat Khan
Skye Krzykwa
Alexis Lee
Brendan Ohr
Shirley O'Mara
Matthew Prince
Michael Robinson
Pranav Vaya
Carol Wang
Rachel Weldy
Jess Wu
Meiyi Yang
Jichao Zhang

INVITED TALKS

Sarah Sebo
Group Dynamics
University of Chicago

Mike Forst
Character Design
Freelancer Sounds and HRI Designer

Sindhu Kutty
Fairness & Bias in Machine Learning
University of Michigan

Michal Luria
Policies in AI Agents
Center for Democracy & Technology

Hannah Pelikan
Conversational Analysis
Linköping University

Mafalda Gamboa
Ethnography & autoethnography
Chalmers University of Technology

Filipa Correia
Multi-agents evaluation
Interactive Technologies Institute

Yasmin Keats
Inclusive Design
Open Style Lab

GROUP PROJECTS
ROB 340: Human Evaluations of Robot Systems
Instructors: Patricia Alves-Oliveira, Xiaoxiao Du
GSI: Miranda Mittleman
IA: Margaret Kempe

Robotics Department
University of Michigan Ann Arbor
Office: 2252 FRB

This course is running in Winter 2025

Lectures

Monday & Wednesday 1:30 - 3:00 PM GGBL 1025

Office Hours

Wednesday 10:30 - 11:30 AM 4150 FMCRB

Lab

Thursday 3:30 - 5:30 PM 4150 FMCRB

Course Description:
This course covers the fundamentals of how robots can be evaluated by human users to better understand the success of robotic systems. Topics include metrics and analysis for human studies of robots, measuring human performance in the context of robotic systems, and exploring how robot behaviors such as gaze, movement, etc, are perceived by humans. By the end of this class, you will know the types of metrics, studies, and analyses you can conduct to understand the performance of a robot system informed by human feedback.
Learning Objectives:
  • Evaluate Human-Robot Interaction Systems.
  • Learn to work in teams.
  • Critique studies in user Human-Robot Interaction.
  • Select the appropriate study method for a human-robot interaction study.
  • Recognize differences between formative and summative studies.
  • Reflect on the difference between system metrics and user metrics (when metrics collide).
  • Create a research plan to evaluate robot systems
  • Compare and contrast method in user studies
  • Understand the connection between theory, design, implementation, and evaluation of a robot system
Group Project:
The group project is a hands-on research project whose research topic and protocol are defined by the students and culminates in a written report and an oral group presentation. Students are expected to choose from the below puzzle piece to create a research study. Students submit a 5-6 page report and give an oral presentation covering their project at the end of the term.
Puzzle Piece:
Labs:
Lab 1
Lab 1: Basics of Human-Robot Interaction

The goal of this lab is to create an interaction using the given robotic arm embodiment and explore the role of human integration in robot task execution.

Lab 2
Lab 2: Emotive Capabilities using a Robotic Interface

The goal of this lab is to communicate an emotion using only an RC car by creating an emotive routine to understand aspects of expressive robotic behavior.

Lab 3
Lab 3: Group dynamics with a Robot

The goal of this lab is to understand the nuances present in group dynamics and how a robot may influence them through a role-play activity.

Lab 4
Lab 4: Conversational Robotic Gaze

The goal of this lab is to evaluate implemented conversational gaze behaviors in a robot and explore how the differences in human perception.

Lab 5
Lab 5: LLMs in Robotics

The goal of this lab is to analyze the pros and cons of LLMs when used in robotics and test their limitations.

Lab 6
Lab 6: Algorithm performance versus user experience

The goal of this lab is to understand the trade-offs between the user perception of robot performance and algorithmic optimal performance.

Class Lecture Summary

CLASS 1
Lecture: Introduction for the Course
This class was an introduction to Human Evaluations of Robot Systems
 SLIDES
Readings:  GenAI
CLASS 2
Lecture: Evaluation Protocol
Evaluation protocol for experimental studies in human-robot interaction (part 1).
 SLIDES
CLASS 3
Lecture: Evaluation Protocol cont. & Emotions
Evaluation protocol for experimental studies in human-robot interaction (part 2).
 SLIDES
CLASS 4
Lecture: Non-verbal behaviors
The role of non-verbal behaviors, such as gaze, in human-robot interaction.
 SLIDES
CLASS 5
Lecture: Emotions
History of the study of emotions and their role during interactions with robots.
 SLIDES
CLASS 6
Lecture: Wizard of Oz & Peripheral interactions
Exploring the creation of alternative interactions with robots.
 SLIDES
Readings:  Dieter Rams
CLASS 7
Lecture: Verbal interactions
The role of verbal interactions, inclusing trust, self-disclosure, and components of speech towards robots.
 SLIDES
CLASS 8
Guest Lecture: Sarah Sebo
 SLIDES
Readings:  Sarah's website
CLASS 9
Guest Lecture: Mike Forst
 SLIDES
Readings:  Mike's website
CLASS 10
Lecture: Ethics in HRI
 SLIDES
CLASS 12
Guest Lecture: Sindhu Kutty
 SLIDES
CLASS 13
Guest Lecture: Michal Luria
 SLIDES
CLASS 14
Guest Lecture: Hannah Pelikan
 SLIDES
Readings:  Hannah's Website
CLASS 16
Guest Lecture: Filipa Correia
 SLIDES
Readings:  Filipa's Website
CLASS 17
Lecture: Case Studies & Behavioral Analysis
 SLIDES
CLASS 18
Group Project Feedback
 SLIDES
CLASS 19
Group Project Feedback
 SLIDES
CLASS 20
Lecture: Crowdsourcing data in Machine Learning
 SLIDES
CLASS 21
Lecture: Collaboration and Learning
 SLIDES
CLASS 23
Guest Lecture: Yasmin Keats
 SLIDES
CLASS 24
Group Project Work
 SLIDES
CLASS 25
Surveys & Statistics / Project Work
 SLIDES