Course Overview

This graduate course gives a broad overview of various models for combining human and machine intelligence to solve computational problems. Through weekly seminars and a class project, we will examine three roles that humans play in computational systems -- humans as computers, humans as teachers, and humans as collaborators. This research-focused course covers literature from a variety of research areas, and includes topics such as human computation and crowdsourcing, learning by demonstration, mixed initiative systems, active learning from human teachers, interactive machine learning, human-in-the-loop security, etc.


During a typical seminar session, we will discuss three papers. Each paper will be assigned a discussant. The discussant is responsible for (a) writing a blog post by Tuesdays 12pm noon discussing the key ideas behind the paper, and what the findings teach us about designing effective human-in-the-loop systems, and (b) giving a short presentation based on the blog post (10 minutes) and leading a discussion (20 minutes). The blog post and presentation should contain answers to the following questions: Students who are not acting as discussants must read all the assigned papers before coming to class. In addition, for each paper, students must enter by Tuesdays 12pm noon a blog post "comment" describing one unique idea on how to extend the work presented in the paper. These ideas will be discussed during class.

Attendance and Classroom Etiquette

You are expected to attend every class. Your participation grade will be based, in part, on how many classes you missed. Special consideration can be made for a few exceptions (e.g., academic travel, ilnesses and family emergencies). However, you must discuss your anticipated absence with the instructor, and provide the necessary justification and documentation.

During class, students are strongly discouraged to use their laptops and mobile devices.


The class project is a major component of this course. Students will work in groups on a project where they will design, prototype and/or evaluate a human-in-the-loop system. During each class, there is a 30 minute "working session" dedicated towards the project. During the working session, students can brainstorm ideas, work on specific project tasks, exchange ideas with other groups, and seek help and advice from the instructor.

The project will have four deliverables: (a) a 1-page preliminary proposal, (b) a 6-page paper draft written in conference proceedings format (without results), and (c) a final paper (with results), and (d) a project presentation. Students are encouraged to submit their work as short papers, workshop papers, or full conference papers.


Students will be evaluated on the quality of their participation (both online and in-class) and project write-up and presentation.