Below is a reading list, which serves as a reference guide about human-in-the-loop systems and highlights various topics that are covered in the course.

A. Interactive Machine Learning

• Fails, J. A., and Olsen Jr, D. R. 2003. Interactive machine learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces, pg 39-45.
• Fiebrink, R., D. Trueman, and P. R. Cook. A meta-instrument for interactive, on-the-fly machine learning. Proceedings of New Interfaces for Musical Expression (NIME), 2009.
• Power to the People: The Role of Humans in Interactive Machine Learning" by Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza. AAAI Magazine, 2013.
• Kapoor, A., Lee, B., Tan, D., and Horvitz, E. 2010. Interactive optimization for steering machine classification. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pg. 1343-1352.

B. Human-in-the-Loop Supervised Learning

• "Minimally-Supervised Learning" by Sanjoy Dasgupta
• "Active Learning"by Burr Settles, Morgan & Claypool, 2012.
• Maya Cakmak, Crystal Chao, and Andrea L. Thomaz. Designing Interactions for Robot Active Learners. IEEE Transactions ON Autonomous Mental Development, 2(2), 2010.
• S. Vijayanarasimhan and K. Grauman. What's it going to cost you? Predicting effort vs. informativeness for multi-label image annotations. In CVPR, 2009.
• B. Settles, M. Craven and L. Friedland. Active Learning with Real Annotation Costs. In Proceedings of NIPS Workshop on Cost-Sensitive Learning, 2008.
• Branson S., Wah C., Babenko B., Schroff F., Welinder P., Perona P., Belongie S., Visual Recognition with Humans in the Loop. European Conference on Computer Vision (ECCV), 2010.

C. Human-Robot Interaction

• "Human-Robot Interaction: A Survey" by Michael Goodrich and Alan Schultz. Foundations and Trends in Human-Computer Interaction, 1(3), 2007.
• "A survey of robot learning from demonstration" by Brenna D. Argall, Sonia Chernova, Manuela Veloso and Brett Browning, Robotics and Autonomous Systems, 2009.
• Reinforcement Learning with Human Teachers: Evidence of Feedback and Guidance with Implications for Learning Performance. Andrea L. Thomaz and Cynthia Breazeal
• Stephanie Rosenthal, Manuela Veloso. Mobile Robot Planning to Seek Help with Spatially-Situated Tasks. In AAAI 2012.
• C. Breazeal & A. Thomaz. Learning from human teachers with socially guided exploration. Proceedings of the 2008 IEEE International Conference on Robotics and Automatio, ICRA 2008.

D. Human Computation and Crowdsourcing

• "Human Computation" by Edith Law and Luis von Ahn, Morgan & Claypool 2011.
• Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. Exploring Iterative and Parallel Human Computation Processes. In HCOMP 2010.
• E. Law and L. von Ahn. Input-agreement: A New Mechanism for Data Collection using Human Computation Games. In CHI 2009.
• H. Zhang, E. Law, K. Gajos, E. Horvitz, R. C. Miller and D. Parkes. Human Computation Tasks with Global Constraints: A Case Study. In CHI 2012.
• Rajan Vaish, Keith Wyngarden, Jingshu Chen, Brandon Cheung, Michael Bernstein. Twitch Crowdsourcing: Crowd Contributions in Short Bursts of Time. In CHI 2014
• Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, and Adam Kalai. Adaptively Learning the Crowd Kernel. In ICML, 2011.
• Panos Ipeirotis and Evgeniy Gabrilovich. Quizz: Targeted Crowdsourcing with a Billion (Potential) Users. In WWW 2014.
• Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatrowicz, A., White, B., White, S., & Yeh, T. VizWiz: nearly real-time answers to visual questions. In UIST 2010.
• Daniel S. Weld., Mausam, and Peng Dai. Human Intelligence Needs Artificial Intelligence. In HCOMP 2011.
• Courses at Other Universities:
  • Crowdsourcing & Human Computation at University of Pennslyvania (Chris Callison-Burch)
  • Crowd Programming at Carnagie Mellon University (Jeff Bigham)

E. Mixed-Initiative Systems

• Hoffmann, R., Amershi, S., Patel, K., Wu, F., Fogarty, J., & Weld, D. S. (2009). Amplifying community content creation with mixed initiative information extraction. Proceedings of the 27th International Conference on Human Factors in Computing Systems, 1849-1858.
• "Principles of Mixed-Initiative User Interfaces" by Eric Horvitz, CHI 1999.
• A Framework for Mixed-Initiative Clustering. Yifen Huang and Tom Mitchell. NESCAI 2007, Ithaca, NY, April 2007.
• Shilman, M., Tan, D., & Simard, P. (2006). CueTIP: a mixed-initiative interface for correcting handwriting errors. UIST '06: Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology.

F. Brain-Human Interfaces

• Shenoy, P., Tan, D. S., & Tan, D. (2008). Human-aided computing: utilizing implicit human processing to classify images. CHI '08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems.
• E.T. Solovey, P. Schermerhorn, M. Scheutz, A. Sassaroli, S. Fantini, R.J.K. Jacob. Brainput: Enhancing Interactive Systems with Streaming fNIRS Brain Input. In CHI 2012.
• Szafir, D. and Mutlu, B. Pay attention! Designing Adaptive Agents that Monitor and Improve User Engagement. In CHI 2012.

G. Human-Agent Teamwork

• Human-Agent Team Formation in Dynamic Environments" by Arlette van Wissen, Ya'akov Gal, Bart Kamphorst, Virginia Dignum. Computers in Human Behavior 28:23-33, 2012
• Bradshaw et al. Adjustable Autonomy and Human-Agent Teamwork in Practice: An Interim Report on Space Applications.

H. Human-in-the-loop Security

• A Framework for Reasoning About Human in the Loop by Lori Cranor.
• I. Flechais, J. Riegelsberger and A. Sasse. Divide and Conquer: The Role of Trust and Assurance in the Design of Secure Socio-Technical Systems. In 2005 Workshop on New Security Paradigm 2005.
• E. Choe, S. Consolvo, J. Jung, B. Harrison, S. Patel and J. Kientz. Investigating Receptiveness to Sensing and Inference in the Home using Sensor Proxies. In Ubicomp 2012.
• S. Uellenbeck, M. Durmuth, C. Wolf, and T. Holz. Quantifying the Security of Graphical Passwords: The Case of Android Unlock Patterns. In Proceedings of the 2013 ACM SIGSAC conference on Computer & Communications Security, 2013. (CCS '13)
• C. Bravo-Lillo, L. Faith Cranor, J. Downs, S. Komanduri, R. W. Reeder, S. Schechter, and M. Sleeper. Your Attention Please: Designing security-decision UIs to make genuine risks harder to ignore. In Proceedings of the Ninth Symposium on Usable Privacy and Security, 2013. (SOUPS '13)

I. Research Methods in Human-in-the-loop Systems

• Gergle, D., & Tan, D. (2014). Experimental Research in HCI. In Ways of Knowing in HCI. Eds: J.S. Olson and W. Kellogg. Springer.

J. Human-Automation Interaction

• V. Riley. A General Model of Mixed-Initiative Human-Machine Systems. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting October 1989 vol. 33 no. 2 124-128.
• R. Parasuraman and V. Riley. Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society 1997 39: 230.
• N.B. Sarter, D. D. Woods, and C.E. Billings. Automation Surprises. Handbook of Human Factors & Ergonomics, second edition, G. Salvendy (Ed.), Wiley, 1997.
• R. Parasuraman, T. B. Sheridan and C. D. Wickens. A Model for Types and Levels of Human Interaction with Automation. In IEEE Transactions On Systems, Man and Cybernetics - Part A: Systems and Humans, Vol 30, No. 3, May 2000.
• C. D. Wickens, H. Li, A. Santamaria, A. Sebok and N. B. Sarter. Stages and Levels of Automation: An Integrated Meta-analysis. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2010 54: 389.
• D. Kaber, J. Riley, K. Tan and M. Endsley. On the Design of Adaptive Automation for Complex Systems. International Journal of Cognitive Ergonomics, 2001, 5(1), 37--57.
• H. Furukawa and R. Parasuraman. Supporting System-Centered View of Operators Through Ecological Interface Design: Two Experiments on Human-Centered Automation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2003 47: 567.

K. Ethics

• Irani, L. and M.S. Silberman. (2013) Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. Proceedings of CHI 2013.
• David Martin, Benjamin V. Hanrahan, Jacki O'Neill, Neha Gupta. (2014) Being a Turker. In CSCW.

Design by OS Templates