September 10

Handouts: Slides
Homework: By Friday Sept 12, Sign-up on doodle for your discussant seession.

September 17

Human Computation I
  1. G. Little, L. B. Chilton, M. Goldman, and R. C. Miller. Exploring Iterative and Parallel Human Computation Processes. In HCOMP 2010.
    (Discussant: Xiang)
  2. O. Tamuz, C. Liu, S. Belongie, O. Shamir, and A. Kalai. Adaptively Learning the Crowd Kernel. In ICML, 2011.
    (Discussant: Menna)
  3. R. Vaish, K. Wyngarden, J. Chen, B. Cheung, M. Bernstein. Twitch Crowdsourcing: Crowd Contributions in Short Bursts of Time. In CHI 2014.
    (Discussant: Adriel)

September 24

Interactive Machine Learning
  1. Fails, J. A., and Olsen Jr, D. R. 2003. Interactive machine learning. In IUI, pg 39-45.
    (Discussant: Shu)
  2. Fiebrink, R., D. Trueman, and P. R. Cook. A meta-instrument for interactive, on-the-fly machine learning. Proceedings of New Interfaces for Musical Expression (NIME), 2009.
    (Discussant: Jingjie)
  3. Kapoor, A., Lee, B., Tan, D., and Horvitz, E. 2010. Interactive optimization for steering machine classification. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pg. 1343-1352.
    (Discussant: Peng)

October 1

Mixed-Initiative Systems
  1. Hoffmann, R., Amershi, S., Patel, K., Wu, F., Fogarty, J., & Weld, D. S. (2009). Amplifying community content creation with mixed initiative information extraction. Proceedings of the 27th International Conference on Human Factors in Computing Systems, 1849-1858.
    (Discussant: Qifan)
  2. Horvitz, E. Principles of Mixed-Initiative User Interfaces. In CHI 1999.
    (Discussant: Adriaan)
  3. Shilman, M., Tan, D., & Simard, P. (2006). CueTIP: a mixed-initiative interface for correcting handwriting errors. UIST '06: Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology.
    (Discussant: Lalit)

Due: project ideas

October 8

Human-in-the-loop Supervised Learning
  1. Maya Cakmak, Crystal Chao, and Andrea L. Thomaz. Designing Interactions for Robot Active Learners. IEEE Transactions ON Autonomous Mental Development, 2(2), 2010.
    (Discussant: Sharon)
  2. Branson S., Wah C., Babenko B., Schroff F., Welinder P., Perona P., Belongie S., Visual Recognition with Humans in the Loop. European Conference on Computer Vision (ECCV), 2010.
    (Discussant: Devin)
  3. B. Settles, M. Craven and L. Friedland. Active Learning with Real Annotation Costs. In Proceedings of NIPS Workshop on Cost-Sensitive Learning, 2008.
    (Discusant: Steven)

October 15

Human-Robot Interaction
  1. Stephanie Rosenthal, Manuela Veloso. Mobile Robot Planning to Seek Help with Spatially-Situated Tasks. In AAAI 2012.
    (Discussant: Vishal)
  2. Andrea L. Thomaz and Cynthia Breazeal. Reinforcement Learning with Human Teachers: Evidence of Feedback and Guidance with Implications for Learning Performance. In AAAI, 2006.
    (Discussant: Brian)
  3. C. Breazeal & A. Thomaz. Learning from human teachers with socially guided exploration. Proceedings of the 2008 IEEE International Conference on Robotics and Automatio, ICRA 2008.
    (Discussant: Pragnya)

Due: project proposal, 1-2 pages, in SIGCHI format. The proposal should include (a) problem statement, (b) system description, (c) a plan for evaluating the system performance, to characterize how involving human in the loop enables the system to achieve better computational outcomes, and (d) a plan for evaluating the human component of the system. Your description for (d) should address the following questions: What will you consider when designing the interfaces to elicit human input and feedback? Can you think of multiple interfaces to elicit the same feedback? Which interface works better and why? You can consider using heuristic evaluation as a methodology.

October 22

  1. Irani, L. and M.S. Silberman. (2013) Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. Proceedings of CHI 2013.
    (Discussant: Alexandra)
  2. David Martin, Benjamin V. Hanrahan, Jacki O'Neill, Neha Gupta. (2014) Being a Turker. In CSCW.
    (Discussant: Caroyln)
  3. Watch "Minds for Sale" by Jonathan Zittrain.

Special Guest: SpamGirl (community manager for TurkerNation)

Due: (1) prepare questions for SpamGirl, (2) sign-up for a project meeting time on Monday Oct 20 or Oct 21 using this Doodle.

October 29

Brain-Human Interfaces
  1. Shenoy, P., Tan, D. S., & Tan, D. Human-aided computing: utilizing implicit human processing to classify images. In CHI 2008.
    (Discussant: Jasnoor)
  2. E.T. Solovey, P. Schermerhorn, M. Scheutz, A. Sassaroli, S. Fantini, R.J.K. Jacob. Brainput: Enhancing Interactive Systems with Streaming fNIRS Brain Input. In CHI 2012.
    (Discussant: Atri)
  3. Szafir, D. and Mutlu, B. Pay attention! Designing Adaptive Agents that Monitor and Improve User Engagement. In CHI 2012.
    (Discussant: Yeti)

November 5

Human-in-the-loop Security
  1. E. Choe, S. Consolvo, J. Jung, B. Harrison, S. Patel and J. Kientz. Investigating Receptiveness to Sensing and Inference in the Home using Sensor Proxies. In Ubicomp 2012.
    (Discussant: Xiang)
  2. S. Uellenbeck, M. Durmuth, C. Wolf, and T. Holz. Quantifying the Security of Graphical Passwords: The Case of Android Unlock Patterns. In Proceedings of the 2013 ACM SIGSAC conference on Computer & Communications Security, 2013. (CCS '13)
    (Discussant: Lalit)
  3. C. Bravo-Lillo, L. Faith Cranor, J. Downs, S. Komanduri, R. W. Reeder, S. Schechter, and M. Sleeper. Your Attention Please: Designing security-decision UIs to make genuine risks harder to ignore. In Proceedings of the Ninth Symposium on Usable Privacy and Security, 2013. (SOUPS '13)
    (Discussant: Pragnya)

Due: 4-page project paper draft (introduction, related work, system description, experimental design)

November 12

Human Computation II
  1. Panos Ipeirotis and Evgeniy Gabrilovich. Quizz: Targeted Crowdsourcing with a Billion (Potential) Users. In WWW 2014.
    (Discussant: Sharon)
  2. Daniela Retelny et al. Expert Crowdsourcing with Flash Teams. In UIST 2014.
    (Discussant: Jie Peng)
  3. Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatrowicz, A., White, B., White, S., & Yeh, T. VizWiz: nearly real-time answers to visual questions. In UIST 2010.
    (Discussant: Peng Peng)

November 19

User Studies and System Evaluation
  1. R. Parasuraman and V. Riley. Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors: The Journal of the Human Factors and Ergonomics Society 1997 39: 230. (Optional Reading, No Presentation)

Due: You will bring your prototype system to class, and evaluate 4+ users interacting with your system. Before the session, you should (1) select a few heuristics from Nielsen's Heuristics and use them to guide your observations of user behavior. (2) design a questionnaire/survey that users can complete to provide evaluation/feedback to your system, (3) ensure that your system is actually logging user actions, so that you can analyze that data later.

Your Evaluation Teams are listed here.

Guest Speaker: Burr Settles (Duolingo)

November 26 (Last Class)

Project Showcase
Students present their work.

December 3

Due: project report