Current Research Projects
  • Small Group Crowdsourcing

    We are developing a framework that enables small group of people to annotate scientific, historical and medical artifacts, and deliberate on ambiguous cases. In CrowdEEG, for example, we study how EEG specialists can be supported in collaborative analysis of ambiguous cases, how to combine active learning algorithms with crowdsourcing to reduce the need for expert involvement, and how to enable non-experts to perform such complex medical annotation tasks. This project is in collaboration with Prof. Joelle Pineau from McGill University and Dr. Andrew Lim from Sunnybrook Hospital. (IJCAI 2016, WebConf 2019 Workshop, CHI 2019 Workshop, CHI 2020).

  • Teachable Robot Project

    Curiosity a well-studied psychological phenomenon and a powerful motivating factor designers can leverage to increase user engagement, particularly in applications requiring long-term interaction (e.g., tutoring systems, health tracking tools). In the teachable robot project, we aim to design a teachable robot agent that acts as a less knowledgeable peer to be taught by elementary-school students, with the explicit goal of inducing their curiosity about the material being taught, and use the teachable robot agent as an experimentation platform to study technology-mediated peer learning. (CHI 2019, CHI 2019 Workshop).

Conference and Journal Papers

J. Ceha, K. J. Lee, E. Nilsen, J. Goh and E. Law. Can a Humorous Conversational Agent Enhance Learning Experience and Outcomes? To Appear in CHI 2021. * Best Paper Honorable Mention

S. Suh, K. J. Lee, B. Cheng, C. Latulipe and E. Law. Using Comics to Introduce and Reinforce Programming Concepts in CS1. In SIGCSE 2021.

A. Vtyurina, C. Clarke, E. Law, J. Trippas, and H. Bota. A Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity. In ACM ICTIR 2020.

S. Suh, M. Lee, G. Xia and E. Law. Coding Strip: A Pedagogical Tool for Teaching and Learning Programming Conceptsthrough Comics. In VL/HCC 2020.

H. Chen, R. Cohen, K. Dautenhahn, E. Law, K. Czarnecki. Autonomous Vehicle Visual Signals for Pedestrians: Experiments and Design Recommendations. In IEEE Intelligent Vehicles Symposium (IV) 2020.

S. Suh, M. Lee and E. Law. How Do We Design for Concreteness Fading? Survey, General Framework, and Design Dimensions. In IDC 2020.

M. Alaimi, E. Law, K. Pantasdo, P-Y. Oudeyer and H. Sauzeon. Pedagogical Agents for Fostering Question-Asking Skills in Children. In CHI 2020.

M. Schaekermann, G. Beaton, E. Sanoubari, A. Lim, K. Larson and E. Law. Ambiguity-aware AI Assistants for Medical Data Analysis. In CHI 2020.

N. Dillen, M. Ilievski, E. Law, L. Nacke, K. Czarnecki and O. Schneider. Keep Calm and Ride Along: Passenger Comfort and Anxiety as Physiological Responses to Autonomous Driving Styles. In CHI 2020.

A. Williams, G. Mark, K. Milland, E. Lank and E. Law. The Perpetual Work Life of Crowdworkers: How Tooling Practices Increase Fragmentation in Crowdwork. In CSCW 2019.

M. Schaekermann, G. Beaton*, M. Habib, A. Lim, K. Larson and E. Law. Understanding Expert Disagreement in Medical Data Analysis through Structured Adjudication. In CSCW 2019.

G. d’Eon, J. Goh, K. Larson and E. Law. Paying Crowd Workers for Collaborative Work. In CSCW 2019.

J. Ceha, N. Chhibber, J. Goh, P-Y Oudeyer, C. McDonald, D. Kulic, E. Law. "Expression of Curiosity in Social Robots: Design, Perception, and Effects on Behaviour." In CHI 2019.

M. Schaekermann, J. Goh, K. Larson and E. Law. "Resolvable vs. Irresolvable Disagreement: A Study on Worker Deliberation in CrowdWork." In CSCW 2018.
* Best Paper Award

W. Callaghan, J. Goh, M. Mohareb, A. Lim and E. Law. "MechanicalHeart: A Human-Machine Framework for the Classification of Phonocardiograms." In CSCW 2018.

N. Chhibber, R. Syed, M. Teng, J. Goh, K. Collins-Thompson and E. Law. "Human Perception of Surprise: A User Study." In Proceedings of SIGIR Computational Surprise Workshop, 2018.

A. Vtyurina, A. Fourney and E. Law. "5 Seconds After: Exploring User Actions with Voice Assistants in the Moments After a System Response." In Proceedings of CHI Voice-based Conversational UX Studies and Design, 2018.

Cartwright, M., Seals, A., Salamon, J., Williams, A., Mikloska, S., MacConnell, D., Law, E., Bello, J.P., Nov, O. "Seeing Sound: Investigating the Effects of Visualizations and Complexity on Crowdsourced Audio Annotations." In Proceedings of the ACM on Human-Computer Interaction, vol. 1(1): Computer-Supported Cooperative Work and Social Computing 2017, 2017.

A. Williams, J. Goh, C. Willis, A. Ellison, J. Brusuelas, C. Davis and E. Law. Deja Vu: Characterizing Worker Reliability Using Task Consistency. To appear in AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2017.

E. Law, V. Cai, E. Liu, S. Sasy, J. Goh, A. Blidaru and D. Kulic. A Wizard-of-Oz Study of Curiosity in Human-Robot Interaction. To appear in IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2017.

M. A. O'Leary, K. Alphonse, M. Arce H., D. Cavaliere, A. Cirranello, T Dietterich, M. Julius, S. Kaufman, E. Law, M. Passarotti, A. Reft, J. Robalino, N. B. Simmons, S. Smith, D. Stevenson, E. Theriot, P. M. Velazco, R. Walls, M. Yu and M. Daly. Crowds Replicate Performance of Scientific Experts Scoring Phylogenetic Matrices of Phenotypes. To appear in Systematic Biology, 2017. (The definitive version is available here)

C. Willis, E. Law, A. Williams, B. Franzone, R. Bernardos, L. Bruno, C. Hopkins, C. Schorn, E. Weber, D. Park and C. Davis. CrowdCurio: an online crowdsourcing platform to facilitate climate change studies using herbarium specimens. In New Phytologist, 2017. (The definitive version is available here)

P. Jaini, Z. Chen, P. Carbajal, E. Law, L. Middleton, K. Regan, M. Schaekermann, G. Trimponias, J. Tung, and P. Poupart. Online Bayesian Transfer Learning for Sequential Data Modeling. In 5th International Conference on Learning Representations (ICLR), 2017.

E. Law, K. Z. Gajos, A. Wiggins, M. Gray and A. Williams. Crowdsourcing as a Tool for Research: Implications of Uncertainty. In CSCW 2017.

T. Tse, J. Salamon, A. Williams, H. Jiang and E. Law. Ensemble: A Hybrid Human-Machine System for Generating Melody Scores from Audio. In ISMIR 2016.

S. Pan, K. Larson, J. Bradshaw and E. Law. Dynamic Task Allocation Algorithm for Hiring Workers that Learn. In IJCAI 2016.

E. Law, M. Yin, J. Goh, K. Chen, M. Terry and K. Gajos. "Curiosity Killed the Cat, but Makes Crowdwork Better." In CHI 2016.
* Best Paper Honorable Mention

E. Law, B. Grosz, L. M. Sanders and S. H. Fischer. "SimplyPut: Leveraging a Mixed-Expertise Crowd to Improve Health Literacy." In AAMAS Workshop on Human-Agent Interaction Design and Models 2013.

E. Law, C. Dalton, N. Merrill, A. Young, K. Z. Gajos. "Curio: A Platform for Supporting Mixed-Expertise Crowdsourcing." In HCOMP 2013.

O. Amir, B. Grosz, E.Law and R. Stern. "Collaborative Health Plan Support." In AAMAS 2013.
* Challenges and Visions Track, Best Paper Second Prize

H. Zhang, E. Law, K. Gajos, E. Horvitz, R. C. Miller and D. Parkes. "Human Computation Tasks with Global Constraints: A Case Study." In CHI 2012.
* Best Paper Honorable Mention

E. Law and L. von Ahn. Human Computation. Morgan & Claypool Synthesis Lectures on Artificial Intelligence and Machine Learning, edited by Ron Brachman, Tom Dietterich and William Cohen, June 2011.

E. Law. "Human Computation for Music Classification." In Music Data Mining, edited by T. Li, M. Ogihara and G. Tzanetakis. CRC Press/Chapman Hall, 2011.

E. Law, B. Settles, A. Snook, H. Surana, L. von Ahn and T. Mitchell. "Human Computation for Attribute and Attribute Value Acquisition." In CVPR Workshop on Fine-Grained Visual Categorization 2011.

E. Law, P. Bennett, and E. Horvitz. "The Effects of Choice in Routing Relevance Judgments." In SIGIR 2011.

E. Law and H. Zhang. "Towards Large-Scale Collaborative Planning: Answering High-Level Search Queries Using Human Computation." In AAAI 2011.

E. Law, B. Settles and T. Mitchell. "Learning to Tag using Noisy Labels." In ECML 2010.

E. Law, K. West, M. Mandel, M. Bay and S. Downie. "Evaluation of Algorithms Using Games: The Case of Music Tagging." In ISMIR 2009. EXPERIMENTS

E. Law and L. von Ahn. "Input-agreement: A New Mechanism for Data Collection using Human Computation Games." In CHI 2009. VIDEO DATASET BLOG
* Best Paper Honorable Mention

J. Betteridge, A. Carlson, S. Hong, E. Hruschka Jr., E. Law, T. Mitchell and S. Wang. "Towards Never Ending Language Learning." In AAAI Spring Symposium on Learning by Reading and Learning to Read 2009.

E. Law, L. von Ahn and T. Mitchell. "Search Wars: A Game for Improving Web Search." In HCOMP 2009.

E. Law, A. Mityagin and M. Chickering. "Intentions: A Game for Classifying Search Query Intent." In CHI 2009 Work-in-Progress.

E. Law. The Problem of Accuracy as An Evaluation Criterion." In ICML Workshop on Evaluation Methods for Machine Learning 2008.

E. Law, L. von Ahn, R. Dannenberg and M. Crawford. "TagATune: a Game for Sound and Music Annotation." In ISMIR 2007.

Exciting Work-in-Progress: Extended Abstract, Workshop Papers, etc

E. Law, K.J. Lee and P. Baghaei. The Curiosity Notebook: A Platform for Supporting Learning with Teachable Robots. In HRI 2020 Workshop on Exploring Creative Contents in Social Robots.

E. Law, E. Nilsen, K.J. Lee, A. Chauhan, J. Ceha, P. Baghaei, A. Debbane and K. Yang. Learning and Teaching with Conversational Agents. In CHI 2020 Workshop on Grand Challenges in Conversational UI.

E. Law, P. Baghaei, N. Chhibber, D. Kulic, S. Lin, K. D. Pantasdo, J. Ceha, S. Suh, N. Dillen. Curiosity Notebook: A Platform for Learning by Teaching Conversational Agents. In CHI 2020 Late Breaking Work.

C. Vandenhof and E. Law. Contradict the Machine: A Hybrid Approach to Identifying Unknown Unknowns. AAMAS 2019 Extended Abstract.

E. Law and X. Kuang. The Role of Experimentation in Human-AI Interaction. CHI 2019 Workshop on Where is the Human? Bridging the Gap Between AI and HCI.

E. Law and D. Kulic. How to Know When a User is Curious: A Challenge for HRI in Social Robotics. CHI 2019 Workshop on The Challenges of Working on Social Robots that Collaborate with People

A. Williams, H. Kaur, E. Law and E. Lank. Guiding Attention with Tasks and Emotions in Conversational Agents. CHI 2019 Workshop on Conversational Agents: Constructing Action Plans from a Wave of Research and Development.

N. Chhibber and E. Law. Using Conversational Agents To Support Learning By Teaching. CHI 2019 Workshop on Conversational Agents: Constructing Action Plans from a Wave of Research and Development.

N. Dillen, E. Law, O. Schneider and K. Czarnecki. Passenger-vehicle Interaction in Fully Autonomous Vehicles: current and future trends. CHI 2019 Workshop on Looking into the Future: Weaving the Threads of Vehicle Automation.

M. Schaerkemann, G. Beaton, M. Habib, A. Lim, K. Larson, E. Law. Capturing Expert Arguments from Medical Adjudication Discussions in a Machine-readable Format. WebConf 2019 Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing.

M. Schaerkemann, G. Beaton, M. Habib, A. Lim, K. Larson, E. Law. crowdEEG: A Platform for Structured Consensus Formation in Medical Time Series Analysis. CHI 2019 Workgroup on Interactive Systems in Healthcare (WISH).