Current Research Projects
  • CrowdEEG

    We are developing a framework that combines machine and human intelligence to classify human clinical EEG recordings, which aims to increase accuracy, cost efficiency, and capacity of analysis. We study how EEG specialists can be supported in collaborative analysis of ambiguous cases, how to combine active learning algorithms with crowdsourcing to reduce the need for expert involvement, and how to enable non-experts to perform such complex medical annotation tasks. This project is in collaboration with Prof. Joelle Pineau from McGill University and Dr. Andrew Lim from Sunnybrook Hospital. (IJCAI 2016, WebConf 2019 Workshop, CHI 2019 Workshop).

  • Teachable Robot Project

    Curiosity a well-studied psychological phenomenon and a powerful motivating factor designers can leverage to increase user engagement, particularly in applications requiring long-term interaction (e.g., tutoring systems, health tracking tools). In the teachable robot project, we aim to design a teachable robot agent that acts as a less knowledgeable peer to be taught by elementary-school students, with the explicit goal of inducing their curiosity about the material being taught, and use the teachable robot agent as an experimentation platform to study technology-mediated peer learning. (CHI 2019, CHI 2019 Workshop).

Conference and Journal Papers

J. Ceha, N. Chhibber, J. Goh, P-Y Oudeyer, C. McDonald, D. Kulic, E. Law. "Expression of Curiosity in Social Robots: Design, Perception, and Effects on Behaviour." To appear in CHI 2019.

M. Schaekermann, J. Goh, K. Larson and E. Law. "Resolvable vs. Irresolvable Disagreement: A Study on Worker Deliberation in CrowdWork." In CSCW 2018.
* Best Paper Award

W. Challaghan, J. Goh, M. Mohareb, A. Lim and E. Law. "MechanicalHeart: A Human-Machine Framework for the Classification of Phonocardiograms." In CSCW 2018.

N. Chihhibber, R. Syed, M. Teng, J. Goh, K. Collins-Thompson and E. Law. "Human Perception of Surprise: A User Study." In Proceedings of SIGIR Computational Surprise Workshop, 2018.

A. Vtyurina, A. Fourney and E. Law. "5 Seconds After: Exploring User Actions with Voice Assistants in the Moments After a System Response." In Proceedings of CHI Voice-based Conversational UX Studies and Design, 2018.

Cartwright, M., Seals, A., Salamon, J., Williams, A., Mikloska, S., MacConnell, D., Law, E., Bello, J.P., Nov, O. "Seeing Sound: Investigating the Effects of Visualizations and Complexity on Crowdsourced Audio Annotations." In Proceedings of the ACM on Human-Computer Interaction, vol. 1(1): Computer-Supported Cooperative Work and Social Computing 2017, 2017.

A. Williams, J. Goh, C. Willis, A. Ellison, J. Brusuelas, C. Davis and E. Law. Deja Vu: Characterizing Worker Reliability Using Task Consistency. To appear in AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2017.

E. Law, V. Cai, E. Liu, S. Sasy, J. Goh, A. Blidaru and D. Kulic. A Wizard-of-Oz Study of Curiosity in Human-Robot Interaction. To appear in IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2017.

M. A. O'Leary, K. Alphonse, M. Arce H., D. Cavaliere, A. Cirranello, T Dietterich, M. Julius, S. Kaufman, E. Law, M. Passarotti, A. Reft, J. Robalino, N. B. Simmons, S. Smith, D. Stevenson, E. Theriot, P. M. Velazco, R. Walls, M. Yu and M. Daly. Crowds Replicate Performance of Scientific Experts Scoring Phylogenetic Matrices of Phenotypes. To appear in Systematic Biology, 2017. (The definitive version is available here)

C. Willis, E. Law, A. Williams, B. Franzone, R. Bernardos, L. Bruno, C. Hopkins, C. Schorn, E. Weber, D. Park and C. Davis. CrowdCurio: an online crowdsourcing platform to facilitate climate change studies using herbarium specimens. In New Phytologist, 2017. (The definitive version is available here)

P. Jaini, Z. Chen, P. Carbajal, E. Law, L. Middleton, K. Regan, M. Schaekermann, G. Trimponias, J. Tung, and P. Poupart. Online Bayesian Transfer Learning for Sequential Data Modeling. In 5th International Conference on Learning Representations (ICLR), 2017.

E. Law, K. Z. Gajos, A. Wiggins, M. Gray and A. Williams. Crowdsourcing as a Tool for Research: Implications of Uncertainty. In CSCW 2017.

T. Tse, J. Salamon, A. Williams, H. Jiang and E. Law. Ensemble: A Hybrid Human-Machine System for Generating Melody Scores from Audio. In ISMIR 2016.

S. Pan, K. Larson, J. Bradshaw and E. Law. Dynamic Task Allocation Algorithm for Hiring Workers that Learn. In IJCAI 2016.

E. Law, M. Yin, J. Goh, K. Chen, M. Terry and K. Gajos. "Curiosity Killed the Cat, but Makes Crowdwork Better." In CHI 2016.
* Best Paper Honorable Mention

E. Law, B. Grosz, L. M. Sanders and S. H. Fischer. "SimplyPut: Leveraging a Mixed-Expertise Crowd to Improve Health Literacy." In AAMAS Workshop on Human-Agent Interaction Design and Models 2013.

E. Law, C. Dalton, N. Merrill, A. Young, K. Z. Gajos. "Curio: A Platform for Supporting Mixed-Expertise Crowdsourcing." In HCOMP 2013.

O. Amir, B. Grosz, E.Law and R. Stern. "Collaborative Health Plan Support." In AAMAS 2013.
* Challenges and Visions Track, Best Paper Second Prize

H. Zhang, E. Law, K. Gajos, E. Horvitz, R. C. Miller and D. Parkes. "Human Computation Tasks with Global Constraints: A Case Study." In CHI 2012.
* Best Paper Honorable Mention

E. Law and L. von Ahn. Human Computation. Morgan & Claypool Synthesis Lectures on Artificial Intelligence and Machine Learning, edited by Ron Brachman, Tom Dietterich and William Cohen, June 2011.

E. Law. "Human Computation for Music Classification." In Music Data Mining, edited by T. Li, M. Ogihara and G. Tzanetakis. CRC Press/Chapman Hall, 2011.

E. Law, B. Settles, A. Snook, H. Surana, L. von Ahn and T. Mitchell. "Human Computation for Attribute and Attribute Value Acquisition." In CVPR Workshop on Fine-Grained Visual Categorization 2011.

E. Law, P. Bennett, and E. Horvitz. "The Effects of Choice in Routing Relevance Judgments." In SIGIR 2011.

E. Law and H. Zhang. "Towards Large-Scale Collaborative Planning: Answering High-Level Search Queries Using Human Computation." In AAAI 2011.

E. Law, B. Settles and T. Mitchell. "Learning to Tag using Noisy Labels." In ECML 2010.

E. Law, K. West, M. Mandel, M. Bay and S. Downie. "Evaluation of Algorithms Using Games: The Case of Music Tagging." In ISMIR 2009. EXPERIMENTS

E. Law and L. von Ahn. "Input-agreement: A New Mechanism for Data Collection using Human Computation Games." In CHI 2009. VIDEO DATASET BLOG
* Best Paper Honorable Mention

J. Betteridge, A. Carlson, S. Hong, E. Hruschka Jr., E. Law, T. Mitchell and S. Wang. "Towards Never Ending Language Learning." In AAAI Spring Symposium on Learning by Reading and Learning to Read 2009.

E. Law, L. von Ahn and T. Mitchell. "Search Wars: A Game for Improving Web Search." In HCOMP 2009.

E. Law, A. Mityagin and M. Chickering. "Intentions: A Game for Classifying Search Query Intent." In CHI 2009 Work-in-Progress.

E. Law. The Problem of Accuracy as An Evaluation Criterion." In ICML Workshop on Evaluation Methods for Machine Learning 2008.

E. Law, L. von Ahn, R. Dannenberg and M. Crawford. "TagATune: a Game for Sound and Music Annotation." In ISMIR 2007.

Exciting Work-in-Progress: Extended Abstract, Workshop Papers, etc

C. Vandenhof and E. Law. Contradict the Machine: A Hybrid Approach to Identifying Unknown Unknowns. AAMAS 2019 Extended Abstract.

E. Law and X. Kuang. The Role of Experimentation in Human-AI Interaction. CHI 2019 Workshop on Where is the Human? Bridging the Gap Between AI and HCI.

E. Law and D. Kulic. How to Know When a User is Curious: A Challenge for HRI in Social Robotics. CHI 2019 Workshop on The Challenges of Working on Social Robots that Collaborate with People

A. Williams, H. Kaur, E. Law and E. Lank. Guiding Attention with Tasks and Emotions in Conversational Agents. CHI 2019 Workshop on Conversational Agents: Constructing Action Plans from a Wave of Research and Development.

N. Chhibber and E. Law. Using Conversational Agents To Support Learning By Teaching. CHI 2019 Workshop on Conversational Agents: Constructing Action Plans from a Wave of Research and Development.

N. Dillen, E. Law, O. Schneider and K. Czarnecki. Passenger-vehicle Interaction in Fully Autonomous Vehicles: current and future trends. CHI 2019 Workshop on Looking into the Future: Weaving the Threads of Vehicle Automation.

M. Schaerkemann, G. Beaton, M. Habib, A. Lim, K. Larson, E. Law. Capturing Expert Arguments from Medical Adjudication Discussions in a Machine-readable Format. WebConf 2019 Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing.

M. Schaerkemann, G. Beaton, M. Habib, A. Lim, K. Larson, E. Law. crowdEEG: A Platform for Structured Consensus Formation in Medical Time Series Analysis. CHI 2019 Workgroup on Interactive Systems in Healthcare (WISH).