Research
Focus Areas
Programming Education at Scale
Instructors can see the code a class turns in, but not how any of it got written. We build tools that make a whole class's problem-solving process visible in time to act on it -- including how students use AI.
People:
Publications:
- Editrail: Understanding AI Usage by Visualizing Student-AI Interaction in Code
- CodeStream: Augmenting Timelines with Code Annotation for Navigating Large Coding Histories
- SPARK: Real-Time Monitoring of Multi-Faceted Programming Exercises
- Co-Advisor: Learning Programming Strategies in Context
- CFlow: Supporting Semantic Flow Analysis of Students' Code in Programming Problems at Scale
- RunEx: Augmenting Regular-Expression Code Search with Runtime Values
- VizProg: Identifying Misunderstandings by Visualizing Students' Coding Progress
- PuzzleMe: Leveraging Peer Assessment for In-Class Programming Exercises
- EdCode: Towards Personalized Support at Scale for Remote Assistance in CS Education
Web Automation and Programming by Demonstration
People want to automate work on websites they do not control. We study what makes that hard, and build tools that let them demonstrate what they want and then see what was inferred from the demonstration.
People:
Publications:
- Multi-Click: Cross-Tab Web Automation via Action Generalization
- ScrapeViz: Hierarchical Representations for Web Scraping Macros
- ParamMacros: Creating UI Automation Leveraging End-User Natural Language Parameterization
- Understanding the Challenges and Needs of Programmers Writing Web Automation Scripts
Building and Testing Interactive Interfaces
Interactive behavior is scattered across event handlers, which makes it hard to write, hard to read, and hard to test. We design notations, editors, and prototyping tools for expressing behavior directly, and techniques for finding where an interface breaks.
People:
Publications:
- Simulating Human Cursor Trajectories for Path-Sensitive GUI Evaluation
- CoCapture: Effectively Communicating UI Behaviors on Existing Websites by Demonstrating and Remixing
- Improving Crowd-Supported GUI Testing with Structural Guidance
- Implementing Multi-Touch Gestures with Touch Groups and Cross Events
- Expresso: Building Responsive Interfaces with Keyframes
- InterState: A Language and Environment for Expressing Interface Behavior
- ConstraintJS: Programming Interactive Behaviors for the Web by Integrating Constraints and States
- Playbook: Revision Control & Comparison for Interactive Mockups
- How to Support Designers in Getting Hold of the Immaterial Material of Software
- FireCrystal: Understanding Interactive Behaviors in Dynamic Web Pages
Explaining Code and Capturing Process
Code records the result of a decision but not the reasoning behind it. We build editors, notebooks, and visualizations that capture the intent, discussion, and process behind an artifact, and keep them attached to it.
People:
Publications:
- ConvoMap: Interactive Visualizations for Exploring Complex Conversations in Multi-Agent Systems
- Colaroid: A Literate Programming Approach for Authoring Explorable Multi-Stage Tutorials
- Think-Aloud Computing: Supporting Rich and Low-Effort Knowledge Capture
- Callisto: Capturing the "Why" by Connecting Conversations with Computational Narratives
- How Data Scientists Use Computational Notebooks for Real-Time Collaboration
- Creating Guided Code Explanations with chat.codes
- Codelets: Linking Interactive Documentation and Example Code in the Editor
Accessibility of Programming and the Web
Programming tools are visual by default, which shapes who can contribute to software and in what role. We study how blind and low-vision developers actually work, alone and on mixed-ability teams, and what their tools would have to change.
People:
Publications:
- Towards Inclusive Source Code Readability Based on the Preferences of Programmers with Visual Impairments
- UI Development Experiences of Programmers with Visual Impairments in Product Teams
- Accessibility of UI Frameworks and Libraries for Programmers with Visual Impairments
- Understanding Accessibility and Collaboration in Programming for People with Visual Impairments
- Explore, Create, Annotate: Designing Digital Drawing Tools with Visually Impaired People
- Arboretum and Arbility: Improving Web Accessibility Through a Shared Browsing Architecture
Immersive Authoring
Building 3D content means describing spatial things through a 2D screen. We build tools for authoring, versioning, and co-creating scenes with generative models from inside VR.
People:
Publications:
- VRCopilot: Authoring 3D Layouts with Generative Models in VR
- VRGit: A Version Control System for Collaborative Content Creation in Virtual Reality
- FlowMatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality
- Studying the Benefits and Challenges of Immersive Dataflow Programming
On-Demand Help for Programmers
A stuck programmer needs a helper who already has their context. We built systems that package that context, route it to an expert, a crowd, or a program synthesizer, and get the answer back into the editor -- mostly between 2016 and 2020.
People:
All Publications
UIST 2026
-
Editrail: Understanding AI Usage by Visualizing Student-AI Interaction in Code
Editrail visualizes AI contributions alongside students’ code editing histories to help instructors understand how students use and adapt generated code. In a comparison study, participants identified AI usage patterns more accurately than with the baseline interface and used the histories to decide where targeted guidance could help.
CHI 2026
-
CodeStream: Augmenting Timelines with Code Annotation for Navigating Large Coding Histories
CodeStream combines code annotations, semantic zooming, and cumulative effort indicators to help instructors navigate students’ programming histories. In a study with 12 participants, it improved accuracy in identifying coding patterns compared with a baseline timeline while requiring comparable analysis time.
TSE 2026
-
Navigating Complexity: How Context Shapes Debugging Strategy Choices Among Expert Developers
A survey of 35 web developers and interviews with 16 expert web developers examine how context shapes debugging strategy choices. The study identifies static and dynamic contextual factors and a descriptive state-transition model showing how experts adapt their strategies as clarity, reproducibility, and constraints change.
VL/HCC 2025
-
Co-Advisor: Learning Programming Strategies in Context
Co-Advisor uses code context, runtime information, and programmer actions to provide debugging strategies with personalized feedback in Visual Studio Code. In a formative study with 14 programmers, participants made more debugging progress with contextual feedback than with non-adaptive strategies, suggesting a way to support strategic knowledge during problem solving.
-
ConvoMap: Interactive Visualizations for Exploring Complex Conversations in Multi-Agent Systems
ConvoMap helps developers inspect conversations among AI agents through automated topic coding, a semantic map, and linked message views. In a comparison study with 16 participants, it improved accuracy in understanding agent behavior and conversation patterns, supporting investigation of recurring problems across multiple runs.
-
SPARK: Real-Time Monitoring of Multi-Faceted Programming Exercises
SPARK helps instructors monitor programming exercises with interdependent steps and varied outputs by organizing progress around instructor-defined checkpoints. It suggests tests, visualizes progress, and exposes intermediate outputs; a study with 16 participants found more accurate identification of student difficulties than with the baseline dashboard.
UIST 2025
-
Multi-Click: Cross-Tab Web Automation via Action Generalization
Multi-Click lets users select analogous elements within or across browser tabs and apply clicks or text entry to them together. Structural and visual matching, target previews, and an editable data table help users control repeated web actions and vary inputs without writing an automation script.
UIST 2024
-
VRCopilot: Authoring 3D Layouts with Generative Models in VR
VRCopilot combines generative AI, speech and pointing, and editable wireframes to help users create room layouts in virtual reality. Its studies found that wireframe-based creation supported greater perceived agency than automatic generation, while manual creation through multimodal specification offered the strongest sense of creativity and control.
VL/HCC 2024
-
ScrapeViz: Hierarchical Representations for Web Scraping Macros
ScrapeViz lets users create web scraping macros by demonstration and inspect their behavior through a hierarchical visual storyboard. In a study with 12 participants, its linked page views and output table helped users trace scraped data, identify anomalies, and check macros while authoring.
L@S 2024
-
CFlow: Supporting Semantic Flow Analysis of Students' Code in Programming Problems at Scale
CFlow helps instructors compare large collections of student programs through groups of semantically related statements, aligned code views, and visual summaries of common mistakes. In a study with 16 participants analyzing more than 6,000 submissions, participants answered questions about mistakes and patterns faster and identified more valid mistakes than with the baseline system.
CHI 2024
-
Towards Inclusive Source Code Readability Based on the Preferences of Programmers with Visual Impairments
A qualitative study with 16 blind and visually impaired developers examines preferences for code formatting, including naming conventions, line length, and indentation. The findings show where screen reader use changes readability needs and suggest improvements to programming languages, development environments, and style guides.
VL/HCC 2023
-
RunEx: Augmenting Regular-Expression Code Search with Runtime Values
RunEx helps programming instructors search collections of student code by combining regular expressions with constraints on values produced during execution. In a study with experienced Python programmers, most with teaching experience, participants identified coding patterns and mistakes more accurately than with tools limited to text matching.
CHI 2023
-
Colaroid: A Literate Programming Approach for Authoring Explorable Multi-Stage Tutorials
Colaroid is a Visual Studio Code extension for authoring programming tutorials that connect explanations, code differences, and runnable versions of a project. Studies found that authors could readily create tutorials and that readers used intermediate versions to explore the code at their own pace.
-
VizProg: Identifying Misunderstandings by Visualizing Students' Coding Progress
VizProg visualizes students’ changing code on a two-dimensional map to help instructors follow progress and recognize different approaches during programming exercises. In a comparison using replays of students’ work, participants identified more misunderstandings and answered questions about students’ problems more accurately than with an adapted OverCode baseline.
-
VRGit: A Version Control System for Collaborative Content Creation in Virtual Reality
VRGit brings version control into virtual reality through a graph of miniature 3D scenes that users can navigate, branch, and reuse. A laboratory study found that its previews, shared history views, and portals helped participants manage design alternatives and discuss changes while collaborating on virtual room layouts.
VL/HCC 2022
-
Accessibility of UI Frameworks and Libraries for Programmers with Visual Impairments
This paper examines the accessibility of UI frameworks and libraries through a mailing-list analysis and interviews with 18 programmers with visual impairments. It shows how interactions among frameworks, development tools, operating systems, and screen readers complicate coding and collaboration, and recommends clearer accessibility documentation and better support for seeking help.
-
ParamMacros: Creating UI Automation Leveraging End-User Natural Language Parameterization
ParamMacros lets people create website automation by marking parameters and possible values in a natural language question, then demonstrating how to answer it. The system uses that input to generalize the demonstration, and a laboratory study found that participants could specify meaningful parameters and saw value in the approach.
CSCW 2021
-
PuzzleMe: Leveraging Peer Assessment for In-Class Programming Exercises
PuzzleMe supports peer assessment during programming exercises by letting students share lightweight tests and review code in groups matched by their solutions. Classroom deployments showed how these features helped students identify bugs, correct misunderstandings, and explore alternative approaches, while also revealing that peer discussions sometimes needed encouragement.
-
Understanding Accessibility and Collaboration in Programming for People with Visual Impairments
This interview study examines the workplace experiences of 22 programmers with visual impairments, including how they set up tools and collaborate with sighted colleagues. It shows that inaccessible software and visually oriented team practices create additional work, from developing technical workarounds to explaining access needs and negotiating changes to shared workflows.
VL/HCC 2021
-
Understanding the Challenges and Needs of Programmers Writing Web Automation Scripts
This paper investigates how programmers write automation scripts for third-party websites through two studies, including an evaluation of a prototype editor that shows page context and execution feedback. It identifies difficulties selecting elements consistently across pages and recommends feedback about selected elements, interaction results, and changes to websites over time.
CHI 2021
-
CoCapture: Effectively Communicating UI Behaviors on Existing Websites by Demonstrating and Remixing
CoCapture lets designers communicate changes to an existing website by recording its behavior, demonstrating and editing new behavior, and linking written requests to the resulting mockups. Study participants described desired UI behavior more accurately than with conventional sketching and communication tools, and reviewers found those descriptions easier to follow.
-
Think-Aloud Computing: Supporting Rich and Low-Effort Knowledge Capture
Think-aloud computing captures spoken explanations, design decisions, and unresolved problems while people work, using prompts and links to application context. Studies across programming, slide creation, and 3D modeling found that it captured information missing from traditional documentation, with participants reporting similar overall effort but different sources of disruption.
UIST 2020
-
FlowMatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality
FlowMatic lets programmers create interactive virtual reality scenes while immersed in them, using visual dataflow connections to define reactions to events, create objects, and reuse behaviors. A study with eight programmers found that participants could build the requested scenes and valued the direct manipulation and immediate feedback.
VL/HCC 2020
-
A Hybrid Crowd-Machine Workflow for Program Synthesis
BashOn combines automated shell-command synthesis with crowd workers who identify parts of natural language requests and check the outputs of candidate commands. Experiments showed that these targeted contributions improved command-generation accuracy over the automated system alone, even though the workers lacked Bash expertise, while leaving some requests without a returned answer.
-
EdCode: Towards Personalized Support at Scale for Remote Assistance in CS Education
EdCode lets programming students request help within their code editor and lets instructors link explanations to relevant code, follow up through chat, and share selected answers with the class. Usability studies found that students valued the contextual support and could understand shared answers even when unrelated code was hidden.
IMX 2020
-
Sifter: A Hybrid Workflow for Theme-based Video Curation at Scale
Sifter combines automated video filtering with crowd workers who select videos and reach agreement on material for themed compilations. A comparison covering 12 compilations found faster video selection than staff curators and similar relevance ratings for 11 compilations, showing how the workflow can assist editorial curation.
CHI 2020
-
Callisto: Capturing the "Why" by Connecting Conversations with Computational Narratives
Callisto connects discussions to notebook cells and their editing history so collaborators can recover the reasoning behind a data analysis. In a study with data science students, these links helped newcomers understand an ongoing project more accurately, including earlier alternatives that no longer appeared in the final notebook.
-
Explore, Create, Annotate: Designing Digital Drawing Tools with Visually Impaired People
This study examines how 11 adults with visual impairments explore, create, and label tactile maps using braille and audio. Audio labels supported faster answers to questions about the maps, while braille supported more varied exploration strategies; the findings inform drawing tools that accommodate both labeling methods and familiar interactions.
-
Improving Crowd-Supported GUI Testing with Structural Guidance
This paper presents interactive event-flow graphs and guidance within the interface to help crowd testers find unexplored interaction sequences. In a study with 30 testers across 11 test pages, the techniques increased transition coverage and prevented repeated transitions, helping both trained and untrained testers make more thorough tests.
CSCW 2019
-
How Data Scientists Use Computational Notebooks for Real-Time Collaboration
This study combines a survey of 195 data scientists with observations of 24 people collaborating remotely in pairs to compare shared and individual computational notebooks. Synchronous editing helped participants maintain shared context and explore alternatives, while also creating challenges with uneven participation, interfering edits, and coordinating work.
VL/HCC 2019
-
Studying the Benefits and Challenges of Immersive Dataflow Programming
This exploratory study examines immersive dataflow programming through a prototype authoring tool and interviews with seven participants who used it to build virtual scenes. Participants valued direct interaction and immediate feedback but struggled with connecting nodes, cluttered diagrams, and navigation, motivating design changes that combine two-dimensional program layouts with three-dimensional scenes.
CHI 2019
-
Implementing Multi-Touch Gestures with Touch Groups and Cross Events
This paper introduces touch groups, which summarize multiple fingers, and cross events, which detect when fingers cross a specified path, to simplify programming custom multi-touch gestures. A readability study found that programmers understood gesture implementations faster and more accurately with these primitives than with standard touch events.
CSCW 2018
-
Creating Guided Code Explanations with chat.codes
The chat.codes tool links written explanations to specific code regions, records intermediate versions, and displays code changes within the conversation. Studies of authors and readers found these features useful for creating and following step-by-step explanations, with introductory programming students completing comprehension tasks faster when code references and history were available.
UIST 2018
-
Adasa: A Conversational In-Vehicle Digital Assistant for Advanced Driver Assistance Features
Adasa is a conversational assistant that uses natural-language questions, commands, and vehicle signals to explain and control driver assistance features. In a road study with 15 drivers, participants found it useful for understanding lane keeping and adaptive cruise control and reported that its spoken responses helped them use these features.
-
Arboretum and Arbility: Improving Web Accessibility Through a Shared Browsing Architecture
Arboretum shares selected parts of a browsing session with remote helpers while letting users accept or reject proposed actions, and Arbility applies this architecture to web accessibility. In a study with nine blind users, collaboration with sighted crowd workers helped retrieve information that was difficult or impossible to access using screen readers alone.
VL/HCC 2018
-
Expresso: Building Responsive Interfaces with Keyframes
Expresso lets users create responsive web interfaces by arranging elements at representative viewport sizes and defining transitions between these keyframes. Smooth transitions interpolate property values as the viewport changes, while jump transitions create abrupt layout changes, allowing custom responsive behavior without writing CSS.
CHI 2017
-
Codeon: On-Demand Software Development Assistance
Codeon lets developers request help aloud within their editor, automatically shares the relevant code context, and integrates asynchronous responses from remote programmers. In a laboratory study, developers completed more programming tasks than with synchronous voice and code sharing, supported by the ability to hand off work and continue in parallel.
UIST 2016
-
CodeMend: Assisting Interactive Programming with Bimodal Embedding
CodeMend uses natural-language requests and existing code to recommend API functions and parameters for editing Python visualizations. Its interface highlights code to change, previews suggested edits, and lets users choose parameter values drawn from code examples, helping them find and integrate changes within their current program.
CHI 2016
-
Towards Providing On-Demand Expert Support for Software Developers
This paper reports two studies of the support developers want from remote experts: requests to an imagined ideal assistant and collaboration with human helpers. The findings identify eight requirements for future support tools, including sharing code context, responding promptly, and tailoring help to the developer’s goals.
UIST 2014
-
InterState: A Language and Environment for Expressing Interface Behavior
InterState combines state machines, constraints, and a live visual editor to help programmers write and reuse user interface behaviors. In a study with 20 experienced programmers, participants completed two interface modification tasks in roughly half the time required with conventional JavaScript event-callback code.
CHI 2013
-
ZoomBoard: A Diminutive QWERTY Soft Keyboard Using Iterative Zooming for Ultra-Small Devices
ZoomBoard uses repeated zooming to make a familiar QWERTY keyboard usable on very small touchscreens. In a study with six participants using a tiny keyboard displayed on an iPad, typing speed reached an average of 9.3 words per minute by the eighth trial, with fewer errors than a non-zooming keyboard.
UIST 2012
-
ConstraintJS: Programming Interactive Behaviors for the Web by Integrating Constraints and States
ConstraintJS combines constraints and finite state machines with HTML, CSS, and JavaScript to express relationships that change with an interface’s state. Example applications show how this approach can coordinate layout, asynchronous data, and interaction behaviors while reducing the need for separate event-callback code.
ICSE 2012
-
Inferring Method Specifications from Natural Language API Descriptions
This paper presents a method for turning natural language API descriptions into formal code contracts. In an evaluation of C# file-system and Facebook API documentation, it identified contract sentences with about 92% precision and 93% recall, then inferred their specifications with about 83% accuracy.
CHI 2012
-
Codelets: Linking Interactive Documentation and Example Code in the Editor
Codelets keep example code connected to interactive explanations and customization tools inside the programmer’s editor, including after the example has been inserted and modified. In a study with 20 programmers, participants completed guided jQuery Mobile tasks faster and with fewer preview refreshes than participants using conventional web documentation.
IS-EUD 2011
-
Playbook: Revision Control & Comparison for Interactive Mockups
Playbook helps interface designers keep versions of interactive mockups created from layered Photoshop files and simple behavior scripts. Designers can compare alternatives side by side by mapping equivalent actions and responses, allowing a single interaction to drive multiple mockups even when their interfaces differ.
CHI 2010
-
How to Support Designers in Getting Hold of the Immaterial Material of Software
This paper reports two participatory design workshops on how interaction designers conceive, refine, and communicate novel interface controls. The findings motivate tools that preserve gestures, examples, usage scenarios, and design rationale so designers and developers can continue refining an interaction together instead of treating static sketches as complete specifications.
VL/HCC 2009
-
FireCrystal: Understanding Interactive Behaviors in Dynamic Web Pages
FireCrystal is a Firefox extension for recording and inspecting interactive behavior in web pages. Its replay timeline lets developers revisit visible changes, select elements of interest, and inspect associated HTML, CSS, and JavaScript, using execution history to help locate potentially relevant code for understanding and reuse.