Research

Focus Areas

Programming Education at Scale

Instructors can see the code a class turns in, but not how any of it got written. We build tools that make a whole class's problem-solving process visible in time to act on it -- including how students use AI.

People: Ashley Zhang, Steve Oney, and Maryam Arab

Publications:

Web Automation and Programming by Demonstration

People want to automate work on websites they do not control. We study what makes that hard, and build tools that let them demonstrate what they want and then see what was inferred from the demonstration.

People: Rebecca Krosnick, Jiacheng Zhang, Maryam Arab, and Steve Oney

Publications:

Building and Testing Interactive Interfaces

Interactive behavior is scattered across event handlers, which makes it hard to write, hard to read, and hard to test. We design notations, editors, and prototyping tools for expressing behavior directly, and techniques for finding where an interface breaks.

People: Steve Oney and Rebecca Krosnick

Publications:

Explaining Code and Capturing Process

Code records the result of a decision but not the reasoning behind it. We build editors, notebooks, and visualizations that capture the intent, discussion, and process behind an artifact, and keep them attached to it.

People: April Wang, Steve Oney, and Christopher Brooks

Publications:

Accessibility of Programming and the Web

Programming tools are visual by default, which shapes who can contribute to software and in what role. We study how blind and low-vision developers actually work, alone and on mixed-ability teams, and what their tools would have to change.

People: Mauli Pandey and Steve Oney

Publications:

Immersive Authoring

Building 3D content means describing spatial things through a 2D screen. We build tools for authoring, versioning, and co-creating scenes with generative models from inside VR.

People: Lei Zhang and Steve Oney

Publications:

On-Demand Help for Programmers

A stuck programmer needs a helper who already has their context. We built systems that package that context, route it to an expert, a crowd, or a program synthesizer, and get the answer back into the editor -- mostly between 2016 and 2020.

People: Yan Chen and Steve Oney

Publications:

All Publications

  • Co-Advisor: Learning Programming Strategies in Context

    Maryam Arab, Hanning Li, Rushal Butala, and Steve Oney

    A context-aware learning tool that helps programmers develop strategic problem-solving skills like debugging and program design. Integrated with Visual Studio Code, it analyzes the learner's code, runtime state, and actions to recommend expert strategies tailored to a developer's context. As learners follow these strategies, Co-Advisor provides real-time feedback to make it easier for developers to understand strategic knowledge in context.

    VL/HCC 2025  
     PDF

  • ConvoMap: Interactive Visualizations for Exploring Complex Conversations in Multi-Agent Systems

    Ashley Zhang, Victor Bursztyn, Gromit Chan, Shunan Guo, Eunyee Koh, Steve Oney, and Jane Hoffswell

    A visualization system that helps developers understand and debug large collections of conversations between AI agents in Multi-Agent Systems (MASs). ConvoMap contains visualizations to show the flow of interactions between agents and the topics covered. ConvoMap allows MAS developers to spot behavioral patterns, trace errors, and compare outcomes at scale.

    VL/HCC 2025  
     PDF

  • SPARK: Real-Time Monitoring of Multi-Faceted Programming Exercises

    Yinuo Yang, Ashley Zhang, Steve Oney, and April Wang

    A dashboard for programming instructors that lets them monitor student progress on complex, multi-step programming exercises. SPARK lets instructors define checkpoints, automatically generates tests for each step, and visualizes progress across students. SPARK also enables inspection of intermediate outputs and runtime states to reveal common challenges and misconceptions.

    VL/HCC 2025  
    Honorable mention  
     PDF

  • Multi-Click: Cross-Tab Web Automation via Action Generalization

    Jiacheng Zhang, Jiawen Li, Maryam Arab, and Steve Oney

    Proposes "action generalization"---a technique for applying the same action across multiple contexts automatically. Multi-click lets users select a "generalization strategy" to determine which elements a given action (e.g., click or text entry) will apply to within a given page or across multiple tabs. Unlike prior web automation tools, multi-click works at the *interaction* level, rather than requiring a separate script representation of automation macros, which can improve the directness and understandability of bulk actions.

    UIST 2025  
     PDF

  • How Data Scientists Use Computational Notebooks for Real-Time Collaboration

    April Wang, Anant Mittal, Christopher Brooks, and Steve Oney

    We reported how synchronous editing in computational notebooks changes the way data scientists work together compared to working on individual notebooks through a formative survey and an observational study. Working on the synchronous notebooks improves collaboration by creating a shared context, encouraging more exploration, and reducing communication costs. However, the current synchronous editing features may lead to unbalanced participation and activity interference without strategic coordination.

    CSCW 2019  
    Best paper  
     PDF