Steve Oney
Pronouns: he/him
Steve Oney is an Associate Professor at the University of Michigan School of Information and Computer Science and Engineering (by courtesy). His research focuses on enabling and encouraging more people to write and customize computer programs by creating new programming tools and exploring usability issues in programming environments. Steve completed his Ph.D in Carnegie Mellon’s Human-Computer Interaction Institute where he was advised by Professor Brad Myers and Dr. Joel Brandt. He also attended MIT (CS & math S.B. in 2007, CS M.Eng in 2008).
Publications
-
Editrail: Understanding AI Usage by Visualizing Student-AI Interaction in Code
Editrail visualizes AI contributions alongside students’ code editing histories to help instructors understand how students use and adapt generated code. In a comparison study, participants identified AI usage patterns more accurately than with the baseline interface and used the histories to decide where targeted guidance could help.
-
CodeStream: Augmenting Timelines with Code Annotation for Navigating Large Coding Histories
CodeStream combines code annotations, semantic zooming, and cumulative effort indicators to help instructors navigate students’ programming histories. In a study with 12 participants, it improved accuracy in identifying coding patterns compared with a baseline timeline while requiring comparable analysis time.
-
Navigating Complexity: How Context Shapes Debugging Strategy Choices Among Expert Developers
A survey of 35 web developers and interviews with 16 expert web developers examine how context shapes debugging strategy choices. The study identifies static and dynamic contextual factors and a descriptive state-transition model showing how experts adapt their strategies as clarity, reproducibility, and constraints change.
-
Co-Advisor: Learning Programming Strategies in Context
Co-Advisor uses code context, runtime information, and programmer actions to provide debugging strategies with personalized feedback in Visual Studio Code. In a formative study with 14 programmers, participants made more debugging progress with contextual feedback than with non-adaptive strategies, suggesting a way to support strategic knowledge during problem solving.
-
ConvoMap: Interactive Visualizations for Exploring Complex Conversations in Multi-Agent Systems
ConvoMap helps developers inspect conversations among AI agents through automated topic coding, a semantic map, and linked message views. In a comparison study with 16 participants, it improved accuracy in understanding agent behavior and conversation patterns, supporting investigation of recurring problems across multiple runs.
-
SPARK: Real-Time Monitoring of Multi-Faceted Programming Exercises
SPARK helps instructors monitor programming exercises with interdependent steps and varied outputs by organizing progress around instructor-defined checkpoints. It suggests tests, visualizes progress, and exposes intermediate outputs; a study with 16 participants found more accurate identification of student difficulties than with the baseline dashboard.
-
Multi-Click: Cross-Tab Web Automation via Action Generalization
Multi-Click lets users select analogous elements within or across browser tabs and apply clicks or text entry to them together. Structural and visual matching, target previews, and an editable data table help users control repeated web actions and vary inputs without writing an automation script.
-
VRCopilot: Authoring 3D Layouts with Generative Models in VR
VRCopilot combines generative AI, speech and pointing, and editable wireframes to help users create room layouts in virtual reality. Its studies found that wireframe-based creation supported greater perceived agency than automatic generation, while manual creation through multimodal specification offered the strongest sense of creativity and control.
-
ScrapeViz: Hierarchical Representations for Web Scraping Macros
ScrapeViz lets users create web scraping macros by demonstration and inspect their behavior through a hierarchical visual storyboard. In a study with 12 participants, its linked page views and output table helped users trace scraped data, identify anomalies, and check macros while authoring.
-
CFlow: Supporting Semantic Flow Analysis of Students' Code in Programming Problems at Scale
CFlow helps instructors compare large collections of student programs through groups of semantically related statements, aligned code views, and visual summaries of common mistakes. In a study with 16 participants analyzing more than 6,000 submissions, participants answered questions about mistakes and patterns faster and identified more valid mistakes than with the baseline system.
-
Towards Inclusive Source Code Readability Based on the Preferences of Programmers with Visual Impairments
A qualitative study with 16 blind and visually impaired developers examines preferences for code formatting, including naming conventions, line length, and indentation. The findings show where screen reader use changes readability needs and suggest improvements to programming languages, development environments, and style guides.
-
RunEx: Augmenting Regular-Expression Code Search with Runtime Values
RunEx helps programming instructors search collections of student code by combining regular expressions with constraints on values produced during execution. In a study with experienced Python programmers, most with teaching experience, participants identified coding patterns and mistakes more accurately than with tools limited to text matching.
-
Colaroid: A Literate Programming Approach for Authoring Explorable Multi-Stage Tutorials
Colaroid is a Visual Studio Code extension for authoring programming tutorials that connect explanations, code differences, and runnable versions of a project. Studies found that authors could readily create tutorials and that readers used intermediate versions to explore the code at their own pace.
-
VizProg: Identifying Misunderstandings by Visualizing Students' Coding Progress
VizProg visualizes students’ changing code on a two-dimensional map to help instructors follow progress and recognize different approaches during programming exercises. In a comparison using replays of students’ work, participants identified more misunderstandings and answered questions about students’ problems more accurately than with an adapted OverCode baseline.
-
VRGit: A Version Control System for Collaborative Content Creation in Virtual Reality
VRGit brings version control into virtual reality through a graph of miniature 3D scenes that users can navigate, branch, and reuse. A laboratory study found that its previews, shared history views, and portals helped participants manage design alternatives and discuss changes while collaborating on virtual room layouts.
-
Accessibility of UI Frameworks and Libraries for Programmers with Visual Impairments
This paper examines the accessibility of UI frameworks and libraries through a mailing-list analysis and interviews with 18 programmers with visual impairments. It shows how interactions among frameworks, development tools, operating systems, and screen readers complicate coding and collaboration, and recommends clearer accessibility documentation and better support for seeking help.
-
ParamMacros: Creating UI Automation Leveraging End-User Natural Language Parameterization
ParamMacros lets people create website automation by marking parameters and possible values in a natural language question, then demonstrating how to answer it. The system uses that input to generalize the demonstration, and a laboratory study found that participants could specify meaningful parameters and saw value in the approach.
-
PuzzleMe: Leveraging Peer Assessment for In-Class Programming Exercises
PuzzleMe supports peer assessment during programming exercises by letting students share lightweight tests and review code in groups matched by their solutions. Classroom deployments showed how these features helped students identify bugs, correct misunderstandings, and explore alternative approaches, while also revealing that peer discussions sometimes needed encouragement.
-
Understanding Accessibility and Collaboration in Programming for People with Visual Impairments
This interview study examines the workplace experiences of 22 programmers with visual impairments, including how they set up tools and collaborate with sighted colleagues. It shows that inaccessible software and visually oriented team practices create additional work, from developing technical workarounds to explaining access needs and negotiating changes to shared workflows.
-
Understanding the Challenges and Needs of Programmers Writing Web Automation Scripts
This paper investigates how programmers write automation scripts for third-party websites through two studies, including an evaluation of a prototype editor that shows page context and execution feedback. It identifies difficulties selecting elements consistently across pages and recommends feedback about selected elements, interaction results, and changes to websites over time.
-
CoCapture: Effectively Communicating UI Behaviors on Existing Websites by Demonstrating and Remixing
CoCapture lets designers communicate changes to an existing website by recording its behavior, demonstrating and editing new behavior, and linking written requests to the resulting mockups. Study participants described desired UI behavior more accurately than with conventional sketching and communication tools, and reviewers found those descriptions easier to follow.
-
Think-Aloud Computing: Supporting Rich and Low-Effort Knowledge Capture
Think-aloud computing captures spoken explanations, design decisions, and unresolved problems while people work, using prompts and links to application context. Studies across programming, slide creation, and 3D modeling found that it captured information missing from traditional documentation, with participants reporting similar overall effort but different sources of disruption.
-
FlowMatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality
FlowMatic lets programmers create interactive virtual reality scenes while immersed in them, using visual dataflow connections to define reactions to events, create objects, and reuse behaviors. A study with eight programmers found that participants could build the requested scenes and valued the direct manipulation and immediate feedback.
-
A Hybrid Crowd-Machine Workflow for Program Synthesis
BashOn combines automated shell-command synthesis with crowd workers who identify parts of natural language requests and check the outputs of candidate commands. Experiments showed that these targeted contributions improved command-generation accuracy over the automated system alone, even though the workers lacked Bash expertise, while leaving some requests without a returned answer.
-
EdCode: Towards Personalized Support at Scale for Remote Assistance in CS Education
EdCode lets programming students request help within their code editor and lets instructors link explanations to relevant code, follow up through chat, and share selected answers with the class. Usability studies found that students valued the contextual support and could understand shared answers even when unrelated code was hidden.
-
Sifter: A Hybrid Workflow for Theme-based Video Curation at Scale
Sifter combines automated video filtering with crowd workers who select videos and reach agreement on material for themed compilations. A comparison covering 12 compilations found faster video selection than staff curators and similar relevance ratings for 11 compilations, showing how the workflow can assist editorial curation.
-
Callisto: Capturing the "Why" by Connecting Conversations with Computational Narratives
Callisto connects discussions to notebook cells and their editing history so collaborators can recover the reasoning behind a data analysis. In a study with data science students, these links helped newcomers understand an ongoing project more accurately, including earlier alternatives that no longer appeared in the final notebook.
-
Explore, Create, Annotate: Designing Digital Drawing Tools with Visually Impaired People
This study examines how 11 adults with visual impairments explore, create, and label tactile maps using braille and audio. Audio labels supported faster answers to questions about the maps, while braille supported more varied exploration strategies; the findings inform drawing tools that accommodate both labeling methods and familiar interactions.
-
Improving Crowd-Supported GUI Testing with Structural Guidance
This paper presents interactive event-flow graphs and guidance within the interface to help crowd testers find unexplored interaction sequences. In a study with 30 testers across 11 test pages, the techniques increased transition coverage and prevented repeated transitions, helping both trained and untrained testers make more thorough tests.
-
How Data Scientists Use Computational Notebooks for Real-Time Collaboration
This study combines a survey of 195 data scientists with observations of 24 people collaborating remotely in pairs to compare shared and individual computational notebooks. Synchronous editing helped participants maintain shared context and explore alternatives, while also creating challenges with uneven participation, interfering edits, and coordinating work.
-
Studying the Benefits and Challenges of Immersive Dataflow Programming
This exploratory study examines immersive dataflow programming through a prototype authoring tool and interviews with seven participants who used it to build virtual scenes. Participants valued direct interaction and immediate feedback but struggled with connecting nodes, cluttered diagrams, and navigation, motivating design changes that combine two-dimensional program layouts with three-dimensional scenes.
-
Implementing Multi-Touch Gestures with Touch Groups and Cross Events
This paper introduces touch groups, which summarize multiple fingers, and cross events, which detect when fingers cross a specified path, to simplify programming custom multi-touch gestures. A readability study found that programmers understood gesture implementations faster and more accurately with these primitives than with standard touch events.
-
Creating Guided Code Explanations with chat.codes
The chat.codes tool links written explanations to specific code regions, records intermediate versions, and displays code changes within the conversation. Studies of authors and readers found these features useful for creating and following step-by-step explanations, with introductory programming students completing comprehension tasks faster when code references and history were available.
-
Adasa: A Conversational In-Vehicle Digital Assistant for Advanced Driver Assistance Features
Adasa is a conversational assistant that uses natural-language questions, commands, and vehicle signals to explain and control driver assistance features. In a road study with 15 drivers, participants found it useful for understanding lane keeping and adaptive cruise control and reported that its spoken responses helped them use these features.
-
Arboretum and Arbility: Improving Web Accessibility Through a Shared Browsing Architecture
Arboretum shares selected parts of a browsing session with remote helpers while letting users accept or reject proposed actions, and Arbility applies this architecture to web accessibility. In a study with nine blind users, collaboration with sighted crowd workers helped retrieve information that was difficult or impossible to access using screen readers alone.
-
Expresso: Building Responsive Interfaces with Keyframes
Expresso lets users create responsive web interfaces by arranging elements at representative viewport sizes and defining transitions between these keyframes. Smooth transitions interpolate property values as the viewport changes, while jump transitions create abrupt layout changes, allowing custom responsive behavior without writing CSS.
-
Codeon: On-Demand Software Development Assistance
Codeon lets developers request help aloud within their editor, automatically shares the relevant code context, and integrates asynchronous responses from remote programmers. In a laboratory study, developers completed more programming tasks than with synchronous voice and code sharing, supported by the ability to hand off work and continue in parallel.
-
CodeMend: Assisting Interactive Programming with Bimodal Embedding
CodeMend uses natural-language requests and existing code to recommend API functions and parameters for editing Python visualizations. Its interface highlights code to change, previews suggested edits, and lets users choose parameter values drawn from code examples, helping them find and integrate changes within their current program.
-
Towards Providing On-Demand Expert Support for Software Developers
This paper reports two studies of the support developers want from remote experts: requests to an imagined ideal assistant and collaboration with human helpers. The findings identify eight requirements for future support tools, including sharing code context, responding promptly, and tailoring help to the developer’s goals.
-
InterState: A Language and Environment for Expressing Interface Behavior
InterState combines state machines, constraints, and a live visual editor to help programmers write and reuse user interface behaviors. In a study with 20 experienced programmers, participants completed two interface modification tasks in roughly half the time required with conventional JavaScript event-callback code.
-
ZoomBoard: A Diminutive QWERTY Soft Keyboard Using Iterative Zooming for Ultra-Small Devices
ZoomBoard uses repeated zooming to make a familiar QWERTY keyboard usable on very small touchscreens. In a study with six participants using a tiny keyboard displayed on an iPad, typing speed reached an average of 9.3 words per minute by the eighth trial, with fewer errors than a non-zooming keyboard.
-
ConstraintJS: Programming Interactive Behaviors for the Web by Integrating Constraints and States
ConstraintJS combines constraints and finite state machines with HTML, CSS, and JavaScript to express relationships that change with an interface’s state. Example applications show how this approach can coordinate layout, asynchronous data, and interaction behaviors while reducing the need for separate event-callback code.
-
Inferring Method Specifications from Natural Language API Descriptions
This paper presents a method for turning natural language API descriptions into formal code contracts. In an evaluation of C# file-system and Facebook API documentation, it identified contract sentences with about 92% precision and 93% recall, then inferred their specifications with about 83% accuracy.
-
Codelets: Linking Interactive Documentation and Example Code in the Editor
Codelets keep example code connected to interactive explanations and customization tools inside the programmer’s editor, including after the example has been inserted and modified. In a study with 20 programmers, participants completed guided jQuery Mobile tasks faster and with fewer preview refreshes than participants using conventional web documentation.
-
Playbook: Revision Control & Comparison for Interactive Mockups
Playbook helps interface designers keep versions of interactive mockups created from layered Photoshop files and simple behavior scripts. Designers can compare alternatives side by side by mapping equivalent actions and responses, allowing a single interaction to drive multiple mockups even when their interfaces differ.
-
How to Support Designers in Getting Hold of the Immaterial Material of Software
This paper reports two participatory design workshops on how interaction designers conceive, refine, and communicate novel interface controls. The findings motivate tools that preserve gestures, examples, usage scenarios, and design rationale so designers and developers can continue refining an interaction together instead of treating static sketches as complete specifications.
-
FireCrystal: Understanding Interactive Behaviors in Dynamic Web Pages
FireCrystal is a Firefox extension for recording and inspecting interactive behavior in web pages. Its replay timeline lets developers revisit visible changes, select elements of interest, and inspect associated HTML, CSS, and JavaScript, using execution history to help locate potentially relevant code for understanding and reuse.
Teaching
-
Practical JavaScript: From Scratch to React (Coursera Specialization)
-
Co-Instructor – SI 506 (Programming I)
-
Co-Instructor – SI 347 (Human-Computer Interaction)
-
Instructor (and creator) – SI 379 (Building Interactive Applications)
-
Instructor (and creator) – SI 579 (Building Interactive Applications)
-
Instructor – SI 106 (Programs, Information, & People)
-
Python 3 Programming Specialization