ParamMacros: Creating UI Automation Leveraging End-User Natural Language Parameterization

Rebecca Krosnick and Steve Oney

VL/HCC 2022

PDF Publisher page

Abstract

Prior work in programming-by-demonstration (PBD) has explored ways to enable end-users to create custom automation without needing to write code. We propose a new end-user specification model – asking the end-user to explicitly identify parts of their natural language query that can be generalized. We built a PBD system, ParamMacros, where users first generalize a concrete natural language question – identifying parameters and their possible values – and then create a demonstration of how to answer the question on the website of interest. ParamMacros then infers a generalized program by using the user-provided parameter values to identify relevant patterns in the website’s structure. In a lab study we found that participants were able to meaningfully parameterize natural language queries and felt such a parameterization and demonstration process would be useful for creating custom automation.

Cite this paper (BibTeX)

@inproceedings{krosnick2022parammacros,
  author    = {Krosnick, Rebecca and Oney, Steve},
  title     = {{ParamMacros}: Creating {UI} Automation Leveraging End-User Natural Language Parameterization},
  booktitle = {IEEE Symposium on Visual Languages and Human-Centric Computing},
  year      = {2022},
  doi       = {10.1109/vl/hcc53370.2022.9833005},
  url       = {https://doi.org/10.1109/vl/hcc53370.2022.9833005}
}