ScrapeViz: Hierarchical Representations for Web Scraping Macros

Rebecca Krosnick and Steve Oney

VL/HCC 2024

PDF Markdown Publisher page

Abstract

Programming-by-demonstration (PBD) makes it possible to create web scraping macros without writing code. However, it can still be challenging for users to understand the exact scraping behavior that is inferred and to verify that the scraped data is correct, especially when scraping occurs across multiple pages. We present ScrapeViz, a new PBD tool for authoring and visualizing hierarchical web scraping macros. ScrapeViz’s key novelty is in providing a visual representation of web scraping macros—the sequences of pages visited, generalized scraping behavior across similar pages, and data provenance. We conducted a lab study with 12 participants comparing ScrapeViz to the existing web scraping tool Rousillon and saw that participants found ScrapeViz helpful for understanding high-level scraping behavior, tracing the source of scraped data, identifying anomalies, and validating macros while authoring.

Cite this paper (BibTeX)

@inproceedings{krosnick2024scrapeviz,
  author    = {Krosnick, Rebecca and Oney, Steve},
  title     = {{ScrapeViz}: Hierarchical Representations for Web Scraping Macros},
  booktitle = {IEEE Symposium on Visual Languages and Human-Centric Computing},
  year      = {2024},
  doi       = {10.1109/vl/hcc60511.2024.00040},
  url       = {https://doi.org/10.1109/vl/hcc60511.2024.00040}
}