A Hybrid Crowd-Machine Workflow for Program Synthesis

Despite advances in machine learning, there has been little progress towards creating automated systems that can reliably solve difficult tasks, such as programming or scripting. In this paper, we propose techniques for increasing the reliability of automated systems for program synthesis task via a hybrid workflow that augments the system with input from crowds of human workers. Unlike previous hybrid workflow systems, which have been focused on less complex tasks that crowd workers can do in their entirety (e.g., image labeling), our pro- posed workflow handles tasks that untrained crowd workers cannot do alone (i.e., scripting). We show that we can improve the performance of an automated system by integrating crowd workers into targeted portions of the task workflow. We evaluate our approach by creating BashOn, a system that increases the accuracy of an automated program that generates Bash shell commands from natural language descriptions by nearly 30%.