Paper Reading #6: TurKit

TurKit

by: Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller


Authors
Greg Little and Max Goldman are both graduate students doing research at MIT in the CSAIL along with Robert Miller, an associate professor.  Lydia Chilton is a graduate student at the University of Washington.

Presentation Venue
"UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology" as per the document specs from the ACM digital library, dl.acm.org. The presentation took place in New York City, NY, 2010.

Summary


TurKit introduces a programming model called Crash-and-rerun for algorithms that require human computation.  It is intended to allow developers to avoid rerunning high cost operations when developing.  The technology for this already exists, but was developed as a wrapper to facilitate usability.

Hypothesis
Algorithms which require human computation are hard to fit into a standard algorithm.  TurKit was designed as a wrapper for MTurk that will store expensive operations so that the program can crash, be altered, and rerun without needing to redo things such as ask users for an answer to some problem.  In essence, this means their hypothesis was to make this system usable.

Methods
TurKit uses a database and a new primitive to store expensive operations.  This means that an expensive function is wrapped by the once primitive which stores the value to the database and simply uses that value in the future.  Then, when the program crashes, it automatically restarts and avoids recalculating the values by using their stored values in the database.  There were a couple of examples of people using TurKit in their applications with moderate success.

Results
There is little indication concerning preference for TurKit and its Crash-and-rerun model over some other model, and there was indication of confusion with using TurKit.  They were successful in creating a wrapper system that could store expensive items, however at some scalability costs.  TurKit sacrificed scalability for program usability.

Discussion


I have mixed feelings about this toolkit and programming model.  On the one hand, I really love the idea of being able to save pieces of a program but allow debug printing and altering along the way for on-the-fly iterative development.  But, up until the end of the report I was wondering what sort of performance TurKit would actually be able to handle since size was limited.  I was skeptical at best, and their declaration that it was done for usability didn't really smooth my ruffled feathers.  Their testing numbers of "1000 HITs" suggested to me that it would be feasible only for relatively small individual research programs, but would not be usable for larger projects at all.  Since I consider most worthwhile projects, aside from small apps and games, that require user input to be quite large and complex, I don't think it's useful for that.  I hope that it's a leap board for similar programming models in the future - although I do still acknowledge that storing individual expensive operations does not solve asymptotically expensive algorithms.