Paper Reading #6: TurKit: Human Computation Algorithms on Mechanical Turk

Reference Information
TurKit: Human Computation Algorithms on Mechanical Turk
Greg Little, Lydia B. Chilton, Max Goldman, Robert C. Miller
Presented at UIST'10, October 3-6, 2010, New York, New York, USA

Author Bios

• Greg Little is a professor in MIT's Computer Science and Artificial Intelligence Lab. He hopes to build a computer that can house consciousness.
• Lydia B. Chilton is a graduate student at the University of Washington and interned for Microsoft Research Asia.
• Max Goldman is a graduate student at MIT's Computer Science and Artificial Intelligence Lab and User Interface Design Group.
• Robert C. Miller is an Associate Professor at MIT and leads their User Interface Design Group. He has a PhD from Carnegie Mellon.

Summary
Hypothesis
That TurKit toolkit provides an effective way of integrating iterative programming with human computation.

Methods
The authors tested various example applications in lab and evaluated performance from 20 scripts running nearly 30,000 tasks in a year. The applications included iterative writing, blurry text recognition, decision theory experimentation, psychophysics experimentation

Results

Results claim that the programming model is good for prototyping algorithmic tasks on MTurk, and that it sacrifices efficiency for programming usability.Users were unclear about which aspects of a TurKit script were stored in the execution trace, and which parts could be modified or re-ordered while others were unaware of the parallelism feature.

People tend to keep to the original style and formatting while working on paragraphs. In the blurry text recognition the final result was nearly perfect. In the decision theory experimentation TurKit was useful in coordinating the iterative nature of the process, but not necessarily very good at simulating actual human behavior. It was effective in the area of psychophysics experimentation, since calls to MTurk were embedded within a larger application.

Contents
TurKit is a toolkit to facilitate human computation algorithms, in which certain functions are delegated to a human. It uses Amazon's Mechanical Turk (MTurk) platform, which relies on paid humans to collect and organize data. TurKit uses crash-and-rerun to allow programmers to write imperative programs that rely on MTurk. Crash-and-rerun allows for modification and re-execution of a script without rerunning the more costly side functions. TurKit's implementation of this stores the trace leading to the current state. Certain actions can be designated to only run once, saving time and resolving non-determinism. Crashing a script is similar to blocking.

An extension of JavaScript, TurKit Script, provides a wrapper for MTurk. It allows for parallel computing through forks. An online interface facilitates script design and execution. Three primary MTurk functions used are text entry, voting, and sorting.

Discussion
I think the application of a Mechanical Turk system is very interesting and I like the possibility of making them easier and better to implement.