Experiment Feedback Page

Background

This page has been developed to provide feedback to participants in the Tag-a-Texture experiment on Amazon Mechanical Turk. This trial used multilevel textures to investigate how the human visual system distinguishes textured from completely random patterns.


Progress of the Experiment, Statistics and Feedback from workers.

We had in total 13 batches. Each of these batches contained 30 HITs. Each HIT had 32 assignments. This gives a total of 12480 HITs. We thanks the workforce of 512 turkers, we very much appreciated your work and time. Also we gathered many good suggestions on how to improve the experiment management.


We revised both the approval and bonus polices for this experiment. As the instructions were too verbose, many workers had the first HIT of a series expire during execution. The average time of completion was about 3 min, so it was indeed very tight to complete all 30 in order to get a bonus. So we revised that for anyone that completed a set of 28 or more, meaning 41 workers got a bonus.


Previously we had a system of catch trials for this experiment, in order to estabilish if a worker was performing the task unatentive. These catch trials were 8 very easy to spot bands. We expected users to get 3 of the 8 right. The rationale was that the 4 catch trials on 0.2s presentation time was harder to spot. So anyone missing just one catch trial on the 2s presentation time part would get his work accepted. About 20% fail the catch trials. But because we did not explain this to the workers beforehand we decide to accept all work done. We will expurge work that fail the catch trials in our analysis.



Sneak peek: performances

The figures below reflect the percentage of correct answers for each of the 30 HITs within a Batch. The labeling on these graphs will look mysterious, as it is a code for how the textures were prepared and their mathematical properties. Because we have 4 band positions, for a texture family very close (indistinguishable) to random (or if a dodger were to click randomly the buttons) the average performance for all workers would be near 0.25. When we have a preprint of the first scientific paper related to this experiment, we will update this page.



Average performance against texture type. Batch[1]




Average performance against texture type. Batch[2]




Average performance against texture type. Batch[3]




Average performance against texture type. Batch[4]




Average performance against texture type. Batch[5]




Average performance against texture type. Batch[6]




Average performance against texture type. Batch[7]




Average performance against texture type. Batch[8]




Average performance against texture type. Batch[9]




Average performance against texture type. Batch[10]




Average performance against texture type. Batch[11]




Average performance against texture type. Batch[12]




Average performance against texture type. Batch[13]




Hits completed per worker, all batches.



Links to Publications and Conference Presentations

Watch this space!!!!


Related Background Publications

  1. BUHRMESTER, M., KWANG, T. & GOSLING, S. D. 2011. Amazon's Mechanical Turk : A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6, 3-5.
  2. CRUMP, M. J., MCDONNELL, J. V. & GURECKIS, T. M. 2013. Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLoS One, 8, e57410.
  3. HORTON, J. J., RAND, D. G. & ZECKHAUSER, R. J. 2011. The online laboratory: conducting experiments in a real labor market. Experimental Economics, 14, 399-425.
  4. MADDESS, T. & NAGAI, Y. 2001. Discriminating isotrigon textures [corrected]. Vision research, 41, 3837-60.
  5. MADDESS, T., NAGAI, Y., VICTOR, J. D. & TAYLOR, R. R. 2007. Multilevel isotrigon textures. Journal of the Optical Society of America. A, Optics, image science, and vision, 24, 278-93.
  6. MASON, W. & SURI, S. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behav Res Methods, 44, 1-23.
  7. PAOLACCI, G., CHANDLER, J. & IPEIROTIS, P. 2010. Running experiments on Amazon Mechanical Turk. Judgment and decision making, 5, 411-419.
  8. Tkacik, G., Prentice, J. S., Victor, J. D. & Balasubramanian, V. Local statistics in natural scenes predict the saliency of synthetic textures. Proceedings of the National Academy of Sciences 107, 18149 –18154 (2010).
  9. Victor, J. D. & Conte, M. M. Local image statistics: maximum-entropy constructions and perceptual salience. J. Opt. Soc. Am. A 29, 1313–1345 (2012).
  10. Barbosa, M. S., Bubna-Litic, A. & Maddess, T. Locally countable properties and the perceptual salience of textures. J. Opt. Soc. Am. A 30, 1687–1697 (2013).
  11. Victor, J. D., Thengone, D. J. & Conte, M. M. Perception of second- and third-order orientation signals and their interactions. J Vis 13, 21 (2013).