@JohnHolbein1
“These findings provide clear evidence that data collected on MTurk simply cannot be trusted.” Researchers have long argued about whether Amazon Mechanical Turk (MTurk) survey data can be trusted. This paper takes a simple approach to evaluating the quality of data currently produced by MTurk. The author gives respondents pairs of questions that are obviously contradictory. For example: "I talk a lot" and "I rarely talk." Or: "I like order" and "I crave chaos." If people are paying attention, agreeing with one should mean disagreeing with the other. At minimum, the two answers shouldn’t move together. The same exact survey is fielded on three platforms: Prolific, CloudResearch Connect, and MTurk. On Prolific and Connect, things behave normally: most contradictory items are negatively correlated, just as common sense predicts. On MTurk, however, the results are the opposite. Over 96% of these clearly opposite item pairs are positively correlated. In other words, many respondents give similar answers to statements that literally contradict each other. The authors then try what most researchers would do next: -restrict the sample to "high-reputation" MTurk workers -apply standard attention checks -drop fast responders and straight-liners None of it fixes the problem. Even after aggressive screening, many contradictory items remain positively correlated on MTurk. The implication is severe: careless responding on MTurk isn’t rare noise; it’s systematic enough to flip the sign of relationships and generate results that are the opposite of what they really are. Wow; this is damning.
