Yet another valiant attempt...: What does Turker Nation think of your brand?

I’ve completed the information gathering phase of my current research effort—exploring the relationship between brand opinion and crowdworker engagement. While I’m still churning through the data analysis, I thought I’d share some descriptive statistics with you to give you a sense of what’s coming. For an overview of how I produced these numbers, see the Methodology Overview (below).

Turker opinions on US consumer brands

As part of the survey methodology, each participating “turker” (a worker on Amazon Mechanical Turk, a crowdworker platform) was asked their opinion of two (2) well-known US consumer brands. They were asked to rank their opinion on a scale from “1 – Poor” to “5 – Excellent”. While the average score of each brand is not specifically relevant to the research question, the results do paint an interesting picture, as indicated by the following graphic.

What’s interesting is the manner in which some of the ratings skew when compared to other measures of brand performance. Consider Volkswagen, which respondees scored third highest out of the 35 brands evaluated: or Nintendo, which scored fifth highest. In the 2011 Interbrand Global Top 100 Brand Rankings⁽¹⁾, these two brands placed 47th and 48th, respectively. This is well back in the pack from Google (Interbrand #4), Nike (#25) and Coca-Cola (#1). The survey did include an open response question, asking for a rationale of their opinion. I hope that an analysis of this information will provide additional insight and testable hypotheses.

Knowing the identity of the task provider

While some companies are attempting to maintain anonymity within Amazon Mechanical Turk (“AMT”), this is likely to be counterproductive. As the following chart indicates, a third of Turkers think it’s “very important” to know for whom they’re working, and another two-fifths are at least interested.

There are a variety of reasons for wanting to know, and the subject deserves further analysis. An obvious reason, identified by reviewing discussions on the Turker Nation discussion site (http://turkernation.com/) is that there have been issues with payment from certain requestors. Also, new requestors are viewed with suspicion until they’ve demonstrated their consistency and reliability. Other Turkers are selective about the organizations for whom their willing to work—this is the focus of my research, and will be covered in a future post.

Methodology Overview

Over the period from April 5th to April 19th, 2012, I published a worker task for completion on Amazon Mechanical Turk (“AMT” - http://mturk.com). The task was a survey comprised of five (5) questions to gather information on brand / company opinion and task selection drivers. The task was published to AMT workers (“Turkers”) located in the United States (to provide for consistency of brand awareness) who have completed more than 500 tasks with a 90%+ approval rate on their completed tasks. These filters were implemented per suggested best practices to avoid potential bots and spammers.

Regarding the representativeness of AMT workers, there have been numerous studies performed that indicate that “Mechanical Turk workers are at least as representative of the U.S. population as traditional subject pools, with gender, race, age and education of Internet samples all matching the population more closely than college undergraduate samples and internet samples in general.”⁽²⁾ As result, this initial study chose not to collect additional demographic data, although future research may extend into this area.

Out of a pool of 1,200 potential responses, I received 1,002 task submissions. After eliminating incomplete responses and multiple responses from the same worker (to reduce sample bias) I was left with 728 valid responses on which to base analysis. On average, a worker spent 1 minute and 48 seconds completing the survey.

Notes

(1) "Best Global Brands 2011." Interbrand. 2011. Web. 22 Apr. 2012.

(2) Paolacci, Gabriele, Jesse Chandler, and Panagiotis Ipeirotis. "Running Experiments on Amazon Mechanical Turk." Judgment and Decision Making. Society for Judgement and Decision Making, Aug. 2010. Web. 22 Apr. 2012.