Seven percent of Americans are so gullible they think chocolate milk comes straight from brown cows, right? Wrong. The gullible ones are the news reporters and their readers who unquestioningly swallowed the incredible survey result on the matter.
I have no doubt that real surveyors asked real Americans where they thought chocolate milk comes from and that seven percent said brown cows. The question is whether those respondents were telling the truth or playing along with what they reasonably concluded was a joke.
There may actually be people so ignorant that they think brown cows squirt chocolate-flavored milk from their tits. But to conclude from the survey in question that seven percent of the population thinks that you’d have to be, well, ignorant—or else a “journalist” intentionally peddling fake news to promote page clicks.
The story presents a good object lesson for critically reading—and professionally producing—the news.
One giant red flag is that the raw survey results, along with the wording of the questions, are apparently nowhere to be found. On June 1, Food and Wine published a story by Elisabeth Sherman claiming (link in the original):
The Innovation Center for U.S. Dairy conducted a survey of more than 1,000 adults 18 and over in April of this year. They uncovered some shocking facts about how people think about—and drink—milk.
First off, 48% of respondents said that they aren’t sure where chocolate milk comes from. Um, guys, it comes from cows—and not just the brown kind.
Still, 7% of people—and remember, this survey talked to actual, grown-up adults—still think that chocolate milk only comes from brown cows.
First off, if you’re not immediately skeptical of the percentages, you’re not reading or reporting the news with a sufficiently critical eye.
Let’s try to track down the survey to see how the questions were worded, because obviously survey results depend radically on the wording of the questions.
The Food and Wine story takes us to undeniablydairy.org, which redirects to dairygood.org/undeniably-dairy. And what is Undeniably Dairy? According to an article at Ad Age linked by Undeniably Dairy, the purpose of the organization is this: “Dairy Industry Ready to Remind Us Just How Much We Love It [Dairy].” So presumably a purpose of the survey is to help persuade people to buy more dairy. Might a responsible journalist wonder whether an industry-funded survey uses questions to generate sensational results?
As of this date, the relevant segment at Undeniably Dairy reads, “Udder-ly shocking: Some Americans think chocolate milk comes from brown cows!”
This links takes us to a June 15 story by Today’s Tracy Saelinger, which in turn references two sources: an article originally published in 2014 by the National Dairy Council which does not list survey results, and the June 1 story by Sherman. Huh?
Any time you see circular or ambiguous references you should become nervous as a news reader or reporter. Where is the original survey? Anyway, I’m not sure what documentation Sherman originally saw that served as the basis of her story.
Notably, neither Sherman nor Saelinger voice the slightest bit of skepticism regarding the survey results. Saelinger condescendingly writes:
Uh, yeah—sorry people, but unless you’re vegan or have a dairy allergy, don’t most of us learn that when you first squeeze Hershey’s syrup into a glass of milk at 6 years old? And aside from that, how on earth would a mammal be able to manufacture cocoa inside one’s organs?
Apparently Saelinger never learned another tidbit that most six-year-olds know: You can’t believe everything you read on the internet.
Numerous other media sites also picked up the story. For example, a June 15 Washington Post story by Caitlin Dewey (referencing Sherman, of course) uncritically cites the industry-funded survey results. So does a June 16 CNN story by Nancy Coleman (as do others).
So what does the actual survey question state? I have not been able to find that information anywhere. On June 16 and 17, I contacted Lisa McComb from (the government-affiliated) Dairy Management Inc. She wrote:
The full survey currently isn’t posted anywhere. The survey was conducted by Edelman Intelligence to kick off our Undeniably Dairy campaign on behalf of the Innovation Center for U.S. Dairy. It polled 1,000 American adults online between May 5 and May 9, 2017.
I asked her how the question about chocolate milk was worded. She replied, “The question was phrased as: Where does chocolate milk come from?” However, I have not heard back regarding my follow-up question about whether the question was open-ended or included multiple-choice answers.
The best indication of the question wording I’ve seen comes from NPR. Host Audie Cornish says, “Jean Ragalie-Carr is president of the National Dairy Council, which commissioned the survey. She says they put that question to a thousand people and gave them several options for how to answer.” Ragalie-Carr says, “Well, there was brown cows or black-and-white cows, or they didn’t know.” NPR’s report is entirely uncritical of the survey results.
Glendora Meikle, an actual journalist with the Columbia Journalism Review, was understandably perplexed by the NPR exchange:
A careful listener’s ear may have perked up at this exchange. How exactly was the question phrased? Were those the only three options—two cow colors or “I don’t know”? And did this mean that even someone who plainly knew that chocolate milk was simply any milk that had been mixed with chocolate and sugar was not given the option of choosing anything resembling the correct response? “Comes from” is heavy-handed phrasing in and of itself, implying the chocolate milk emerges as is, without human intervention.
My best guess is that the question frequently came across as a joke, something like, “Where does chocolate milk come from, brown cows or black cows?” Understandably, some people smiled and delivered the well-known punch-line. The survey result is humorous, but it says nothing reliable about what the American public actually believes.
Meikle’s further remarks are well worth pondering:
[J]ournalists don’t have the luxury of playing fast and loose with the facts.
We’ve become accustomed to seeing these kinds of poorly phrased survey questions pop up and go viral because of some bonkers statistic they claim to support. The build-up/tear-down cycle is exhausting, and “the media” come off looking either lazy and gullible, or malicious for trying to mislead the public.
Those of us concerned with news literacy and public trust in media feel let down when one of these stories fools us across such a wide array of platforms. The problem isn’t this one survey and subsequent coverage. The evergreen problem is that if we feel like we can’t trust journalists to vet the small stuff for us, we worry that we can’t trust them with the big stuff, either. And we don’t need to be reminded that public trust in media is incredibly fragile right now.
Here’s another quibble: As Meikle indicates, the question is ambiguous. In fact, chocolate milk does come, in part, from brown cows, as well as from other sources (other cows, chocolate, and perhaps sweeteners and various additives). For example, the photograph I selected to accompany this article shows brown cows on a dairy farm. Dairy farmers do not somehow segregate out the brown cows when they produce chocolate milk, so some portion of the chocolate milk that people drink comes from brown cows. Also, depending on how broadly we interpret milk, chocolate milk does not necessarily come from cows at all; for example, I could mix up some chocolate “milk” from the coconut milk currently sitting in my refrigerator.
I’d love to see a new survey from the dairy industry that asks people where American journalism comes from. Let’s include “the ass-end of cows” as one of the possible choices and see what the survey says. Some journalists, such as Meikle, would not deserve the results.
Image: Department of Agriculture