In September, we released the findings of our August 2020 Blind Bias Survey, in which over 2,000 people across the political spectrum rated the media bias of five different news outlets blindly, seeing only their content and story choice but not the name of the news publication. Below are visualizations of the data we collected.
In a Blind Bias Survey, we ask people to self-report their own political bias (Left, Lean Left, Center, Lean Right, or Right) and then present them with content from media outlets. We show them numerous front page headlines and the first few paragraphs from two articles from a given media outlet, stripping them of any identifying information, and ask respondents to rate the outlet’s bias.
While our Blind Bias Surveys are a uniquely powerful and fair way to rate the bias of news sources without giving more weight to one group over another, they alone do not always capture the full picture. They are essentially snap shots of the top stories on one day. That misses the bias that may show itself in other stories, in pictures, and on other days.
We had roughly equal numbers of people who identified as having a personal bias of Lean Left, Center, or Lean Right take the survey (529, 605, and 595, respectively), but much less people with a Left or Right bias take the survey (233 on the Left and just 153 on the Right). Therefore, the charts below show normalized data.
We calculated the data so that equal weight is given to the responses of people from each political group, so that the Lean Left, Center, and Lean Right folks aren’t over-represented. These graphics show how the majority of all survey respondents perceive each outlet, if we’d had an equal number of people from each group rating them. (For more detailed charts showing how liberals and conservatives rated these outlets differently, jump to the bottom of this page here.)
While the Blind Bias Survey is an excellent tool because those being surveyed aren’t biased by the publication’s brand or pre-existing biases about it, it is important to note that majority rule does not determine AllSides Media Bias Ratings.
When we apply these findings to our ratings, we examine survey data by looking at pluralities within bias groups, as well as average ratings across all bias groups. In addition, these surveys are not the only methodology we use. Final bias ratings are typically arrived at via a combination of methods, including surveys, Editorial Reviews (conducted by a team that includes a balance of people from the Left, Center and Right of the political spectrum), third party data, and independent research.