Does Media Bias Determine Trust?

MBFC Data Deep Dive

In my previous post about the rise of misinformation, I talked about Media Bias Fact Check (MBFC) as source for determining the integrity of a media outlet. Intrigued by the amount of data they had (almost 9,000 media outlets rated in the world) I decided to do a meta analysis of those in the United States.

The United States enjoys a high degree of press freedom, protected by the First Amendment to the Constitution, which guarantees freedom of speech and the press. This freedom fosters a diverse media landscape, encompassing everything from major national outlets to local newspapers, independent blogs, and social media platforms. As a result, Americans are exposed to a broad spectrum of political views and biases. This diversity has both strengths and challenges. On one hand, it encourages vibrant public discourse, enabling individuals to access a variety of perspectives and make informed decisions. On the other hand, the multiplicity of viewpoints can lead to polarization, as some media outlets emphasize sensationalism or partisan narratives to attract audiences and generate revenue. The rise of digital media and algorithms further amplifies this effect, often creating “echo chambers” where individuals are exposed primarily to views that align with their own.

This is why media literacy, fact checking, and self introspection are so important. If we maintain the status quo and take things at face value then we are doomed to misinformation. We must always ask ourselves if we can trust the information.

MBFC uses a sophisticated methodology to determine the credibility of an outlet using four primary categories: bias, factual reporting, traffic/longevity, and press freedom. Given the relatively high press freedom in the United States, the latter does not apply. Within each of these they assign a score and the total score determines high, medium, and low credibility ratings.

Number of Media Outlets by Bias

---
config:
  xyChart:
    width: 1200
    height: 600
    xAxis:
      labelPadding: 0
      labelFontSize: 12
  themeVariables:
    xyChart:
      backgroundColor: "#161618"
---
xychart-beta
   title "Count of Media Outlets by Bias"
	x-axis "Bias" ["Left", "Left-Center", "Least Biased", "Right-Center", "Right", "Pro-Science", "Conspiracy", "Questionable"]
	y-axis "Count" 0 --> 1700
	bar [487, 1182, 1479, 517, 363, 283, 380, 1540]

Insights

• Conspiracy-Pseudoscience and Questionable Sources accounts for a shocking third of the MBFC dataset (1,920 out of 6,231). This simply cannot be ignored as it lends credence to the systemic and parasitic nature of misinformation.
• Least Biased outlets account for about a quarter of the dataset (1,479 out of 6,231) indicating that there is still a good amount of unbiased information out there.
• Left leaning publications outnumber right by almost 2:1.

Average Credibility Rating by Bias

---
config:
  xyChart:
    width: 1200
    height: 600
    xAxis:
      labelPadding: 0
      labelFontSize: 12
  themeVariables:
    xyChart:
      backgroundColor: "#161618"
---
xychart-beta
	title "Average Credibility Rating of Media Outlet by Bias"
	x-axis "Bias" ["Left", "Left-Center", "Least Biased", "Right-Center", "Right", "Pro-Science", "Conspiracy", "Questionable"]
	y-axis "Average Credibility" 0 --> 10
	bar [4.69, 6.45, 7.64, 6.23, 3.61, 7.72, 1.08, 1.90]

Insights

• Not surprisingly, Conspiracy-Pseudoscience and Questionable Sources are the least credible and should ultimately be avoided while Pro-Science are the most credible and should be sought out whenever possible.
• Least Biased publications have second highest average credibility rating, indicating that they are good source of information.
• On average, right leaning publications have lower credibility than left leaning.

Now, it is easy to fall into the trap of looking at numbers and drawing broad conclusions without any sort of statistical analysis that can help weed out randomness. I’m curious to know if media bias, to some extent, determines trust. In order to objectively do this, we need to know if the differences between the average credibility rating of a bias group is due to chance or not. The Analysis of Variance (ANOVA) statistical method is a good choice for this.

Hypothesis

There is a statistically significant difference between the average credibility rating of media bias groups.

Analysis of Variance (ANOVA) Method

Given the number of groups, independence, and slight skewness of the data, using ANOVA, seems to be practical. Generally speaking, you want the data to follow a normal distribution, however, the one-way ANOVA can tolerate data that is non-normal (skewed or kurtotic distributions) with only a small effect on the Type I error rate (the probability of incorrectly stating that there is a difference between the averages of each group)

ANOVA

A statistical method used to examine and compare the means of two or more data groups, ANOVA determines whether the observed differences between group means are statistically significant or merely attributable to random variation.

Parameters

To calculate the result of the ANOVA test, we need to calculate the following for each media bias group:

• number of samples
• sum of the crediting ratings
• average of the credibility ratings
• standard deviation of the credibility ratings

Once we have these we can then calculate the ANOVA which consists of:

• degrees of freedom between and within groups
• sum of squares between and within groups
• mean of squares between and within groups
• the F statistic which represents the ratio of the variance between the groups and the variance inside the groups.

Data Tables

Bias	Size	Sum	Average	Standard Deviation
Left	487	2015	4.686046512	1.029422948
Left-Center	1182	7470	6.445211389	0.8038502572
Least Biased	1479	11115	7.639175258	0.7237221392
Right-Center	517	3138	6.226190476	0.8462456976
Right	363	1112	3.61038961	0.7641110166
Pro-Science	283	2130	7.717391304	1.095372949
Conspiracy-Pseudoscience	380	354	1.0821.081.08568807	1.06357806
Questionable Sources	1540	2673	1.90113798	0.5888998001
Total	6231	30007	5.116283035

Source	Degrees of Freedom	Sum of Squares	Mean of Squares	F-Stat	P-Value
Between Groups	7	37069.7829	561.997583	889	< .00001
Within Groups	6223	3933.983081	0.6321682599
Total	6230

Calculation

For those not interested in the specifics of the math, you are welcome to skip on down to the conclusion

Given the somewhat complex calculation for ANOVA, I’ve decided to break down each step in the table above for clarity.

Degrees of freedom (DF) between groups is simply the number of bias groups (g) minus one

$$\begin{gathered} D{F}_{between}=g-1 \\ D{F}_{between}=8-1=7 \end{gathered}$$

Degrees of freedom (DF) within groups is the total number of observations (n) across all bias groups minus the number of bias groups (g)

$$\begin{gathered} D{F}_{within}=\sum _{i=1}^k{n}_i-g \\ D{F}_{within}=(487+1182+1479+517+363+283+380+1540)-8=6231-8=6223 \end{gathered}$$

The sum of squares (SS) between groups is the sum of observations within each bias group (n) multiplied by the difference between the mean of the bias group and the mean of the entire data set squared

$$\begin{gathered} S{S}_{between}=\sum _{i=1}^k{n}_i({\bar{X}}_i-\bar{X}{)}^2 \\ S{S}_{between}=(487\ast (4.69-5.12{)}^2)+(1182\ast (6.45-5.12{)}^2)+(1479\ast (7.64-5.12{)}^2)+(517\ast (6.23-5.12{)}^2)+(363\ast (3.61-5.12{)}^2)+(283\ast (7.72-5.12{)}^2)+(380\ast (1.08-5.12{)}^2)+(1540\ast (1.90-5.12{)}^2) \\ S{S}_{between}=37069.7829 \end{gathered}$$

The sum of squares (SS) within groups is the sum of observations within each bias group (n) minus one, multiplied by the standard deviation of the bias group (s) squared

$$\begin{gathered} S{S}_{within}=\sum _{i=1}^k(n_i-1)s_i^2 \\ S{S}_{within}=(487-1)\ast 1.03^2+(1182-1)\ast 0.80^2+(1479-1)\ast 0.72^2+(517-1)\ast 0.85^2+(363-1)\ast 0.76^2+(283-1)\ast 1.1^2+(380-1)\ast 1.06^2+(1540-1)\ast 0.59^2 \\ S{S}_{within}=3933.983081 \end{gathered}$$

The mean of squares (MS) between groups is the sum of squares (SS) between groups divided by the number of groups (g) minus one

$$\begin{gathered} M{S}_{between}=\frac{S{S}_{between}}{g-1} \\ M{S}_{between}=\frac{37069.7829}{8-1}=561.997583 \end{gathered}$$

The mean of squares (MS) within groups is the sum of squares (SS) within groups divided by the total number of samples (n) minus the number of bias groups (g)

$$\begin{gathered} M{S}_{within}=\frac{S{S}_{within}}{n-g} \\ M{S}_{within}=\frac{3933.983081}{6231-8}=0.6321682599 \end{gathered}$$

The F-Statistic is the mean of squares between groups divided by the mean of squares within groups

$$F=\frac{M{S}_{between}}{M{S}_{within}}F=\frac{561.997583}{0.6321682599}=889$$

You can determine the p-value by doing a lookup in an F-distribution table with the degrees of freedom between and within. Most tables will not go up to very large values such as the case here, however, it can be calculated with statistical software. The resulting p-value is less than 0.00001. In other words, there is less than a 1 in 100,000 chance that the average credibility rating differences are due to chance.

$$p\_value<0.00001$$

Result

The bias of a media publication can be an indicator of trust given that the difference between the average credibility ratings of each bias group are extremely statistically significant, meaning it is very unlikely due to chance.

Conclusion

Yes, media bias does determine trust, to an extent. Unsurprisingly, you should seek out Pro-Science and Least Biased publications if you want the most credible sources and avoid like the plague anything classified as Conspiracy-Pseudoscience and Questionable Sources

Interestingly, on average, left leaning publications produce more trustworthy content. This is more pronounced when looking at Left vs Right with 4.69 and 3.61 credibility ratings. Given the rise of MAGA and Trumpism, which are notorious for producing divisive propaganda, this may be influencing the lower credibility on the right. The right also tends to be more susceptible to misinformation and is less-informed which could encourage Right bias outlets to produce lower quality content. In a sense, this susceptibility is self-serving.

People tend to seek out information that aligns with their ideals. This doesn’t seem to be a great way to exclusively source your information given the lower credibility of polarized content relative to those that are Least Biased or Pro Science.

Just some thoughts.

When possible, all supporting articles or references I linked to were fact checked and credibility verified. Below is the data from MBFC

Source	Bias	Reporting Accuracy	Credibility Rating
Media Matters	Left	High	High
Science.org	Pro-Science	High	Very High