Are Nielsen Ratings Accurate?
This article seeks to show that how Nielsen Media Research chooses its ratings families may alter the accuracy of the Nielsen’s ratings. More than this, this article seeks to uncover just how antiquated and unreliable Nielsen’s household rating system actually is. Let’s explore.
What is Nielsen?
I’ll give a small synopsis here, but Wikipedia does a much better job at describing who and what Nielsen Media Research (one of this company’s many names) is. For all intents and purposes, I will refer to Nielsen Media Research as simply Nielsen for the purpose of this article.
Nielsen is a research group who seeks to identify how viewers, among other avenues of information that they gather, watch Television. During the 70s, this was the primary means by which TV executives learned the ratings fate of their television programs.
How does Nielsen work?
Nielsen still relies on its Nielsen households to provide the vast majority of its television ratings information. It does this by sending out unsolicited mail to households around the country attempting to solicit a household into becoming a Nielsen household. By using this moniker, it means the family who resides at a specific household must do certain things to not only participate in the Nielsen program, but must also provide feedback to Nielsen around its viewing habits.
How does Nielsen collect its ratings information?
According to Nielsen’s own site, it says the following:
To measure TV audiences and derive our viewing metrics (i.e., ratings, reach, frequency), we use proprietary electronic measuring devices and software to capture what content, network or station viewers are watching on each TV and digital devices in the homes of our Nielsen Families. In total, we measure hundreds of networks, hundreds of stations, thousands of programs and millions of viewers. In the U.S., electronic measuring devices and millions of cable/satellite boxes are used to provide local market-level viewing behaviors, enabling the media marketplace to gain a granular view of TV audiences.
What that means is that, as a Nielsen household, they will send you a device and/or require you to install certain software on your existing devices which will “measure” your viewing habits. In other words, they spy on what you’re watching and it reports back to Nielsen what you specifically watched and for how long. For example, Nielsen might install software onto your smart TV device, Roku, TiVO, Apple TV or possibly even your cable TV provider’s supplied box.
Nielsen may even be willing to supply you with their own device, which you will place in-line with your existing TV and devices. It does say “devices and software”, meaning one or both can be used.
Rural vs Urban
Typically, larger urban city areas tend to vote Democrat more often than Republican. These urban areas are also typically more densely populated. On the flip side, rural areas tend to vote Republican more often than Democrat. Why is this information important? It’s important to understand these facts because it can drastically alter the accuracy of Nielsen’s ratings. Let’s understand why.
For participating in being a Nielsen household, you’re given a stipend. In other words, you’re paid for this service. Let’s understand more about this pay. You’re paid around $10 a month to participate. If you remain a Nielsen household for a certain period, around 6 months, Nielsen will pay you a bonus. All told, for 6 months of service, a Nielsen household will receive around $200.
Here’s where the Urban vs Rural comes into play. Rural areas tend to be more depressed economically. Meaning, income is generally less and the need for extra money is, therefore, higher. Urban areas tend to boom more economically meaning the need for extra money is, therefore, lessened.
If a rural household receives a card inviting them to become part of the Nielsen family, explaining all of the “benefits” (including the pay), rural viewers are much more likely to take Nielsen up on their pitch. It seems easy enough to get paid simply for watching TV. On the other hand, urban areas are less likely to take Nielsen up on their offer not only because the pay is so low, but because urban viewers are much more savvy around their privacy.
Who would intentionally invite a company into your household to spy on you, even for money? One might say, well there’s Alexa. Alexa offers benefits to the user far greater than what Nielsen provides. Nielsen provides spying for cash. Alexa offers app features, smart house features, music, calling features, recipe helpers, and the list goes on. Nielsen’s device(s) and software(s) don’t provide those much extended features.
Nielsen’s spying is one tracked and only helps out TV executives. I might add that those TV executives PAY Nielsen to gain access to this information. Which means that if you’re a Nielsen household, you’re getting paid out of money collected from TV executives. In effect, it is the TV executives who effectively sign your Nielsen paycheck that you receive. I digress.
Random Solicitation
Make no mistake, Nielsen solicits households through a random mail selection process. It sends pitch cards out to inform and solicit households to participate. They may even include a crisp $1 bill to entice the household. Nielsen knows that a certain percentage of people will take Nielsen up on their offer to participate in the program.
The difficulty is that this selection process relies on random chance for whomever chooses to participate. This goes back to Urban vs Rural argument. Because depressed areas are more “hard up” for cash, they are more likely to take Nielsen up on their offer than Urban areas, who urban viewers are not only likely to be mistrustful of spying using digital devices, these people also don’t necessarily need the small-ish amount of cash that Nielsen is offering… considering the amount of time required to watch TV (and do whatever else Nielsen requires). Yes, Nielsen requires you to watch TV to participate. The whole thing doesn’t work unless you actually watch TV.
This ultimately means that it is more likely rural Republican areas of the country are over represented in Nielsen’s households and equally likely Democrat areas to be under-represented in Nielsen’s ratings. While Nielsen has no control over who chooses to accept the “Nielsen Household” solicitation, Nielsen does control the parameters to entice people into the program. Thus, their parameters are skewed toward lower income households, which are likely to be in predominantly rural areas.
In other words, depressed rural areas are far more likely to need the extra cash and be willing to jump through Nielsen’s hoops than more affluent urban areas. That’s not to say that there won’t be a percentage of viewers in urban areas as some households in those areas may elect to participate.
Disposable Income
Urban areas can be a bit more affluent than rural areas. Urban area residents may have more in disposable income, but also because it’s a larger city, it has more entertainment options. This means entertainment options besides watching TV. When you live in a small rural town, entertainment options can be extremely limited even if disposable income is available. Rural townships tend to encourage more TV watching more often than urban areas where night clubs, restaurants, theme parks, opera, live theater events, shopping and large cinemas are common. More entertainment options means less need to watch TV as often.. except for specific shows.
Thus, urban viewers are less likely to want to participate in Nielsen’s household program than rural viewers, whose entertainment options may be limited by both what’s available near them and by their disposable income.
Extrapolation
Here’s the crux of Nielsen’s problems. Based on the over and under represented areas due to Nielsen’s flawed selection process, they attempt to make up for this by extrapolating data. Regardless of how the households may be skewed, Nielsen intends to extrapolate its data anyway.
Nielsen estimates that it has around 42,000 households participating in 2022. Though, I’d venture to guess that that number is not completely accurate. I’d suggest Nielsen may have perhaps half that number actively participating at any one time. There might be 42,000 households signed up as a Nielsen household, but likely only a fraction actively participate at any specific moment in time.
For example, not every household will watch a specific sporting event when it’s on. Only those who truly enjoy watching a specific football game might be watching a specific game. This could drop that 42,000 households down to under 5,000 viewers. If it’s a local sporting event, it could drop that number down well below 1,000 and maybe even below 200 actively watching.
200 equals 1 million, 5 million, 100 million?
How does this affect the ratings? Good question. Only Nielsen really knows. The problem is, as I stated above, Nielsen uses extrapolation.
What is extrapolation? Extrapolation is the process of using 1 viewer to represent many viewers. How many is a matter of debate. It is a process that Nielsen has employed for many years, and it is highly inaccurate. It makes the assumption that for every one viewer watching, there will be a specific number also watching. How many are extrapolated is really up to Nielsen. Nielsen must come up with those numbers and herein lies the inaccuracy.
Effectively, Nielsen fudges the numbers to appear great (or poor) depending on how it decides to cull the number together. In other words, extrapolation is an exceedingly poor and inaccurate way to determine actual viewership numbers. Yet, here we are.
Digital Media Streaming
With digital streaming services, such as Netflix, Hulu, Amazon and Crackle… more specifically, devices such as DVRs like TiVO and devices like Apple TV, Nielsen’s numbers may be somewhat more accurate when using these devices. However, one thing is certain. Nielsen still doesn’t have 100% accuracy because it doesn’t have 100% of every TV household participating.
Again, Nielsen’s numbers may be somewhat more accurate because we now have active digital streaming devices, but Nielsen still employs extrapolation to inflate the data they collect. Nielsen takes the numbers they collect, then guess at how many might be watching based on each single viewer’s behaviors.
Why Extrapolation over Interpolation?
Interpolation requires two distinct sets of data points in which to fill in the interior data gap between those two sets. Filling in data between two distinct sets of data is a bit more accurate than attempting to guess at data points outside of them.
With viewership numbers, it’s only one set of data at a single point in time. Everything that is gleaned from that single set of data is always considered “outside” or “extrapolated” data. There’s nothing in a single data set to interpolate. You have 42,000 households. You have a smaller number watching a TV program at any point in time. That’s all there is.
If Nielsen ran two unique and separate sets of 42,000 households of viewers (a total of 84,000 viewers), interpolation would be possible between those to separate sets of 42,000. Nielsen doesn’t utilize this technique, thus making interpolation of its collected data is impossible.
How Accurate is Extrapolation?
Not very. I’ll point to this StackExchange article to explain the details as to exactly why. In short, the larger the number gets outside of the original sample size, the larger the margin for error… to the point where the error outweighs the value of the extrapolation.
One answer provides this quote:
[Extrapolation] is a theoretical result, at least for linear regression. Indeed, if one computes the so-called ”prediction error” (see this link, slide 11), one can easily see that the further the independent variable 𝑥 is away from the sample average 𝑥¯ (and for extrapolation one may be far away), the larger the prediction error. In the link that I referred to one can also see that in a graphical way.
In a system where there is no other option, such as during the 70s when computers were room-sized devices, extrapolation may have been the only choice. Today, with palm sized internet enabled phones containing compute power orders of magnitudes faster than many of those 70s room-size computers, continuing to use extrapolation honestly makes zero sense… especially when accuracy is exceedingly important and, indeed, required.
Extrapolation Examples
If 1 Nielsen viewer represents 1,000 viewers extrapolated (1:1,000), then 100 Nielsen households watching suggests 100,000 viewers may actually be watching. If 100 Nielsen viewers watch a program and each household represents 100,000 viewers (1:100,000), then this suggests 10,000,000 viewers may be watching. Just by changing the ratio, Nielsen can alter how many it suggests may be watching. Highly inaccurate and completely beholden to Nielsen making up these ratios. As stated above, the larger the number diverges from the original sample size, the larger the margin of error… possibly making this data worthless.
These suggested extrapolated viewership numbers do not actually mean that that many viewers were, in reality, watching. In fact, the real viewership number may be far, far lower than the extrapolated numbers suggest. This is why extrapolation is a bad, bad practice. Extrapolation is always error prone and usually in the wrong way. It makes too many assumptions that are more than likely to be wrong.
Unless the person doing the extrapolation has additional data points which logically suggest a specific ratio is at play, then it’s all “best guess” and “worst error”.
How many businesses would choose run their corporation on “best guess”? Yet, that’s exactly what TV executives are doing when they “rely” (and I use this term loosely) on Nielsen.
Biased
Even above the fact that extrapolation has no real place in business, because of its highly inaccurate and “best guess” nature, these numbers can be highly biased. Why? Because of the Urban vs Rural acceptance rates.
Unless Nielsen explicitly goes out of their way to take the under vs over represented nature of Nielsen households into account when extrapolating, what Nielsen suggests is even more inaccurate than I even suggest just from the use of extrapolation alone.
CNN vs Fox News
CNN has tended to be a more liberal and, thus, a Democrat favorable news organization. Though, I’d say CNN tends to be more moderate in its liberal Democrat leanings. Fox News, on the other hand, makes no bones about their viewpoint. Fox News is quite far right and Republican in too much of its of leanings. Fox News is not always as far right as, for example, Alex Jones or other extremist right media. However, some of its leanings can be as far right as some quite far right media. Here’s an image from the Pew Research Center that visually explains what I’m describing:
Whether Pew’s research and datapoints are spot on, I’ll leave that for you to decide. I’ve reviewed this chart and believe it to be mostly accurate in terms of each outlet’s political leanings. Though, I personally have found PBS to be somewhat closer to the “Average Respondent” location than this chart purports… which is why even Pew might not have this chart 100% correct. For the purpose of CNN, Fox News and Hannity, I’ve found this chart to be spot on.
As you can see in the chart above, Fox News itself is considered a right leaning news organization, but not far off of center at around a 2. However, the Sean Hannity show is considered just as far right as Breitbart at about 6-7. CNN is considered slightly left leaning at around a 1 (less left leaning than Fox News is right leaning at 2).
What does all this mean for Nielsen? It means that those who are Republican, which tends to include more rural viewers than urban, those rural viewers tend to be conservative. Because Nielsen is more likely to see participation from rural viewers than urban viewers, due to its enticement practices, this skews Nielsen’s accuracy towards conservative viewership and away from liberal viewership. Nielsen’s enticement practice isn’t the only problem which can lead to this skew, though.
Meaning, Fox News viewership numbers as stated by Nielsen may be highly overestimated and inaccurate. Quantifying that more specifically, Fox News viewers may be over-represented where CNN viewers may be severely under-represented. It further means that unless Nielsen actually realizes this liberal vs conservative under vs over representation disparity in its Nielsen households (respectively) and, thus, alters its extrapolated numbers accordingly, then its viewership numbers published for CNN vs Fox News are highly suspect and are likely to be highly inaccurate.
Worse, Fox News is owned by Rupert Murdoch. Because this man is in it for the cash that he can milk from the Fox News network, he’s more than willing to pay-for-play. Meaning, if he can get companies to favor Fox News by asking them for favors in exchange for money, he (or one of his underlings) will do it. Murdoch can then make more money because more advertisers will flock to Fox News under the guise of more viewership. Fake viewership is most definitely lucrative. Because Nielsen extrapolates data, this makes faking data extremely easy.
Unlike YouTube where Google has no reason to lie about its reported views, Fox News has every reason to lie about its viewership, particularly if it can game other companies into complying with its wishes.
Nielsen Itself
Nielsen purports to offer objective data. Yet, we know that businesses are helmed by fallible human CEOs who have their own viewpoints and political leanings and who are in it for the money. One only needs to look at Rupert Murdoch and Fox News to understand this problem. Some CEOs also choose to micromanage their company’s products. Meaning, if Nielsen’s current CEO is micromanaging its ratings product, which is also likely to be Nielsen’s highest moneymaking product, then it’s entirely possible that the ratings being reported are biased, particularly in light of the above about Rupert Murdoch (who is also a Republican).
Conflict of Interest
When money gets involved, common sense goes out the window. What I mean by this statement is that since TV executives / networks pay Nielsen to receive its ratings results periodically, Nielsen is beholden to its customers. The word “beholden” can have many meanings in this “sales” context. Typically in business, “beholden” means the more you pay, the more you get. In the case of Nielsen, it’s possible that paying more to Nielsen potentially means that business may get more / better ratings. That sort of breaks the “objective” context of Nielsen’s data service. It’s called “Conflict of Interest”.
In essence, in this case it could represent a pay-for-play solution, a true conflict of interest. There’s honestly no way to know what deals Nielsen has brokered with its clients, or more specifically with Rupert Murdoch’s Fox News Network. Most companies who do sales deals keep those details close to the vest and under non-disclosure binding contracts. The only way these deals ever get exposed is during court trials when those contracts can become discovery evidence for a trial. Otherwise, they remain locked in digital filing cabinets between both parties. Even then, such contracts are very unlikely to contain words disclosing any “back room” verbal handshake deals discussed. Those deal details will be documented in a separate system or set of systems describing how to handle that customer’s account.
Let me count the ways
There are many problems in the Nielsen’s rating services that may lead to highly inaccurate information being released. Let’s explore them:
- Nielsen’s solicitation of households can easily lead to bias due to its probability of luring in people who are hard up for cash (e.g., rural Republicans) vs those who are not (e.g., urban Democrats).
- Nielsen’s products and software spy on knowing users about viewership habits. Spying of any variety is usually viewed with skepticism and disdain, especially these days and especially by certain types of people in the population (usually liberal leaning individuals). Rural Republicans are less likely to understand the ramifications of this spying (and more willing to accept it) than urban Democrats (who tend to be more likely to work in tech based businesses and who see this type of spying as too intrusive).
- Nielsen’s numbers are “fortified” using extrapolation. Fortified is a nice way of saying “padded”. By padding their numbers, Nielsen staff can basically gyrate the numbers any way they want and make any channels viewership numbers look any particular way. Which ties directly into…
- Nielsen sells its ratings product to TV producers and networks. Because these deals are brokered separately for varying amounts of money, the network who pays the most is likely to see the best results (i.e., pay-for-play).
- Nielsen moved away from its “on paper” auditing system to the use of digital device auditing. Because Nielsen removed the human factor from this ratings equation (and fired people as a result), it also means that fewer and fewer people can see the numbers to know what they truly are (or at least were before the extrapolation). Fewer people seeing the numbers means higher chances of fabrication.
Looking at all of these above, it’s easy to see how Nielsen’s numbers could be seriously inaccurate, possibly even intentionally. I won’t go so far as to say, fake, although that’s entirely possible. However, because Nielsen employs extrapolation, it would be easy for a Nielsen staffer (or even Nielsen’s very CEO) to make up anything they want and justify it based on its “proprietary” extrapolation techniques. Meaning, numbers stated for any network’s viewership could be entirely fabricated by Nielsen, possibly even at a network’s request or possibly even as part of that network’s deal with Nielsen.
In fact, fabrication is possible based entirely on number 4 above. A TV network could pay significantly to make sure their network and their programming is always rated the highest, at least until they stop paying for it. With Nielsen’s extrapolation system and when data can get played fast and loose, it’s entirely possible for such a sales scenario to manifest.
Why are Nielsen’s Numbers Important?
Advertising. That’s the #1 reason. Companies using TV advertising wish to invest their advertising dollars into channels with the highest viewership. The higher, the better. Nielsen’s ratings are, therefore, indicative that a higher ratings share means higher viewership. The problem is, Nielsen’s extrapolation gets in the way of that. Regardless of whether or not cheating or fabrication is involved, the sheer fact that extrapolation is used should be considered a problem.
The only thing Nielsen really knows is that of the 42,000 Nielsen households that it has devices in, only a fraction of those households watched a given program or channel at any specific time. Meaning, the real numbers of viewership from Nielsen offers a maximum of 42,000 viewers at any moment in time… no where close to the millions that they claim. Any number higher than 42,000 is always considered fabricated whether extrapolation or any other means is used to inflate that number.
That companies like Procter and Gamble rely on those 42,000 Nielsen households to determine whether to invest perhaps millions of dollars in advertising on a channel is suspect. That companies have been doing this since the 70s is a much bigger problem.
In the 70s, when there was no other way to really determine TV viewership, Nielsen’s system may have held some measure of value, even though it used extrapolation. However, in 2022 with live always-on internet enabled phone, tablet, computer, game console and other smart TV devices, measuring actual live viewers seems quite feasible directly from each device tuned in. If someone is live streaming CNN over the Internet, for example, it’s not hard to determine and count this at all. If hundreds of people are streaming, that should be easy to count. If millions, it’s also easy. Why extrapolate when you can use real numbers?
The days of extrapolation should have long ended, replaced by live viewer tallies from various digital streaming devices, such as phones, computers and Apple TVs. Whether these devices are allowed to phone home to provide that data, that’s on each viewer to decide. If the viewer wishes to opt-in to allowing their viewership metrics to be shared with each TV station, then that’s far more realistic viewership numbers than Nielsen’s extrapolated numbers. If they opt-out, then those stations can’t see the numbers. Opting in and out should be the choice of the viewer.
That’s where privacy meets data sharing. Some people simply don’t want any of their private data to be shared with companies… and that’s okay. That then means some level of extrapolation (there’s that word again) must be used to attempt inflate the numbers accordingly.
Let’s consider that 42,000 is 0.01273% of 330 million. That’s trying to represent the entire population of TV viewers in the United States from less than 0.01% of people watching. Insane! With always-on digital devices, if 10% opt out, that’s still provides 90% more accurate viewership numbers than relying on Nielsen’s tiny number of households. Which means there’s way less amount of data to attempt to extrapolate. That advertisers don’t get this point is really surprising.
Auditing
You might think, “Well, isn’t Nielsen audited?”
Most companies dealing with numbers are typically audited. Unfortunately, I’ve found that working in a tech business which sees regular audits can still have fabrication. How? Because those who work on the technical side of the house are not those who get audited. Meaning, those systems administrators who maintain the logs and records (i.e. databases) aren’t under the scrutiny that the financial side of the house gets.
If it relates to money and sales, auditing of the accounting books is a regular occurrence and must uphold specific standards due to legal requirements. Auditing when it relates to anything else is catch-as-catch-can, particularly when laws don’t exist. Meaning, the auditors must rely on the statements of staffers to be accurate. There’s no way for an auditor to know if something has or hasn’t been fabricated when viewing a log.
Worse, if the company employs a proprietary algorithm (read private) to manage its day to day operations, auditors typically are unable to break through its proprietary nature to understand if there’s a problem afoot. In other words, auditors must take what’s told to them at face value. This is why auditing is and can be a highly inaccurate profession. I should also point out that auditing isn’t really intended to uncover treachery and deception. It’s intended to document what a company states about specific questions, whether true or false. Treachery and deception may fall out of an audit, but usually only if legal action is brought against the company.
In the case of money, it’s easy to audit records of both the company and third parties to ensure the numbers match. In the case of proprietary data, there’s no such records to perform this sort of matching. What an auditor sees is what they must accept as genuine. The only real way that such deception and fabrication becomes known is if an employee performing such fabrication blows the whistle. An independent auditor likely won’t be able to find it without a whistleblower. Because jobs tend to be “on the line” around such matters, employees are usually told what they can and cannot say to an auditor by their boss. Meaning, the boss might be acutely aware of the fabrication and may instruct their employees not to talk about it, even if directly asked.
In fact, employees performing such fabrication of data may intentionally be shielded from audits, instead throwing employees who have no knowledge at the auditors. It’s called, plausible deniability.
Overall
None of the above is intended to state that Nielsen fabricates numbers maliciously. However, know that extrapolation of data is actually the art of data fabrication. It takes lower numbers and then applies some measure of logic and reasoning that “makes sense” to deduce a larger number. For example, if one person complains of a problem, it’s guaranteed a number of other people have also encountered the same exact problem, but didn’t complain.
The art is in deducing how many didn’t complain. That’s extrapolation by using logic and reasoning to deduce the larger number. Extrapolation clearly isn’t without errors. Everyone who deals in extrapolation knows there’s a margin of error, which might be as high as 10% or possibly higher and which grows as the extrapolation data size increases.
Are Nielsen’s ratings numbers accurate? Not when you’re talking about 42,000 households attempting to represent the around 122 million households with TVs. This data doesn’t even include digital phones, tablets and computers which are capable of streaming TV… which smartphones alone account for about 7.26 billion devices. Yes, billion. In the United States, the number of smart phone owners is around 301 million. There are more smart phones in existence in the United States (and the rest of the world) than there are TV’s in people’s homes.
So, exactly why does Nielsen continue to cling to its extremely outdated business model? Worse, why do advertisers still rely on it? 🤷♂️
↩︎
Why Rotten Tomatoes is rotten
When you visit a site like Rotten Tomatoes to get information about a film, you need to ask yourself one very important question, “Is Rotten Tomatoes trustworthy?”
Rotten Tomatoes as a movie review service has come under fire many times for revitew bombing and manipulation. That is, Rotten Tomatoes seem to allow shills to join the service to review bomb a movie to either raise or lower its various scores by manipulating the Rotten Tomatoes review system. In the past, these claims couldn’t be verified. Today, they can.
As of a change in May 2019, Rotten Tomatoes has made it exceedingly easy for both movie studios and Rotten Tomatoes itself to game and manipulate the “Audience Score” ratings. Let’s explore.
Rotten Tomatoes as a Service
Originally, Rotten Tomatoes began its life as an independent movie review service such that both critics and audience members can have a voice in what they think of a film. So long as Rotten Tomatoes remained an independent and separate service from movie studio influence and corruption, it could make that claim. Its reviews were fair and for the most part accurate.
Unfortunately, all good things must come to an end. In February of 2016, Fandango purchased Rotten Tomatoes. Let’s understand the ramifications of this purchase. Because Fandango is wholly owned by Comcast and in which Warner Brothers also holds an ownership stake in Fandango, this firmly plants Rotten Tomatoes well out of the possibility of remaining neutral in film reviews. Keep in mind that Comcast also owns NBC as well as Universal Studios.
Fandango doesn’t own a stake in Disney as far as I can tell, but that won’t matter based on what I describe next about the Rotten Tomatoes review system.
Review Bombing
As stated in the opening, Rotten Tomatoes has come under fire for several notable recent movies as having scores which have been manipulated. Rotten Tomatoes has then later debunked those claims by stating that their system was not manipulated, but then really offering no proof of that fact. We simply have to take them at their word. One of these allegedly review bombed films was Star Wars: The Last Jedi… where the scores inexplicably dropped dramatically in about a 1 month period of time. Rotten Tomatoes apparently validated the drop as “legitimate”.
Unfortunately, Rotten Tomatoes has become a bit more untrustworthy as of late. Let’s understand why.
As of May of 2019, Rotten Tomatoes introduced a new feature known as “verified reviews”. For a review’s score to be counted towards the “Audience Score”, the reviewer must have purchased a ticket from a verifiable source. Unfortunately, the only source from which Rotten Tomatoes can verify ticket purchases is from its parent company, Fandango. All other ticket purchases don’t count… thus, if you choose to review a film after purchasing your ticket from the theater’s box office, from MovieTickets.com or via any other means, your ticket won’t count as “verified” should you review or rate the movie. Only Fandango ticket purchases count towards “verified” reviews, thus altering the audience score. This change is BAD. Very, very bad.
Here’s what Rotten Tomatoes has to say from the linked article just above:
Rotten Tomatoes now features an Audience Score made up of ratings from users we’ve confirmed bought tickets to the movie – we’re calling them “Verified Ratings.” We’re also tagging written reviews from users we can confirm purchased tickets to a movie as “Verified” reviews.
While this might sound like a great idea in theory, it’s ripe for manipulation problems. Fandango also states that “IF” they can determine “other” reviews as confirmed ticket purchases, they will mark them as “verified”. Yeah, but that’s a manual process and is impossibly difficult to determine. We can pretty much forget that this option even exists. Let’s list the problems coming out of this change:
- Fandango only sells a small percentage of overall ticket sales for a film. If the “Audience Score” is calculated primarily and solely from Fandango ticket sales alone, then this metric is a horribly inaccurate metric to rely on.
- Fandango CAN handpick “other” non-Fandango ticket purchased reviews to be included. Not likely to happen often, but this also means they can pick their favorites (and favorable) reviews to include. This opens Rotten Tomatoes up to Payola or “pay for inclusion”.
- By specifying exactly how this process works, this change opens the Rotten Tomatoes system to being gamed and manipulated, even by Rotten Tomatoes staff themselves. Movie studios can also ask their employees, families and friends to exclusively purchase their tickets from Fandango and request these same people to write “glowing, positive reviews” or submit “high ratings” or face job consequences. Studios might even be willing to pay for these positive reviews.
- Studios can even hire outside people (sometime known as shills) to go see a movie by buying tickets from Fandango and then rate their films highly… because they were paid to do so. As I said, manipulation.
Trust in Reviews
It’s clear that while Rotten Tomatoes is trying to fix its ills, it is incredibly naive at it. It gets worse. Not only is Rotten Tomatoes incredibly naive, this company is also not at all tech savvy. Its system is so ripe for being gamed, the “Audience Score” is a nearly pointless metric. For example, 38,000 verified reviews based on millions of people who watched it? Yeah, if that “Audience Score” number isn’t now skewed, I don’t know what is.
Case in point. The “Audience Score” for The Rise of Skywalker is 86%. The difficulty with this number is the vast majority of the reviews I’ve seen from people on chat forums don’t rate the film anywhere close to 86%. What that means is that the new way that Rotten Tomatoes is calculating scores is effectively a form of manipulation itself BY Rotten Tomatoes.
To have the most fair and accurate metric, ALL reviews must be counted and included in all ratings. You can’t just toss out the vast majority of reviews simply because you can’t verify them has holding a ticket. Even still, holding a ticket doesn’t mean someone has actually watched the film. Buying a ticket and actually attending a showing of the film are two entirely separate things.
While you may have verified a ticket purchase, did you verify that the person actually watched the film? Are you withholding brand new Rotten Tomatoes account reviewers out of the audience score? How trustworthy can someone be if this is their first and only review on Rotten Tomatoes? What about people who downloaded the app just to buy a ticket for that film? Simply buying a ticket from Fandango doesn’t make the rating or reviewer trustworthy.
Rethinking Rotten Tomatoes
Someone at Rotten Tomatoes needs to drastically reconsider this change and they need to do it fast. If Rotten Tomatoes wasn’t guilty of manipulation of review scores before this late spring change in 2019, they are now. Rotten Tomatoes is definitely guilty of manipulating the “Audience Score” by the sheer lack of reviews covered under this “verified review” change. Nothing can be considered valid when the sampling size is so small as to be useless. Verifying a ticket holder also doesn’t validate a review author’s sincerity, intent or, indeed, legitimacy. It also severely limits who can be counted under their ratings, thus reducing the trustworthiness of “Audience Score”.
In fact, only by looking at past reviews can someone determine if a review author has trustworthy opinions.
Worse, Fandango holds a very small portion of all ticket sales made for theaters (see below). By showing all (or primarily) scores tabulated by people who bought tickets from Fandango, this definitely eliminates well over half of the written reviews on Rotten Tomatoes as valid. Worse, because of the way the metric is calculated, nefarious entities can game the system to their own benefit and manipulate the score quickly.
This has a chilling effect on Rotten Tomatoes. The staff at Rotten Tomatoes needs roll back this change pronto. For Rotten Tomatoes to return it being a trustworthy neutral entity in the art of movie reviews, it needs a far better way to determine trustworthiness of its reviews and of its reviewers. Trust comes from well written, consistent reviews. Ratings come from trusted sources. Trust is earned. The sole act of buying a ticket from Fandango doesn’t earn trust. It earns bankroll.
Why then are ticket buyers from Fandango any more trustworthy than people purchasing tickets elsewhere? They aren’t… and here’s where Rotten Tomatoes has failed. Rotten Tomatoes incorrectly assumes that by “verifying” a sale of a ticket via Fandango alone, that that somehow makes a review or rating more trustworthy. It doesn’t.
It gets worse because while Fandango represents at least 70% of online sales, it STILL only represents a tiny fraction of overall ticket sales, at just 5-6% (as of 2012).
“Online ticketing still just represents five to six percent of the box office, so there’s tremendous potential for growth right here.” –TheWrap in 2012
Granted, this TheWrap article is from 2012. Even if Fandango had managed to grab 50% of the overall ticket sales in the subsequent 7 years since that article, that would leave out 50% of the remaining ticket holder’s voices, which will not be tallied into Rotten Tomatoes current “Audience Score” metric. I seriously doubt that Fandango has managed to achieve anywhere close to 50% of total movie ticket sales. If it held 5-6% overall sales in 2012, in 7 years Fandango might account for growth between 10-15% at most by 2019. That’s still 85% of all reviews excluded from Rotten Tomatoes’s “Audience Score” metric. In fact, it behooves Fandango to keep this overall ticket sales number as low as possible so as to influence its “Audience Score” number with more ease and precision.
To put this in a little more perspective, a movie theater might have 200 seats. 10% of that is 20. That means that for every 200 people who might fill a theater, just less than 20 people have bought their ticket from Fandango and are eligible for their review to count towards “Audience Score”. Considering that only a small percentage of that 20 will actually take the time to write a review, that could mean out of every 200 people who’ve seen the film legitimately, between 1 and 5 people might be counted towards the Audience Score. Calculating that up, for very 1 million people who see a blockbuster film, somewhere between 5,000 and 25,000’s reviews may contribute to the Rotten Tomatoes “Audience Score”… even if there are hundreds of thousands of reviews on the site.
The fewer the reviews contributing to that score, the easier it is to manipulate that score by adding just a handful of reviews to the mix… and that’s where Rotten Tomatoes “handpicked reviews” come into play (and with it, the potential for Payola). Rotten Tomatoes can then handpick positive reviews for inclusion. The problem is that while Rotten Tomatoes understands all of this this, so do the studios. Which means that studios can, like I said above, “invite” employees to buy tickets via Fandango before writing a review on Rotten Tomatoes. They can even contact Rotten Tomatoes and pay for “special treatment”. This situation can allow movie studios to unduly influence the “Audience Score” for a current release… this is compounded because there are so few reviews that count to create the “Audience Score”.
Where Rotten Tomatoes likely counted every review towards this score before this change, after they implemented the new “verified score” methodology, this change greatly drops the number of reviews which contribute to tallying this score. This lower number of reviews means that it is now much easier to manipulate its Audience Score number either by gaming the system or by Rotten Tomatoes handpicking reviews to include.
Fading Trust
While Rotten Tomatoes was once a trustworthy site for movie reviews, it has greatly reduced its trust levels by instituting such backwards and easily manipulable systems.
Whenever you visit a site like Rotten Tomatoes, you must always question everything you see. When you see something like an “Audience Score”, you must question how that number is calculated and what is included in that number. Rotten Tomatoes isn’t forthcoming.
In the case of Rotten Tomatoes, they have drastically reduced the number of included reviews in that metric because of their “verified purchase” mechanism. Unfortunately, the introduction of that mechanism at once destroys Rotten Tomatoes trust and trashes the concept of their site.
It Gets Worse
What’s even more of a problem is the following two images:
From the above two images, it is claimed Rotten Tomatoes has 37,956 “Verified Ratings”, yet they only have 3,342 “Verified Audience” reviews. That’s a huge discrepancy. Where are those other 34,614 “Verified” reviews? You need to calculate the Audience Score not solely on a phone device using a simplistic “rate this movie” alone. It must be calculated in combination with an author writing a review. Of course, there are 5,240 reviews that didn’t at all contribute to any score at all on Rotten Tomatoes. Those audience reviews are just “there”, taking up space.
Single number ratings are pointless without at least some text validation information. Worse, we know that these “Verified Ratings” likely have nothing to do with “Verified Audience” as shown in the images above. Even if those 3,342 audience reviews are actually calculated into the “Verified Ratings” (they probably aren’t), that’s still such a limited number when considered with the rest of the “Verified Ratings” so as to be skewed by people who may not have even attended the film.
You can only determine if someone has actually attended a film by asking them to WRITE even the smallest of a review. Simply pressing “five star” on the app without even caring is pointless. It’s possible the reviews weren’t even tabulated correctly via the App. The App itself may even submit star data after a period of time without the owner’s knowledge or consent. The App can even word its rating question in such a way as to manipulate the response in a positive direction. Can we say, “Skewed”?
None of this leads to trust. Without knowing exactly how that data was collected, the method(s) used and how it was presented on the site and on the app, how can you trust any of it? It’s easy to see professional critic reviews because Rotten Tomatoes must cite back to the source of the review. However, with audience metrics, it’s all nebulous and easily falsified… particularly when Rotten Tomatoes is intentionally obtuse and opaque for exactly how it collects this data and how it is presents it.
Even still, with over one million people attending and viewing The Rise of Skywalker, yet Rotten Tomatoes has only counted just under verified 38,000 people, something doesn’t add up. Yeah, Rotten Tomatoes is so very trustworthy (yeah right), particularly after this “verified” change. Maybe it’s time for those Rotten Tomatoes to finally be tossed into the garbage?
↩︎
Rant Time: Google doesn’t understand COPPA
We all know what Google is, but what is COPPA? COPPA stands for the Children’s Online Privacy Protection Act and is legislation designed to incidentally protect children by protecting their personal data given to web site operators. YouTube has recently made a platform change allegedly around COPPA, but it is entirely misguided. It also shows that Google doesn’t fundamentally understand the COPPA legislation. Let’s explore.
COPPA — What it isn’t
The COPPA body of legislation is intended to protect how and when a child’s personal data may be collected, stored, used and processed by web site operators. It has very specific verbiage describing how and when such data can be collected and used. It is, by its very nature, a data protection and privacy act. It protects the data itself… and, by extension, the protection of that data hopes to protect the child. This Act isn’t intended to protect the child directly and it is misguided to assume that it does. COPPA protects personal private data of children.
By the above, that means that the child is incidentally protected by how their collected data can (or cannot) be used. For the purposes of COPPA, a “child” is defined to be any person under the age of 13. Let’s look at a small portion of the body of this text.
General requirements. It shall be unlawful for any operator of a Web site or online service directed to children, or any operator that has actual knowledge that it is collecting or maintaining personal information from a child, to collect personal information from a child in a manner that violates the regulations prescribed under this part. Generally, under this part, an operator must:
(a) Provide notice on the Web site or online service of what information it collects from children, how it uses such information, and its disclosure practices for such information (§312.4(b));
(b) Obtain verifiable parental consent prior to any collection, use, and/or disclosure of personal information from children (§312.5);
(c) Provide a reasonable means for a parent to review the personal information collected from a child and to refuse to permit its further use or maintenance (§312.6);
(d) Not condition a child’s participation in a game, the offering of a prize, or another activity on the child disclosing more personal information than is reasonably necessary to participate in such activity (§312.7); and
(e) Establish and maintain reasonable procedures to protect the confidentiality, security, and integrity of personal information collected from children (§312.8).
This pretty much sums up the tone for what follows in the body text of this legislation. What it essentially states is all about “data collection” and what you (as a web site operator) must do specifically if you intend to collect specific data from someone under the age of 13… and, more specifically, what data you can and cannot collect.
YouTube and Google’s Misunderstanding of COPPA
YouTube’s parent company is Google. That means that I may essentially interchange “Google” for “YouTube” because both are one-in-the-same company. With that said, let’s understand how Google / YouTube fundamentally does not understand the COPPA body of legislation.
Google has recently rolled out a new feature to its YouTube content creators. It is a checkbox both as a channel wide setting and as an individual video setting. This setting sets a flag whether the video is targeted towards children or not (see image below for this setting’s details). Let’s understand Google’s misunderstanding of COPPA.
COPPA is a data protection act. It is not a child protection act. Sure, it incidentally protects children because of what is allowed to be collected, stored and processed, but make no mistake, it protects collected data directly, not children. With that said, checking a box on a video whether it is appropriate for children has nothing whatever to do with data collection. Let’s understand why.
Google has, many years ago in fact, already implemented a system to prevent “children” (as defined by COPPA) to sign up for and use Google’s platforms. What that means is when someone signs up for a Google account, that person is asked questions to ascertain the person’s age. If that age is identified as under 13, that account is classified by Google as in use by a “child”. Once Google identifies a child, it is then obligated to uphold ALL laws governed by COPPA (and other applicable child privacy laws) … that includes all data collection practices required by COPPA and other applicable laws. It can also then further apply Google related children protections against that account (i.e. to prevent the child from viewing inappropriate content on YouTube). Google would have needed to uphold these data privacy laws since the year 2000, when COPPA was enacted. If Google has failed to protect a child’s collected data or failed to uphold COPPA’s other provisions, then that’s on Google. It is also a situation firmly between Google and the FTC … the governmental body tasked with enforcing the COPPA legislation. Google solely collects the data. Therefore, it is exclusively on Google if that data is used or collected in inappropriate ways, counter to COPPA’s requirements.
YouTube’s newest “not appropriate for children” flag
As of November 2019, YouTube has implemented a new flag for YouTube content creators. The channel-wide setting looks like so:
This setting, for all intents and purposes, isn’t related to COPPA. COPPA doesn’t care whether video content is targeted towards children. COPPA cares about how data is collected from children and how that data is then used by web sites. COPPA is, as I said above, all about data collection practices, not about whether content is targeted towards children.
Let’s understand that in the visual entertainment area, there are already ratings systems which apply. Systems such as the ESRB ratings system founded in 1994. This system specifically sets ratings for video games depending on the types of content contained within. For TV shows, there is the TV Parental Guidelines which began in 1996 and was proposed between the US Congress, the TV industry and FCC. These guidelines rate TV shows such as TV-Y, TV-14 or TV-MA depending, again, on the content within. This was mandated in 1997 by the US Government due to its stranglehold on TV broadcast licenses. For theatrical films, there’s the MPAA’s movie ratings system which began in 1968. So, it’s not as if there aren’t already effective content ratings systems available. These voluntary systems have been in place for many years already.
For YouTube, marking your channel or video content as “made for kids” has nothing whatever to do with COPPA legislated data collection practices.
YouTube Creators
Here is exactly where we see Google and YouTube’s fundamental misunderstanding of COPPA. COPPA is about the protection and collection of data from children. Google collects, stores and uses this and all data it collects. YouTube creators have very, very limited access to any of this Google-collected data. YouTube creators have no hand in its collection or its use. Google controls all of the data collection on YouTube. With the exception of comments and the list of subscribers of a channel, the majority of the data collected and supplied by Google to the creators is almost exclusively limited to aggregate unpersonalized statistical data. Even then, this data can be inaccurate depending on what the Google account ID stated when they signed up. Still, the limited personal subscriber data it does supply to content creators is limited to the subscriber’s ID only. Google offers its content creators no access to deeper personal data, not even the age of its subscribers.
Further, Google (and pretty much every other web site) relies on truthfulness when people sign up for services. Google does not in any way verify the information given to Google during the signup process or that this information is in any way accurate or truthful. Indeed, Google doesn’t even verify the identity of the person using the account or even require the use of real names. The only time Google does ANY level of identity verification is when using Google Wallet. Even then, it’s only as a result of needing identity verification due to possible credit card fraud issues. Google Wallet is a pointless service that many other payment systems do better, such as Apple Pay, Amazon Checkout and, yes, PayPal. I digress.
With that said, Google is solely responsible for all data collection practices associated with YouTube (and its other properties) including storing, processing and managing of that data. YouTube creators have no control over what YouTube (or Google) chooses to collect, store or disseminate. Indeed, YouTube creators have no control over YouTube’s data collection or storage practices whatsoever.
This new alleged “COPPA mechanism” that YouTube has implemented has nothing whatever to do with data collection practices and everything to do with content which might be targeted towards “children”. Right now, this limited mechanism is pretty much a binary system (a very limited system). The channel either does or it doesn’t target content towards children (either channel as a whole or video by video). It’s entirely unclear what happens when you do or don’t via YouTube, though some creators have had seeming bad luck with their content, which has been manually reviewed by YouTube staff and misclassified as “for children” when the content clearly is not. These manual overrides have even run counter to the global channel settings, which have been set to “No, set this channel as not made for kids.”
Clearly, this new mechanism has nothing to do with data collection and everything to do with classifying which content is suitable for children and which isn’t. This defines a …
Ratings System
Ratings systems in entertainment content are nothing new. TV has had a content rating systems since the mid 90s. Movies have had ratings systems since the late 60s. Video games have had them since the mid 90s. COPPA, on the other hand, has entirely nothing to do with ratings or content. It is legislation that protects children by protecting their data. It’s pretty straightforward what COPPA covers, but one thing it does not cover is whether video content is appropriate to be viewed by children. Indeed, COPPA isn’t a ratings system. It is child data protection legislation.
How YouTube got this law’s interpretation so entirely wrong is anyone’s guess. I can’t even fathom how Google could have been led this astray. Perhaps Google’s very own lawyers are simply inept and not at all versed in COPPA? I have no idea… but whatever led YouTube’s developers to thinking the above mechanism in any way relates to COPPA is entirely wrong thinking. No where does COPPA legislate YouTube video content appropriateness. Categorizing content is entirely up to a ratings system to handle.
Indeed, YouTube is trudging on very thin ice with the FTC. Not only did they interpret the COPPA legislation completely wrong, they have implemented “a fix” even more wrongly. What Google and YouTube has done is shoot themselves in the foot… not once, but twice. The second time is that Google has fully admitted that they don’t even have a functional working ratings system. Indeed, it doesn’t… and now everyone knows it.
Google has now additionally admitted that children under the age of 13 use YouTube by the addition of this “new” mechanism. With this one mechanism, Google has admitted to many things about children using its platform… which means YouTube and Google are both now in the hot seat with regards to COPPA. They must now completely ensure that YouTube (and Google by extension) is fully and solely complying with the letter of COPPA’s verbiage by collecting children’s data.
YouTube Creators Part II
YouTube creators have no control over what Google collects from its users, that’s crystal clear. YouTube creators also don’t have access to view most of this data or access to modify anything related to this data collection system. Only Google has that level of access. Because Google controls its own data collection practices, it is on Google to protect any personal information it may have received by children using its platform.
That also means that content creators should be entirely immune from prosecution over such data collection practices… after all, the creators don’t own or control Google’s data collection systems.
This new YouTube mechanism seems to imply that creators have some level of liability and/or culpability for Google’s collection practices, when creators simply and clearly do not. Even the FTC made a striking statement that they may try to “go after” content creators. I’m not even sure how that’s possible under COPPA. Content creators don’t collect, store or manage data about children, regardless of the content that they create. The only thing content creators control is appropriateness of the content towards children… and that has nothing to do with COPPA and everything to do with a ratings system… a system that Google does not even have in place within YouTube.
Content creators, however, can voluntarily label their content as TV-MA or whatever they deem is appropriate based on the TV Parental Guidelines. After all, YouTube is more like TV than it is like a video game. Therefore, YouTube should offer and have in place the same ratings system as is listed in the TV Parental Guidelines. This recent COPPA-attributed change is actually YouTube’s efforts at enacting a content ratings system, albeit an extremely poor attempt at one. As I said, creators can only specify the age appropriateness of the content that they create. YouTube is simply the platform where it is shown.
FTC going after YouTube Creators?
Google controls its data collections systems, not its content creators (though YouTube does hold leverage over whether content is or remains monetized). What that means is that it makes absolutely no sense for the FTC to legally go after content creators based on violations of COPPA. There may be other legislation they can lean on, but COPPA isn’t it. COPPA also isn’t intended to be a “catch all” piece of legislation to protect children’s behaviors on the Internet. It is intended to protect how data is collected and used by children under 13 years of age… that’s it. COPPA isn’t intended to be used as a “ratings system” for appropriateness by video sharing platforms like YouTube.
I can’t see even one judge accepting, let alone prosecuting such a clear cut case of legal abuse of the justice system. Going after Google for COPPA violations? Sure. They stored and collected that data. Going after the YouTube content creators? No, I don’t think so. They created a video and uploaded it, but that had nothing whatever to do with how Google controls, manages or collects data from children.
If the US Federal Government wants to create law to manage appropriateness of Internet content, then they need to draft it up and pass it. COPPA isn’t intended for that purpose. Voluntary ratings systems have been in place for years including within motion pictures, TV and now video games. So then why is YouTube immune from such rating systems? Indeed, it’s time YouTube was forced to implement a proper ratings system instead of this haphazard binary system under the false guise of COPPA.
Content Creator Advice
If you are a YouTube content creator (or create on any other online platform), you should take advantage of the thumbnail and describe the audience your content targets. The easiest way to do this is to use the same ratings system implemented by the TV Parental Guidance system… such as TV-Y, TV-14 and TV-MA. Placing this information firmly on the thumbnail and also placing it onto the video at the beginning of your video explicitly states towards which age group and audience your content is targeted. By voluntarily rating not only the thumbnail, but also the content itself in the first 5 minutes of the video opening, your video cannot be misconstrued for any other group or audience. This means that even though your video is not intended for children, placing the TV Parental Guidance rating literally onto the video intentionally states that fact in plain sight.
If a YouTube employee manually reclassifies your video as being “for children” even when it isn’t, labeling your content in the video’s opening as TV-MA explicitly states that the program is not suitable for children. You might even create an additional disclaimer as some TV programs do stating:
This content is not suitable for all audiences. Some content may be considered disturbing or controversial. Viewer or parental discretion is advised.
Labeling your video means that even the FTC can’t argue that your video somehow inappropriately targeted children… even though this new YouTube system has nothing to do with COPPA. Be cautious, use common sense and use best practices when creating and uploading videos to YouTube. YouTube isn’t there to protect you, the creator. The site is there to protect YouTube and Google. In this case, this new creator feature is entirely misguided as a COPPA helper, when it is clearly intended to be a ratings system.
Before you go…
One last thing… Google controls everything about the YouTube platform including the “recommended” lists of videos. If, for whatever reason, Google chooses to promote a specific video towards an unintended audience, the YouTube creator has no control over this fact. In point of fact, the content creator has almost no control over any promotion or placement of their video within YouTube. The only exception is if YouTube allows for paid promotion of video content (and they probably do). After all, YouTube is in it for the $$$. If you’re willing to throw some of your money at Google, I’m quite sure they’d be willing to help you out. Short of paying Google for video placement, however, all non-paid placement is entirely at the sole discretion of Google. The YouTube creator has no control over their video’s placement within “recommended” lists or anywhere else on YouTube.
↩︎
Random Thoughts - Randocity! 
leave a comment