Random Thoughts – Randocity!

Why Rotten Tomatoes is rotten

Posted in botch, business, california by commorancy on December 31, 2019

cinema-popcornWhen you visit a site like Rotten Tomatoes to get information about a film, you need to ask yourself one very important question, “Is Rotten Tomatoes trustworthy?”

Rotten Tomatoes as a movie review service has come under fire many times for review bombing and manipulation. That is, Rotten Tomatoes seem to allow shills to join the service to review bomb a movie to either raise or lower its various scores by manipulating the Rotten Tomatoes review system. In the past, these claims couldn’t be verified. Today, they can.

As of a change in May 2019, Rotten Tomatoes has made it exceedingly easy for both movie studios and Rotten Tomatoes itself to game and manipulate the “Audience Score” ratings. Let’s explore.

Rotten Tomatoes as a Service

Originally, Rotten Tomatoes began its life as an independent movie review service such that both critics and audience members can have a voice in what they think of a film. So long as Rotten Tomatoes remained an independent and separate service from movie studio influence and corruption, it could make that claim. Its reviews were fair and for the most part accurate.

Unfortunately, all good things must come to an end. In February of 2016, Fandango purchased Rotten Tomatoes. Let’s understand the ramifications of this purchase. Because Fandango is wholly owned by Comcast and in which Warner Brothers also holds an ownership stake in Fandango, this firmly plants Rotten Tomatoes well out of the possibility of remaining neutral in film reviews. Keep in mind that Comcast also owns NBC as well as Universal Studios.

Fandango doesn’t own a stake in Disney as far as I can tell, but that won’t matter based on what I describe next about the Rotten Tomatoes review system.

Review Bombing

As stated in the opening, Rotten Tomatoes has come under fire for several notable recent movies as having scores which have been manipulated. Rotten Tomatoes has then later debunked those claims by stating that their system was not manipulated, but then really offering no proof of that fact. We simply have to take them at their word. One of these allegedly review bombed films was Star Wars: The Last Jedi… where the scores inexplicably dropped dramatically in about a 1 month period of time. Rotten Tomatoes apparently validated the drop as “legitimate”.

Unfortunately, Rotten Tomatoes has become a bit more untrustworthy as of late. Let’s understand why.

As of May of 2019, Rotten Tomatoes introduced a new feature known as “verified reviews”. For a review’s score to be counted towards the “Audience Score”, the reviewer must have purchased a ticket from a verifiable source. Unfortunately, the only source from which Rotten Tomatoes can verify ticket purchases is from its parent company, Fandango. All other ticket purchases don’t count… thus, if you choose to review a film ater you’ve purchased your ticket via the box office, from MovieTickets.com or directly at theaters won’t count as “verified” should you review or rate the movie.

Here’s what Rotten Tomatoes has to say from the linked article just above:

Rotten Tomatoes now features an Audience Score made up of ratings from users we’ve confirmed bought tickets to the movie – we’re calling them “Verified Ratings.” We’re also tagging written reviews from users we can confirm purchased tickets to a movie as “Verified” reviews.

While this might sound like a great idea in theory, it’s ripe for manipulation problems. It also states that “IF” Fandango can confirm “other” reviews as confirmed ticket purchases, they will mark them as “verified”. Yeah, but that’s a manual process and is impossibly difficult to determine. Let’s list the problems coming out of this change:

  1. Fandango only sells a small percentage of overall ticket sales for a film. If the “Audience Score” is calculated primarily and solely from Fandango ticket sales alone, then this metric is a horribly inaccurate metric to rely on.
  2. Fandango CAN handpick “other” non-Fandango ticket purchased reviews to be included. Not likely to happen often, but this also means they can pick their favorites (and favorable) reviews to include. This opens Rotten Tomatoes up to Payola or “pay for inclusion”.
  3. By specifying exactly how this process works, this change opens the Rotten Tomatoes system to being gamed and manipulated, even by Rotten Tomatoes staff themselves. Movie studios can also ask their employees, families and friends to exclusively purchase their tickets from Fandango and request these same people to write “glowing, positive reviews” or submit “high ratings” or face job consequences.
  4. Studios can even hire outside people (sometime known as shills) to go see a movie by buying tickets from Fandango and then rate files highly because they were paid to do so. As I said, manipulation.

Trust in Reviews

It’s clear that while Rotten Tomatoes is trying to fix its ills, it is incredibly naive at doing so. Not only is Rotten Tomatoes incredibly naive, it’s also not at all tech savvy. Its system is so ripe for being gamed, the Audience Score now is almost a pointless metric.

Case in point. The “Audience Score” for The Rise of Skywalker is 86%. The difficulty with this number is the vast majority of the reviews I’ve seen from people on chat forums don’t rate the film anywhere close to 86%. What that means is that the new way that Rotten Tomatoes is calculating scores is effectively a form of manipulation itself BY Rotten Tomatoes.

To have the most fair and accurate metric, ALL reviews must be counted and included in all ratings. You can’t just toss out the vast majority of reviews simply because you can’t verify them has holding a ticket. Even still, holding a ticket doesn’t mean someone has actually watched the film. Buying a ticket and actually attending a showing of the film are two entirely separate things.

While you may have verified a ticket purchase, did you verify that the person actually watched the film? Are you withholding brand new Rotten Tomatoes account reviewers out of the audience score? How trustworthy can someone be if this is their first and only review on Rotten Tomatoes? What about people who downloaded the app just to buy a ticket for that film? Simply buying a ticket from Fandango doesn’t make the rating or reviewer trustworthy.

Rethinking Rotten Tomatoes

Someone at Rotten Tomatoes needs to drastically reconsider this change and they need to do it fast. If Rotten Tomatoes wasn’t guilty of manipulation of review scores before this late spring change in 2019, they are now. Rotten Tomatoes is definitely guilty of manipulating the “Audience Score” by the amount of reviews covered under this “verified review” change. Verifying a ticket holder also doesn’t validate the review authors sincerity, intent or, indeed, legitimacy. It also severely limits who can be counted under their ratings.

In fact, only looking at past reviews can someone determine if a review author has trustworthy opinions.

Worse, Fandango holds a very small portion of all ticket sales made for theaters (see below). By showing all (or primarily) scores tabulated by people who bought tickets from Fandango, this definitely eliminates well over half of the written reviews on Rotten Tomatoes as valid. Worse, because of the way the metric is calculated, nefarious entities can game the system to their own benefit and manipulate the score quickly.

This has a chilling effect on Rotten Tomatoes. The staff at Rotten Tomatoes needs roll back this change pronto. For Rotten Tomatoes to return it being a trustworthy neutral entity in the art of movie reviews, it needs a far better way to determine trustworthiness of its reviews and of its reviewers. Trust comes from well written, consistent reviews. Ratings come from trusted sources. Trust is earned. The sole act of buying a ticket from Fandango doesn’t earn trust. It earns bankroll.

Why then are ticket buyers from Fandango any more trustworthy than people purchasing tickets elsewhere? They aren’t… and here’s where Rotten Tomatoes has failed. Rotten Tomatoes incorrectly assumed that by “verifying” a sale of a ticket via Fandango alone, that somehow makes a review or rating more trustworthy. It doesn’t.

It gets worse because while Fandango represents at least 70% of online sales, it STILL only represents a tiny fraction of overall ticket sales, at just 5-6% (as of 2012).

“Online ticketing still just represents five to six percent of the box office, so there’s tremendous potential for growth right here.” –TheWrap in 2012

Granted, this TheWrap article is from 2012. Even if Fandango had managed to grab 50% of the overall ticket sales in the subsequent 7 years since that article, that would leave out 50% of the remaining ticket holder’s voices, which will not be tallied into Rotten Tomatoes current “Audience Score” metric. I seriously doubt that Fandango has managed to achieve anywhere close to 50% of total movie ticket sales. If it held 5-6% overall sales in 2012, in 7 years Fandango might account for growth between 10-15% at most by 2019. That’s still 85% of all reviews excluded from Rotten Tomatoes’s “Audience Score” metric.  In fact, it behooves Fandango to keep this overall ticket sales number as low as possible so as to influence its “Audience Score” number with more ease and precision.

To put this in a little more perspective, a movie theater might have 200 seats. 10% of that is 20. That means that for every 200 people who might fill a theater, just less than 20 people have bought their ticket from Fandango and are eligible for their review to count towards “Audience Score”. Considering that only a small percentage of that 20 will actually take the time to write a review, that could mean out of every 200 people who’ve seen the film legitimately, between 1 and 5 people might be counted towards the Audience Score. Calculating that up, for very 1 million people who see a blockbuster film, somewhere between 5,000 and 25,000’s reviews may contribute to the Rotten Tomatoes “Audience Score”… even if there are hundreds of thousands of reviews on the site.

The fewer the reviews contributing to that score, the easier it is to manipulate that score by adding just a handful of reviews to the mix… and that’s where Rotten Tomatoes “handpicked reviews” come into play (and with it, the potential for Payola). Rotten Tomatoes can then handpick positive reviews for inclusion. The problem is that while Rotten Tomatoes understands all of this this, so do the studios. Which means that studios can, like I said above, “invite” employees to buy tickets via Fandango before writing a review on Rotten Tomatoes. They can even contact Rotten Tomatoes and pay for “special treatment”. This situation can allow movie studios to unduly influence the “Audience Score” for a current release… this is compounded because there are actually so few reviews that actually count to create this score.

Where Rotten Tomatoes likely counted every review towards this score before this change, after they instituted its “verified score” methodology, that greatly dropped the number of reviews which now contribute to tallying that score. This lower number of reviews means that it is now much easier to manipulate its Audience Score number either by gaming the system or by Rotten Tomatoes handpicking reviews to include.

Fading Trust

While Rotten Tomatoes was once a trustworthy site for movie reviews, it has greatly reduced its trust levels by instituting such backwards and easily manipulable systems.

Whenever you visit a site like Rotten Tomatoes, you must always question everything you see. When you see something like an “Audience Score”, you must question how that number is calculated and what is included in that number. Rotten Tomatoes isn’t forthcoming.

In the case of Rotten Tomatoes, they have drastically reduced the number of included reviews in that metric because of their “verified purchase” mechanism. Unfortunately, the introduction of that mechanism at once destroys Rotten Tomatoes trust and trashes the concept of their site.

It Gets Worse

What’s even more of a problem is the following two images:

Screen Shot 2019-12-23 at 7.26.58 AM

Screen Shot 2019-12-23 at 7.26.24 AM

From the above two images, it is claimed Rotten Tomatoes has 37,956 “Verified Ratings”, yet they only have 3,342 “Verified Audience” reviews. That’s a huge discrepancy. Where are those other 34,614 “Verified” reviews? You need to calculate the Audience Score not solely on a phone device using a simplistic “rate this movie” alone. It must be calculated in combination with an author writing a review. Of course, there are 5,240 reviews that didn’t at all contribute to any score at all on Rotten Tomatoes. Those audience reviews are just “there” taking up space.

Single number ratings are pointless without at least some text validation information. Worse, we know that these “Verified Ratings” likely have nothing to do with “Verified Audience” as shown in the images above. Even if those 3,342 audience reviews are actually calculated into the “Verified Ratings” (they probably aren’t), that’s still such a limited number when considered with the rest of the “Verified Ratings” so as to be skewed by people who may not have even attended the film.

You can only determine if someone has actually attended a film by asking them to WRITE even the smallest of a review. Simply pressing “five star” on the app without even caring is pointless. It’s possible the reviews weren’t even tabulated correctly via the App. The App itself may even submit star data after a period of time without the owner’s knowledge or consent. The App can even word its rating question in such a way as to manipulate the response in a positive direction. Can we say, “Skewed”?

None of this leads to trust. Without knowing exactly how that data was collected, the method(s) used and how it was presented on the site and on the app, how can you trust any of it? It’s easy to see professional critic reviews because Rotten Tomatoes must cite back to the source of the review. However, with audience metrics, it’s all nebulous and easily falsified… particularly when Rotten Tomatoes is intentionally obtuse and opaque for exactly how it collects this data and how it is presents it.

Even still, with over one million people attending and viewing The Rise of Skywalker, yet Rotten Tomatoes has only counted just under verified 38,000 people, something doesn’t add up. Yeah, Rotten Tomatoes is so very trustworthy (yeah right), particularly after this “verified” change. Maybe it’s time for those Rotten Tomatoes to finally be tossed into the garbage?

↩︎

Rant Time: Adobe VoCo’s ethical dilemma

Posted in best practices, botch, business, california, ethics by commorancy on February 28, 2018

I have to wonder about Adobe’s business ethics at times. First, there’s Photoshop. While I can admit that photo editing has a legitimate purpose, such as correcting red eye or removing telephone lines or removing reflections of the camera man from a photo, there is the much seedier and ethically murky purpose for Photoshop. Now comes Adobe VoCo. It is a product idea that does for spoken audio what Photoshop does for images. Let’s explore this YouTube clip from 2016:

Skip to 3:18 for the meat of this video.

VoCo’s Use Cases and Ethics

Though, yes, I will concede that the demonstration above was funny and we all laughed, the demonstration has a deep seated ethically murky undertone once the laughing stops. In fact, that’s what prompted this blog article.

Unlike Photoshop which has actual real world use cases (yes, other than making models thinner and glowier for the cover of Vogue), VoCo is one of those unnecessary tools that, while cool in theory, makes Adobe seem that it’s now in the business of causing world disruption instead of actually solving creative problems. After the ethical problems created by Photoshop, Adobe has to know the ethical quandary it introduces by bringing the VoCo audio editing tool to market. Adobe decides to go ahead with demoing this tool anyway. So much for business ethics. Instead, Adobe should have patented and shelved this product idea and never shown it off.

There’s no effective real world use case for this product other than for making someone say things that they actually didn’t say. The only use case where this technology might even be somewhat useful, depending on output quality, is in the voice over industry where an actor might be unavailable at a time when a line needs to be changed to fit continuity better. The voice over industry is the only industry where VoCo could have even the smallest glimmer of hope of a use case. This is such a tiny niche market segment to introduce this tool in such a public spectacle way.

The only other use case would be to sample all of the audio from a particular dead actor or actress’s productions and then recreate lines of new spoken dialog based on that. Again, this is one of those entertainment areas that fits firmly into the uncanny valley, particularly if the spoken lines are attached to a CG actor. Again, this is not a substantial use case in my opinion and is most definitely creepy. It’s definitely not a big enough use case to warrant this public release spectacle. Do we really want to see Marilyn Monroe or Elvis brought back to life on the big screen using CG and VoCo dialog?

There is no other legitimate use case for this product. It’s like Adobe intentionally wants to flaunt its lack of ….

Business Ethics and Self-Editing

Businesses today have no ability to self-edit or recognize ethics. That is, stop ethically bad product ideas from making it to the market. Just thinking about this product and how it could possibly be used, it doesn’t have legitimate use cases (other than the voice over use case I mentioned above). However, there are perhaps thousands of illegitimate uses for this tool. Let’s list a few of them, shall we:

  • Falsifying a deposition to make the person being deposed say something they didn’t say
  • Falsifying a statement of non-confession to make a person confess to a crime when they didn’t actually confess
  • Falsifying a phone conversation
  • Changing any spoken words from non-incriminating to incriminating evidence

In legal circles, the use for this tool is ripe for abuse and has use cases as wide as the Grand Canyon and as deep as the Mariana Trench. In other words, while VoCo has no substantial legitimate use cases, it has thousands of illegitimate use cases. There is no way Adobe couldn’t see this. There is no way for Adobe to feign ignorance about this tool or the ethical problems it imposes if released.

Legal Evidence

Some have theorized that this tool would become just as Photoshop has. Basically, because evidence can now be manufactured in products like VoCo, it means that audio evidence would no longer be easily admissible. While that idea has some soundness to it, the legal system is not always technically savvy and can sometimes move at a snail’s pace. Eventually, the courts and lawyers will be on board with this ‘manufactured evidence’ sound clip idea, but not before several someones are incriminated over manufactured evidence that isn’t caught in time.

Some have theorized that Adobe should watermark the sound clip. The difficulty with audio watermarking is that it ruins the audio. No one would buy a professional audio tool that intentionally makes the audio sound bad or introduces something that is audibly noticeable, strictly because Adobe wants to insert a watermark to legally cover their collective butts. No. No one would buy a tool that causes damage to the audio output. This means that only a silent kind of watermark could be introduced. Such a watermark would consist primarily as a tag within the saved audio clip file. Any tags introduced in a save file can easily be stripped away by converting the audio clip to a new format or by playing the audio clip back and recording it on analog equipment. In fact, a whole industry and set of tools would likely appear to strip out any watermarks imposed by Adobe onto the saved files.

Unless there is a substantial way to identify that the clip has been edited, and I don’t know how Adobe could even solve this problem fully, VoCo is a tool that would end up more abused than legitimately used.

Flawed Product Ideas

While this is somewhat of a cool technological advancement, it doesn’t need to exist. It doesn’t need to exist because it has basically one limited use case. I’d argue that as a production runner, you can just wait until the voice actor becomes available and ask them to re-record the lines you need. That is, instead of using a tool like this. A tool like VoCo might save you some time, but by demanding such a tool for your use, it means the rest of the world must also endure the consequences of a world full of falsified evidence. Is that the world you want to live in? Evidence that could even be used against you, the audio editor. No, thanks.

However, it’s clear that prototype code has been written based on the video above. This means that Adobe could release such a product into the wild in the future. Thankfully, as of this article in 2018, this product does not yet exist. Unfortunately, Adobe has already opened Pandora’s box. A working prototype means that any coder with leanings towards audio engineering could produce a similar tool and release it into the wild without the help of Adobe. Thanks Adobe.

It is as yet unclear when or if this product could ever be released. Note that this video segment apparently showcases experimental product ideas (products that may never see the light of day) and not actual products. After all, such a legally murky product would have to clear Adobe’s legal team before release. Considering the many negative use cases for such an audio editing product and the legal liability that Adobe might endure as a result, I’d hope that Adobe’s legal team has shelved this product idea permanently.

Agree or disagree? Please leave a comment below. Also, don’t miss any new Randocity articles by subscribing to this blog via clicking the blue follow button at the top right.

%d bloggers like this: