Random Thoughts – Randocity!

I’ve been playing Fallout 76 on and off since its release. However, Bethesda has not only miscalculated the quality of the game itself, Bethesda’s devs have repeatedly introduced more bugs than they have fixed. So far, the patches have been a strategy of one step forward and three steps back. This game has all of the signs of code outsourcing and illustrates all of the dangers of this practice. Let’s explore.

Game Development

Having worked at many different high tech companies that write code for their business to succeed, I have seen many different code writing practices… some good, many more that are bad.

Typically, when code is written “in-house” (meaning, by developers on site at the headquarters), the quality control remains at a “standard bar” set by the development manager. This doesn’t mean that every piece of code written is great, but it does mean that the bad code likely won’t make it into production after “code review”. The “code review” process is a process by which all code is peer reviewed by other developers to make sure the code is up to formatting standards, that it doesn’t make any egregious mistakes and that such things as math calculations make sense. Comments in the code are usually optional and up to the development team to set how code gets documented.

I’ve worked at many companies where code is not documented at all. Instead, the documentation is written in a Wiki or similar internal web site describing the design goals of the code. I don’t particularly like this practice when working on the production side of the house, but it’s generally not a practice we can win a fight against. Reading documentation in the code is sometimes the only defense when code acts up in production. If they choose not to write inline documentation, that’s on the development team. Though, I will say that this practice leads to technical debt and is not recommended.

Without diving too deeply into code development practices, let’s apply all of what I’ve said to Bethesda’s Fallout 76.

Bad Coding Practices

I don’t even know where to begin with how Bethesda is managing this product. Let’s just say that having worked in several large Software-as-a-Service (SaaS) organizations, how Bethesda is handling Fallout 76 is so behind the times, it’s not even funny.

Today, the current practice is to use the following code development cycle otherwise known as Systems Development Life Cycle (SDLC). This cycle has the following phases:

1. Planning
2. Analysis and Requirements
3. Design
4. Development
5. Integration and Testing
6. Implementation / Release
7. Operational Maintenance

These 7 phases are a never ending cycle in a continuing software product. In fact, there could be several releases all running in concert each at different phases. Meaning, the current release is at phase 7, the next release is at phase 4 and two or three future releases are at at any of the phases prior to release.

The SDLC process has grown out of bad coding practices used during the 90s and has been adopted to counter those bad coding and release practices. This life cycle is a way to ensure quality of code when it is finally released. It’s also a way to ensure that the end user has the highest quality experience possible with the end product. Quality assurance is the name of the game. As a development company, the key to success is to minimize disruptions via bad code and maximize user experience with high quality features. No code is ever perfect, but you can reduce problems by following best coding practices and implementing solid SDLC processes.

Bethesda’s Coding Practices

Unfortunately, Bethesda has chosen a poor coding cycle for Fallout 76. Instead of treating Fallout 76 like a professionally produced product using SLDC practices, they are simply slinging code as fast as possible without actually performing any sanity checks or, indeed, performing any quality assurance on the end product.

In fact, with each new Fallout 76 release, the product has become increasingly worse, less reliable, less performant and increasingly more unstable. By “worse”, I mean that they’re introducing not only regression bugs that break previously and correctly working features, they’re introducing new bugs and not even fixing the bugs they claimed to have fixed. Indeed, the product is actually getting worse.

While I realize that coding a game like Fallout 76 is probably reasonably complex, the difficulty I have with this game is why a developer is touching code in portions of the game where bugs did not exist. What were they doing touching that part of the code? And yet, here we are, this newly broken code is being rolled out to their production servers?

Clearly, Bethesda performs absolutely zero testing. These bugs are so basic, anyone spending 5 minutes using the game would spot them instantly. It’s crystal clear that Bethesda is NOT following an SDLC process. They’re just releasing code by the seat of their pants and hoping it does “something positive”.

Outsourcing

Because Fallout 76 gets worse with each successive release, this has all of the telltale signs of Bethesda outsourcing their software development efforts to an off-shore team (possibly in India). Having worked with outsourced developers in India in the past, you MUST micromanage these outsourced companies at every tiny step. You also need to be extremely explicit with how you want the implementation and you need to 100% test every piece of delivered code.

Failing to micromanage an outsourced software development company leads to the exact problems seen in Fallout 76. While I can’t be 100% certain that Bethesda is outsourcing, their release practices certainly have all of the earmarks of using this practice for Fallout 76. There’s absolutely no reason why previously working features in the game should inexplicably become broken in the next release.

And believe me, I’ve become exceedingly tired and irritated of fighting these compounding stupid bugs in this game. Not only does it show Bethesda as a low quality developer, it says they have no quality standards of any kind. You don’t intentionally roll out broken features in a formerly working product… you just don’t do this.

Chasing Abusers

Bethesda has clearly bitten off more than they can chew. They certainly have no one on their team who understands SaaS product scaling. If one gamer on the server crafting boiled water can bring the server to its knees, there’s a major problem with this product. In a properly designed multi-user product, no single user should be able to overload the server with any “standard” user interface activity. By “standard”, I mean features that the product is supposed to properly support.

There are many instances where a single user can craft foods at a crafting table which causes “Server Not Responding” or incurs major lag for other users on the server. These are sanctioned activities intended to be used by the users, yet they can break the server?

Abusers, on the other hand, find loopholes to allow them to perform activities that the software was not designed to do. For example, duplicating items by logging on and off in very unusual ways… ways in which the developer didn’t test or consider during the design phase.

Right now, Bethesda is chasing down these unintentional holes at all costs… and by that I mean, by introducing game breaking bugs that affect standard users who are not abusing. And, they’re attempting to fix these holes at the cost of ignoring the design failings of the game that also need to be addressed. Many of these design failings were introduced at release and are still waiting in the queue to be addressed by Bethesda. Yet, instead of taking care of these long standing bugs, the devs are flying by the seat of their pants fixing the holes… which honestly don’t need to be fixed as a fire drill.

Penalizing Players

Bethesda doesn’t understand the dangers of reduction. Removing or degrading product features is always a negative for the end user, never a positive. For example, the Two Shot Explosive weapons are what players long to find. These rarely dropped highly powerful weapons are, in fact, one of the sole reasons players come back to the Fallout franchise.

Sure, the questing is fun, but it’s the Legendary dropped loot from these difficult bosses that is the actual Win. It’s the trophy that says, “Hey look what I got after spending all that time defeating the Scorchbeast Queen”. By degrading, limiting and/or removing these highly sought after items from the game, this removes a substantial reason to even play Fallout 76.

If you spend an hour defeating a boss only to see it drop a 12 damage Pipe Pistol (the same as a Level 1 enemy kill), what have you really accomplished? How does that make the gamer feel? Does it make the gamer feel good about what they’ve just done? No. Does it make the gamer want to come back and do it again? No.

On the Wrong Track

Bethesda is entirely on the wrong track. If you have abusers in the game, chase them down and ban them… no holds barred. If you find a player who is carrying 300,000 weight in duplicated items, ban them. Remove them from the game. Find them all and remove them. Logic dictates that anyone carrying 25,000 Stimpaks along with 25,000 guns stacked didn’t create them through legitimate means.

You ban the abusers. You don’t code around them. You don’t hobble your universe to make the duplicated guns worthless. Instead of spending precious time alienating your intended gaming audience, you focus on making the game better for legitimate users willing to stay within the game’s design framework. For those who stray and choose to test the coding boundaries of the game, you ban them… permanently. You also make a warning statement that any persons intending to cause harm to or disrupt the services will be banned without warning. In-game abuse can only be dealt with one way, the ban hammer.

Yes, you can fix the bugs along the way that enabled that abuse, but you don’t make that your sole and entire means of existing. You focus on fixing the bugs that are getting in the way of your legitimate paying gamers who are willing to stay on the game’s “golden path”. By “golden path”, this is a software development phrase that means the track designed by the developers for end users of the software product to follow. Anyone who strays from the “golden path” may encounter bugs, unexpected consequences or crash the software system. Though, your developers should have coded proper error handling so that crashing is nearly impossible.

Yes, some users can unintentionally stray from the “golden path” occasionally. These users are not the target. It’s the users who intentionally stray from the “golden path” to exploit holes in the software to gain access, privilege or items which are unintended. Speaking of gaining access….

Dev Room

There’s been much controversy over this room. Personally, I don’t care if it exists or not. However, that this room made it into Bethesda’s Fallout 76 production servers is entirely a design miss. Such dev environments should never make it onto production servers. That this room rolled out onto the production network is a problem Bethesda needs to address internally. Users who stray from the golden path into this room isn’t the fault of the gamer. Bethesda, you left the room in the game. It was your responsibility to ensure such rooms don’t exist on the production servers. That users ended up in there, that’s your mistake, Bethesda.

Sure, you can drop the ban hammer on these users, but that’s not good public relations. In fact, dropping the ban hammer on users for entry into this room is severe. If they didn’t cause damage to the game or take anything from the room, there’s no damage done. Those users who took items from the dev room and duplicated them should be banned… not necessarily for entering the room, but for exploiting the duplication bug which disrupts a server for other players.

Again, it comes back to disruption. Any gamer intentionally causing disruption to the game outside of the “golden path” should be perma-banned. This act of disruption should be spelled out as abuse in the terms and conditions for the game.

Fallout 76 is SaaS

Even though Fallout 76 is a game, it’s also a Software-as-a-Service product and it should be treated in the same way as any SaaS product. Yet, Bethesda hasn’t the first clue of how to build or operate a SaaS product. That’s crystal clear.

Bethesda’s SDLC seems non-existent. Without any kind of software quality assurance team, there’s no way to ensure the product lives up to any kind of quality standard. Right now, this game is a piss poor attempt by a game studio at a SaaS product. A product that is on the verge of being a spectacular failure. I might even argue, it’s already reached the failure point.

Bethesda, you have a hard choice to make. If you continue to chase the abusers at the cost of fixing the REAL problems with this game, your game WILL DIE. The choice you need to make is whether to stay on this insane path of chasing abuse bugs or stop this insanity and begin fixing the real reliability and stability problems with this game. Such real problems include severe frame rate drops, enemies can spawn in unkillable states, invisibility problems (enemies and players alike), the problem with quests that can’t be completed, the problem where Legendary enemies drop without any loot at all.

Game Economy and Systems Design

Bethesda continually argues that the abusers caused disruption to the economy in the game. What economy? There is none. If you call vendors with 200 caps an economy, that’s not an economy. An economy is players buying, selling and trading with one another. You know, the whole reason you designed the game with 24 players in each “World”. Yet, when players actually tried to create an economy, you shut them down with patches and then released many of the rare items to the vendors to make them “less rare”.

Part of the reason items were rare was entirely due to incidence of spawn rates. Spawn rates, I might add, that you designed into the game intentionally. Spawn rates intended to force players to hunt for stuff. Yet now you’re all butthurt over the fact that players actually created an economy around this.

What exactly are you wanting the players to do in this game then? Aren’t the players supposed to “rebuild” the wasteland? Setting up trading shops and whatnot is exactly what players would do in a world like this. In fact, in the ruthless wild-west of the wasteland, players would likewise be ruthless in obtaining anything and everything they could. That players used duplication exploits comes with the ruthlessness of wasteland territory. The problem with the duplication exploit isn’t the duplication. It’s the disruption it causes to other player’s games. That’s the abuse vector. That’s the reason to ban-hammer the player. The server disruption is the abuse, not the duplication.

Still, you should have been warning players all along the way when their weight got too high. That you didn’t have anything in place to monitor this part of the game is a design miss. A miss that wouldn’t have been missed if you had had a proper Systems Engineer reviewing the design all along the way. Yet, you chose to rush the game to market unfinished and now you have to redesign it along the way… a redesign that is causing player unrest and player abandonment.

Patch Upsides vs Downsides

The last several patches have been attempts at thwarting the abusers by fixing the exploit vectors at the cost of not fixing long standing disrupting bugs… bugs that have existed since the game’s release (i.e., getting stuck in power armor, unkillable enemies, invisible enemies, loading screen problems, etc). This strategy has been to the entire detriment of the Fallout 76 gaming community. Not only have you alienated so many users from the game, you continue to alienate more and more with each new patch.

If you’re planning on releasing a patch, you need to focus on the upsides of patching. You know, like fixing bugs that players NEED to have fixed… like frame rate issues, like audio glitching, like server lags, like a bigger stash, like improved features. Sure, you can throw in fixes like nerfing the Two Shot and Explosive weapons, but you also need to offset these heavily negative gaming experiences by adding positive new things to the game to entice gamers back… like adding new weapons to the game to take the place of those heavily nerfed Two Shot Explosive weapons.

There’s no reason for gamers to play Fallout 76 if the Legendary dropped loot is now no better than standard dropped loot. Focusing entirely on downside patches isn’t going to win you new players. It’s simply a quick way to the death of Fallout 76… as if the game needed any more help in this department.

Overall

Bethesda, you need to rethink your strategy for Fallout 76 and future MMO endeavors. The current strategy you are taking to address the issues in this game will not bring more players to this game. In fact, you’re likely to turn this game into a wasteland with only a handful of players ever playing.

If you stay on this path, I predict that you will end up shutting down your servers for this game by the end of 2019. Gamers won’t continue to play in an environment where the loot is not worth their time.

And what the hell? Serum recipes cost 19,000+ caps? Considering you can only hold 24,000 caps in the game, this is insane. Even 6,000 caps would be excessive.

Bethesda, figure it out quick or the game ends.

↩︎