Should I beta test Fallout 76?
While I know that beta testing for Fallout 76 is already underway, let’s explore what it means to beta test a game and whether or not you should participate.
Fallout 76
Before I get into the nitty gritty details of beta testing, let’s talk about Fallout 76. Fallout 76, like The Elder Scrolls Online before it, is a massively multiplayer online role playing game (MMORPG). Like The Elder Scrolls Online which offered an Elder Scrolls themed universe, Fallout 76 will offer a Fallout themed universe in an online landscape.
How the game ultimately releases is yet to be determine, but a beta test gives you a solid taste of how it will all work. Personally, I didn’t like The Elder Scrolls Online much. While it had the flavor and flair of an Elder Scrolls game entry, the whole thing felt hollow and unconnected to the franchise. It also meant that Bethesda spent some very valuable time building this online game when they could have been building the next installment of the Elder Scrolls.
It is as yet undetermined how these online games play into the canon of The Elder Scrolls or, in Fallout 76’s case, in the Fallout universe. Personally, I see them as offshoots with only a distant connection. For example, The Elder Scrolls Online felt Elder Scrollsy, but without the deep solid connections and stories that go with building that universe. Instead, it was merely a multiplayer playground that felt like The Elder Scrolls in theme, but everything else was just fluff. I’m deeply concerned that we’ll get this same treatment from Fallout 76.
The Problem with Online Games
Online games have, in recent years, gotten a bad rap… and for good reason. The reason that this is so is because the game developers focus on the inclusion of silly things like character emoting and taking selfies. While these are fun little inclusions, they are by no means intrinsic to the fundamental game play of an actual game.
Games should be about the story that unfolds… about why your character is there and how your character is important in that universe. When the game expands to include an online component, now it’s perhaps tens of thousands of people all on the server at the same time. So, how can each of these characters be important to that universe? The answer is, they can’t.
Having many characters all running around doing the “same” things in the universe all being told by the game that they are “the most important thing” to the survival of that universe is just ludicrous.
This leads to the “importance syndrome” which is present in any MMORPG. As a developer, you either acknowledge the importance syndrome and avoid it by producing a shallow multiplayer experience that entirely avoids player importance (i.e., Fortnite, Overwatch, Destiny, etc) or you make everyone important each in their own game (i.e., The Elder Scrolls Online). Basically, the game is either a bunch of people running around doing nothing important at all and simply trying to survive whatever match battles have been set up (boring and repetitive) or the game treats each user as if they are individually important in their own single player game, except there are a bunch of other users online, all doing the same exact thing.
The Elder Scrolls Online fell into the latter camp which made the game weird and disconnected, to say the least. It also made the game feel less like an Elder Scrolls game and more like any cheap and cheesy iPad knockoff game you can download for free… except you’ve paid $60 + DLC + online fees for it.
I’ve played other MMORPG games similar to The Elder Scrolls Online including Defiance. In fact, Defiance played so much like The Elder Scrolls Online, I could swear that Bethesda simply took Defiance’s MMORPG engine and adapted it to The Elder Scrolls Online.
Environments and Users
The secondary problem is how to deal with online users. Both in the Elder Scrolls Online and Defiance, there were areas that included player versus environment (PvE). PvE environments mean that players cannot attack other players. Only NPCs can attack your player or your character can die by the environment (i.e., falling onto spikes). There were also some areas of the online map that were player versus player (PvP). PvP means any online player can attack any other online player in any way they wish.
In The Elder Scrolls Online, the PvP area was Cyrodiil, which was unfortunate for ESO. The PvP made this territory mostly a dead zone for the game. Even though there were a few caves in the area and some exploring you could do, you simply couldn’t go dungeon diving there because as soon as you tried, some player would show up and kill your player. Yes, the NPCs and AI enemies could also show up and kill your player, but so could online players.
The difficulty with Cyrodiil was that if another player killed your player in the PvP area, that player death was treated entirely differently than if they died by the environment. If another player killed your character, you had to respawn at a fort, which would force your character to respawn perhaps half a map away from where you presently were. If your character died by the environment or another NPC, you could respawn in the same location where your character died. This different treatment in handling the character death was frustrating, to say the least.
With Fallout 76, I’m unsure how all of this will work, but it’s likely that Bethesda will adopt a similar strategy from what they learned in building The Elder Scrolls Online. This likely means both PvE areas and PvP area(s). Note that ESO only had one PvP zone, but had many PvE zones. This made questing easier in the PvE zones, but also caused the “importance syndrome”. This syndrome doesn’t exist in single player offline games, but is omnipresent in MMORPGs.
MMORPGs and Characters
The difficulty with MMORPGs is that they’re primarily just clients of a server based environment. The client might be a heavier client that includes handling rendering character and environment graphics, but it is still nonetheless a client. This means that to use an MMORPG, you must log into the server to play. When you login, your character information, bank account, level ups, weapons, armor and so on are kept on the server.
This means that you can’t save off your character information. It also means you can’t mod your game or mod your character through game mods. Online games are strict about how you can change or manage your game and your character. In fact, these systems are so strict that if a new version of the game comes out, you must first download and install the game before they’ll let you back onto the server… unlike standalone games that let you play the game even if networking components are disabled. This means that you cannot play an MMORPG until your client is most current, which could mean 50GB and hours later.
This means that you’ll need an always on Internet connection to play Fallout 76 and you’ll need to be able to handle very large client downloads (even if you own the game disc).
Beta Testing
Many game producers like to offer, particularly if it’s a server based MMORPG, the chance for players to beta test their new game. Most online games allow for this.
However, I refuse to do this for game developers. They have a team they’ve hired to beta test their environments, quests and landscapes. I just don’t see any benefit for my player to get early access to their game environment. Sometimes, characters you build and grow in a beta won’t even carry over into the released game. This means that whatever loot you have found and leveling you may have done may be lost when release day comes. For that early access, the developer will also expect you to submit bug reports. I won’t do that for them. I also don’t want to feel obligated to do so.
Bethesda stands to make millions of dollars off of this game. Yet, they’re asking me to log into their game early, potentially endure huge bugs preventing quest progress, potentially lose my character and all of its progress and also spend time submitting bug reports? Then, spend $60 to buy the game when it arrives? Then, rebuild my character again from scratch?
No, I don’t think so. I’m not about to spend $60 for the privilege of spending my time running into bugs and submitting bug reports for that game. You, the game developer, stand to make millions from this game. So, hire people to beta test it for you. Or, give beta testers free copies of the game in compensation for the work they’re doing for you.
If you’re a gamer thinking of participating in beta testing, you should think twice. Not only are you helping Bethesda to make millions of dollars, you’re not going to see a dime of that money and you’re doing that work for free. In addition, you’re still going to be expected to spend $60 + DLC costs to participate in the final released game. No, I won’t do that. If I’m doing work for you, you should pay me as a contractor. How you pay me for that work is entirely up to you, but the minimum payment should consist of a free copy of the game. You can tie that payment to work efforts if you like.
For example, for each report submitted and verified as a new bug, the beta tester will get $5 in credit towards the cost of the game up to the full price of the game. This encourages beta testers to actually submit useful bug reports (i.e., duplicates or useless reports won’t count). This also means you earn your game as you report valid and useful bugs. It also means that you won’t have to pay for the game if you create enough useful, genuine reports.
Unfortunately, none of these game developers offer such incentive programs and they simply expect gamers to do it “generously” and “out of the kindness of their hearts”. No, I’m not doing that for you for free. Pay me or I’ll wait until the game is released.
Should I Participate in Beta Tests?
As a gamer, this is why you should not participate in beta tests. Just say no to them. If enough gamers say no and fail to participate in beta releases, this will force game developers to encourage gamers to participate with incentive programs such as what I suggest above.
Unfortunately, there are far too many unwitting gamers who are more than willing to see the environment early without thinking through the ramifications of what they are doing. For all of the above reasons, this is why you should NEVER participate (and this is why I do not participate) in any high dollar game beta tests.
↩︎
Software Engineering and Architecture
Here’s a subject of which I’m all too familiar and is in need of commentary. Since my profession is technical in nature, I’ve definitely run into various issues regarding software engineering, systems architecture and operations. Let’s Explore.
Software Engineering as a Profession
One thing that software engineers like is to be able to develop their code on their local laptops and computers. That’s great for rapid development, but it causes many problems later, particularly when it comes to security, deployment, systems architecture and operations.
For a systems engineer / devops engineer, the problem arises when that code needs to be productionalized. This is fundamentally a problem with pretty much any newly designed software system.
Having come from from a background of systems administration, systems engineering and devops, there are lots to be considered when wanting to deploy freshly designed code.
Designing in a Bubble
I’ve worked in many companies where development occurs offline on a notebook or desktop computer. The software engineer has built out a workable environment on their local system. The problem is, this local eneironment doesn’t take into account certain constraints which may be in place in a production environment such as internal firewalls, ACLs, web caching systems, software version differences, lack of compilers and other such security or software constraints.
What this means is that far too many times, deploying the code for the first time is fraught with problems. Specifically, problems that were not encountered on the engineer’s notebook… and problems that sometimes fail extremely bad. In fact, many of these failures are sometimes silent (the worst kind), where everything looks like it’s functioning normally, but the code is sending its data into a black hole and nothing is actually working.
This is the fundamental problem with designing in a bubble without any constraints.
I understand that building something new is fun and challenging, but not taking into account the constraints the software will be under when finally deployed is naive at best and reckless at the very worse. It also makes life as a systems engineer / devops engineer a living hell for several months until all of these little failures are sewn shut.
It’s like receiving a garment that looks complete, but on inspection, you find a bunch of holes all over that all need to be fixed before it can be worn.
Engineering as a Team
To me, this is situation means that software engineer is not a team player. They might be playing on the engineering team, but they’re not playing on the company team. Part of software design is designing for the full use case of the software, including not only code authoring, but systems deployment.
If systems deployment isn’t your specialty as a software engineer, then bring in a systems engineer and/or devops engineer to help guide your code during the development phase. Designing without taking the full scope of that software release into consideration means you didn’t earn your salary and you’re not a very good software engineer.
Yet, Silicon Valley is willing to pay “Principal Engineers” top dollar for these folks failing to do their jobs.
Building and Rebuilding
It’s an entirely a waste of time to get to the end of a software development cycle and claim “code complete” when that code is nowhere near complete. I’ve had so many situations where software engineers toss their code to us as complete and expect the systems engineer to magically make it all work.
It doesn’t work that way. Code works when it’s written in combination with understanding of the architecture where it will be deployed. Only then can the code be 100% complete because only then will it deploy and function without problems. Until that point is reached, it cannot be considered “code complete”.
Docker and Containers
More and more, systems engineers want to get out of the long drawn out business of integrating square code into a round production hole, eventually after much time has passed, molding the code into that round hole is possible. This usually takes months. Months that could have been avoided if the software engineer had designed the code in an environment where the production constraints exist.
That’s part of the reason for containers like Docker. When a container like Docker is used, the whole container can then be deployed without thought to square pegs in round holes. Instead, whatever flaws are in the Docker container are there for all to see because the developer put it there.
In other words, the middle folks who take code from engineering and mold it onto production gear don’t relish the thought of ironing out hundreds of glitchy problems until it seamlessly all works. Sure, it’s a job, but at some level it’s also a bit janitorial, wasteful and a unnecessary.
Planning
Part of the reason for these problems is the delineation between the engineering teams and the production operations teams. Because many organizations separate these two functional teams, it forces the above problem. Instead, these two teams should be merged into one and work together from project and code inception.
When a new project needs code to be built that will eventually be deployed, the production team should be there to move the software architecture onto the right path and be able to choose the correct path for that code all throughout its design and building phases. In fact, every company should mandate that its software engineers be a client of operations team. Meaning, they’re writing code for operations, not the customer (even though the features eventually benefit the customer).
The point here is that the code’s functionality is designed for the customer, but the deploying and running that code is entirely for the operations team. Yet, so many software engineers don’t even give a single thought to how much the operations team will be required support that code going forward.
Operational Support
For every component needed to support a specific piece of software, there needs to be a likewise knowledgeable person on the operations team to support that component. Not only do they need to understand that it exists in the environment, the need to understand its failure states, its recovery strategies, its backup strategies, its monitoring strategies and everything else in between.
This is also yet another problem that software engineers typically fail to address in their code design. Ultimately, your code isn’t just to run on your notebook for you. It must run on a set of equipment and systems that will serve perhaps millions of users. It must be written in ways that are fail safe, recoverable, redundant, scalable, monitorable, deployable and stable. These are the things that the operations team folks are concerned with and that’s what they are paid to do.
For each new code deployment, that makes the environment just that much more complex.
The Stacked Approach
This is an issue that happens over time. No software engineer wants to work on someone else’s code. Instead, it’s much easier to write something new and from scratch. It’s easy for software engineer, but it’s difficult for the operations team. As these new pieces of code get written and deployed, it drastically increases the technical debt and burden on the operations staff. Meaning, it pushes the problems off onto the operations team to continue supporting more and more and more components if none ever get rewritten or retired.
In one organization where I worked, we had such an approach to new code deployment. It made for a spider’s web mess of an environment. We had so many environments and so few operations staff to support it, the on-call staff were overwhelmed with the amount of incessant pages from so many of these components.
That’s partly because the environment was unstable, but that’s partly because it was a house of cards. You shift one card and the whole thing tumbles.
Software stacking might seem like a good strategy from an engineering perspective, but then the software engineers don’t have to first line support it. Sometimes they don’t have to support it at all. Yes, stacking makes code writing and deployment much simpler.
How many times can engineering team do this before the house of cards tumbles? Software stacking is not an ideal any software engineering team should endorse. In fact, it’s simply comes down to laziness. You’re a software engineer because writing code is hard, not because it is easy. You should always do the right thing even if it takes more time.
Burden Shifting
While this is related to software stacking, it is separate and must be discussed separately. We called this problem, “Throwing shit over the fence”. It happens a whole lot more often that one might like to realize. When designing in a bubble, it’s really easy to call “code complete” and “throw it all over the fence” as someone else’s problem.
While I understand this behavior, it has no place in any professionally run organization. Yet, I’ve seen so many engineering team managers endorse this practice. They simply want their team off of that project because “their job is done”, so they can move them onto the next project.
You can’t just throw shit over the fence and expect it all to just magically work on the production side. Worse, I’ve had software engineers actually ask my input into the use of specific software components in their software design. Then, when their project failed because that component didn’t work properly, they threw me under the bus for that choice. Nope, that not my issue. If your code doesn’t work, that’s a coding and architecture problem, not a component problem. If that open source component didn’t work in real life for other organizations, it wouldn’t be distributed around the world. If a software engineer can’t make that component work properly, that’s a coding and software design problem, not an integration or operational problem. Choosing software components should be the software engineer’s choice to use whatever is necessary to make their software system work correctly.
Operations Team
The operations team is the lifeblood of any organization. If the operations team isn’t given the tools to get their job done properly, that’s a problem with the organization as a whole. The operations team is the third hand recipient of someone else’s work. We step in and fix problems many times without any knowledge of the component or the software. We do this sometimes by deductive logic, trial and error, sometimes by documentation (if it exists) and sometimes with the help of a software engineer on the phone.
We use all available avenues at our disposal to get that software functioning. In the middle of the night the flow of information can be limited. This means longer troubleshooting times, depending on the skill level of the person triaging the situation.
Many organizations treat its operations team as a bane, as a burden, as something that shouldn’t exist, but does out of necessity. Instead of treating the operations team as second class citizens, treat this team with all of the importance that it deserves. This degrading view typically comes top down from the management team. The operations team is not a burden nor is it simply there out of necessity. It exists to keep your organization operational and functioning. It keeps customer data accessible, reliable, redundant and available. It is responsible for long term backups, storage and retrieval. It’s responsible for the security of that data and making sure spying eyes can’t get to it. It is ultimately responsible to make sure the customer experience remains at a high excellence standard.
If you recognize this problem in your organization, it’s on you to try and make change here. Operations exists because the company needs that job role. Computers don’t run themselves. They run because of dedicated personnel who make it their job and passion to make sure those computers stay online, accessible and remain 100% available.
Your company’s uptime metrics are directly impacted by the quality of your operations team staff members. These are the folks using the digital equivalent of chewing gum and shoelaces to keep the system operating. They spend many a sleepless night keeping these systems online. And, they do so without much, if any thanks. It’s all simply part of the job.
Software Engineer and Care
It’s on each and every software engineer to care about their fellow co-workers. Tossing code over the fence assuming there’s someone on the other side to catch it is insane. It’s an insanity that has run for far too long in many organizations. It’s an insanity that needs to be stopped and the trend needs to reverse.
In fact, by merging the software engineering and operations teams into one, it will stop. It will stop by merit of having the same bosses operating both teams. I’m not talking about at a VP level only. I’m talking about software engineering managers need to take on the operational burden of the components they design and build. They need to understand and handle day-to-day operations of these components. They need to wear pagers and understand just how much operational work their component is.
Only then can engineering organizations change for the positive.
As always, if you can identify with what you’ve read, I encourage you to like and leave a comment below. Please share with your friends as well.
↩︎
Rant Time: Apple Music vs Twitter
I know I’ve been on a tirade with the number of rants recently, but here we are. I rant when there’s something to rant about. This time it’s about sharing Apple Music playlists on Twitter… and just how badly this feature is broken. Worse, just how Apple itself is broken. Let’s explore.
Twitter Cards
Twitter has a feature they call Twitter cards. It’s well documented and requires a number of meta tags to be present in an HTML page. When the page is shared via Twitter, Twitter goes looking at the HTML for its respective Twitter meta tags to generate a Twitter card.
A Twitter card comes in two sizes and looks something like this:
Small Twitter Card
Large Twitter Card
What determines the size of the Twitter card seems to be the size and ratio of the image. If the image is square in size (144×144 or larger), Twitter creates a small card as shown at the top. If the image ratio is not square and larger than 144×144, Twitter produces a large Twitter card. The difference between the cards is obvious:
- Small card has an image to the left and text to the right
- Large card has image above and text below
It’s up to the person sharing on Twitter to decide which size is most appropriate. Personally, I prefer the larger size because it allows for a much larger image.
Apple Music Playlist Sharing
Here’s where the RANT begins… hang onto your hat’s folks. Apple’s engineering team doesn’t get Twitter cards…. AT. ALL! Let me give an example of this. Here’s a playlist I shared on Twitter:
What’s wrong with this Twitter card? If you guessed the image is way too tiny, you’d win. Apple doesn’t understand the concept of producing a 144×144 image properly. Here’s the fundamental problem. In iTunes, my playlist image is uploaded with a 1200×1200 size image. This image is well large enough for any use on the net. Here’s how it looks in iTunes, albeit scaled somewhat small:
Note, iTunes retains the full image size, but scales the image as needed. If you look at the playlist on the web, it looks like this with a much larger scaled image:
As you can see, the image scales properly and still looks good even larger. Yes, even large enough to produce a 144×144 image on a Twitter card.
Here’s the Twitter card metadata on that Apple Music Preview page:
meta id="1" name="twitter:title" content="AstroWorld Pioneer by Klearnote" class="ember-view"
meta id="2" name="twitter:description" content="Playlist · 22 Songs" class="ember-view"
meta id="3" name="twitter:site" content="@appleMusic" class="ember-view">
meta id="4" name="twitter:domain" content="Apple Music" class="ember-view">
meta id="5" name="twitter:image"
content="https://is5-ssl.mzstatic.com/image/thumb/SG-S3-US-Std-Image-000001/v4/a2/c6/6f/a2c66fc6-a63b-f590-c6db-e41aebfc327c/image/600x600wp.png"
class="ember-view"
meta id="6" name="twitter:card" content="summary" class="ember-view"
You’ll notice that the text in red above is the piece that is relevant. Let’s look at that image now…
You’ll notice that the playlist image content is centered at 213×213 pixels in size centered in a light grey box that’s 600×600. Yes, that thick light grey border is part of the image. This is actually how the image is being produced by Apple on their servers. That would be okay if the image were scaled to the full 600×600 pixels. Unfortunately, it isn’t. Twitter will scale any image to its preferred size of 144×144 pixels for small Twitter cards. Here’s what a 144×144 image looks like when scaled by WordPress:
Small, but reasonably clear. Here’s Twitter’s crap scaled and unreadable version:
I have no idea what Twitter is using to scale its images, but it looks like absolute trash. The bigger problem isn’t that Twitter has scaled this image down, it’s that Apple has provided Twitter with such an already small and crap looking playlist image. Why have a 144×144 image if you’re only going to use 1/9th of the entire space? Apple, why wouldn’t you not want to use the entire 144×144 image space to make the image look like this:
That sized image would make the Twitter card look like this…
… instead of this absolute shit looking card…
How the Mighty Have Fallen
Apple used to be a well respected company who always prided itself on doing things correctly and producing high quality products. Today, they’re a shadow of their former selves. Producing products as crap as this only serves as a detriment to all of the other products they now offer. It’s clear, Apple Music is an afterthought and Apple seems to have only one engineer assigned to this software product… maybe none.
It’s also clear, Apple doesn’t respect the standards of anyone, not even themselves. I consider this absolute crap attention to detail. Seriously, who wants their images to be scaled to the point of being unreadable? No one!
Yet, when I called Apple Support to report this issue, I was told, “This is expected behavior”. Expected by whom? Who would ever expect an image to be scaled the point of nonrecognition? No one. If this is the level of software development effort we’re now seeing from Apple, then I don’t even want to think what corners are being cut on their hardware products.
What’s next? Apple watches catching on fire and exploding on people’s wrists? Phones taking out people’s ears? If I can no longer trust Apple to uphold the standards of high quality, then the mighty have truly fallen. There is no hope for Apple no matter how much crap they try to peddle.
Apple, Hear Me!
If you are serious about your business, then you need to be serious about all aspects including offering high quality products, services and features. This goes all the way to playlist sharing on Twitter. My experience with dealing with Apple in this matter was so amateur, including the way Apple Music itself is being handled, why should I continue to use your products? Give me a reason to pay you $99 for such shit service! Seriously, in addition to the above, I’m also finding what appear to be bootlegged music products on Apple Music and yet you’re pawning it off as official releases?
And as suggested by your representative, why should I contact Twitter for this issue? Twitter’s features work properly when provided with the correct information. As has been stated for years in software engineering, “Garbage In, Garbage Out”. It is you, Apple, who are providing Twitter with garbage information. It’s not a Twitter problem, it’s an Apple problem. Also, because this is an Apple engineering problem to solve, why should I contact Twitter on Apple’s behalf? I don’t work for you. You need to have YOUR engineering team contact Twitter and have them explain to you the errors of your ways.
This is just the tip of the iceberg here. There’s so much wrong at Apple, if you continue to entrust your family’s safety into Apple’s products, you may find one of your family members injured or dead. Apple, wake up and learn to take quality seriously.
The next time you are shopping for a computer or a watch device, you need to ask yourself, “Do I really trust Apple to provide safe choices for me or my family?”
Apple has now officially and truly reached the level of shit!
Broken Apple Image credit: The King of The Vikings via DeviantArt
↩︎
Random Thoughts - Randocity! 
leave a comment