Random Thoughts – Randocity!

Software Engineering and Architecture

Posted in botch, business, Employment by commorancy on October 21, 2018

ExcellenceHere’s a subject of which I’m all too familiar and is in need of commentary. Since my profession is technical in nature, I’ve definitely run into various issues regarding software engineering, systems architecture and operations. Let’s Explore.

Software Engineering as a Profession

One thing that software engineers like is to be able to develop their code on their local laptops and computers. That’s great for rapid development, but it causes many problems later, particularly when it comes to security, deployment, systems architecture and operations.

For a systems engineer / devops engineer, the problem arises when that code needs to be productionalized. This is fundamentally a problem with pretty much any newly designed software system.

Having come from from a background of systems administration, systems engineering and devops, there are lots to be considered when wanting to deploy freshly designed code.

Designing in a Bubble

I’ve worked in many companies where development occurs offline on a notebook or desktop computer. The software engineer has built out a workable environment on their local system. The problem is, this local eneironment doesn’t take into account certain constraints which may be in place in a production environment such as internal firewalls, ACLs, web caching systems, software version differences, lack of compilers and other such security or software constraints.

What this means is that far too many times, deploying the code for the first time is fraught with problems. Specifically, problems that were not encountered on the engineer’s notebook… and problems that sometimes fail extremely bad. In fact, many of these failures are sometimes silent (the worst kind), where everything looks like it’s functioning normally, but the code is sending its data into a black hole and nothing is actually working.

This is the fundamental problem with designing in a bubble without any constraints.

I understand that building something new is fun and challenging, but not taking into account the constraints the software will be under when finally deployed is naive at best and reckless at the very worse. It also makes life as a systems engineer / devops engineer a living hell for several months until all of these little failures are sewn shut.

It’s like receiving a garment that looks complete, but on inspection, you find a bunch of holes all over that all need to be fixed before it can be worn.

Engineering as a Team

To me, this is situation means that software engineer is not a team player. They might be playing on the engineering team, but they’re not playing on the company team. Part of software design is designing for the full use case of the software, including not only code authoring, but systems deployment.

If systems deployment isn’t your specialty as a software engineer, then bring in a systems engineer and/or devops engineer to help guide your code during the development phase. Designing without taking the full scope of that software release into consideration means you didn’t earn your salary and you’re not a very good software engineer.

Yet, Silicon Valley is willing to pay “Principal Engineers” top dollar for these folks failing to do their jobs.

Building and Rebuilding

It’s an entirely a waste of time to get to the end of a software development cycle and claim “code complete” when that code is nowhere near complete. I’ve had so many situations where software engineers toss their code to us as complete and expect the systems engineer to magically make it all work.

It doesn’t work that way. Code works when it’s written in combination with understanding of the architecture where it will be deployed. Only then can the code be 100% complete because only then will it deploy and function without problems. Until that point is reached, it cannot be considered “code complete”.

Docker and Containers

More and more, systems engineers want to get out of the long drawn out business of integrating square code into a round production hole, eventually after much time has passed, molding the code into that round hole is possible. This usually takes months. Months that could have been avoided if the software engineer had designed the code in an environment where the production constraints exist.

That’s part of the reason for containers like Docker. When a container like Docker is used, the whole container can then be deployed without thought to square pegs in round holes. Instead, whatever flaws are in the Docker container are there for all to see because the developer put it there.

In other words, the middle folks who take code from engineering and mold it onto production gear don’t relish the thought of ironing out hundreds of glitchy problems until it seamlessly all works. Sure, it’s a job, but at some level it’s also a bit janitorial, wasteful and a unnecessary.


Part of the reason for these problems is the delineation between the engineering teams and the production operations teams. Because many organizations separate these two functional teams, it forces the above problem. Instead, these two teams should be merged into one and work together from project and code inception.

When a new project needs code to be built that will eventually be deployed, the production team should be there to move the software architecture onto the right path and be able to choose the correct path for that code all throughout its design and building phases. In fact, every company should mandate that its software engineers be a client of operations team. Meaning, they’re writing code for operations, not the customer (even though the features eventually benefit the customer).

The point here is that the code’s functionality is designed for the customer, but the deploying and running that code is entirely for the operations team. Yet, so many software engineers don’t even give a single thought to how much the operations team will be required support that code going forward.

Operational Support

For every component needed to support a specific piece of software, there needs to be a likewise knowledgeable person on the operations team to support that component. Not only do they need to understand that it exists in the environment, the need to understand its failure states, its recovery strategies, its backup strategies, its monitoring strategies and everything else in between.

This is also yet another problem that software engineers typically fail to address in their code design. Ultimately, your code isn’t just to run on your notebook for you. It must run on a set of equipment and systems that will serve perhaps millions of users. It must be written in ways that are fail safe, recoverable, redundant, scalable, monitorable, deployable and stable. These are the things that the operations team folks are concerned with and that’s what they are paid to do.

For each new code deployment, that makes the environment just that much more complex.

The Stacked Approach

This is an issue that happens over time. No software engineer wants to work on someone else’s code. Instead, it’s much easier to write something new and from scratch. It’s easy for software engineer, but it’s difficult for the operations team. As these new pieces of code get written and deployed, it drastically increases the technical debt and burden on the operations staff. Meaning, it pushes the problems off onto the operations team to continue supporting more and more and more components if none ever get rewritten or retired.

In one organization where I worked, we had such an approach to new code deployment. It made for a spider’s web mess of an environment. We had so many environments and so few operations staff to support it, the on-call staff were overwhelmed with the amount of incessant pages from so many of these components.

That’s partly because the environment was unstable, but that’s partly because it was a house of cards. You shift one card and the whole thing tumbles.

Software stacking might seem like a good strategy from an engineering perspective, but then the software engineers don’t have to first line support it. Sometimes they don’t have to support it at all. Yes, stacking makes code writing and deployment much simpler.

How many times can engineering team do this before the house of cards tumbles? Software stacking is not an ideal any software engineering team should endorse. In fact, it’s simply comes down to laziness. You’re a software engineer because writing code is hard, not because it is easy. You should always do the right thing even if it takes more time.

Burden Shifting

While this is related to software stacking, it is separate and must be discussed separately. We called this problem, “Throwing shit over the fence”. It happens a whole lot more often that one might like to realize. When designing in a bubble, it’s really easy to call “code complete” and “throw it all over the fence” as someone else’s problem.

While I understand this behavior, it has no place in any professionally run organization. Yet, I’ve seen so many engineering team managers endorse this practice. They simply want their team off of that project because “their job is done”, so they can move them onto the next project.

You can’t just throw shit over the fence and expect it all to just magically work on the production side. Worse, I’ve had software engineers actually ask my input into the use of specific software components in their software design. Then, when their project failed because that component didn’t work properly, they threw me under the bus for that choice. Nope, that not my issue. If your code doesn’t work, that’s a coding and architecture problem, not a component problem. If that open source component didn’t work in real life for other organizations, it wouldn’t be distributed around the world. If a software engineer can’t make that component work properly, that’s a coding and software design problem, not an integration or operational problem. Choosing software components should be the software engineer’s choice to use whatever is necessary to make their software system work correctly.

Operations Team

The operations team is the lifeblood of any organization. If the operations team isn’t given the tools to get their job done properly, that’s a problem with the organization as a whole. The operations team is the third hand recipient of someone else’s work. We step in and fix problems many times without any knowledge of the component or the software. We do this sometimes by deductive logic, trial and error, sometimes by documentation (if it exists) and sometimes with the help of a software engineer on the phone.

We use all available avenues at our disposal to get that software functioning. In the middle of the night the flow of information can be limited. This means longer troubleshooting times, depending on the skill level of the person triaging the situation.

Many organizations treat its operations team as a bane, as a burden, as something that shouldn’t exist, but does out of necessity. Instead of treating the operations team as second class citizens, treat this team with all of the importance that it deserves. This degrading view typically comes top down from the management team. The operations team is not a burden nor is it simply there out of necessity. It exists to keep your organization operational and functioning. It keeps customer data accessible, reliable, redundant and available. It is responsible for long term backups, storage and retrieval. It’s responsible for the security of that data and making sure spying eyes can’t get to it. It is ultimately responsible to make sure the customer experience remains at a high excellence standard.

If you recognize this problem in your organization, it’s on you to try and make change here. Operations exists because the company needs that job role. Computers don’t run themselves. They run because of dedicated personnel who make it their job and passion to make sure those computers stay online, accessible and remain 100% available.

Your company’s uptime metrics are directly impacted by the quality of your operations team staff members. These are the folks using the digital equivalent of chewing gum and shoelaces to keep the system operating. They spend many a sleepless night keeping these systems online. And, they do so without much, if any thanks. It’s all simply part of the job.

Software Engineer and Care

It’s on each and every software engineer to care about their fellow co-workers. Tossing code over the fence assuming there’s someone on the other side to catch it is insane. It’s an insanity that has run for far too long in many organizations. It’s an insanity that needs to be stopped and the trend needs to reverse.

In fact, by merging the software engineering and operations teams into one, it will stop. It will stop by merit of having the same bosses operating both teams. I’m not talking about at a VP level only. I’m talking about software engineering managers need to take on the operational burden of the components they design and build. They need to understand and handle day-to-day operations of these components. They need to wear pagers and understand just how much operational work their component is.

Only then can engineering organizations change for the positive.

As always, if you can identify with what you’ve read, I encourage you to like and leave a comment below. Please share with your friends as well.



Is Obama hostile towards big business?

To answer this question, we need to delve a little deeper. Note, I am neither condoning nor praising Obama’s handling of his regulatory efforts. However, I would like to point out certain corrections that do need to be made.

“The truth is that not even the Franklin Roosevelt administration was as hostile to and ignorant about free enterprise as this [Obama’s] administration is.”
–Steve Forbes.

But, is Obama really hostile towards business? Or, is he making needed corrections? There is a fine line here. This issue also points out a serious problem in politics today. That problem is, you guessed it, money. Without money, the world doesn’t work. Without money, candidates don’t get elected. Without money, businesses don’t sell things and make money. Back up the train.. Businesses make plenty of money without governmental help. The trouble is that businesses want to be able to make laws that enable their businesses to make more money and then have the government be lenient with them when issues arise.

The reality, though, is that like the separation of church and state, the government now needs separation of business and state. The two are oil and water, they don’t mix. Government needs to be able to make law without interference from any party. But, businesses have deep pockets and hefty lawyers. These two elements help elect officials and help sway these same officials into making good on promises they made towards these businesses during the election.

Obama’s corrections

While I don’t agree with every single thing Obama has done, I do agree that change is necessary. The change that he is making is intended to correct the issues that led to the economic downturn. The trouble comes with statements from people like Steve Forbes. Mr. Forbes believes that he is the end-all-be-all-know-it-all when it comes to all-things-business. The trouble is, he doesn’t. Yes, he runs a successful magazine, but that doesn’t make him an authority. That makes him a successful business owner.

Obama is walking that fine line. A fine line that shouldn’t even be necessary. But, there it is. The line that’s there to help Obama help the economy, help spur business and growth and reduce the chances of a repeated failure. At the same time, the line is there to show that government values business, but isn’t there to socialize it. The trouble is, this economic downturn was of our own making. By our, I mean Wall Street. The housing bubble was just that, a bubble. Bubbles eventually burst and this bubble was no exception. It’s not as if analysts and intelligent minded people couldn’t see the handwriting on the wall. When the mortgage interest rates got down to 1% and all of those ARM and specialty loans were being issued like water flowing down the Mississippi, trouble was inevitable. We just didn’t know that banks and insurance companies were tying their financial soundness to these extremely risky loans using credit default swaps.

Until the bubble burst, no one really knew just how deep the rabbit hole went. Then, everything came crashing down and all of the nasty subprime mortgage and credit default swap issues came into view in their all fugly detailed glory. The first evidence of that was Bear Stearns followed by AIG (and the subsequent governmental bailout). I still think they should have let AIG fold, I digress.

Government and Business

It’s high time that government distanced itself from corporate businesses. It’s high time congress made laws to separate government from business (including political support). It’s high time that government stopped being a pawn for corporate businesses. Forbes clearly seems to think that Free Enterprise requires socialism to function. Free Enterprise is not part of and does not need socialism. Free Enterprise means that businesses can do whatever they need to do (within the limits of the laws) to make their business succeed. Clearly, there have not been laws enabled that have dramatically impacted Free Enterprise. The laws that have been enacted have been placed there to prevent corporations from producing risky investment vehicles with a high likelyhood of crashing down again. If businesses are now floundering, it’s not because of laws. It’s because corporations have lost their way and are still expecting handouts. Well, you can keep your hand out, but don’t expect the government to be dropping any coin in it.

Corporations have relied, no… depended on the US Government for handouts. That time needs to end. Subsidies for business need to go away. Businesses need to fend for themselves just like Free Enterprise mandates. If a business can’t make it on its own, then let it fail. I’ll repeat, LET IT FAIL. Failure is also part of Free Enterprise. Businesses that will succeed, will succeed because they produce a good product or service. Businesses that fail, will fail because they don’t produce good products or services.

Lost our way

America, and specifically corporate enterprises, have lost their way. For far too long have big corporations depended on favorable governmental conditions (sounds like a weather report) to help them stay in business. Well, that train has left (and must leave). It should be solely up to you and your business practices alone to make or break your company. It is the quality of your products, services and support that makes people want to buy your products or invest in your company. Nothing has changed about this aspect of Free Enterprise.

We need to go back to a time when quality was the key. When providing a superior product was the answer to getting people to buy things. If that also means deflation, then so be it. Businesses need to find their way by learning how to do more with less. How to manage their staff better and stop over-hiring. At the same time, many of them need to stop under-hiring and also value the employees that they have right now.

The key to keeping your business flowing is by keeping your employees active, productive and happy. Morale is a big problem in companies during any downturn. Once fear sets in over the next reduction in force (RIF), then morale falls to all-time-lows. No, taking the employees on an outing doesn’t boost morale. The way to boost morale is to stop RIFing the staff out the door. Yes, I know it gives a temporary boost to the stock price and makes the shareholders happy, but that’s a temporary fix with limited effects. Once the dust settles, the employees who are left become disgruntled, unhappy and produce less. This is completely backwards thinking. Which is why business has lost its way.

Shareholder value vs quality products

I know, someone’s going to say that it is all about ‘shareholder value’. That may be the way things seem now, but it is wrong. Currently accepted actions that lead to improved shareholder value tend to undercut production, stifle innovation, reduce profit margins and lower productivity. Why would you intentionally do this to your business? So, while these measures may seem to help the stock price, it does nothing to help the company improve its quality of products and services. In fact, in the long run, these actions almost always negatively impact the bottom line. So, the fundamental question is, are you in business to make the shareholders happy or are you in business to sell quality products and services? This fundamental question must be answered.

The true answer to this question also shows that Free Enterprise priorities today are all wrong. It used to be that the customer is #1. Now, shareholders are #1 and customers are #2. This is both wrong and stupid. Until businesses go back to the idea that the customer is #1, corporations will continue to fail and need governmental subsidies. While shareholders are considered #1, there is really no such thing as Free Enterprise when it comes to multi-million dollar corporations… which is why they always need a handout from the government.

%d bloggers like this: