Software Engineering and Architecture
Here’s a subject of which I’m all too familiar and is in need of commentary. Since my profession is technical in nature, I’ve definitely run into various issues regarding software engineering, systems architecture and operations. Let’s Explore.
Software Engineering as a Profession
One thing that software engineers like is to be able to develop their code on their local laptops and computers. That’s great for rapid development, but it causes many problems later, particularly when it comes to security, deployment, systems architecture and operations.
For a systems engineer / devops engineer, the problem arises when that code needs to be productionalized. This is fundamentally a problem with pretty much any newly designed software system.
Having come from from a background of systems administration, systems engineering and devops, there are lots to be considered when wanting to deploy freshly designed code.
Designing in a Bubble
I’ve worked in many companies where development occurs offline on a notebook or desktop computer. The software engineer has built out a workable environment on their local system. The problem is, this local eneironment doesn’t take into account certain constraints which may be in place in a production environment such as internal firewalls, ACLs, web caching systems, software version differences, lack of compilers and other such security or software constraints.
What this means is that far too many times, deploying the code for the first time is fraught with problems. Specifically, problems that were not encountered on the engineer’s notebook… and problems that sometimes fail extremely bad. In fact, many of these failures are sometimes silent (the worst kind), where everything looks like it’s functioning normally, but the code is sending its data into a black hole and nothing is actually working.
This is the fundamental problem with designing in a bubble without any constraints.
I understand that building something new is fun and challenging, but not taking into account the constraints the software will be under when finally deployed is naive at best and reckless at the very worse. It also makes life as a systems engineer / devops engineer a living hell for several months until all of these little failures are sewn shut.
It’s like receiving a garment that looks complete, but on inspection, you find a bunch of holes all over that all need to be fixed before it can be worn.
Engineering as a Team
To me, this is situation means that software engineer is not a team player. They might be playing on the engineering team, but they’re not playing on the company team. Part of software design is designing for the full use case of the software, including not only code authoring, but systems deployment.
If systems deployment isn’t your specialty as a software engineer, then bring in a systems engineer and/or devops engineer to help guide your code during the development phase. Designing without taking the full scope of that software release into consideration means you didn’t earn your salary and you’re not a very good software engineer.
Yet, Silicon Valley is willing to pay “Principal Engineers” top dollar for these folks failing to do their jobs.
Building and Rebuilding
It’s an entirely a waste of time to get to the end of a software development cycle and claim “code complete” when that code is nowhere near complete. I’ve had so many situations where software engineers toss their code to us as complete and expect the systems engineer to magically make it all work.
It doesn’t work that way. Code works when it’s written in combination with understanding of the architecture where it will be deployed. Only then can the code be 100% complete because only then will it deploy and function without problems. Until that point is reached, it cannot be considered “code complete”.
Docker and Containers
More and more, systems engineers want to get out of the long drawn out business of integrating square code into a round production hole, eventually after much time has passed, molding the code into that round hole is possible. This usually takes months. Months that could have been avoided if the software engineer had designed the code in an environment where the production constraints exist.
That’s part of the reason for containers like Docker. When a container like Docker is used, the whole container can then be deployed without thought to square pegs in round holes. Instead, whatever flaws are in the Docker container are there for all to see because the developer put it there.
In other words, the middle folks who take code from engineering and mold it onto production gear don’t relish the thought of ironing out hundreds of glitchy problems until it seamlessly all works. Sure, it’s a job, but at some level it’s also a bit janitorial, wasteful and a unnecessary.
Planning
Part of the reason for these problems is the delineation between the engineering teams and the production operations teams. Because many organizations separate these two functional teams, it forces the above problem. Instead, these two teams should be merged into one and work together from project and code inception.
When a new project needs code to be built that will eventually be deployed, the production team should be there to move the software architecture onto the right path and be able to choose the correct path for that code all throughout its design and building phases. In fact, every company should mandate that its software engineers be a client of operations team. Meaning, they’re writing code for operations, not the customer (even though the features eventually benefit the customer).
The point here is that the code’s functionality is designed for the customer, but the deploying and running that code is entirely for the operations team. Yet, so many software engineers don’t even give a single thought to how much the operations team will be required support that code going forward.
Operational Support
For every component needed to support a specific piece of software, there needs to be a likewise knowledgeable person on the operations team to support that component. Not only do they need to understand that it exists in the environment, the need to understand its failure states, its recovery strategies, its backup strategies, its monitoring strategies and everything else in between.
This is also yet another problem that software engineers typically fail to address in their code design. Ultimately, your code isn’t just to run on your notebook for you. It must run on a set of equipment and systems that will serve perhaps millions of users. It must be written in ways that are fail safe, recoverable, redundant, scalable, monitorable, deployable and stable. These are the things that the operations team folks are concerned with and that’s what they are paid to do.
For each new code deployment, that makes the environment just that much more complex.
The Stacked Approach
This is an issue that happens over time. No software engineer wants to work on someone else’s code. Instead, it’s much easier to write something new and from scratch. It’s easy for software engineer, but it’s difficult for the operations team. As these new pieces of code get written and deployed, it drastically increases the technical debt and burden on the operations staff. Meaning, it pushes the problems off onto the operations team to continue supporting more and more and more components if none ever get rewritten or retired.
In one organization where I worked, we had such an approach to new code deployment. It made for a spider’s web mess of an environment. We had so many environments and so few operations staff to support it, the on-call staff were overwhelmed with the amount of incessant pages from so many of these components.
That’s partly because the environment was unstable, but that’s partly because it was a house of cards. You shift one card and the whole thing tumbles.
Software stacking might seem like a good strategy from an engineering perspective, but then the software engineers don’t have to first line support it. Sometimes they don’t have to support it at all. Yes, stacking makes code writing and deployment much simpler.
How many times can engineering team do this before the house of cards tumbles? Software stacking is not an ideal any software engineering team should endorse. In fact, it’s simply comes down to laziness. You’re a software engineer because writing code is hard, not because it is easy. You should always do the right thing even if it takes more time.
Burden Shifting
While this is related to software stacking, it is separate and must be discussed separately. We called this problem, “Throwing shit over the fence”. It happens a whole lot more often that one might like to realize. When designing in a bubble, it’s really easy to call “code complete” and “throw it all over the fence” as someone else’s problem.
While I understand this behavior, it has no place in any professionally run organization. Yet, I’ve seen so many engineering team managers endorse this practice. They simply want their team off of that project because “their job is done”, so they can move them onto the next project.
You can’t just throw shit over the fence and expect it all to just magically work on the production side. Worse, I’ve had software engineers actually ask my input into the use of specific software components in their software design. Then, when their project failed because that component didn’t work properly, they threw me under the bus for that choice. Nope, that not my issue. If your code doesn’t work, that’s a coding and architecture problem, not a component problem. If that open source component didn’t work in real life for other organizations, it wouldn’t be distributed around the world. If a software engineer can’t make that component work properly, that’s a coding and software design problem, not an integration or operational problem. Choosing software components should be the software engineer’s choice to use whatever is necessary to make their software system work correctly.
Operations Team
The operations team is the lifeblood of any organization. If the operations team isn’t given the tools to get their job done properly, that’s a problem with the organization as a whole. The operations team is the third hand recipient of someone else’s work. We step in and fix problems many times without any knowledge of the component or the software. We do this sometimes by deductive logic, trial and error, sometimes by documentation (if it exists) and sometimes with the help of a software engineer on the phone.
We use all available avenues at our disposal to get that software functioning. In the middle of the night the flow of information can be limited. This means longer troubleshooting times, depending on the skill level of the person triaging the situation.
Many organizations treat its operations team as a bane, as a burden, as something that shouldn’t exist, but does out of necessity. Instead of treating the operations team as second class citizens, treat this team with all of the importance that it deserves. This degrading view typically comes top down from the management team. The operations team is not a burden nor is it simply there out of necessity. It exists to keep your organization operational and functioning. It keeps customer data accessible, reliable, redundant and available. It is responsible for long term backups, storage and retrieval. It’s responsible for the security of that data and making sure spying eyes can’t get to it. It is ultimately responsible to make sure the customer experience remains at a high excellence standard.
If you recognize this problem in your organization, it’s on you to try and make change here. Operations exists because the company needs that job role. Computers don’t run themselves. They run because of dedicated personnel who make it their job and passion to make sure those computers stay online, accessible and remain 100% available.
Your company’s uptime metrics are directly impacted by the quality of your operations team staff members. These are the folks using the digital equivalent of chewing gum and shoelaces to keep the system operating. They spend many a sleepless night keeping these systems online. And, they do so without much, if any thanks. It’s all simply part of the job.
Software Engineer and Care
It’s on each and every software engineer to care about their fellow co-workers. Tossing code over the fence assuming there’s someone on the other side to catch it is insane. It’s an insanity that has run for far too long in many organizations. It’s an insanity that needs to be stopped and the trend needs to reverse.
In fact, by merging the software engineering and operations teams into one, it will stop. It will stop by merit of having the same bosses operating both teams. I’m not talking about at a VP level only. I’m talking about software engineering managers need to take on the operational burden of the components they design and build. They need to understand and handle day-to-day operations of these components. They need to wear pagers and understand just how much operational work their component is.
Only then can engineering organizations change for the positive.
As always, if you can identify with what you’ve read, I encourage you to like and leave a comment below. Please share with your friends as well.
↩︎
CanDo: An Amiga Programming Language
At one point in time, I owned a Commodore Amiga. This was back when I was in college. I first owned an Amiga 500, then later an Amiga 3000. I loved spending my time learning new things about the Amiga and I was particularly interested in programming it. While in college, I came across a programming language by the name of CanDo. Let’s explore.
HyperCard
Around the time that CanDo came to exist on the Amiga, Apple had already introduced HyperCard on the Mac. It was a ‘card’ based programming language. What that means is that each screen (i.e., card) had a set of objects such has fields for entering data, placement of visual images or animations, buttons and whatever other things you could jam onto that card. Behind each element on the card, you could attach written programming functions() and conditionals (if-then-else, do…while, etc). For example, if you had an animation on the screen, you could add a play button. If you click the play button, a function would be called to run the animation just above the button. You could even add buttons like pause, slow motion, fast forward and so on.
CanDo was an interpreted object oriented language written by a small software company named Inovatronics out of Dallas. I want to say it was released around 1989. Don’t let the fact that it was an interpreted language fool you. CanDo was fast for an interpreted language (by comparison, I’d say it was proportionally faster than the first version of Java), even on the then 68000 CPU series. The CanDo creators took the HyperCard idea, expanded it and created their own version on the Amiga. While it supported very similar ideas to HyperCard, it certainly wasn’t identical. In fact, it was a whole lot more powerful than HyperCard ever dreamed of being. HyperCard was a small infant next to this product. My programmer friend and I would yet come to find exactly how powerful the CanDo language could be.
CanDo
Amiga owners only saw what INOVATronics wanted them to see in this product. A simplistic and easy to use user interface consisting of a ‘deck’ (i.e, deck of cards) concept where you could add buttons or fields or images or sounds or animation to one of the cards in that deck. They were trying to keep this product as easy to use as possible. It was, for all intents and purposes, a drag-and-drop programming language, but closely resembled HyperCard in functionality, not language syntax. For the most part, you didn’t need to write a stitch of code to make most things work. It was all just there. You pull a button over and a bunch of pre-programmed functions could be placed onto the button and attached to other objects already on the screen. As a language, it was about as simple as you could make it. I commend the INOVATronics guys on the user-friendly aspect of this system. This was, hands down, one of the easiest visual programming languages to learn on the Amiga. They nailed that part of this software on the head.
However, if you wanted to write complex code, you most certainly could do this as well. The underlying language was completely full featured and easy to write. The syntax checker was amazing and would help you fix just about any problem in your code. The language had a full set of typical conditional constructs including for loops, if…then…else, do…while, while…until and even do…while…until (very unusual to see this one). It was a fully stocked mostly free form programming language, not unlike C, but easier. If you’re interested in reading the manual for CanDo, it is now located at this end of this section below.
As an object oriented language, internal functions were literally attached to objects (usually screen elements). For example, a button. The button would then have a string of code or functions that drove its internal functionality. You could even dip into that element’s functions to get data out (from the outside). Like most OO languages, the object itself is opaque. You can’t see its functions names or use them directly, only that object that owns that code can. However, you could ask the object to use one of its function and return data back to you. Of course, you had to know that function existed. In fact, this would be the first time I would be introduced to the concept of object oriented programming in this way. There was no such thing as free floating code in this language. All code had to exist as an attachment to some kind of object. For example, it was directly attached to the deck itself, to one of the cards in the deck, to an element on one of the cards or to an action associated with that object (mouse, joystick button or movement, etc).
CanDo also supported RPC calls. This was incredibly important for communication between two separately running CanDo deck windows. If you had one deck window with player controls and another window that had a video you wanted to play, you could send a message from one window to another to perform an action in that window, like play, pause, etc. There were many reasons to need many windows open each communicating with each other.
The INOVATronics guys really took programming the Amiga to a whole new level… way beyond anything in HyperCard. It was so powerful, in fact, there was virtually nothing stock on the Amiga it couldn’t control. Unfortunately, it did have one downside. It didn’t have the ability to import system shared libraries on AmigaDOS. If you installed a new database engine on your Amiga with its own shared function library, there was no way to access those functions in CanDo by linking it in. This was probably CanDo’s single biggest flaw. It was not OS extensible. However, for what CanDo was designed to do and the amount of control that was built into it by default, it was an amazing product.
I’d also like to mention that TCP/IP was just coming into existence with modems on the Amiga. I don’t recall how much CanDo supported network sockets or network programming. It did support com port communication, but I can’t recall if it supported TCP/IP programming. I have no doubt that had INOVATronics stayed in business and CanDo progressed beyond its few short years in existence, TCP/IP support would have been added.
CanDo also supported Amiga Rexx (ARexx) to add additional functionality to CanDO which would offer additional features that CanDo didn’t support directly. Though, ARexx worked, it wasn’t as convenient as having a feature supported directly by CanDo.
Here are the CanDo manuals if you’re interested in checking out more about it:
- Original CanDo Manual
- Searchable CanDo Manual
- Original CanDo ProPak Manual
- Searchable CanDo ProPak Manual
Here’s a snippet from the CanDo main manual:
CanDo is a revolutionary, Amiga specific, interactive software authoring system. Its unique purpose is to let you create real Amiga software without any programming experience. CanDo is extremely friendly to you and your Amiga. Its elegant design lets you take advantage of the Amiga’s sophisticated operating system without any technical knowledge. CanDo makes it easy to use the things that other programs generate – pictures, sounds, animations, and text files. In a minimal amount of time, you can make programs that are specially suited to your needs. Equipped with CanDo, a paint program, a sound digitizer, and a little bit of imagination, you can produce standalone applications that rival commercial quality software. These applications may be given to friends or sold for profit without the need for licenses or fees.
As you can see from this snippet, INOVATronics thought of it as an ‘Authoring System’ not as a language. CanDo itself might have been, but the underlying language was most definitely a programming language.
CanDo Player
The way CanDo worked its creation process was that you would create your CanDo deck and program it in the deck creator. Once your deck was completed, you only needed the CanDo player to run your deck. The player ran with much less memory than the entire CanDo editor system. The player was also freely redistributable. However, you could run your decks from the CanDo editor if you preferred. The CanDo Player could also be appended to the deck to become a pseudo-executable that allowed you to distribute your executable software to other people. Also, anything you created in CanDo was fully redistributable without any strings to CanDo. You couldn’t distribute CanDo, but you could distribute anything you created in it.
The save files for decks were simple byte compiled packages. Instead of having to store humanly readable words and phrases, each language keyword had a corresponding byte code. This made storing decks much smaller than keeping all of the human readable code there. It also made it a lot more tricky to read this code if you opened the deck up in a text editor. It wasn’t totally secure, but it was better than having your code out there for all to see when you distributed a deck. You would actually have to own CanDo to decompile the code back into a humanly readable format… which was entirely possible.
The CanDo language was way too young to support more advanced code security features, like password encryption before executing the deck, even though PGP was a thing at that time. INOVATronics had more to worry about than securing your created deck’s code from prying eyes, though they did improve this as they upgraded versions. I also think the INOVATronics team was just a little naïve about how easily it would be to crack open CanDo, let alone user decks.
TurboEditor — The product that never was
A programmer friend who was working towards his CompSci masters owned an Amiga, and also owned CanDo. In fact, he introduced me to it. He had been poking around with CanDo and found that it supported three very interesting functions. It had the ability to decompile its own code into humanly readable format to allow modification, syntactically check the changes and recompile it, all while still running. Yes, you read that right. It supported on-the-fly code modification. Remember this, it’s important.
Enter TurboEditor. Because of this one simple little thing (not so little actually) that my friend found, we were able to decompile the entire CanDo program and figure out how it worked. Remember that important thing? Yes, that’s right, CanDo is actually written in itself and it could actually modify pieces that are currently executing. Let me clarify this just a little. One card could modify another card, then pull that card into focus. The actual card wasn’t currently executing, but the deck was. In fact, we came to find that CanDo was merely a facade. We also found that there was a very powerful object oriented, fully reentrant, self-modifying, programming language under that facade of a UI. In fact, this is how CanDo’s UI worked. Internally, it could take an element, decompile it, modify it and then recompile it right away and make it go live, immediately adding the updated functionality to a button or slider.
While CanDo could modify itself, it never did this. Instead, it utilized a parent-child relationship. It always created a child sandbox for user-created decks. This sandbox area is where the user built new CanDo Decks. Using this sandbox approach, this is how CanDo built and displayed a preview of your deck’s window. The sandbox would then be saved to a deck file and then executed as necessary. In fact, it would be one of these sandbox areas that we would use to build TurboEditor, in TurboEditor.
Anyway, together, we took this find one step further and created an alternative CanDo editor that we called TurboEditor, so named because you could get into it and edit your buttons and functions much, much faster than CanDo’s somewhat sluggish and clunky UI. In fact, we took a demo of our product to INOVATronics’s Dallas office and pitched the idea of this new editor to them. Let’s just say, that team was lukewarm and not very receptive to the idea initially. While they were somewhat impressed with our tenacity in unraveling CanDo to the core, they were also a bit dismayed and a little perturbed by it. Though, they warmed to the idea a little. Still, we pressed on hoping we could talk them into the idea of releasing TurboEditor as an alternative script editor… as an adjunct to CanDo.
Underlying Language
After meeting with and having several discussions with the INOVATronics team, we found that the underlying language actually has a different name. The underlying language name was AV1. Even then, everyone called it by ‘CanDo’ over that name. Suffice it to say that I was deeply disappointed that INOVATronics never released the underlying fully opaque, object oriented, reentrant, self-modifying on-the-fly AV1 language or its spec. If they had, it would have easily become the go-to deep programming language of choice for the Amiga. Most people at the time had been using C if they wanted to dive deep. However, INOVATronics had a product poised to take over for Amiga’s C in nearly every way (except for the shared library thing which could have been resolved).
I asked the product manager while at the INOVATronics headquarters about releasing the underlying language and he specifically said they had no plans to release it in that way. I always felt that was shortsighted. In hindsight for them, it probably was. If they had released it, it could have easily become CanDo Pro and they could sold it for twice or more the price. They just didn’t want to get into that business for some reason.
I also spoke with several other folks while I was at INOVATronics. One of them was the programmer who actually built much of CanDo (or, I should say, the underlying language). He told me that the key pieces of CanDo he built in assembly (the compiler portions) and the rest was built with just enough C to bootstrap the language into existence. The C was also needed to link in the necessary Amiga shared library functions to control audio, animation, graphics and so on. This new language was very clever and very useful for at least building CanDo itself.
It has been confirmed by Jim O’Flaherty, Jr. (formerly Technical Support for INOVATronics) via a comment that the underlying language name was, in fact, AV1. This AV portion meaning audio visual. This new, at the time, underlying object oriented Amiga programming language was an entirely newly built language and was designed specifically to control the Amiga computer.
Demise of INOVAtronics
After we got what seemed like a promising response from the INOVATronics team, we left their offices. We weren’t sure it would work out, but we kept hoping we would be able to bring TurboEditor to the market through INOVATronics.
Unfortunately, our hopes dwindled. As weeks turned into months waiting for the go ahead for TurboEditor, we decided it wasn’t going to happen. We did call them periodically to get updates, but nothing came of that. We eventually gave up, but not because we didn’t want to release TurboEditor, but because INOVATronics was apparently having troubles staying afloat. Apparently, their CanDo flagship product at the time wasn’t able to keep the lights on for the company. In fact, they were probably floundering when we visited them. I will also say their offices were a little bit of a dive. They weren’t in the best area of Dallas and in an older office building. The office was clean enough, but the office space just seemed a little bit well worn.
Within a year of meeting the INOVATronics guys, the entire company folded. No more CanDo. It was also a huge missed opportunity for me in more ways than one. I could have gone down to INOVATronics, at the time, and bought the rights to the software on a fire sale and resurrected it as TurboEditor (or the underlying language). Hindsight is 20/20.
We probably could have gone ahead and released TurboEditor after the demise of INOVATronics, but we had no way to support the CanDo product without having their code. We would have had to buy the rights to the software code for that.
So, there you have it. My quick history of CanDo on the Amiga.
If you were one of the programmers who happened to work on the CanDo project at INOVATronics, please leave a comment below with any information you may have. I’d like to expand this article with any information you’re willing to provide about the history of CanDo, this fascinating and lesser known Amiga programming language.
Random Thoughts - Randocity! 
leave a comment