Random Thoughts – Randocity!

Rant Time: Bloomberg and Hacked Servers

Posted in best practices, botch, data security, reporting by commorancy on October 5, 2018

Bloomberg has just released a story claiming SuperMicro motherboards destined for large corporations may have been hacked with a tiny “spy” chip. Let’s explore.

Bloomberg’s Claims

Supposedly the reporters for Bloomberg have been working on this story for months. Here’s a situation where Bloomberg’s reporters have just enough information in hand to be dangerous. Let’s understand how this tiny chip might or might not be able to do what Bloomberg’s alarmist view claims. Thanks Bloomberg for killing the stock market today with your alarmist reporting.

Data Compromise

If all of these alleged servers have been compromised by a Chinese hardware hack, someone would have noticed data streaming out of their server to Chinese IP addresses, or at least some consistent address. Security scans of network equipment require looking through inbound and outbound data logs for data patterns. If these motherboards had been compromised, the only way for the Chinese to have gotten that data back is through the network. This means data passing through network cards, switches and routers before ever hitting the Internet.

Even if such a tiny chip were embedded in the system, many internal only servers have no direct Internet access. This means that if these servers are used solely for internal purposes, they couldn’t have transmitted their data back to China. The firewalls would prevent that.

For servers that may have had direct access to the Internet, these servers could have sent payloads, but eventually these patterns would have been detected by systems administrators, network administrators and security administrators in performing standard security checks. It might take a while to find the hacks, but they would be found just strictly because of odd outbound data being sent to locations that don’t make sense.

Bloomberg’s Fantasy

While it is definitely not out of the realm of possibility that China could tamper with and deliver compromised PCB goods to the US, it’s doubtful that this took place in the numbers that Bloomberg has reported.

Worse, Bloomberg makes the claim that this so-called hacked hardware was earmarked for specific large companies. I don’t even see how that’s possible. How would a Chinese factory know the end destination of any specific SuperMicro motherboard? As far as I know, most cloud providers like AWS and Google buy fully assembled equipment, not loose motherboards. How could SuperMicro board builders possibly know it’s going to end up in a server at AWS or Google or Apple? If SuperMicro’s motherboard products have been hacked, they would be hacked randomly and everywhere, not just at AWS or Google or whatever fantasy Bloomberg dreams up.

The Dangers of Outsourcing

As China’s technical design skills grow, so will the plausibility of receiving hacked goods from that region. Everyone takes a risk ordering any electronics from China. China has no scruples about any other country than China. China protects China, but couldn’t give a crap about any other country outside of China. This is a dangerous situation for China. Building electronics for the world requires a level of trust that must exist or China won’t get the business.

Assuming this alleged “spy chip” is genuinely found on SuperMicro motherboards, then that throws a huge damper on buying motherboards and other PCBs made in China. China’s trust level is gone. If Chinese companies are truly willing to compromise equipment at that level, they’re willing to compromise any hardware built in China including cell phones, laptops and tablets.

This means that any company considering manufacturing their main logic boards in China might want to think twice. The consequences here are as serious as it can get for China. China has seen a huge resurgence of inbound money flow into China. If Bloomberg’s notion is true, this situation severely undermines China’s ability to continue at this prosperity level.

What this means ultimately is that these tiny chips could easily be attached to the main board of an iPhone or Android phone or any mobile device. These mobile devices can easily phone home with data from mobile devices. While the SuperMicro motherboard problem might or might not be real, adding such a circuit to a phone is much more undetectable and likely to provide a wealth more data than placing it onto servers behind corporate firewalls.

Rebuttal to Bloomberg

Statements like from this next reporter is why no one should take these media outlets seriously. Let’s listen. Bloomberg’s Jordan Robertson states, “Hardware hacking is the most effective type of hacking an organization can engineer… There are no security systems that can detect that kind of manipulation.” Wrong. There are several security systems that look for unusual data patterns including most intrusion detection systems. Let’s step back for a moment.

If the point in the hardware hacking is to corrupt data, then yes, it would be hard to detect that. You’d just assume the hardware is defective and replace it. However, if the point to the hardware hack is to phone data home, then that is easily detected via various security systems and is easily blocked by firewalls.

The assumption that Jordon is making is that we’re still in the 90s with minimal security. We are no longer in the 90s. Most large organizations today have very tight security around servers. Depending on the role of the server, it might or might not have direct trusted access to secured data. That server might have to ask an internal trusted server to get the data it needs.

For detection purposes, if the server is to be used as a web server, then the majority of the data should have a 1:1 relationship. Basically, one request inbound, some amount of data sent outbound from that request. Data originating from the server without an inbound request would be suspect and could be detected. For legitimate requests, you can see these 1:1 relationships in the logs and when watching the server traffic on a intrusion detection system. For one-sided transactions sending data outbound from the server, the IDS would easily see it and could block it. If you don’t think that most large organizations don’t have an IDS even simply in watch mode, you are mistaken.

If packets of data originate from the server without any prompting, that would eventually be noticed by a dedicated security team performing regular log monitoring and regular server security scans. The security team might not be able to pinpoint the reason (i.e. a hardware hack) for unprompted outbound data, but they will be able to see it.

I have no idea how smart such tiny chip could actually be. Such a tiny chip likely would not have enough memory to store any gathered payload data. Instead, it would have to store that payload either on the operating systems disks or in RAM. If the server was cut off from the Internet as most internal servers are, that disk or RAM would eventually fill its data stores up without transfer of that data to wherever it needed to go. Again, systems administrators would notice the spike in usage of /tmp or RAM due to the chip’s inability to send its payload.

If the hacking chip simply gives remote control access to the server without delivering data at all, then that would also be detected by an IDS system. Anyone attempting to access a port that is not open will be blocked. If the chip makes an outbound connection to a server in China and leaves it open would eventually be detected. Again, a dedicated security team would see the unusual data traffic from/to the server and investigate.

If the hacking chip wants to run code, it would need to compiled it first. That implies having a compiler in that tiny chip. Doubtful. If the system builder installs a compiler, the spy chip might be able to leverage it, assuming it has any level of knowledge about the current operating system installed. That means that chip would have to know about many different versions of Linux, BSD, MacOS X, Windows and so on, then have code ready to deploy for each of these systems. Unlikely.

Standards and Protocols

Bloomberg seems to think there’s some mystery box here that allows China to have access to these servers without bounds. The point to having multi-layer security is to prevent such access. Even if the motherboards were compromised, most of these servers would end up behind multiple firewalls in combination with continuous monitoring for security. Even more than this, many companies segregate servers by type. Servers performing services that need a high degree of security have very limited ability to do anything but their one task. Even getting into these servers can be challenge even for administrators.

For web servers in a DMZ which are open to the world, capturing data here might be easier. However, even if the hacker at SuperMicro did know which company placed an order for motherboards, they wouldn’t know how those servers would ultimately be deployed and used. This means that these chips could be placed into server roles behind enough security to render their ability to spy as worthless.

It’s clear, these reporters are journalists through and through. They really have no skill at being a systems administrator, network engineer or security administrator. Perhaps it’s now time to hire technical consultants at Bloomberg who can help you guide your articles when they involve technical matters? It’s clear, there was no guidance by any technical person who could steer Jordan away from some of the ludicrous statements he’s made.

Bloomberg, hire a technical consultant the next time you chase one of these “security” stories or give it up. At this point, I’m considering Bloomberg to be nothing more a troll looking for views.


If you enjoy reading Randocity, please like, subscribe and leave a comment below.

↩︎

 

Advertisements

What’s wrong with Vista / Windows?

Posted in microsoft, tanking, windows by commorancy on July 6, 2009
This post comes from a variety of issues that I’ve had with Vista (specifically Vista 64 Home Premium).  And, chances are, these problems will not be resolved in Windows 7.  Yet, here they are in all their glory.
Memory Leaks
Vista has huge and horrible memory leaks.  After using Vista for a period of time (a week or two without a reboot) and using a variety of memory intensive 3D applications (Daz Studio, Carrara, The Gimp and Poser.. just to name a few), the system’s memory usage goes from 1.69GB to nearly 3GB in usage.  To answer the burning question… yes, I have killed all apps completely and I am comparing empty system to empty system.  Worse, there is no way to recover this memory short of rebooting.  If you had ever wondered why you need to reboot Windows so often, this is the exact reason.  For this reason alone, this is why Windows is not considered ‘stable’ by any stretch and why UNIX outperforms Windows for this reason alone.
Startup and Shutdown
Microsoft plays games with both of these procedures.
On Startup, Microsoft’s engineers have tricked you into thinking the system is functional even when it isn’t.  Basically, once the desktop appears, you think you can begin working.  In reality, even once the desktop appears, you still cannot work.  The system is still in the process of starting up the Windowing interface on top of about 100 background services (on many of which the windowing interface relies).  This trick makes Windows appear snappier to start up than it really is.  In fact, I would prefer it to just ready the system fully, then present the Windowing interface when everything is 100% complete.  I don’t want these tricks.  When I see the windowing interface, I want to know I can begin using it immediately… not before.
On Shutdown, we have other issues.  With Vista, Microsoft Engineers have done something to this process to make it, at times, ridiculously slow.  I have seen 8-15 minute ‘Shutting Down’ screens where the hard drive grinds the entire time.  I’m sorry, but shutdown time is not housekeeping time.  That needs to be done when the system is running.  It should not be done during shutdown procedures.  A shutdown should take no more than about 1-2 minutes to complete flushing buffers to disk and killing all processes.  If it can’t be done in 1-2 minutes, shut the system down anyway as there is nothing that can be done to finish those tasks anyway.
Windows Updates
Microsoft was supposed to eliminate the need to shutdown/reboot for most Windows updates.  For some updates, this is true.  For the majority of Windows updates, this is still not true.  In fact, Microsoft has, once again, made this process multistep and tediously slow in the process.  Don’t get me wrong, I’m grateful that they are now at least verbose in, sort of, what’s going on.. but that doesn’t negate the fact that it’s horribly slow.  The steps now are as follows:
Windows installation process (downloading and installation through the Windows dialog box).  You think it’s over when you..
Restart the system and it goes through finishing Step 2 of this process during shutdown… and then you think it’s over again when
The system starts back up and goes through Step 3 of the update process.
Ok, I’m at a loss.  With Windows XP, we had two steps.  Those first during Windows updater and the second when the system starts back up.   Now with Vista, we have to introduce another step?
Windows Explorer
For whatever reason, Windows Explorer in Vista is horribly broken.  In Window XP, you used to be able to configure your Windows how you liked then lock it in with Tools->Folder Options  and then View->Apply to Folders.  This would lock in exactly how every window should appear (list or icon format, size of icons, etc).  With Windows Vista, this is completely and uterly broken.  This functionality just no longer works.  I’ve tried many many times to lock in a format and Windows just randomly changes the folders back to whatever it feels like doing.
For example, I like my windows to look like this:
Unfortunately, Windows has its down agenda.  If I open a file requester (the standard Vista requester… the one that looks like the above) and I change the view to ANY other folder than this one, it randomly changes folders on the system.  So, I might open the above folder and it will later look like any of these:

This post comes from a variety of issues that I’ve had with Vista (specifically Vista 64 Home Premium).  And, chances are, these problems will not be resolved in Windows 7.  Yet, here they are in all their glory.

Memory Leaks

Vista has huge and horrible memory leaks.  After using Vista for a period of time (a week or two without a reboot) and using a variety of memory intensive 3D applications (Daz Studio, Carrara, The Gimp and Poser.. just to name a few), the system’s memory usage goes from 1.69GB to nearly 3GB in usage.  To answer the burning question… yes, I have killed all apps completely and I am comparing empty system to empty system.  Worse, there is no way to recover this memory short of rebooting.  If you had ever wondered why you need to reboot Windows so often, this is the exact reason.  For this reason alone, this is why Windows is not considered ‘stable’ by any stretch and why UNIX outperforms Windows for this reason alone.

Startup and Shutdown

Microsoft plays games with both of these procedures.

On Startup, Microsoft’s engineers have tricked you into thinking the system is functional even when it isn’t.  Basically, once the desktop appears, you think you can begin working.  In reality, even once the desktop appears, you still cannot work.  The system is still in the process of starting up the Windowing interface on top of about 100 background services (on many of which the windowing interface relies).  This trick makes Windows appear snappier to start up than it really is.  In fact, I would prefer it to just ready the system fully, then present the Windowing interface when everything is 100% complete.  I don’t want these tricks.  When I see the windowing interface, I want to know I can begin using it immediately… not before.

On Shutdown, we have other issues.  With Vista, Microsoft Engineers have done something to this process to make it, at times, ridiculously slow.  I have seen 8-15 minute ‘Shutting Down’ screens where the hard drive grinds the entire time.  I’m sorry, but shutdown time is not housekeeping time.  That needs to be done when the system is running.  It should not be done during shutdown procedures.  A shutdown should take no more than about 1-2 minutes to complete flushing buffers to disk and killing all processes.  If it can’t be done in 1-2 minutes, shut the system down anyway as there is nothing that can be done to finish those tasks anyway.

Windows Updates

Microsoft was supposed to eliminate the need to shutdown/reboot for most Windows updates.  For some updates, this is true.  For the majority of Windows updates, this is still not true.  In fact, Microsoft has, once again, made this process multistep and tediously slow in the process.  Don’t get me wrong, I’m grateful that they are now at least verbose in, sort of, what’s going on.. but that doesn’t negate the fact that it’s horribly slow.  The steps now are as follows:

  1. Windows installation process (downloading and installation through the Windows dialog box).  You think it’s over when you..
  2. Restart the system and it goes through finishing Step 2 of this process during shutdown… and then you think it’s over again when
  3. The system starts back up and goes through Step 3 of the update process.

Ok, I’m at a loss.  With Windows XP, we had two steps.  Those first during Windows updater and the second when the system starts back up.   Now with Vista, we have to introduce another step?

Windows Explorer

For whatever reason, Windows Explorer in Vista is horribly broken.  In Window XP, you used to be able to configure your Windows how you liked then lock it in with Tools->Folder Options  and then View->Apply to Folders.  This would lock in exactly how every window should appear (list or icon format, size of icons, etc).  With Windows Vista, this is completely and utterly broken.  Basically, this functionality simply no longer works.  I’ve tried many many times to lock in a format and Windows just randomly changes the folders back to whatever it feels like doing.

For example, I like my windows to look like this:

Favorite Format

Favorite Format

Unfortunately, Windows has its own agenda.  If I open a file requester (the standard Vista requester… the one that looks like the above) and I change the view to ANY other style than the one above, this change randomly changes other folder views on the system permanently.  So, I might open the above folder and it will later look like any of these:

Format Changed 1

Format Changed 1

Format Changed 2

Format Changed 2

or even

Format Changed 3

Format Changed 3

All of which is highly frustrating.  So, I’ll visit this folder later and see the entire headers have changed, or it’s changed to icon format or some other random format.  Worse, though, is that I’ve specifically changed to the folder to be my favorite format with Tools->Options.  In fact, I’ve gone through this permanent change at least 3-4 times after random changes  have happened and inevitably it changes to some other format later.  Again, highly frustrating.

Access Denied / Enhanced Security

For whatever reason, Microsoft has made shortcuts to certain folders.  Like for example, in your profile directory they have renamed ‘My Documents’ to simply ‘Documents’.  Yet, for whatever reason, Microsoft has created shortcuts that don’t work.  For example, if I click on ‘My Documents’ shortcut, I see ‘Access Denied’.  I don’t get why they would create a shortcut and then prevent it from working.

The only thing the enhanced security has done for Windows users is make it more of a problem to work.  Security goes both ways.  It helps protect you from malicious intent, but it can also get in the way of usability.  Security that ultimately gets in the way, like UAC, has failed to provide adequate security.  In fact, it has gone too far.  UAC is a complete and utter failure.  Combining this with making nearly every security issue tied to the SYSTEM user (with practically zero privileges), makes for stupid and exasperating usability.

Filesystem

To date, Windows still relies heavily and ONLY on NTFS.  Linux has about 5-6 different filesystems to choose from (Reiser, VxFS, XFS, Ext2, Ext3, JFS, BSD and several others).  This allows systems administrators to build an operating system that functions for the application need.  For example, some filesystems perform better for database use than others.   On Windows, you’re stuck with NTFS.  Not only is NTFS non-standard and proprietary (written by Veritas), it also doesn’t perform as well as it should under all conditions.  For database use, this filesystem is only barely acceptable.  It has hidden limits that Microsoft doesn’t publish that will ultimately bite you.  Microsoft wants this to become a pre-eminent datacenter system, but that’s a laugh.  You can’t trust NTFS enough for that.  There are way too many hidden problems in NTFS.  For example, if you hit a random limit, it can easily and swiftly corrupt NTFS’ MFT table (directory table).  Once the MFT table is corrupt, there’s no easy way to repair it other than CHKDSK. Note that CHKDSK is the ONLY tool that can truly and completely fix NTFS issues.  And, even CHKDSK doesn’t always work.  Yes, there are third party tools from Veritas and other companies, but these aren’t necessarily any better than CHKDSK.  Basically, if CHKDSK can’t fix your volume, you have to format and restore.

Note, however, that this isn’t a general Vista issue.  This problem has persisted back to the introduction of NTFS in Windows NT.  But, Microsoft has made no strides to allow or offer better more complete filesystems with better repair tools.  For example, Reiser and EXT3 both offer more complete repair tools than NTFS ever has.

Registry

The registry has got to be one of the most extensive hacks ever placed into any operating system.  This kludge of a database system is so completely botched from a design perspective, that there’s really nothing to say.  Basically, this system needs to be tossed and redesigned.  In fact, Microsoft has a real database system in MSSQL.  There is no reason why the registry is not based on MSSQL rather than that stupid hack of a thing call a hive/SAM.  Whomever decided on this design, well.. let’s just hope they no longer work at Microsoft.

Failure

For the above reasons (and others), Microsoft has completely failed with Windows Vista.  This failure was already in the making, though, when Longhorn was announced ages ago.  In fact, Microsoft had planned even more draconian measures to enable heavy DRM on Windows.  Thankfully, that was removed from Vista.  But, what remains makes Vista so encumbered and exasperating to use, it’s no wonder users are frustrated using Vista.  Combining that with its incredibly large footprint (1.6GB of memory just to boot the OS), and you have a complete loser of an OS.

Windows 7 is a glimmer of hope, but it is still heavily tied to Vista.  If UAC and these stupid SYSTEM user security measures remain, then nothing will really change.  Microsoft needs to take Windows back to the drawing board and decide what is necessary and what isn’t.  Preventing the user from actually using the operating system is not and should not be a core value, let alone part of security.  Yet, here we are.

Microsoft, you need to take a look at the bigger picture.  This is your final chance to get Windows right.  There are plenty of other unencumbered operating systems out there that do not get in the way of desktop computing.  These operating systems are definitely a threat to Microsoft’s continued viability… especially with blundering mistakes like Vista.  Windows will never win any awards for Best Operating System with issues such as these.  Consider Microsoft’s stupid filesystem layout that allows operating system and application files to be thrown all over the hard drive and you’ll begin to understand why Windows continues to fail.

The single reason why Microsoft continues to exist is because users feel compelled to buy this antiquated dog of an operating system strictly due to application support.  If developers would finally and completely jump ship to other more thoughtfully designed operating systems, then Windows would finally wither and die… eventually, this will happen.

%d bloggers like this: