Control Systems. Are they possibly the biggest security vulnerability?
By now almost everyone has seen the news media on the cyberattacks stemming from Russia. So far nothing dastardly has occurred at the same scale of the Colonial Pipeline fiasco in 2021. That event involved the deployment of ransomware across the corporate network. As a precaution, the company shutdown its network that monitored the pipeline which, in effect, shut down the pipeline itself. The end result was a frightening interruption in the gasoline supply in the southeast part of the U.S. So when it was announced that Russia was on the prowl, it obviously raised concerns that vital infrastructure could be at risk.
As of this day, not much has been released regarding the how and why of the Colonial Pipeline event. It is one of the fatal flaws of the IT security industry. People tend to hold things close to the chest, what we call “security through obscurity.” Reveal too much, people may learn how your network is designed, how your security is structured and how competent or incompetent your personnel are in regards to IT security. But this much I have picked up. There is no divulged evidence that the “control systems” of the Colonial Pipeline were effected. But there is some indication of other serious faults in their IT security. In the end, Colonial had to pay a ransom to regain control of their systems.
So I can’t help be a bit curious when I see flowing from CISA an unending listing of vulnerable control systems. CISA stands for the Cybersecurity and Infrastructure Security Agency. It is part of the Department of Homeland Security. There have been seven advisories posted warning of security vulnerabilities on control systems through March 2022. Every aspect of IT has its own risk factors. Windows and Apple, for example, are operating systems that reside on almost every computer on the planet. Android and Apple are operating systems on smartphones. Certain applications have vulnerabilities and require routine updates, such as Word or the Firefox browser. But control systems – they are in a class by themselves.
What is a Control System?
Control systems are specialized computer technologies designed to perform specific functions. In my experience, the most common control systems I have encountered are air conditioners and power conditioners. These two items are a part of every computer center and are vital to the performance and reliability of computer services. Remote monitoring is an important component of that. If an air conditioning unit malfunctions, it can automatically send notification to operators of the computer center. Temperature alerts can be transmitted as well. And when power services become unstable, the data directed to the computer center operators enables a pro-active timely adjustment and better communication to affected customers and staff.
Not all control systems are sophisticated creatures on the Internet.1 I have programmable thermostats in my home and they are not interconnected. Alarm systems are another common element of a house, not all of which are connected to a network. But once you get “smart” and connect household devices to the Internet, you expose those components to cyberattacks. There have been several stories posted regarding Ring. I see the Ring doorbell or the Ring label in people’s windows beside their front door. While it is super convenient to have suspect activity relayed to your telephone, it can literally open your entire house to malicious actors. The same applies to your garage door opener. While downloading an app to your phone is amusing, it can also expose your home to cyberattacks.
Pipelines, power lines and nuclear reactors are no different. The intelligence provided by control systems enable corporations the ability to deliver a more reliable service at less cost with better safety. Network engineers go to considerable lengths to secure the infrastructure. But they are human and it has been demonstrated time and time again that if there is a vulnerability, it will be found.
The Problem of Obscurity
Anyone who installs a device with a control system, they have one objective – to get it to work. It must do a specific task. When air conditioners were installed in the regional computer center, it had one primary purpose – to keep our computer center cool so that our equipment would perform at optimal levels. Of the 150 people in the office who knew anything about those air conditioners, it was usually two of us at any time. The rest of the office never thought about it.
In a similar manner, the power conditioners that were installed were designed for two reasons – to provide stabilized power to our computer systems and to provide at least fifteen minutes of backup power. At one time in my career, power conditioners were designed to communicate with our servers to coordinate power reductions and then system shutdowns in the event of a major power disruption. Critical systems were placed on a 24/7 circuit backed up by diesel generators, designed for automated restart upon power restoration. Pretty fancy stuff. When it worked, nobody thought about it! And that was the point. It was dependable, virtually eliminated extended downtime, removed entirely faults due to brownouts, and maintained at near 100% levels critical communication systems. Again, only two people in the building knew about all that stuff.
And that is the problem with control systems. Business managers are usually clueless about these sort of devices. In fact, I have worked with some IT Security managers who appear to be clueless about control systems. It was interesting to see how security managers were surprised when “unknown” devices started showing up in computer scans, only to learn that they were power conditioners, air conditioners, telephone relays, radio dispatch centers, security cameras, and environmental controls. All these devices had a purpose. But security was the last thing on anyone’s mind when they installed them. Even more fascinating was that the people who installed them and managed them were not IT personnel, but HVAC specialists, electricians and radio technicians.
The Problem of Obsolescence
In my article, Ransomware and the Mechanical Pencil, I point out one of the major reasons why some obsolete IT systems are out there. Not everything connected to computers receives a monthly security update. The primary reason is that control system manufacturers focus on the device, not the software. I will take as an example the air conditioning system I integrated into our computer center in about 2014. It was a new system and it came with a network card that plugged into our network. Nice. Gave it an IP address. Tested the connection OK and then configured the air conditioner. After an evaluation period of about a month I invoked secured HTTP. Suddenly I lost connection. Instead I got this nasty error “SSL_ERROR_UNSUPPORTED_VERSION.” After decades of working on the Internet, this was my first encounter with this problem. The best I could find on the subject was buried in the archives of Mozilla’s support documentation. It provided the following definition:
“Peer using unsupported version of security protocol.” On a client socket, this means the remote server has attempted to negotiate the use of a version of SSL that is not supported by the NSS library, probably an invalid version number. On a server socket, this means the remote client has requested the use of a version of SSL older than version 2.
Upon further research I discovered that all approved browsers did not recognize older versions of SSL (Secured Socket Layer). It was considered highly unsecured. And, get this, the document was posted in 2008!
I said to myself, “You have got to be kidding. This is a brand new air conditioner!”
After consulting with the A/C unit’s customer support, it was determined the card needed a firmware update. The software that ran on the network card was embedded in a chip. In the meantime, I had some fun with this situation. I located on the Internet a solution to this problem. I installed a shell program that put me back in time, creating a virtual session of Windows 98! I was using Windows 7 at the time, so I was in effect going back almost 20 years. Within this shell, I installed a version of the Firefox browser that was about ten years old. With this combination, I created a connection that predated the security block in the browser, connecting to the AC unit. Through that connection, I turned off HTTP security, regaining access to the air conditioner control system.
Seeing the old Windows 98 graphics and the ancient Firefox program was a fun trip down memory lane. To load the firmware was a bit more entertaining. I had to connect to the AC unit through a serial port. This is 1990’s technology! Our laptops no longer had serial ports. We were not even certain if the bevy of cables in the computer center were serial-compliant. So I asked a friend from the local radio station to assist. Alas, there is still tons of stuff running on old standards in the radio world. Together we tested the serial cables, tested the USB-to-serial dongles, and then played with the Telnet2 command interface. I uploaded the firmware package to the AC unit and updated the firmware on the network adapter. Viola – problem solved.
This dramatically demonstrates the nature of control systems. The primary objective of control system designers is to provide a trustworthy and dependable operation of a service. In my case, it was the air conditioner. Their concern is to assure that the room is cooled at the desired level. Traditionally technicians have configured air conditioners through a primitive LCD panel on the unit. Security is not exactly their forte. The software on the network card which provided the intelligence to manage the unit remotely was a secondary design, almost an after-thought. So it is no surprise that they sold a product that was already obsolete in the world of cybersecurity. The subsequent updates were often months, if not years, behind in regards to security. This, in essence, explains why CISA has posted so many alerts on control systems.
What IT Security Overlooks
On the flip side are the IT Security nerds. The lazy response is to force control systems off-line. That did not go far simply because control systems are vital to operations. When I worked for the USFS, control systems were to be found on everything from power stabilization to security cameras, to scientific instruments on Mt. St. Helens and the webcam inside the beaver lodge. Our Alaskan radio towers were often isolated atop mountains bedecked with remote monitoring technology. As regards taking a control system off-line, the answer is often “No” and “That’s Impossible.”
The second thing IT Security overlooks is the reality of today’s IT support world. When I worked at the University of Missouri School of Medicine, I supported a network of about 100 people. When I moved up to Alaska, the objective was to increase that to one support specialist per 450 users. In the USFS, it was in the neighborhood of one in 4500 by the time I retired! Related to that dynamic is the fact that some of us moved about a lot. I wasn’t even in town four months of the year. With teleworking, I was only in the office two days a week. To be physically present to make daily inspections of equipment was impossible. So operating a computer center “in the blind” made no sense. Remote monitoring systems were vital. It explained in many respects why the regional computer center could operate at 99% performance. With Juneau’s famous winters and Taku winds, power disruptions were frequent and it was vital to have data coming in that confirmed that systems were back up and operating as expected after each event.
Ignoring the problem is not a solution, nor is totally disconnecting control systems. The solution is to take the bull by the horns and progressively resolve the problem.
The Solution
So the challenge of IT Security is to come up with a workable solution and endeavor to mitigate risk. Here are some suggestions.
- Recognize that Control Systems are in a class all by themselves. Expect them to be behind the curve when it comes to IT security.
- Build in guarantees of security updates when bidding or purchasing control systems. Do this on a grading system, rather than mandated requirements. IT Security personnel will then learn how the marketplace is aware of security vulnerabilities, identifying vendors who are strong in IT security.
- For large enterprises, develop a direct link to software engineers. Forget the routine customer service stuff. When a security vulnerability hits the headlines, IT security administrators don’t need “Press 1 for air conditioners …” They need a direct line to the engineering group responsible for the product. Again – build that into the request for quotes. Establish the relationship prior to bid acceptance, reviewing security concerns pro-actively.
- Generate a unique subnet for control systems. Control systems will always be behind in security updates. Isolate them from the normal corporate network. With this arrangement, you can more effectively hide control systems, construct more restrictive firewalls that limits who can access these systems.
- Even better – develop an independent network for control systems. More on this below. This is the purest solution, where the control systems are connected to workstations and servers that are exclusively dedicated to the control systems, with no connection to the Internet.
- In all respects, block control systems from communicating over the Internet. Unless you are running highly specialized systems directly connected to manufacturers (like GE jet engines), control systems have no business communicating with anyone over the Internet. Turn off all automated interconnections with vendors and set up firewalls accordingly.
- Track CISA alerts for all control system vulnerabilities. IT security personnel need to note that control systems often share common elements. If one control system alert announces that Telnet is ON by default (which is a huge vulnerability), you can bet that the the control systems you use may have the same issue.
- Customize monitoring software for control systems. Control systems produce a unique array of information. Security monitoring can be customized to track those elements.
The Radical Solution
As mentioned above, the radical solution is to put workstations, servers and control systems on their own network. The chief characteristic of this arrangement is that it is strictly an intranet, with no Internet features.
Back in 1994, some users I supported had IBM 3270 terminals sitting on their desks, alongside the IBM PS-2 computers. I thought it was a bit curious since IBM had the capacity to use the PS-2 for terminal emulation. But it was a carry-over from the days before PC’s could do much; and the keyboard layout, the software they used and the training they received all revolved around these terminals. What was peculiar about this arrangement was that there were no security issues. They connected directly to the mainframe. There was no interconnection to anything else.
I am surprised to observe how IT managers forget that this same arrangement can be done with workstations today. It clutters up the cubicle, but it is doable. It produces some challenges in network design. It may involve setting up an unique Active Directory domain that is not connected to the corporate network, requiring manual insertion of updates into a central site that updates all workstations in the control system network. Or it may even involve manually updating each workstation.
The vulnerability is still there, but vastly diminished. It is interesting to note that the Stuxnet malware targeted Siemens controllers running centrifuges that purified uranium for the Iranian nuclear program and these computers were on a stand-alone network. The malware was introduced through a technician’s USB thumbdrive! So nothing is perfectly secured. Derivatives of the Stuxnet malware are still “in the wild” and can be used to structure similar incursions into targeted control systems.
The solution to the Stuxnet vulnerability is ultimately controlling file transfers through a USB drive. This can be done by using group policies in Windows (which requires Active Directory), but my guess is that it can also be custom designed into the Windows registry for stand-alone computers. For a long time Windows had no such capabilities. Alas, only Linux figured out that riddle!
Which leads to another level of radicalization – using computers running the Linux operating system. This option is usually not feasible because the software designers for the control systems almost all exclusively code through Windows. Moving over to Linux is a dispensational change, affecting everything from network security to personnel training. But Linux is inherently more secure. Very few malware programs are designed for Linux and clawing down to the root level of a Linux system is not easy for non-administrative users. Linux can be configured to restrict almost anything and monitoring Linux systems is vastly easier than Windows.
And probably the most revolutionary consideration is to use a non-TCP/IP network protocol. Sounds crazy? Before 2000, we used IPX/SPX that interconnected the Novell network. It was a bit kinky in interconnecting our regional clinics, but it worked. We forget that before TCP/IP, there was networking. Using a non-TCP/IP protocol would really put down the slammer on intrusions since there are not many intruders who understand anything other than TCP/IP.
Make Control Systems A Security Priority
I have studied security events in the past few years. They often have a common factor – obsolete remote access accounts that were never deleted. That is a huge security failure. But it points to how easy it is to overlook something as mundane as a routine review of an employee or contractor’s account termination. What these situations involved was remote access. It is something that is a fact of life in today’s world. System administrators, engineers and programmers will invariably need to access systems from remote locations. Control systems are almost always overlooked. They need to be placed into the same reviews that workstations, servers and network routers undergo. Their vulnerabilities need to be listed, prioritized and placed into a timeline. When a vendor cannot address a vulnerability, put the device on a short list for replacement. Simple as that.
IT managers and their employers need to pro-actively engage manufacturers of control systems. This needs to be a top-down commitment. As a specialist, if I questioned the security of a control system, I would generally get a polite non-substantive answer. But when the CIO asks the same question and the manufacturer realizes his opinion can be the difference of landing a multi-million dollar contract, things happen.
© Copyright 2022 to Eric Niewoehner