Skip to content

EricN Publications

Publications by Eric Niewoehner

Menu
  • Non-Fiction
    • A God Thing
      • Rachel’s Prairie
      • Adolescent Meditations
    • Oakland
      • Old Buildings Can be Creepy
      • Food for Thought
      • Dust in the Wind
      • The Arc of Travel
    • Iconium
    • Thinking Out Loud
      • Is Juneau Running Out of Diesel?
      • The Volkmer Effect on Alaska’s Congressional Representation
      • The Tragedy of Russia
      • Are We In A Recession?
      • Where Does Inflation Come From?
      • The Mail-In Ballot Experience
      • The Debt We Owe
      • The Solution to Paying Off Student Debt
      • Student Debt: Are Colleges to Blame?
      • The Problem of Student Debt
      • Fighting Inflation – A Trip to the Grocery Store
      • Ukraine – Poland 1939 All Over Again
      • Broken: Is Public Education Beyond Repair?
      • Transparency and Critical Race Theory
      • The New Wave
      • The Pro Act and YouTube
      • Almond Abstract and the Pursuit of Happiness
      • Open Letter to Major League Baseball
      • Tribute to Rush Limbaugh
      • Why Parler?
      • Adults in the Room
      • The Invisible Hand
      • Transparency
      • Fake News Update
      • Solution to Fake News
    • Old Friends
      • The Ideological Origins of the American Revolution
        • Sources and Traditions
        • Power and Liberty
        • The Logic of Rebellion
        • A Note On Conspiracy
        • Transformation — From Reaction to Revolution
        • The Contagion of Liberty
      • The Road to Serfdom
        • Introductions — Or Before You Begin
        • The Road Once Traveled
        • Are You A Nazi?
        • Are You A Socialist?
        • Individualism
    • The Pandemic Journal
      • When Does Ten Percent Matter?
      • Federalism — Or Intentional Chaos
      • Faith (Part II)
      • Faith
      • The Big Surprise
      • Teleworking
      • The Invisible Hand
      • Unbelievable
      • Rethinking Education
      • Perception
      • Selfie and the Mask
      • China
      • Strategic Globalism
      • Risk
      • Media
      • Unknowable
      • Home Schooling
      • Grocery Chronicles
    • When Few Were Watching
    • The Advent Conspiracy
      • What’s Missing
      • Cash Only Please
      • Worship Fully
      • Give More
      • Christmas — It’s a Tide, not a Day
  • Fiction by Eric Niewoehner
  • Technology Publications
    • Technology Blogs
      • A Play on Words
      • Spam Update — How to Avoid Spam
      • Facebook — A Sickening Feeling
      • Can PayPal Be Trusted?
      • Are You Ready to Rumble?
      • The Alaskan Congressional Rodeo
      • The Gift of Gab
      • Facebook – What Say You?
      • What’s Up with Parler
      • Out of Control? – Security Vulnerabilities in Control Systems
      • The Case of Lorie Smith
      • Facebook — Impersonated Accounts
      • The Case of Josh Renaud
      • SHAKEN, But Not STIRred
      • Can GoDaddy Be Trusted?
      • Ransomware and the Mechanical Pencil
      • Tracking Scams in Phone Messages
      • Disaster Recovery — The Case of Parler
      • CVS Update
      • Why Parler?
      • Computer Vision Syndrome
      • Passwords – the Gremlins of Cyberspace
      • Spam Filtering: Mastering Your E-mail
    • The Facebook Safe Space
    • Technical Documentation
      • Welcome to DOGland
      • Windows 10 and Computer Vision Syndrome
      • Sustainable Printing
    • The Tech Community
    • Introducing Substack and Locals.Com
  • Contact
  • About
  • Lessons Learned
Menu
Colonial Pipeline Tanks

Out of Control? – Security Vulnerabilities in Control Systems

Posted on April 5, 2022December 30, 2022 by Eric Niewoehner

Control Systems.  Are they possibly the biggest security vulnerability?

By now almost everyone has seen the news media on the cyberattacks stemming from Russia. So far nothing dastardly has occurred at the same scale of the Colonial Pipeline fiasco in 2021. That event involved the deployment of ransomware across the corporate network. As a precaution, the company shutdown its network that monitored the pipeline which, in effect, shut down the pipeline itself. The end result was a frightening interruption in the gasoline supply in the southeast part of the U.S. So when it was announced that Russia was on the prowl, it obviously raised concerns that vital infrastructure could be at risk.

As of this day, not much has been released regarding the how and why of the Colonial Pipeline event. It is one of the fatal flaws of the IT security industry. People tend to hold things close to the chest, what we call “security through obscurity.” Reveal too much, people may learn how your network is designed, how your security is structured and how competent or incompetent your personnel are in regards to IT security. But this much I have picked up. There is no divulged evidence that the “control systems” of the Colonial Pipeline were effected. But there is some indication of other serious faults in their IT security. In the end, Colonial had to pay a ransom to regain control of their systems.

So I can’t help be a bit curious when I see flowing from CISA an unending listing of vulnerable control systems. CISA stands for the Cybersecurity and Infrastructure Security Agency. It is part of the Department of Homeland Security. There have been seven advisories posted warning of security vulnerabilities on control systems through March 2022. Every aspect of IT has its own risk factors. Windows and Apple, for example, are operating systems that reside on almost every computer on the planet. Android and Apple are operating systems on smartphones. Certain applications have vulnerabilities and require routine updates, such as Word or the Firefox browser. But control systems – they are in a class by themselves.

A/C Control Panel
Example of an A/C Control Panel

What is a Control System?

Control systems are specialized computer technologies designed to perform specific functions. In my experience, the most common control systems I have encountered are air conditioners and power conditioners. These two items are a part of every computer center and are vital to the performance and reliability of computer services. Remote monitoring is an important component of that. If an air conditioning unit malfunctions, it can automatically send notification to operators of the computer center. Temperature alerts can be transmitted as well. And when power services become unstable, the data directed to the computer center operators enables a pro-active timely adjustment and better communication to affected customers and staff.

Not all control systems are sophisticated creatures on the Internet.1 I have programmable thermostats in my home and they are not interconnected. Alarm systems are another common element of a house, not all of which are connected to a network. But once you get “smart” and connect household devices to the Internet, you expose those components to cyberattacks. There have been several stories posted regarding Ring. I see the Ring doorbell or the Ring label in people’s windows beside their front door. While it is super convenient to have suspect activity relayed to your telephone, it can literally open your entire house to malicious actors. The same applies to your garage door opener. While downloading an app to your phone is amusing, it can also expose your home to cyberattacks.

Pipelines, power lines and nuclear reactors are no different. The intelligence provided by control systems enable corporations the ability to deliver a more reliable service at less cost with better safety. Network engineers go to considerable lengths to secure the infrastructure. But they are human and it has been demonstrated time and time again that if there is a vulnerability, it will be found.

The Problem of Obscurity

Anyone who installs a device with a control system, they have one objective – to get it to work. It must do a specific task. When air conditioners were installed in the regional computer center, it had one primary purpose – to keep our computer center cool so that our equipment would perform at optimal levels. Of the 150 people in the office who knew anything about those air conditioners, it was usually two of us at any time. The rest of the office never thought about it.

In a similar manner, the power conditioners that were installed were designed for two reasons – to provide stabilized power to our computer systems and to provide at least fifteen minutes of backup power. At one time in my career, power conditioners were designed to communicate with our servers to coordinate power reductions and then system shutdowns in the event of a major power disruption. Critical systems were placed on a 24/7 circuit backed up by diesel generators, designed for automated restart upon power restoration. Pretty fancy stuff. When it worked, nobody thought about it! And that was the point. It was dependable, virtually eliminated extended downtime, removed entirely faults due to brownouts, and maintained at near 100% levels critical communication systems. Again, only two people in the building knew about all that stuff.

And that is the problem with control systems. Business managers are usually clueless about these sort of devices. In fact, I have worked with some IT Security managers who appear to be clueless about control systems. It was interesting to see how security managers were surprised when “unknown” devices started showing up in computer scans, only to learn that they were power conditioners, air conditioners, telephone relays, radio dispatch centers, security cameras, and environmental controls. All these devices had a purpose. But security was the last thing on anyone’s mind when they installed them. Even more fascinating was that the people who installed them and managed them were not IT personnel, but HVAC specialists, electricians and radio technicians.

The Problem of Obsolescence

In my article, Ransomware and the Mechanical Pencil, I point out one of the major reasons why some obsolete IT systems are out there. Not everything connected to computers receives a monthly security update. The primary reason is that control system manufacturers focus on the device, not the software. I will take as an example the air conditioning system I integrated into our computer center in about 2014. It was a new system and it came with a network card that plugged into our network. Nice. Gave it an IP address. Tested the connection OK and then configured the air conditioner. After an evaluation period of about a month I invoked secured HTTP. Suddenly I lost connection. Instead I got this nasty error “SSL_ERROR_UNSUPPORTED_VERSION.” After decades of working on the Internet, this was my first encounter with this problem. The best I could find on the subject was buried in the archives of Mozilla’s support documentation. It provided the following definition:

“Peer using unsupported version of security protocol.” On a client socket, this means the remote server has attempted to negotiate the use of a version of SSL that is not supported by the NSS library, probably an invalid version number. On a server socket, this means the remote client has requested the use of a version of SSL older than version 2.

Upon further research I discovered that all approved browsers did not recognize older versions of SSL (Secured Socket Layer). It was considered highly unsecured. And, get this, the document was posted in 2008!

I said to myself, “You have got to be kidding. This is a brand new air conditioner!”

After consulting with the A/C unit’s customer support, it was determined the card needed a firmware update. The software that ran on the network card was embedded in a chip. In the meantime, I had some fun with this situation. I located on the Internet a solution to this problem. I installed a shell program that put me back in time, creating a virtual session of Windows 98! I was using Windows 7 at the time, so I was in effect going back almost 20 years. Within this shell, I installed a version of the Firefox browser that was about ten years old. With this combination, I created a connection that predated the security block in the browser, connecting to the AC unit. Through that connection, I turned off HTTP security, regaining access to the air conditioner control system.

Control Systems still utilize DB-9 Serial Cable for communications.
DB-9 Serial Cable

Seeing the old Windows 98 graphics and the ancient Firefox program was a fun trip down memory lane. To load the firmware was a bit more entertaining. I had to connect to the AC unit through a serial port. This is 1990’s technology! Our laptops no longer had serial ports. We were not even certain if the bevy of cables in the computer center were serial-compliant. So I asked a friend from the local radio station to assist. Alas, there is still tons of stuff running on old standards in the radio world. Together we tested the serial cables, tested the USB-to-serial dongles, and then played with the Telnet2 command interface. I uploaded the firmware package to the AC unit and updated the firmware on the network adapter. Viola – problem solved.

This dramatically demonstrates the nature of control systems. The primary objective of control system designers is to provide a trustworthy and dependable operation of a service. In my case, it was the air conditioner. Their concern is to assure that the room is cooled at the desired level. Traditionally technicians have configured air conditioners through a primitive LCD panel on the unit. Security is not exactly their forte. The software on the network card which provided the intelligence to manage the unit remotely was a secondary design, almost an after-thought. So it is no surprise that they sold a product that was already obsolete in the world of cybersecurity. The subsequent updates were often months, if not years, behind in regards to security. This, in essence, explains why CISA has posted so many alerts on control systems.

Mountaintop Radio Tower

What IT Security Overlooks

On the flip side are the IT Security nerds. The lazy response is to force control systems off-line. That did not go far simply because control systems are vital to operations. When I worked for the USFS, control systems were to be found on everything from power stabilization to security cameras, to scientific instruments on Mt. St. Helens and the webcam inside the beaver lodge. Our Alaskan radio towers were often isolated atop mountains bedecked with remote monitoring technology. As regards taking a control system off-line, the answer is often “No” and “That’s Impossible.”

The second thing IT Security overlooks is the reality of today’s IT support world. When I worked at the University of Missouri School of Medicine, I supported a network of about 100 people. When I moved up to Alaska, the objective was to increase that to one support specialist per 450 users. In the USFS, it was in the neighborhood of one in 4500 by the time I retired! Related to that dynamic is the fact that some of us moved about a lot. I wasn’t even in town four months of the year. With teleworking, I was only in the office two days a week. To be physically present to make daily inspections of equipment was impossible. So operating a computer center “in the blind” made no sense. Remote monitoring systems were vital. It explained in many respects why the regional computer center could operate at 99% performance. With Juneau’s famous winters and Taku winds, power disruptions were frequent and it was vital to have data coming in that confirmed that systems were back up and operating as expected after each event.

Ignoring the problem is not a solution, nor is totally disconnecting control systems. The solution is to take the bull by the horns and progressively resolve the problem.

The Solution

So the challenge of IT Security is to come up with a workable solution and endeavor to mitigate risk. Here are some suggestions.

  • Recognize that Control Systems are in a class all by themselves. Expect them to be behind the curve when it comes to IT security.
  • Build in guarantees of security updates when bidding or purchasing control systems. Do this on a grading system, rather than mandated requirements. IT Security personnel will then learn how the marketplace is aware of security vulnerabilities, identifying vendors who are strong in IT security.
  • For large enterprises, develop a direct link to software engineers. Forget the routine customer service stuff. When a security vulnerability hits the headlines, IT security administrators don’t need “Press 1 for air conditioners …” They need a direct line to the engineering group responsible for the product. Again – build that into the request for quotes. Establish the relationship prior to bid acceptance, reviewing security concerns pro-actively.
  • Generate a unique subnet for control systems. Control systems will always be behind in security updates. Isolate them from the normal corporate network. With this arrangement, you can more effectively hide control systems, construct more restrictive firewalls that limits who can access these systems.
  • Even better – develop an independent network for control systems. More on this below. This is the purest solution, where the control systems are connected to workstations and servers that are exclusively dedicated to the control systems, with no connection to the Internet.
  • In all respects, block control systems from communicating over the Internet. Unless you are running highly specialized systems directly connected to manufacturers (like GE jet engines), control systems have no business communicating with anyone over the Internet. Turn off all automated interconnections with vendors and set up firewalls accordingly.
  • Track CISA alerts for all control system vulnerabilities. IT security personnel need to note that control systems often share common elements. If one control system alert announces that Telnet is ON by default (which is a huge vulnerability), you can bet that the the control systems you use may have the same issue.
  • Customize monitoring software for control systems. Control systems produce a unique array of information. Security monitoring can be customized to track those elements.
IBM-3270 Terminal
IBM-3270 Terminal

The Radical Solution

As mentioned above, the radical solution is to put workstations, servers and control systems on their own network. The chief characteristic of this arrangement is that it is strictly an intranet, with no Internet features.

Back in 1994, some users I supported had IBM 3270 terminals sitting on their desks, alongside the IBM PS-2 computers. I thought it was a bit curious since IBM had the capacity to use the PS-2 for terminal emulation. But it was a carry-over from the days before PC’s could do much; and the keyboard layout, the software they used and the training they received all revolved around these terminals. What was peculiar about this arrangement was that there were no security issues. They connected directly to the mainframe. There was no interconnection to anything else.

I am surprised to observe how IT managers forget that this same arrangement can be done with workstations today. It clutters up the cubicle, but it is doable. It produces some challenges in network design. It may involve setting up an unique Active Directory domain that is not connected to the corporate network, requiring manual insertion of updates into a central site that updates all workstations in the control system network. Or it may even involve manually updating each workstation.

The vulnerability is still there, but vastly diminished. It is interesting to note that the Stuxnet malware targeted Siemens controllers running centrifuges that purified uranium for the Iranian nuclear program and these computers were on a stand-alone network. The malware was introduced through a technician’s USB thumbdrive! So nothing is perfectly secured. Derivatives of the Stuxnet malware are still “in the wild” and can be used to structure similar incursions into targeted control systems.

The solution to the Stuxnet vulnerability is ultimately controlling file transfers through a USB drive. This can be done by using group policies in Windows (which requires Active Directory), but my guess is that it can also be custom designed into the Windows registry for stand-alone computers. For a long time Windows had no such capabilities. Alas, only Linux figured out that riddle!

Which leads to another level of radicalization – using computers running the Linux operating system. This option is usually not feasible because the software designers for the control systems almost all exclusively code through Windows. Moving over to Linux is a dispensational change, affecting everything from network security to personnel training. But Linux is inherently more secure. Very few malware programs are designed for Linux and clawing down to the root level of a Linux system is not easy for non-administrative users. Linux can be configured to restrict almost anything and monitoring Linux systems is vastly easier than Windows.

And probably the most revolutionary consideration is to use a non-TCP/IP network protocol. Sounds crazy? Before 2000, we used IPX/SPX that interconnected the Novell network. It was a bit kinky in interconnecting our regional clinics, but it worked. We forget that before TCP/IP, there was networking. Using a non-TCP/IP protocol would really put down the slammer on intrusions since there are not many intruders who understand anything other than TCP/IP.

Make Control Systems A Security Priority

I have studied security events in the past few years. They often have a common factor – obsolete remote access accounts that were never deleted. That is a huge security failure. But it points to how easy it is to overlook something as mundane as a routine review of an employee or contractor’s account termination. What these situations involved was remote access. It is something that is a fact of life in today’s world. System administrators, engineers and programmers will invariably need to access systems from remote locations. Control systems are almost always overlooked. They need to be placed into the same reviews that workstations, servers and network routers undergo. Their vulnerabilities need to be listed, prioritized and placed into a timeline. When a vendor cannot address a vulnerability, put the device on a short list for replacement. Simple as that.

IT managers and their employers need to pro-actively engage manufacturers of control systems. This needs to be a top-down commitment. As a specialist, if I questioned the security of a control system, I would generally get a polite non-substantive answer. But when the CIO asks the same question and the manufacturer realizes his opinion can be the difference of landing a multi-million dollar contract, things happen.

© Copyright 2022 to Eric Niewoehner

Read More Technology Blogs

Related

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

EricN Publication Logo
  • Facebook page for EricN Publications
  • LinkedIn page for EricN Publications
  • Twitter page for EricN Publications

Recent Posts

  • Why The Facebook Safe Space
  • Facebook — Impersonated Accounts
  • Individualism
  • A Play on Words
  • Spam Update — How to Avoid Spam

Categories

  • A God Thing
  • Advent Conspiracy
  • Alaska
  • Bernard Bailyn
  • Documentation
  • Economics
  • Education
  • FA Hayek
  • Facebook
  • Faith
  • History
  • Life
  • Missouri
  • Non-Fiction
  • Oakland
  • Old Friends
  • Pandemic Journal
  • Politics
  • Security
  • Social Media
  • Tech Blogs
  • Technology
  • Thinking Out Loud

Archives

  • January 2023
  • December 2022
  • November 2022
  • October 2022
  • September 2022
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • December 2021
  • September 2021
  • August 2021
  • July 2021
  • April 2021
  • February 2021
  • January 2021
  • August 2020
  • July 2020
  • June 2020
  • May 2020
  • April 2020
  • November 2018
  • August 2018
  • April 2018
  • February 2017
  • November 2016
  • October 2016
  • July 2016
Copyright Notice

All articles are copyrighted material from Eric Niewoehner.

  • Facebook
  • LinkedIn
  • Twitter
© 2023 EricN Publications | Powered by Minimalist Blog WordPress Theme