Industrial Systems: To patch or not to patch?

There are many peculiarities that must be taken into account when considering the safety of industrial systems and SCADA systems. One especially relevant is patching or updating the systems or software that they support. When through a security assessment of this type of system you get to the question: "And how do you carry out maintenance of systems to patch known vulnerabilities?” We can find very different answers. Some examples:

Option 1: Poker face

We do not apply security patches. It is not necessary since our industrial network is completely isolated, we rely in the ‘air GAP’ to protect our systems and anyway, most manufacturers use not publish security updates. On the other hand, sometimes the software upgrade also involves hardware change, so that budgetary constraints do not permit such updates.

This answer or other similars are quite common. And I do not think it is a crazy strategy to follow to not apply security patches when these conditions are met:

1. A risk analysis was performed to clearly understand what the threats that may affect the non-patched systems and what impact could have such threats. Note that I do not mean to make a superficial risk analysis, but I mean analyzing risks in depth. That is, know exactly what vulnerabilities are not patched up, how it could be exploited by an attacker and what compensatory measures are implemented to mitigate the risk of no patching it. When considering the threats should pay particular attention to the perimeter of industrial systems, points of interaction with traditional networks and access points that are easily accessible by visitors or the general public.
2. Once done this risk analysis, if the problems, costs or difficulties that result from applying the patches are greater than the risk of non-patching, it make sense don’t apply the patch.
3. This decision should be carried out in an informed and conscious way by the risk owner.

On the other hand, it is clear that we must put pressure on manufacturers to implement vulnerabilities management processes in their products and this point should be a key criterion in the selection of these technologies.

Option 2: The quiet man

Well, it depends on the manufacturer, the device and the technician is responsible for the update. We don’t have actually documented, but we use different methods such as direct download of patches from the manufacturer’s website (who doesn’t publish a signed hash of the file for verification after download, or when they do, we don’t check it anyway). Sometimes, to save time, we even download the patches from our home where bandwidth is higher than in the office. We recorded it in our USB and connect to the network of industrial systems that is completely isolated from the IT network. You know, the famous ‘air gap’. Other times, is a partner or the manufacturer who comes with its USB or with his laptop and connect it directly to our industrial network to apply updates or perform any other maintenance task.

In these cases, as you can imagine, the problem is that the ' isolation ' ceases to be such as USB, portable CD or turn connects to our isolated network. With these practices our systems are exposed to so many threats. Some of them could be:

• Malware that can cause performance problems or even a denial of service on these systems.
• Advanced Malware can even allow remote control or data theft. Although a priori this seems impossible in an isolated network, today we can find numerous proofs of concept on how they could perform these attacks dodging the 'Air GAP ‘.
• Fraudulent updates or patches downloaded from internet whose system changes can be different than expected.
• Connections from third parties to our network using their own laptops which may have a lower level of our security. Also, if we do not control what activities performed in our systems can be a source of threat to be considered. Do not forget that it is very likely that our partners also work for our direct competition, so it is a source of risk to consider.

Option 3: The outstanding

In our organization we have documented and secure processes for carrying out the update of all our industrial systems. We have different systems to be informed when any new vulnerability is discovered which could affect our systems. A comparative analysis of the risks between upgrade or leave unpatched the systems is performed, so the risk owner can set the criteria used for deciding whether to patch and in which term it should be done. Once we decided that a patch should be applied, we obtain it from a secure source verifying its integrity and authenticity, we deploy it in our test environments to verify that the update will not compromise the functionality or security of systems and, only after this, and under our strict control and supervision, the update is deployed in production within the time limit set by the risk owner.

Well, if you obtain this answer, you can’t do anything else but congrats the client. However, I never found this answer yet.

Of course I'm simplifying the possibilities and using hyperboles in this article, but my goal is make you think about the fact that manage critical vulnerabilities are a key aspect to consider in security evaluations, especially when you are evaluating Industrial or SCADA systems. If you don’t update your systems, you will be accumulating vulnerabilities but if you update it in the wrong way, the update process itself can be a (big) risk source. Therefore, to plan, to document and to establish a continuous improvement process over the vulnerability management process should be in the agenda of any CSO who intends improve the security of his organization.