The last few weeks have continued to demonstrate the impact of a compromised software supply chain – whether that’s realised by malicious threat actors or a lack of software quality control.

CrowdStrike, one of the leading endpoint detection and response platform vendors, released a software update that effectively disabled computers running Windows globally. A poorly tested piece of code made its way via various established and trusted patching policy routes and caused a global catastrophe by stopping over 24,000 firms from functioning effectively. The repercussions are still being felt and costs and losses continue to mount. Not bad for a simple pointer error.

Guy Carpenter’s report estimates that the insured loss of the CrowdStrike incident is worth between $300 million and $1 billion US dollars. What does that mean to society? Over 7,000 flights were cancelled or delayed in the immediate aftermath. Thousands of holidaymakers suddenly found themselves without a holiday. Employees suddenly found themselves delaying meetings, missing key milestones, and unable to meet deadlines for stakeholders. Hospitals and other healthcare providers were forced to cancel surgeries – even for those who critically needed those treatments to survive. Critical National Infrastructure (CNI) operators had to deal with the potential of being unable to control critical systems. Banks and other financial firms were unable to process transactions. This is the sort of impact that governments expect from sophisticated threat actors embedding zero-day exploits into software – just like when the Ukrainian power grid was attacked nearly ten years ago. These events don’t just have an impact on business, they impact society, and they influence the global economy too.

One of the biggest tragedies is that the impact was unintentional. CrowdStrike released a flawed software update that caused a catastrophic global impact. This shows, once again, what happens if poorly validated software is released upstream in the supply chain – it expedites havoc for downstream organisations. This latest example clearly demonstrates how the software supply chain can have a significant impact on the security and stability of a nation.

Industries leveraging information technology must remember that sophisticated threat actors are using the same approaches to embed zero-day exploits in the software that powers our daily lives. These threat actors exfiltrate sensitive information for intelligence and gain the information advantage. They inject disruptive and de-stabilising misinformation to compromise our decision-making capability. They access critical systems that can cause catastrophic damage (think of Stuxnet). The reality remains that any firm involved in supplying CNI software and services is a target for sophisticated threat actors.

Sadly, the CrowdStrike outage comes just weeks before other headlines reporting similar ‘DevSecOps’ events performed by sophisticated threat actors. If we are not careful then DevSecOps will become DevSecOOPS! A Chinese state-sponsored hacking group – StormBamboo – compromised an undisclosed internet service provider to poison the automatic updates that are shipped to their customers’ equipment. While it’s not clear whether this was realised by a software supply chain attack or an adversary-in-the-middle attack, it is itself a software supply chain attack designed to open a backdoor to enable downstream maleficence on that ISP’s customers.

In another escapade of ‘DevSecOOPS’, a firm employed by a UK prime defence contractor to create a new staff intranet for submarine engineers, managed to outsource to developers in Siberia and Minsk. The possible security implications are that Russia may have planted zero-day exploits in the software to detonate at some future point as well as compromising the security-cleared personnel who maintain British submarines. This not only puts the personnel maintaining submarines at risk – as they may become targets for espionage themselves – but the security of our nation too. The Ministry of Defence has supply chain rules in place with threats of fines and sanctions against industry players to deter their deviance from the rules, however, these recent examples show that deterrence is not working, and the risk continues to rise.

However, these attack methods are not new. We saw a sophisticated threat actor target the SolarWinds development environment in 2020 so they could embed their malware into an application that thousands of global businesses relied on.

My growing worry is why the IT industry appears to fail in effectively vetting how software, libraries, and updates that are built and shipped. If nothing else, it lacks the necessary levels of transparency and end-users still blindly trust published software and updates. Why are we still burying our heads in the sand when we know the risks of using poor quality or vulnerable software?

At its core principle, a Zero-Trust way of thinking is like saying “prove it” before granting the access required to a system. We ask users of IT systems to prove who they are and where they are before we grant them access to sensitive data and if they aren’t where they should be or can’t prove their identity then we deny them access to that data.

How do we apply this principle to software? Software suppliers need to provide us with evidence as to the quality and reliability of their software so a risk management decision can be made on whether to use it and inherently trust it. One way of producing this evidence is to test the software in a ‘sandbox’ or ‘pre-production’ test environment. Another option is to collect software provenance information using Software Bills of Materials (SBOMs) to build a picture of the known vulnerabilities an item of software may inherit or incorporate. Tools like slsa.dev allow developers to incorporate this attestation information in the software through the development environment so end-users could understand if the toolchain used to build a software package may have been compromised and potentially malicious code injected into the resulting software package.

Nexor led a research project in conjunction with the UK Government and partners, which proved how a development environment can be made to adopt a Zero-Trust approach, by implementing a policy decision point that analyses the embedded records of a software development project against a given security and development policy across several development tools and at each stage in the software project. This research proved that making a security decision on importing past or previous software packages and the history of how they were produced could significantly reduce the likelihood of importing vulnerable software.

The learning from this research provides valuable insight as to how we can reduce the likelihood of negative software supply chain security events in the future. If a zero-trust development environment could inspect the provenance of the CrowdStrike software, analysing the embedded attestations, it would demonstrate that tests were not fully performed on the software. The development environment would enforce the policies, not permitting the software to be bundled together for release beyond the test environments. Similarly, if the software did manage to make its way into production usage and provenance records of that detail were embedded in it, a software consumer would not allow it to be imported, deployed, and used.

While you may argue this can be done in a (human) process, human processes are fallible to not being carefully followed. A zero-trust, provenance analysing development environment builds the process into the technology and automates it – and thus makes it much more reliable than any developer-intensive human process. It continuously enforces the security policy to ensure compliance during each interaction with the development environment. It is like using robots to build machines – much more reliable and efficient than ‘manual construction’. When software automates so much in the modern world why should the processes of quality software production be so human-intensive? I strongly believe, as a cybersecurity professional, that transparent automation as I’ve described above is a much more reliable route to achieving policy and process compliance that people will find easier to comply and more understandable and trustworthy.

Back to the point of zero-trust and software. You should:

• Prove that you’ve vetted your software updates before you import and execute them. Make sure they haven’t been tampered with and test them in a reference environment first.

• Prove that the code you intend to release complies with your security and development policies and let the consumers in your supply chain know it as openly and transparently as possible.

• Embed the provenance information of where your toolchain and libraries came from in your software package, to enable consumers to manage the risks and vulnerabilities of that software.

• Provide the data of how that software has been built and tested, so that consumers can do their own checks on the software quality and reliability.

• Use modern software development techniques, tools, and approaches such as embracing DevSecOps.

• Do not blindly trust your software suppliers. You should implement a delivery and quality checking procedure typical of any “Goods-In” process but for software as much as any material goods and pay close attention to the supply chain of the vendor.

• Reject any software that does not meet the quality criteria and send it back to the supplier.

• Ensure your business maintains a resilience strategy that minimises the blast radius of incidents within your IT infrastructure.

• Implement Zero-Trust enabled Development Environments.

Nexor are experts in secure software development and managing the risks of developing software in the high assurance sector. Whether you need to understand the risks of the software supply chain to your business or how to make your development environment secure with the minimum fuss, Nexor’s research and practice in the field can help you meet your assurance requirements. If you would like to learn more about how Nexor can help you please contact us today.