Over a month ago, the CrowdStrike outage caused so much havoc across public and private entities. In the first blog post on this topic, our Cybersecurity expert, Sales Engineer Eric Beller concentrated on how organizations impacted could address the issue, why some organizations were spared, and the importance of backup, redundancy, and disaster recovery preparations. Read/reread “Partner Brief: Windows Host Outage/Crashes Due to CrowdStrike Update” to catch up on what happened and what next steps you should take with your customers.
There have been many headlines and stories on the economic costs of the outage, lawsuits, and explanations for the cause of the outage (which is interesting on an academic level but also creates many questions). Let’s focus on keeping the conversation moving forward. One comment we’ve heard from customers is “Why bother doing updates and patches, if they’re going to cause the same kind of damage as an actual cyberattack”. The quick answer? This did not create the same kind of damage as a cyberattack. No information was compromised. The breakdown remained a single area that was rectified without creeping into other business areas. There was a great deal of inconvenience, and much of the damage could have been prevented with the appropriate policy and procedures – particularly the ability to revert back to data saved securely offsite.
Compare and Contrast What Happens With An Attack | Updates and patches let organizations avoid events like ransomware attacks. If the patch from CrowdStrike fixed a zero-day vulnerability, the pain of downtime from this incident was well worth it. The alternative scenario, an actual ransomware attack, would see extended downtime, extortion, compromised data, ongoing disruption of customer confidence, and a much larger recovery bill and tough road forward.
Patch Management is Crucial in Ensuring Software Security and Reliability, Have Policies in Place and Test Before You Implement | Software updates and patches are a vital component of maintaining system security and reliability. Patch management is the process of distributing and applying updates to software, addressing errors, vulnerabilities, and bugs that can compromise the integrity of an organization’s technology. There is a level of trust organizations place in their technology vendors, and it’s a tough balancing act between dedicating in-house resources vs. relying on outside processes and policies to test thoroughly. This can be a strong argument for choosing to work with an MSSP, which conducts its own testing before implementation. Many ISPs also take the approach with their network OS, to ensure nothing breaks with an update.
“An effective Managed Patching Service reduces your attack surface to zero for all but the most advanced criminals and threat actors. Effective Patch Management makes you a less attractive target for criminals and is a vital part of your risk management and resiliency strategies.”
-Matt Douglass, CBTS, Senior Director of Solution Engineering
Whether the customer’s IT team, vendor, or a Managed Service Provider performs patching, make sure to have an initial test mode environment before it’s fully rolled out. This is sometimes referred to as sandboxing, allowing testing to be performed in a safe and isolated way. Always ensure there are proven methodologies and practices to roll back a patch update easily in case the patch causes a major disruption or worse. The NIST SP-800-4 policy outlines what is regarded as the standard for patching policy, and it’s a great place to start and see the complexity that can be involved. With Cyber-insurance premiums and deductibles rising significantly over the past few years, organizations can minimize increases by demonstrating a commitment to addressing vulnerabilities and achieving a strong security posture.
MSSPs can be a great option to gain industry-leading expertise, economies of scale, and reduce the burden on your IT team. Providers like CBTS, Cybersafe, DYOPATH, and TPx for example, all have extremely robust patching policies and reputations for their attention to detail, minimizing the risk to their clients while providing timely support to stay ahead of any issues. It’s a great idea to check if your chosen MSSPs perform patching and a well-developed strategy for implementing this service. Eric detailed in his first Partner Brief on the Outage how Cybersafe, in particular, stays a patch back (for the most part) to ensure they test each patch thoroughly before it goes live, which is why their customers were largely unaffected. This type of caution can help avoid cost and frustration, paired with the ability to revert back to data saved securely offsite, and organizations in this situation can minimize or even completely avoid network and data downtime.
“Patch and Vulnerability Management needs to be designed so new releases are tested in an environment without impacting operations should a fatal error occur. Once tested and proven, stagger and roll out updates across the environment in stages, leaving the most critical systems until the end of the patch cycle.”
-Thomas Pioreck, CISO at Cybersafe
The outage illustrated the importance many organizations place by relying on certified components in critical systems, and the ‘workarounds’ vendors are using to streamline their updates. CrowdStrike failed to have a stable/secure kernel which is essential, hammering home the need for careful software update management, especially kernel updates, to address vulnerabilities promptly.
“The software industry and the security industry, in particular, must rethink their respective approach to using kernels, certification of stability/security in the software development lifecycle, and change management.”
-Sam Bourgeois, Director Cybersecurity Services at DYOPATH
A well-implemented patch management process ensures that software is updated regularly, fixing errors and vulnerabilities that can be exploited by cybercriminals. Without patch management, systems become increasingly vulnerable to attacks, data breaches, and downtime. The consequences can be severe, including:
- Security risks: Unpatched software leaves doors open for hackers to gain unauthorized access, steal sensitive data, and disrupt operations
- System downtime: Failure to apply patches can cause software crashes, freezes, or compatibility issues, resulting in lost productivity and revenue
- Compatibility issues: Incompatible patches (like what happened with CrowdStrike) can lead to conflicts between software components, causing system instability and errors
The Purposes of Different Types of Patches
- Security patches: Fix vulnerabilities and errors that can be exploited by attackers
- Bug fixes: Resolve glitches and performance issues that affect software usability
- Compatibility patches: Ensure seamless integration between different software components
- Feature updates: Enhance software functionality and adding new features
Best Practices, Effective Patch Management
- Plan ahead: Schedule updates during maintenance windows to minimize disruptions
- Collaborate with technical teams: Ensure a common language and understanding of patching among security, IT, and DevOps teams
- Establish a disaster recovery process: Have a backup plan in case patching goes awry
- Stay informed: Monitor patch reports and advisories
Patch management is a critical component of software maintenance, ensuring the security, reliability, and performance of an organization’s data, network, and systems. Understanding the importance of patch management and implementing best practices helps:
- Protect against cyber threats and data breaches
- Minimize downtime and lost productivity
- Ensure smooth software operation and compatibility
- Maintain a happy and engaged Customer Experience (CX)
Patch management is not a luxury, it’s a necessity. Prioritize patch management strategy, policy, and practices to safeguard technology and maintain overall customer and stakeholder trust with your clients.
Your Sandler Partners support team can be as upfront or behind the scenes as you want – helping discovery/solution design with your customers, being less visible, or remaining hands-off. Involve your dedicated Sales Engineer if you need help engaging your customers in these incredibly important and often technical conversations. Hopefully events like this are infrequent, but we can learn from them, seizing the opportunity to discuss business continuity, patch management, backup, disaster recovery, and overall Cybersecurity policies to help mitigate downtime and risk. You can be crucial to helping your customer’s organizations maintain operations and remain secure while also adding to your revenue.
As a Partner, you have many resources to help you – from your dedicated Channel Manager and Sales Engineer (find them listed on your Sandler Portal homepage) to resources in your Portal like conversation starters, pre-built Marketing Center Campaigns, and my everyday favorite, the click-based Solutions Finder discovery sales tool, with searchable criteria from every one of our 200+ Providers. Blue Mantis, CBTS, Cybersafe, DYOPATH, and TPx are great Cybersecurity-Managed Service Provider options to review along with several other Partner favored options you can keep top of mind when discussing how to improve your customer’s threat posture and overall continuity practices.
Not a member of the independent Sandler Partners community? There’s never been a better time to Become a Sandler Partner, we’re ready to welcome you, and introduce you to all the benefits you have to gain!
Thanks for reading.
Gerry Davis
