In cybersecurity, time is critical.
Whether you’re chasing down a threat actor who’s laterally moving through your networks or working to isolate a host to stop a ransomware attack in its tracks, even one second can make all the difference.
I’m fortunate to work with incredible technology and the best cybersecurity experts in the industry, as they afford us the agility to respond quickly to events.
We saw this firsthand during the first holiday weekend in 2022. Surprise, surprise—the first holiday weekend of the year happened to mark the date of our first threat advisory of the year as well.
A Recap of Events
1448 ET - January 14, 2022
It’s a normal Friday afternoon. Our 24/7 ThreatOps team is hard at work, threat hunting and sending incident reports to our partners to ensure their customers are protected.
At 2:48pm ET, one of our analysts investigates a Windows Defender alert (we review these to weed out Defender false positives). After a quick investigation, it’s determined that it’s Cobalt Strike, a remote access tool (RAT) that attackers use to maintain unauthorized access to either install additional malicious software (e.g. ransomware) or attempt to laterally move through the network to other systems to increase their ability to cause havoc. Once confirmed, our analyst sends a critical report to the partner.
Within minutes, another Windows Defender alert comes in from another system from a completely different organization. Once again, the analyst performs a quick investigation and sends a critical incident report to the partner.
In both cases, Windows Defender did its job and protected the endpoint from the RAT being executed. But one thing to consider, and a key reason we added our Managed Antivirus solution, is that attackers will often try something else if they are blocked. Monitoring antivirus detections is a critical aspect of a defense-in-depth strategy and provides context to call the bigger picture into view.
1700 ET - January 14, 2022
As our analysts continue to do their job, one analyst notices a commonality between the two previous reports that both are VMware Horizon servers. After further review and some open-source research, we determine that this may be a bigger incident.
Our Head of Threat Operation Center quickly puts out a message on Twitter to let others know we’re observing Cobalt Strike activity as well.
At this point, it’s starting to look like this is related to the UK’s National Health Service (NHS) alert from January 5 that hackers were actively targeting Log4Shell vulnerabilities in VMware Horizon servers in an effort to establish persistent access via web shells.
And then, it clicks: It’s not just another Friday, but it's a holiday weekend—a prime time for cyberattacks. For defenders, that can often mean a long and busy weekend is ahead.
We start letting our entire ThreatOps team and others at Huntress know that it might be a rough night. Luckily, we have such an amazing team that instead of shutting down the computer for the holiday weekend, many of our teammates jump back online to help out a lending hand—although I’m sure we were all feeling the sentiment described below.
1938 ET - January 14, 2022
After a quick team-wide debrief, one of our leaders on the Huntress product team recommends we roll out our managed endpoint detection and response (EDR) capability, Process Insights, to all systems with VMware Horizon. This new endpoint detection and response (EDR) capability is based on an acquisition we made in early 2021 that allows us to proactively detect and respond to non-persistent malicious behavior by giving us the ability to collect detailed information about processes. After a quick chat, we contacted a few of our private beta participants and let them know our plan to roll them out to these servers due to the growing possibility of a large-scale incident.
1945 ET - January 14, 2022
Now that we had confidence that this was not a false alarm, we decided to get the word out to the community. We post on r/msp, notify many in the MSP peer group and begin working on notifying our partners. The ThreatOps team puts together a spreadsheet that includes all of our partners with customers using VMware Horizon servers. We give that to our sales team, executives and others who were anxious to give a lending hand to our partners.
After a short period of time, ThreatOps is able to update the list to include other key details, including the version of VMware Horizon server which allows us to further pinpoint the affected systems. After giving a few thankful partners the heads up—as well as inadvertently scaring a few partners that had already patched—we stopped the calls so we could use this new spreadsheet and give a more focused notification.
At this time, with the additional information on versions of VMware Horizon servers that are vulnerable to Log4Shell, we decide to proactively roll out our managed EDR feature to all of these systems.
2105 ET - January 14, 2022
We decided to start digging into VMware’s articles as well as VMware’s Security Advisory to better understand their recommendation. As we dug into VMware’s mitigation tool, we realized that if a web shell is discovered, the tool does not do anything to remove the web shell.
We continued to analyze this and decided that the best way to ensure that no web shells exist on the system would be to restore from backups. In order to do this, we really needed to download every JavaScript file and analyze it to determine which had web shells and when they were inserted so we had a good timeline of the original attack.
0054 ET - January 15, 2022
After long hours of digging into every vulnerable host, updating our spreadsheet to include which ones had already been exploited and had web shells, one of our researchers recreated the backdoor trigger and was able to execute a simple “whoami” command.
We were now able to confirm this wasn’t just a theory—this was bad. The web shell gave system access just as the earlier research had indicated.
The good news was that within a couple of minutes of processing time, our new managed EDR capability was able to not only see this command but also give us a visualization of the process tree.
Within 15 minutes, our team was able to create a detector based on this behavior and similar behaviors which allowed Process Insights to alert us of web shell triggers giving us visibility into previously unknown execution.
This was a moment we all realized the hard work that has been put into Process Insights over the last year was worth it. We now not only had our beliefs that Process Insights would be game-changing for our partners and their customers, but we now had a real-world example of it in action. We also solidified why our Managed Antivirus capability was so vital because we were able to turn an alert into a detector that can give us an earlier warning on malicious behavior. This was all done in less than a 12-hour window.
0530 ET - January 15, 2022
At this point, we’d sent out reports to all of our customers, we’d updated our Reddit post with the most up-to-date information including mitigation techniques and we had called every partner with web shells installed. Our analyst in Australia was finishing up his shift while our analyst in the United Kingdom was fully up-to-speed and ready to go for the day. At this point, our incident response team decided to call it a night as we knew it would be a busy day with follow-up questions and details.
The Rest of the Holiday Weekend
Our team was hard at work continuing to send incident reports for Cobalt Strike, cryptominers and other malware being installed on the VMware Horizon servers. We continued to contact our partners to let them know that they really needed to patch before this escalated into something more serious—like attackers spreading throughout the network or even launching ransomware. Many of our partners believed they had patched, but most of the time, they were just running the mitigation script and not actually patching.
Lessons Learned
We’ve said it many times before, but it bears repeating: No security vendor is going to be able to stop something 100 percent of the time. As many cybersecurity experts say, there is no silver bullet in security, so it’s important that you have patching, security-focused processes, multi-factor authentication (MFA), and other defense-in-depth practices.
And as for Process Insights, our managed EDR capability? It’s now generally available to all new and existing partners! If you're a Huntress partner, please reach out to your account manager or support for more information.
