July 2, 2021 was one of the more complex and successful ransomware attacks in recent memory. As we’ve learned, REvil performed a sophisticated, timed attack on Kaseya VSA servers to distribute Sodinokibi ransomware out to clients and remove traces of them being there.
Throughout it all, our team played an active role in investigating and analyzing exactly how this attack was carried out while trying to help the businesses that were affected. We also did what we could to keep the community updated on all of our findings and research via Reddit and our blog.
In one of our updates on July 2, we mentioned:
“For our Huntress partners using VSA, we took proactive steps to help protect your systems. We will send out a follow-up with details.”
We decided to be intentionally vague until Kaseya released the required patch. Now, we can share those “proactive steps” we took, explain why we took them and show exactly what we mean when we say “our offense is your defense.”
Initial Attack Findings
About two hours after the ransomware incidents started, we were alerted to the payload that was used, obfuscated as “Kaseya VSA Agent Hot-fix”:
Our ThreatOps and Engineering teams reviewed the payload and were able to pull out some bits of information that would eventually lead to a way to “vaccinate” Huntress partners from getting encrypted. We looked at the following:
1. copy /Y C:\Windows\System32\certutil.exe C:\Windows\cert.exe
Make a copy of the legitimate Windows certutil.exe utility and place it in the C:\Windows\ folder with a new name: cert.exe.
2. echo %RANDOM% >> C:\Windows\cert.exe
Append a "random" value to the end of cert.exe. This is accomplished with a built-in feature of DOS where the %RANDOM% environment variable will produce a random value when called. Appending this to the end of a legit executable doesn't prevent that executable from running, but it changes the hash which may be used in automated detection platforms to detect its use.
3. C:\Windows\cert.exe -decode c:\kworking\agent.crt c:\kworking\agent.exe
Using the modified but legitimate Windows certutil, the attackers decoded the malicious payload, agent.exe from the agent.crt file that was sent down to endpoints via the VSA server.
4. del /q /f c:\kworking\agent.crt C:\Windows\cert.exe
Delete the agent.crt and cert.exe files.
5. c:\kworking\agent.exe
Execute the ransomware payload, agent.exe.
In these examples, c:\kworking was displayed and the default directory, but this is actually a configurable variable known as #vAgentConfiguration.AgentTempDir# in a given VSA deployment. If this is changed, the attack would've been carried out in the configured directory.
What is certutil.exe?
According to Microsoft...
“Certutil.exe is a command-line program, installed as part of Certificate Services. You can use certutil.exe to dump and display certification authority (CA) configuration information, configure Certificate Services, backup and restore CA components, and verify certificates, key pairs, and certificate chains.”
Essentially, if you give certutil.exe a certificate to decode, it does just that. In this case, cert.exe would base64 decode agent.crt to agent.exe. REvil understood this and used it maliciously. However, if there’s a file that has the same name as the decoded name, then certutil won’t decode it because it’s “in the way.”
Therefore, to “vaccinate” VSA servers from running a malicious program, all that was needed was to add an innocent file of the same name as agent.exe.
Our Response
The Huntress platform allows us to pull files in case we need to run some extra investigation on suspicious activity, something we did a lot during this attack, but it also allows us to push files down to endpoints with the Huntress service. With that knowledge, our engineers got to work creating a fake agent.exe to send to the C:\kworking\ dir.
However, there were a few problems with this vaccine:
- If REvil caught wind of the vaccine, they could just change the name of the file or directory, and it would run as intended.
- The kworking dir is configurable, so if it is named something different, the vaccine wouldn’t work.
- The Huntress agent.exe could be confused with the REvil agent.exe.
Taking all of these into account, we decided it would be best to just push it out.
The decision to push out the vaccine as soon as we had it wasn’t something we took lightly. However, we saw what felt like an opportunity to help in the time of a crisis, and we knew the vaccine wouldn’t cause any damage. Because of this, we acted fast and pushed it out to our partners.
The vaccine was initially pushed out before 1830 ET that evening to all Huntress agents as long as they were checking in. The Huntress agent.exe is a text file that includes instructions for how to contact us.
We let Kaseya and other vendors know what the Huntress agent.exe file hashes were so they didn’t block it and wouldn’t have any false positives for any detectors:
Some partners have since brought up that they thought they were hacked because they saw the agent.exe file but then realized that it was the Huntress version. We never want to give our customers an unnecessary reason to panic, but in this emergency situation, we were okay with people being a bit shocked with what turned out to be an innocent file rather than being fully encrypted. Even so, we felt it wise to make another version of the agent.exe text file.
• • •
Hopefully, this update helps contribute to the efforts of cybersecurity researchers so we can all be more prepared for the next event.
If you need assistance—even if you're not a current Huntress partner—please contact our support team at support@huntress.com. We're working around the clock to support those who have been impacted by this attack.
