July 2, 2021 was one of the more complex and successful ransomware attacks in recent memory. As we’ve learned, REvil performed a sophisticated, timed attack on Kaseya VSA servers to distribute Sodinokibi ransomware out to clients and remove traces of them being there.

Throughout it all, our team played an active role in investigating and analyzing exactly how this attack was carried out while trying to help the businesses that were affected. We also did what we could to keep the community updated on all of our findings and research via Reddit and our blog. 

In one of our updates on July 2, we mentioned:

“For our Huntress partners using VSA, we took proactive steps to help protect your systems. We will send out a follow-up with details.” 

We decided to be intentionally vague until Kaseya released the required patch. Now, we can share those “proactive steps” we took, explain why we took them and show exactly what we mean when we say “our offense is your defense.”

Initial Attack Findings

About two hours after the ransomware incidents started, we were alerted to the payload that was used, obfuscated as “Kaseya VSA Agent Hot-fix”:

Our ThreatOps and Engineering teams reviewed the payload and were able to pull out some bits of information that would eventually lead to a way to “vaccinate” Huntress partners from getting encrypted. We looked at the following:

1. [.highlight]copy /Y C:\Windows\System32\certutil.exe C:\Windows\cert.exe[.highlight]

Make a copy of the legitimate Windows [.highlight]certutil.exe[.highlight] utility and place it in the [.highlight]C:\Windows\[.highlight] folder with a new name: [.highlight]cert.exe.[.highlight]

2. [.highlight]echo %RANDOM% >> C:\Windows\cert.exe[.highlight]

Append a "random" value to the end of cert.exe. This is accomplished with a built-in feature of DOS where the %RANDOM% environment variable will produce a random value when called. Appending this to the end of a legit executable doesn't prevent that executable from running, but it changes the hash which may be used in automated detection platforms to detect its use.

3. [.highlight]C:\Windows\cert.exe -decode c:\kworking\agent.crt c:\kworking\agent.exe[.highlight]

Using the modified but legitimate Windows certutil, the attackers decoded the malicious payload, [.highlight]agent.exe[.highlight] from the [.highlight]agent.crt[.highlight] file that was sent down to endpoints via the VSA server.

4. [.highlight]del /q /f c:\kworking\agent.crt C:\Windows\cert.exe[.highlight]

Delete the [.highlight]agent.crt[.highlight] and [.highlight]cert.exe[.highlight] files.

5. [.highlight]c:\kworking\agent.exe[.highlight]

Execute the ransomware payload, [.highlight]agent.exe[.highlight].

In these examples, [.highlight]c:\kworking[.highlight] was displayed and the default directory, but this is actually a configurable variable known as #vAgentConfiguration.AgentTempDir# in a given VSA deployment. If this is changed, the attack would've been carried out in the configured directory.

What is certutil.exe?

According to Microsoft...

“[.highlight]Certutil.exe[.highlight] is a command-line program, installed as part of Certificate Services. You can use [.highlight]certutil.exe[.highlight] to dump and display certification authority (CA) configuration information, configure Certificate Services, backup and restore CA components, and verify certificates, key pairs, and certificate chains.”

Essentially, if you give [.highlight]certutil.exe[.highlight] a certificate to decode, it does just that. In this case, [.highlight]cert.exe[.highlight] would base64 decode [.highlight]agent.crt[.highlight] to [.highlight]agent.exe[.highlight]. REvil understood this and used it maliciously. However, if there’s a file that has the same name as the decoded name, then [.highlight]certutil[.highlight] won’t decode it because it’s “in the way.”

Therefore, to “vaccinate” VSA servers from running a malicious program, all that was needed was to add an innocent file of the same name as [.highlight]agent.exe[.highlight].

Our Response

The Huntress platform allows us to pull files in case we need to run some extra investigation on suspicious activity, something we did a lot during this attack, but it also allows us to push files down to endpoints with the Huntress service. With that knowledge, our engineers got to work creating a fake [.highlight]agent.exe[.highlight] to send to the [.highlight]C:\kworking\[.highlight] dir.

However, there were a few problems with this vaccine:

• If REvil caught wind of the vaccine, they could just change the name of the file or directory, and it would run as intended.
• The [.highlight]kworking[.highlight] dir is configurable, so if it is named something different, the vaccine wouldn’t work.
• The Huntress [.highlight]agent.exe[.highlight] could be confused with the REvil [.highlight]agent.exe[.highlight].

Taking all of these into account, we decided it would be best to just push it out.

The decision to push out the vaccine as soon as we had it wasn’t something we took lightly. However, we saw what felt like an opportunity to help in the time of a crisis, and we knew the vaccine wouldn’t cause any damage. Because of this, we acted fast and pushed it out to our partners.

The vaccine was initially pushed out before 1830 ET that evening to all Huntress agents as long as they were checking in. The Huntress [.highlight]agent.exe[.highlight] is a text file that includes instructions for how to contact us.

We let Kaseya and other vendors know what the Huntress [.highlight]agent.exe[.highlight] file hashes were so they didn’t block it and wouldn’t have any false positives for any detectors:

Some partners have since brought up that they thought they were hacked because they saw the [.highlight]agent.exe[.highlight] file but then realized that it was the Huntress version. We never want to give our customers an unnecessary reason to panic, but in this emergency situation, we were okay with people being a bit shocked with what turned out to be an innocent file rather than being fully encrypted. Even so, we felt it wise to make another version of the [.highlight]agent.exe[.highlight] text file.

• • •

Hopefully, this update helps contribute to the efforts of cybersecurity researchers so we can all be more prepared for the next event.

If you need assistance—even if you're not a current Huntress partner—please contact our support team at support@huntress.com. We're working around the clock to support those who have been impacted by this attack.