Background
On 29 March 2023, CrowdStrike released a social media post and then a follow-on blog describing a supply chain compromise involving 3CXDesktopApp softphone software. Subsequent analysis and investigation indicated a complex, multi-stage supply chain intrusion of indeterminate length and unknown impact. As shown below, preparation for this event (particularly creation of network infrastructure associated with the infection chain) goes back to early 2022 in some cases, with more extensive action starting in early December 2022.
3CX Related Events Timeline
Initial reporting indicates behavioral detection of suspicious activity surrounding 3CXDesktopApp in Windows environments starting on the 21st and 22nd of March 2023. However, subsequent reporting from 3CX indicates that the earliest vulnerable versions of the software appeared in January 2023 with the 18.11.1213 macOS Electron application. Thus actual activity based on vulnerable versions may go back to January 2023 in at least macOS environments, with subsequent actions in Windows environments in March 2023 resulting in identification.
As indicated by 3CX’s CEO in an interview with CyberScoop,
“The macOS version only has a couple of thousands of users.”
Thus one possibility is that initial access and modification started with the less-prevalent macOS version, resulting in significantly less activity in an operating system typically featuring a lower degree of monitoring, before a Windows variant was deployed with the March updates (18.12.407 and 18.12.416). At this time, extension to a much larger installation base (that also typically features a higher degree of security tool monitoring), resulted in identification and subsequent mitigation work.
As of this writing, much remains to be learned about this event, not the least of which being what was the purpose of this supply chain intrusion. The entity identified by CrowdStrike as responsible, Labyrinth Chollima (also referred to more generally, broadly, and potentially problematically as “Lazarus Group”), is linked to cyber operations ranging from traditional espionage to cryptocurrency theft to destructive attacks. As a result, the potential outcome of this intrusion could vary widely - and given the large footprint of 3CX customers, one could easily hypothesize a use case for each in such a large and varied target set.
Interestingly, the prime initial function of the payload distributed with the malicious 3CXDesktopApp update appears to be browser information gathering. While this could be leveraged for harvesting items such as stored passwords, access tokens and similar information, current analysis indicates this information theft focuses on gathering browsing history. As noted by researchers at Volexity, it is possible that this mechanism is used to “fingerprint” infected hosts to enable triage and selection of which infected endpoints merit further operations.
Irrespective of attribution, motivation and purpose, the incident in question contains multiple lessons with respect to supply chain intrusions, defensive possibilities, and organization response operations. While we anticipate additional information on this campaign will emerge in the coming weeks and months, sufficient items are available now to inform defenders on what possibilities exist to counter activity such as this incident, and ensure continuity of operations in the face of supply chain-based cyber threats.
Contextualizing Supply Chain Intrusions
Supply chain intrusions contain a certain mystique within information security circles. Supply chain incidents, such as the incident involving SolarWinds Orion software in 2020, gather significant headlines and maintain an air of invincibility. If an adversary is able to inject a capability into a legitimate application, the popular imagination posits that such intrusions become near impossible to detect or mitigate, giving adversaries an almost unassailable advantage.
However, as noted in previous research, supply chain intrusions are more nuanced than many realize. Intruders must complete two intrusions: one into the organization or entity that will serve as a “vector,” then another via that entity to actual targets of interest. Furthermore, while the deployment from the supply chain mechanism to victims may be automated or relatively easy, widespread exploitation may result in “tipping off” analysts and defenders. More nuanced approaches to targeting and delivery through various checks and other items may be more stealthy, but as viewed in the SUNBURST payload injected into SolarWinds Orion, even these will leave behind artifacts for follow-up, such as the anomalous network connections used for victim fingerprinting in that campaign.
Supply chain intrusions, though difficult to identify, are not immune to defender identification. More interestingly, such intrusions may be just as difficult for adversaries to successfully weaponize as it may be for defenders to identify their initial presence.
In this case, we look to adversary dependencies. A supply chain intrusion mechanism delivers an implant or some access mechanism to an entity within the victim network. For that point of access to be useful or actionable, the adversary must satisfy one of two possibilities: developing a way to successfully communicate with that new point of access to ensure command and control (C2); or build a capability that can operate near autonomously within victim environments to avoid the need for positive control. The former means that classic mechanisms for monitoring C2 activity, such as identifying new or anomalous domains or network traffic analysis and monitoring, can identify the access point and related activity. The latter, while potentially quite powerful, is very difficult to develop and can either result in such a tool being “too aggressive” (thus overspreading and announcing its presence), or “not being aggressive enough” (leading to a failed mission).
As we continue to dig into intruder lifecycle dependencies and requirements to achieve objectives, ever more touchpoints such as those described above to identify, mitigate, or even prevent such activity emerge. Through considering these options, the scales begin to tip significantly, from an initial perception of overwhelming intruder advantage to potential equivalence, if not identifying outright defender superiority in well-designed, well-instrumented environments.
Orienting Adversary Actions In Space & Time
Building on the above, a supply chain compromise, except in the very rare instance where such a capability is fully autonomous and is designed to directly impact the device, software, or other that is also leveraged for access, becomes useless absent its purpose in achieving adversary objectives. As highlighted below, adversary operations take place across time and network “space.” Any intrusion requires proceeding through multiple steps, typically defined in the Cyber Kill Chain, from initial information gathering to ultimate actions on objectives.
Supply Chain Intrusions Mapped To The Cyber Kill Chain
Looking at the above, supply chain intrusions certainly form part of this sequence of events, and in some instances may result in very effective, difficult-to-detect mechanisms for satisfying preliminary stages of operations. However, absent autonomous capabilities requiring no further operator intervention, that implant will require follow-on communication. Even for an autonomous capability, it will need to move through the victim network to reach ultimate targets to achieve objectives.
All of these actions shift the intrusion from a stealthy, difficult-to-detect circumvention of multiple defensive layers into a more typical post-exploitation engagement. Even in past high-profile cases such as the SolarWinds Orion supply chain intrusion, intruders moved from an initial access point achieved through the supply chain vector to the use of Cobalt Strike Beacon for post-exploitation activity. Other mechanisms, such as credential harvesting for abusing remote logins, remote scheduled tasks, PowerShell and WMI-based mechanisms, and other standard lateral movement techniques are equally relevant. Defenders may lose some initiative in that “border defenses” are circumvented through a supply chain intrusion, but layered defenses including host and internal network visibility will still be effective in identifying post-compromise activity.
Notably, there are exceptions to the above. For example, the Kaseya supply chain compromise in 2021 resulted in timed, direct movement from initial access to the on-premise Kaseya VSA servers to hitting clients with ransomware. Effectively, this represents a case of a nearly autonomous payload directly impacting the very systems on which the compromised software is installed. In these cases, defense does become significantly more difficult, especially given the permissions and general allowlisting associated with remote management and maintenance (RMM) tools like Kaseya VSA. Defensive operations here become more limited, and require difficult decisions in terms of what extent such applications should be trusted and how much power should be given to security products that may identify abuse of these products - but in the process may also impact legitimate operations when preemptively acting.
Designing & Implementing Defense
While exceptions such as the Kaseya event exist, overall supply chain compromises require adversaries to satisfy multiple, interconnected, and interdependent requirements to achieve mission success. Emphasis on visibility, layered defenses, and implementing emergency plans for business continuity all factor into preparing for and responding to compromises of any type, including supply chain vectors.
Defense in depth focused on potential supply chain challenges begins with first understanding the defended environment:
- What software is installed?
- What versions are currently in use?
- What functions do these items perform?
Developing and maintaining a reasonable asset and application inventory can streamline response and recovery in the event of a supply chain incident. Being able to rapidly determine whether the organization is impacted, and to what degree, can enable significant efficiencies in the response, mitigation, and recovery process, assisting all further steps. In many instances, organizations spend valuable time attempting to determine whether they are even impacted in scenarios such as the 3CX event - time that could be better used to isolate assets and begin recovery.
Once applicability and scope are determined, defenders can move into mitigation and response steps. This includes being able to perform root cause analysis (RCA) on any activity identified that may be associated with installation of modified software or similar supply chain vectors. For example, defenders should be able to identify any attempts to “break out” and move laterally from the potentially compromised asset, indicating a “hands-on keyboard” intrusion has or is currently taking place. Similarly, reviewing network logs for the potentially compromised asset can identify potential C2 or exfiltration channels indicative of adversary success.
Mature, well-instrumented environments can apply some of these mechanisms to identify supply chain intrusions as they occur provided appropriate investigation takes place. For example, an environment with proper asset classification could identify irregular external network activity or authentication attempts (or similar lateral movement activity) from a server to clients or other devices indicating unexpected actions from that device, such as in the SolarWinds SUNBURST scenario. For circumstances such as 3CXDesktopApp users, identifying anomalous process chains, parent-child process relationships, or similar, while difficult, can flag potentially malicious activity of interest for further investigation.
Review Of Defensive Steps Through Adversary Phases & Kill Chain
An important caveat to the above recommendations is that none of this is easy. While mature organizations with significant security resources can (and should) strive to implement the above, the same is not the case for smaller organizations or entities that are resource constrained. For examples like 3CX’s telephony services, these applications are in widespread use across both enterprise and small business environments, meaning the capacity to respond to such events will vary greatly. Constrained organizations should look to prioritizing attack surface management to minimize and understand residual risk, and identify commercial and public partners that can assist in the event of compromise where possible. But most importantly for all entities, extending response planning and execution to business continuity and recovery is critical.
Balancing Security & Business Functionality
If a breach is identified and properly scoped, remediation requires not only recovering impacted systems and restoring operations, but doing so in a way that maintains business functionality and continuity to the greatest extent possible. In the case of 3CX software, this application represents a critical component of core business functionality: phone communication. While evidence is incomplete at this time, preliminary review of initial responses to antimalware services quarantining 3CX applications on 21/22 March, without providing significant explanation or context as to why this action took place, resulted in organizations allowlisting the application to ensure continuity of operations.
Example Communications Concerning 3CXDesktopApp Antimalware Identification
Many may wish to engage in post-incident, after-action assessment that many organizations, including third party entities from security organizations to system platform managers, failed in this instance in allowing matters to continue for several days from initial identification. Some may be even further tempted to shame victims for allowing this activity to occur after initial, ambiguous security notifications. However, analysts, asset owners, and end users must all understand the necessity of balancing continuity of operations with organizational security. Absent thorough RCA as documented above, which multiple parties failed to do for several days, such events will occur, and absent appropriate context we should assume business owners will default to maintaining critical application availability.
The above being said, organizations must plan for the loss or degradation of such services to enable appropriate, risk-based decision making in the future as part of their response plan to identified (or suspected) security incidents. In the case of 3CX, customers have the option of switching from the desktop application to browser-based versions to ensure operational continuity.
In this case, options from 3CX may minimize potential disruption. But organizations must assess the footprint of critical services and plan for their loss or degradation in advance across multiple systems and functions. Failure to do so means that a security incident metastasizes from an IT event to an event potentially impacting the ongoing survival of the organization.
Conclusions
Supply chain intrusions are difficult problems - but they are not impossible to counter or identify. While the 3CXDesktopApp incident continues to play out and there remains much to learn about this event, including how 3CX itself was compromised and what the adversary’s intentions were for this intrusion, defenders can nonetheless adopt a set of common controls, actions, and planning suggestions to minimize the impact of this and future supply chain events.
Most importantly, understanding the relationships between applications and their implications for operations may be vital in advanced planning and defensive action to minimize potential damage and disruption. Historically, some supply chain events, such as the Kaseya incident, resulted in near immediate impacts in terms of ransomware deployment. In the case of 3CXDesktopApp, rapid movement from initial action to effects could have led to results ranging from ransomware to deployment of a destructive wiper.
Thankfully, once the broader community became aware of events and could properly contextualize them following CrowdStrike’s disclosure, multiple entities were able to work together to research, assess, and potentially contain this threat before objectives could be accomplished. There is much work that remains to be done, and 3CX customers are strongly advised to follow the company’s current guidance to migrate operations to their browser-based platform. But absent immediate disruptions and business dislocations, this and similar events provide examples for how and why organizations must invest in the visibility and defensive capabilities necessary to identify, contain, and respond to future supply chain intrusions.
