UPDATE 2/27/23: As recently spotted by Fox-IT and subsequently reported in SecurityWeek, a critical vulnerability discovered last year in ConnectWise’s R1Soft Server Backup Manager software has now been seen exploited in the wild to deploy backdoors on hundreds of servers via CVE-2022-36537. The article below was originally published in late October by Senior Security Researchers John Hammond and Caleb Stewart detailing the ways in which this vulnerability could be exploited, and it has been confirmed this method is now being utilized by threat actors.
Community education and outreach are critical to driving awareness of these situations and avoiding damage by offering expert, data-driven insights for remediation. It's both a privilege and a responsibility to partner with the broader security ecosystem to stop hackers in their tracks and mitigate risks to organizations across the globe. Huntress will continue to share far and wide the findings of our in-house experts on both the ThreatOps and Research & Development teams so the community can benefit.
Huntress has validated an initial report for an authentication bypass and sensitive file leak present in the Java framework “ZK”, used within the ConnectWise R1Soft software Server Backup Manager SE. This post details the severity of this supply chain finding, emphasizes the immediate need to patch and highlights the importance of close vendor collaboration within the SMB/MSP communities.
The video below demonstrates the Huntress POC exploit being used to these aforementioned steps: 1) bypass authentication, 2) upload a backdoored JDBC database driver to gain code execution, and 3) use the REST API to trigger commands to registered agents to ultimately push the recently leaked Lockbit 3.0 ransomware to all downstream endpoints.
Initial Discovery
Markus Wulftange of Code White GmbH originally discovered a vulnerability in the ZK library, and following responsible disclosure, helped encourage the patched release of ZK version 9.7.2 in May 2022. This authentication bypass vulnerability was dubbed CVE-2022-36537.
A colleague of Markus, Florian Hauser (@frycos), saw that the ZK library was bundled with ConnectWise R1Soft Server Backup Manager software and tried to notify ConnectWise in July 2022. Without a response after the 90 days of disclosure standard, @frycos teased screenshots of how to replicate the issue on Twitter. Days later, the post was removed; ConnectWise kindly asks researchers to use the disclosure form located on their Trust Center homepage.
Huntress Research
From this tweet, Huntress has reproduced this issue and furthered the proof of concept exploit, leveraging the initial vulnerability to leak server private key files, software licenses, system configuration files and ultimately gain Remote Code Execution (RCE) as the system superuser and manipulate R1Soft to push further arbitrary code execution downstream to all registered endpoints.
We have worked closely with ConnectWise to advise them of these issues and they have released a patch for Server Backup Manager SE software. Huntress has validated their patch and confirms it is effective against stopping our own proof-of-concept (POC) exploit.
Potential Impact
Our team was able to demonstrate the impact and severity of this issue by running our POC exploit to:
- Bypass authentication
- Upload a backdoored JDBC database driver to gain code execution
- Use the REST API to trigger commands to registered agents to ultimately push the recently leaked Lockbit 3.0 ransomware to all downstream endpoints
Our research identified upwards of 5,000 exposed server manager backup instances via Shodan—all of which had the potential to be exploited by threat actors, along with their registered hosts. Considering that Server Backup Manager SE is predominately used by Hosting and Managed Service Providers who specialize in outsourced IT services for many businesses, this vulnerability has the potential to impact significantly more than 5,000 SMBs.
At the time of this writing, there has been no evidence of exploitation in the wild discovered by Huntress or ConnectWise.
We’ve been working closely with the ConnectWise team throughout this process–their teams have been responsive and collaborative and deserve credit for taking the appropriate steps as we’ve engaged with them.
This vulnerability serves as another reminder of the importance of vendors and community members working together to respond to security issues—and to act quickly before bad actors take advantage of known exploits.
Our Findings
Huntress security researchers John Hammond and Caleb Stewart conducted the work to recreate the proof-of-concept and explore its impact. We were able to retrieve a free trial of the R1Soft Server Backup Manager and test locally within our own lab environment.
The screenshots that @frycos shared online displayed different HTTP requests and responses present in Burpsuite, which was not as telling as a full writeup and detailed explanation. We had to do a bit of experimentation and trial-and-error to fully achieve the authentication bypass, and within a few hours we had successfully recreated the POC.
With this authentication bypass, it was clear from the upstream ZK vulnerability disclosure that we could reach sensitive internal server files like /WEB-INF/web.xml and zk.xml. While this was one accomplishment, it wasn’t a weaponized exploit, and we were curious if more damage could be done.
From the initial vulnerability, we could seemingly leverage the internal forward to perform multiple HTTP POST requests, but only POST requests. This limited the access we had as an authenticated user—but we could still chain together some creative tricks.
It should be noted that ZK is a peculiar framework, so “traditional” HTTP requests that one might think to perform with a classic Cross-Site Scripting attack wouldn’t work. ZK handles all processing server-side, retrieving events each time you click on an individual button or widget, and every time you edit an input field. Every single element on every single page is attached a unique identifier, and all actions related to it are sent to the server sequentially.
That meant a stronger exploit would need a bit of hand-holding, simulating each action with another request and sort of “swinging” from one widget or page identifier to another.
After a night of tinkering, we had a functioning primitive to freely explore the R1Soft application, taking advantage of the authentication bypass vulnerability. We then looked to see what we might latch onto to push the envelope and develop this proof-of-concept further.
At this point, we deviated from the ZK vulnerability itself and were leveraging features and functionality of the R1Soft application itself. ZK opened the door with the authentication bypass—but now, as an admin user, we had the same power and privileges as any system administrator orchestrating their backup solution with R1Soft specifically.
We went down rabbit hole after rabbit hole, decompiling Java classes and digging through code, trying different routes or attack chain avenues. Ultimately, we were most successful via the simple feature of adding your own database driver.
Server Backup Manager SE offers the administrator the ability to provide their own Java Database Connectivity (JDBC) database driver. The provided JAR file is uploaded to the server’s temporary directory, and then added to the class path to be invoked. With some tricks, this could be leveraged to gain code execution!
We did our due diligence and research, and quickly found an online primitive for a backdoored JDBC driver (huge thanks for your work, Airman604!). Chaining this together with our HTTP traversal code, we could automate uploading a database driver and trigger arbitrary code. The Server Backup Manager SE service runs as root, and with that, we had fully compromised the backup server.
Additionally, as the administrator user, we could view system info including operating system/kernel version, hardware specifications and even download “Logs for Tech Support” which included server configuration files, logs, the product subscription key, credential keystores and server private keys.
Throughout this effort, we explored the ZK REST API and grew familiar with other aspects of the software suite that could be manipulated (including a sweet Konami cheat code easter egg 😉). Once again chaining these features together, we developed a proof-of-concept to run commands and arbitrary code on not only the backup server itself, but also all of the registered hosts and devices managed by the Server Backup Manager.
Stronger Together
Immediately following our development of the remote code execution POC, we reached out to ConnectWise to express that this vulnerability has more significant ramifications than initially believed (special thanks to a community champion and our great friend Jason Slagle of CNWR who helped coordinate communication and connect us with the right people at ConnectWise).
We met with the ConnectWise team on October 20 and demonstrated our proof-of-concept, explaining how things fit together. We asked about the potential for a patch and how we might be able to collaborate to help improve the security landscape through this together.
ConnectWise guaranteed that a patch was in the works, as they have been communicating with ZK personally for a tailored security update for Java 7. The official ConnectWise security advisory explains that ConnectWise Recover SBMs have been automatically updated to the latest version of Recover version (v2.9.9). R1Soft specific instances must manually upgrade the server backup manager to SBM v6.16.4 with reference to the R1Soft upgrade wiki and official release notes. Huntress has confirmed the SBM v6.16.4 patch security release successfully remediates our own proof-of-concept exploit in our local testing environment.
Huntress reached out to and worked with internal partners that were willing to assist in the testing and validation of our proof-of-concept. This effort was treated with sincere caution and care and we are very grateful for each partner's gracious permission, transparency, and generosity.
Indicators of Compromise
Again, at the time of this writing, there has been no evidence of exploitation in the wild discovered by Huntress or ConnectWise. While developing a proof of concept, Huntress did not identify any indicators for the initial bypass of authentication. However, the following indicators were observed relating to the malicious JDBC driver and downstream agent execution:
- The server log file (/usr/sbin/r1soft/log/server.log) indicates that a new database driver was uploaded as well as commands being sent to the individual agents. Note: the commands that are executed are not logged. Additionally, while running our exploit we also observed a Java exception and traceback in the logs. The following log excerpt includes the exception message, but excludes the traceback.
- The agent logs (C:\Program Files\Server Backup Agent\log\cdp.log) indicate that commands were received from the server. Note that unfortunately this may not be an explicit indicator of compromise as this could be normal behavior.
- To note, the Server Backup Manager SE server does not have HTTP request logging by default... but if the maintainer has a proxy in front of the server that is logging requests and captures requested URLs, you might see activity repeatedly posting to /zkau/upload and/or a nextURI query parameter (which should never be used). This triggers the initial ZK vulnerability.
It is important to note that the upstream ZK vulnerability not only affects R1Soft, but also anyapplication utilizing an unpatched version of the ZK framework. The access an attacker can gain by using this authentication bypass vulnerability is specific to the application being exploited, however there is serious potential for other applications to be affected in a similar way to R1Soft Server Backup Manager. Huntress is working closely with our DIVD partners to continue the larger hunt and help secure other ZK applications that are at risk.
Complete Timeline
- May 2022 - Markus Wulftange of Code White GmbH discloses ZK auth bypass
- July 2022 - Florian Hauser of Code White GmbH attempts to report to ConnectWise
- 14 October 2022 - Florian Hauser teases initial authentication bypass screenshots on Twitter
- 16 October 2022 - Huntress security researchers recreate the authentication bypass proof-of-concept
- 17 October 2022 - Huntress security researchers expand the exploit to gain remote code execution
- 18 October 2022 - Jason Slagle introduces Huntress security researchers to ConnectWise via email
- 20 October 2022 - Huntress security researchers meet with ConnectWise to demonstrate RCE
- 28 October 2022 - ConnectWise releases security advisory and patch
- 31 October 2022 - Huntress writeup is released
