Written by Phill Moore with contributions from Zach Stanford and Ross Brittain February 15
2023 was not a great year for Citrix NetScaler products. 17 vulnerabilities were posted, four of which have a Common Vulnerability Scoring System (CVSS) score above nine.
This increase of vulnerabilities has resulted in CyberCX clients asking the question – is our NetScaler compromised?
CyberCX’s Digital Forensics and Incident Response (DFIR) team reviewed dozens of Citrix NetScaler devices in 2023 and noticed a similar theme. Some of the NetScaler devices had been operating in a compromised state for months, if not years with no obvious signs of impairment. Some of these compromises had been dormant for years but were high-risk and could have facilitated a significant cyber criminal or nation-state attack in the future.
Off the back of CVE-2023-3519, which allowed an unauthenticated attacker to execute code on an unpatched NetScaler, Mandiant produced an indicator of compromise scanner which does a ‘best effort job at identifying existing compromise’. Thankfully, due to the production of this script, organisations can quickly scan their devices and may find that they were compromised but not always in relation to the current NetScaler CVEs.
The main lesson to take away from this is that these devices have been vulnerable to attack for a very long time. If you look at the number of CVEs against edge/border devices more broadly, it quickly becomes apparent that threat actors can target these devices which may be misconfigured, out-of-warranty, unpatched, or otherwise vulnerable as a way of getting inside the gates. Unfortunately, a lot of the time these devices may sit there for years, still serving their intended purpose, and still be vulnerable.
Make sure to check out the bottom of this post to understand how CyberCX can assist you in the never-ending list of tasks to reduce your attack surface, including vulnerability management, and identifying, securing, investigating and remediating compromised devices.
While this blog post primarily covers long-compromised NetScalers, we have also included a case study for a NetScaler device that was compromised due to more recently disclosed vulnerabilities.
CVE-2019-19781 → Mass exploitation for cryptocurrency mining and backdoors
A proof-of-concept (PoC) script demonstrating CVE-2019-19781 was released in January 2020, which put vulnerable Citrix NetScaler devices immediately at risk. This vulnerability was used notably to deploy webshells or backdoors in multiple organisations and evidence of the PoC script was commonly seen across the NetScalers we examined.
During a review of NetScalers used by an organisation CyberCX identified that this vulnerability had been used on three NetScalers to configure a scheduled task, otherwise called a “cron” job, under the nobody account in January 2020. The cron job ran every minute and performed a curl request to remote IPs to download a shell script. At the time of deployment, these IPs likely hosted Monero coin miners, however as threat actors exploited vulnerable devices across the Internet, this likely did not last very long. Therefore, even though the cron job was still running, it was not downloading anything.
More interestingly, one of the NetScaler devices was not exhibiting this same behaviour and instead we found evidence that the NOTROBIN backdoor had been placed on this device. This was of significant concern as this malware provides backdoor access to the NetScaler, rather than a coin miner which poses less risk to the organization.
The httpd process name was chosen by the threat actor as there is a legitimate httpd process that is expected to be run from a separate location. Execution from the hidden .nscache directory is a clear indicator of compromise in this case and this folder specifically has been seen in other unrelated compromises.
Thankfully in this case, there was no evidence that the malware was actually running on the device, even though it should have.
CVE-2019-19781→TOR!
A similar investigation identified initial exploitation of the CVE-2019-19781 vulnerability, however resulted in the deployment of a webshell in March 2020.
The webshell was PHP-based and the name was a fairly long and randomly generated string, which we assess to be an identifier. This file was placed in the \var\vpn\theme directory, and an additional cron job was also configured.
A review of the \var\nstmp\.nscache directory provided us with something much more interesting.
This showed a number of ELF binaries, shell scripts, and configuration files. Interestingly some of the dates recorded against these files suggested that they were created a month prior to the webshell, however our assessment is this is due to the method of data transfer – there was a direct correlation between this set of files and the webshell; the name of the webshell was recorded within the ident file!
Quite a bit of the malware was available on VirusTotal, and by selecting “Re-Analyze” we could take advantage of the new Code Insights feature to summarise the shell scripts that had previously been uploaded. For example, the httpd-nscache_clean script
A thorough review of the available files identified that this set of malware existed on the system for approximately a year, however we had limited evidence to identify what it was used for, and we can say with confidence that at the time of analysis it had not been functional for the years prior.
CVE-2023-3519 → Credential Harvesting
Proactive scanning by a client identified historic compromise that had been remediated, however further investigation identified the same device was exploited more recently using CVE-2023-3519, and the threat actor made modifications to an existing JavaScript file to capture usernames and credentials as they were entered. This is an example of a more advanced tactic, more likely associated with Initial Access Brokers (IAB) and nation state adversaries.
Unfortunately, this is an example of a compromise that the existing CVE hunting script didn’t find – this can be supplemented with file integrity monitoring or custom detections for modifications to files on the device.
The threat actor added additional lines to the following file, which is imported in the login page of the NetScaler Gateway. This was identified by noticing a discrepancy in the modification time of the file.
\var\netscaler\logon\LogonPoint\plugins\ns-gateway\nsg-setclient.js
Default:
Post Modification:
As well as modification to the JavaScript, the threat actor created a credential processor file \vpn\themes\imgs\netscaler.php which received the POST request made above and wrote the information to a file called netscaler.1.
The contents of \vpn\themes\imgs\netscaler.php can be seen below
As the victim users of the NetScaler continued to login, their credentials were saved to the netscaler.1 file.
Using the httpaccess-vpn.log from the NetScaler device we were able to track when the threat actor retrieved the credential log from the device.
The threat actor used the stolen credentials to login to the NetScaler and establish an SSLVPN connection.
Exploited NetScaler to get DA
In this investigation the threat actor was able to gain access to the Citrix environment with legitimate credentials which allowed them to compromise the NetScaler. Unfortunately, the environment itself was quite old and susceptible to multiple vulnerabilities. The TA created (and then deleted) a webshell, which allowed them to extract the contents of the ns.conf file. The only evidence of this was found within a HTTP web access log, however the GET request came from a previously identified malicious IP address:
From here they were able to obtain a service account credential, which unfortunately was a Domain Administrator, as well as a list of available domain controllers to laterally move to. By getting access to the ns.conf, an attacker can easily run a Metasploit module that will extract all stored credentials, part of which was contributed by CyberCX NZ’s own Principal Security Consultant Daniel Jensen. As defenders, understanding that these credentials are easily decrypted allows us to understand how critical it is to rotate these credentials after an attack and hunt for signs of misuse within the environment.
Thankfully the client had implemented some defensive mechanisms in the form of Canary devices which alerted them to the unauthorised scanning activity. Honeypots such as those built by ThinkstCanary are a fantastic way of detecting an adversary in your network, and we have recommended deploying them on networks in the past that would be of high value to certain APT groups.
We understand that organisations may be required to support legacy technology, however this client showed that they were still able to add defensive technologies to allow for early detection.
One notification, two compromises
An external notification led to a client investigating their NetScaler device, which they had been advised was compromised. During our analysis we identified multiple webshells related to two separate breaches across two months.
Multiple “China Chopper” webshells with randomised names had been placed in the \vpn\theme directory in early August 2023. These webshells all followed a similar format and provided the threat actor with the ability to evaluate additional PHP code.
Although access logging did not extend to this time, error logs on the NetScaler indicated immediate attempts to execute commands failed due to PHP syntax errors in the content provided to the webshells.
The secondary compromise of the device was then performed in September 2023 the following webshell was placed in the \netscaler\ns_gui\vpn directory:
Within five minutes of this webshell being created, a malicious JavaScript ‘include’ tag was appended to the end of the system’s logon page, \netscaler\ns_gui\vpn\index.html. This tag retrieved a remote JavaScript resource which would result in any credentials entered into the logon page also being sent to a remote website. The observed evidence aligns with the campaign documented by IBM X-Force where these modifications were made to vulnerable NetScaler instances around the globe.
Approximately two weeks after the second breach, further reconnaissance including an attempt to log into the NetScaler using a small subset of the accounts exposed via the malicious JavaScript “include” was observed originating from a known TOR exit node. Based on the client, this increased our confidence that this was a nation state adversary, however we had no indication that they were successful.
Lessons learned
Unfortunately securing edge devices is hard, and another task on top of the generally overburdened IT operations and security teams. These devices typically run for long periods of time without reboots, they have a large attack surface, and they can’t typically run endpoint monitoring.
Ideally, we would have appropriate network monitoring (for example by putting the edge device behind a third-party Web Application Firewall), ensure all required patches and file integrity monitoring is configured, and periodically review the device file system for signs of compromise.
Patching
It’s critical to ensure that the device is kept up to date, and one can’t assume that once a patch has been applied that the device hasn’t previously been exploited. Sometimes a patch will remediate any previous exploitation, but other times the persistence will remain, not to mention if credentials were stolen then they can still be re-used or sold. The only way to know for sure is to have a look at the device and check for signs of compromise – and when in doubt, well, reset everything…which isn’t ideal.
A device that is no longer supported by the vendor will likely continue working for years into the future, however that doesn’t mean it’s safe to use!
Enable File Integrity Monitoring
FIM can be setup on a NetScaler, however, while useful, this may not identify all malicious modifications to the system. In our experience, FIM is available on other types of appliances however may be easily bypassed or provide a false sense of security. That being said, if it can be enabled with minimal impact and possibly increase your ability to detect a malicious modification it should be considered.
Unfortunately, NetScaler do not publish hashes for the individual application files to aid identification of manipulated files.
Data collection and triage
Collecting data from a NetScaler is not the simplest process. The device itself may be a physical or virtual appliance and has two primary components – a RAM disk that is flushed on reboot, and persistent storage. During an update, the device may be rebooted and that will remove any evidence that solely exists in the RAM disk. This may however remediate a previous compromise, which is a double-edged sword as it does not afford the business the knowledge that their system had been compromised.
Data collection can be performed using UAC for triage or a DD command for a full collection, however limited triage can also be performed live and Manuel Winkel at Deyda Consulting has produced some excellent guides for CVE-2023-3519 and CVE-2023-4966. Additionally, many of the NetScalers that came through to our team were identified because of Mandiant’s scanner script, which is going to be a good first step to identifying whether there might be something malicious on the system. Unfortunately, while an excellent script, in our examinations we have found that sometimes a scan will return no findings and may still have been compromised. An automated scanner may not replace the need for a skilled forensic analyst.
Another great way to triage a system has been documented by Nextron-Systems about how to use their free and paid toolkits.
Log forwarding
Unfortunately, the logging on NetScalers can be fairly short lived, and often lacks the depth to fully identify the vulnerability exploited to compromise the device. Only the ns.log file can easily be forwarded to a SIEM, which will at least cover some parts of an investigation – and without forwarding this log, exploitation of things like CitrixBleed will be difficult to identify due to log rotation. The device logs themselves will need to be collected from the device and these can show everything from webshell interaction to actual commands that were run.
If all else fails, hopefully the appliance is regularly backed up, however this approach is very cumbersome if analysis is required.
Conclusion
When in doubt, always adopt a defence in depth strategy – your perimeter is under attack, and while you can’t defend on all fronts, proactively identifying vulnerabilities or compromises can help prevent incidents in your network. While some of these incidents appeared to be harmless, we did find backdoors into environments that could be exploited years down the track. In some cases, these exploits could have been conducted by initial access brokers seeking to leverage and sell their access into a network and in others they could have been nation states gaining access into a key target.
Proactively reviewing the available vulnerabilities in your border devices and identifying whether there are signs of existing compromise can be a major increase to your security.
If you suspect your network device is compromised or has previously been compromised, ensure it is investigated in order to identify any potential risks before rebuilding the device. In the age of n-day vulnerabilities and delayed patches, CyberCX recommends rebuilding your devices from scratch when a critical exploit is discovered after your investigation is complete.
If you want to know how CyberCX can assist you, reach out at the links below.
