CVE-2022-27510, CVE-2022-27518: Citrix ADC and Internet Gateway version assumption measurement (2023)

written byyun zhenghu

Two critical vulnerabilities were recently reported in Citrix ADC and Citrix Gateway; where one of them was exploited by a threat actor in the wild. Because these vulnerabilities are remotely exploitable and the situation is historicalCitrix vulnerabilitiesRIFT began investigating how to identify the exact version of the Citrix ADC and Gateway servers on the Internet so that we can let customers know if they still haven't fixed the issue.

Accurate version information is useful in determining if a server is still vulnerable to specific vulnerabilities. We also use version information to obtain version statistics and version assumptions over time.

In the first part of this blog, we explained how we collect and analyze Citrix ADC disk images to identify the version. Below we explain how we found a way to determine the version build date and how we use the build dates to download the missing Citrix ADC images to help in our version identification quest. The last part covers version statistics for Citrix ADC servers on the Internet and how we use them to measure version adoption on the Internet.

Skip to the bottom if you are more interested in the statistics than the technical details of Citrix ADC version identification.

CVE-2022-27510: Unauthorized access to gateway user resources

On November 8, 2022, Citrix released aSecurity Bulletin for CVE-2022-27510, a critical authentication bypass vulnerability that affects Citrix ADC (formerly known as NetScaler) and Citrix Gateway. To take advantage of this, the server must be configured as a gateway (SSL VPN, ICA Proxy, CVPN, RDP Proxy).

CVE-2022-27518: Unauthenticated remote execution of arbitrary code

Less than a month later, on December 13, 2022, the National Security Agency (NSA) released aCyber ​​Security Consultingthat APT5 is actively exploiting Citrix ADC servers. However, this advisory does not mention a specific CVE abuse, but rather a potential new vulnerability, as Citrix released one on the same day.how-to blogand a new onesecurity bulletinCVE-2022-2751-Details8, which is a new vulnerability and should not be confused with CVE-2022-27510. For this to be taken advantage of, the Citrix ADC or Gateway server must be configured as a SAML service provider or SAML identity provider.

Locating Citrix ADC and Gateway servers on the Internet

Citrix ADC and Gateway servers are generally Internet facing due to the nature of the device. For example, services like Shodan and Censys regularly scan the Internet and identify these servers. Using this information, we can create a list of Citrix ADC and Citrix Gateway servers with SSL VPN/Gateway service that are exposed to the Internet. We used this to create an initial server list and found around 28,000 servers on the internet as of November 11, 2022.

version identification

Unfortunately, exact version information is not available in the HTTP response from a Citrix ADC or Gateway server. However, we notice that there is an MD5 hash value in the HTTP body when we request it./vpn/index.htmlURL:

Here we see the parameter?v=6e7b2de88609868eeda0b1baf1d34a7eattached to various resource URLs. We extracted these hashes from Censys scan data to create a list of the most common version hashes. We found around 100 hashes of a single version.

To see if we can map the version hash to exact versions, we first start our own Citrix ADC server to examine if this version hash is static or generated in this HTML page.

new market

There are more server appliances available in the cloud, and Citrix ADC is no exception. You can easily find it on your favorite cloud marketplace. Gone is the slow process of downloading images and starting a virtual machine to install the application; We can do it directly in the cloud! We use the Google Cloud Marketplace, but it is also available on AWS and Azure.

After deploying Citrix ADC from the Cloud Marketplace with a single click, let's log in to the shell and browse the file system to see if we can find them.index.htmlpage and whether it contains a hash value.

SSH into the VM and see the version, looks like we're at version13,1 Bau 33,47:

yun@cloudshell:~ (rift-citrix-362712)$ gcloud ssh citrix-adc-vpx-instance############################ # # # ########################################### ## # # ## ## WARNING: Access to this system is restricted to authorized users only. ## Log out IMMEDIATELY if you are not an authorized user! ############################################# # # ## ########################## Done > View NetScaler NS13.1 Version: Build 33.47.nc, Date: 23 Sep 2022 13:12: 49 (64-bit) Done

modelshellto enter the shell and we will find them allindex.htmlRecords:

> shellroot@ns# uname -aFreeBSD ns 11.4-NETSCALER-13.1 FreeBSD 11.4-NETSCALER-13.1 #0 e5f9d90507ab(heads/artesa_33_47)-dirty: sex set 23 13:13:05 PDT 2022 root@sjc-bld-bsd114-228 :/usr/obj/usr/home/build/adc/usr.src/sys/NS64 amd64root@ns# find / -name 'index.html'/netscaler/ns_gui/admin_ui/gui_v2/swagger_ui/index.html/netscaler /ns_gui/vpn/index.html/var/netscaler/gui/vpn/index.html/var/netscaler/gui/admin_ui/gui_v2/swagger_ui/index.html/var/netscaler/logon/LogonPoint/index.html/var /python/lib/python3.7/site-packages/djangorestframework-3.11.0-py3.7.egg/rest_framework/templates/rest_framework/docs/index.html/var/python/lib/python3.7/site-packages /Django-3.0.5-py3.7.egg/django/contrib/admin/templates/admin/index.html/var/python/lib/python3.7/site-packages/Django-3.0.5-py3.7 .egg/django/contrib/admindocs/templates/admin_doc/index.html

It seems that there are several directories withindex.html, Let's check/vpn/index.html:

root@ns# head /netscaler/ns_gui/vpn/index.html<!DOCTYPE html PÚBLICO "-//W3C//DTD XDEV_HTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/ DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="X-UA-Compatible" content="IE= edge"><title>Citrix Gateway</title><link rel="ICONO DE ACCESO DIRECTO" href="/vpn/images/AccessGateway.ico" type="image/vnd.microsoft.icon"><META http-equiv= "Content-Type" content="text/html; charset=UTF-8"><META content=noindex,nofollow,noarchive name=robots><link href="/vpn/js/rdx/core/css/rdx. css?v=7afe87a42140b566a2115d1e232fdc07" rel="hoja de estilo" type="text/css"/><link href="/logon/themes/Default/css/base.css?v=7afe87a42140b566a2115d1e232fdc07" rel="hoja de estilo" type=" texto/css" mídia="tela" />

¡Bingo! oindex.htmlThe file contains the hash7afe87a42140b566a2115d1e232fdc07, and when we look up this value in Censys, we get several promising results. So we can assume this version.13.1-33.47is assigned to this hash value.

Let's see what other files contain this hash value:

root@ns# grep -r 7afe87a42140b566a2115d1e232fdc07 /var/netscaler | cortar -d: -f1 | ordenar -u/var/netscaler/gui/epa/epa.html/var/netscaler/gui/epa/errorpage.html/var/netscaler/gui/epa/posterrorpage.html/var/netscaler/gui/vpn/index. html/var/netscaler/gui/vpn/loading.html/var/netscaler/gui/vpn/logout.html/var/netscaler/gui/vpn/tmindex.html/var/netscaler/gui/vpn/tmlogout.html/ var/netscaler/gui/vpns/choices.html/var/netscaler/gui/vpns/f_ndisagent.html/var/netscaler/gui/vpns/f_services1.html/var/netscaler/gui/vpns/f_services_linux.html/var/ netscaler/gui/vpns/j_services.html/var/netscaler/gui/vpns/m_services.html/var/netscaler/gui/vpns/navui/refresh.html/var/netscaler/gui/vpns/nohomepage.html/var/ netscaler/gui/vpns/postepa.html

If you install the app from the Google Cloud Marketplace, you don't have the option to choose a version to install. But of course we want to install other versions to start building a hash list of known versions.

We notice that aBereitstellung.zipThe file can be downloaded from the Google Cloud Marketplace:

after unpackingBereitstellung.zipWe see that it contains Terraform scripts to deploy the application to the cloud and points to the disk image it uses for installation. Fortunately, Citrix also left a list of other disk image names in a Jinja template that points to different Citrix ADC versions.

Can these disk images be downloaded? Yes, it turns out that you can!

Retrieving images from the cloud

we use the followinggcloudCommands to download Citrix ADC disk images:

• create gcloud compute imagesto download the image to our own Google Cloud project. This way, you can select this image when creating a new virtual machine. However, this doesn't make the image directly accessible for reading in other tools, so we need to export it first.
• Exporting gcloud compute images, the provided image is exported to a Google Cloud Storage (GCS) bucket in a different format, such asqvaca2ovmdk.

The raw disk image is 20 GB, but when exported asqvaca2format we can reduce it to just 2 GB.

The following shell script executes both steps in a single script:

#!/bin/sh## Export a Citrix ADC image to a Google Cloud Storage bucket## Usage:# ./export-citrix-image.sh <image name> <bucket_path># Example:# ./ export - citrix-image.sh citrix-adc-vpx-10-standard-13-0-85-19 gs://my-bucketimage="$1"gcs_bucket="$2"gcloud compute images create "$image" -- image source https://www.googleapis.com/compute/v1/projects/citrix-master-project/global/images/$image --verbosity debuggcloud compute images export --destination-uri "$gcs_bucket/$image. qcow2 " - -imagem "$image" --export format qcow2

Now that we've exported some familiar images fromBereitstellung.zipto a GCS warehouse, we can process them in bulk. Of course, archiving and saving can also be useful for further research, especially vulnerable versions, as they tend to be deleted.

This technique can also be used for other devices on the Google Cloud Marketplace.

Dissect to the rescue!

What better way to process disk images en masse than to dogfood our own?Open SourcedissectStructure. While Citrix ADC runs on FreeBSD,dissectcan read yoursUFS/FFSFile systems pretty good. after ushaveaSomemistake that is.

We can also do all in oneshell Cloud,without activating an additional VM.

Let's assemble our GCS cube containing theqvaca2use images firstgcsfuse:

yun@cloudshell:~ (rift-citrix-362712)$ mkdir Bucketyun@cloudshell:~ (rift-citrix-362712)$ gcsfuse my-citrix-adc-bucket Bucket2022/12/11 15:48:25.442402 Iniciar gcsfuse/0.41. 9 (Go versão go1.18.4) para el aplicativo "" usando ponto de montaje: /home/yun/bucket2022/12/11 15:48:25.462744 Abrindo conexão GCS...2022/12/11 15:48:25.557883 Montando System de Arquivos "my-citrix-adc-bucket"...2022/12/11 15:48:25.563799 1 Eimer/Citrix*. qcow2citrix-adc-vpx-10-estándar-13-0-83-27.qcow2citrix-adc-vpx-10-estándar-13-0-87-9.qcow2citrix-adc-vpx-10-estándar-13-0- 88-14.qcow2citrix-adc-vpx-10-standard-13-1-21-50.qcow2citrix-adc-vpx-10-standard-13-1-33-52.qcow2citrix-adc-vpx-byol-13- 0-76-31.qcow2citrix-adc-vpx-byol-13-0-79-64.qcow2citrix-adc-vpx-byol-13-0-82-45.qcow2citrix-adc-vpx-byol-13-0- 88-16.qcow2citrix-adc-vpx-byol-13-1-33-49.qcow2citrix-adc-vpx-byol-13-1-33-52.qcow2citrix-adc-vpx-byol-13-1-33- 54.qcow2citrix-adc-vpx-byol-13-1-37-38.qco w2citrix-adc-vpx-express-13-0-83-29.qcow2...

Now we are going to install the latest version ofdissectusePip3in a virtual environment:

yun@cloudshell:~ (rift-citrix-362712)$ python3 -mvenv dissectyun@cloudshell:~ (rift-citrix-362712)$ source dissect/bin/activate(dissect) yun@cloudshell:~ (rift-citrix-362712) $ pip3 install --pre diseccionar

Dissect installs several command line tools. One of these tools is calledShell-alvo, which can read disk images and file systems in a variety of formats and provides a shell-like interface for browsing the file system.

Now we are going to open a disk image withShell-alvo, we specify or-qCheck to hide some warnings:

Dissect is not supported by Citrix ADC (yet!); Therefore, we have access to recognized file systems. We see two partitions and note that the first is the/ to throwpartition and the second is the/guerraShare.Shell-alvohas a basethinkcommand, but the output can be sent to an external tool likegrep:

citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /> cd fs1citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> encontrar . | grep index.html/fs1/netscaler/gui/vpn/index.html/fs1/netscaler/gui/vpn/tmindex.html/fs1/netscaler/gui/admin_ui/gui_v2/swagger_ui/index.html^C

Let's take the firstindex.htmlFile, Archive (grepis used to limit the output for this example):

citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> cat /fs1/netscaler/gui/vpn/index.html | grep ?v= | head -n1<enlace href="/vpn/js/rdx/core/css/rdx.css?v=c9e95a96410b8f8d4bde6fa31278900f" rel="hoja de estilo" type="text/css"/>citrix-adc-vpx-10-estándar -13-0-83-27.qcow2 /fs1>

Great, we thought for the version13.0-83-27is the hashed versionc9e95a96410b8f8d4bde6fa31278900f.

Not everyone realizes this, but with just this command we can load a FreeBSD imageqvaca2Disk format with differentUFS/FFS2Partitions with Python anddissectin a cloud layer. No more annoying installation of additional tools and tedious steps to mount images. The future is now!

To further automate this, we can use thedissectPython API or we can use a simple oneliner shellgoal fs, which can run some basic commands on a disk image:

It's good to mention that at this point we also started to investigate whether the version hash can be computed using MD5 addition variants of the version string, but to no avail.

Determine the date of construction

After processing our purchased cloud images, we discovered that we still had version hashes from across the web with no known version, so apparently not all versions were listed or not available as a cloud image. We were able to capture some images of clouds by guessing the name of the image, but that wasn't enough to fill in the gaps.

Finally, we turned to the Citrix download page to find other versions we didn't already have. A Citrix account is required, but is open for registration. However, it seems that not all released builds are listed on the Citrix download page, especially older builds that have been superseded by a newer build. we found oneGitHub projectwho removed the Citrix download links that were helpful to our investigation and we found that these links are still valid (after login).

As our list of known versions continues to grow, we are still missing some very common hashes in the data set. Of course, we wanted to know what version it is, and at this point we found an interesting way to get the approximate build date of the Citrix ADC server version.

In the disk image we find the gzip compressed file calledrdx_en.json.gzin betweenvpnSubdirectory:

citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> dagegen. | grep rdx_en.json.gz/fs1/netscaler/gui/vpn/js/rdx/core/lang/rdx_en.json.gz/fs1/netscaler/gui/admin_ui/rdx/core/lang/rdx_en.json.gz^Ccitrix -adc-vpx-10-standard-13-0-83-27.qcow2 /> ls -la /fs1/netscaler/gui/vpn/js/rdx/core/lang | grep rdx_en-rw-r--r-- 1001 513 35 2021-09-27T14:01:20 rdx_en.json.gz

let us runFile, ArchiveCommand in this gzip file:

citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> file /fs1/netscaler/gui/vpn/js/rdx/core/lang/rdx_en.json.gz/fs1/netscaler/ gui/vpn/js/rdx/core/lang/rdx_en.json.gz: data compressed with gzip, it was "rdx_en.json", last modified: Monday Sep 27, 2021 14:01:20 by Unix

Last editionlun 27 sep 14:01:20 2021. Great, we got the timestamp of when this gzip file was created and found that the timestamp is an accurate description of when the version was created/released, very cool! This JSON file appears to be used for translation purposes, but in newer versions, this file is just a gzipped JSON dictionary.

Why thisrdx_en.json.gzThe file is invpnsubdirectory, it can also be downloaded remotely by accessing the following URI/vpn/js/rdx/core/lang/rdx_en.json.gz

We now have a list of version hashes, known versions, and approximate build dates. Using the build data, we can now derive the approximate version from a version hash whose version we don't yet know.

For example, the version hash4434db1ec24dd90750ea176f8eab213cThe version number was still missing and we have already processed all the cloud images and download links available on the Citrix download page. But armed with the knowledge of the date of construction2022-06-29 13:46:08From this version hash, we can deduce that this build date is among the build dates of known versions.12.1-65.15mi12.1-65.25. Explanation table:

Well, the possible versions could be:12.1-65.16Pro12.1-65.24.

These versions are not listed on the download page, but is there a version within this range and is it still downloadable? If we look at the Citrix download links for known releases, we see that the download ID appears to be incremental and the files are in a specific format. For example:

Example URL format: https://downloads.citrix.com/[DOWNLOAD_ID]/build-[VERSION]_nc_64.tgz https://downloads.citrix.com/20651/build-12.1-65.15_nc_64.tgz <-- Known URL https://downloads.citrix.com/20???/build-12.1-65.??_nc_64.tgz <-- list this URL https://downloads.citrix.com/21408/build-12.1-65.25_nc_64 .tgz <-- well-known URL

Can we list the url and download the file? The answer is yes. Using a small python script to list the download link and version, we found that the following URL returns a200 good:

https://downloads.citrix.com/20929/build-12.1-65.17_nc_64.tgz

after downloadbuild-12.1-65.17_nc_64.tgz, we confirm this version12.1-65.17maps to version hash4434db1ec24dd90750ea176f8eab213c. This technique has proven very useful in closing most of the version gaps in our data set, with a few missing ones.

To see our compiled list of version hashes, approximate build dates, and versions, check out this summary:https://gist.github.com/fox-srt/c7eb3cbc6b4bf9bb5a874fa208277e86

version statistics

After assigning the best known version hashes to a version, we can measure how many versions are active on the web and if they are still vulnerable.CVE-2022-27510oCVE-2022-27518.

The following table shows the top 20 active versions on the Internet and also shows whether that version is vulnerable to either of the two most recent CVEs:

We see that most are in the version13,0-88,14, which is not vulnerable to any of the CVEs. The finalist is the version12.1-65.21that is not vulnerableCVE-2022-27510, but it is forCVE-2022-27518. There are also many servers that do not return a version hash, so we cannot identify the exact version of those servers.

Please note that forCVE-2022-27518a SAML prerequisite is required for it to be exploitable. So just knowing the version doesn't fully indicate if the server is exploitable, but it's still a good indicator that it needs to be updated.

The chart below shows the top 9 countries using Citrix ADC/Gateway and how many servers are still vulnerable to the two critical CVEs. In most countries we are seeing an obvious decrease in vulnerable serversCVE-2022-27518following the recent advisory release from the NSA and Citrix.

This chart shows the top 20 countries using Citrix ADC/Gateway and how many servers successfully upgrade to be protected against both CVEs.

Conclution

In this blog, we show how we perform version identification of Citrix ADC and Citrix Gateway servers by analyzing disk images exported from Google Cloud Marketplace.dissect. We also show that gzip files can be useful for timestamp information and how we use them to find and download missing Citrix ADC builds.

Finally we use theversion identification datato measure Internet-facing versions of Citrix ADC and Gateway servers over time and determine thatNSAmiCitrix Consultingreally helped with the updates. However, some servers are still vulnerable toCVE-2022-27510oCVE-2022-27518.

We hope that this blog will provide additional information on these two Citrix CVEs and that our research on version identification will contribute to future studies.