Racing and Fuzzing HTTP/3: Open-sourcing QuicDraw(H3)

QuicDraw evaluation – can we race HTTP/3 servers?

To evaluate QuicDraw performance against other tools, we used the following setup (see the figure 9 below):

• We set up a Keycloak (version 23) system with a known race condition on an AWS EC2 machine.
  • Note: Keycloak is an open-source identity and access management solution, currently a CNCF project.
  • The Keycloak instance used in our tests includes a known race condition discovered in previous CyberArk Labs research. This issue is fixed in current versions.
• We used AWS CloudFront because it gives us a real-world HTTP/3 server with a relatively easy setup process.

Figure 9 – Our test setup: using QuicDraw on Keycloak behind CloudFront (configured to serve HTTP/3)

QuicDraw performance vs other tools

We used the above test setup and ran five different tools from the same machine (using the same internet connectivity).

In our tests, we used curl. Although curl is not built specifically for race conditions testing, it can support HTTP/3 (special build), and it is highly optimized; nevertheless, we use it as a benchmark indicating a “baseline” to which we can compare.

We’ll start with some background for the test:

• In Keycloak, an admin can create an API Token for developers to use on the system.
• On creation, an admin can limit the number of clients that can be created by the developer (API Token).
• The developer uses this API Token to create clients via the Keycloak API.
• When using the API Token, the token’s “remaining count” is updated.

Figure 10 – Keycloak – admin sets the number of clients that the token can create.

For our tests, we set the count limit to one on all tokens, meaning each number below zero causes a race condition on the server side.

Below, we detail each tool and the essential parameters used for each tool’s execution.

Tool no. 1: racing using curl loop over HTTP/2

For this test, we used the static version of curl and commands similar to the following:

for i in {1..220};
do
/static-curl/curl -k -d ‘{“clientId”:”curl\_client\_’$i'”}’ -H ‘Authorization:…’ “https://keycloak23.cloudfront.net/path/” –http2 &
done

Tool no. 2: racing using curl loop over HTTP/3

For this test, we used the static version of curl (with HTTP/3 support) and commands similar to the following:

for i in {1..220};
do
/static-curl/curl -k -d ‘{“clientId”:”curl\_client\_’$i'”}’ -H ‘Authorization:…’ https://keycloak23.cloudfront.net/path/ –http3-only &
done

Tool no. 3: racing using QuicDraw (v0.8.0) over HTTP/3

For this test, we used commands similar to the following:

quicdraw_v08 “https://keycloak23.cloudfront.net/path/” -l /ssl_key_log_file.log -d ‘{“clientId”:”QuicDraw\_POC\_v08\_\_\_”}’ -H ‘Authorization: bearer …’

In this version (v0.8), we set the script to issue 220 requests.

In our specific setup, 117 requests were sent within a single (Quic-Fin-Sync) QUIC packet. (see Figure 11 below)

Figure 11 – QuicDraw in action (evaluation) with 117 (DATA) streams sent on a single QUIC packet

On (QUIC) stream limit (MAX_STREAMS)

In QUIC, set the maximum number of streams that can be “online” (sent with no response) between the client and the server via the MAX_STREAMS frame.

In our client (QuicDraw), we sent the stream limit (MAX_STREAMS frame) to 128 (see figure 12 below)

Figure 12 – QuicDraw sets the stream limit (MAX_STREAMS frame) to 128.

Tool no. 4: racing using Turbo Intruder (HTTP/2)

As of the writing of this paper, Burp Suite does not support HTTP/3.

For this test, we set the for loop within Burp Turbo Intruder (a slightly modified version of race-single-packet-attack.py script to modify the payload sent in each request) to 110 requests. Since larger values (220) got our burp to hang and not respond, we decided to run this test with a lower value (110).

Figure 13 – Traffic inspected using Turbo Intruder,30 streams sent over a single-packet.

As seen in the figure 13 above, 30 requests are sent in a single-packet (this limit is also mentioned in PortSwigger’s paper).

Tool no. 5: racing using Burp Intruder (HTTP/2)

We set the number of payloads to 220 requests for this test. According to our tests, Burp Intruder does not use the single-packet mechanism.

In the next section, we dive into our racing (tests) results.

Evaluation results – QuicDraw racing the Keycloak

Below are the results of our testing:

Figure 14 – Results racing vs our test setup using QuicDraw, curl, Burp Intruder, and Turbo Intruder.

As seen in our test results (Figure 14), we set a count limit to one on all tokens. This means each number below zero caused a race condition on the server side. Moreover, the lower the number, the more effective the tool at exploiting the race condition on the server side.

We used the above command/tool to evaluate its performance, testing each tool three times with different (Keycloak initial access) tokens.

Key insights from our tests:

• Using tool no. 1, curl (HTTP/2) in a for loop gave us the following results: (-22, -2, -13).
  • Not using the single-packet mechanism.
• Using tool no. 2, curl (HTTP/3) in a for loop gave us the following results: (-1, 0, -6).
  • Not using the single-packet mechanism.
• Using tool no. 3, QuicDraw (v0.8) gave us the following results: (-66, -38, -79).
  • Using our implementation of Quic-Fin-Sync on HTTP/3.
• Using tool no. 4, Burp Turbo-Intruder (slightly modified) race-single-packet-attack.py (HTTP/2) gave us the following results: (-5, -4, -1).
  • Using the single-packet mechanism behind the scenes.
• Using tool no. 5, Burp Intruder’s (HTTP/2) gave us the following results: (-6, -6, -6).
  • This one is used as a benchmark without using the single-packet mechanism.

We are aware that this setup cannot be taken as scientific results. Moreover, some of the tools (Burp Intruder and curl) are not explicitly designed to exploit race conditions, making these results “not fair.” (Yes, we were also surprised by the efficiency and results returned with curl).

However, we do suggest treating the Burp Intruder and curl results as a benchmark, stating “the best result without a specialized tool” We also think that our tool’s results (QuicDraw(H3)), which performed significantly better than the non-specialized tools tested, suggest it is very effective in triggering race conditions.

As seen above, when using Turbo Intruder, 30 requests are sent in a single-packet. While using QuicDraw, 117 requests were sent within a single (Quic-Fin-Sync) packet. This, together with the fact that QuicDraw outperformed Turbo Intruder (by a factor of “x6”) suggests that when HTTP/3 is enabled, you should try using QuicDraw to fuzz and exploit race conditions.

QuicDraw usage and demos

In the following section, we mention how to use QuicDraw for racing (exploiting race conditions) and fuzzing (multiple different URLs or POST data payloads) HTTP/3 servers. For complete documentation, consult the QuicDraw(H3) repository.

Racing HTTP/3 applications

To use the same request multiple times (using the Quic-Fin-Sync), use the -tr/–total-requests argument.

quicdraw -tr TOTAL_REQUESTS
# example
quicdraw https://www.cyberark.com/ -tr 7

Demo: QuicDraw racing demo

Fuzzing HTTP/3 applications

Fuzzing in QuicDraw is based on a simple concept, just like other web fuzzers (Ffuf, Wfuzz). Go over the data section (-d) and replace any reference to the FUZZ keyword with the value given in the wordlist (-w) as the payload.

quicdraw -w WORDLIST -d DATA_with_FUZZ_keyword
quicdraw <https://http3_server.com/path> -w path/to/wordlist -d ‘{“jsonkey”:”FUZZ”}’

Known limitations

The MAX_STREAMS parameter is a limit set by the QUIC implementations (client and server). It can limit our or other “racing” implementations.