GHSA-hpcc-26xq-25fv: Netty: Memory Exhaustion via HTTP/3 Reserved Frame Types

Summary Netty's Http3FrameCodec buffers incoming data for HTTP/3 reserved frame types up to the specified payload length without any limits. The payload length is read directly from the wire and trusted without validation. A bad actor can send a reserved frame with a payload length of up to Integer.MAX_VALUE, causing the server to buffer the data in memory. This leads to an OOM and a gradual Denial of Service due to memory exhaustion as multiple streams are opened. Details io.netty.handler.codec.http3.Http3FrameCodec#decodeFrame handles reserved frame types as follows: // Handling reserved frame types // https://tools.ietf.org/html/draft-ietf-quic-http-32#section-7.2.8 if (in.readableBytes() < payLoadLength) { return 0; } The payLoadLength is read directly from the wire and trusted implicitly. Since payLoadLength can be up to Integer.MAX_VALUE and there is no maximum payload length enforcement for reserved frames, the decoder will accumulate bytes in memory until the wire-provided length is reached. This allows a bad actor to exhaust server memory by opening multiple QUIC streams and sending reserved frames with large payload lengths, followed by a small amount of data (e.g., up to the defined limit) on each stream. PoC @Test public void test() throws Exception { EventLoopGroup group = new MultiThreadIoEventLoopGroup(1, NioIoHandler.newFactory()); try { X509Bundle cert = new CertificateBuilder() .subject("cn=localhost") .setIsCertificateAuthority(true) .buildSelfSigned(); QuicSslContext serverContext = QuicSslContextBuilder.forServer(cert.toTempPrivateKeyPem(), null, cert.toTempCertChainPem()) .applicationProtocols(Http3.supportedApplicationProtocols()) .build(); CountDownLatch serverConnectionClosed = new CountDownLatch(1); ChannelHandler serverCodec = Http3.newQuicServerCodecBuilder() .sslContext(serverContext) .maxIdleTimeout(5000, TimeUnit.MILLISECONDS) .initialMaxData(10_000_000) .initialMaxStreamDataBidirectionalLocal(1_000_000) .initialMaxStreamDataB