Skip to content

fix(http): cap decompressed body to MaxBodySize to block gzip bombs#886

Open
Shinku-Chen wants to merge 1 commit into
gocolly:masterfrom
Shinku-Chen:fix(http)-cap-decompressed-body-to-MaxBodySize-to-block-gzip-bombs
Open

fix(http): cap decompressed body to MaxBodySize to block gzip bombs#886
Shinku-Chen wants to merge 1 commit into
gocolly:masterfrom
Shinku-Chen:fix(http)-cap-decompressed-body-to-MaxBodySize-to-block-gzip-bombs

Conversation

@Shinku-Chen

Copy link
Copy Markdown
Contributor

The body-size limit applied io.LimitReader to the raw response body
before gzip.NewReader, so MaxBodySize only bounded the compressed
bytes pulled from the network. A gzip-bombed response — small
compressed payload (KBs) that decompresses to many GBs of repeated
bytes — passed through the limit untouched: the decompressed
io.ReadAll then exhausted memory. This is a classic DoS shape when
scraping untrusted hosts.

Move LimitReader to after the gzip wrap so it caps the decompressed
stream. Non-gzip responses are still bounded because bodyReader
remains res.Body in that path. For gzip responses, the gzip reader
stops pulling more compressed bytes once the LimitReader sees
MaxBodySize decompressed bytes, so both memory and bandwidth remain
bounded.

Add TestMaxBodySizeCapsDecompressedGzipBody: serve ~1 KiB of
gzip-compressed zeros that expand to 1 MiB, set MaxBodySize to 4 KiB,
and assert the body received by OnResponse is at most 4 KiB. The
test reports "body length 1048576 exceeds MaxBodySize 4096" on the
previous implementation and passes with the fix. DisableCompression
on the test transport prevents net/http from gunzipping internally
so the colly gzip path is actually exercised.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant