Skip to content

fix(clone): give cloned collector its own redirect handler#904

Open
Noethix55555 wants to merge 1 commit into
gocolly:masterfrom
Noethix55555:fix/clone-redirect-handler
Open

fix(clone): give cloned collector its own redirect handler#904
Noethix55555 wants to merge 1 commit into
gocolly:masterfrom
Noethix55555:fix/clone-redirect-handler

Conversation

@Noethix55555

Copy link
Copy Markdown

Fixes #875

Clone shared the parent's httpBackend, which holds a single *http.Client. That client's CheckRedirect closure is created in Init and bound to the parent collector. Consequently, when a clone followed a redirect, the redirect check ran against the parent's AllowURLRevisit, AllowedDomains/DisallowedDomains, DisallowedURLFilters and store instead of the clone's.

This gives the clone its own httpBackend and http.Client that reuse the parent's Transport, cookie jar, timeout and limit rules (so connection pooling and cookies still behave as documented), but rebinds CheckRedirect to the clone via clone.checkRedirectFunc().

Regression test: parent has no filters; clone sets DisallowedURLFilters to block /blocked; server redirects /a -> 302 -> /blocked. The clone must not follow the redirect. The test fails on master (the clone reaches /blocked) and passes with the fix. Full suite and -race pass.

Clone shared the parent's httpBackend, whose http.Client CheckRedirect
closure was bound to the parent collector. A clone's AllowURLRevisit,
filters and store were therefore ignored when following redirects.

Give the clone its own httpBackend and http.Client that reuse the
parent's Transport, cookie jar, timeout and limit rules, but rebind
CheckRedirect to the clone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cloned Collectors inherit checkRedirectFunc (so it uses the parent's AllowURLRevisit)

1 participant