Skip to content

Fix clone redirect revisit checks#882

Open
StantonMatt wants to merge 1 commit into
gocolly:masterfrom
StantonMatt:fix-clone-redirect-revisit
Open

Fix clone redirect revisit checks#882
StantonMatt wants to merge 1 commit into
gocolly:masterfrom
StantonMatt:fix-clone-redirect-revisit

Conversation

@StantonMatt

Copy link
Copy Markdown

Summary

  • fix cloned collectors following redirects with the parent collector's redirect checks
  • store the request-owning collector in the request context before dispatch
  • keep the shared HTTP backend while routing CheckRedirect through the collector that created the request

Fixes #875.

Verification

go test . -run 'TestCollector(CloneRedirectUsesCloneRevisitSetting|URLRevisit|Depth)|TestCollectorRedirect' -count=1 -v
go test ./... -count=1
gofmt -l colly.go colly_test.go
git diff --check origin/master...HEAD

I also checked the regression test against current master before the fix; it failed with AlreadyVisitedError when the clone had AllowURLRevisit = true and the parent did not.

Signed-off-by: Matthew Stanton <stantonmatthewj@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cloned Collectors inherit checkRedirectFunc (so it uses the parent's AllowURLRevisit)

1 participant