feat(core): provide requests to skipped callbacks#3867
Conversation
barjin
left a comment
There was a problem hiding this comment.
Thank you @AkhilTrivediX !
See my comment below regarding the interface change, I'm otherwise fine with the changes 👍
Btw another thing to consider is the larger memory footprint (see screenshot).
This is imo expected (as we're constructing Request instances now) and likely wouldn't be as striking in average cases (I ran the benchmark on a skip-heavy scenario), so it's not a blocker for me. Still, something to keep in mind.
Thanks!
| export type SkippedRequestReason = 'robotsTxt' | 'limit' | 'enqueueLimit' | 'filters' | 'redirect' | 'depth'; | ||
|
|
||
| export type SkippedRequestCallback = (args: { url: string; reason: SkippedRequestReason }) => Awaitable<void>; | ||
| export type SkippedRequestCallback = (args: { request: Request; reason: SkippedRequestReason }) => Awaitable<void>; |
There was a problem hiding this comment.
This is a breaking change for users passing onSkippedRequest: ({ url }) => { ... }.
If we agree on this interface, this PR needs to be targeting the v4 branch (the upcoming major). Even then, I'd prefer to only deprecate the url field (perhaps with a warning message informing about the new request field?
Summary
RequesttoonSkippedRequestinstead of a URL stringuserDatareaches the callbackWhy
onSkippedRequestpreviously discarded request metadata, which made it impossible for handlers to inspect fields such asuserData. This follows the v4 API direction in #3801: callers receive the request object and can read its URL fromrequest.url.Validation
corepack yarn eslint packages/basic-crawler/src/internals/basic-crawler.ts packages/browser-crawler/src/internals/browser-crawler.ts packages/core/src/enqueue_links/enqueue_links.ts packages/core/src/enqueue_links/shared.ts packages/http-crawler/src/internals/http-crawler.ts packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts test/core/crawlers/basic_crawler.test.tscorepack yarn tsc-check-testscorepack yarn vitest run test/core/crawlers/basic_crawler.test.ts --silent(68 passed)Closes #3801