Selective link crawling
Restrict a CrawlSpider to URLs matching patterns such as product or article paths.
Scrapy · Class
Rule
Rule defines how a CrawlSpider extracts, processes, follows, and handles links. Configure it with a LinkExtractor plus optional callback, request/link processors, follow behavior, and errback.
How it works
Rule configures how CrawlSpider handles links found by a LinkExtractor. It can select links, dispatch matching responses to a callback, pass callback keyword arguments, transform links or requests, handle failures, and control whether subsequent links are followed.
Syntax
scrapy.spiders.Rule(link_extractor: LinkExtractor | None = None, callback: CallbackT | str | None = None, cb_kwargs: dict[str, Any] | None = None, follow: bool | None = None, process_links: ProcessLinksT | str | None = None, process_request: ProcessRequestT | str | None = None, errback: Callable[[Failure], Any] | str | None = None)Parameters
| Name | Type | Description |
|---|---|---|
link_extractor | LinkExtractor | None | LinkExtractor defining which links are extracted. |
callback | CallbackT | str | None | Callback callable or spider method name for extracted links. |
cb_kwargs | dict[str, Any] | None | Keyword arguments passed to the callback. |
follow | bool | None | Whether to follow links from matching responses. |
process_links | ProcessLinksT | str | None | Callable or method name processing extracted links. |
process_request | ProcessRequestT | str | None | Callable or method name processing generated requests. |
errback | Callable[[Failure], Any] | str | None | Callable or method name handling request failures. |
Key points
- A Rule describes one crawling behavior within a CrawlSpider.
- Requests created through rules retain the matching rule's index in their metadata for response dispatch.
- Omitting link_extractor uses a default extractor, which extracts links without custom extractor arguments.
- A callback receives the matching response as its first positional argument.
- cb_kwargs supplies named arguments that CrawlSpider passes into the callback.
- process_links can be a callable or a spider method name used to process extracted link lists.
- process_request can be a callable or method name that processes each request generated from a matching link.
- errback can be a callable or spider method name for handling request failures.
Common mistakes
- Rule order matters: when several rules match one link, CrawlSpider uses the earliest matching rule.
- Leaving follow unset changes its default based on callback presence: no callback follows links, while a callback does not.
- Do not copy the rule metadata key into requests created by another rule, because response dispatch can select the wrong callback.
Example: Crawl matching links with a callback
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class DocsSpider(CrawlSpider):
name = "docs"
allowed_domains = ["example.com"]
start_urls = ["https://example.com/"]
rules = (
Rule(LinkExtractor(allow=r"/articles/"), callback="parse_article", follow=True),
)
def parse_article(self, response):
yield {"url": response.url, "title": response.css("title::text").get()}
The rule limits extracted links to article paths, sends matching responses to parse_article, and continues following links found there.
When to use it
Response parsing
Send matching pages to a named spider method that yields extracted items.
Link preprocessing
Apply custom filtering or normalization to extracted links before requests are created.
Request and failure handling
Customize generated requests or handle request failures for a rule.
Interview questions
QWhat is Rule used for?
Rule is a CrawlSpider configuration object that associates link extraction with optional callback, processing, error, and follow behavior. CrawlSpider applies the first matching rule when multiple rules match a link.
Follow-up: What happens if a generated request preserves another rule's metadata? It can cause the response to be dispatched to the wrong callback, so rule metadata must not be copied across rules.
QWhat does link_extractor control?
link_extractor determines which links are extracted from each crawled page. Each extracted link becomes a Request, and its link text is stored in the request metadata under link_text.
Follow-up: How would you restrict crawling to product pages? Supply a LinkExtractor configured with an allow pattern for product URLs.
QHow does callback interact with follow?
callback may be a callable or the name of a spider method and receives a Response. It should return an item, request, or iterable containing items and/or requests.
Follow-up: How does callback choice affect follow when follow is omitted? follow defaults to true without a callback and false when a callback is supplied.
Related
CrawlSpiderLinkExtractorSpiderRequestResponse