Rule-based site traversal
A site has predictable internal URL patterns and pages should be crawled automatically.
Scrapy · Class
CrawlSpider
CrawlSpider is a Spider subclass for regular websites that follows links according to ordered Rule objects. Subclass it, define standard spider attributes, and use rules to describe link selection and callbacks.
How it works
CrawlSpider is a Spider subclass for crawling regular websites through declarative Rule objects. It extracts links, selects the first matching rule in declaration order, creates requests using that rule, and processes the resulting responses with the rule callback; start responses are handled through parse_start_url.
Syntax
class scrapy.spiders.CrawlSpider(Spider):Key points
- Use CrawlSpider as a convenient starting point for ordinary website crawling.
- Link traversal is declared through rules instead of being fully hand-coded.
- The class retains the standard Spider attributes, which the subclass must define as appropriate.
Common mistakes
- Rule order matters: when several rules match a link, only the first matching rule is selected.
Example: CrawlSpider with an ordered link rule
from scrapy import Spider
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class DocsSpider(CrawlSpider):
name = "docs"
allowed_domains = ["example.com"]
start_urls = ["https://example.com/"]
rules = (
Rule(LinkExtractor(allow=(r"/docs/",)), callback="parse_doc"),
)
def parse_doc(self, response):
yield {"url": response.url, "title": response.css("title::text").get()}
A CrawlSpider applies its ordered rules to links extracted from responses, sending matching links to the configured callback. Subclass it, define the usual spider attributes, and provide rules describing link-following behavior.
When to use it
Pattern-specific parsing
Several link patterns need different callbacks or extraction behavior.
Reusable crawling scaffold
A project needs a reusable generic crawler starting from one or more URLs.
Interview questions
QWhat problem does CrawlSpider solve?
CrawlSpider is a Spider subclass designed for regular websites. Its rules describe link-following behavior, so the crawler can generate requests without manually implementing every link traversal.
Follow-up: How does it choose among overlapping rules? It uses the first matching rule according to the order in the rules attribute.
QWhat is the purpose of rules?
The rules attribute is a sequence of Rule objects. Each Rule defines a particular crawling behavior, such as which links to select and how to process their responses.
Follow-up: What should be changed for highly custom traversal? Override the relevant behavior or implement a regular Spider when the generic rule model is not suitable.
QHow are start responses handled?
parse_start_url handles responses from start_urls and can parse initial content before rule-driven crawling continues. Its result must be an item, a request, or an iterable containing those types.
Follow-up: How can a spider customize processing after a rule callback? Use the rule's callback and the CrawlSpider response-processing hooks rather than relying on the default parser alone.
Related
SpiderRuleLinkExtractorparse_start_url