API reference
Scrapy API reference
Scrapy web crawling framework
- Identifiers
- 12
- With full guides
- 12
- Types
- 3
Topics covered
- Spiders
- Requests & responses
- Selectors
- Items
- Item pipelines
- Settings
- Middlewares
- Crawling at scale
Class
| Name | What it does |
|---|---|
Spider | scrapy.Spider is the base class for defining Scrapy crawlers. Configure initial URLs, parse responses, extract items, and yield requests to continue crawling. |
Request | scrapy.Request represents an HTTP request created by a spider and executed by Scrapy's Downloader. Configure its URL, method, callback, payload, metadata, headers, scheduling priority, and error handling. |
Response | scrapy.http.Response represents an HTTP response downloaded by Scrapy and passed to a spider. It stores the URL, status, headers, raw byte body, and optional flags. |
Item | scrapy.Item is a dictionary-like container for structured data extracted by spiders. Define its permitted fields with scrapy.Field() and populate instances using normal mapping operations. |
Field | scrapy.Field is a dictionary-like metadata container used to declare fields on Scrapy Item subclasses. It can store configuration such as a serializer, while Scrapy records declared fields in Item.fields. |
CrawlSpider | CrawlSpider is a Spider subclass for regular websites that follows links according to ordered Rule objects. Subclass it, define standard spider attributes, and use rules to describe link selection and callbacks. |
Selector | Selector wraps a response or markup string and lets you select document content with XPath or CSS. Results are extracted with get() or getall(). |
ItemLoader | ItemLoader is Scrapy’s convenient, response-aware API for collecting scraped values into items while applying field processors. Add values from selectors or directly, then call load_item() to produce the populated item. |
Rule | Rule defines how a CrawlSpider extracts, processes, follows, and handles links. Configure it with a LinkExtractor plus optional callback, request/link processors, follow behavior, and errback. |
Function
| Name | What it does |
|---|---|
parse | parse is a Scrapy spider’s default callback for processing responses without an explicitly assigned callback. It extracts items and can yield requests to continue crawling. |
xpath | xpath() applies an XPath expression to a Scrapy response or selector and returns the matching nodes. Extract their values with get() or getall(). |
Module
| Name | What it does |
|---|---|
settings | Scrapy settings are a shared key-value configuration namespace for the framework and its components. Projects normally define common settings in settings.py, with the active module selected through SCRAPY_SETTINGS_MODULE. |
Search across every package and filter by type in your dashboard.
Start learning