Item extraction
Extract structured fields from HTML or other downloaded content.
Scrapy · Function
parse
parse is a Scrapy spider’s default callback for processing responses without an explicitly assigned callback. It extracts items and can yield requests to continue crawling.
How it works
Define parse on a spider to handle a downloaded Response, extract data, and optionally produce more requests. Scrapy uses it by default when a request has no callback, including the requests generated from start URLs; callback keyword arguments supplied through cb_kwargs are also accepted.
Syntax
parse(response: Response, **kwargs: Any) -> AnyParameters
| Name | Type | Description |
|---|---|---|
response | Response | The downloaded response to process and extract data from. |
**kwargs | Any | Additional keyword arguments supplied through the request’s cb_kwargs. |
Returns
A Request, item, iterable of requests and/or items, or None.
Key points
- Scrapy selects parse automatically when a downloaded request has no explicit callback.
- Its two core jobs are extracting scraped data and generating additional URLs for the crawl.
- Values placed in a request’s cb_kwargs reach parse as keyword arguments.
- The method combines response processing with optional crawl expansion.
- Selectors are the typical extraction mechanism, while items and requests represent the callback’s outputs.
Common mistakes
- The callback must yield or return only supported Scrapy outputs: requests, items, iterables containing them, or None.
- A spider can omit parse only when every request it sends has its own callback.
Example: Extract items with parse
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com/"]
def parse(self, response):
for quote in response.css(".quote"):
yield {
"text": quote.css(".text::text").get(),
"author": quote.css(".author::text").get(),
}
The spider’s default callback extracts fields from each matching element and yields one item per quote.
When to use it
Link traversal
Follow pagination, detail links, or discovered URLs.
Context-aware callbacks
Propagate request-specific context into callback processing.
Interview questions
QWhat is parse responsible for?
parse is the default callback for responses whose requests do not specify a callback. It receives the Response, extracts data, and can return or yield items and further requests.
Follow-up: What happens to requests yielded by parse? Scrapy schedules them, downloads their responses, and invokes the callback assigned to each request.
QWhat arguments can parse receive?
Its first argument is the Response being processed. It may also receive keyword arguments passed through the request’s cb_kwargs.
Follow-up: How can a callback remain compatible with arbitrary callback data? Accept keyword parameters, typically through **kwargs, in addition to response.
QWhat may parse return?
parse may produce a Request, an item, an iterable containing requests or items, or None. Yielding is commonly used to emit multiple results incrementally.
Follow-up: Why yield rather than build a list? Yielding lets Scrapy consume results incrementally and avoids requiring all outputs to be held at once.
Related
scrapy.SpiderResponseRequeststart()start_urlscb_kwargscallbackSelectors