Response parsing
Extract fields from an HTTP response in a spider.
Scrapy · Class
Selector
Selector wraps a response or markup string and lets you select document content with XPath or CSS. Results are extracted with get() or getall().
How it works
Selector wraps response content or supplied markup so XPath, CSS, and related extraction operations can be applied. When created from a response, Scrapy infers the selector type from that response; when created from text, it defaults to HTML unless a type is specified.
Syntax
scrapy.Selector(**args: Any, **kwargs: Any)Parameters
| Name | Type | Description |
|---|---|---|
response | TextResponse | None | A TextResponse or response-compatible object whose content is selected and extracted. |
text | str | None | Markup supplied directly when a response is unavailable. |
type | str | None | Parsing mode; supported values include html, xml, json, or None for inference. |
Returns
A Selector instance wrapping the supplied response or text content.
Key points
- Selector builds on parsel while integrating selection and extraction with Scrapy response objects.
- A Scrapy response already provides its selector through the response.selector attribute.
- Initialize it from a response or provide markup directly with the text argument.
- Selection alone produces selector results; call get() or getall() to obtain extracted values.
- Selectors target document parts using XPath or CSS expressions.
Common mistakes
- Use either a response object or markup supplied through the text argument; do not assume a response is required.
- Leaving type unset delegates parser selection to Scrapy, which may infer HTML, XML, or JSON from the response.
Example: Select text from markup
from scrapy import Selector
html = "<html><body><h1>Scrapy</h1></body></html>"
selector = Selector(text=html)
print(selector.xpath("//h1/text()").get())Creates a selector from markup and extracts the heading text with XPath.
When to use it
Standalone markup parsing
Parse a saved HTML fragment or fixture without issuing a request.
Nested document selection
Traverse nested matches using chained XPath or CSS selections.
Interview questions
QWhat is Selector used for?
Selector is Scrapy’s wrapper for selecting parts of response content. It can be constructed from a TextResponse or from markup passed through text.
Follow-up: How would you create one without a response? Pass the markup string using Selector(text=...).
QHow do you extract data with Selector?
Use xpath() or css() to locate content, then get() for one result or getall() for all matching results. XPath and CSS selections return selector objects that support further chaining.
Follow-up: What happens when no result is available from get()? It returns None, while getall() returns an empty list.
QHow is the parsing type determined?
The selector type can be inferred from the response, or forced with type. Supported types include html, xml, and json, with HTML used as the default when text is supplied without an explicit type.
Follow-up: Why force type? To ensure the supplied content is parsed using the intended rules instead of automatic inference.
Related
SelectorListResponse.selectorxpath()css()get()getall()parsel