Read text nodes
Extracting page text such as headings, titles, or paragraph content.
Scrapy · Function
xpath
xpath() applies an XPath expression to a Scrapy response or selector and returns the matching nodes. Extract their values with get() or getall().
How it works
Call xpath() on a Scrapy response or selector with an XPath query. It returns a flattened SelectorList; use get() for one string or getall() for all matching values, and use a leading dot for paths relative to a nested selector.
Syntax
response.xpath(query, namespaces=None, **kwargs)Parameters
| Name | Type | Description |
|---|---|---|
query | str | XPath query string to apply to the response or selector. |
namespaces | Mapping[str, str] | None | Optional namespace mapping for the XPath query. |
**kwargs | Any | Additional keyword arguments accepted by the selector implementation. |
Returns
A flattened SelectorList containing selectors for nodes matching the XPath query.
Key points
- XPath expressions select nodes in both XML and HTML documents.
- XPath is the underlying selection mechanism on which Scrapy selectors are built.
- Matching selectors do not become strings automatically; call get() or getall() to extract their contents.
- A text-node XPath can be combined with getall() to collect every matching text value.
- The selector documentation describes descendant text selection as a supported selection pattern.
- Attribute values can be selected from descendant links, including href.
Common mistakes
- A leading slash makes a nested XPath absolute to the document rather than relative to the current selector; use .// for descendants.
- For XPath string functions, pass the current node with . instead of selecting descendant text nodes through .//text().
- Namespaced XML may require removing namespaces before an XPath query can match elements.
Example: Extract title text with XPath
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
def parse(self, response):
for title in response.xpath("//title/text()").getall():
yield {"title": title}
Selects every title text node and emits each extracted value.
When to use it
Read attributes
Collecting URL or metadata values stored in HTML/XML attributes.
Use XPath filtering
Selecting records with conditions or paths that are awkward to express in CSS.
Query nested content
Traversing a selected element's descendants while preserving the current context.
Interview questions
QWhat does Scrapy's xpath() method return?
xpath() evaluates an XPath query against a response or selector and returns matching nodes as a flattened SelectorList. The resulting selectors still support extraction methods such as get() and getall().
Follow-up: How do you select relative descendants from a nested selector? Use .//, because an XPath beginning with / is absolute to the document.
QHow do you extract values after calling xpath()?
Use get() when one extracted string is needed and getall() when all matching strings are needed. For example, response.xpath("//title/text()").get() returns one title value.
Follow-up: What happens if no nodes match? The selector list is empty, so get() yields None while getall() yields an empty list.
QWhat kinds of data can an XPath expression select?
The argument is an XPath expression, which can select HTML or XML nodes, text nodes, and attributes. For example, //a/@href selects link attributes and //span/text() selects span text.
Follow-up: How can XML namespace problems be handled? Remove namespaces with Selector.remove_namespaces() before applying the XPath.
Related
SelectorSelectorListget()getall()Selector.remove_namespaces()css()