Normalize scraped fields
A spider extracts product fields from HTML and needs consistent normalization before creating the item.
Scrapy · Class
ItemLoader
ItemLoader is Scrapy’s convenient, response-aware API for collecting scraped values into items while applying field processors. Add values from selectors or directly, then call load_item() to produce the populated item.
How it works
ItemLoader is Scrapy’s response-aware extension of itemloaders for collecting scraped values and applying field processors before producing an item. Instantiate it with an existing item or let it create one from default_item_class, add values through extraction or direct methods, and call load_item() when collection is complete.
Syntax
ItemLoader(item=None, selector=None, response=None, **context)Parameters
| Name | Type | Description |
|---|---|---|
item | Item or None | Existing item to populate; when omitted, an item is created using default_item_class. |
selector | Selector or Response | Selector or response used by extraction methods such as add_xpath() and add_css(). |
Returns
load_item() returns the populated item after collected values have been processed.
Key points
- ItemLoader centralizes the process of filling scraped items instead of requiring direct field assignment.
- Scrapy’s class extends the underlying itemloaders functionality with response-oriented integration.
- A field may receive several values during collection; the configured processor determines how those values are combined.
- Processing is configured independently per field, with separate input and output stages.
- Values can be collected from XPath selections, CSS selections, or direct values using the corresponding add methods.
Common mistakes
- Input processing happens immediately when values are added, so the loader stores processed input rather than the original value.
- The item is not finalized merely by adding values; call load_item() after collection to obtain the populated item.
Example: Populate an item with ItemLoader
from scrapy.loader import ItemLoader
from scrapy.item import Item, Field
class Product(Item):
name = Field()
price = Field()
loader = ItemLoader(item=Product())
loader.add_value("name", "Widget")
loader.add_value("price", "12.50")
product = loader.load_item()
print(product["name"], product["price"])Creates a loader for a Product item, collects values, then materializes the populated item with load_item().
When to use it
Combine repeated values
A field is assembled from several extracted fragments that must be combined by a processor.
Customize parsing rules
Different spiders or source formats require independently configurable field parsing rules.
Interview questions
QWhat problem does ItemLoader solve?
ItemLoader provides a structured API for populating scraped items while applying field processors. It accepts an item or can create one automatically, collects values, and returns the populated item through load_item().
Follow-up: How are multiple values for one field handled? The loader accumulates them and later combines them through the field’s configured processing function.
QWhat happens if ItemLoader is created without an item?
You can instantiate it with an item object, or omit the item and rely on the class configured by default_item_class. The latter causes an item object to be created during initialization.
Follow-up: How would you choose a different item type for a particular loader? Configure or override default_item_class for that loader class or use an explicitly supplied item.
QWhen do ItemLoader processors run?
Each item field has an input processor and an output processor. Input processing occurs as values arrive, while output processing is applied when the collected field value is loaded into the final item.
Follow-up: Why can adding values from different sources remain maintainable? Field-specific processors centralize normalization and combination rules instead of scattering them through spider code.
Related
ItemLoader.default_item_classItemLoader.add_xpath()ItemLoader.add_css()ItemLoader.add_value()ItemLoader.load_item()Input and Output processorsSelectors