Scrapy · Class

Request

scrapy.Request represents an HTTP request created by a spider and executed by Scrapy's Downloader. Configure its URL, method, callback, payload, metadata, headers, scheduling priority, and error handling.

How it works

scrapy.Request represents an HTTP request generated by a spider and executed by Scrapy's Downloader. Its options control the URL, HTTP method, callback or error callback, request data, metadata, priority, filtering, and related request behavior.

Syntax

scrapy.Request(**args: Any, **kwargs: Any)

Parameters

Name	Type	Description
`url`	`str`	URL of the request; invalid URLs raise ValueError.
`callback`	`collections.abc.Callable`	Callable invoked with the downloaded response.
`method`	`str`	HTTP method; defaults to GET.
`meta`	`dict`	Initial values for Request.meta; shallow-copied when supplied.
`body`	`bytes or str`	Request body as bytes or string; strings are encoded using encoding.
`encoding`	`str`	Request encoding, defaulting to utf-8.
`headers`	`dict-like`	Request headers.
`cookies`	`dict or list`	Request cookies.
`priority`	`int`	Scheduling priority.
`dont_filter`	`bool`	Whether duplicate-request filtering is disabled.
`errback`	`collections.abc.Callable`	Callable used for request-processing errors.

Returns

A scrapy.Request object representing the HTTP request; its execution produces a Response.

Key points

A Request models work that a spider schedules for the Downloader.
callback receives the response after the request is downloaded.
method selects the HTTP verb and defaults to GET.
meta supplies request-scoped values accessible through Request.meta.
body carries payload data; string payloads are encoded to bytes.
Without callback, the spider's parse method handles the response.

Common mistakes

A request without callback uses the spider's parse() method.
An invalid URL raises ValueError during request creation.
Request metadata is shallow-copied, so nested objects remain shared.

Example: Request with callback and metadata

import scrapy

class ExampleSpider(scrapy.Spider):
    name = "example"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.org/",
            callback=self.parse,
            headers={"Accept": "text/html"},
            meta={"source": "homepage"},
        )

    def parse(self, response):
        yield {"title": response.css("title::text").get()}

Creates a GET request, routes its downloaded response to parse, and carries custom headers and metadata.

When to use it

Follow links with a callback

A spider needs to follow a discovered link and process the resulting page.

Set request headers and cookies

A request needs authentication, content negotiation, or custom HTTP behavior.

Pass crawl state with meta

A callback needs context while processing a response.

Send a custom method and body

A spider submits data or calls an endpoint other than the default GET request.

Interview questions

QWhat is scrapy.Request and how does it fit into crawling?

Request represents an HTTP request that a spider usually creates and the Downloader executes, producing a Response. The callback receives that downloaded response.

Follow-up: What happens when callback is omitted? Scrapy uses the spider's parse() method.

QHow do callback and errback differ?

The callback handles a successful downloaded response, while errback handles errors raised during processing. The request can also carry headers, cookies, body data, metadata, and an encoding.

Follow-up: Why use meta? It passes request-scoped data through the crawl, but nested mutable values are not independently copied.

QWhich core arguments control a Request?

The method controls the HTTP method and defaults to GET. The URL is required, and invalid URLs raise ValueError.

Follow-up: How can request scheduling behavior be adjusted? Use priority and dont_filter, as documented request options.

scrapy.http.Response
Request.meta
Request.replace
Request.copy
scrapy.FormRequest

Source: Requests and Responses — Scrapy 2.19.0 documentation. Checked 2026-10-15.