Skip to content
masterclass Buddy
Start learning

Scrapy · Option

ROBOTSTXT_OBEY

RobotsTxtMiddleware checks Scrapy requests against a site's robots.txt rules. Enable the middleware and ROBOTSTXT_OBEY to block requests that the site forbids.

Last updated

Syntax

async process_request(request: Request, spider: Spider | None = None) -> None

In-depth API docs are only for subscribers: parameters, examples, common mistakes and interview practice.

Subscribe
  • ROBOTSTXT_OBEY
  • ROBOTSTXT_PARSER
  • ROBOTSTXT_USER_AGENT
  • DOWNLOADER_MIDDLEWARES
  • scrapy.robotstxt.ProtegoRobotParser
  • scrapy.robotstxt.PythonRobotParser

Source: Downloader Middleware — Scrapy 2.13.4 documentation. Checked 2026-11-03.