Skip to content
masterclass Buddy
Start learning

Scrapy · Option

CONCURRENT_REQUESTS

CONCURRENT_REQUESTS sets Scrapy’s global limit for simultaneous downloader requests. For broad crawls, the documentation suggests starting at 100 and tuning it based on CPU and memory use.

Last updated

Syntax

CONCURRENT_REQUESTS = integer

In-depth API docs are only for subscribers: parameters, examples, common mistakes and interview practice.

Subscribe
  • CONCURRENT_REQUESTS_PER_DOMAIN
  • REACTOR_THREADPOOL_MAXSIZE

Source: Broad Crawls — Scrapy 2.15.2 documentation. Checked 2026-11-03.