Scrapy · Module

settings

Scrapy settings are a shared key-value configuration namespace for the framework and its components. Projects normally define common settings in settings.py, with the active module selected through SCRAPY_SETTINGS_MODULE.

How it works

Scrapy settings provide a shared key-value configuration namespace used by the framework, extensions, pipelines, and spiders. A project normally stores its common configuration in settings.py, while the active project settings module can be selected with SCRAPY_SETTINGS_MODULE.

Syntax

SCRAPY_SETTINGS_MODULE=myproject.settings

Key points

Settings act as a shared configuration namespace that Scrapy components can query by key.
The configuration applies across Scrapy's core, extensions, item pipelines, and spiders.
A standard project generally keeps its common configuration in settings.py.
Command-line values override settings supplied by lower-precedence sources.

Common mistakes

Set SCRAPY_SETTINGS_MODULE to an importable Python module path, such as myproject.settings, rather than a filesystem path.
The selected settings module must be importable by Python.

Example: Select a project settings module

import os
from scrapy.crawler import CrawlerProcess

os.environ["SCRAPY_SETTINGS_MODULE"] = "myproject.settings"
process = CrawlerProcess()
process.crawl("quotes")
process.start()

Selects a project settings module through the environment and starts a crawler process using those settings.

When to use it

Project-wide configuration

Configure behavior shared by all spiders in a project.

Select the active settings module

Choose which Scrapy project configuration to load.

Command-line overrides

Override configuration for one execution without editing project files.

Interview questions

QWhat are Scrapy settings and where are project settings normally stored?

Settings are a global key-value mapping used to configure Scrapy components. A project usually places shared values in its settings.py module, and SCRAPY_SETTINGS_MODULE identifies which project settings module is active.

Follow-up: How do conflicting setting sources resolve? Command-line settings have the highest precedence, followed by spider, project, add-on, command-specific default, and global default settings.

QHow does SCRAPY_SETTINGS_MODULE work?

The environment variable selects the active settings module using Python import syntax, for example myproject.settings. That module must be available on Python's import search path.

Follow-up: What would you check if selection fails? Verify both the dotted module name and that the project is importable from the current Python path.

QWhich settings source wins when values conflict?

Settings are populated through several mechanisms with defined precedence. Command-line values override spider, project, add-on, command-specific default, and global default values.

Follow-up: Why does this matter operationally? A command-line override can change runtime behavior without editing the project's settings.py file.

SCRAPY_SETTINGS_MODULE
scrapy.crawler.Crawler.settings
settings.py
scrapy.settings.default_settings

Source: Settings. Checked 2026-10-15.