Seed a crawl with start_urls
A crawl begins from a fixed list of pages.
Scrapy · Class
Spider
scrapy.Spider is the base class for defining Scrapy crawlers. Configure initial URLs, parse responses, extract items, and yield requests to continue crawling.
How it works
Subclass scrapy.Spider to define a crawl. Scrapy uses the spider's initial URLs to create requests, invokes parse for each response by default, and lets callbacks yield extracted items or more requests to continue the crawl.
Syntax
class scrapy.Spider(**args: Any*, ***kwargs: Any*)Parameters
| Name | Type | Description |
|---|---|---|
*args, **kwargs | Any | Arguments supplied when the spider is instantiated; the documentation shows the class accepting arbitrary positional and keyword arguments. |
Returns
A spider callback can yield item objects and Request objects; start() yields initial Request objects.
Key points
- A spider packages the crawl rules for one site or a related collection of sites.
- Its two central responsibilities are scheduling requests and interpreting downloaded responses.
- The crawl begins by consuming requests produced by start().
- Providing start_urls is the simplest way to declare the initial crawl targets.
- Selectors are the usual mechanism for extracting information from a Response.
- A callback communicates extracted results and further traversal through items and Requests.
Common mistakes
- The default initial requests use parse as their callback, so implement parse unless you assign another callback.
- Callbacks can produce items or continuation requests; yielding neither means the crawl produces no extracted data from that response.
Example: Minimal Spider
import scrapy
class Spider(scrapy.Spider):
name = "quotes"
start_urls = ["https://example.com/"]
def parse(self, response):
yield {"title": response.css("title::text").get()}
This spider starts at the configured URL, parses the downloaded response, and yields an item containing the page title.
When to use it
Extract items from responses
Downloaded pages contain structured information.
Follow continuation requests
Links or API pages lead to additional content.
Group site-specific crawling logic
Several related sites share crawl behavior.
Interview questions
QWhat is scrapy.Spider and what is its default crawl flow?
A Spider is a class that defines which requests to send and how to process responses. It normally starts from start_urls, and parse handles each initial response by default.
Follow-up: How does a callback continue a crawl? It yields Request objects, whose callbacks process later responses.
QHow are a spider's initial requests created?
Set start_urls to the initial URL strings. Scrapy's default start() creates a Request for each one and assigns parse as the callback.
Follow-up: What if startup requires generated or customized requests? Override start() and yield the required Request objects.
QWhat may a spider callback yield?
A callback parses the Response, commonly with Selectors, and yields item objects for extracted data. It may also yield Request objects to follow additional pages.
Follow-up: Can one callback return both? Yes; it can yield extracted items and continuation requests during the same response processing.
Related
startstart_urlsparseRequestResponseSelectorsfrom_crawler