Web crawler at scale

I would use a two-level design. Front queues are ordered by priority, computed from link signals, domain quality and freshness need. A router moves URLs from front queues into back queues, one per host, and each back queue has a next-allowed-fetch timestamp in a min-heap. A fetcher pops the host whose time has come, takes one URL, and the host's time is pushed forward by its crawl delay, adjusted up after 429 or 5xx responses. Because there are millions of hosts, there are always ready hosts, so fetchers stay busy without any single host being hit hard. I would shard by host hash so each host lives on exactly one worker, which avoids distributed locking. I would also apply limits per IP, since many small sites share servers.

There are two separate problems. For URLs, I normalize first: lowercase scheme and host, remove fragments and default ports, resolve relative paths, and strip known tracking parameters. Then I check a Bloom filter; a miss means definitely new, a hit is confirmed in a sharded key-value store, so memory stays small. For content, exact hashes catch identical bodies, but most duplicates differ by ads or timestamps. So I extract main text, compute a 64-bit SimHash, and treat pages within a few bits of Hamming distance as near-duplicates. To search quickly, I split the fingerprint into bands and index each band, so candidates share at least one band. Duplicates are stored as aliases pointing to a canonical page, and rel=canonical tags are used as a hint, not trusted blindly.

I split it into a loop of stages. Seeds go into a URL frontier. The frontier has priority queues and per-host back queues. A politeness gate releases a URL only when the host's delay has passed and robots.txt allows the path. Async fetchers use a local DNS cache and send conditional GETs. The parser extracts and normalizes links and applies trap rules. A URL-seen check, a Bloom filter backed by a sharded key-value store, decides which links enter the frontier. Content is fingerprinted for exact and near duplicates, and unique pages are written as WARC records to object storage. A recrawl scheduler reads fetch history and re-enqueues URLs as they come due. Each stage scales out independently, and work is partitioned by host.

A 30-day month is about 2.6 million seconds, so one billion pages means roughly 385 fetches per second on average. I would plan for two to three times that at peak, because of retries and uneven host availability. Bandwidth and storage depend on the average page size, so I state it as an assumption. If pages average 100 KB uncompressed, that is about 38 MB/s, around 300 Mbps sustained, and about 100 TB of raw HTML per month before compression. Gzip in WARC typically shrinks HTML several-fold. Each fetch spends most of its time waiting on the network, so async I/O lets one machine hold thousands of open connections. The real limits are politeness and DNS, not CPU.

I model each page's changes as a Poisson process with rate λ. Each fetch tells us whether the content changed since the last fetch, using a 304 response, an unchanged hash or a SimHash distance. From that history I estimate λ, with estimators that correct for changes missed between visits. The next recrawl time comes from λ and the page's value, such as importance, traffic or how often it links to new URLs. Pages that rarely change are backed off exponentially up to a maximum interval. Volatile, high-value pages are revisited sooner. Research by Cho and Garcia-Molina shows that pages which change too fast to keep fresh should not get unlimited budget. Sitemap lastmod, feeds and push signals can trigger early recrawls.

A trap is any URL space that generates effectively infinite pages: calendars with next-month links forever, session IDs in URLs, faceted filters combining endlessly, or relative-link bugs that keep growing the path. Some are malicious, most are accidental. I defend in layers. Normalization removes session parameters. Rule filters reject URLs over a length limit, with too many query parameters, or with repeated path segments. Each host gets a crawl budget tied to its importance, so a trap can only waste that host's share. I track depth from seeds and the ratio of new unique content per host; if a host keeps producing near-duplicate pages, I lower its priority. Finally, I monitor hosts whose discovered URL counts grow abnormally fast and review or block patterns.

I partition by host, or by registrable domain, using consistent hashing. Each host is then owned by exactly one node, so its back queue, next-allowed-fetch time and robots.txt cache stay local. Politeness can be enforced without distributed locks. When a node joins or leaves, only about 1/N of the hosts move. Links the parser finds are routed to the owning node of their host, usually in batches through a message bus, to keep cross-node traffic cheap. Very large hosts can be a hotspot, but politeness caps per-host throughput anyway, so that is rarely a CPU problem. Virtual nodes even out how many hosts each machine owns. Ownership changes must hand over queue state, or the new owner must rebuild it from durable storage.

robots.txt is cached per host, keyed by scheme, host and port, because each origin has its own file. RFC 9309 says crawlers should not use a cached copy for more than 24 hours, unless the file is unreachable. So I set a TTL of 24 hours or less and refetch it when the TTL expires. A 4xx response, such as 404, means the file is unavailable, and the crawler may access any resource. Redirects should be followed for at least five hops. Crawlers must parse at least 500 KiB of the file. Rules are matched by the longest matching path. If an Allow and a Disallow rule are equally specific, Allow wins. Crawl-delay is not part of the standard, but many crawlers honour it as a courtesy.

Each host record stores a crawl delay, and the gate uses it to set the next-allowed-fetch time. On a 429, I honour the Retry-After header if it is present. Otherwise I apply multiplicative backoff, for example doubling the delay up to a cap. Repeated 5xx responses or timeouts also increase the delay. After sustained failures, the host is paused for a cool-down period. When responses are healthy again, the delay shrinks slowly, which is the AIMD pattern from congestion control. I also scale the delay with observed response time, because a slow server is a stressed server. Failed URLs are re-enqueued with a retry count, and they are dropped or parked after a limit. The host never sees a retry storm.

Safe rules follow RFC 3986 equivalence. Lowercase the scheme and host. Remove the default port, such as :80 for http. Resolve dot segments. Normalize the percent-encoding of unreserved characters. Drop the fragment, because it is never sent to the server. Risky rules change meaning on some sites. Paths can be case-sensitive. Some query parameters actually select content, so I only strip known tracking parameters like utm_* or session IDs, and I learn per-host rules where possible. Sorting query parameters is usually safe, but not always. Removing a trailing slash or a www prefix should be verified, for example by the redirects you observe. Over-normalizing loses pages. Under-normalizing wastes the crawl budget on duplicates.

The false-positive rate is about (1 − e^(−kn/m))^k, where n is the number of items, m is the number of bits and k is the number of hash functions. The optimal k is (m/n)·ln 2. A 1% false-positive rate needs about 9.6 bits per element, and 0.1% needs about 14.4. A false positive means an unseen URL is skipped, so the cost is lost coverage, not a correctness bug. If more URLs go in than planned, the rate climbs fast. So I size for the expected URL count with headroom, shard the filter by URL hash, and monitor its fill ratio. For long-running crawls I use scalable Bloom filters or rotate by generation. The backing key-value store holds the authoritative record.

Each URL gets a score from several signals. Examples are link-based importance such as an approximate PageRank, domain quality, depth from the seeds, freshness need for recrawls and business priorities. The score maps to one of several front priority queues. A biased selector picks from higher queues more often but still serves lower ones, so nothing starves. Per-host back queues add fairness, because no single host can dominate. I also cap the number of URLs per host and per domain admitted into each round. Without these caps, a giant site or a trap could crowd out the long tail. Priorities are recomputed periodically, because link graphs change as the crawl discovers more pages.

The frontier is durable, either a replicated log or a persistent queue with periodic checkpoints. A fetcher leases a URL instead of deleting it. When the fetch result is committed, the lease is acknowledged. If the node dies, the leases expire and the URLs become available again. This gives at-least-once fetching. A few pages may be fetched twice, which is acceptable because writes are idempotent. The WARC record is keyed by URL and fetch time, and dedup absorbs repeats. The per-host next-allowed-fetch time must also be persisted or conservatively reset, so recovery does not hit hosts too fast. Host ownership moves to another node through consistent hashing. Partially written WARC files are closed or discarded, and the index points only to committed records.

There are four main entities. A URL record is keyed by a hash of the normalized URL. It holds the URL, host ID, canonical URL, discovery source, depth, priority, status, last fetch time, HTTP status, ETag, Last-Modified, content hash, SimHash, estimated change rate and next due time. A host record holds the robots.txt rules and fetch time, crawl delay, next-allowed-fetch time, error counters, the resolved IP and the URL budget used. A content record is keyed by exact hash. It holds the SimHash, the canonical URL and a pointer to the WARC file and offset. Alias records map duplicate URLs to their canonical page. Fetch history is append-only for change estimation. Records are sharded by host or URL hash.

A crawler fetches attacker-controlled URLs from inside your network, so SSRF is the main risk. Before connecting, I resolve the hostname and block private, loopback, link-local and cloud metadata addresses, such as 10.0.0.0/8, 127.0.0.0/8 and 169.254.169.254. I connect to the validated IP itself, so a DNS rebinding answer cannot swap it, and I re-check every redirect hop. Fetchers run in an isolated network segment with egress-only rules and no credentials. Parsers handle hostile input, so they run sandboxed with memory and time limits. Decompression is capped to stop zip bombs. Only allowed schemes are fetched, usually http and https.

Every fetch either refreshes a known page or discovers a new one, so I split the budget explicitly. For example, a share goes to recrawls ranked by value multiplied by the probability of change, and the rest to discovery ranked by priority. The split is tuned with metrics such as freshness of important pages, the rate of new valuable URLs and the share of 304 or unchanged responses. A high unchanged rate means I am recrawling too much. Hubs such as home pages, category pages and feeds are worth frequent recrawls, because they yield new links. Sitemaps and push signals improve both goals cheaply. Politeness caps still apply per host whatever the split.

First I find which limit binds. Fetching and parsing scale horizontally: add nodes, and consistent hashing moves about 1/N of the hosts. The frontier and URL store need more shards, and per-node memory for Bloom filters must be re-planned for the new URL count. DNS caching capacity and egress bandwidth grow linearly, and their cost often dominates. Politeness does not scale with hardware, so tenfold volume must come from more hosts, or from fewer wasted fetches through better recrawl prediction, conditional GETs and sitemaps. Storage grows linearly, so compression, dedup and revisit records matter more. Parser and rendering tiers scale separately. Multi-region fetching places fetchers near hosts to cut latency.

For each URL I store the ETag and Last-Modified values from the previous response. On a recrawl, I send If-None-Match with the ETag, or If-Modified-Since with the date. An unchanged resource returns 304 Not Modified with no body. That saves bandwidth for both sides, skips parsing and storage, and still counts as a successful fetch for change-rate estimates. Many servers ignore these headers or generate a new ETag on every request. In that case I fall back to my own exact content hash. If the hash matches the last fetch, I record the page as unchanged, possibly as a WARC revisit record. The scheduler learns that the page is stable, even though the server never sent a 304.

I treat canonical as a strong hint, not a command, because site owners often get it wrong. Common errors are canonicals pointing to the home page, to a 404 or across unrelated domains. When a page declares a canonical, I record the fetched URL as an alias of the canonical URL. I enqueue the canonical URL if it has not been seen. I lower the priority for recrawling the alias. I trust it more when the canonical and alias content are near-duplicates by SimHash and the same site controls both. If the canonical is broken, I ignore it and keep the fetched URL as its own canonical. Canonical signals, redirects and duplicate clusters together decide which URL represents the page.

The Bloom filter is a fast, in-memory, probabilistic filter. It cannot be listed, it has false positives and a standard one cannot delete entries. The key-value store keeps the authoritative per-URL record: canonical URL, status, last fetch time, ETag, content hash and change statistics. The recrawl scheduler needs that record anyway. The flow is simple. If the Bloom filter says the URL is absent, it is definitely new, so I insert it. If the filter says present, I can check the store when the URL looks valuable. I shard both by a hash of the normalized URL, or by host, to match frontier ownership and keep checks local. Writes are batched. If a node loses its filter, it can be rebuilt from the store.

DNS is a classic bottleneck. Blocking system resolvers serialize lookups, and public resolvers will rate-limit a crawler. I run local caching resolvers on each crawler node or cluster. Lookups use an async resolver library and respect record TTLs, within a minimum and maximum bound. Hostnames can be prefetched when URLs enter a host's back queue, so the address is ready by the time the gate releases the URL. Negative results such as NXDOMAIN are cached for a shorter time. Resolution failures count toward host health. The resolved IP also matters for politeness, because many hostnames can share one server. Security checks against private address ranges run on the resolved address.

Shared hosting and CDNs mean thousands of hostnames may resolve to one server. Per-host delays alone could still overload that machine. So I add a second limiter keyed by IP address or IP prefix, which caps concurrent connections and request rate to that address. A URL is released only when both its host gate and its IP gate allow it. CDNs complicate this, because one IP fronts huge capacity and is not one fragile server. So IP limits for known large CDN ranges can be looser, and they still adapt to 429 and 5xx responses. The resolved IP can change over time, so the mapping is refreshed with DNS TTLs.

Rendering with a headless browser costs far more than a plain HTTP fetch in CPU, memory and latency. It also fetches many sub-resources, which affects politeness. So I make it a separate, selective tier. The normal pipeline fetches the raw HTML first. Heuristics or learned signals pick pages for rendering, for example a near-empty body, a known single-page-app framework or high value. The renderer runs in sandboxed pools with strict timeouts. It caches shared scripts and stylesheets, and it counts sub-resource requests against the host's politeness budget. Links found after rendering go through the same normalization, trap rules and seen-check pipeline. Most of the web can still be crawled without rendering.

Public pages can still contain personal data. In India the Digital Personal Data Protection Act, 2023 applies, and other regions have their own laws. I minimize what is kept and why. The crawler fetches only public pages, without logging in or bypassing paywalls, and respects robots.txt plus robots meta and X-Robots-Tag directives. The user-agent string identifies the crawler and links to a contact page. There must be a takedown and removal process that deletes or suppresses content and stops future crawling. Access to raw WARC archives is restricted and audited. Retention is defined, and data is encrypted at rest. Legal review covers copyright use and the purpose of the crawl.

WARC is the ISO 28500 standard archive format used by the Internet Archive and Common Crawl. It stores the full HTTP request and response, with headers and timestamps, so you can reprocess, replay and audit later. Many gzip-compressed records are appended into large files, which suits object storage. Each record compresses on its own, so a reader can seek to a byte offset without unpacking the whole file. The main record types are warcinfo, request, response, metadata and revisit. A revisit record notes that the content matches an earlier capture, so duplicates do not store the payload again. A separate index maps URL and timestamp to file and offset.

I treat a redirect target as a new URL, not something fetched inline. It goes through normalization, trap rules, the seen-check and politeness for its own host. That stops redirects from bypassing rate limits or robots.txt on another domain. I cap redirect chains, at around five to ten hops, and detect loops. Permanent redirects (301, 308) update the source URL's record to point at the target and lower the recrawl priority of the source. Temporary ones (302, 307) keep the original as the canonical URL to recrawl. Each hop is re-checked for SSRF. Redirects to login pages or soft-error pages are marked so they do not pollute the index.

The per-host limit sets the ceiling, so speed comes from using that limit well, not from exceeding it. I follow the site's robots.txt and any published crawl guidance. I reuse keep-alive or HTTP/2 connections so each request costs the server less. Conditional GETs keep recrawls cheap. Sitemaps help target new and changed URLs instead of rediscovering them through links. Many large sites offer database dumps or APIs, Wikipedia's dumps for example, which are better than crawling. Concurrency rises only while response times and error rates stay healthy, and backs off on 429 or 5xx. For partners, a negotiated rate or a push feed is the right approach.

Throughput: fetches per second by status class, bytes downloaded and pages stored. Politeness: requests per host and per IP against limits, 429 and 5xx rates per host, and robots.txt fetch failures. Frontier health: queue depth by priority, age of the oldest due URL, host queue counts and the fraction of fetchers idle while waiting on politeness. Quality: duplicate rate, trap-rule hits per host, the Bloom filter fill ratio and the share of unchanged recrawls. Latency: DNS, connect, time to first byte and parse time. I alert on sudden spikes in errors from one host, URL counts per host growing out of bounds, stalled queues and storage write failures. I also keep a contact channel for webmaster complaints.

Every fetch has hard limits. There is a connect timeout and a total timeout, so a slow-drip server cannot hold a connection open. The body size is capped, and decompressed output is limited so zip bombs fail fast. There is a maximum header size and a redirect limit. Only expected content types are parsed, and others are dropped or stored as metadata only. Infinite streaming responses hit the size or time cap. Parsers run sandboxed with memory limits, because malformed HTML or PDFs can crash them. Hosts that repeatedly trigger these limits get longer delays or are blocklisted. Async I/O means one bad host ties up only its own connection, not a whole thread pool.

No. Eventual consistency is fine, which is a big part of why the system scales. If two parsers discover the same new URL at the same moment and both pass the check, the worst case is a duplicate fetch. Dedup and idempotent storage absorb it. Sharding by URL hash routes each URL to one owner, which serializes checks per key and makes races rare. Batched, asynchronous inserts are acceptable. What must be durable is the authoritative URL record and the frontier, so a crash does not cause a full re-crawl. Politeness state needs stronger guarantees, and host-based ownership gives that locally, without global coordination.

A linear scan across billions of fingerprints is impossible. The standard approach, from Manku and colleagues at Google, uses permuted tables. For 64-bit fingerprints with a threshold of 3, split the bits into blocks. By the pigeonhole principle, any match within distance 3 must exactly equal the query on some blocks. Build several tables, each sorted by a different permutation that puts some blocks first. For a query, look up each table on those leading bits, then check the full Hamming distance on the small candidate set. More tables mean fewer candidates but more storage. The tables can be sharded by prefix and queried in parallel.

Sitemaps list a site's URLs, optionally with lastmod. A sitemap file may hold up to 50,000 URLs and 50 MB uncompressed, and index files point to more sitemaps. They help discovery, especially for deep pages with few links. Fresh lastmod values can trigger early recrawls, but I weight them by how accurate each site's lastmod has been historically. Robots.txt can declare sitemap locations. Push protocols such as IndexNow let sites notify participating search engines when URLs change, which cuts wasted recrawls. All of these are hints. URLs from them still go through normalization, the seen-check, trap rules, politeness and robots.txt checks.

A soft 404 is an error or empty page served with a 200 status, for example 'product not found' with a normal template. Storing them wastes space, and recrawling them wastes budget. Detection signals include very short main content, typical error phrases and many different URLs on one host sharing a near-identical SimHash. A strong check is to fetch a deliberately random, non-existent URL on the host and compare its fingerprint to the suspect page. If they match, the page is a soft 404. Detected pages are marked as errors, kept out of the content store and recrawled rarely. Hosts that send many of them get lower budgets.

1. A304 Not ModifiedCorrect answer
2. B412 Precondition Failed
3. C200 OK
4. D204 No Content

Why: 304 Not Modified tells the client its cached version is still valid, and the response carries no body.

1. ADiscovery time
2. BPage priority score
3. CHost, so each queue holds URLs for a single hostCorrect answer
4. DContent type

Why: Back queues map to hosts so the crawler can enforce one request at a time and a delay per host.

1. AFalse positives
2. BHash collisions
3. CSaturation as it fills up
4. DFalse negativesCorrect answer

Why: If an item was inserted, all its bits are set, so a Bloom filter never reports an inserted item as absent.

1. AAbout 3,850
2. BAbout 38,500
3. CAbout 385Correct answer
4. DAbout 39

Why: One billion divided by about 2.59 million seconds is roughly 385 per second on average.

1. AAssume the site is fully disallowed until robots.txt can be fetchedCorrect answer
2. BCrawl only the home page
3. CIgnore the error and use a robots.txt from another host
4. DCrawl the whole site as if no robots.txt exists

Why: RFC 9309 treats server errors as unreachable and says the crawler must assume complete disallow. A 4xx, by contrast, lets the crawler proceed as if no rules exist.

1. AThe filter is full and must be rebuilt
2. BThe URL has definitely been crawled
3. CThe URL has definitely not been crawled
4. DThe URL was probably seen, with a small false-positive chanceCorrect answer

Why: Bloom filters have no false negatives but can have false positives, so a hit means 'probably seen'. A miss means the URL is certainly new.

1. AStop crawling the host for 30 days
2. BCrawl any resource on the hostCorrect answer
3. CAssume complete disallow
4. DRetry every minute until it succeeds

Why: A 4xx status means robots.txt is unavailable, and RFC 9309 lets crawlers access any resource.

1. AUntil the server sends a 304
2. BForever, until a 4xx is returned
3. CNo more than 24 hours, unless the file is unreachableCorrect answer
4. DUp to 7 days

Why: The RFC says crawlers should not use a cached robots.txt for more than 24 hours, except when the file is unreachable.

1. ARetry-AfterCorrect answer
2. BCache-Control
3. CETag
4. DVary

Why: Retry-After tells the client how long to wait before sending more requests.

1. ARemoving a trailing slash
2. BLowercasing the path
3. CRemoving all query parameters
4. DLowercasing the hostCorrect answer

Why: Hostnames are case-insensitive, but paths and queries can be case-sensitive or carry meaning.

1. AIt encrypts traffic between nodes
2. BIt removes the need for robots.txt
3. CIt ensures pages are fetched in priority order
4. DEach host's politeness state lives on one node, and a membership change moves only about 1/N of hostsCorrect answer

Why: Single ownership lets politeness be enforced locally, and consistent hashing limits reshuffling when nodes change.

1. AFetch it over HTTPS instead
2. BFetch it like any other URL
3. CBlock it, because it is a link-local cloud metadata address and an SSRF targetCorrect answer
4. DFetch it only if robots.txt allows it

Why: Metadata endpoints can leak credentials, so crawlers must block private and link-local addresses.

1. AA robots.txt file with many rules
2. BA sitemap index file
3. CA calendar page with endless 'next month' linksCorrect answer
4. DA 301 redirect from http to https

Why: Dynamically generated calendars can produce an unlimited number of unique URLs.

1. AExactly-once fetching with no duplicates
2. BPermanent removal of those URLs
3. CAt-least-once fetching, because expired leases make the URLs available againCorrect answer
4. DAt-most-once fetching, so those URLs are lost

Why: Leases expire and the URLs return to the queue, and idempotent writes plus dedup absorb repeat fetches.

1. ASimHash is cryptographically stronger
2. BSimilar documents produce fingerprints with small Hamming distanceCorrect answer
3. CSimHash output is shorter so it uses less disk
4. DMD5 cannot hash HTML

Why: SimHash is locality-sensitive, so pages that differ by small edits get fingerprints differing in few bits. MD5 changes completely with a one-byte change, so it only catches exact duplicates.

1. A10 MiB
2. B64 KiB
3. C500 KiBCorrect answer
4. D1 MiB

Why: Crawlers must parse at least 500 KiB of a robots.txt file.

1. ANo, because Disallow always wins
2. BNo, because the first rule in the file wins
3. CYes, because the longest matching rule winsCorrect answer
4. DOnly if Crawl-delay is set

Why: RFC 9309 uses the most specific (longest) match, and /public is longer than /.

1. ADisallow
2. BCrawl-delayCorrect answer
3. CUser-agent
4. DAllow

Why: Crawl-delay is a widely used extension, but RFC 9309 does not define it.

1. ABloom filters are read-only after creation
2. BClearing its bits could also clear bits shared by other URLs and create false negativesCorrect answer
3. CDeletion needs a full rehash
4. DThe hash functions are not reversible

Why: Bits are shared between items, so deletions need a variant such as a counting Bloom filter.

1. Ak = n/m
2. Bk = m·n
3. Ck = (m/n)·ln 2Correct answer
4. Dk = log2(n)

Why: The optimal k is (m/n)·ln 2, which leaves about half of the bits set.

1. A3Correct answer
2. B16
3. C32
4. D0

Why: Manku and colleagues found that k = 3 on 64-bit fingerprints worked well for billions of pages.

1. AUniform revisit allocation can beat proportional allocation on average freshnessCorrect answer
2. BPages should be recrawled at random
3. CRecrawl frequency does not affect freshness
4. DProportional revisits (more often for faster-changing pages) always maximize average freshness

Why: Their analysis showed that over-investing in pages that change very often hurts average freshness.

1. Ametadata
2. Bwarcinfo
3. Cresponse
4. DrevisitCorrect answer

Why: Revisit records point to an earlier capture so duplicate payloads are not stored twice.

1. ARFC 9309 requires it
2. BFragments break DNS resolution
3. CThe fragment is not sent to the server, so it does not change the fetched resourceCorrect answer
4. DFragments are always tracking parameters

Why: Fragments are handled on the client side, so URLs that differ only by fragment fetch the same resource.

1. AUse only public DNS resolvers
2. BDisable DNS caching
3. CResolve the name once, validate the IP, and connect to that exact IPCorrect answer
4. DResolve the name twice and compare the answers

Why: Connecting to the validated IP stops a second lookup from returning an internal address.

1. AAs a binding directive that must always be followed
2. BAs something to ignore
3. CAs a hint to weigh against redirects and content similarityCorrect answer
4. DAs equivalent to a robots.txt Disallow

Why: Canonical tags are often wrong, so they are treated as strong hints and cross-checked.

1. AOnly byte-identical contentCorrect answer
2. BPages with the same topic
3. CPages that differ only in ads or timestamps
4. DPages with the same title

Why: Changing even one byte changes the exact hash, so near-duplicates need SimHash.

1. ATo speed up DNS lookups
2. BTo enforce robots.txt rules
3. CTo cap how many URLs one host can add, limiting traps and crowd-outCorrect answer
4. DTo compress WARC files

Why: Budgets stop one host or a generated URL space from eating the crawl.

1. ALast-Modified
2. BContent-Length
3. CExpires
4. DETagCorrect answer

Why: If-None-Match carries the ETag from the earlier response, and If-Modified-Since uses Last-Modified.

1. AA Bloom filter
2. BA hash map keyed by URL
3. CA FIFO queue of hosts
4. DA min-heap keyed by next-allowed-fetch timeCorrect answer

Why: A min-heap returns the host that becomes eligible soonest in O(log n) per operation.

1. AZipf distribution over hosts
2. BGaussian process over page size
3. CMarkov chain over links
4. DPoisson process with a per-page change rateCorrect answer

Why: Page changes are typically modelled as a Poisson process, with λ estimated from fetch history.

1. A50,000 URLs or 50 MB uncompressedCorrect answer
2. B10,000 URLs or 10 MB
3. C100,000 URLs or 100 MB
4. D1,000 URLs or 1 MB

Why: The sitemaps protocol caps each file at 50,000 URLs and 50 MB uncompressed.

1. AThe crawler reads noindex first, then obeys Disallow
2. BThe crawler never fetches the page, so it never sees the noindex tagCorrect answer
3. Cnoindex overrides Disallow
4. DBoth are ignored

Why: Robots.txt blocks the fetch, so on-page directives such as noindex are never read.