Exa Web Crawling Engineer — Learning & Application Kit

Built September 2026. 14 documents, ~6,750 lines: every phase of the 12-week Rust + crawling roadmap, with verified links, crate versions, hands-on exercises, project specs, and Exa-specific application intel. Companion research prompts live in .research-prompts/ and raw agent logs in .research-logs/ (both safe to delete).

Read it as one page: open index.html — a single offline file with sidebar navigation, section search (/ or ⌘K), scrollspy, and dark mode. Rebuild it with node build_guide.mjs after editing any markdown.

File map

First week (start here)

1. Day 0: install the toolchain using the commands in 13-toolchain-verified-2026.md (rustup, components, cargo-nextest/watch/flamegraph).
2. Days 1–3: Rust Book + Rustlings from 01; build the URL-normalizer CLI.
3. Days 4–7: finish the 4 CLI drills (robots.txt fetcher, title extractor, CSV/JSON exporter) and start 02.

12-week schedule → files

Dependency rule from 09: B reuses A's politeness/frontier modules; C consumes A's crawl.db (≥2k pages from a permissive corpus); D quotes A+B+C numbers. Each milestone ends with a tagged release and README results table.

Overqualified track (files 16 + 17). 16 adds a 6-level Rust mastery path (~140-170h core, +40-50h extension) run as a morning track across weeks 1-12: borrow-checker katas, crawler error taxonomy, MPMC queue with Miri/loom receipts, profiling + zero-copy/SIMD passes, production-grade testing/supply chain, Tokio internals. 17 upgrades the projects: A+ adds a 100M pages/day capacity model, 5k-host politeness simulation and OPIC scheduling; B+ adds a TypeScript Playwright sidecar and challenge/anti-bot resilience; C+ adds a self-built HNSW/IVF+PQ ANN index with a 1B-vector extrapolation; E/F cover the Backend JD (mini exa-d pipeline, 10B-page search feasibility package). Sequence and cut lines are in 17 §8; the proof-of-scale checklist remains 15 §6.

Application fast facts (verify before applying — listings change weekly)

• Crawler roles were live in San Francisco ($180K–$350K + equity) and Singapore as of Sep 2026; the SF req has been open over a year, per 14.
• Referrals convert at 40–70% vs 0.1–2% for cold portals; the playbook is proof-of-work + specific cold email/X DM + PRs to exa-labs repos + their Discord.
• Resume/outreach language: "web-scale crawling," "distributed crawler," "crawl politeness / RFC 9309," "JavaScript rendering, CDP/Playwright," "high-performance Rust," "search/retrieval quality."
• Never use "anti-bot bypass" framing; say politeness-aware and responsible crawling (08 §legal, 14).
• Smart questions to ask in the interview: 14, interview-loop details in 11.

Accuracy notes

• 13 audited every pinned version and ~90 links across files 01–08 against crates.io/docs.rs on 2026-09-11; only a few genuine corrections found (e.g., console-subscriber 0.4 → 0.5, criterion repo moved to criterion-rs/criterion.rs, don't bump html5ever past scraper's pin).
• Files 01–08 were researched by 8 parallel phase agents; files 09–14 by Kimi K3 and Muse 1.3 xhigh deep-research runs. Treat job/comp specifics as perishable; everything link-based was spot-checked.

Rust Fundamentals for a Crawler Engineer

Exa's crawler team runs 100M+ pages/day in Rust/C++, so ownership/borrowing, zero-copy parsing, and predictable performance are job requirements, not trivia. Weeks 1-2 of your roadmap need to get you to "I can ship a tested CLI" — not to Rust mastery. Everything below is chosen for that single goal: enough Rust to write parsing, error-handling, and benchmarking tools, with the language rabbit holes deferred to weeks 3+.

Core path in one line: rustup -> The Book + Rustlings (parallel) -> 100 Exercises -> The Rust CLI Book -> thiserror/anyhow -> tests + proptest -> criterion. Everything else is lookup or enrichment.

Start Here (core path)

Do this, not everything: finish Book ch. 1-13 with Rustlings, do the first ~60 of 100 Exercises, then build the exercises in this file. Only then branch into videos and paid books.

Books (free + paid)

Free

Paid

Video Courses & Series

Interactive Exercises

Tooling Setup

Minimal setup, copy-paste:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup component add clippy rustfmt
cargo install bacon cargo-nextest cargo-expand
cargo new crawlkit --bin && cd crawlkit
cargo add anyhow thiserror
cargo fmt && cargo clippy -- -D warnings && cargo test

Project hygiene worth adopting immediately (Rust 1.74+):

# Cargo.toml
[lints.clippy]
unwrap_used = "warn"
todo = "warn"

Error Handling, Testing & Benchmarking

Error handling (learn this in week 1, not month 3)

Core rule: typed errors in libraries (thiserror), dynamic errors in binaries (anyhow). Use ? everywhere; unwrap() only in tests and genuinely unreachable cases.

Small, realistic pattern:

#[derive(Debug, thiserror::Error)]
pub enum CrawlError {
    #[error("invalid url: {0}")]
    Url(#[from] url::ParseError),
    #[error("robots.txt unavailable")]
    Robots(#[source] std::io::Error),
}

fn load(path: &str) -> anyhow::Result<String> {
    std::fs::read_to_string(path).context("reading robots.txt")
}

Testing

Benchmarking

Cargo + criterion skeleton:

[dev-dependencies]
criterion = { version = "0.8", features = ["html_reports"] }

[[bench]]
name = "normalize"
harness = false

use criterion::{criterion_group, criterion_main, Criterion};
use std::hint::black_box;

fn bench(c: &mut Criterion) {
    let input = "HTTPS://Exa.AI:443/a/../b?q=1#frag";
    c.bench_function("normalize", |b| b.iter(|| black_box(black_box(input))));
}

criterion_group!(benches, bench);
criterion_main!(benches);

Communities

Recommended Order (weeks 1-2 and beyond, with time estimates)

Assumes ~3 focused hours/day, ~40 hours total for Phase 1. If you have less time, cut the videos, not Rustlings or the exercises.

Week 1 — syntax, ownership, first CLIs (~20h)

Week 2 — idioms, errors, tests, benchmarks (~20h)

Phase 1 exit criteria (do not move on until all are true)

• You can split a crate into lib.rs + main.rs and organize modules without fighting the compiler.
• You explain move/copy/borrow rules and fix borrow errors without cloning everything.
• You design a thiserror enum, propagate with ?, and add .context() at binary boundaries.
• Unit + integration + at least one proptest pass; cargo clippy -- -D warnings is clean.
• A criterion benchmark has a saved baseline and a hyperfine run compares CLI before/after.
• One public GitHub repo with tests, README, and honest commit history.

Beyond weeks 1-2 (keep it time-boxed)

• Weeks 3-4 (solidify, ~15h/week): finish Command-Line Rust; Effective Rust; Comprehensive Rust; Crust of Rust (Iterators, Smart Pointers, Channels); skim mini-redis. Then start Phase 2 (async Rust/Tokio) — do not spend more than 4 weeks on pure fundamentals.
• Months 2-3 (as phases require): Rust for Rustaceans; Rust Atomics and Locks (before distributed crawler); The Rust Performance Book; Zero To Production (Phase 7 reference).

Hands-on Exercises

Put all of these in one cargo workspace named crawlkit, one crate per tool, each with a README and tests. None require network access; the real fetching starts in Phase 3.

1. unorm — URL normalizer CLI

• Reads URLs from argv or stdin (one per line); lowercases scheme/host, removes default ports (:80/:443), resolves ./.. segments, drops fragments; --json emits {original, normalized}.
• Acceptance: invalid URLs print to stderr and exit non-zero; unit tests cover IPv6 hosts, IDN/percent-encoding, query preservation; proptest proves normalize(normalize(u)) == normalize(u); never panics. Crate: url (https://github.com/servo/rust-url).

2. robots-check — robots.txt policy checker

• Parse User-agent, Allow, Disallow, Crawl-delay groups; support * wildcards and $ anchors; longest-match wins; unknown agent falls back to *; no rules = allow.
• Acceptance: fixture suite with mixed-case directives, comments, empty groups, duplicate agents; --user-agent and file args; proptest: any random rule set yields deterministic decisions and never panics. Implement the parser yourself; compare against the robotstxt crate (https://docs.rs/robotstxt) in tests.

3. title — HTML title/meta extractor

• Reads HTML from file/stdin; outputs JSON {title, description, link_count}; collapses whitespace, decodes entities, handles uppercase tags and malformed documents.
• Acceptance: 10 fixtures including truncated HTML, missing <title>, and nested comments; integration tests invoke the binary; no panic on arbitrary bytes. Crates: scraper (https://docs.rs/scraper), serde_json (https://github.com/serde-rs/json).

4. links — link extractor + deduper

• Extract <a href>, resolve relative URLs against --base, drop javascript:, mailto:, and fragment-only links, dedupe by normalized URL, sort, print one per line.
• Acceptance: fixture with protocol-relative (//host/path) and entity-encoded URLs; proptest: output has zero duplicates and every line parses as a URL; handles Unicode paths unchanged.

5. crawl-export — streaming NDJSON to CSV

• Reads crawl records {url, status, depth, fetched_at} (one JSON per line), filters status == 200, writes CSV with a stable header; malformed lines are reported to stderr with line numbers and skipped.
• Acceptance: streams 1M synthetic lines with RSS < 100 MB (measure with /usr/bin/time -l or hyperfine); unit tests for CSV escaping (quotes, commas, newlines); BufReader/BufWriter only.

6. frontier — in-memory polite crawl scheduler

• Library crate (no network): Frontier::push(url, depth) and Frontier::next(); respects per-host minimum delay and max in-flight count; BFS by depth; dedupes by normalized URL; injectable clock for tests.
• Acceptance: deterministic unit tests prove per-host spacing and BFS order; criterion benchmark scheduling 1M URLs; zero unsafe; document the design in the README as if reviewing a crawler PR.

7. fetch-fixture — minimal HTTP client against a local test server

• In tests, build a std::net::TcpListener fixture server serving canned responses; the client sends GET with User-Agent, Host, Connection: close, parses status line and Content-Length, and enforces set_read_timeout.
• Acceptance: offline tests for 200, 404, slow-server timeout, and connection reset; no external HTTP crates yet. This is the warm-up for Phase 3's reqwest/HTTP work.

8. bench-normalize — performance pass on Exercise 1

• Add a criterion bench over 100k URLs; save baseline; make one measured optimization (fewer allocations, Cow, faster percent-decoding) and document the delta.
• Acceptance: cargo bench -- --save-baseline main works in CI mode; README shows throughput and p-value from criterion plus hyperfine before/after wall times; no regression versus baseline.

Common Pitfalls / What to Skip

• Do not chase lifetime mastery in week 1. Learn elision and when annotations are required; the rest comes from reps. rustc --explain E0502 is a built-in teacher.
• Clones to get unstuck are fine; clones in hot paths are not. Do not reach for Rc<RefCell<>> to make Rust feel like Python.
• Never unwrap() I/O, network, or parser input in production code. A panic in a crawl worker silently drops work. Reserve unwrap for tests.
• Do not put anyhow in library APIs. Callers cannot match variants; use thiserror in libraries, anyhow in binaries.
• Skip macros, unsafe, pinning, and Send/Sync internals for now. They belong to weeks 6+, and crawler work rarely needs hand-written unsafe.
• Do not learn async simultaneously with ownership. Phase 2 is dedicated to Tokio for a reason.
• Do not buy a bootcamp before finishing the free path. Reviews of the ~$500 Let's Get Rusty bootcamp are mixed; free material covers fundamentals.
• Distrust pre-2021 tutorials. clap v2, criterion 0.4, lazy_static, pre-? error handling, and pre-edition-2024 idioms will teach you bad habits.
• Do not benchmark with hand-rolled Instant loops or single runs. Use criterion/divan with black_box, and hyperfine for whole binaries.
• Do not spread across five books. One core path plus lookups; more input is not more skill.
• Do not defer fmt/clippy/tests. They are part of the language culture and the interview signal.
• Skip for now: embedded, WASM/frontend, blockchain, game dev, macro-heavy crates, tokio before Phase 2, and "100-day" content-mill courses.

Open-Source Repos to Read

Read actively: start at main.rs, run cargo doc --open, use cargo expand on confusing derives, trace one feature end-to-end, and run the test suite.

Async Rust & Tokio for Crawlers

Exa's crawler team keeps 100M+ pages/day moving through a Rust/C++ stack, and the difference between "a script that fetches pages" and "a crawler that runs for months" is almost entirely async runtime engineering: bounded concurrency, backpressure, per-host politeness, timeouts, cancellation, and observability. This phase turns Tokio from a runtime you call into a system you understand, so that the later phases (polite crawling, JS fallback, distributed scheduling) have something solid to build on. Everything below is chosen for one target artifact: a concurrent fetcher that can pull 10,000 URLs without falling over.

Start Here (core path)

Work these in order. Do not start at the bottom.

Books & Long-Form Guides

Video Courses & Talks

Official Docs & Examples

Concurrency Patterns for Crawlers

The core question is always: how many requests are in flight, and what happens when one of them hangs?

1. Semaphore + JoinSet (recommended default for the 10k fetcher). Acquire the permit before spawning so the spawning loop itself applies backpressure.

use std::sync::Arc;
use tokio::{sync::Semaphore, task::JoinSet};

let sem = Arc::new(Semaphore::new(64));
let mut set = JoinSet::new();

for url in urls {
    let permit = sem.clone().acquire_owned().await.unwrap();
    set.spawn(async move {
        let _permit = permit;
        fetch(url).await
    });
}

while let Some(res) = set.join_next().await {
    match res {
        Ok(outcome) => stats.record(outcome),
        Err(join_err) => stats.record_panic(join_err),
    }
}

2. buffer_unordered (least code, no task handles). Concurrency is the buffer size; results arrive as they complete. Beware: the whole stream must be 'static or you must .collect() before locals drop.

use futures::{stream, StreamExt};

let outcomes: Vec<Outcome> = stream::iter(urls)
    .map(|url| async move { fetch(url).await })
    .buffer_unordered(64)
    .collect()
    .await;

3. Bounded worker pool (explicit backpressure). Best when the URL source is unbounded (a frontier queue). mpsc::channel(N) blocks the producer when the queue is full, which is what you want.

let (tx, rx) = tokio::sync::mpsc::channel::<Url>(1_000);
let rx = Arc::new(tokio::sync::Mutex::new(rx));

let mut workers = JoinSet::new();
for _ in 0..64 {
    let rx = rx.clone();
    workers.spawn(async move {
        loop {
            let next = { rx.lock().await.recv().await };
            match next {
                Some(url) => process(url).await,
                None => break,
            }
        }
    });
}

4. Per-host politeness. One semaphore per host + a fixed delay gives "1 in-flight per host, >=200ms apart" without a global bottleneck.

let per_host: DashMap<String, Arc<Semaphore>> = DashMap::new();
let gate = per_host
    .entry(host.clone())
    .or_insert_with(|| Arc::new(Semaphore::new(1)))
    .clone();
let _permit = gate.acquire_owned().await.unwrap();
tokio::time::sleep(politeness_delay).await;
fetch(url).await;

Choosing between them: Semaphore + JoinSet when you need per-task cancellation and metrics; buffer_unordered when you already have a finite URL list and want minimal code; bounded mpsc when the frontier is endless; FuturesUnordered (https://docs.rs/futures/latest/futures/stream/struct.FuturesUnordered.html) only when you need a set of unspawned futures for non-'static borrows.

Timeouts, Retries, Cancellation, Graceful Shutdown

Budget every network operation. A crawler without deadlines slowly fills up with hung sockets.

async fn fetch_with_deadline(client: &reqwest::Client, url: &str) -> Result<Bytes, Error> {
    tokio::time::timeout(Duration::from_secs(10), async {
        let resp = client.get(url).send().await?;
        let status = resp.status();
        if status.as_u16() == 429 || status.is_server_error() {
            return Err(Error::Retryable(status));
        }
        if status.is_client_error() {
            return Err(Error::Permanent(status));
        }
        Ok(resp.bytes().await?)
    })
    .await?
}

Retry only what deserves it, with jitter. 429/408/5xx/connect errors are retryable; 404/403 are not. Jitter prevents synchronized retry storms across 10k tasks.

let mut attempt = 0u32;
loop {
    match fetch_with_deadline(&client, url).await {
        Ok(bytes) => break Ok(bytes),
        Err(e) if e.is_retryable() && attempt < 3 => {
            let base = Duration::from_millis(200 * 2u32.pow(attempt) as u64);
            let jitter = Duration::from_millis(rand::random::<u64>() % 100);
            tokio::time::sleep(base + jitter).await;
            attempt += 1;
        }
        Err(e) => break Err(e),
    }
}

Graceful shutdown = stop accepting -> signal -> drain. The Tokio shutdown guide (above) is the authoritative walkthrough; this is the shape:

let token = CancellationToken::new();
let tracker = TaskTracker::new();

for url in urls {
    let token = token.clone();
    tracker.spawn(async move {
        tokio::select! {
            _ = token.cancelled() => { /* stop early */ }
            result = fetch(url) => stats.record(result),
        }
    });
}
tracker.close();

tokio::select! {
    _ = tokio::signal::ctrl_c() => {
        token.cancel();
        let _ = tokio::time::timeout(Duration::from_secs(5), tracker.wait()).await;
    }
    _ = tracker.wait() => {}
}
stats.print_summary();

Cancellation-safety warning: dropping a future cancels it at its current .await. That is fine mid-GET (the request is aborted), but dangerous if you're mid-write to a checkpoint file. select! picks a random ready branch unless you add biased;; only use cancel-safe operations inside select! branches (the docs label each one).

Observability (tracing, tokio-console)

Instrument the fetch, not the loop. Attach host/URL/status/bytes as span fields so you can filter and aggregate; the span name becomes the unit in tokio-console and trace viewers.

use tracing::{info, instrument};

#[instrument(skip(client), fields(host = %url.host_str().unwrap_or("unknown")))]
async fn fetch(client: &reqwest::Client, url: &url::Url) -> anyhow::Result<usize> {
    let resp = client.get(url.clone()).send().await?;
    let status = resp.status();
    let bytes = resp.bytes().await?.len();
    info!(status = status.as_u16(), bytes, "fetched");
    Ok(bytes)
}

tokio-console setup (two lines + one build flag):

# Cargo.toml
tokio = { version = "1", features = ["full", "tracing"] }
console-subscriber = "0.4"

fn main() {
    console_subscriber::init(); // replaces tracing_subscriber::fmt::init() in dev
    // ...
}

Build/run with RUSTFLAGS="--cfg tokio_unstable" cargo run and connect with cargo install --locked tokio-console && tokio-console. The console flags self-wakes, tasks that never yield, lost wakers, and large futures: exactly the failure modes of a 10k-request crawler. Keep it as a dev-only feature.

Communities

Recommended Order (weeks 3-4, with time estimates)

Assumes ~7-8 focused hours per week on this phase; adjust to your schedule but keep the dependency order.

Week 3 — Fundamentals and the runtime (~14h)

Week 4 — Patterns, production hardening, and proof (~14h)

If you only have one week, do days 1-5 and Exercises 1-2, then treat the rest as week 4 of Phase 2 continued in parallel with Phase 3.

Hands-on Exercises

Build one crate (crawler-lab) with a mock-server binary (e.g. axum/wiremock) that can inject delays, 429s, 500s, and hangs, and a fetcher binary that evolves across exercises.

1. Sequential baseline. Fetch 200 URLs sequentially; print total wall time and per-request latency.

• Acceptance: no panics; report shows p50/p95/p99 and total time; mock server logs 200 requests.
• Why: you need a number to beat for every later claim.

2. Concurrent 10k-URL fetcher with a hard concurrency cap.

• Acceptance: processes 10,000 URLs against the mock server; a shared AtomicUsize high-water mark never exceeds MAX_CONCURRENCY (start at 64, then 512); handles 404, connection refused, malformed URL, and DNS failure without exiting; prints success/error/timeout counts, p50/p95/p99 latency, total bytes, and requests/sec; p95 at least 10x better than Exercise 1.
• Uses: Semaphore + JoinSet, one shared reqwest::Client, timeout per request.

3. Bounded worker pool with visible backpressure.

• Acceptance: producer feeds 1,000,000 synthetic URLs through mpsc::channel(1_000); queue length never exceeds 1,000 (assert via a counter); RSS stays under 200 MB; all workers exit cleanly after tx.drop(); worker JoinSet is fully drained.
• Why: this is the shape of Phase 5's frontier queue.

4. Timeouts, retries, and error classification.

• Acceptance: a 404 produces zero retries; a server that fails twice then succeeds produces exactly two retries and one success; every request completes within a global deadline; delays show jitter (no two retries in a 100ms bucket share the same delay); a 429 with Retry-After: 1 waits >=1s.
• Uses: tokio::time::timeout, exponential backoff + jitter, retryable/permanent error enum.

5. Graceful shutdown on Ctrl+C.

• Acceptance: SIGINT mid-run stops new requests, in-flight requests finish (or time out) within 5s, a summary prints after draining, process exits 0; a second Ctrl+C force-exits within 200ms; tokio-console shows zero live tasks after the summary.
• Uses: CancellationToken, TaskTracker, tokio::signal::ctrl_c, select!.

6. Per-host politeness.

• Acceptance: across 100 hosts the fetcher issues at most one in-flight request per host and timestamps per host are >=200ms apart (parse a request log to prove it); global concurrency still respects the 64 cap; one artificially slow host (10s delay) does not delay other hosts; host state map doesn't grow unboundedly (evict idle hosts or accept a documented bound).
• Uses: DashMap<String, Arc<Semaphore>>, per-host delay, global semaphore.

7. Observability drill with tokio-console.

• Acceptance: every fetch emits a span with host, status, bytes, and a per-task ID; console_subscriber::init() runs under RUSTFLAGS="--cfg tokio_unstable"; deliberately inject std::thread::sleep(2s) and screenshot the blocked task in tokio-console; replace it with tokio::time::sleep and show busy-time drop; commit a report.md with both observations.
• Uses: #[instrument], tracing-subscriber EnvFilter, tokio-console.

8. Manual Future + mini executor (mental-model capstone).

• Acceptance: implement a Future that returns Ready on its third poll; write a ~30-line executor loop that polls a Vec<Box<dyn Future>> with a no-op waker until all complete; a unit test asserts exactly 3 polls; a written note (<=200 words) explains what would break without Pin and why tokio::spawn requires Send + 'static; finally, rewrite the same logic with async + Tokio and show identical output.
• Why: interviews for systems roles at companies like Exa go here.

Stretch (optional, only if week 4 goes fast): rewrite Exercise 2 using buffer_unordered and benchmark it against the JoinSet version; write a one-page comparison of throughput, memory, and error-handling ergonomics.

Common Pitfalls / What to Skip

Pitfalls that will actually bite you

1. Blocking the executor. std::thread::sleep, blocking file I/O, and heavy HTML parsing inside async fn stall every task on that worker. Rule: no more than ~10-100 microseconds between .awaits; move long sync work to spawn_blocking or rayon.
2. Holding a std::sync::MutexGuard across .await. Deadlocks and non-Send futures. Either use tokio::sync::Mutex (slower) or keep the critical section short and drop the guard before awaiting. mini-redis shows the right pattern.
3. Unbounded concurrency. 10k tokio::spawns with no semaphore, or an unbounded channel, turns into file-descriptor exhaustion and OOM. Everything gets a cap: concurrent requests, queue depth, per-host rate, retries.
4. No per-request deadline. One hung TCP connection can pin a semaphore permit forever. Every network call gets tokio::time::timeout (and reqwest gets connect_timeout).
5. New reqwest::Client per request. Kills connection pooling and TLS reuse. Create once, clone (it's an Arc inside).
6. Fighting 'static on tokio::spawn. If a task borrows locals, use JoinSet/futures::future::join_all without spawning, or move owned data in. Don't reach for unsafe or Arc<Mutex<>> first.
7. Ignoring task failures. JoinHandle returns Result<T, JoinError>; panics don't crash the process, they vanish into that error. Always match on it.
8. Synchronized retries (thundering herd). Same backoff for 10k tasks = a second self-inflicted denial of service. Add jitter and cap attempts.
9. select! cancellation surprises. Dropping a future can cancel an in-flight operation; know which branches are cancel-safe before relying on loop behavior.
10. Retrying non-idempotent mistakes. Don't retry 4xx (except 408/429); don't retry POSTs that may have succeeded. Politeness and correctness are part of crawl quality.

What to skip during weeks 3-4

• async-std / smol deep dives. Tokio is the ecosystem and the hiring market; a one-page comparison is enough.
• Actor frameworks (Actix, ractor). Learn bounded channels and task-per-component first; frameworks hide the mechanics you're being hired to know.
• Manual runtime internals beyond Exercise 8 (epoll/io_uring, pin-project internals, writing a reactor). Good Phase 6+ reading, not a hiring signal right now.
• #[async_trait] everywhere. Async fn in traits has been stable since Rust 1.75; only pull in async-trait when you need dyn compatibility.
• Blocking HTTP clients, thread-per-request designs, or Python-style async porting. This phase is about the Tokio way.
• Premature distributed design. Don't build a scheduler in Phase 2; Exercises 2-6 are the foundation Phase 6 will generalize.

Open-Source Repos to Read

Read with a question in hand ("how is the permit released?"), not cover to cover.

HTTP-First Scraping in Rust

Most of the web at Exa's scale (100M+ pages/day) must be fetched and parsed without a browser — it is cheaper, faster, and parallelizes cleanly across machines. This phase gives you the production HTTP + HTML stack for that majority: reqwest for pooled, compressed, timeout-bounded fetching; scraper/dom_query for browser-grade CSS extraction; serde for typed records; and the url crate plus content hashing for crawler-grade URL and duplicate handling. Getting this layer excellent — retries, redirects, content-type filtering, canonicalization, dedup, fixture tests — is exactly the "extreme optimization" the Exa crawler role asks for, and it is the foundation for the polite-crawling and distributed phases that follow.

Versions and links verified September 2026. Stack snapshot: reqwest 0.13.5, scraper 0.27, dom_query 0.28 (rising alternative), serde 1.0.229, serde_json 1.0.151, url 2.5.8, lol_html 3.0.1, dom_smoothie 0.18, blake3 1.8.7, wiremock 0.6.5, texting_robots 0.2.2.

Start Here (core path)

If you only do five things, do these in order (roughly one sitting each):

1. Cookbook: fetch a page with a configured reqwest::Client — pooled, timed out, gzip+brotli, real User-Agent, non-2xx classified. Official docs and examples are the ground truth.
2. Parse it with scraper — Html::parse_document, precompiled Selectors, ElementRef text/attr extraction.
3. Reshape it with serde — typed structs, Option fields, #[serde(rename_all = "camelCase")], JSONL output.
4. Normalize URLs with the url crate — Url::parse + join, then your own policy layer (fragments, tracking params, trailing slash).
5. Test everything offline — fixture HTML with include_str!, response mocking with wiremock, snapshot assertions.

Core Crates (reqwest, scraper, serde, url, etc.)

The dependency set

# Cargo.toml — versions current as of Sep 2026
[dependencies]
tokio      = { version = "1", features = ["macros", "rt-multi-thread", "time", "sync", "fs"] }
reqwest    = { version = "0.13", features = ["gzip", "brotli", "zstd", "deflate", "cookies", "json", "stream"] }
scraper    = "0.27"
serde      = { version = "1", features = ["derive"] }
serde_json = "1"
url        = "2.5"
mime       = "0.3"
blake3     = "1.8"
thiserror  = "2"
anyhow     = "1"

[dev-dependencies]
wiremock          = "0.6"
insta             = "1"
pretty_assertions = "1"

Notes that save hours:

• reqwest 0.13 defaults: charset, http2, system-proxy, and default-tls now resolves to rustls (with rustls-platform-verifier). The old "works with curl, fails in Rust" corporate-TLS story is mostly gone; still verify against real hosts in containers.
• gzip/brotli/zstd/deflate enable automatic decompression: reqwest sets Accept-Encoding if you haven't and strips Content-Encoding/Content-Length from decompressed responses.
• stream enables Response::bytes_stream() — required for lol_html streaming later in this guide.
• serde_json and blake3 are pulled in directly; don't re-export them from the HTTP layer.

reqwest: the one Client factory you will reuse everywhere

Create one Client per process/worker and clone it (it is an Arc internally). A new Client per request destroys connection pooling and TLS session reuse.

use std::time::Duration;
use reqwest::{Client, redirect::Policy};
use reqwest::header::{HeaderMap, HeaderValue, USER_AGENT, ACCEPT, ACCEPT_LANGUAGE};

pub const BOT_UA: &str = "FerrisCrawler/0.1 (+https://example.com/bot; contact@example.com)";

pub fn build_client() -> anyhow::Result<Client> {
    let mut headers = HeaderMap::new();
    headers.insert(ACCEPT, HeaderValue::from_static(
        "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.1"));
    headers.insert(ACCEPT_LANGUAGE, HeaderValue::from_static("en-US,en;q=0.9"));

    let client = Client::builder()
        .user_agent(BOT_UA)
        .default_headers(headers)
        .connect_timeout(Duration::from_secs(5))
        .read_timeout(Duration::from_secs(15))      // per-read, catches stalled sockets
        .timeout(Duration::from_secs(30))           // total deadline; default is NONE
        .pool_idle_timeout(Duration::from_secs(90)) // default 90s
        .pool_max_idle_per_host(16)                 // default usize::MAX — cap it
        .redirect(Policy::limited(5))               // default 10; be explicit
        .cookie_store(true)                         // cookies feature
        .https_only(true)
        .gzip(true).brotli(true).zstd(true).deflate(true)
        .build()?;
    Ok(client)
}

Facts to internalize:

• No default timeout. A crawler without .timeout(...) eventually hangs forever. Always set both total and connect timeouts; add read_timeout for stalled bodies.
• Redirects are followed up to 10 by default and silently. Use Policy::limited(n) and record the final URL (response.url()), because the redirect target — not your input URL — is what actually got fetched (and it may be off-domain).
• Cookies are off by default. .cookie_store(true) needs the cookies feature; required for session/logged-in flows and for sites that bounce you through a cookie wall.
• Content-type filtering happens after the response. Check before parsing:

use reqwest::header::CONTENT_TYPE;

let resp = client.get(url).send().await?;
if !resp.status().is_success() {
    anyhow::bail!("HTTP {} for {}", resp.status(), url);
}
let ctype = resp.headers().get(CONTENT_TYPE)
    .and_then(|v| v.to_str().ok()).unwrap_or("");
if !ctype.starts_with("text/html") && !ctype.starts_with("application/xhtml+xml") {
    anyhow::bail!("skipping non-HTML content-type: {ctype}");
}
let body = resp.bytes().await?;                // bytes first, not text()
let html = String::from_utf8_lossy(&body);     // or decode via charset/encoding_rs

• text() vs bytes(): text() decodes using the Content-Type charset (the charset feature is on by default). For crawling, prefer bytes() so you can hash the exact payload, enforce size caps, and control charset recovery (encoding_rs + meta sniffing) yourself.
• Retries: reqwest 0.13 added a built-in, scoped retry policy (ClientBuilder::retry, module reqwest::retry) that retries protocol NACKs by default with a 20% request budget. For crawl-specific status handling (429/503 with Retry-After), either configure that classifier or layer reqwest-retry/backoff on top. See reqwest::retry.
• reqwest::blocking is a trap inside async code (it panics/deadlocks in a Tokio runtime). This phase is async end to end.

scraper: selectors, text, attributes

use scraper::{Html, Selector};
use std::sync::LazyLock;

static ITEM: LazyLock<Selector> = LazyLock::new(|| Selector::parse("article.product_pod").unwrap());
static LINK: LazyLock<Selector> = LazyLock::new(|| Selector::parse("h3 a").unwrap());

#[derive(Debug, serde::Serialize)]
struct Item { title: String, url: String, price_gbp: f64 }

fn parse_items(html: &str, base: &url::Url) -> Vec<Item> {
    let doc = Html::parse_document(html);
    doc.select(&ITEM).filter_map(|el| {
        let a = el.select(&LINK).next()?;
        let title = a.value().attr("title")?.to_string();
        let href = a.value().attr("href")?;
        let url = base.join(href).ok()?.to_string();
        let price_gbp = el.text().collect::<String>()
            .split('£').nth(1)?.trim().parse().ok()?;
        Some(Item { title, url, price_gbp })
    }).collect()
}

Rules:

• Selector::parse is fallible — parse once into LazyLock<Selector> statics or prepare them in a struct; never inside per-element loops.
• element.text() yields descendant text nodes; element.value().attr("href") gets attributes; ElementRef::inner_html()/html() serialize. See ElementRef.
• scraper::Html is not Send (Tendril-backed). Extract owned Strings in a synchronous scope, then await; never hold Html across .await.
• Html::parse_document takes &str only — decode bytes yourself if the page is not UTF-8.
• Always resolve relative URLs with Url::join against the final response URL (post-redirect).

serde: typed records, not Value soup

use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
struct PageRecord {
    url: String,
    final_url: String,
    status: u16,
    title: Option<String>,               // absent selector -> None, no panic
    lang: Option<String>,
    #[serde(default)]
    links: Vec<String>,
    #[serde(skip_serializing_if = "Option::is_none")]
    canonical: Option<String>,
}

// JSONL output: one record per line, append-only, crash-friendly.
let mut out = std::io::BufWriter::new(std::fs::File::create("pages.jsonl")?);
for rec in records {
    serde_json::to_writer(&mut out, &rec)?;
    use std::io::Write; writeln!(out)?;
}

Use Value only for unknown/growing shapes (JSON-LD, API responses); convert to typed structs as soon as the shape stabilizes. Beware #[serde(flatten)] with Value — it forces buffering and is slower; never combine with deny_unknown_fields.

Extraction Patterns (selectors, text, JSON-LD, readability)

Selector patterns worth memorizing

Edge case that bites everyone: link[rel="canonical"] also matches rel="canonical alternate" and casing variants. The HTML parser normalizes rel token casing; treat the attribute as a whitespace-separated token list and test fixtures for REL="Canonical".

Canonical link extraction (do it on every page)

fn canonical_of(html: &str, base: &url::Url) -> Option<url::Url> {
    let doc = Html::parse_document(html);
    let sel = Selector::parse(r#"link[rel~="canonical"][href], meta[property="og:url"][content]"#).ok()?;
    for el in doc.select(&sel) {
        let raw = el.value().attr("href").or_else(|| el.value().attr("content"))?;
        if let Ok(u) = base.join(raw.trim()) { return Some(u); }
    }
    None
}

This matters for a search engine: the canonical URL is the identity a page claims for itself, and it is a strong dedup signal alongside your own normalization.

JSON-LD / schema.org extraction

Modern sites embed structured data as <script type="application/ld+json">. Parse it as Value, then match on @type; @graph wrappers are common.

use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct ArticleLd {
    #[serde(rename = "@type")]
    kind: TypeOrVec,
    headline: Option<String>,
    author: Option<AuthorOrVec>,
    datePublished: Option<String>,
    url: Option<String>,
}
// Start loose: Value -> find object with @type Article/NewsArticle/WebPage,
// then deserialize that subtree into a typed struct.
fn first_json_ld(html: &str) -> Option<serde_json::Value> {
    let doc = Html::parse_document(html);
    let sel = Selector::parse(r#"script[type="application/ld+json"]"#).unwrap();
    doc.select(&sel)
        .filter_map(|s| serde_json::from_str::<serde_json::Value>(s.inner_html().as_ref()).ok())
        .next()
}

Treat malformed JSON-LD as expected: a huge fraction of the web ships invalid JSON in these scripts. Never fail a page because of it.

Readability: main-content extraction with dom_smoothie

For "boilerplate to clean text" (the exact preprocessing Exa does before embedding), dom_smoothie is a faithful port of Mozilla's Readability.js with a serde feature and a test corpus you can reuse.

use dom_smoothie::{Article, Config, Readability};

let cfg = Config { max_elements_to_parse: 9000, ..Default::default() };
let mut r = Readability::new(html, Some(final_url.as_str()), Some(cfg))?;
let article: Article = r.parse()?;
println!("{} | {} chars", article.title, article.length);
let text = article.content.unwrap_or_default();   // cleaned HTML

Also in the family: dom_query (same author) as a scraper alternative with :has, :contains, :only-text, base-URI lookup, and a markdown feature that serializes a document to Markdown — very relevant to AI-facing extraction.

URL Canonicalization & Deduplication

What url gives you for free, and what is your policy

url implements the WHATWG URL Standard, so Url::parse already applies spec normalization: lowercases scheme and host, drops default ports (:80/:443), resolves dot segments (/a/../b), IDNA/punycode-encodes hosts, percent-encodes consistently, and serializes empty paths as /. Url implements Hash/Eq/Ord on that normalized serialization, so it is a valid dedup key as-is.

The remaining decisions are crawler policy, not spec: fragments, tracking params, query ordering, trailing slashes, www vs apex, index.html vs directory. Put them in one function with tests:

use url::Url;

pub fn normalize(url: &Url) -> Url {
    let mut u = url.clone();
    u.set_fragment(None);                       // never sent to servers anyway

    // Drop tracking/session params; keep the rest in deterministic order.
    let kept: Vec<(String, String)> = u.query_pairs()
        .filter(|(k, _)| !matches!(k.as_ref(),
            "utm_source"|"utm_medium"|"utm_campaign"|"utm_term"|"utm_content"
            |"fbclid"|"gclid"|"mc_cid"|"mc_eid"|"sessionid"|"sid"))
        .map(|(k, v)| (k.into_owned(), v.into_owned()))
        .collect();
    if u.query().is_some() {
        let mut q = u.query_pairs_mut();
        q.clear();
        let mut kept = kept;
        kept.sort();
        for (k, v) in kept { q.append_pair(&k, &v); }
    }

    // Optional policy: strip a trailing "index.html" and collapse "//".
    // Do NOT strip trailing slashes site-wide: most servers treat /a and /a/ as different.
    u
}

Normalization checklist — each item is a test case:

Url::join is the other workhorse: resolve every href against the base before dedup, or you will store /about, about, and https://site/about as three pages.

Two-layer dedup

Layer 1 — URL identity (before fetch). Keep a HashSet<Url> (or DashMap across tasks) of normalized URLs. At scale, a Bloom filter (fastbloom::AtomicBloomFilter, concurrent and serializable) gives approximate membership in bits-per-URL; false positives only cost you a missed fetch, never a duplicate. Real crawlers shard this by host (Common Crawl's SURT format reorders https://a.com/x → com,a)/x so same-site URLs cluster; useful once you go distributed).

Layer 2 — content identity (after fetch). Hash the payload and skip storing duplicates served at different URLs (mirrors, session URLs, pagination aliases).

use blake3;

// Hash normalized content, not raw bytes: strips volatile template bits.
let text = dom_smoothie_text_or_fallback(&html);
let digest = blake3::hash(text.as_bytes());   // 32 bytes, fast, parallelizable
let key = digest.to_hex().to_string();

Practical notes:

• Hash normalized text (post-boilerplate) for semantic dedup, but also keep a raw-HTML hash to detect true byte-identical duplicates cheaply.
• A HashMap<ContentHash, Url> gives you a canonical URL for every duplicate ("revisit" semantics). Browsertrix, the production crawler from Webrecorder, does exactly this with SHA-256 payload digests.
• Near-duplicates (same article, different ad slots) need SimHash/MinHash or embeddings — out of scope here, but content hashing is the hook you will extend.
• Test your normalizer as a table and assert idempotence: normalize(normalize(u)) == normalize(u).

Testing Scrapers (fixtures, wiremock, selector tests)

The rule for this phase: network is tested separately from parsing. Parsing tests are pure functions over strings; network behavior is tested against a local mock server. Unit tests should never touch the real web.

1. Fixture HTML + golden outputs

tests/
  fixtures/
    books_list.html
    malformed_unclosed_tags.html
    jsonld_article.html
    canonical_variants.html
  parse_books.rs

fn parse_fixture() -> Vec<Item> {
    let html = include_str!("fixtures/books_list.html");
    parse_items(html, &url::Url::parse("https://books.toscrape.com/").unwrap())
}

#[test]
fn extracts_every_card() {
    let items = parse_fixture();
    assert_eq!(items.len(), 20);
    assert_eq!(items[0].title, "A Light in the Attic");
    assert!(items.iter().all(|i| i.url.starts_with("https://books.toscrape.com/")));
}

#[test]
fn malformed_html_never_panics() {
    let html = include_str!("fixtures/malformed_unclosed_tags.html");
    let _ = parse_items(html, &url::Url::parse("https://example.com").unwrap());
}

Save fixtures with the date and source URL in an HTML comment; when a site redesigns, you add a new fixture rather than mutating the old one — that is how you keep regression coverage across layouts.

2. Mock the HTTP boundary with wiremock

use wiremock::{MockServer, Mock, ResponseTemplate};
use wiremock::matchers::{method, path, header_exists};

#[tokio::test]
async fn sends_bot_ua_and_accepts_gzip() {
    let server = MockServer::start().await;
    Mock::given(method("GET"))
        .and(path("/page"))
        .and(header_exists("user-agent"))
        .respond_with(ResponseTemplate::new(200)
            .set_body_raw("<html><title>ok</title></html>", "text/html"))
        .expect(1)
        .mount(&server)
        .await;

    let client = build_client().unwrap();
    let resp = client.get(format!("{}/page", server.uri())).send().await.unwrap();
    assert_eq!(resp.status(), 200);
}

#[tokio::test]
async fn retries_then_succeeds() {
    let server = MockServer::start().await;
    // First two calls fail, third succeeds. wiremock >= 0.6 supports sequence responders.
    Mock::given(method("GET")).and(path("/flaky"))
        .respond_with(ResponseTemplate::new(503))
        .up_to_n_times(2)
        .mount(&server).await;
    Mock::given(method("GET")).and(path("/flaky"))
        .respond_with(ResponseTemplate::new(200).set_body_string("done"))
        .mount(&server).await;

    // ... call your fetch_page_with_retry(&client, url) ...
    // assert attempts == 3 and body == "done"
}

Testing strategy checklist:

• Selector tests: one fixture per real layout; assert exact field values, not just counts.
• Negative tests: missing selector, empty element, malformed HTML, non-UTF-8 page.
• Network tests: redirect chain, 429 with Retry-After, 503-then-200, timeout, non-HTML content-type, gzip/brotli body.
• Dedup tests: alias URLs collapse to one key; identical bodies from two URLs produce one content record.
• Snapshot the full Vec<Record> JSON per fixture; diffs make site changes obvious.

Performance (streaming, compression, connection pooling)

Connection pooling and concurrency

• Reuse one Client; keep pool_max_idle_per_host high enough for your per-host concurrency (e.g., 16-32) and pool_idle_timeout at 90s (default). HTTP/2 (default feature) multiplexes one connection for many requests to the same host.
• Use futures::stream::buffer_unordered(n) (or FuturesUnordered) for global concurrency, and a per-host tokio::sync::Semaphore for politeness. That per-host gate is also where rate limiting lands in Phase 4.
• tcp_nodelay is on by default; leave it.
• Consider .hickory_dns(true) (feature hickory-dns) when crawling many domains from one process to avoid blocking getaddrinfo calls.

Compression

• Enable gzip, brotli, zstd, deflate. Brotli typically saves 15-25% over gzip on HTML at a comparable crawl speed; zstd wins on decompression speed. Let reqwest negotiate via Accept-Encoding.
• For manual pipelines (e.g., tee-ing raw bytes to storage while decompressing), use async-compression with Tokio codecs.
• Budget CPU: decompression at 100M pages/day is a real line item. Measure before choosing to prefer brotli-only.

Streaming parsing (the big memory win)

If you only need links or a few fields, do not buffer the whole body. reqwest gives you a byte stream and lol_html consumes it incrementally, running CSS-selector handlers as chunks arrive.

use futures_util::StreamExt;
use lol_html::{element, HtmlRewriter, Settings};

pub async fn stream_links(client: &reqwest::Client, url: &str) -> anyhow::Result<Vec<String>> {
    let resp = client.get(url).send().await?.error_for_status()?;
    let mut body = resp.bytes_stream();

    let mut links: Vec<String> = Vec::new();
    let mut rewriter = HtmlRewriter::new(
        Settings::new().element_content_handlers(vec![element!("a[href]", |el| {
            if let Some(href) = el.get_attribute("href") {
                links.push(href);
            }
            Ok(())
        })]),
        |_chunk: &[u8]| {}, // discard output; we only collect side effects
    );

    while let Some(chunk) = body.next().await {
        rewriter.write(&chunk?)?;
    }
    rewriter.end()?;
    Ok(links)
}

Why this matters for Exa-scale crawling: peak memory becomes O(chunk + extracted state) instead of O(page), the first links are available before the last byte lands, and you can abort a page mid-stream the moment it is clearly low-value (spam, wrong language, huge payload).

Cap your inputs regardless: reject on Content-Length when present, and enforce a hard byte cap on the decompressed stream (a Content-Encoding zip bomb can otherwise expand hundreds of times).

Conditional requests (free bandwidth savings)

Store ETag and Last-Modified per URL; send If-None-Match/If-Modified-Since on refreshes. A 304 Not Modified costs almost nothing and keeps content hashes stable.

let resp = client.get(url)
    .header(reqwest::header::IF_NONE_MATCH, etag)
    .send().await?;
if resp.status() == reqwest::StatusCode::NOT_MODIFIED { /* keep prior record */ }

Recommended Order (Weeks 5-6)

Assumes your Rust basics and Tokio phase are done (you can write async fns and use Result fluently). Total ≈ 26-30 focused hours.

Week 5 — Fetch, parse, type, normalize (≈ 14h)

Week 6 — Hardening, dedup, streaming, structure (≈ 14h)

Hands-on Exercises

Each exercise is scoped for 1-3 hours and maps directly to a crawler-team concern.

Exercise 1 — Harden the client factory (2h) Build build_client() with: bot UA, Accept/Accept-Language defaults, connect/read/total timeouts, Policy::limited(5), cookie store, all four compression features, https_only(true). Acceptance: wiremock tests assert the outgoing user-agent contains a contact URL, gzip is negotiated, a redirect loop fails within the limit, a server that stalls past read_timeout returns your Timeout error, and non-2xx is classified (retryable vs terminal) without panicking.

Exercise 2 — Scrape 5 mostly-static sites into clean JSON with retries/gzip/redirects/timeout budgets/content-type filtering (3h) Targets: books.toscrape.com (catalogue + pagination), quotes.toscrape.com, www.scrapethissite.com/pages/simple/ (countries), Hacker News front page (30 links + points), www.paulgraham.com/articles.html (essay index). Acceptance: one JSONL record per page with url, final_url, status, title, links[], and site-specific typed fields; relative links resolved absolute; non-HTML responses skipped with a logged reason; the run completes five sites in under 60s total; exit code non-zero if any site fails permanently; a rerun produces identical records modulo volatile fields (snapshot-tested).

Exercise 3 — URL normalizer + torture table (2h) Implement normalize() and a table-driven test with at least 15 (input, expected) cases. Acceptance: covers scheme/host case, default ports, dot segments, IDNA host, fragment stripping, tracking-param removal, empty-path slash, and query sorting; a proptest asserts idempotence; invalid URLs fail with a typed error; 10 real cases from crawler-test.com's URL duplication pages are copied into fixtures.

Exercise 4 — Two-layer dedup index (2h) Add URL-set dedup before fetch and content-hash dedup after fetch. Output a canonical_url -> [alias_urls] map plus duplicate stats. Acceptance: three aliases of the same URL cause one fetch; two different URLs returning byte-identical bodies produce one content record with one alias; different content types never collide; a test uses wiremock to verify only one HTTP request occurs for aliased URLs; stats report url_dupes, content_dupes, fetched.

Exercise 5 — Resilient fetch: backoff, Retry-After, budget (2.5h) Implement fetch_with_retry with jittered exponential backoff, terminal vs transient status classification, Retry-After parsing for 429/503, and a max-attempts budget. Acceptance: wiremock sequence 503,503,200 yields success with exactly 3 calls; 429 with Retry-After: 1 waits at least 1s (assert via tokio::time::pause/pause-aware sleeps or recorded attempt timestamps); 404 returns immediately with no retry; attempts never exceed the budget for a permanently failing host.

Exercise 6 — Selector fixture suite (2h) Create tests/fixtures/ with well-formed, malformed, missing-field, and "site redesign" variants for one target, plus insta snapshots of the extracted Vec<Record>. Acceptance: every fixture has a passing test; malformed HTML never panics; a fixture where the main selector matches nothing returns an empty Vec and logs a warning (no unwrap); cargo insta test shows zero pending snapshots; one test proves selectors are compiled once (static LazyLock) by reusing them across 1,000 iterations.

Exercise 7 — Streaming extractor with a memory ceiling (2.5h) Use bytes_stream + lol_html to extract all links from a generated large HTML fixture (concatenate a page 10,000x, ~20-50MB) served by a local file server or wiremock. Acceptance: links emitted before end() is called; a unit test where a tag is split across two chunks still extracts the href; peak RSS (measure with /usr/bin/time -l on macOS or -v on Linux) stays under ~100MB while the full parse run does; running the same extraction with Html::parse_document on the full body for comparison shows materially higher memory.

Exercise 8 — Readability + structured data pipeline (2.5h) For 3 fixture pages, extract readable article text with dom_smoothie, parse Article/NewsArticle JSON-LD with serde, and extract the canonical/og:url with the precedence rule: link[rel=canonical] > og:url > final response URL. Acceptance: title and body match golden snapshots; JSON-LD given as @graph and as a top-level array both parse; invalid JSON-LD is skipped without failing the page; canonical precedence is covered by three tests including conflicting tags; extracted text length is reported and under a sane cap.

Definition of done for the phase: a crawler-http crate whose cargo test suite passes offline, plus a CLI that scrapes the exercise-2 sites to JSONL with dedup stats — this is the artifact you show in your portfolio from Phase 3.

Common Pitfalls / What to Skip

Do not use / avoid:

• select (0.6.1) — not formally deprecated, but effectively legacy: last release Mar 2025 (previous Dec 2022) and it still depends on html5ever 0.26, which the ecosystem lists as obsolete. scraper (0.27, html5ever 0.39) and dom_query are the maintained paths. Read it for history; don't start new code on it. https://github.com/utkarshkukreti/select.rs
• kuchiki — explicitly unmaintained on lib.rs (last release Aug 2020); it was the ancestor of scraper. Do not use.
• Raw html5ever — use only if you need tree-builder hooks; scraper/dom_query give you selectors without the boilerplate.
• reqwest::blocking in async code — panics inside a Tokio runtime. Use async, or keep blocking strictly in sync CLI tools/tests.
• spider (the full crawler framework) as your Phase 3 tool — it is excellent and you should read it, but adopting it now skips the primitives (scheduling, politeness, dedup) you need to demonstrate. Build the HTTP layer yourself first; graduate to spider in Phase 5 when you need CDP fallback and distributed workers.

Open-Source Repos to Read

Ordered by reading value for this phase. Read actively: run their tests, trace one page from bytes to parsed struct.

Companion rationale: Exa's crawler posting asks for "distributed crawler at 100M+ pages/day", "crawl politeness", "JS rendering", and "extreme optimization" in Rust/C++. This phase covers the HTTP-first substrate all of that sits on: a pooled client that never hangs, typed extraction that never panics, canonical identity for every URL, content hashing for dedup, and streaming parsing that keeps memory flat. Browser/CDP fallback, per-host scheduling, and politeness are built directly on these primitives in the next phases.

Polite Crawling & URL Frontier Design

Exa runs a web-scale crawler (100M+ pages/day) and its crawler engineers are judged on whether the system is polite, resumable, and scheduler-smart at that scale — not merely fast. This phase is where you learn the contract between a crawler and the web: robots.txt/sitemaps, host-partitioned frontiers, adaptive rate limiting, conditional GETs, and durable crawl state. Everything here is Rust-first, with classic papers and other-language crawlers included because their designs transfer directly.

Start Here (core path)

If you only do six things, do these in order. Everything else in this file is reference material.

Context for the target role: Exa crawler postings and the team's public writing are at https://exa.ai/careers (postings) and https://exa.ai/blog (engineering posts). Read the careers page once to anchor your portfolio narrative.

Robots.txt, Sitemaps & Standards

Key implementation rules to encode (verify each against RFC 9309 before shipping):

• Match the most specific User-agent group (longest product-token match); fall back to *.
• Path rules are prefix matches supporting * and $; the longest matching rule wins, and Allow wins ties.
• Crawl-delay is not in RFC 9309 — honor it when present (many sites and Common Crawl rely on it), but never treat its absence as permission to hammer a host.
• Cache parsed robots per host for at most 24 h; on 4xx (except 429) treat as allow-all; on 5xx treat as temporarily unreachable and back off while using the cached copy if available.
• Cap robots.txt reads at 500 KiB and fail closed on malformed input.
• Always send a descriptive User-Agent, e.g. YourBot/0.1 (+https://yourdomain.example/bot).

Crawler Architecture & URL Frontier

Design summary to implement:

• Front queues implement priority (depth, URL quality, lastmod, change rate). K priority levels, biased random selection so high-priority queues drain faster without starving the rest.
• Back queues implement politeness: one queue per host, at most one in-flight request per host, and a per-host next_allowed_at.
• A host→queue table routes new URLs to the right back queue; a min-heap of next-allowed times picks the next eligible host.
• Back queues should outnumber workers (~3× per Mercator) so workers are never idle while a host cools down.
• Frontier size outgrows RAM: keep head/tail buffers in memory and spill the bulk to SQLite/RocksDB.
• In a distributed crawler, hash by host (not URL) so all state for a host stays on one node.

Rate Limiting, Backoff & Politeness

Politeness policy to implement:

1. Per-host token bucket (governor keyed limiter or your own) with a conservative default (e.g. 1 rps) and per-host overrides from Crawl-delay.
2. Adaptive AIMD: on 429/5xx multiply the host rate by ~0.5 (multiplicative decrease); after sustained success add back slowly (additive increase) up to a cap. Scrapy's latency/N is the same idea using response time as the signal.
3. 429 handling: sleep at least Retry-After (parse both formats); do not cap server instructions downward. If Retry-After exceeds your budget, durably reschedule the URL instead of hot-looping.
4. 5xx / connection errors: retry with capped exponential backoff plus full jitter (sleep = rand(0, min(cap, base·2^attempt))), max attempts and max elapsed time both bounded.
5. 4xx: 404/410 are terminal — remove/mark URLs; 408/429 are retryable.
6. Adaptive per-host concurrency: start at 1, grow only while latency and error rate stay healthy (Scrapy's target-concurrency model).
7. Never retry until the retry storm is jittered — 1000 goroutines waking at the same instant is a self-inflicted DDoS.

Conditional GETs, Caching & Freshness

Freshness policy to implement:

• Store etag and last_modified per URL; send If-None-Match and, when absent, If-Modified-Since.
• On 304: update last_fetched_at, do not touch content, and treat it as a success signal for rate adaptation.
• On 200: update validators, content_hash, and change counters used for scheduling.
• On 404/410: schedule deletion/removal after N consecutive failures.
• Adaptive recrawl interval: if a URL changed, shorten its interval; after repeated 304s, lengthen it geometrically (cap it). Seed initial intervals from sitemap lastmod when available.
• 304s also count against politeness — a 304 is still a request; keep per-host limiting in force.

Persistent Crawl State (SQLite/Postgres in Rust)

Minimal frontier schema that survives kill -9:

PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL;
PRAGMA busy_timeout = 5000;

CREATE TABLE hosts (
  host           TEXT PRIMARY KEY,
  next_allowed_at INTEGER NOT NULL DEFAULT 0,  -- unix millis, monotonic-ish
  rate_per_sec   REAL NOT NULL DEFAULT 1.0,
  robots_state   TEXT NOT NULL DEFAULT 'unknown', -- ok|allow_all|unreachable
  robots_fetched_at INTEGER
);

CREATE TABLE urls (
  id            INTEGER PRIMARY KEY,
  url           TEXT NOT NULL UNIQUE,
  host          TEXT NOT NULL REFERENCES hosts(host),
  priority      INTEGER NOT NULL DEFAULT 5,
  state         TEXT NOT NULL DEFAULT 'pending',   -- pending|in_progress|done|failed|gone
  next_fetch_at INTEGER NOT NULL DEFAULT 0,
  retry_count   INTEGER NOT NULL DEFAULT 0,
  etag          TEXT,
  last_modified TEXT,
  last_status   INTEGER,
  content_hash  TEXT
);
CREATE INDEX urls_due   ON urls(state, next_fetch_at, priority);
CREATE INDEX urls_host  ON urls(host, state);

Operational rules: one writer connection (SQLite allows only one at a time — batch enqueues in transactions), N reader connections via deadpool-sqlite; claim work with UPDATE ... WHERE state='pending' AND next_fetch_at <= ? LIMIT 1 RETURNING * inside a transaction; checkpoint hosts.next_allowed_at after every fetch; keep in_progress rows with a lease timestamp so a crashed worker's claims can be reclaimed. Move to Postgres via SQLx only when a second crawler node appears.

Open-Source Crawlers to Study

Note on "crawlee-rust": there is no official Crawlee port for Rust as of September 2026 (the apify/crawlee-rust repository does not exist and no crawlee crate is published). The Rust ecosystem analogues are spider-rs/spider and Apify's Rust component impit (https://github.com/apify/impit), which is the browser-impersonation client Crawlee's TS stack itself uses. Study Crawlee's design, implement it on spider/Tokio.

Recommended Order (weeks 7–8, with time estimates)

Assumes the Rust/Tokio/HTTP phases are done. Budget ~14 h/week; all "deliverable" items are concrete files in your crawler repo.

Week 7 — robots, discovery, frontier

Week 8 — politeness hardening, freshness, persistence

Hands-on Exercises

Each exercise builds on the previous one in a single polite-crawler repo. Acceptance criteria are testable in CI without live network access unless stated.

1. Robots gatekeeper (2–3 h) Build RobotsCache::new(host) that fetches /robots.txt once per host, parses with texting_robots, and answers allowed(url). Acceptance: passes a fixture suite covering wildcard/$ paths, longest-match and Allow-wins-ties, case sensitivity, missing file (allow all), 5xx (disallow + backoff), 429 (retry), >500 KiB truncation, and 24 h cache expiry — all with a mocked HTTP client.

2. Sitemap-first URL discovery (2–3 h) Given a domain, discover seed URLs via (a) Sitemap: lines in robots.txt, then (b) /sitemap.xml fallback; recurse through sitemap indexes; parse lastmod. Acceptance: on a fixture site with a gzipped sitemap index, discovers every URL exactly once (normalized, fragments stripped, 50k-per-file limit enforced); stores lastmod for scheduling.

3. Host-partitioned URL frontier (4–6 h) Implement front queues per priority and one back queue per host with a next_allowed_at heap. Workers pull only from hosts that are due; never two concurrent fetches to one host. Acceptance: property test with a mocked clock: load 10 000 URLs across 50 hosts; assert zero host-overlap, per-host order preserved, and higher-priority URLs complete earlier (p95 latency lower) than low-priority.

4. Adaptive politeness limiter (3–4 h) Start at 1 rps/host; apply Crawl-delay when present; on 429 parse Retry-After (both formats) and sleep max(backoff, Retry-After); on 5xx halve the rate; after 20 consecutive 2xx/304s, increase rate by 10% up to 5 rps. Acceptance: against a local mock server that 429s above 2 rps, the crawler converges below 2 rps within 10 s; a Retry-After: 2 response is honored within ±100 ms; unit tests pin every transition.

5. Backoff correctness with full jitter (1–2 h) Wrap fetches in retry logic for 5xx/timeouts only; 404/410 are terminal; max 5 attempts or 60 s total. Acceptance: unit tests with seeded RNG assert each sleep ∈ [0, min(cap, base·2^n)]; 404 attempts == 1; a timeout sequence never exceeds the 60 s budget; all retries are recorded per host in stats.

6. Conditional GET + freshness scheduling (2–3 h) Persist etag/last_modified; send validators on every re-crawl; on 304 bump freshness without refetching the body; implement recrawl interval adaptation (shorten on change, geometric lengthening on 304 streaks). Acceptance: second pass over an unchanged fixture returns ≥90% 304s and zero body downloads; a URL modified between passes gets a 200 and its interval halves; intervals are visible in the DB and a report.

7. Persistent frontier in SQLite (4–5 h) Port the frontier to SQLite with WAL, claim/lease semantics, and host rate state. Acceptance: start a crawl, kill -9 mid-run, restart: no URL is fetched twice (verified by server-side request log), no duplicate enqueue, and total pending+done counts match the input URL count; write throughput ≥ 1 000 enqueues/s in a micro-benchmark.

8. Capstone: polite domain crawler + run report (3–4 h) Crawl 5 small, crawl-friendly real sites (e.g. your own site, example.com, a docs site) to depth 3 with all features enabled; emit per-host stats. Acceptance: zero robots violations, zero per-host rate violations (verified from the request log), ≥95% of fetches succeed or 304, all state persisted, Ctrl-C mid-crawl resumes cleanly, and crawl-report.md lists requests/host, 429/5xx counts, 304 rate, average delay, and top disallowed paths hit from links.

Common Pitfalls / What to Skip

• Don't build anti-bot evasion for this role. Stealth plugins, proxy rotation to dodge blocks, CAPTCHA solvers, and TLS fingerprint spoofing are Phase 5+ questions at best and ethically/legally fraught; Exa's crawler team hires for politeness, reliability, and scheduling at web scale. Getting blocked is a politeness failure, not a fingerprinting problem.
• Legality framing: robots.txt is a convention, not a law, but it is evidence of site intent. Terms of service, the CFAA (see hiQ v. LinkedIn), EU DSM Art. 4 TDM opt-outs, GDPR (personal data), and paywall/login boundaries all matter; don't crawl behind auth, don't bypass technical protections, and get review before crawling a new target class.
• A global semaphore is not politeness. Limiting total concurrency to 100 still hammers a single host with 100 requests. Politeness is per-host and time-based.
• Ignoring Retry-After or clamping it down. Never retry sooner than the server asked; choose max(backoff, Retry-After) and reschedule durably if it exceeds your budget.
• Un-jittered exponential backoff. Thundering herds are the classic failure; use full jitter and bound total attempts and elapsed time.
• Resolving DNS per request. Cache with TTLs and negative results (hickory-resolver); DNS becomes a bottleneck at scale.
• Memory-only frontier. You will lose days of crawl on restart; persist from day one (SQLite WAL, single writer).
• Weak URL canonicalization. Fragments, tracking params, trailing slashes, percent-encoding, and http→https variants all cause duplicates; build the canonicalizer early with fixture tests.
• Spider traps. Calendars, infinite pagination, and session IDs can explode the frontier; cap depth and URLs-per-host and detect near-infinite patterns.
• Treating Crawl-delay as universal. Google ignores it; Bing/Yandex/Common Crawl honor it. Honor it anyway, and never use its absence as license.
• Confusing 5xx with "disallow". For robots.txt, 5xx means temporarily unreachable — back off and reuse cache; for resources, retry with backoff, don't drop the URL.
• What to skip for now: building a distributed queue (Redis/Kafka/RocksDB) before a single-node crawler is correct; Postgres before multi-node; implementing your own jitter math before reading AWS's article; reading all of IR Book; browser rendering (Phase 5).

Communities

JS Rendering, Browsers & CDP (HTTP-First, Browser-Fallback)

Exa crawls and parses tens of billions of pages, refreshing the index continuously (see exa-d), so the crawler team's job is coverage per dollar: fetch everything you can over HTTP, and spend a browser only where the content truly is not in the response. Browser rendering costs 10-50x a plain fetch (CPU, RAM, latency, per-request overhead), so the routing decision — "does this URL need a browser?" — is the highest-leverage code you will write. This phase gets you fluent in CDP, Playwright, Rust browser crates, and a hybrid architecture where the browser is a fallback for dynamic content, never an anti-bot tactic.

Version snapshot: verified September 2026. chromiumoxide 0.9.1, headless_chrome 1.0.22, fantoccini 0.22.1, playwright-rs 0.18.0, thirtyfour 0.37.5, spider 2.53.9.

Start Here (core path)

Read in this order; everything here is free.

A 30-line Python probe that answers "does this page need a browser?" from the raw HTTP response: https://dev.to/0012303/a-30-line-probe-that-tells-you-if-a-page-needs-a-browser-1pj

Chrome DevTools Protocol Fundamentals (docs + guides)

CDP is Chromium's native instrumentation protocol: JSON-RPC over a WebSocket, organized into domains of methods and events. Chrome DevTools itself, Puppeteer, and Playwright's Chromium path all speak it. It is Chromium-only (Firefox and WebKit have their own debug protocols), and it has stable plus experimental surfaces.

Core mental model (from Getting Started with CDP):

• Start Chrome with --remote-debugging-port=9222 (or =0 for a random port; the WS URL is printed to stderr and written to DevToolsActivePort).
• HTTP endpoints: GET /json/version (browser WS URL), GET /json/list (targets), GET /json/protocol (full protocol JSON), PUT /json/new?{url}, WS /devtools/page/{targetId}, WS /devtools/browser/{id}.
• Commands are {id, method, params}; responses echo id; events have no id. Message order is meaningful (e.g., Target.setDiscoverTargets reports all current targets before its response).
• Targets are pages, workers, iframes, service workers, the browser itself. You attach with Target.attachToTarget {targetId, flatten:true} and get a sessionId; every subsequent message for that target carries sessionId. Flattened sessions are the only mode worth learning; non-flat is deprecated.
• Sessions form a hierarchy: the WebSocket connection is an implicit root browser session; page sessions are children and close when the parent detaches.
• Protocol state is per session (Network.enable, Runtime.enable, Page.enable must be called on the session you listen on).
• Stable vs experimental: prefer stable; expect experimental churn. High-level libraries exist partly to absorb that churn.

Domains that matter for crawling:

Fetch interception mechanics: Fetch.enable with patterns of {urlPattern, resourceType, requestStage} (wildcards * = zero or more, ? = one; stages Request or Response). Matching requests emit Fetch.requestPaused and stall until you call continueRequest, failRequest, or fulfillRequest. Response-stage interception (requestStage: "Response") is how you read bodies before the page gets them. This is the low-level version of Playwright's page.route().

Useful extra resources:

Protocol choice at a glance:

Playwright & TypeScript/Python Baseline

Playwright is the industry baseline for browser automation and the right way to learn rendering behavior before touching Rust. It drives Chromium, Firefox, and WebKit with one API, ships first-class auto-waiting, and exposes network interception cleanly. For crawler work you will use the library API, not the test runner.

Semantics to internalize before writing any Rust:

• page.goto(url, waitUntil) supports commit, domcontentloaded, load, networkidle. networkidle is discouraged even by Playwright's own docs — prefer waiting for a specific response/selector/locator state.
• Locators auto-wait and auto-retry; a locator re-resolves on every action. Use locator.waitFor({state: 'visible'}), page.waitForResponse, page.waitForFunction for explicit readiness.
• Registration order matters for routes: handlers run in reverse registration order; use route.fallback() to delegate rather than ending a request, and route.abort('blockedbyclient') (not bare abort()) so pages treat it as a content blocker rather than a network fault.
• Resource types from request.resourceType() (equivalently CDP Network.ResourceType): document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, other. Block image/media/font for text extraction; test stylesheet before blocking (layout-dependent JS can break).
• If routes appear to miss requests, a service worker is serving them; set serviceWorkers: 'block' on the context.
• browser.newContext() is the isolation unit. One browser process can serve many contexts — reuse the process, isolate the contexts.
• connectOverCDP() attaches to an existing browser; useful when another component owns the browser lifecycle, but lower fidelity than Playwright's own protocol.

WebDriver vs CDP in one line: WebDriver is a cross-browser W3C standard that needs a driver binary and cannot intercept network traffic; CDP is Chromium-only, WebSocket-based, and exposes everything (interception, console, emulation, profiling). For crawling, CDP (via Playwright/Puppeteer/chromiumoxide) is the technology; WebDriver/BiDi only matters if you need non-Chromium engines.

Rust Browser Crates (chromiumoxide, fantoccini, headless_chrome, playwright) — comparison table

Engineering notes (choose with these in mind):

• chromiumoxide — version 0.9.1 (Feb 2026). Async only, tokio-only. Every CDP domain is generated from the protocol PDL into chromiumoxide_cdp (~60K lines), so expect slow first compiles and a large dependency tree. Page::execute accepts any generated Command, so Fetch/Network/Emulation are all reachable even when no convenience wrapper exists. Best fit for a long-lived, high-throughput browser worker.
• headless_chrome — version 1.0.22 (Jun 2026). Synchronous, thread-based; closest to Puppeteer ergonomically. Has request interception, JS coverage, element/page screenshots, PDF, headful mode, and a fetch feature that downloads a known-good Chromium. Known gaps (stated in its README): frames, file choosers, touch, network/CPU emulation, request timing, SSL inspection, EventSource/WebSocket inspection. Great for tools and tests; awkward when you need thousands of concurrent fetches in one async runtime.
• fantoccini — version 0.22.1 (Feb 2026). WebDriver client; you run geckodriver/chromedriver yourself on port 4444. Tokio-based, CSS/XPATH locators, forms, waits, and Client::source for post-JS HTML. Because WebDriver has no interception or lifecycle-event API, your waiting strategy degrades to polling. Choose it when cross-browser compatibility matters more than depth.
• playwright-rs — version 0.18.0 (Sep 2026), pre-1.0 but rapidly released. Follows Microsoft's binding architecture: it downloads the Playwright driver (Node 18+ required) and speaks to it, so you get Playwright's auto-waiting, contexts, cross-browser support, and traces with Rust ergonomics. Browser install is a separate npx playwright@<matching> install step, and the crate pins an exact driver version. Good choice if you value Playwright semantics over raw CDP control; expect API churn before 1.0.
• Do not use the old playwright crate (octaltree lineage) — unmaintained; playwright-rs is the current project. (crates.io names: playwright-rs, docs at https://docs.rs/playwright-rs.)
• Recommendation for this roadmap: prototype in Playwright TS/Python, then build the Rust worker on chromiumoxide. Use headless_chrome for quick one-off scripts and fantoccini/thirtyfour only if a WebDriver requirement appears.

Crate pages: chromiumoxide · headless-chrome · fantoccini · playwright-rs · thirtyfour · spider

Hybrid Routing: When to Render vs HTTP Parse

The architecture that scales to 100M+ pages/day is boring on purpose: most URLs never touch a browser.

frontier ──► scheduler ──► HTTP fetcher pool (reqwest, high concurrency)
                │                    │
                │                    ▼
                │            parse + extract (scraper/lol-html)
                │                    │
                │                    ▼
                │            shell detector ──► static / parseable  ──► index
                │                    │
                │                    └──► likely_shell ──► browser queue (bounded)
                │                                              │
                │                                              ▼
                │                                    browser worker pool (CDP)
                │                                    render + wait + extract
                │                                              │
                └◄────────────── decision cache (per URL + per host) ◄──────┘

Rules that make it work:

1. Always HTTP first. One cheap GET answers both "can I get the data?" and "does this host need a browser?". Only the parse result decides escalation.
2. Try embedded data before a browser. A huge share of "JavaScript sites" are SSR frameworks that ship the data in the first response: __NEXT_DATA__ (Next.js), window.__NUXT__ (Nuxt), window.__INITIAL_STATE__, JSON-LD. Parsing that JSON is a superset of rendering for extraction purposes and costs ~0 extra milliseconds. A browser is the last resort, not the second.
3. Prefer capturing the XHR/fetch API over the DOM when the SPA loads JSON. With Playwright, page.waitForResponse('**/api/**') + response.json(); with CDP, Network.responseReceived + Network.getResponseBody (or Fetch at response stage). Structured JSON is more stable than DOM selectors and cheaper than waiting for paint.
4. Cache the decision at host level. If 3 sampled URLs on shop.example are all shells, route subsequent shop.example URLs to the browser queue directly; if a host is static, never render it again. Expire decisions (e.g., 7 days) and keep a small per-host confidence counter.
5. Keep one politeness budget. Browser-rendered pages fire dozens of subresource requests. Per-domain rate/token buckets must cover both HTTP and browser requests, or rendering will silently multiply your load by 10-30x.
6. ambiguous is not render. Send ambiguous URLs to a sampled browser check, then decide for the batch — do not default to Chrome.

SPA-shell detection signals (cheapest first, short-circuit):

Reference implementations to mirror: ketch detect.go (text + markers + ratio, returns static/likely_shell/ambiguous), StormCrawler's JsRenderingDetector (fingerprints → noscript → empty root → thin-content fallback, stamps a routing key), and the 30-line Python probe (visible-text size, hydration blob, needle).

Render when: content is client-only; data arrives via authenticated/signed XHR you cannot reproduce; interaction (clicks, scroll, pagination) is required; login/session state matters; or you are sampling to validate your HTTP extractor.

Do not render when: the data is already in the HTML or an embedded JSON blob; a public JSON API/feed/sitemap provides it; you only need head metadata; the page is a bot wall (rendering won't change the outcome and attempting to defeat it is out of scope); or robots.txt/ToS disallow the fetch in the first place.

Cost, Latency & Browser Pool Operations

Numbers worth memorizing (with sources): a browser fetch is typically 10-50x slower than an HTTP fetch and is concurrency-limited by RAM (StormCrawler); a headless Chrome tab runs ~200-500MB and takes ~1-5s to start; blocking image/media/font cut transfer by 67% on an image-heavy catalogue but can add latency because every request passes through a handler (measure medians, not one run). Hosted browser APIs land around $6-18 per 1,000 usable pages at medium volume once proxies/ops are included; self-hosting wins on steady high volume but only if you can staff the fleet. Lightpanda claims ~20MB/instance and 9-16x less memory than Chrome by skipping graphical rendering — treat as vendor benchmarks and validate on your targets.

Pool design checklist:

• Fixed number of browser processes; each process hosts N contexts, one job per context. Never launch() per URL.
• Acquire/release semantics with a semaphore and a bounded queue; backpressure beats crashing.
• Recycle by maxUses (e.g., 25-50 pages) and maxAge (e.g., 5-10 minutes); recreate on crash or missed health check (ping via Browser.getVersion or a blank navigation with timeout).
• One watchdog per process (kill -9 + respawn), and a global cap on concurrent browsers derived from RAM (instances ≈ available_RAM / 400MB).
• Launch flags: --headless (new headless; old headless was removed from Chrome 132), --disable-dev-shm-usage in containers, explicit --user-data-dir per process for isolation. Keep --no-sandbox out of production unless the container is the sandbox.
• Metrics: render ratio, queue wait p95, render p50/p95, pages per browser-hour, crashes/recycles per 1K pages, bytes per page with/without blocking, extraction yield after render.
• Resource blocking is a cost tool first; keep it honest as a behavioral consideration second (see next section).
• One renderer can often be replaced by one JSON capture: always test the XHR path before scaling browsers.

Legal/Ethical Framing (responsible rendering, not anti-bot bypass)

Browser rendering exists here to see dynamic content, not to defeat access controls. Write this distinction into your README and portfolio — it is exactly how Exa frames its own crawler.

• Identify yourself. Use a product token (e.g., YourBot/1.0 (+https://your.site/bot, contact@)) and a real contact page. Exa’s ExaSearchBot policy is the model to imitate: purpose stated, identification documented, robots.txt respected, rate-limited per site.
• robots.txt is crawl guidance, not authorization. Per RFC 9309: longest-match wins; 4xx = unrestricted, 5xx = treat as fully disallowed until retried; cache up to 24 hours. A Disallow is a strong stop signal; an Allow does not override ToS, auth, or rate limits.
• Stop signals are stops. 429/403/CAPTCHA/paywall/login walls mean back off or stop. Do not rotate IPs, swap fingerprints, or automate CAPTCHA solving to continue. A rendered challenge page is not "coverage".
• Rendering increases load. A browser session multiplies requests per URL; account for subresources in the same per-host budget and honor Retry-After and Crawl-delay across both HTTP and browser paths.
• Public visibility is not permission. hiQ v. LinkedIn and the CFAA line of cases are fact-specific, not blanket clearance; ToS and copyright still apply. Get counsel for consequential collection.
• Keep anti-bot circumvention out of scope. Stealth plugins, fingerprint spoofing, proxy rotation to evade blocks, and CAPTCHA solving are a different project with a different risk profile; nothing in this roadmap requires them.
• Prefer the sanctioned channel. API, sitemap, feed, partner agreement — every one of these beats rendering, both ethically and on cost.

Recommended Order (weeks 9-10, with time estimates)

Assumes Phases 1-4 done (Rust, Tokio, HTTP scraping, polite crawling). Total: ~38-42 hours over two weeks, ~2-3h weekdays and a 4-6h weekend block.

Week 9 — Learn the stack and measure rendering (≈18-20h)

Week 10 — Build the hybrid crawler and browser pool (≈20h)

Hands-on Exercises

Each exercise is self-contained and builds toward the final hybrid crawler. Test targets: https://quotes.toscrape.com/ (static) and https://quotes.toscrape.com/js/ (client-rendered), plus any Next.js/Nuxt site you are permitted to crawl.

1. Raw CDP client (no libraries). Launch Chrome headless with a debugging port; use the HTTP endpoints to find the page target; open a WebSocket; Target.attachToTarget (flatten:true); Page.enable; Page.navigate to example.com; wait for Page.loadEventFired; Page.captureScreenshot; decode base64 to PNG. Acceptance: PNG opens and visibly shows the page; running twice produces two PNGs; logs correlate request ids to responses and show the sessionId on every page command; zero Playwright/Puppeteer/chromiumoxide imports.

2. Rendering delta measurement. Extract quotes from / and /js/ twice: once with HTTP-only (reqwest + scraper) and once with Playwright (page.goto + locator text). Acceptance: HTTP-only finds 10/10 on / and 0/10 on /js/; Playwright finds 10/10 on both; a table records transport, wall ms, bytes, quotes found for all four runs; p50 of 3 runs each.

3. Resource blocking with honest measurement. Register context.route('**/*') blocking image/media/font with route.abort('blockedbyclient'); count blocked requests; measure transfer bytes and total wall time over 3 runs on 3 different pages. Acceptance: Blocked counter > 0 on pages that have images; bytes table shows the reduction per page; a one-paragraph note identifies at least one page where blocking did not meaningfully reduce time, with numbers.

4. SPA shell detector in Rust. Implement classify(html: &str) -> Verdict (Static | LikelyShell | Ambiguous) using at least four signals from the detection table (visible text length, empty hydration root, framework fingerprint, noscript JS-required, script-to-text ratio). Build 12 fixtures: static blog, static docs, Next.js shell, Next.js SSR with __NEXT_DATA__, Nuxt, SvelteKit, Angular, Qwik, empty app root, noscript page, thin error page, real content page. Acceptance: 12 fixtures compile into unit tests; ≥11/12 verdicts correct; the SSR-with-blob fixture returns Static (do not render); every decision prints the triggering reason string; cargo test green.

5. Embedded-data extraction (no browser). Pick a Next.js or Nuxt site you may crawl; extract structured data from __NEXT_DATA__ / window.__NUXT__ in the raw HTML. Separately load the same URL once in Playwright and extract the same fields from the DOM. Acceptance: The extraction run provably never launches a browser (guard flag or assert no Chrome process); ≥95% field match on ≥20 records; README documents the blob markers and how your detector will recognize them.

6. Hybrid crawler that detects SPA shells and routes to a browser worker. Input: 100 mixed URLs (static pages, shells, SSR-with-blob). Pipeline: HTTP fetch → extract → detect → route shells to a bounded browser queue (Rust + chromiumoxide worker) → extract rendered DOM. Include per-URL and per-host decision caching and one shared per-host token bucket. Acceptance: Browser renders ≤ (number of known shells + 10%); static and SSR pages have render count of exactly 0; all 100 URLs produce output with at least one non-empty extracted field; metrics line prints http_count, render_count, render_ratio, cache_hits, render_p50/p95; one command runs the whole crawl.

7. Browser pool hardening. Run 2 browser processes with context-per-job, acquire/release, maxUses=25, maxAge=5min, periodic health checks. Mid-run, kill -9 one browser. Acceptance: Crawl of 300 URLs completes unattended; replacement spawns within 30s; counters report recycles, crashes, and timeouts; output rows == input rows (no dropped URLs); peak RSS and browsers-per-GB documented.

8. Capture the JSON API instead of the DOM. On an SPA that fetches JSON, use CDP Network.responseReceived + Network.getResponseBody (or Playwright waitForResponse) to capture the payload, and extract records from JSON. Acceptance: Extracted record count equals rendered-DOM record count; JSON path median latency < DOM path median over 10 runs; extraction code contains no HTML selectors for the data fields; the captured endpoint and schema are documented.

Stretch: Run Exercise 6 with Lightpanda as a second CDP engine behind the same worker interface; compare memory per instance, startup time, and extraction yield, and write down where fidelity broke.

Common Pitfalls / What to Skip

• Launching a browser per URL. 1-5s startup and 200-500MB per instance. Reuse processes and contexts; recycle on a budget, not per request.
• networkidle as a default wait. Playwright itself discourages it. Wait for a specific response, locator state, or lifecycle event.
• sleep() as a wait strategy. Race conditions and wasted seconds; use the framework's auto-waiting plus explicit conditions.
• Blocking script or stylesheet reflexively. Scripts are the content on SPAs; CSS can drive layout-dependent JS. Start with image/media/font, test stylesheet per target.
• Forgetting service workers. They can serve requests outside page.route; set serviceWorkers: 'block'.
• Bare route.abort(). Prefer route.abort('blockedbyclient') so retry-on-error page logic doesn't loop.
• Trusting --headless=old. Removed from the Chrome binary in 132; chrome-headless-shell is the standalone legacy binary, not the new headless mode.
• Using the unmaintained playwright crate. Use playwright-rs (padamson).
• Expecting CDP cross-browser. CDP is Chromium-only; use WebDriver/WebDriver BiDi for Firefox/Safari.
• Rendering anti-bot challenges. Out of scope; treat 403/429/CAPTCHA as stop signals, not problems for Playwright to solve.
• Stealth/anti-detect stacks as a strategy. puppeteer-extra-plugin-stealth, fingerprint spoofers, and proxy rotation to evade blocks add risk without improving indexing quality; skip them in this roadmap.
• Screenshot-based extraction. Screenshots are for debugging/QA; extract from DOM or JSON.
• Leaving Network.enable/Fetch.enable on with no consumer. They add per-request overhead; enable only while capturing or intercepting.
• Counting renders as success. Track render ratio, bytes, and cost per usable page; a high render rate is a routing failure.
• Ignoring the subresource budget. Rendered pages generate 10-30x the requests; your politeness layer must count them.

Open-Source Repos to Read

Distributed Crawlers & Observability

Exa runs a distributed crawl fleet over 500B+ webpages, targeting 100M+ pages/day, and its crawler team owns the unglamorous core: scheduling, queues, partition ownership, at-least-once delivery, and the telemetry that tells you why the frontier stopped draining at 3am. Phase 6 is where your single-machine crawler becomes a fleet and where you learn to prove it is healthy: queues and partitioning, idempotent writes, checkpointing and backpressure, metrics/tracing/dashboards, containerized deployment, and profiling. Everything else in your portfolio is judged through this lens — "can this person operate a web-scale crawl?"

How to use this file: work the Start Here path first, use the tables as a menu, and build the week-11 exercises at the bottom. Links were checked in September 2026; prefer docs.rs over crates.io (crates.io blocks automated checks).

Start Here (core path)

Do these in order. Everything else in this file is optional depth.

Message Queues & Streams

The frontier is a queue — but the abstraction matters more than the broker. Design a Queue trait in your project so a reviewer sees you understand the tradeoffs, then implement one backend well.

Broker comparison (for a web-crawl frontier)

Key crawler-specific tradeoffs: domain affinity beats uniform hashing (you need per-domain ordering/serialization for politeness), ack-after-side-effect is non-negotiable, replay (Kafka/NATS/Redis) lets you reprocess after a parser bug, and managed queues can't give you replay. Pulsar/RabbitMQ/Redpanda are out of scope for week 11 — don't learn a fourth broker.

Rust client resources

Recommendation for week 11: NATS JetStream. One docker run nats -js, official async Rust client, at-least-once acks, replay, and KV for coordination. If you want the AWS interview signal, SQS maps 1:1 to the Hello Interview design; if you want the "distributed log" signal, Kafka + rdkafka.

Partitioning, Idempotency, Checkpointing, Backpressure

Domain partitioning / consistent hashing

Crawler-specific rule: hash the registrable domain (e.g. news.ycombinator.com → ycombinator.com) to a worker, not the URL. That gives you (a) politeness/serialization per domain, (b) stable ownership across scale-ups, and (c) no two workers hammering the same host. URLs from one domain are then queued in order to that worker.

Idempotency (at-least-once is the only honest delivery guarantee)

Pattern to implement: canonicalize URL (strip fragments, normalize host/scheme) → url_hash = sha256(canonical) → message carries the hash → worker does INSERT ... ON CONFLICT (url_hash) DO NOTHING in the same transaction as the fetch-result write → ack the message last. Duplicates become no-ops; crashes become redeliveries.

Checkpointing

Checkpoint at two levels: queue position (broker ack/offset) and crawl state (per-URL status: discovered / queued / fetched / parsed / dead, plus retry_count and next_attempt_at). Checkpointing only the broker offset means a parser bug forces a full re-fetch; checkpointing state lets you replay parsing from S3/WARC.

Backpressure & graceful degradation

Phase-6 invariants (memorize these):

1. Ack/delete the message after the fetch result is durably written (S3 + DB), never before.
2. Every queue payload carries a deterministic idempotency key (canonical URL hash); consumers are idempotent.
3. Checkpoint = broker ack + per-URL state transition; both must survive kill -9.
4. All channels/queues are bounded; when full, the producer waits or sheds — never buffer to disk silently.
5. Per-domain token bucket + global concurrency semaphore; deferred URLs go back to the queue with next_attempt_at, not into a sleeping worker.
6. Retries use exponential backoff + jitter + a max attempt count; after that, DLQ and move on.
7. Circuit-break domains that return repeated 5xx/429 so one broken host can't consume the fleet.
8. Graceful shutdown: stop taking new work, drain in-flight, ack/flush, exit within a deadline.

Observability

If Phase 6 has one deliverable an interviewer will remember, it's a crawl that is legible: dashboards, traces, and a console that show queue depth, per-domain health, and where time goes.

Metrics

Crawler metrics worth emitting (name them like this): pages_fetched_total{status}, fetch_duration_seconds (histogram), bytes_downloaded_total, frontier_queue_depth, in_flight_requests, domain_rate_limit_wait_seconds, retries_total{reason}, dlq_total, dedup_hits_total, scheduler_lag_seconds, worker_heartbeat_timestamp, parse_errors_total. Metrics labels are only for bounded-cardinality dimensions — never label by URL (domain only if you accept the cardinality; it's acceptable in a portfolio with a known domain set).

Tracing, logs, and OpenTelemetry

Spans to create: crawl.url (root per URL: url_hash, domain, depth), fetch (status, bytes, duration), parse (links found, content hash), enqueue (partition, queue), dedup_check. Propagate trace context through queue message headers so a single URL shows one trace across scheduler → worker → parser → indexer. Structured logs (tracing + JSON) with trace_id/span_id close the loop.

Minimal local observability stack (Compose services)

nats (or kafka/redis) + scheduler + worker × N + parser + prometheus + grafana + otel-collector + jaeger. Four of those are off-the-shelf images; only the crawler services are yours.

Deployment (Docker Compose, Kubernetes basics)

Compose is the week-11 target; Kubernetes is an optional half-day if you want the resume line. Do not let k8s delay the crawler.

Compose checklist: one compose.yml, service names as DNS, healthcheck + depends_on: condition: service_healthy, named volumes for broker data, .env for config, restart: unless-stopped for workers, and deploy.resources.limits so a runaway worker doesn't eat the laptop. If (and only if) that works: translate each service to a Deployment + Service in kind, add a liveness/readiness probe, scale workers with kubectl scale, and kill a pod to watch it recover.

Performance Profiling & Benchmarking Rust

Measure before you optimize, and benchmark the system, not just the function. On macOS, use samply for sampling (better macOS story); cargo-flamegraph works but rides xctrace and can be finicky.

Canonical Cargo.toml for profiling (both samply and flamegraph need symbols):

[profile.profiling]
inherits = "release"
debug = true

Workflow: hyperfine the whole crawl for a baseline → samply record a realistic run to find the widest stacks (usually JSON/HTML parsing, DNS, or accidental blocking) → write a criterion bench for the exact hot function → fix → re-run both and commit the before/after graphs to the README. Also run tokio-console under load: if a task shows seconds of busy time in a loop, you found a blocking call or a spin.

System Design Reading (DDIA, crawler architecture references)

Reading priority for week 11: Hello Interview crawler (design vocabulary) → Mercator + IR Book frontier (architecture) → DDIA stream processing / partitioning chapters (why the primitives exist) → SRE overload chapters (operational realism). Skip the rest until the project runs.

Recommended Order (week 11, with time estimates)

Total ≈ 18h core + 2h optional k8s stretch.

Hands-on Exercises (5-8)

Each builds on your Phase 5 crawler. Acceptance criteria are the definition of done.

1. Split the monolith into a Compose fleet. Services: scheduler (produces URL jobs), frontier/broker, worker (fetch), parser (extract links + text), metrics (Prometheus scrape), plus Prometheus/Grafana. Acceptance: docker compose up -d starts everything healthy; docker compose up -d --scale worker=3 triples fetch throughput in the dashboard; docker compose kill worker drops throughput but the crawl resumes with no lost URLs; README shows the architecture diagram.

2. Prove at-least-once + idempotency. Publish every URL twice, and add a test hook that crashes a worker after fetch but before ack. Acceptance: a dedup_hits_total metric/row count increments on the duplicate; the URL-state table contains exactly one row per canonical URL; the crashed URL is redelivered and succeeds; a test (#[tokio::test]) asserts duplicate processing produces identical state.

3. Domain partitioning with minimal reshuffling. Implement rendezvous hashing over registrable domains with a worker list from config. Acceptance: unit test maps 10k sample domains across 8 workers and asserts < 2% variance in load; removing one worker remaps only its ~1/8 share; two workers never fetch the same domain concurrently (assert via per-domain lock/logs); per-domain fetch order is preserved.

4. Checkpoint/resume after hard kill. Run a 1k-URL crawl, kill -9 the scheduler and a worker midway, restart the stack. Acceptance: zero URLs reported as permanently lost; pending (unacked) message count returns to zero after drain; a run report shows discovered → fetched → parsed counts reconcile; no double-fetch beyond the duplicate window allowed by at-least-once.

5. Backpressure and graceful degradation under a slow consumer. Make the parser sleep (env-configurable latency) until queues saturate; simultaneously point the crawler at a host that always 503s. Acceptance: worker RSS stays under a configured bound (e.g. 512MB) because channels are bounded; the crawler slows instead of OOMing; the bad domain's circuit breaker opens and the rest of the crawl continues; retries show jittered delays and a retry budget cap; over-capacity fetch requests are load-shed with a 429/503 counted in metrics.

6. Build the crawl dashboard and one full trace. Emit the metric set above; add spans crawl.url, fetch, parse, enqueue and propagate trace context in queue headers. Acceptance: Grafana dashboard JSON committed with throughput, error rate, p50/p95 fetch latency, frontier depth, and retries by reason; Jaeger shows a single trace spanning scheduler → worker → parser for one URL; tokio-console connects and shows no task with multi-second busy time.

7. Profile and optimize one hot path. Baseline with hyperfine for end-to-end and a criterion bench for HTML link extraction. Acceptance: samply flamegraph produced for a release+debug run; one identified hotspot improved ≥20% on the criterion bench with before/after output committed; flamegraph SVG/README note explains the fix and what you deliberately did not optimize.

8. (Stretch) Deploy the fleet to kind. Translate each compose service to a Deployment + Service; add readiness probes and resource limits. Acceptance: kubectl get pods all Ready; deleting a worker pod self-heals; kubectl scale deployment worker --replicas=4 increases throughput; kubectl port-forward reaches Grafana; a short README section lists what k8s added over Compose and what it cost.

Common Pitfalls / What to Skip

Pitfalls (and the fix):

• Chasing exactly-once delivery. It doesn't exist over a network; at-least-once + idempotent writes does. Stop reading about transactional outboxes for a crawler and add a unique index.
• Unbounded channels. mpsc::unbounded turns a slow parser into an OOM kill. Use bounded channels and let backpressure propagate.
• Acking too early. Ack before the durable write and a crash loses the URL silently. Always: write → fsync/confirm → ack.
• Partitioning by URL hash. You get no per-domain ordering and can hammer one host from N workers. Hash domains.
• Putting page bodies in queue messages. Queues are not blob stores; pass IDs/keys, store bytes in S3 or on disk. (Hello Interview calls this out explicitly.)
• Retry storms. Retries without jitter, budgets, or circuit breakers amplify outages and get your IP range banned. Add jitter, caps, and DLQ from day one.
• A single hot domain. One giant host can saturate a partition; cap per-domain in-flight requests and let other domains flow around it.
• Visibility timeout too short. Slow fetches get redelivered while still running, causing duplicate work. Size timeouts to p99.9 fetch latency (SQS) or use heartbeats/InProgress acks.
• Profiling debug builds. Results are meaningless; always --profile profiling (release + debug symbols).
• Regex-parsing HTML. Use an HTML parser; the CPU and correctness costs are real.
• Observability as an afterthought. If it isn't on a dashboard, it didn't happen. Emit metrics and spans while building each stage, not after.

Skip for now (do not spend week 11 here):

• Writing your own broker, WAL, or Raft; benchmark-driven custom lock-free frontier — use Redis/NATS/Kafka.
• Pulsar, RabbitMQ, ksqlDB, Flink, Spark — a fourth broker/stream processor adds no signal at this stage.
• Kubernetes operators, Helm charts, service mesh, ingress controllers, multi-region — Compose plus a kind cameo is plenty.
• Prometheus federation/Thanos/Loki clusters; OpenTelemetry sampling deep dives; custom tracing backends.
• Bloom filters and HyperLogLog for dedup before a plain unique index proves insufficient; crawler traps solved by max-depth.
• Browser rendering/CDP in this phase — that was Phase 5; the distributed system is the point here.
• Autoscaling in the cloud, spot fleets, and cost optimization — interesting in the interview, out of scope for the portfolio week.

Communities

If you want interview signals beyond code: post your Compose stack and a Grafana screenshot in r/rust, write up your idempotency test on your blog, and read one distributed-systems paper per week with Papers We Love. For Exa specifically, be ready to talk about the frontier, domain partitioning, politeness at fleet scale, and how you'd detect a stuck crawl from metrics alone.

Search & Retrieval for Crawled Data

Exa's Web Crawling Engineer role sits on a pipeline whose value is only realized at retrieval: crawl -> parse -> index -> search. This file covers the retrieval half — BM25/full-text search with Tantivy, embeddings and vector search in Rust, boilerplate removal and chunking, snippets/highlights, ranking evaluation, and Exa's own technical write-ups so your add-on mirrors how they think about search quality.

Start Here (core path)

Core path in order: (1) run Tantivy basic_search on 20 of your crawled pages, (2) read IIR Ch. 1 + 6 + 11 skim, (3) read Exa's vector-DB and BM25 posts, (4) run fastembed-rs + Qdrant quickstart, (5) chunk with text-splitter, (6) expose POST /search with Tantivy highlights, (7) evaluate and add hybrid + reranking.

IR Fundamentals (free books/papers)

Tantivy (full-text search in Rust)

Embeddings & Vector Search (Rust tooling + vector DBs)

Parsing, Boilerplate Removal & Chunking

Snippets, Highlights & Ranking Evaluation

Exa's Public Research (what to read)

Read in this order; each post maps directly to a subsystem the role touches.