Architecture and invariants

← Maintainer handbook · Classification · Database lifecycle · Security policy

One permission-gated LinkedIn guest-HTML adapter feeds a deterministic classifier and one canonical SQLite lifecycle store. The website, API, feeds, downloads, and bounded README are read-only projections.

Open the annotated repository maps for five subsystem views and links to implementation. The overview below connects configuration, lifecycle state, public projections, and supporting tools.

Architectural principles

Contract	Required behavior
Identity	Numeric LinkedIn job IDs uniquely identify listings; distinct IDs remain distinct
Canonical state	SQLite alone owns lifecycle truth; public files cannot reconstruct it
Strict acceptance	Ambiguous type, role, seniority, cycle/posting date, or geography is excluded
Safe closure	Search-page disappearance never closes a job
Failure isolation	Failed searches record diagnostics, not lifecycle mutations
One writer	Canonical application writes occur through Repository; Alembic owns schema evolution
Bounded access	Requests, retries, pages, results, concurrency, response sizes, and pacing remain limited
Unauthenticated source	No source credentials, cookies, browser collection, private endpoints, or evasion
Read-only projections	Presentation does not collect, classify, migrate, or mutate canonical state
Determinism	The same evidence and settings produce the same classification and publication

These are contracts, not suggestions. Review SECURITY.md before proposing a changed boundary.

Component map

Owner	Path	Responsibility
CLI	src/opportunities/cli/	Preconditions, command input, orchestration, output and exit codes
Configuration	src/opportunities/config/, configs/	Settings, rules, recursive search registry and limits
Transport	src/opportunities/scrapers/http.py	Authorization interlock, approved endpoints, no redirects/cookies, bounded retries and pacing
Source adapter	src/opportunities/scrapers/linkedin.py	Guest URL construction, cards, detail identity, structured metadata and closure alerts
Normalization	src/opportunities/normalization/	Stable title and conservative location signals
Classification	src/opportunities/pipeline/classification.py	Pure acceptance/rejection decisions
Collection orchestration	src/opportunities/pipeline/runner.py	Bounded concurrent discovery, isolated outcomes, serialized persistence
Collection quality	src/opportunities/pipeline/data_quality.py	Pure aggregate drift analysis and publication-gate findings
Availability	src/opportunities/pipeline/availability.py	Bounded rotating explicit evidence audit
Persistence	src/opportunities/database/, migrations/	Transactions, provenance, lifecycle, bounded aggregate quality baselines and schema evolution
Projections	src/opportunities/readme.py, readme_localization.py, public_exports.py, search_registry_docs.py	Localized generated regions, public serialization and validation
Website	site/src/lib/, site/src/components/	Server-only read-only queries; client-only interaction
Operations	scripts/, .github/workflows/	Verified recovery, validation, review handoff and release-pointer deployment

Read one listing through the code

For a first code review, follow one listing rather than reading every module in directory order:

1. Start with CollectionPipeline.run in src/opportunities/pipeline/runner.py. It connects search scheduling, fetching, classification, and per-search persistence. The CLI in src/opportunities/cli/app.py adds command preconditions and publication gates; it is not the place to start learning the classification rules.
2. Follow LinkedInScraper.scrape in src/opportunities/scrapers/linkedin.py into parse_search_page and parse_job_detail. Then read HttpFetcher.get_text in scrapers/http.py for the separate authorization, pacing, retry, and source-stop policy.
3. Read Classifier.classify alongside normalize_locations. Trace an accepted role and a rejected one through tests/unit/test_classification.py; the tests distinguish discovery hints from sufficient publication evidence.
4. Follow Repository.persist_success in src/opportunities/database/repository.py. Read the stale-observation and failed-transaction cases in tests/integration/test_repository.py before changing lifecycle behavior. tests/integration/test_runner.py connects these boundaries with mocked collection and a real disposable database.
5. Follow an open row into public_exports.py and readme.py, then into site/src/lib/opportunities.ts. The website does not rerun classification. Its directory component delegates URL state to use-opportunity-directory.ts and private browser state to use-local-opportunities.ts.

For an offline end-to-end example, read scripts/testing/create_site_fixture.py: it creates synthetic listings through real migrations, repository methods, and exporters. Run it only through the documented fixture setup, never against production state. Use the test inventory to find the failure cases for a particular boundary.

Discovery is not acceptance

Search YAML controls where to look, not what may be published. Search cards undergo company/title prefilters before bounded detail fetching. Detail responses must independently contain title and company identity; a current card cannot make a generic HTTP-success page valid. Detail location may fall back to card location.

Overlapping searches share in-flight detail requests by numeric ID. Completed response bodies are evicted; one cancelled search must not cancel a request another search uses. Accepted DiscoveredJob values cross the repository boundary only after normalization and classification.

Search registry owns discovery schema and pagination; Configuration owns transport tuning. Never add acceptance rules to the website, YAML workflow, or README renderer.

Failure isolation

Searches fetch within the existing concurrency bound, in oldest-completed order with stable registry ties, then outcomes persist in finish-time order. Each successful search commits independently, including its private scheduling timestamp. Failed and source-skipped searches never advance that timestamp. A failed search records bounded, sanitized diagnostics but does not upsert jobs, apply absence, confirm unavailability, or close rows. A partial batch preserves successful work.

HTTP 429 stops all new source requests. Completed searches or confirmed availability checks may qualify for partial publication, but other denials and independent failures block it. The source-access policy defines eligibility. An audit rate limit also prevents nightly scraping from starting.

Partial updates still require integrity, projection, and snapshot validation and manual review. Collection must also pass its quality gate. Neither partial nor blocked runs establish a full-registry quality baseline; audit-only runs produce no collection metrics or report.

Rotation becomes durable only through verified snapshots; the reviewed-state barrier still blocks new mutations while a partial proposal is unmerged. CLI exit codes are canonical.

The search start is a conservative observation lower bound: a detail page fetched early in a slow search must not overrule a newer observation merely because its search finished later. Repository stale-evidence guards and monotonic timestamps enforce this. Exact transitions, the separate rotating audit, and migrations belong to Database and lifecycle.

Public projections

Projection	Owner and contract
Website	Short-lived read-only SQLite connections; all open rows and latest successful collection time; website engineering
API	Shared release selection, bounded inputs, and explicit field allowlists; listing queries and read-only dataset status; v1 contract
RSS / Atom feeds	Same read-only open rows, latest 50 entries, exact type/country/category filters, and escaped XML; subscriber guide
CSV / JSON / metadata	Python renderer selects approved fields, neutralizes CSV formulas, validates schema/bytes/counts/hashes, and atomically replaces files; download contract
READMEs	English and five translations share canonical counts, seals, and listing data with localized renderer-owned presentation; maximum five newest rows per employment type in each file
Registry documentation	Renderer owns the search-directory count block, not the surrounding guide

Optional collection-time data-quality monitoring compares aggregate observations with recent matching runs and stores bounded counts through Repository. Its gate runs after search commits and can withhold publication, but never changes classification or rolls back lifecycle state. Reports contain no listing text or raw source responses; website and export projections do not read monitoring baselines. The CLI quality-report reference owns thresholds and scope.

The README includes a hidden SHA-256 seal covering every website-visible open row and the exact last-successful-collection timestamp, including rows outside its preview. Validation reconstructs both regions from SQLite using fixed presentation strings for each supported language and requires exact localized output equality. Rendering preflights all files before atomic per-file writes and preserves authored text; README localization defines file discovery and ownership. Automation's reviewed-state barrier prevents deploying unreviewed public state. A hash alone does not explain a change: review the sanitized artifact and protected evidence when necessary.

Coverage metrics are a separate generated region in Testing, owned by scripts/docs/coverage_docs.py. No production database is needed to refresh them.

Dependency direction

CLI → configuration / pipeline / repository / projection renderers. Pipeline → transport + parser / normalization + classifier / repository. Repository → models + database session. Website → read-only SQLite; downloads → fixed generated files. The parser does not write SQL, the repository does not parse HTML, and renderers do not discover jobs.

Maintainer add-job / add-jobs is an offline input path through the same classifier and repository upsert. It validates supplied facts but cannot authenticate them, rejects manual reopening of closed rows, and invents no search provenance. CLI owns the input contract; Database owns transaction semantics.

Browser-local state is not lifecycle state

Saved, applied, hidden, and visit timestamps exist only in browser localStorage. They contain IDs, not job metadata or application details, and never reach SQLite or the API. Website engineering owns storage semantics; Privacy owns the disclosure.

Extension policy

Focused searches, stricter classification with nearby rejection tests, sanitized fixtures, read-only views, accessibility improvements, and safer recovery checks can preserve these boundaries. Additional providers, source authentication/browser collection, multiple writers, server-stored application state, mutation APIs, user content, or new authentication surfaces require explicit architecture and security review. Do not introduce them as incidental refactoring.