Automation and operations¶

← Maintainer handbook · Database lifecycle · Containers and deployment · Troubleshooting · Security policy

Configure and operate the GitHub Actions workflows that validate changes, collect authorized listing evidence, preserve canonical SQLite state, propose README updates, and deploy reviewed state. Use the workflow inventory to identify an entry point or the operations map to follow the handoffs.

Validation and collection remain separate: normal CI never contacts LinkedIn.

Contents¶

Routine operation from GitHub
Workflow overview
Validation workflows
Reusable automation boundaries
Workflow permissions
Protected environments and repository settings
First production activation
First public-state review seal
Recover a missing README state proposal
Collection authorization
Schedule and concurrency
Manual collection inputs
Manual job addition
Collection and deployment flow
Exit-code handling
State continuity and artifacts
Restricted VPS snapshot configuration
README update pull requests
Post-nightly production runbook
VPS deployment
Coordinated first rollout and rollback
Recovery and migration failures
Disabling collection
Operational checklist

Routine operation from GitHub¶

After first production activation, routine listing updates need no local checkout or SSH session. Use Actions, review the generated pull request, and verify the served dataset. Always select main when starting an operational workflow; after a merge, start a fresh run rather than rerunning an old commit.

The normal loop is:

Let Nightly full update run on its three-day cadence, or run it manually for an additional update. Leave skip_availability unchecked unless you intentionally want collection without the separate audit. A successful daily cadence check with skipped processing is an expected off-day, not a dataset update.
Follow the post-update review checklist. Require successful processing, verified snapshot publication, and validation on the proposal's exact head. Fully successful nightly proposals may merge automatically under the configured policy; partial proposals always require manual review and merge.
After the matching README proposal reaches main, run Scrape jobs or deploy reviewed state with deploy_to_vps=true. Approve the production environment if configured. Merging the PR alone does not update the website.
Verify production status against the reviewed artifact and deployment identifier. A green workflow, matching count, or recent export-generation time alone does not establish that the correct dataset is live.

Use these exceptions instead of repeatedly collecting:

Situation	Action
A correct snapshot has no open README proposal	Run Verify canonical state recovery with `recover_readme_proposal=true`; follow proposal recovery
Add independently reviewed listings without discovery	Use manual job addition, then review and deploy normally
Source permission expires or a source denial occurs	Disable collection and review authorization; never retry to bypass a denial
A migration, checksum, integrity, or publication gate fails	Preserve evidence and use operator troubleshooting; do not initialize empty state or bypass the failed check

Do not start another state-changing update while its README proposal is unmerged. The reviewed-state barrier intentionally stops it. The default recovery drill publishes a snapshot; only its explicit README recovery modes are read-only.

Maintenance that remains necessary¶

Actions automate the listing lifecycle, not every responsibility of operating a public service:

Review Dependabot proposals, failed security checks, and known tooling advisories. Application and dependency changes need normal validation and an application release; dataset deployment does not install new application code.
Keep both credential-bearing environments restricted to main, maintain required checks, and review source permission and SSH key validity. Check that scheduled workflows remain enabled after inactivity; GitHub can disable schedules in inactive public repositories.
Monitor VPS disk capacity and retain an independently protected backup outside that VPS. Snapshot retention is metadata, not automatic cleanup, and the same-host snapshot store cannot survive complete host loss. Follow state continuity before any cleanup or replication.
Opportunity counts and collection dates are generated by dataset workflows. Historical traffic figures are authored text, not live counters; use the statistics maintenance rules, without editing generated regions.

A failed gate or security advisory can require engineering or host-administration work. Do not treat this routine as a guarantee of unattended operation or as permission to change repository settings, publish state, or access the source.

Review maintenance pull requests¶

Dependabot proposes changes; it does not establish compatibility or install them in production. For each proposal, inspect the changed packages, lockfile, and upstream release notes, then require all applicable checks on the latest PR commit. A green check from an earlier commit is not sufficient. Do not remove a required check or dependency override merely to make the PR mergeable.

Change	After validation and merge
Generated listing counts, preview, or seal	Deploy the reviewed dataset from fresh `main`, then verify it against the reviewed public artifact and deployment identifier
Authored documentation or historical traffic figures	Let Documentation site publish from `main`; check the rendered page. No dataset deployment is needed
Website runtime dependency, framework, Docker base image, or application code	Follow the application release and rollback procedure; merging alone does not update the running container
Python pipeline dependency	Fresh Actions runs use the merged lockfile; release any separately operated CLI image before using it. If migrations or generated output change, review their compatibility and state implications before the next operational run
CI action or development-only tool	Fresh validation runs use the new version; check its permissions and output changes. Do not deploy data solely because a tooling update merged

Docker base-image updates may also require changes to the exact Debian security-package pins; Dependabot does not coordinate those edits automatically. Major upgrades, failing checks, new advisories, source-format changes, and recovery failures remain maintenance work rather than routine merge decisions.

Workflow overview¶

Workflow	Trigger	Responsibility
`python-ci.yml`	Push to `main`, pull request, manual	Python formatting, linting, typing, thresholded combined statement-and-branch coverage, parsing/classification benchmarks, migrations, and generated-document checks
`site-ci.yml`	Push to `main`, pull request, manual	Prettier, ESLint, strict TypeScript, the Next.js production build, unit tests, browser checks, axe-core accessibility scans, and Lighthouse CI on the built standalone site against synthetic SQLite state
`codeql.yml`	Push to `main`, pull request, Monday 05:31 UTC, manual	CodeQL `security-extended` analysis for Python and TypeScript, with findings uploaded to GitHub code scanning
`docker-ci.yml`	Push to `main`, pull request, manual	Action and Dockerfile linting, production-image builds, SPDX SBOMs and build evidence, vulnerability scans, plus migrated read-only SQLite and production-header smoke tests
`gitleaks.yml`	Push to `main`, pull request, manual	Scan relevant Git history for committed secrets with redacted output and no finding artifacts or pull-request comments
`dependency-review.yml`	Pull request, manual on a README automation branch with `pull_request_number`	Reject newly introduced dependencies with high or critical known vulnerabilities
`scorecard.yml`	Saturday 06:17 UTC, manual from `main`	Assess repository supply-chain practices, publish the Scorecard result, and upload supported SARIF findings to code scanning
`docs-links.yml`	Wednesday 06:41 UTC, manual	Lychee checks root Markdown and documentation Markdown/HTML links without adding external-link failures to pull-request validation
`documentation.yml`	Push to `main`, pull request, manual	Digest-pinned Markdown and prose lint, strict MkDocs Material build, and local rendered-link check; deploy only the allowlisted public documentation artifact from `main` to GitHub Pages
`canonical-state-drill.yml`	Manual	Recover, validate, republish, and round-trip a canonical snapshot without source access; explicit read-only modes propose the first public-state review seal or recover a missing README proposal without publishing a snapshot
`nightly.yml`	Daily 04:23 UTC cadence check; full update every three days; manual from `main`	Availability audit (manually skippable), scrape-time drift checks and aggregate report artifact, with one narrowly scoped automated README pull request
`scrape.yml`	Manual	Scrape-only update with its own review pull request, or deployment-only publication of reviewed state from `main`
`check-availability.yml`	Manual	Bounded availability-only audit with its own review pull request
`add-job.yml`	Manual	Add 1–10 maintainer-reviewed listings to durable canonical state without source access, then open one README-only review pull request

Two workflow_call-only files are implementation building blocks, not operator entry points: reusable-process-state.yml owns canonical restore, migration, selected source phases, validation, snapshot publication, optional protected deployment, and sanitized artifacts; reusable-readme-pr.yml owns the narrowly scoped README branch and pull-request mutation. The pinned Python/uv setup is shared through .github/actions/setup-python/action.yml.

Workflow files under .github/workflows/ are the executable source of truth. Update this guide when triggers, inputs, outputs, retention, permissions, or deployment behavior changes.

Validation workflows¶

The seven validation workflows dispatched for generated README proposals require no LinkedIn access:

Python CI checks lockfile consistency and validates the pipeline, CLI, migrations, lifecycle behavior, README projection, and documentation contracts; it publishes critical-path coverage and benchmark reports for 30 days. Current measured values are in the coverage baseline.
Site CI uses the documented Node.js and Bun versions to validate formatting, linting, strict TypeScript, the production Next.js build, unit tests, Playwright behavior, and axe-core accessibility checks against synthetic SQLite state. Lighthouse CI reuses that build, fixture, and installed Chromium to audit the homepage and indexable second directory page without contacting third-party analytics; reports are uploaded for seven days only when the Lighthouse audit fails. Thresholds and scope are documented in the testing guide.
CodeQL runs GitHub's extended security query suite independently for Python and TypeScript on pushes, pull requests, manual runs, and every Monday at 05:31 UTC. It uses interpreted-language no-build extraction and uploads results only to GitHub code scanning.
Docker CI runs actionlint, audits every workflow and composite action with blocking zizmor, and runs Hadolint. It builds both production targets, generates SPDX JSON SBOMs and build evidence with image IDs, available repository digests, and pinned builder-tool versions, and uploads those files for 30 days. Pushes and manual runs on main additionally create GitHub-signed SLSA provenance attestations for the evidence files; pull-request jobs do not receive OIDC or attestation permissions. Trivy rejects high or critical vulnerabilities for which a fix is available, and CI verifies migration, read-only website access, public-export delivery, Content Security Policy, and HTTP Strict Transport Security. Unfixed findings are excluded from the image-vulnerability gate. CI does not publish container images or attest a GHCR image digest; see Containers and deployment for the evidence scope and verification commands.
Gitleaks checks commits introduced by pull requests and pushes to main; manual runs scan the full fetched history. Checkout fetches full history without persisting credentials. Findings fail the job, but comments, summaries, and report artifacts are disabled to avoid publishing sensitive values. No detection allowlist is configured.
Documentation site lints maintained Markdown with markdownlint-cli2 and Vale before strict building and link validation; it does not deploy from an automation branch.
Dependency Review rejects newly introduced dependencies with high or critical known vulnerabilities, including development and unknown scopes. Ordinary pull requests use their event's base/head commits; dispatched README proposals resolve and verify those commits from the pull request.

The weekly Documentation links workflow uses Lychee to check root Markdown and documentation Markdown/HTML. Its configuration excludes only numeric LinkedIn job URLs, which can expire or reject automated checks, and skips documentation image files as inputs. Broken internal and other external links fail the check. Maintainers can also run it manually.

The Documentation site workflow runs markdownlint-cli2 and the local Vale style over maintained Markdown, strictly builds the MkDocs Material site, and checks rendered local links, anchors, and images on pull requests. Both linters run in digest-pinned containers without network access or write access to the checkout. Only a successful main build deploys its allowlisted public artifact to GitHub Pages; the deploy job alone receives the Pages write and OIDC permissions. For first-time setup, set GitHub Pages' source to GitHub Actions, configure docs.techopportunities.eu as the custom domain, and create a DNS CNAME record from docs to simonesiega.github.io. The build adds the matching CNAME file to the artifact. DNS changes and GitHub repository settings are maintainer-managed and are not performed by CI.

Dependency Review requires the repository dependency graph and only contents: read plus pull-requests: read permissions. License enforcement is disabled because the project has no dependency-license policy; show-openssf-scorecard: false keeps this gate focused. Dependabot proposes version updates and CodeQL analyzes project code separately.

Its workflow_dispatch input is pull_request_number, not caller-selected comparison refs. Dispatch on that PR's automated/ head branch. Before the real dependency-review action runs, the job requires an open, same-repository PR targeting main, an exact README-only diff, and a head branch and SHA matching the workflow run. It obtains the immutable base SHA from GitHub and rejects a missing, malformed, or identical base/head comparison. Invalid scope or unavailable metadata fails the check rather than skipping the review. The normal pull_request path, including fork contributions, remains unchanged.

OpenSSF Scorecard findings and triage¶

scorecard.yml runs weekly and on manual dispatch from main. The official SHA-pinned action assesses repository supply-chain practices and produces SARIF for GitHub code scanning; it also publishes results to OpenSSF's public Scorecard API. This assessment is separate from the OpenSSF Best Practices project's self-assessment. Its job is limited to contents: read, security-events: write, and id-token: write; the OIDC permission is used for Scorecard publication. The workflow does not use a personal access token or repository-write permission. It runs independently of Dependency Review and is not included in the seven validation workflows dispatched for generated README proposals. Keep show-openssf-scorecard: false in Dependency Review so its pull-request gate remains focused on introduced dependencies.

Review Scorecard alerts individually under Security → Code scanning. Scorecard provides a supply-chain assessment, not a replacement for CodeQL, Gitleaks, Dependency Review, zizmor, Trivy, or CI. The workflow fails if Scorecard analysis or SARIF publication fails, but individual Scorecard findings are informational: there is no aggregate score threshold or Scorecard finding configured as a merge gate. Address real weaknesses through the relevant repository settings or project controls; do not broaden permissions, disable existing checks, or add suppressions simply to improve a score.

Use the following context when triaging:

Branch protection: verify the default branch, rulesets or branch-protection rules, required checks, and review policy in GitHub settings. Repository rulesets are visible to Scorecard's default token; some classic branch-protection details may not be, so verify them directly rather than granting an administrative token.
Best Practices (#34): the OpenSSF Best Practices project records its current assessment and badge status. Scorecard reads that external project record; adding a badge image alone cannot resolve the alert. Review the evidence for each claim, distinguish in-progress from passing, and rerun Scorecard after changes to the record.
Pinned dependencies: this repository pins third-party Actions to full commit SHAs, container images to digests, and application dependencies with uv.lock and site/bun.lock; Dependabot is configured for updates. The $/... syntax resolves repository-owned workflows and actions; Scorecard may misidentify these as unpinned third-party actions. Do not change them to ./... merely to silence Scorecard: blocking zizmor recommends $/... for self-repository references.
Fuzzing: the website unit suite runs seeded, bounded fast-check properties against the public API query parser. This is offline input fuzzing, not a live-source test. Compare any Scorecard fuzzing finding with the suite's actual scope.
SAST (#31) and CI-Tests (#35): Scorecard samples recent commits and merged pull requests. CodeQL and the offline test workflows already run on every pull request and push to main; the README automation explicitly dispatches and waits for those checks because GITHUB_TOKEN pushes do not trigger normal CI. A historical missing check is not repaired by changing triggers; investigate failed or absent checks on future commits before merging, and let the sample roll forward rather than fabricating status results.
Token permissions and workflows: compare repository-wide token defaults with each workflow's explicit permissions. Actionlint and blocking zizmor audits are the dedicated workflow checks; Scorecard complements them and does not replace them.
Packaging and releases: v1.0.0 establishes the first stable release line; SECURITY.md documents security support. Check release-related findings against actual tags, published artifacts, and validation evidence; package metadata alone does not establish publication. Do not publish a release solely to raise the Scorecard result; record the rationale when triaging such an alert.

Repository security settings and required-check configuration are maintained in GitHub, not by this workflow. Follow Protected environments and repository settings for the repository's required settings.

Validation jobs have explicit timeouts, cancel superseded runs only on the same workflow/ref, and checkout without persisted Git credentials. Third-party actions and CI tool images are pinned to immutable revisions where practical and should remain pinned. Runtime and package-manager versions should stay explicit rather than being resolved through latest-release APIs.

Equivalent local commands are documented in the testing guide.

Reusable automation boundaries¶

The operator-facing workflows remain small mode selectors:

nightly.yml selects availability followed by scrape unless the audit stops source access, or scrape only when a manual run sets skip_availability=true;
check-availability.yml selects availability only;
collection mode in scrape.yml selects scrape only;
deployment mode in scrape.yml selects projection regeneration followed by deployment;
add-job.yml selects one atomic manual batch upsert followed by full projection regeneration;
canonical-state-drill.yml normally selects restore, export, validation, and round-trip publication without source access; its mutually exclusive seal-adoption and README-recovery modes read verified state and propose a README-only review without durable publication.

All canonical-state modes call reusable-process-state.yml. This keeps restore/bootstrap, migration, CLI exit-code handling, validation, checkpoint, durable publication, and sanitized-artifact ordering identical, except that the two README review modes deliberately skip durable publication. Deployment mode selects the production environment and runs the final locked deployment in that same protected job, so canonical SQLite never crosses jobs through a GitHub cache or artifact. scripts/database/restore_canonical_state.sh encapsulates restricted-snapshot reconciliation and the verified live-database fallback. scripts/database/canonical_state_store.sh owns immutable SFTP snapshot restore/publication. scripts/deployment/deploy_canonical_state.sh and scripts/deployment/activate_canonical_release.sh own locked, checksum-verified versioned deployment with a single atomic release-pointer cutover.

The wrappers, rather than the reusable processor, decide which source phase runs, which protected environment applies, and whether README mutation or deployment is needed. Canonical processing never pushes Git branches, and the README mutation workflow never receives VPS credentials or canonical SQLite state.

Workflow permissions¶

The repository's default GITHUB_TOKEN permission is read-only. Workflow defaults are read-only or empty; permissions are elevated at job boundaries only. The reusable README workflow defaults to no token permissions; its single mutation job declares the required writes, and callers explicitly pass those permissions to that job. Scorecard may still warn about these necessary job-level writes; do not replace them with a broader token just to suppress a warning.

Job class	Token permissions	Credentials and data
Validation and canonical processing	`contents: read`	Source checkout; `canonical-state` jobs receive restore/publication credentials only
Protected deployment processing	`contents: read`	`production` job restores, validates, snapshots, and deploys without a database handoff artifact
Main-branch container evidence attestation	`actions: read`, `contents: read`, `attestations: write`, `id-token: write`	Runs only on `main`; signs CI SBOM and build-evidence files, not unpublished container images
README pull-request mutation	`actions: write`, `contents: write`, `pull-requests: write`	One-day English and five localized README handoff only; no VPS credentials or SQLite artifact; explicitly dispatches validation on the generated commit
Dependency Review	`contents: read`, `pull-requests: read`	Dependency comparison and dispatched README proposal metadata only; no canonical state or VPS credentials
CodeQL analysis	`actions: read`, `contents: read`, `security-events: write`	Security-analysis upload only
Scorecard analysis	`contents: read`, `id-token: write`, `security-events: write`	Default-branch assessment, OIDC-authenticated public Scorecard result publication, and SARIF upload only

GitHub cache entries on the default branch are readable by pull-request workflows, including forks, and public-repository artifacts are available to anyone with repository read access. Canonical SQLite therefore never enters Actions cache or artifacts. Thirty-day artifacts include the already-public README, sanitized CSV/JSON projections and metadata, aggregate-only collection-quality reports, and Docker CI's production-image SBOMs and build-evidence manifest. Quality reports contain counts, rates, search slugs, and check outcomes, never listing content. The one-day cross-job artifact contains only the English README and its five supported translations. Docker evidence records image IDs, any available repository digests, and SBOM checksums; main-only GitHub attestations bind the evidence files to their workflow run. Checkout never persists Git credentials.

GitHub suppresses push-triggered workflow recursion for GITHUB_TOKEN; pull-request workflows created by that token can require manual approval. See GitHub's workflow-trigger behavior. After verifying that the automation pull request targets main, uses a same-repository fixed head branch and the expected title, and changes only paths in the six-README allowlist, the mutation workflow therefore dispatches python-ci.yml, site-ci.yml, docker-ci.yml, codeql.yml, gitleaks.yml, documentation.yml, and dependency-review.yml explicitly on the generated head commit. The dependency-review dispatch also receives the PR number. It identifies those exact dispatch runs and waits for all seven to succeed before requesting the nightly merge against that validated head SHA. This keeps the branch alive long enough for GitHub to create every job even when branch protection is missing or misconfigured; configured required checks and human review remain additional merge controls. Manual scrape and availability pull requests receive and await the same validation but still require human merge.

Protected environments and repository settings¶

Production automation requires two GitHub environments. Restrict both environments to the main branch; do not allow arbitrary tags or branches.

Environment	Used by	Secrets	Review policy
`canonical-state`	Restore, migration, authorized collection/audit or manual insertion, validation, and snapshot publication	`VPS_HOST`, `VPS_USER`, `VPS_SSH_PRIVATE_KEY`, `VPS_BACKUP_SSH_PRIVATE_KEY`, `VPS_SSH_KNOWN_HOSTS`	Must permit unattended scheduled runs; use a main-only branch policy and no required reviewer
`production`	Restore, validate, snapshot, and checksum-verified VPS deployment	`VPS_HOST`, `VPS_USER`, `VPS_SSH_PRIVATE_KEY`, `VPS_BACKUP_SSH_PRIVATE_KEY`, `VPS_SSH_KNOWN_HOSTS`	Main-only; a required maintainer approval is recommended

Configure the branch restriction in Settings → Environments, separately for canonical-state and production:

Under Deployment branches and tags, choose Selected branches and tags.
Add a Branch rule for the exact name main, and remove other branch or tag rules. No restriction does not protect environment secrets; Protected branches only is not equivalent to allowing only main.
Keep canonical-state free of required reviewers so scheduled runs can start. For production, configure the intended maintainer approval policy.
Reopen each environment and verify the saved policy before supplying credentials or enabling scheduled collection.

A branch check inside a workflow is defense in depth, not a replacement: another branch can change its workflow code. These settings are not created or repaired by any checked-in workflow.

VPS_SSH_PRIVATE_KEY is present in canonical-state only for the reviewed live-database bootstrap used when durable snapshot state is absent. Normal restore/publication uses the restricted backup key. Deployment mode needs both keys in production: the restricted key restores and republishes verified canonical state, while the deployment key performs the final replacement. After moving these values into environments, delete identically named repository secrets and remove this repository's access to equivalent organization secrets so another branch cannot access them outside environment protection.

Keep these non-secret values as repository variables:

Variable	Required	Production configuration / fallback
`LINKEDIN_CRAWL_AUTHORIZED`	For source access	No enabled default
`OPPORTUNITIES_MAX_CONCURRENCY`	Yes	Set to `1`
`OPPORTUNITIES_RATE_LIMIT_SECONDS`	Yes	Set to `10`
`OPPORTUNITIES_AVAILABILITY_INTERVAL_DAYS`	Yes	Set to `5`
`OPPORTUNITIES_AVAILABILITY_MAX_JOBS`	Yes	Set to `100`; built-in default and workflow fallback remain `50`
`VPS_BACKUP_USER`	No	`opportunities-backup`
`VPS_SSH_PORT`	No	`22`
`CANONICAL_STATE_RETENTION_DAYS`	No	`365`

Repository configuration must also:

set the default GITHUB_TOKEN permission to read-only, permit GitHub Actions to create pull requests, and keep elevated permissions explicit at job boundaries;
require third-party Actions to use full-length commit SHAs and allow squash merging;
protect main and require the check contexts emitted by the current workflows: ruff, python, site, docker, Analyze (Python), Analyze (TypeScript), Gitleaks secret scan, build (Documentation site), and dependency-review;
prevent direct pushes and choose the review policy deliberately—required approvals still block the nightly merge, whether it uses GitHub auto-merge or a normal merge;
enable the dependency graph, Dependabot alerts and security updates, secret scanning and push protection, private vulnerability reporting, and CodeQL code scanning where GitHub makes those controls available;
retain Actions logs and the documented 30-day sanitized projection artifacts according to repository policy.

The environment branch rules are part of the security boundary, not optional documentation. Both reusable workflows also fail explicitly when github.ref is not refs/heads/main, including the repository-mutation boundary that does not receive environment secrets.

First production activation¶

Complete this once before relying on the scheduled run:

Merge the release commit into the default main branch before the scheduled time; schedules always use the default-branch workflow revision.
Create and restrict the canonical-state and production environments exactly as described above, then remove broader copies of their secrets.
Configure the repository variables and verify that LINKEDIN_CRAWL_AUTHORIZED=true reflects current express authorization rather than convenience.
Enable Actions pull-request creation, squash merging, branch protection, and the documented required check contexts.
Complete the one-time public-state seal review from main, then run Verify canonical state recovery with its default input; require successful migration, projection validation, round-trip snapshot verification, and the retained sanitized projection artifact.
Only after that verified durable snapshot exists, delete every legacy opportunities-db-* Actions cache and legacy opportunities-state-* or opportunities-nightly-state-* artifact; these older state bundles are neither approved backups nor safe public artifacts.
Confirm no local or VPS collector can write the same database and no stale operational workflow is still running or queued.
Confirm Nightly full update is enabled and the repository is active enough for GitHub scheduled workflows.

Before switching an existing production website to versioned releases, complete the coordinated first rollout; a successful recovery drill does not switch website readers or populate data/current. Do not use a live scrape as the first test of SSH, environment, snapshot, or branch-protection configuration. The recovery drill exercises those paths without LinkedIn access.

First public-state review seal¶

The generated README includes a hidden SHA-256 seal of every website-visible open row and the exact last successful collection time. The visible tables remain limited to five jobs per type. A below-preview title, location, category, or first-seen change therefore changes the README proposal and must be reviewed before deployment.

For a legacy installation whose reviewed README predates the seal, run Verify canonical state recovery with adopt_public_review_seal=true. This mode restores a verified canonical snapshot, or validates a consistent live-database backup under the first-bootstrap rules, migrates its temporary working copy, checks that the old generated README matches apart from the new seal, and uploads sanitized projections. It performs no LinkedIn request and publishes no new durable snapshot or production release. It opens a manual-review, README-only automated/public-review-seal pull request. Inspect its sanitized CSV/JSON artifact and generated seal, then merge it. A README that already contains a reviewed seal needs no adoption. If the old README differs for any other reason, the run fails and the prior state proposal must be reconciled first. Until this baseline PR merges, normal mutation and deployment modes fail closed at the reviewed-state barrier. Subsequent recovery drills use the default input and continue round-trip snapshot verification.

Recover a missing README state proposal¶

A snapshot can be newer than the reviewed README when snapshot publication succeeds but the proposal job fails, or when its pull request is closed without merging. The reviewed-state barrier then correctly blocks further mutation and deployment. Repeating collection cannot repair this mismatch.

Check for the matching open README proposal first. If it exists, resolve its checks and review it rather than creating a competing proposal.
If the proposal is missing or closed, run Verify canonical state recovery from current main with recover_readme_proposal=true and adopt_public_review_seal=false.
Review the retained sanitized projection artifact and the automated/readme-recovery pull request. Reconcile any older state proposals so they cannot later merge stale projections. Wait for all dispatched validation checks and merge the recovery PR manually only if the restored public state is correct.
Start a fresh workflow from updated main, or wait for the next nightly schedule. GitHub's rerun action uses the original commit and can still compare against the stale README. Deployment remains a separate reviewed-state operation.

Recovery restores and verifies canonical state, migrates only its temporary working copy, renders projections, and validates them before the sanitized handoff. It performs no LinkedIn request, manual job addition, durable snapshot publication, or deployment. The reusable processor rejects recovery combined with any mutation/deployment input, a missing README handoff, or seal adoption. The recovery PR never requests auto-merge; ordinary workflows still require exact agreement with reviewed main.

This includes rejected or unmerged partial proposals: their completion timestamps and lifecycle state are already in the protected verified snapshot, but the review barrier prevents the next run from silently accepting that state. Closing the PR does not roll back SQLite or its rotation. A missing proposal may be recovered read-only; a rejected dataset requires an explicitly approved, verified recovery that reconciles both canonical state and reviewed README before collection resumes. Never restore scheduling metadata separately from its database.

If a prior proposal was rejected because its data was wrong, do not approve the same state merely to unblock automation. Preserve the snapshot and follow canonical-state recovery instead. Neither README recovery nor a sanitized artifact rebuilds or rolls back SQLite.

Collection authorization¶

Every nightly, scrape-only, or availability-only mode that accesses LinkedIn requires this repository variable:

LINKEDIN_CRAWL_AUTHORIZED=true

When the variable is missing or has another value, a LinkedIn-accessing run stops before network access. A manual scrape-workflow run with deploy_to_vps=true skips collection and availability requests, so deployment-only mode does not require this interlock.

Important

This variable is an operator attestation, not permission. Enable it only after express authorization has been obtained, and disable it immediately when authorization is absent, uncertain, expired, or withdrawn.

Local and Docker interlocks are documented in Configuration. The complete source-access boundary is defined in SECURITY.md.

Schedule and concurrency¶

The nightly full update uses a three-day UTC cadence anchored to October 17, 2026: October 20, 23, 26, and 29, then November 1, and so on. Maintainers can request additional manual updates when many new roles appear or hiring-market developments warrant closer monitoring. A daily cron trigger starts only a lightweight date check; canonical processing and README publication are skipped on other days:

on:
  schedule:
    - cron: "23 4 * * *"
  workflow_dispatch:
    inputs:
      skip_availability:
        description: Skip the stored-job availability audit and only scrape listings.
        required: false
        default: false
        type: boolean

concurrency:
  group: opportunity-collection
  cancel-in-progress: false

The date-check job has no source or VPS credentials and compares the current UTC day with the fixed reference date. This avoids the month-boundary reset of a day-of-month cron step such as */3. Only run_update=true permits the state-processing job to start. The nominal time on eligible dates is 04:23 UTC; GitHub may delay or miss scheduled runs, and the gate uses the UTC date when it starts rather than replaying missed updates. A manual run bypasses only this date gate and does not reset the cadence.

Eligible scheduled runs always attempt the availability audit first; skip_availability is effectively false for schedules. An eligible HTTP 429 partial audit skips scraping and proceeds only through validation and manual-review publication. Manual runs default to the same sequence but can skip the separate audit. Authorization, source-stop rules, and the shared one-writer lock apply to both paths.

The processor reads request and availability limits from Actions variables and has a 360-minute default timeout. Set the GitHub Actions repository variable OPPORTUNITIES_AVAILABILITY_MAX_JOBS=100 for production; the built-in default and workflow fallback remain 50 when no value is configured. Check repository, organization, and selected-environment variables, including copies in canonical-state and production, for conflicting overrides such as 50 or legacy 250. Update or remove conflicting overrides so the effective production cap is 100, and verify it in the audit's selected/deferred log. No workflow changes these variables automatically. Keep OPPORTUNITIES_AVAILABILITY_INTERVAL_DAYS=5, concurrency 1, and request spacing 10 seconds.

With the production override, each scheduled audit selects at most 100 due jobs, with at least five days between non-denial attempts. Never-checked and oldest-checked jobs go first; excess due jobs wait for later runs. At one run every three days, 800 initially due jobs require eight completed batches: the last batch is checked 21 days after the first, and the first batch becomes the oldest again at the next run, around day 24. New jobs join the never-checked group. Larger backlogs, new arrivals, source denials, or missed runs can extend the rotation. The five-day minimum is not a promise that every listing is checked within five days.

An audit normally makes up to two requests per selected job. At the production cap of 100 checks per scheduled run, this means up to 200 normal availability HTTP requests every three days plus bounded retries for transient failures; it does not guarantee continued LinkedIn access. Scraping adds separate bounded search, detail, and known-job recheck requests. Existing title/company prefilters and in-flight detail deduplication avoid unnecessary discovery requests without reusing stale completed evidence. HTTP 429 still stops source access without automatic retries. Scraping starts with never-completed and oldest-completed searches, using SQLite completion timestamps and stable registry ties. Denied/skipped searches do not advance, so they precede newly completed groups in the next eligible execution. Rotation survives verified snapshots; interruptions and failed validation do not publish progress. These settings apply on GitHub, not Dokploy.

The nightly, scrape-only, availability-only, manual-add, recovery-drill, and deployment paths share opportunity-collection. This prevents overlapping writers and state replacement while allowing the read-only website to continue serving requests. GitHub concurrency is not a durable FIFO queue: at most one run waits, and a newer pending run can replace it. Avoid dispatching competing operations; confirm the intended run actually starts, and start a fresh run from current main if necessary.

Manual collection inputs¶

Three collection workflows can be run independently from the Actions tab:

Nightly full update audits unless manually skipped, then collects only while source access remains available. Fully successful collections can request a squash merge after validation and configured review requirements; eligible HTTP 429 partial audits skip collection and require manual review and merge.
Check job availability checks a bounded batch of due rows and opens an availability-only pull request.
Scrape jobs or deploy reviewed state runs only the scrape and opens a scrape-only pull request when deployment mode is disabled.

To run the combined update now, open Actions → Nightly full update → Run workflow, select main, leave skip_availability unchecked, and confirm Run workflow. Check it only to collect without the separate stored-job audit. The input must be merged into the default branch before GitHub shows this control. Manual and scheduled updates use the same source-authorization checks, shared concurrency lock, and reviewed-state safeguards; neither deploys to the VPS.

The nightly workflow input is:

Input	Default	Effect
`skip_availability`	`false`	Manual runs only: `true` skips the separate due stored-job availability audit and still scrapes; `false` audits before scraping. Scheduled runs always audit.

Skipping the audit does not disable scraping's bounded known-job rechecks or change its lifecycle evidence rules. Rows not checked by collection remain unchanged, and the separate audit queue does not advance. The generated README proposal states whether the audit was skipped. Full collections retain the existing automatic-merge policy; partial collections still require manual review and merge.

The scrape workflow inputs are:

Input	Default	Effect
`deploy_to_vps`	`false`	Skip collection and deploy reviewed durable SQLite state through locked, checksum-verified versioned release publication after validation against `main`

The availability workflow has no inputs. Availability-only and scrape-only updates open their own README proposals when public state changes; both retain the review handoff after snapshot verification. None of the manual workflows permits an automatic state rebuild: migration or canonical-state validation failures stop the run. VPS deployment is allowed only from main when a manual scrape-workflow run explicitly sets deploy_to_vps=true; scheduled runs preserve state and update the combined pull request but do not deploy.

Manual job addition¶

For on-demand evidence gathering before this operation, use the LinkedIn opportunity scout. It compares against the complete public open dataset and saves an ignored report; it stops for separate candidate and state-changing dispatch approval. The workflow below mutates canonical state before its README review PR exists.

Use Add manually reviewed jobs when a maintainer has independently reviewed 1–10 known public LinkedIn listings that should enter canonical state without running discovery. Run it from main once for the batch and paste a JSON array into jobs_json. Each object requires url, company, title, location, category, and employment_type; optional keys are industries, start_date, and posted_at. JSON names use underscores. The input is limited to 64 KiB and rejects unknown keys or duplicate LinkedIn IDs. See the CLI batch example for the format.

The workflow acquires the shared opportunity-collection lock and enters the main-only canonical-state environment. It restores and verifies the latest canonical snapshot before migration, runs opportunities add-jobs --no-render against the batch, regenerates all normal projections, validates them, checkpoints SQLite, and publishes and restore-verifies one new durable snapshot. The CLI validates every entry and applies them in one repository transaction, so an invalid or closed job cannot leave a partial batch. It applies the deterministic classifier to operator-supplied title, location, employment type, category, cycle, and posting-date evidence; a maintainer must verify the source facts independently, because this offline path cannot authenticate them. It creates no synthetic search, search run, or provenance, performs no LinkedIn request, and does not require LINKEDIN_CRAWL_AUTHORIZED=true.

Only sanitized public projections are retained. A one-day README-only handoff is passed to the existing mutation workflow, which opens or updates automated/manual-jobs-YYYY-MM-DD with a title of the form data: add manually reviewed jobs (YYYY-MM-DD). The date is the workflow run's creation date in Europe/Rome. The pull request changes only the English README and its five localized projections, runs the normal validation workflows, and remains manual-review only; this path never requests auto-merge. Canonical SQLite is not uploaded as an artifact or cache entry and is not written directly to the live VPS database.

Submit all jobs intended for this pull request in the same run. The reviewed-state barrier blocks another canonical mutation while its README proposal is unmerged. After reviewing and merging that README pull request, deploy production separately through Scrape jobs or deploy reviewed state from main with deploy_to_vps=true. Deployment restores the latest snapshot pointer, which now contains the batch, and validation requires the merged README to match that state before the versioned release-pointer cutover. Running deployment before the README merge stops at projection validation rather than dropping or prematurely deploying the manual jobs.

Collection and deployment flow¶

The state-writing workflows share the same recovery, migration, validation, snapshot, and one-writer guarantees, but their source phases differ.

Collection and update path¶

source authorization check, when source access is selected
↓
checkout + pinned runtime setup
↓
restore the latest restricted VPS snapshot manifest and database
↓
optional verified VPS bootstrap when no durable source exists
↓
database migration
↓
availability audit, when selected
↓
bounded collection + classification, when selected and the audit did not stop source access
↓
isolated transactional persistence + aggregate drift checks when scraping
↓
quality warnings continue; blocking drift stops publication and still retains its report
↓
README + sanitized CSV/JSON projection validation
↓
SQLite WAL checkpoint
↓
timestamped SFTP snapshot + round-trip restore verification
↓
retained sanitized public-projection artifact
↓
one-day README-only handoff artifact
↓
separate least-privilege README pull-request job

Nightly runs audit first unless a manual run sets skip_availability=true. An audit source stop prevents scraping. Availability-only runs never scrape; scrape-only runs never audit. Manual additions replace both source phases with the reviewed batch operation.

Every scrape requests data-quality-report.json; the CLI reference defines its thresholds and baseline rules. Warnings continue. Blocking findings or report errors stop projection rendering, snapshots, and the README handoff. The upload retains a report even after a failed scrape; a missing requested report fails the job. If scraping never starts, there is no collection-quality artifact.

Quality checks run after individual search commits and do not roll back successful transactions. Failed searches supply diagnostics, not absence or closure evidence. Partial collections do not advance the quality baseline.

HTTP 429-only partial updates follow the source-access policy. Eligible audits or collections return 4 and emit collection_outcome=partial. An audit partial skips scraping, but still passes SQLite integrity, projection, and round-trip snapshot checks. Preserved public-listing HTTP 301 responses do not disqualify an otherwise eligible audit. Collection also requires its quality gate. Other source denials or failures mixed with HTTP 429 block publication.

Partial README proposals require manual review and merge. Production continues serving its reviewed release until the matching proposal is merged and separately deployed.

Before mutation or deployment, the processor renders the restored snapshot and requires exact agreement with the reviewed English README on main. Its seal covers every public row and exact first-seen and collection timestamps, not just the visible preview. Unmerged or rejected proposals block further ordinary runs. Use seal adoption only for an otherwise matching legacy README, or proposal recovery when the matching proposal is missing or closed. Those modes validate and propose state without collecting, publishing a snapshot, or deploying.

Availability evidence and queue rotation are defined in Database lifecycle. An inconclusive audit without a source denial returns 2 and permits scraping; a source denial never does. Confirmed audit results can become durable only through the validated snapshot path above. Blocked and interrupted working copies are not recovery sources: the next run restores the last verified snapshot. Review authorization before another source-access run.

Deployment-only path¶

A manual scrape.yml run from main with deploy_to_vps=true does not collect or audit LinkedIn. It follows a separate publication path:

restore reviewed durable SQLite state
↓
migrate + generate sanitized CSV/JSON exports
↓
validate README and exports against canonical state
↓
SQLite WAL checkpoint
↓
publish + round-trip verify canonical snapshot
↓
upload database + exports to a unique VPS staging directory
↓
acquire deployment lock, verify checksums and seal the release
↓
atomically switch the release pointer

The deployment mode and README validation prevent newly collected state from being deployed before its README projection has been reviewed and merged. Keeping restore, validation, snapshot publication, and deployment in one approved production job avoids exposing the database through cross-job storage.

Exit-code handling¶

CLI exit code	Workflow behavior
`0`	Continue after complete success
`1`	Stop after a nonqualifying audit source stop, mixed audit failures, scraping with no completed search or a failure other than HTTP `429` or source-skipped work, or a blocked or incomplete quality gate
`2`	Continue only for inconclusive availability auditing; reject this code from scraping as input/configuration failure
`3`	Stop because schema or state preconditions failed
`4`	Validate and preserve eligible HTTP `429` partial scraping or auditing, explicitly labeled partial and requiring manual review and merge; an audit partial skips scraping

Validation must pass after complete or partial success.

Command-level semantics are documented in the CLI reference.

State continuity and artifacts¶

A dedicated, chrooted, SFTP-only VPS account stores recovery snapshots. Each state-writing workflow downloads /state/canonical-state/latest.json, validates its bounded manifest, downloads the referenced timestamped database, and verifies size, SHA-256, schema revision, collection timestamp, SQLite integrity, foreign keys, and required tables. Authentication, host-key, directory-listing, transfer, manifest, checksum, SQLite sidecar, or integrity failures stop the workflow rather than silently using stale state. Only successful directory listings can establish that snapshot storage is absent; a failed or empty manifest download never authorizes bootstrap. An existing snapshots directory without latest.json requires manual reconciliation, even if that directory is empty. A pre-existing local database that does not match the manifest also stops recovery without overwriting it. Local WAL/SHM/journal files and pre-existing restore or bootstrap staging files are preserved for inspection. Publication also rejects an existing local .snapshot-copy replacement path, including directories, symlinks, dangling symlinks, and its SQLite sidecars, before contacting storage. Final replacement staging is created exclusively so a path claimed later cannot be overwritten.

GitHub Actions cache is deliberately not used for canonical SQLite. GitHub documents that pull-request workflows, including forks, can read default-branch caches. During first-time bootstrap, when no snapshot manifest or history exists, the workflow may instead stream a consistent SQLite backup of the reviewed live database over SSH (including committed WAL changes, without copying sidecars). This requires Python 3 on the deployment host. The backup normalizes SQLite to DELETE journal mode so it can later be read as a cold file without WAL/SHM permissions. When no durable snapshot exists, a pre-existing local database is preserved but never accepted as a bootstrap source. The streamed candidate must independently pass integrity, foreign-key, required-table, and Alembic-revision checks before migration.

After validation and WAL checkpointing (and only when no live SQLite sidecars remain), publication creates a consistent SQLite backup through the SQLite backup API and normalizes its journal mode to DELETE before hashing, so the read-only site does not need writable WAL/SHM sidecars. Snapshot paths resemble:

/state/canonical-state/snapshots/2026/07/20/20260720T030000.000000Z-run-123-attempt-1.db
/state/canonical-state/snapshots/2026/07/20/20260720T030000.000000Z-run-123-attempt-1.manifest.json

Every strict JSON manifest records the database path, byte size, SHA-256, Alembic revision, collection and creation timestamps, preceding database and manifest references, workflow source, and retention metadata.

Publication uploads new paths, downloads both files into a clean directory, verifies SQLite and application readability, and only then atomically renames a temporary latest.json. The working and deployment copies are byte-identical to the verified download. Failures before pointer promotion leave the prior pointer in place; a disconnect after promotion may leave the new, already verified pointer in place. The restricted account has no shell, sudo, forwarding, application-database access, or membership in opportunities-site.

Publication retries recognized SSH transport failures (status 255), with at most three reconciliation attempts per write phase and three attempts per read operation, using two- and four-second backoff. It inspects final and run/attempt staging objects before repeating writes, accepts byte-identical immutable objects, repairs only staging bytes proven to be a truncated prefix of the local bundle, and stops on conflicting immutable or staging bytes. A latest pointer that differs from both the verified candidate and the previously fetched pointer stops publication rather than overwriting newer state. Authentication, host-key, permission, checksum, and application-validation failures are not retried.

The local publication bundle is retained for same-run/attempt resumption while the private working directory survives; a fresh runner does not recover that bundle automatically. Unknown staging files are preserved for manual review, not deleted. The shared workflow concurrency lock remains required to prevent concurrent pointer writers.

Thirty-day projection artifacts contain only README.md, the sanitized CSV/JSON exports, and dataset-metadata.json; a separate artifact contains the aggregate quality report. These are verification outputs, not recovery sources. No production database or manifest is uploaded to Actions cache or artifacts. Repositories upgrading from an older cache/artifact state handoff must purge legacy opportunities-db-*, opportunities-state-*, and opportunities-nightly-state-* objects after the first restricted snapshot has passed the recovery drill. By default, VPS manifests declare a 365-day retention window; automation does not delete older snapshots. Capacity must be monitored and expiry reviewed manually after retain_until. Because snapshot storage is on the same VPS as production, it protects against accidental database replacement but not complete VPS, disk, or provider loss. Replication of encrypted or access-controlled snapshots to an independent host remains the recommended next durability layer.

Canonical backup, sidecar, migration, and restoration rules belong to the database lifecycle guide.

Restricted VPS snapshot configuration¶

The account is expected to be named opportunities-backup, chrooted at /srv/opportunities-backup, and forced into internal-sftp -d /state. Its writable directory is /srv/opportunities-backup/state with mode 0700. The account must use a dedicated key, must not have sudo or shell access, and must not belong to opportunities-site.

Dedicated `canonical-state` environment secret¶

Secret	Purpose
`VPS_BACKUP_SSH_PRIVATE_KEY`	Dedicated unencrypted Ed25519 private key for the SFTP-only account

The canonical-state and production environments also supply VPS_HOST and VPS_SSH_KNOWN_HOSTS; production needs the backup key so deployment mode can restore and republish state without a GitHub artifact. VPS_SSH_PORT remains a repository variable. The verified known-hosts entry is mandatory and host-key checking is strict.

Snapshot repository variables¶

Variable	Required	Default
`VPS_BACKUP_USER`	No	`opportunities-backup`
`CANONICAL_STATE_RETENTION_DAYS`	No	`365`

The private key is written only for restore/publication steps and removed with known_hosts afterward. The normal VPS deployment key remains separate. After configuring GitHub, complete the one-time public-state seal review, then run Verify canonical state recovery with its default input. The default drill performs no LinkedIn access. A first bootstrap is permitted only when successful storage listings establish that no snapshot pointer or history exists and no versioned release has been published. An existing snapshots directory, even if empty, blocks bootstrap without latest.json.

README update pull requests¶

Each path uses a separate review branch pattern:

automated/nightly-full-update  # full collection; audit unless manually skipped
automated/nightly-full-update-partial # validated partial; manual review
automated/availability-update  # manually requested availability only
automated/availability-update-partial # validated partial audit; manual review
automated/scrape-update        # fully successful manual scrape
automated/scrape-update-partial # validated partial manual scrape
automated/manual-jobs-YYYY-MM-DD # manually reviewed job batch
automated/public-review-seal   # one-time baseline from verified state
automated/readme-recovery      # manual recovery of a missing state proposal

Dataset proposals include only README.md and the five docs/translations/README.<language>.md files. The renderer refreshes their generated counts, full-state seals, and bounded previews together, preserving authored translation text. Canonical processing uploads these public Markdown files in a one-day README-only handoff, then a separate job with actions: write, contents: write, and pull-requests: write downloads them and performs the GitHub mutation and validation dispatch. SQLite and snapshot manifests never enter that handoff or Git. Processing retains only contents: read; the mutation job receives no VPS credentials or state bundle.

The generated preview remains bounded to five open opportunities per employment type, ordered by immutable first-seen time and then descending ID text, regardless of database size. First-seen time can reflect approximate source posting age rather than exact discovery order. The seal makes every public-directory change produce a README diff, even when all changed rows are outside that preview. Review the sanitized public CSV/JSON artifact linked from the pull-request body before merging a seal-only diff. Those downloads exclude first-seen timestamps, and the displayed collection time has minute precision; if the artifact does not explain the seal change, inspect the relevant values in protected canonical state before approving. The hash alone does not show which rows changed.

A fully successful nightly collection creates or updates its fixed branch and requests a squash merge. Partial collections and audits use the -partial branch suffix, a partial-update/manual-review title and body, and never request a merge. The reusable proposal helper independently disables automatic merging for partial outcomes and cancels any existing auto-merge on a matching partial proposal before updating it. Both paths dispatch and await the same seven validation workflows.

The collection_outcome output identifies full, partial, or none; partial can describe an audit or a collection. Review audit-only results in the command summary. Scraping also produces an aggregate quality report; auditing does not.

Before mutating an existing proposal, the reusable proposal helper verifies the exact base branch, same-repository head branch, title, and changed-file list; the pull request must target main and modify only paths in the explicit six-README allowlist. Translation-only refreshes are permitted, but the root English README remains the reviewed-state barrier. After pushing, it checks that scope and the expected head SHA before dispatching validation. It retries the post-push GitHub scope read briefly to tolerate API propagation, but never relaxes the expected scope. It waits for the exact Python, site, Docker, CodeQL, Gitleaks, documentation, and dependency-review dispatch runs on that head SHA to succeed, then repeats the scope check and supplies the SHA to the merge request so a later head change cannot use stale validation. If generated state already matches main, a matching stale automation pull request is closed rather than left eligible to merge.

The workflow reads the repository's auto-merge setting without changing it. When enabled, it requests GitHub auto-merge; otherwise, it requests a normal squash merge after validation. Neither mode bypasses required checks, up-to-date-branch rules, or review requirements. If the setting cannot be read, the job stops without requesting a merge. A normal merge refused by branch protection fails the job and leaves the proposal open: resolve its checks or reviews and merge that proposal before another canonical-state run. Scrape-only, availability-only, and manual-add pull requests receive the same dispatched validation but remain manual-review paths.

Scheduled runs do not deploy to the VPS. After the nightly pull request merges—or after a maintainer reviews and merges a manual update—start a manual scrape-workflow run from main with deploy_to_vps=true to publish the matching canonical state. Deployment mode skips collection and availability requests, then requires the merged README to validate exactly against restored durable SQLite before deployment. The pull request contains the human-readable README projection; SQLite remains inside protected snapshot storage and the approved production job and is never committed, cached, or uploaded as an artifact. Do not edit generated rows manually; change the renderer or canonical state instead.

Post-nightly production runbook¶

Use this sequence after each scheduled collection. Do not deploy merely because the scheduled job started; deployment is permitted only after the matching README projection reaches main.

Open the Nightly full update run and confirm Audit, collect, and preserve state succeeded. Read the command summaries for partial availability or partial collection. A reported partial result is intentionally preserved and may proceed only because final validation and snapshot verification passed.
Confirm the run retained the 30-day opportunities-nightly-projections-<run-id> artifact and completed round-trip snapshot verification. If scraping ran, also require opportunities-nightly-projections-<run-id>-data-quality and review its warnings. An audit-only HTTP 429 partial instead reports that collection was skipped and creates no collection-quality artifact. No artifact may contain SQLite or a snapshot manifest. A blocking quality report means no README proposal or durable snapshot should have been published.
Inspect automated/nightly-full-update, or automated/nightly-full-update-partial for a partial collection or audit. It must target main, have the matching full or partial/manual-review title, and change only paths in the explicit allowlist: README.md and docs/translations/README.{it,fr,de,es,zh-CN}.md. The one-day multilingual README handoff is separate from the 30-day public-projection artifact, which contains only the root README, sanitized CSV/JSON, and dataset metadata. Partial proposals require explicit human review and manual merge; do not enable auto-merge on them.
Confirm the README mutation job identified and awaited the explicitly dispatched Python, site, Docker, CodeQL, Gitleaks, documentation, and dependency-review runs on the pull-request head SHA. Do not bypass, re-label, or manually broaden the automation pull request to make the merge proceed.
Confirm the pull request squash-merged and that main now contains its generated count, timestamp, and preview. If branch protection requires review, inspect the README-only diff and approve it first. If it remains open afterward, diagnose the failed or missing required check before deployment.
From the Actions tab, run Scrape jobs or deploy reviewed state on main with deploy_to_vps=true.
Approve the production environment deployment if required. Confirm Validate, preserve, and deploy reviewed state restores and validates the matching state before its final deployment step. A README/database mismatch is a safety stop, usually meaning the matching projection was not merged or a newer snapshot exists.
Confirm staging, checksum verification, lock acquisition, immutable release publication, and the single pointer switch completed in the deployment log. Never print secrets or database rows while reviewing logs.
Verify the live directory and both fixed downloads over HTTPS. Check that the visible counts and last successful collection time match main, filtering and pagination still work, and CSV/JSON downloads return the expected attachment filenames. Then verify the deployed dataset against the reviewed artifact.
Keep the nightly and deployment run IDs for the operational record. If authorization will not remain valid for the next scheduled run, immediately set LINKEDIN_CRAWL_AUTHORIZED=false or remove the variable.

If any step before deployment fails, leave production unchanged and diagnose the failed stage. If deployment fails, do not collect again as a repair strategy; inspect the lock, staged uploads, release directory, and current symlink. Before pointer promotion the previous release stays live; afterward select a verified prior release under the lock if rollback is required. Canonical snapshot recovery remains separate.

Verify the deployed dataset¶

After a dataset deployment or application release, request /api/v1/status over HTTPS. The endpoint reads served state only; it does not compare GitHub state or initiate collection.

Require 200 and Cache-Control: no-store. Treat 503 as unavailable data, not an empty dataset.
Compare opportunities and dataset_sha256 with total and json_sha256 in the reviewed update's sanitized dataset-metadata.json artifact. Comparing only with production's metadata cannot detect a consistently stale release.
Check last_successful_collection against the reviewed state. The README displays minute precision; the endpoint preserves microseconds. A partial batch can advance this timestamp without every search succeeding.
Confirm release matches the identifier in the deployment log. null means fixed-path mode; it does not confirm that versioned production readers switched successfully.

Export regeneration can change dataset_generated_at without changing listings. The JSON download omits first-seen and collection timestamps, so its hash cannot replace the README review seal or complete publication validation. A mismatch is a failed verification: inspect the selected release and the deployment result rather than collecting again to repair it.

VPS deployment¶

Required `production` environment secrets¶

Secret	Purpose
`VPS_HOST`	Deployment host
`VPS_USER`	Dedicated SSH user
`VPS_SSH_PRIVATE_KEY`	Ed25519 deployment private key
`VPS_BACKUP_SSH_PRIVATE_KEY`	Restricted snapshot key used to restore and republish validated state
`VPS_SSH_KNOWN_HOSTS`	Pre-verified SSH host-key entry

Repository variables¶

Variable	Required	Default or example
`VPS_SSH_PORT`	No	`22`

Deployment sequence¶

After a review pull request is merged, a deployment-mode run enters the approval-protected production environment, restores the verified durable database state, skips all new collection and availability requests, regenerates the sanitized public exports, validates every projection against main, and round-trip verifies a new snapshot. The same contents: read job then:

uploads the validated database, CSV, JSON, and dataset metadata to an execution-specific staging directory under data/.incoming/ (not to the legacy live paths);
under a nonblocking VPS flock, verifies all four SHA-256 digests again and rejects missing files, sidecars, pre-existing release IDs, and invalid pointers;
writes checksums.sha256 for the four payloads, assigns opportunities-site group read access, renames the complete directory to data/releases/<run-id>-<attempt> on the same filesystem, makes it read-only to the site group, and flushes the published files and directories;
atomically renames a relative symlink data/current to point at the new immutable release, then flushes the directory; failed or completed client runs attempt to clean up their staging uploads, while published releases are never removed automatically. Releases are retained for in-flight readers and manual rollback. A failure after pointer promotion cannot be assumed to have left the old release live: inspect current.

The site resolves current once per server operation to a fixed release directory and opens its SQLite database or export there. A cutover cannot mix files within one directory resolution. Concurrent requests may span two revisions; a page followed by a separate download is not a pinned multi-request transaction. Do not garbage-collect a release until all readers of that release have drained. The fixed legacy paths remain unchanged and will become stale; follow the coordinated rollout below before claiming atomic publication.

Coordinated first rollout and rollback¶

This is not a drop-in update for a VPS serving the old fixed paths:

Merge and validate the site release-reader and deployment scripts, then complete the one-time public-state seal review. Preserve the verified restricted snapshot and the old site configuration. Ensure the deployment user can create data/.incoming, data/releases, and data/.release-deploy.lock; the site group must be able to traverse the data directory.
Run deployment-only once while the old site still reads the unchanged legacy files. On the VPS, verify that data/current points to the new releases/<id> and run (cd data/current && sha256sum -c checksums.sha256). Do not switch readers if deployment or verification fails.
Drain old site instances or temporarily remove traffic. Set OPPORTUNITIES_RELEASE_ROOT=/app/data on every site instance, restart them, and route traffic only to those instances. Confirm the directory and both HTTPS downloads reflect the reviewed release. Do not leave mixed legacy/new instances behind a load balancer.

After current exists, never use the fixed legacy database as a recovery source; restore a verified restricted snapshot instead. A missing snapshot pointer is a stop condition when current or the versioned releases directory exists, even if an old fixed file is present.

For rollback, stop new cutovers, confirm the desired prior release's database schema is readable by the running site and matches a verified restricted snapshot, and verify its four payload hashes with (cd data/releases/<prior-id> && sha256sum -c checksums.sha256). Under the same data/.release-deploy.lock (nonblocking flock), create a fresh temporary relative symlink to releases/<prior-id> in data/ and atomically mv -Tf it over current; flush the data directory and check the live site and both downloads. If a temporary symlink already exists, inspect it rather than overwriting it. Keep both releases until readers drain. This reverts the served projection, not the restricted snapshot history; reconcile that history deliberately before the next deployment. Do not roll back an application image that cannot read the chosen schema, or repoint the site to stale legacy paths. If the symlink is missing/invalid, the versioned site fails closed rather than falling back to legacy state.

Compose topology, volume permissions, Dokploy routing, and container diagnostics belong to the Docker and deployment guide.

Recovery and migration failures¶

Normal workflow migrations use:

uv run opportunities db-upgrade

A migration, integrity, manifest, or canonical-state validation failure stops the workflow. Collection workflows do not expose a state-rebuild input and never delete restored state to recover automatically.

Preserve the failed state, then review durable snapshot history and manifests followed by retained versioned releases (or the legacy previous VPS database only if versioned publication has never occurred). Sanitized projection artifacts cannot restore lifecycle state. Restore a verified compatible snapshot rather than initializing an unrelated empty history. Any intentional rebuild is an exceptional manual recovery decision because it loses original first-seen history, search provenance, closure confirmations, and run diagnostics. Follow Database lifecycle and Troubleshooting.

Disabling collection¶

Set the repository variable to false or remove it:

LINKEDIN_CRAWL_AUTHORIZED

Future collection runs stop at the authorization gate without contacting LinkedIn. The variable is read when a run starts; changing it does not revoke the settings already loaded by an active collector.

If permission is withdrawn while collection or auditing is running, also open that run in Actions and select Cancel workflow. Wait for cancellation to finish; requests already in flight may complete. Do not start another source-access run. Preserve the run's evidence and inspect whether snapshot publication completed before cancellation. If a verified snapshot advanced but its proposal is missing, use the source-free README recovery mode, not another scrape.

Do not bypass the gate by hardcoding an enabled value in workflow or application code.

Operational checklist¶

Before changing or manually running automation, confirm:

[ ] Validation CI remains separate from LinkedIn collection.
[ ] LinkedIn authorization is current and recorded outside the repository.
[ ] Nightly, scrape-only, availability-only, manual-add, recovery, and deployment paths retain the shared one-writer concurrency group.
[ ] Third-party actions remain pinned where practical.
[ ] Logs contain no secrets, GitHub contexts, HTML bodies, .env values, or database rows.
[ ] SQLite is checkpointed and validated before durable snapshot or deployment publication.
[ ] The SFTP-only account restrictions, host-key checks, manifest checksums, retention metadata, and automated restore verification remain enforced.
[ ] Migration or state-validation failures stop without deleting restored canonical state.
[ ] Manual recovery uses a reviewed, checksum-verified source.
[ ] VPS host keys are pre-verified.
[ ] Deployment remains checksum-verified and locked, with a single atomic versioned release pointer and coordinated site rollout.
[ ] SQLite state is never committed to Git.
[ ] Canonical processing remains read-only to GitHub; only the README mutation job has repository and pull-request write permissions.
[ ] No canonical database or manifest enters GitHub cache or artifacts; only the README handoff expires after one day.
[ ] canonical-state and production allow only main, and VPS secrets are unavailable to this repository outside those environments.
[ ] The README mutation job has actions: write only to dispatch the seven validation workflows on the generated commit.
[ ] Artifact visibility, environment protection, and SSH access follow least privilege.