Operator troubleshooting¶
← Maintainer handbook · CLI reference · Database lifecycle · Automation · Security policy
Diagnose pipeline, database, workflow, and container failures without losing evidence. Start with the command's exit code and first sanitized error, and preserve state before making changes. For browsing or browser-list problems, use user help.
Do not delete SQLite, weaken classification rules, increase collection limits blindly, or bypass authorization as a shortcut.
Contents¶
- First diagnostics
- Exit codes
- Configuration failures
- Database and migration failures
- README projection failures
- Collection failures
- Classification and parsing problems
- Lifecycle problems
- GitHub Actions and deployment
- Docker failures
- Requesting help
First diagnostics¶
From the repository root, record:
uv --version
uv run python --version
uv run opportunities --help
uv run opportunities searches
uv run opportunities stats
Also verify:
- the current Git branch and commit;
- the working directory;
- whether
.envor a settings YAML is being loaded; OPPORTUNITIES_DATABASE_URL;OPPORTUNITIES_PUBLIC_EXPORT_DIRwhen downloads are affected;- process-environment overrides;
- whether execution is local, Docker, or GitHub Actions;
- the exact command and exit code.
Never paste a complete environment file, production database, authenticated HTML, secret, or unredacted workflow context into an issue.
Exit codes¶
| Code | Meaning | First action |
|---|---|---|
0 |
Success | No recovery action required |
1 |
Unsafe or completely failed collection, blocking availability source-access denial, validation mismatch, or a blocked or failed quality gate | Preserve state, review authorization after a denial, and inspect the command output or quality report |
2 |
Inconclusive availability audit or rejected configuration/command input | Inspect command context; scraping must stop |
3 |
Database missing tables or not at migration head | Confirm the configured path and recovery source before upgrading; never initialize missing production history |
4 |
Eligible HTTP 429 partial collection or availability audit, not complete success |
Review the command summary and any collection-quality report, validate, and manually review the proposal |
Exit code 2 is intentionally overloaded by command context: the availability audit uses it when checks are inconclusive without a source-access denial, and configuration or selection errors use it for rejected input. Scrape partial completion uses distinct code 4 so configuration errors cannot be mistaken for publication eligibility. An availability source-access denial returns 1 and skips projection rendering unless it qualifies for HTTP 429-only partial publication with code 4; neither code permits follow-on source requests. Review authorization before another run.
Command-specific behavior is documented in the CLI reference.
Configuration failures¶
Invalid configuration or search registry¶
Common causes:
- unknown YAML field;
- YAML root is not a mapping;
- malformed lowercase kebab-case slug;
- duplicate slug or effective query identity;
- unsupported workplace or date filter;
- invalid boolean, timeout, log level, or database URL;
max_resultsgreater thanmax_pages × 25;- unexpected process-environment override.
Run:
For a specific settings file:
Correct the first validation error before investigating later messages.
Configuration precedence, variables, and limits are defined in Configuration.
LinkedIn collection is disabled¶
This is the safe default.
Local or Docker collection requires:
GitHub Actions collection and availability auditing require:
Neither value grants permission.
When the GitHub variable is missing or different, collection and availability workflows stop before LinkedIn network access. Leave the interlocks disabled unless express authorization exists.
setup-uv cannot determine a version¶
Current workflows pin:
- the setup action revision;
uv0.11.6;- Python 3.12.
Logs showing fallback to a latest-release API usually indicate an outdated workflow revision.
Update the branch from current main and rerun. Do not add insecure compatibility flags to hide the problem.
Database and migration failures¶
Database is not migrated¶
Confirm the database URL first. A missing production database requires verified restoration, not initialization. For existing production state, stop other writers and obtain a verified backup before applying pending migrations. A new disposable development database may be initialized directly.
Then run:
When the error remains, confirm both commands use the same OPPORTUNITIES_DATABASE_URL, then run:
Do not delete the database as a repair strategy.
Migration consistency or timestamp failure¶
Treat ORM/Alembic drift or last_seen_at < first_seen_at as a regression or state-corruption signal.
- Stop every writer.
- Preserve the database and sidecars.
- Run the offline test suite.
- Run migration consistency checks.
- Restore a known-good backup when necessary.
- Report the issue using synthetic or redacted evidence.
Do not rewrite old migrations or timestamps merely to make validation pass.
Schema, migration, backup, and restore procedures are canonical in Database lifecycle.
SQLite lock or concurrent-writer error¶
The supported model allows one writer.
Typical causes:
- local collection while GitHub Actions is collecting;
- two pipeline containers using the same volume;
- a VPS collector while Actions owns canonical state;
- overlapping collection and maintenance workflows;
- another process holding a long write transaction.
Stop the additional writer, preserve the current database and sidecars, and return to the one-writer model before retrying.
README projection failures¶
Marker or projection mismatch¶
The root README must contain exactly one opening and one closing marker for each generated region: opportunity counts and opportunity previews.
For an intentional projection update from reviewed representative state, run:
The generated regions contain:
- open-job metadata;
- latest successful collection time;
- the public website link;
- a hidden review seal covering all website-visible open rows and the exact latest successful collection time;
- at most five internships and five New Grad opportunities.
Do not edit generated counts, timestamps, or rows manually; fix the database state or renderer instead. During recovery, do not run render merely to make restored state appear reviewed. Follow README proposal recovery, or database recovery if the state is incorrect.
When mismatch remains, verify:
OPPORTUNITIES_README_PATH;OPPORTUNITIES_DATABASE_URL;- parent-directory write permission;
- absence of concurrent renderers or formatters;
- that the committed projection was not generated from empty state.
Public export is missing or stale¶
Regenerate only the sanitized downloads from migrated state:
Verify OPPORTUNITIES_PUBLIC_EXPORT_DIR, directory write permission for the pipeline, read permission for the website, and that deployment uploaded all three files with matching checksums. Do not generate exports in the website or expose arbitrary filesystem paths as a fallback.
The expected files are open-opportunities.csv, open-opportunities.json, and dataset-metadata.json. They contain no private lifecycle history and can be regenerated from verified canonical state. In versioned production, publish a new reviewed release through deployment-only automation; never regenerate files inside the immutable current release.
README replacement fails¶
Atomic replacement requires a writable parent directory, not only a writable README.md.
In Docker, mount the repository directory rather than README.md as an individual file.
Container permissions are documented in Docker.
Coverage metrics are stale¶
Python CI generates quality-reports/coverage.json from the current test run and then verifies that the generated table in Testing matches it.
Regenerate the report and coverage table from the repository root:
When Make is unavailable, run the complete coverage command documented in Testing, including the JSON report, then run:
Use uv run python scripts/docs/coverage_docs.py --check only after generating a current coverage report. Check mode verifies committed content without rewriting it. Never edit the coverage markers or table manually.
Collection failures¶
No accepted jobs¶
Possible causes:
- no current listing passes strict rules;
- title-prefilter rejection;
- a conflicting cycle year, or a missing cycle combined with ineligible posting-date evidence;
- unknown technology category;
- unknown or non-European location;
- employer allowlist mismatch;
- changed guest markup;
- source challenge or access block.
Compare found and accepted counts:
Identify the rejection stage before changing query limits or classification rules.
Partial collection¶
Exit code 4 preserves completed search transactions without claiming full success.
- Inspect
collection_outcome,publication_eligible,source_stopped, and failed/skipped metrics in the aggregate quality report. HTTP429partial publication requires this gate. - Require SQLite integrity, projection validation, and round-trip snapshot verification. Failed checks withhold publication and scheduling progress.
- Review the labeled
-partialREADME proposal and sanitized artifact; merge manually only when correct. No full-registry quality baseline is created. - Review source authorization before the next eligible execution. Do not immediately retry a denial. Denied and skipped searches keep priority through the verified SQLite rotation.
A failed search does not apply absence or closure evidence. Partial publication permits only HTTP 429 failures and searches skipped after the source stop. Unexpected exceptions, invalid HTML, timeouts, transport or HTTP errors, and other source denials return 1 and block publication, even when other searches completed or a concurrent request was rate-limited.
Partial availability audit¶
Exit 4 means completed checks can qualify for publication after an HTTP 429 stop; it does not mean the audit finished. Read the audit summary and confirm that validation and round-trip snapshot verification passed before reviewing its partial README proposal. Nightly scraping must be skipped. Do not restart it to bypass the rate limit, and do not expect a collection-quality report when no scrape ran.
Denied and unprocessed rows remain eligible for a later authorized run. HTTP 429 before any confirmed available or unavailable listing returns 1 and blocks publication; a public-page 200 alone is not confirmation without validated detail evidence. Exit 2 means an inconclusive audit without a source denial. Use the CLI audit result table to distinguish them, then follow README proposal recovery if a verified snapshot has no matching proposal.
All searches failed¶
When no search completes, exit code 1 can indicate a shared problem involving:
- authorization;
- network access;
- upstream availability;
- access challenge;
- parser behavior;
- configuration.
Failed searches do not apply job lifecycle changes, but the failure does not establish that existing state is valid. Preserve it while diagnosing the shared cause.
Quality gate blocked publication¶
A scrape can exit with code 1 even when some or all searches succeeded. When --quality-report is requested, blocking drift or a failure in quality analysis, report writing, or baseline persistence prevents projection rendering.
- Preserve the working database and the report, if one was written. Successful search transactions have already committed; the gate does not roll them back.
- Inspect the report's scope, baseline readiness, sample sizes, and blocking findings against the quality-report reference.
- Reproduce suspected parser or classification changes with offline fixtures. Treat source blocks as stop conditions, not a reason to increase limits.
- Resolve the cause before publication. Do not run
renderseparately merely to bypass the blocked gate.
Protected automation also stops durable snapshot publication and the README handoff, leaving production on its previously reviewed release. If report generation failed, inspect the first sanitized error rather than treating a missing report as a pass.
Timeout or HTTP 5xx¶
Retries are finite.
- Stop repeated manual runs.
- Keep pacing conservative.
- Retry one search later when the upstream failure is temporary.
- Determine whether the problem is isolated or shared.
A temporary timeout increase may be appropriate:
Do not increase concurrency to evade throttling.
Redirect, HTTP 401, 403, 429, or challenge page¶
An HTTP 301 from an approved numeric public listing preserves only that listing as inconclusive and lets unrelated checks continue. Its destination is never followed or used as evidence.
Stop collection for every other redirect, HTTP 401, 403, or 429, or an access or verification page. These block further requests through the same fetcher; requests already in flight may finish. Review access before starting another run.
Do not add:
- login or session reuse;
- cookies;
- browser automation;
- CAPTCHA services;
- proxies;
- fingerprint evasion;
- private endpoints;
- anti-bot bypasses.
Confirm authorization remains valid. Investigate legitimate parser changes only through minimal sanitized fixtures.
Classification and parsing problems¶
A relevant job is excluded¶
Verify explicit evidence for every required rule:
- internship or New Grad terminology in the title;
- no configured seniority exclusion;
- recognized technology category;
- explicit 2027 cycle, or no conflicting cycle year with posting-date evidence on or after May 1, 2026 serving as the yearless fallback;
- explicit European location.
Description or source employment metadata alone cannot convert a title without Internship or New Grad evidence into an accepted listing.
When authorized, inspect one search without persistence:
Prefer a focused regression test over weakening a global rule.
Industries shows Not specified¶
Industries comes from structured source criteria. Employment type is instead required and classified from the title as Internship or New Grad.
Confirm that:
- the deployed collector includes the current parser;
- collection ran after the field was cleared;
- the source contains structured industries evidence.
Expected structure resembles:
When industries parsing still fails:
- Reproduce with offline fixtures; live reproduction requires express authorization.
- Reduce the markup to a minimal sanitized fixture.
- Add a failing parser regression test.
- Update the parser without broad description-keyword fallbacks.
The website should continue showing Not specified for industries when structured evidence is absent. A missing or unsupported employment type excludes the listing instead of publishing an unspecified value.
Guest markup changed¶
Do not commit full pages.
Create the smallest fixture preserving the changed structure, remove tracking and personal data, use synthetic IDs, and add a regression test.
Parser contribution requirements are defined in CONTRIBUTING.md.
Lifecycle problems¶
Job did not close¶
Search-card absence is intentionally ignored.
During collection, closure requires repeated detail-page 404 or 410 confirmations for every active search association. Separately, the rotating availability audit permanently deletes a row after an explicit 404 or 410 from its public listing or guest detail request, or after a scoped “No longer accepting applications” alert.
max_rechecks selects a deterministic lowest-ID subset, not a rotating queue. Repeating the same search does not guarantee that every absent job will be checked. The separate rotating audit checks due jobs in bounded batches, preserving inconclusive results. A deferred row is not closure evidence.
Check:
closure_confirmation_runs;- active search associations;
- recent successful search runs;
- the bounded recheck limit;
- whether a valid detail page reset confirmations;
- whether the row is due, deferred by the batch cap, or reported as inconclusive.
The complete lifecycle algorithm is documented in Database lifecycle.
Job closed or reopened unexpectedly¶
- Stop writers.
- Back up SQLite.
- Inspect recent search runs and provenance.
- Check the confirmation threshold.
- Confirm whether another active association existed.
- Restore known-good state only when evidence indicates corruption.
Do not manually rewrite timestamps or status values without preserving evidence.
GitHub Actions and deployment¶
Canonical-state job cannot access its environment¶
Confirm that:
- the workflow was dispatched from
main; - the environment is named exactly
canonical-state; - its deployment-branch rule allows
mainand no other branch or tag; - unattended scheduled runs are not waiting for a required reviewer;
- all five environment secrets are present:
VPS_HOST,VPS_USER,VPS_SSH_PRIVATE_KEY,VPS_BACKUP_SSH_PRIVATE_KEY, andVPS_SSH_KNOWN_HOSTS.
Do not copy these secrets back to repository scope to bypass an environment-policy failure. Correct the environment and branch policy. The final deployment separately uses the production environment and should remain approval-gated when practical.
Docker image pull or Trivy scan fails¶
First distinguish infrastructure failure from a vulnerability finding:
- Docker exit code
125, registry authentication errors, connection resets, and image or vulnerability-database download errors indicate that the scan could not run. - A completed Trivy report followed by exit code
1indicates at least one fixable high or critical vulnerability and must be investigated.
The workflow retries the immutable digest-pinned Trivy image pull three times with bounded delays. After an exhausted transient pull or database-download failure, wait for the registry to recover and rerun the failed workflow. If failures persist, inspect Docker Hub and Trivy database registry availability without printing credentials.
Do not remove digest pins, disable Trivy, broaden vulnerability exclusions, or change the scan exit code to turn an infrastructure or security failure into success.
Restored canonical state does not match reviewed main¶
The message “Restored canonical state is ahead of the reviewed README on main” (or “does not match”) means the verified snapshot's generated projection differs from the workflow's checked-out README. Snapshot publication may have succeeded in an earlier run even if its README proposal failed or was closed without merging. This is a review barrier, not permission to discard SQLite or bypass validation.
Merge the matching proposal after its checks and review pass. If it is missing or closed, run Verify canonical state recovery from current main with recover_readme_proposal=true, leaving seal adoption disabled. This explicit mode validates the restored projections and opens a manual-review README-only PR without source access, snapshot publication, or deployment. Follow the recovery procedure, reconcile older proposals, and merge only correct state.
After merging, start a fresh workflow from updated main or wait for the next schedule. Rerunning the old failed run reuses its original commit and may still fail against the old README. Do not use the initial seal-adoption option for an existing seal or a changed dataset.
Canonical snapshot is missing¶
Canonical SQLite is never stored in GitHub Actions cache or artifacts. When restricted VPS snapshot storage is healthy, the workflow downloads latest.json, verifies its timestamped SQLite file, and uses that exact state.
During initial rollout, bootstrap requires successful storage listings that prove no latest.json or snapshot history exists, plus no versioned current pointer or releases directory on the VPS. Only then may the workflow stream a consistent backup from the reviewed legacy live database, followed by independent integrity, foreign-key, required-table, and Alembic-revision checks. An unreferenced local database is never a bootstrap source.
An existing snapshots directory without latest.json stops recovery even when empty. Failed listings, transfers, or empty manifest downloads cannot establish absence. Preserve the evidence and reconcile the pointer instead of starting unrelated history.
The README and sanitized projection artifacts cannot reconstruct lifecycle state.
VPS snapshot restore or publication fails¶
Do not bypass host-key, manifest, checksum, or restore checks to finish a collection run.
Check, without printing credentials:
- the main-only
canonical-stateenvironment and its secrets; VPS_HOST,VPS_BACKUP_USER, andVPS_SSH_PORT;- the
VPS_BACKUP_SSH_PRIVATE_KEYsecret and verifiedVPS_SSH_KNOWN_HOSTSentry; - SFTP-only account access to
/state; - absence of shell, sudo, forwarding,
opportunities-sitemembership, and live-database access; - database size, SHA-256, schema revision, and collection timestamp against the manifest;
- SQLite
integrity_check,foreign_key_check, and required tables.
If restoration reports that local state differs from the latest manifest or that restore/bootstrap staging already exists, preserve those files and any sidecars for investigation; do not remove them just to make a retry pass. A fresh protected job normally starts with no local database.
A publication failure before latest-pointer promotion leaves the prior snapshot authoritative. Preserve failed-run logs and inspect newly uploaded timestamped files; do not repoint latest.json manually until the pair passes:
uv run python scripts/database/canonical_snapshot.py verify \
--database /safe/recovery/opportunities.db \
--manifest /safe/recovery/manifest.json
For recovery, walk the manifest previous_snapshot references newest to oldest. A manifest or checksum failure is a stop condition, not permission to use unverified bytes. Restricted account setup and required settings are documented in Automation.
SSH or VPS deployment fails¶
Verify:
- that the
productionenvironment approved the deployment and allows onlymain; VPS_HOST;VPS_USER;VPS_SSH_PRIVATE_KEY;VPS_BACKUP_SSH_PRIVATE_KEYfor the restore/publication phase;VPS_SSH_KNOWN_HOSTS;- optional
VPS_SSH_PORT; - access to
/srv/european-tech-opportunities-2027/datafor the restricted SSH user; - whether another workflow holds the deployment lock;
- whether the local working database has
-wal,-shm, or-journalsidecars; checkpoint and close it before retrying, without discarding uncheckpointed data; - whether
data/currentis invalid, the run ID has already been published, or a partial upload failed checksum verification. Inspect the pointer and immutable releases under the deployment lock before retrying; never remove an active release.
Do not disable host-key verification.
Deployed database is unchanged¶
First distinguish the pipeline's working file from the versioned release served by the website. Check:
- that deployment-only automation completed and its local and remote payload checksums agree;
- that
data/currentis a valid relative symlink to the intendeddata/releases/<run-id>-<attempt>and(cd data/current && sha256sum -c checksums.sha256)passes; - that the site has
OPPORTUNITIES_RELEASE_ROOT=/app/dataon every instance after the coordinated rollout; without it, the site continues to read the stale legacy fixed paths; - that the site group can traverse
data/releases/and read the selected database, CSV/JSON downloads, and metadata manifest; - that old site instances and cached responses are not being mistaken for the new release. A page and a later download can span a cutover.
A missing or invalid current is a stop condition, not a reason to repoint the site to legacy state. Deployment sequencing and rollback are documented in Automation.
Migration or canonical-state validation fails¶
Collection workflows deliberately provide no state-rebuild input. They stop rather than deleting an incompatible restored database or its sidecars.
- Preserve the failed state and stop additional writers.
- Review verified durable manifests and snapshots first, then retained versioned releases under
data/releases/; a legacyopportunities.db.previousis not current state after rollout. Sanitized projection artifacts cannot restore state. - Verify the selected snapshot’s checksum, schema revision, integrity, and foreign keys.
- Restore it with the procedure in Database lifecycle.
Do not initialize an empty database merely to make automation pass. An intentional manual rebuild loses first-seen history, provenance, closure evidence, and diagnostics. Recovery policy is documented in Automation.
Nightly pull request does not auto-merge¶
A partial collection deliberately uses automated/nightly-full-update-partial, has a partial/manual-review title, and never requests auto-merge. Review and merge manually after checks pass. If rejected or closed, the reviewed-state barrier remains blocking; use proposal recovery only for correct state, or obtain explicit approval for verified canonical recovery. Do not collect again or reset only rotation metadata.
For a fully successful collection, confirm that:
- GitHub Actions may create pull requests and squash merging is enabled;
- the README mutation job has
actions: write,contents: write, andpull-requests: write; - branch protection requires
ruff,python,site,docker,Analyze (Python),Analyze (TypeScript),Gitleaks secret scan,build(Documentation site), anddependency-review; - the pull request targets
main; - its head branch is
automated/nightly-full-update; - its title is
data: nightly availability and scrape update; - changes are restricted to
README.mdand the five supporteddocs/translations/README.<language>.mdfiles; workflow_dispatchruns exist for all seven validation workflows, including Dependency Review, on the automation branch head SHA;- the README mutation job identified those run IDs and waited for each successful conclusion before requesting a merge.
The repository's optional auto-merge setting is not required: when disabled, the workflow requests a normal squash merge after validation. Both paths obey branch protection. A refused normal merge leaves the proposal open and fails the job; resolve its checks or required reviews and merge that proposal rather than collecting again. An older workflow revision can still fail with Auto merge is not allowed for this repository; rerunning that run retains its original revision. See README update pull requests and missing proposal recovery.
A branch push made by GITHUB_TOKEN does not trigger ordinary push workflows, and PR workflows created by that token can wait for Approve workflows to run. The README mutation workflow therefore dispatches all seven validation workflows explicitly after its exact-scope check and keeps the branch alive until they finish. Dependency Review receives the PR number, verifies the open README-only proposal and run SHA, and compares the actual base/head commits. If a run is missing, inspect workflow-dispatch permissions, the selected branch's workflow revision, and its inputs; do not bypass required checks.
A validation run that fails instantly with zero jobs and no logs usually means the automation branch was merged and deleted before GitHub finished creating jobs. The explicit dispatch wait prevents that race without weakening validation. The workflow also refuses validation dispatch and merging when any scope check differs. Do not weaken either safeguard; restore the fixed automation branch to the expected README-only diff instead.
Docker failures¶
Start with:
Expected pipeline database URL:
Database appears empty¶
Confirm both services mount /srv/european-tech-opportunities-2027/data. In versioned production mode the pipeline's working file and the site's current release are intentionally different; compare the site's selected release with the latest reviewed deployment, not with the working file.
For an intentionally new local development directory only, initialize and inspect it explicitly:
A fresh local directory intentionally contains no listings. In production, restore and verify canonical state before migration; website startup never creates or migrates it automatically.
Website cannot read SQLite¶
Check:
- the host state directory is mounted;
- the website bind mount is read-only;
- the database, CSV/JSON downloads, and
dataset-metadata.jsonexist; - UID/GID
10001:10001has read access through the configured host ownership or group mapping; - in versioned mode,
OPPORTUNITIES_RELEASE_ROOT=/app/dataand a validcurrentsymlink target under/app/data/releases/are present; in legacy/local mode only, the configured database path is/app/data/opportunities.db; - in versioned mode, the selected release was produced as a cold, sidecar-free snapshot rather than copied inconsistently from a live WAL database; in legacy mode, preserve any required sidecars and permissions.
README rendering fails in Docker¶
README rendering requires:
- the repository mounted at
/workspace; - writable parent-directory permission;
- write permission on
README.md; - no competing renderer.
Compose configuration is wrong¶
Run:
Verify:
- environment expansion;
- bind-mount source paths;
- the read-only
/app/configsmount; - the
/srv/european-tech-opportunities-2027/datadatabase bind mount; - image targets and service names;
SITE_URLand, after rollout,OPPORTUNITIES_RELEASE_ROOT=/app/datafor every site instance;- absence of an unintended fixed production port.
Dokploy routing fails¶
Verify:
- the domain targets the
siteservice; - the internal port is
3000; - the site container is healthy;
- no conflicting host port is published;
SITE_URLuses the public HTTPS origin.
Container setup and permissions are documented in Docker and deployment.
Requesting help¶
Before opening an issue, reduce the problem to the smallest safe reproduction you can. A useful report includes:
- exact command;
- exit code;
- expected behavior;
- actual behavior;
- operating system;
- relevant Python,
uv, Bun, or Docker version; - affected search slug;
- first sanitized error;
- minimal offline reproduction.
Do not include:
.envcontents;- credentials or tokens;
- LinkedIn cookies;
- authenticated HTML;
- private paths;
- production databases;
- unredacted GitHub Actions contexts.
Report security-sensitive findings privately through SECURITY.md.