Task · PND-0003

Migrate the repo task surface to just and retire Makefiles and ad-hoc scripts

Description

Migrate paperless-ngx-dedupe task surface to just

Follow the fleet just standard (mandatory vocabulary, groups, header, authoring rules, migrate-vs-keep rule, CI integration rules) exactly as frozen. Do not re-litigate it. This repo has no Makefile — the task surface today is package.json scripts (root + two pnpm workspace packages) plus a small set of shell scripts, none of which are dev/CI task orchestration.

Outcome

A top-level justfile becomes the one true task surface: just --list shows setup, fmt, fmt-check, lint, test, check, typecheck, build, dev, test-e2e, docker-validate, docker-dev, audit. just check is exactly what CI’s quality + unit-tests jobs enforce. Root package.json scripts block stays (pnpm workspace filtering still needs it internally, and just recipes call pnpm under the hood) but is no longer the documented entry point — AGENTS.md, README.md and backlog/config.yml all point at just instead. docker-entrypoint.sh, add-correspondent.sh, add-tags.sh, and scripts/cloud-environment-setup.sh are untouched — none of them are dev/CI task orchestration, all four are explicitly out of scope (see §10 below). ci.yml’s quality and unit-tests jobs call just recipes instead of raw pnpm; every other workflow file is untouched.

The complete justfile

Drop this at the repo root as justfile. Adjust only if a command in Step 2 verification does not match what’s actually in this repo at implementation time (toolchain versions, script names).

set shell := ["bash", "-euo", "pipefail", "-c"]

# show the task surface
default:
    @just --list

# install pnpm workspace dependencies (idempotent, frozen lockfile)
[group('dev')]
setup:
    pnpm install --frozen-lockfile

# format source in place (prettier)
[group('check')]
fmt:
    pnpm format:fix
    @just --fmt

# verify formatting without mutating (prettier + just --fmt)
[group('check')]
[no-exit-message]
fmt-check:
    pnpm format
    just --fmt --check

# lint with eslint
[group('check')]
[no-exit-message]
lint:
    pnpm lint

# type-check both packages (core then web, dependency order)
[group('check')]
[no-exit-message]
typecheck:
    pnpm --filter @paperless-dedupe/core check
    pnpm --filter @paperless-dedupe/web check

# run core unit tests (vitest); pass filter="pattern" to narrow
[group('check')]
[no-exit-message]
test filter="":
    pnpm --filter @paperless-dedupe/core exec vitest run {{ filter }}

# run web e2e tests (playwright, needs built packages)
[group('check')]
[no-exit-message]
test-e2e:
    pnpm --filter @paperless-dedupe/web test:e2e

# dependency vulnerability scan (matches CI's audit step)
[group('check')]
audit:
    pnpm audit --audit-level=high

# the full local gate — exactly what CI's quality + unit-tests jobs enforce
[group('check')]
check: fmt-check lint typecheck test

# build both packages in dependency order: core then web
[group('build')]
build:
    pnpm --filter @paperless-dedupe/core build
    pnpm --filter @paperless-dedupe/web build

# start the SvelteKit dev server (http://localhost:5173) — background jobs need docker-dev instead
[group('dev')]
run:
    pnpm dev

# build the local image and run it via compose with a health check
[group('build')]
docker-validate:
    docker build -t paperless-ngx-dedupe:local .
    docker compose -f compose.dev.yml up --force-recreate --abort-on-container-exit

# docker compose dev profile with live rebuild (full workflow incl. background jobs)
[group('dev')]
docker-dev:
    docker compose -f compose.dev.yml --profile dev up dev --build

Notes on choices baked into this file (do not change without re-checking against the standard):

Makefile disposition

None. find . -iname Makefile -o -iname GNUmakefile (excluding node_modules) returns nothing in this repo. Nothing to delete, no table needed.

Script disposition

Script Verdict Recipe / reason
docker-entrypoint.sh KEEP Shipped runtime artifact — Dockerfile’s ENTRYPOINT, runs inside the built container on a target machine with no just. Never called from a dev/CI task.
add-correspondent.sh KEEP Standalone operator utility that bulk-creates correspondents against a live external Paperless-NGX instance via its own hardcoded URL/token — not a repo dev/CI task, has no repo-relative meaning, and is invoked ad hoc by a human, not by setup/build/test/CI.
add-tags.sh KEEP Same as add-correspondent.sh — standalone operator utility against a live external instance, non-trivial control flow (comma-split parsing loop), invoked ad hoc.
scripts/cloud-environment-setup.sh KEEP Real program: functions (log, as_root, install_node), a trap, an architecture-detection case, an HTML-scraping curl+sha256sum verification step, and a retry loop. Its own header says CLOUD AGENTS ONLY: local agents must not execute this. It is invoked by the cloud-agent bootstrap mechanism, not by a developer or CI (§6 “scripts invoked by something other than a developer or CI”) — do not fold it into setup and do not have setup call it.

No scripts are absorbed. All four are real programs, shipped artifacts, or externally-invoked bootstrap — none is a thin sequencing wrapper.

CI changes

Only .github/workflows/ci.yml changes. Every other workflow file (arm-automerge.yml, auto-rc.yml, docker-security.yml, ghcr-cleanup.yml, publish.yml, release-please.yml, scorecard.yml, trigger-docs-sync.yml) is GitHub-native or a reusable-workflow uses: call and must not be touched (see §10).

.github/actions/setup-node-pnpm/action.yml

This composite action is used by every ci.yml job via uses: ./.github/actions/setup-node-pnpm. Add the setup-just step here (once, shared by every consumer) and switch its install step to call the new recipe:

name: Setup Node.js with pnpm
description: Set up pnpm + Node.js and install dependencies (frozen lockfile)

runs:
  using: composite
  steps:
    - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10

    - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
      with:
        node-version: 24
        cache: pnpm

    - uses: extractions/setup-just@<pin-exact-sha> # v4
      with:
        just-version: '1.58.0'

    # Install directly rather than restoring a cross-job node_modules cache: pnpm rewrites
    # pnpm-lock.yaml during install, so the install job's post-install hashFiles() key never
    # matched the consumers' fresh-checkout key — every consumer missed the cache and ran with
    # an empty node_modules (eslint/vitest/playwright "not found"). The pnpm store cache above
    # keeps this fast.
    - name: Install dependencies
      shell: bash
      run: just setup

Resolve <pin-exact-sha> for extractions/setup-just at implementation time (gh api repos/extractions/setup-just/git/refs/tags/v4 or similar) and pin it with a # v4 trailing comment matching this repo’s existing SHA-pin convention (see every other uses: line in ci.yml).

.github/workflows/ci.ymlquality job

Before:

      - uses: ./.github/actions/setup-node-pnpm
      - run: pnpm lint
      - run: pnpm format
      - run: pnpm check
      - name: Audit dependencies
        run: pnpm audit --audit-level=high
        continue-on-error: true

After:

      - uses: ./.github/actions/setup-node-pnpm
      - run: just lint
      - run: just fmt-check
      - run: just typecheck
      - name: Audit dependencies
        run: just audit
        continue-on-error: true

Do not collapse these four into a single run: just check — keep them as separate steps so CI’s per-step annotations/timing stay granular and continue-on-error stays scoped to audit alone. (just check itself must still be proven to equal this sequence locally per Order of Work below — the point is CI keeps step granularity, not that it stops matching check.)

.github/workflows/ci.ymlunit-tests job

Before:

      - name: Run unit tests with coverage
        run: |
          pnpm --filter @paperless-dedupe/core exec vitest run --coverage --reporter=default --reporter=junit --outputFile.junit=./test-results/junit.xml

Leave this step’s run: unchanged. It passes --coverage --reporter=default --reporter=junit --outputFile.junit=... flags that just test does not carry (and should not — those flags are CI-reporting-specific, not part of the developer-facing gate). Do not force this into just test; a recipe’s job is the developer/local gate, not to reproduce every CI reporting flag. This is a deliberate exception — call it out explicitly if a reviewer asks why this one run: didn’t collapse.

.github/workflows/ci.ymle2e-tests job

The “Build packages” step (pnpm --filter @paperless-dedupe/core build && pnpm --filter @paperless-dedupe/web build) becomes:

      - name: Build packages
        run: just build

The “Run E2E tests” step (pnpm test:e2e) becomes:

      - name: Run E2E tests
        run: just test-e2e

Leave the Playwright browser install/cache steps in this job untouched — they are CI-environment provisioning (browser binaries + OS deps), not a repo task a developer runs locally, and are already env-gated (if: steps.playwright-cache...) in ways that don’t map to a single recipe.

.github/workflows/ci.ymldocker-build-verify job

Leave untouched. Its Docker Buildx setup (with OTEL env wiring for build tracing), health-check polling loop, and cleanup step are CI-specific orchestration around docker/build-push-action, not a pnpm/script command this migration is chartered to touch. docker-validate (the new recipe) is the local equivalent a developer runs; it is not what CI’s job does line-for-line and should not be forced to match.

.github/workflows/ci.ymlsource-maps job

The “Build with source maps” step (run: pnpm build, with FARO_SOURCEMAP_API_KEY env) becomes:

      - name: Build with source maps
        run: just build
        env:
          FARO_SOURCEMAP_API_KEY: ${{ secrets.FARO_SOURCEMAP_API_KEY }}

Env passthrough is unaffected — just recipes inherit the step environment (§8).

What must NOT change in ci.yml

Docs and agent-contract changes

AGENTS.md

Replace the ## Commands section (currently lines ~35–47, the fenced pnpm ... block) and the ## Workflow line (Run \pnpm lint && pnpm format –write && pnpm check && pnpm test` after completing work or before pushing…`) with the fleet-standard Task interface block:

## Task interface

This repo's task surface is a `justfile`. Discover it, don't guess it:

    just --list                        # human-readable
    just --dump --dump-format json     # machine-readable
    just --show <recipe>               # what a recipe actually runs

- `just check` is the full gate and is exactly what CI enforces. It must pass before you commit.
- Prefer `just <recipe>` over the underlying tool. If you are typing `pnpm test`, you want `just test`.
- Run `just` with stdin from /dev/null. No recipe in this repo is marked `[confirm]` today; if one is
  added later, stop and ask before running it — never pass `--yes` or `JUST_YES=1`.
- If a task you need does not exist, add a recipe with a `#` doc comment and a `[group(...)]` rather
  than running a bare command.

Keep the rest of AGENTS.md (## Architecture, ## Quality Checks, ## Code Style, ## Svelte 5 Conventions, ## Debugging Guidelines, ## Gotchas & Constraints) unchanged — they’re not task-surface content. Do not paste the recipe list into this file (§9).

README.md

Lines ~68–76 currently read:

pnpm install          # Install dependencies
pnpm dev              # Start dev server (http://localhost:5173)
pnpm build            # Production build
pnpm check            # TypeScript type checking
pnpm lint             # Lint
pnpm test             # Run tests

Replace with:

just setup            # Install dependencies
just run              # Start dev server (http://localhost:5173)
just build            # Production build
just typecheck        # TypeScript type checking
just lint              # Lint
just test             # Run tests

Keep the note immediately after about pnpm docker:dev / background jobs, but update the command name to just docker-dev.

backlog/config.yml

See next section — this also counts as a docs/contract change since it’s what backlog task surfaces as the definition of done.

backlog/config.yml

Current:

definition_of_done:
  - "pnpm lint && pnpm format --write && pnpm check && pnpm test"
  - "pnpm build (core then web, in dependency order)"
  - "pnpm test:e2e (only if packages/web behaviour changed)"

New:

definition_of_done:
  - "just check"
  - "just build"
  - "just test-e2e (only if packages/web behaviour changed)"

Edit this file through the backlog CLI’s config path if it exposes one; if backlog has no CLI verb for editing config.yml fields, this is the one exception where direct edit is correct — config.yml is Backlog.md’s own settings file, not a task/doc record, and the operating rule “never hand-edit a tracker’s markdown; drive it through its CLI” is about task/doc content, not this settings file. State in the PR/commit which route was used.

Order of work

  1. Write the justfile at repo root exactly as specified above.
  2. Run just --fmt --check — must pass with zero diff (the file above is pre-formatted; if it doesn’t, run just --fmt once and re-diff to confirm idempotence).
  3. Run just setup on a clean-ish checkout (or accept the already-installed node_modulespnpm install --frozen-lockfile is idempotent either way) and confirm it succeeds.
  4. Run just fmt-check, just lint, just typecheck, just test, just audit, just build each standalone and confirm each matches what the corresponding pnpm command currently does (same pass/fail, same output shape).
  5. Run just check and confirm it runs fmt-check lint typecheck test in that order and the aggregate result matches running the CI quality + unit-tests steps by hand.
  6. Run just test-e2e after just build and confirm it matches pnpm test:e2e.
  7. Run just docker-validate and just docker-dev (or at minimum dry-review them against the existing pnpm docker:* scripts — they are unchanged commands, just relocated) to confirm the compose files and flags are identical to today’s package.json scripts.
  8. Only once step 2–7 are all green locally: edit .github/actions/setup-node-pnpm/action.yml to add the setup-just step and switch install to just setup.
  9. Edit .github/workflows/ci.yml’s quality, unit-tests (leave run: unchanged, see above), e2e-tests, and source-maps jobs per “CI changes” above.
  10. Push a branch, let CI run, confirm ci-success still passes and job step output for quality/ e2e-tests/source-maps shows the new just <recipe> invocations succeeding identically to the old pnpm calls.
  11. Edit AGENTS.md, README.md per “Docs and agent-contract changes” above.
  12. Edit backlog/config.yml’s definition_of_done per above.
  13. There is nothing to delete — no Makefile exists, and every script is a KEEP. Do not run any git rm as part of this task; if a later audit finds a genuinely orphaned script, that’s a separate task, not this one.

Do not reorder: justfile-and-local-proof always precedes CI wiring, which always precedes docs, so the repo is never red between commits.

Traps specific to this repo

Out of scope

Acceptance Criteria

Definition of Done

Implementation Plan

  1. Verify the current pnpm task surface, retained scripts, workflow constraints, and Just version.
  2. Add the prescribed top-level justfile and prove its formatting, recipe surface, setup, quality gate, build, e2e, audit, and Docker mappings.
  3. Wire the pinned setup-just action and targeted ci.yml calls only after local proof; update task-interface documentation and backlog definition of done.
  4. Review the exact diff and retained-script references, run proportionate workflow/config validation, commit and push main, confirm ci-success at the final SHA, then finalize the task with evidence.

Implementation Notes

Local proof: just --fmt --check and just --dump --dump-format json passed; the clean isolated just check passed with 76 test files and 1,187 tests (9 intentionally skipped); just build passed.

A local just test-e2e attempt did not start because its fixed preview port was occupied by an unrelated local process outside this repository. The process was not touched; the final CI e2e job remains the required proof.

just audit completed with one low and one moderate advisory finding; no high-severity finding caused a failure. The retained Docker recipes were dry-reviewed against the existing package scripts and match exactly.

The justfile follows the current fleet rule that default and setup are ungrouped; this supersedes the older task example that grouped setup.

Final verification after integrating upstream dependency updates:

Comments

author: campaign-ordering created: 2026-08-29 09:18

Fleet ordering — WAVE 2. Starts after the Wave 0 pilot (sf2loki / SFL-0073) and the Wave 1 hubs land.

Within Wave 2 the order is free — these repos do not depend on each other. Batching by language is worthwhile so one lane reuses its Makefile-to-recipe mapping across similar repos.

Do not start before the pilot reports. The standard may be amended off the back of it, and picking this up early risks coding against a superseded seam.

Provisioning just in CI. Which mechanism depends on the runner, and the two must not be mixed:

Runner Mechanism
arc-arm64 (m7kni self-hosted) just is baked into the runner image by m7kni/ci-tools (runner-image/Dockerfile, ARG JUST_VERSION). Do not add extractions/setup-just, and delete the step if this repo already has one — it installs a second just earlier on PATH and turns the image pin into a lie.
GitHub-hosted (all rknightion repos) extractions/setup-just, SHA-pinned, with an explicit just-version:.

Both sides currently sit on 1.58.0 and are Renovate-managed. ci-toolsTool version drift workflow fails if the Dockerfile ARG and the published image ever disagree, and lists any repo still carrying a second pin.

While you are in the workflow files, check the hub pin. On 2026-08-29 Renovate was unfrozen for rknightion/.github in m7kni/renovate-config — it had been enabled: false on the mistaken belief that callers tracked @main, which froze the fleet across 19 different hub SHAs (v1.3.1 June → v1.9.7 August) so that no hub fix ever propagated. Bumps now arrive as one grouped, CI-gated, automerged PR per repo. A uses: whose comment is not a real # vX.Y.Z still cannot be bumped (it resolves to a digest-only update, which the fleet rules disable) — if you find one, repair the comment as part of this task.

author: campaign-ordering created: 2026-08-29 10:43

Standard amendment — ci is the sanctioned superset of check (RATIFIED)

This supersedes the frozen wording check is the complete local gate and reproduces every CI job that can run off a GitHub runner”, which several lanes could not honour without making the pre-commit gate depend on a Docker daemon.

The definitions now are:

Every leg you put in ci must carry a comment naming which of those three it needs. That comment is the guard: without it ci becomes the bin for anything slow or awkward, check quietly stops meaning much, and the fleet is back to a per-repo gate.

Eleven of the 42 lanes arrived at this shape independently before it was ratified, which is why it won.

If this repo has no such legs, it has no ci recipe at all and check is the whole gate. Do not add an empty one.

Final Summary

Migrated the repository task surface to a top-level justfile, wired the pinned CI setup and targeted CI recipe calls, and updated the documented task interface and Backlog definition of done. Local isolated checks and exact-head CI run 33258282391 passed, including ci-success, E2E, source maps, and Docker health verification.

View the source file on GitHub