maand deploy

Deploy pushes job artifacts from the database to worker nodes, runs lifecycle actions (start / restart / stop), executes hook commands (pre_deploy, post_deploy, job_control), and updates allocation hashes so later deploys can skip unchanged jobs or resume after partial failure.

Requires a prior maand build (or maand deploy --build).

CLI

maand deploy [flags]

Examples:

maand deploy
maand deploy -b
maand deploy --jobs api,worker
maand deploy --dry-run
maand deploy -b -n
maand deploy --force --jobs vault
maand deploy --sync-only --jobs prometheus
maand deploy --dry-run --sync-only

Prerequisites

1. Initialized bucket with maand build completed successfully.
2. Host tools: bash, ssh, rsync, and python3 on the CLI host (bun when any hook uses .ts/.js). Deploy checks these before syncing.
3. maand.conf: ssh_user, ssh_key (under secrets/), optional use_sudo.
4. Workers reachable by SSH from the maand CLI host; ensure secrets/worker.key is authorized on workers.
5. Each worker has python3, make, rsync, bash, and timeout on PATH. When use_sudo = true, sudo must work and sudo rsync --version must succeed. Deploy SSH-checks workers before syncing.
6. Each active job has a Makefile unless the job uses only job_control commands.

High-level pipeline

Open DB transaction + kv.Initialize
Start hook HTTP API on host
Reconcile once: stop removed/disabled allocations → after_allocation_stopped → cleanup
  (post-reconcile health_check when a job stopped running allocations but still has survivors)
For deployment_seq = 0 .. max:
  For each job in this sequence (respect --jobs filter):
    pre_deploy (if registered) + stage + plan hash refresh (skip detection)
  When at least one job in the wave needs rollout or post_deploy retry:
    UpdateSeq (+1) — once per deploy invocation, before first rsync
    Prepare worker.json / jobs.json / bin/ for all workers
  For each job to deploy:
    Stage job files + transpile .tpl + certs → tmp/workers/<ip>/jobs/<job>/
    Rsync (filtered per job) to /opt/worker/<bucket_id>/
    Update allocation content hashes (MD5 of staged tree)
    deployJob: batched lifecycle + per-batch promote + health; then post_deploy + promote sweep
    KV checkpoint
Final rsync: per successfully deployed job only (filtered + jobs.json refresh)
Commit transaction (even if some jobs failed — partial deploy)
Return joined errors if any job failed

Deployment sequence

Jobs with the same deployment_seq (from build) are processed in the same wave. Lower sequences complete before higher ones. This respects demands between hooks (e.g. job B depends on job A).

Reference: deployment-sequence.md — demand graph, deployment_seq algorithm, max_concurrent_upgrades within a wave, examples.

Within one sequence, jobs are independent except they share worker staging directories under tmp/workers/<ip>/.

Which jobs run in a deploy wave

A job is considered only if it appears in the sequence and passes the --jobs filter.

JobNeedsRollout and JobNeedsPostDeploy decide whether a job is staged and deployed:

Per-batch promote and deploy completion

During rollout, maand promotes each allocation batch after after_allocation_started hooks succeed and before the health gate for that phase. Hook failure blocks promote for that batch; health failure does not unpromote batches that already promoted.

A job is finished for the current generation only when post_deploy succeeds (or the job has no post_deploy commands). Catalog field post_deploy_status on each allocation hash tracks this:

After a successful deploy wave, promoteAllocationHash (or per-batch promote) sets previous_hash = current_hash and hash.current_version = allocations.new_version. A re-run of maand deploy continues from failed jobs only (partial deploy resume). Jobs with all allocations promoted but post_deploy still pending are staged for step 4 only. Use --force to roll the same content again without a workspace change.

Allocation version tracking

Each active allocation tracks running vs target version alongside content hashes in the hash table (namespace <job>_allocation, key alloc_id).

Defaults: If manifest.json omits version, maand uses 0.0.0 for KV and allocation version fields (build-time dependency rules still require an explicit version when the job participates in the demand graph).

Lifecycle (mirrors content hash promote):

First deploy:     current_version=0.0.0   new_version=2.0.0
                  → make start with CURRENT_VERSION=0.0.0 NEW_VERSION=2.0.0
                  → promote → current_version=2.0.0

Upgrade:          current_version=2.0.0   new_version=2.1.0
                  → restart/reload (per restart_policy) → promote → current_version=2.1.0

Same version, unchanged tree: hash unchanged and `current_version = new_version` → job skipped (no lifecycle)

Version-only bump: **`build`** updates `allocations.new_version`; **`deploy`** runs lifecycle when `hash.current_version != allocations.new_version` even if the content hash is unchanged (typically **`reload`** when policy is **`reload`**)

During a rolling deploy, allocations on different workers can briefly differ (current_version updated per allocation as each wave promotes).

Where to read versions

Per-allocation KV namespace maand/job/<job>/worker/<ip> does not store a version key — deploy clears any legacy copy on promote. Use job-level KV or template context .NewVersion for the build target.

Example Makefile upgrade hook:

restart:
	@echo "Upgrading $(CURRENT_VERSION) -> $(NEW_VERSION)"
	./bin/upgrade.sh

Build-time version rules and demand min_version / max_version are separate — see manifest.md.

Use maand deploy --dry-run to see whether a real deploy would run, without rsync, lifecycle, or hash promotion:

1. Stages job files under tmp/workers/ (same as deploy), including pre_deploy hooks when registered (so plan hashes match real deploy staging). pre_deploy may SSH to workers when the hook runs hooks on allocations.
2. Computes MD5 of each active allocation’s staged tree and compares to previous_hash in the database (rolled back afterward).
3. Prints per job whether deploy is required and per allocation the planned action: start, restart, reload, sync, or skip. When restart_globs forces a restart, the line includes matched= with the changed paths.

maand deploy --dry-run
maand deploy -b -n --jobs api
maand deploy --dry-run --force    # preview forced redeploy

Example output when content changed since last promote:

deploy dry-run: deployment required

deployment sequence 0:
  job "api": deploy required
    10.0.0.1  reload  previous_hash=abc... current_hash=def...
    10.0.0.2  restart previous_hash=abc... current_hash=def... matched=Makefile

When everything is promoted:

deploy dry-run: no deployment required

deployment sequence 0:
  job "api": skip (already promoted on all allocations)

Default deploy path (deployJob steps 1–4)

For each job in the wave that needs rollout or post_deploy-only retry:

1. pre_deploy (optional, before staging)
2. Stage job files to tmp/workers/<ip>/jobs/<job>/
3. Rsync that job to each worker allocation
4. Update allocation hashes for that job
5. deployJob — canonical steps below
6. KV checkpoint after deployJob

When the job has no job_control commands, deployJob runs:

When job_control commands exist, steps 1–3 are replaced by batched job_control scripts; per batch: commands → promote → health (wait). Step 4 is unchanged.

Reconcile (before any job staging): stop removed/disabled allocations → after_allocation_stopped → cleanup. Stop hook failure aborts the entire deploy (no cleanup, no job waves). Post-reconcile health_check runs for jobs that stopped a running allocation and still have active survivors.

--sync-only: skips steps 1–3; runs step 4 only (rsync + post_deploy + promote).

Makefile lifecycle detail

1. handleNewAllocations: Workers where previous_hash IS NULL (new alloc) →
   python3 /opt/worker/<bucket_id>/bin/runner.py <bucket_id> start --jobs <job>
   in batches of max_concurrent_starts (0 = all at once), ordered by rollout_order.
   after_allocation_started hooks run after each batch, then promote that batch. One health check runs after all start batches complete.
2. Pre-rolling health: Before the first upgrade batch when any allocation needs restart/reload.
3. handleUpdatedAllocations: Workers where hash or version changed → lifecycle per restart_policy (see below) in batches of max_concurrent_upgrades, ordered by rollout_order.
   after_allocation_started hooks run after each batch, then promote, then health_check (wait/retry) before the next batch.
4. post_deploy: Hooks with event post_deploy (job-wide wave, batched per allocation).
5. promoteAllocationHash: Idempotent sweep for any allocation not yet promoted (e.g. disabled, sync-only).

When allocations are stopped during reconcile (removed/disabled), after_allocation_stopped hooks run once per stopped batch; cleanup runs only after hooks pass.

Makefile on the worker (under jobs/<job>/) receives CURRENT_VERSION and NEW_VERSION in the environment for start, restart, and reload. Use them for upgrade logic (see Allocation version tracking):

start:
stop:
restart:
reload:

Data/logs/bin on workers are excluded from rsync (--exclude=jobs/*/data, etc.).

Applying changes on workers

Deploy always rsyncs staged files before it decides whether to touch running processes. That split matters: you can push config to disk while choosing how (or whether) the process reacts.

What triggers rollout

New allocations always start. No policy or flag can replace start with rsync-only — use normal deploy for first boot, then tune policy for upgrades.

Default path: Makefile + runner.py

When the job has no job_control commands, deploy calls runner.py on the worker, which runs make targets in the job directory:

Each target receives CURRENT_VERSION and NEW_VERSION in the environment (see Allocation version tracking).

Rolling batches use max_concurrent_starts (starts) and max_concurrent_upgrades (restarts/reloads), ordered by rollout_order. Health checks run after start batches complete and after each update batch when a lifecycle target runs.

restart_policy (manifest)

Set in manifest.json. Default always. Applies to updated allocations only.

Example — Prometheus picks up rule and config changes without restarting the process when only non-critical files change:

{ "restart_policy": "reload" }

reload:
	curl -sf -X POST http://127.0.0.1:$(PROM_PORT)/-/reload

Prometheus needs --web.enable-lifecycle. See guides/prometheus.md.

Stateful jobs (databases, queues) usually keep always. Stateless HTTP services and monitoring stacks often use reload.

restart_globs (manifest, with reload only)

Optional list of job-relative globs (*, ?, ** — same rules as .dashboardignore). maand build rejects restart_globs unless restart_policy is reload.

Maand stores a per-file manifest on each allocation (hash.current_files / hash.previous_files): path → content MD5 of the last staged and promoted trees. On upgrade it diffs those maps:

• If any changed path matches a glob → make restart
• Otherwise → make reload

Example — reload for most edits, restart when compose or binaries change:

{
  "restart_policy": "reload",
  "restart_globs": [
    "docker-compose.yml",
    "docker-compose.yml.tpl",
    "Dockerfile",
    "bin/**"
  ]
}

Dry-run shows which paths triggered restart:

10.0.0.3  restart previous_hash=... current_hash=... matched=Makefile,bin/app
10.0.0.4  reload  previous_hash=... current_hash=...

--sync-only (CLI)

One-deploy override: rsync, post_deploy, and promote without start, restart, or reload. Same effect as restart_policy: never for updated allocations, but chosen on the command line.

Use when the process reads files directly, or when you will run maand job run <job> --target reload yourself afterward.

maand build
maand deploy --dry-run --sync-only --jobs api
maand deploy --sync-only --jobs api

--force --sync-only still skips lifecycle even when force would otherwise roll allocations.

job_control (custom lifecycle)

If the manifest registers job_control, default start / restart / reload are not used. Your script receives NEW_ALLOCATIONS, UPDATED_ALLOCATIONS, CURRENT_VERSION, and NEW_VERSION — implement canary, blue/green, or sync logic there. See hook-api.md.

restart_policy, restart_globs, and --sync-only do not apply on this path (except --sync-only still skips the script and only rsyncs + promotes).

Staging and rsync

Staging (tmp/workers/<worker_ip>/)

Rsync

• From the bucket directory on the CLI host to agent@<worker>:/opt/worker/<bucket_id>/ (user from maand.conf).
• Staging rsync: filter includes only jobs in jobsToStage (+ jobs/<job>/, - jobs/*).
• Final rsync: one pass per successfully deployed job, only to workers with an active allocation for that job; same per-job filter so other jobs on the host are not touched.

Templates (.tpl)

Files under jobs/<job>/ ending in .tpl are rendered at staging time. Full reference: templates.md.

Prometheus job staging

When the staged job ships prometheus.yml or prometheus.yml.tpl, maand assembles monitoring artifacts before rsync (from job_files, not from maand/prometheus KV except scrape expansion):

{{ scrapeConfigs }} reads scrape KV (maand/prometheus/scrape*), expands maand:port/* using active allocations, and skips jobs that would expand to zero targets (does not fail the whole render).

After deploy commits, maand best-effort pushes cert expiry metrics via Prometheus remote write (see certs.md).

Details: prometheus.md.

Removed and disabled allocations

Redeploying the same job on the same worker reuses existing data/ and logs/ (rsync excludes those paths). deploy deletes the allocation hash row when reconciling removed=1 allocations (even if the job is skipped from rollout), so a later redeploy treats it as a new rollout (make start) while worker data/ and logs/ remain. After build only, hashes still show the last promoted state until deploy runs.

When reconcile finishes and a job has no non-removed allocations (every row removed=1), deploy purges all job-scoped KV namespaces. Jobs with disabled-only allocations retain KV. maand build also clears build-owned namespaces when the job is inactive. Run maand gc to delete worker jobs/<job>/ trees and purge removed allocation rows from the catalog.

pre_deploy and post_deploy

• Registered in manifest with executed_on: pre_deploy / post_deploy.
• Run via hooks on the CLI host (Python or Bun).
• pre_deploy failure: job is not added to jobsToStage for this deploy; other jobs continue.
• post_deploy failure: fails that job’s deploy; sets post_deploy_status = failed on all non-removed allocations; earlier jobs in the same run may already be promoted. Retry: maand deploy runs pre-rolling health (if prior failed), then step 4 only.

KV checkpoint

Hooks can write to the in-memory KV store (e.g. connection strings for templates). After each job’s pre_deploy and after deployJob (including post_deploy), maand flushes pending KV changes into the deploy transaction via kv.PersistToSessionTransaction.

Those writes commit when deploy commits (including partial deploy — successful jobs persist even if later jobs fail). If deploy aborts before commit, KV checkpoint writes roll back with the rest of the catalog state.

Partial deploy and retry

1. Job A succeeds → content hashes and current_version promoted; post_deploy_status = success when hooks exist.
2. Job B fails mid-rollout (e.g. health on batch 2) → batch 1 allocations stay promoted; later batches not started; post_deploy skipped.
3. Job C fails post_deploy → all non-removed allocations marked post_deploy_status = failed; content already promoted.
4. Transaction still commits; command returns deploy failed with errors.
5. Fix and run maand deploy again:
   • Job A: skipped (deploy complete).
   • Job B: resumes rollout from unpromoted allocations.
   • Job C: post_deploy-only retry (step 4).

Use maand deploy --force to redeploy promoted jobs without a workspace change.

update_seq

When deploy stages at least one job (rsync to workers), bucket.update_seq increments once before the first staging wave (committed with the deploy transaction). Workers receive the new value in worker.json so they can detect bucket-wide changes.

Skip-only runs (deploy complete on all allocations) do not bump update_seq. maand deploy --dry-run never bumps it either.

Configuration

Uses maand.conf at the bucket root (SSH user, key file under secrets/, sudo for remote rsync). See configuration.md.

Worker key path: secrets/<ssh_key> relative to the bucket root.

Inspect state

maand cat allocations
maand cat deployments
maand cat jobs
maand info

Hash state lives in table hash with namespace <job>_allocation and key alloc_id. maand cat deployments shows current_hash, previous_hash, versions, post_deploy_status, and rollout (removed, disabled, or hash-derived new / restart / promoted / health_failed). Use --active to see only allocations deploy would target.

Common failures

Related

• build.md — catalog and sequences
• templates.md — .tpl rendering
• KV namespaces · KV persistence
• hooks.md · hook-api.md
• disable and drain — disable/re-enable
• rolling-deploy — max_concurrent_upgrades, version upgrades
• debugging-deploy.md — dry-run, cat deployments, failures
• health-check.md — standalone health checks
• job.md — manual start/stop/restart/reload
• gc.md — purge removed allocations