docker stack deploy silently dropped my command (and 3 other swarm lies)

picture the scene: green across the board. the deploy script ran all three of its pre-flight gates, docker compose config came back clean, docker stack deploy converged every service, and the post-deploy verifier printed a wall of OK in cheerful green. i closed the laptop.

then, a day later, i went looking at postgres and found it running with the default max_connections, and no TLS, and none of the eighteen -c tuning flags i’d very carefully written into the stack file. the flags were right there in the yaml. docker compose config printed them back to me, perfectly. and the running container had never seen a single one of them.

this is a footgun i hit later in the same single-vps swarm setup i wrote about in part 1 — traefik at the edge, one app + postgres + a backup sidecar, everything as compose-spec stacks handed to docker stack deploy. part 1 was the happy path. this is the part where i learned that “the yaml is valid” and “the yaml is running” are two completely different claims, and that swarm will happily let you believe the first one while quietly ignoring you on the second.

there are four keys docker stack deploy drops on the floor. here’s each one, why docker compose config won’t warn you, and how i catch them now.

the two commands don’t agree, and that’s the whole problem

here’s the thing nobody puts on the box: docker compose config and docker stack deploy are different code paths reading the same file.

docker compose config is the compose cli’s validator. it parses your stack, merges your x- anchors and ${VAR} interpolation, and renders the full, canonical spec back at you. when it exits 0, all it has told you is “this is well-formed compose and i understood it.” it has said nothing about swarm, because it isn’t the thing that talks to swarm.

docker stack deploy is a different converter. it takes that same compose file and translates it into swarm service objects — a strictly smaller target than the compose spec. anything in the file that doesn’t map onto a swarm service field gets… handled. sometimes with a warning. sometimes not. and my deploy script’s syntax gate was built on the first command:

scripts/deploy.sh

# Gate 3: each stack passes compose syntax validation.
for stack in "${stacks[@]}"; do
  if ! docker compose -f "${stack}" config --quiet; then
    die "Gate 3 failed: invalid syntax in ${stack}"
  fi
done

a green gate 3 means the file is valid. it does not mean the file is deployed as written. i spent a good while conflating those two, so let me walk the four keys that taught me the difference — worst first.

lie #1: the command it swallowed whole

this is the headline, because it’s the only one of the four that swarm ignores with zero warning.

stacks/app.yml

  postgres:
    image: postgres:17.10-alpine3.23@sha256:4f8950c6…   # multi-arch index digest
    user: "70:70"
    cap_drop:
      - ALL
    command:
      - "postgres"
      - "-c"
      - "max_connections=100"
      - "-c"
      - "shared_buffers=192MB"
      # ... a dozen more tuning flags ...
      - "-c"
      - "ssl=on"
      - "-c"
      - "ssl_cert_file=/run/secrets/postgres_server_cert"
      - "-c"
      - "ssl_key_file=/run/secrets/postgres_server_key"

run that through the validator and it’s flawless — the command: list renders straight back at you. deploy it and inspect what actually landed:

# the rendered compose has the command, plain as day
docker compose -f stacks/app.yml config | grep max_connections

# the LIVE service has no args at all
docker service inspect myapp_app_postgres \
  --format '{{json .Spec.TaskTemplate.ContainerSpec.Args}}'
# null

null. not “wrong”. not “partially applied”. the container booted on its image’s default entrypoint args as if my command: block didn’t exist, which for a database means default tuning and, worse, ssl off. no error, no Ignoring unsupported options line, nothing in the deploy output to grep for. it renders correctly and applies not at all, and that combination is the single most dangerous thing a config key can do to you.

what makes it maddening is that swarm clearly can set service args — docker service create and docker service update --args do it every day. the compose-to-swarm converter just… doesn’t carry the command: across on a fresh stack deploy for these services. so the fix isn’t a different key. the fix is to notice the drop and re-apply the args yourself — which is the reconcile loop further down.

lie #2: no-new-privileges, security theatre edition

second key, and this one at least has the decency to warn you — if you’re looking at the right command’s output.

stacks/app.yml

  postgres:
    security_opt:
      - no-new-privileges:true   # documentation marker (enforced in daemon.json)

docker compose config renders security_opt happily. docker stack deploy, however, prints Ignoring unsupported options: cap_add, devices, privileged, security_opt, tmpfs and moves on — swarm services simply have no --security-opt field. that’s not a bug in my file; it’s a nine-year-old gap. moby/moby#30846 ↗ is where the exact ignore string comes from, and moby/moby#41371 ↗ is the still-open feature request asking docker service create to support --security-opt at all. it doesn’t. so the key in your stack file is decoration.

the honest move is to stop pretending the stack file enforces it and enforce it where swarm can’t ignore you: node-wide, in the daemon itself.

scripts/init-server.sh

DAEMON_JSON_DESIRED="$(cat <<'DAEMON'
{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" },
  "no-new-privileges": true,
  "userland-proxy": false,
  "icc": false
}
DAEMON
)"
# write only when the content actually changes, then:
systemctl restart docker

no-new-privileges: true ↗ in /etc/docker/daemon.json makes it the default for every container the engine runs, service or not. one detail cost me an afternoon: a systemctl reload docker does not reliably re-read this — on engine 29.x i watched docker info keep reporting the old value after a reload. a full restart is what applies it. i keep the security_opt lines in the stack file anyway, as a comment-shaped promise to my future self and forward-compat for the day swarm grows the field, but the daemon is the thing doing the work.

lie #3: the short tmpfs list

my services run read_only: true — the root filesystem is immutable, and each one gets exactly the writable scratch it needs back as tmpfs. the obvious way to write that is the short form:

    read_only: true
    tmpfs:
      - /tmp

and the obvious way is on that same Ignoring unsupported options list from lie #2. tmpfs (short form) gets dropped. so you deploy a container with a read-only root and no writable /tmp, and it falls over the first time it tries to write a scratch file — a failure that shows up at runtime, in the app, nowhere near the deploy. the fix is the long-form mount, which goes through the volumes converter that swarm actually honours:

stacks/app.yml

  api:
    read_only: true
    # short `tmpfs: [/tmp]` is dropped by stack deploy; the long form survives.
    volumes:
      - type: tmpfs
        target: /tmp

same intent, different door. one gets ignored, one goes through. there is no reason for that other than “the swarm converter supports the volumes path and not the short key,” which is exactly the kind of thing you only learn by having a container refuse to boot.

lie #4: version, the ghost that at least warns you

the mildest of the four, included because it completes the pattern. the top-level version: field — the version: "3.8" muscle memory from every compose tutorial ever — is, per the compose spec ↗ , “only informative and you’ll receive a warning message that it is obsolete if used.” swarm ignores it; compose validates against the latest schema regardless. it’s harmless, but it’s the same lie in miniature: a key that reads as meaningful and does nothing. so it isn’t in my files at all:

stacks/app.yml

---
# Compose Spec (NO `version:` key — Swarm ignores it; the spec is version-less).
services:
  api:
    # ...

deleting it is the whole fix. i list it because if you’re carrying a version: line out of habit, it’s the cheap tell that you’re thinking in compose-cli terms when you’re deploying to swarm — the exact mindset that lets the other three bite you.

detecting them: check the wire, not the syntax

here’s the lesson under all four: syntax-valid is not deployed. a green docker compose config proves your file parses. it proves nothing about what swarm did with it. so the verifier that runs after every deploy doesn’t re-check the yaml — it interrogates the live system and asserts the properties are actually on the wire.

scripts/verify-deployment.sh

echo -n "Checking no-new-privileges default... "
# current engines list "no-new-privileges" as a bare bullet under Security
# Options — the old "No New Privileges: true" string never matches on 29.x.
if docker info 2>/dev/null | grep -q "no-new-privileges"; then
  echo -e "${GREEN}OK${NC}"
else
  echo -e "${RED}FAILED${NC} (set \"no-new-privileges\": true in /etc/docker/daemon.json)"
  ERRORS=$((ERRORS+1))
fi

that check asks the daemon, not the file. and note the comment — even the shape of docker info’s output changed between engine versions, so an older verifier that grepped for No New Privileges: true would have passed a box where the setting was off. the check tests the property, adapts to the engine, and fails the run when the wire disagrees with the intent. every one of these four keys deserves that treatment: assert the effect, never trust the source.

the fix: reconcile the args swarm forgot

detection is half of it. for lie #1 — the silently-dropped command: — i also want it repaired, automatically, on every deploy, without hardcoding a service name. so deploy.sh, right after the stack deploys, diffs the live args against the rendered compose and patches the difference:

scripts/deploy.sh

if [[ -f "stacks/app.yml" ]]; then
  require_cmd jq
  app_config_json="$(docker compose -f stacks/app.yml config --format json)"
  while IFS= read -r svc; do
    [[ -n "${svc}" ]] || continue
    swarm_service="${PROJECT}_app_${svc}"
    docker service inspect "${swarm_service}" >/dev/null 2>&1 || continue

    args_wanted="$(jq -c --arg svc "${svc}" '.services[$svc].command' <<<"${app_config_json}")"
    args_current="$(docker service inspect "${swarm_service}" \
      --format '{{json .Spec.TaskTemplate.ContainerSpec.Args}}' | jq -c '.')"

    if [[ "${args_current}" != "${args_wanted}" ]]; then
      log "  Patching ${swarm_service} args (stack deploy did not apply command:)..."
      docker service update --args "$(jq -r 'join(" ")' <<<"${args_wanted}")" "${swarm_service}" >/dev/null
    fi
  done < <(jq -r '.services | to_entries[] | select(.value.command != null) | .key' <<<"${app_config_json}")
fi

the shape is: enumerate every service that declares a command: in the rendered compose, look up its live .Spec.TaskTemplate.ContainerSpec.Args, and if they differ, re-apply via docker service update --args ↗ — which, unlike stack deploy, goes straight at the swarm api and does set the args. it’s idempotent: on a deploy where nothing changed, args_current == args_wanted and the loop patches nothing, restarts nothing. it’s also generic — it keys off “does this service have a command,” not “is this service named postgres,” so any future service with a non-default entrypoint gets fixed for free.

the honest part: this is a workaround, and it’s a bit fragile

i want to be straight about what this loop actually is: a patch over a real gap in docker stack deploy, and not a clean one.

• jq -r 'join(" ")' flattens the args into a single space-joined string. that’s fine for -c max_connections=100 and every flag i actually pass. it would quietly mangle any argument that contains a space, because --args re-splits on whitespace. i don’t have such an arg today; the day i do, this breaks and i’ll deserve it.
• patching restarts the task. docker service update --args rolls the service, so the reconcile is a second, sneaky deployment event stapled to the end of the first. for postgres — which i deploy stop-first — that’s a brief blip that the actual stack deploy didn’t cause. the log line is there so at least it’s not a mystery blip.
• it only covers stacks/app.yml. traefik and monitoring don’t get the treatment, on the bet that they don’t rely on a dropped command:. that’s a bet, not a proof.
• it fixes exactly one of the four lies. security_opt is handled out-of-band by the daemon, tmpfs and version are handled by writing the yaml differently. the loop is not a general “make swarm honour compose” machine, and pretending it were would be its own lie.

none of that makes it wrong to ship. at n=1, on one vps, a nine-line reconcile loop that turns a silent database misconfiguration into an automatic, logged, idempotent repair is a very good trade. but it exists because stack deploy won’t carry a command: and won’t tell you it didn’t, and no amount of clever bash upstream turns that into a feature. the deeper fix isn’t in my repo. it’s the two moby issues staying closed.

lessons learned

• docker compose config passing is a claim about your file, not about your cluster. it is the single most misleading green checkmark in the swarm workflow. treat it as “well-formed,” never as “deployed as written.”
• verify the effect on the live system, not the intent in the file. grep docker info for the daemon flag, inspect .Spec.TaskTemplate for the args, assert the property is on the wire. the file lies; the running system can’t.
• when a key is ignored, find where the engine can enforce it. no-new-privileges belongs in daemon.json, not in a security_opt line swarm throws away. push the guarantee down to the layer that owns it.
• write your workarounds down, warts loud. an idempotent reconcile loop with a join(" ") footgun and a one-line comment about why it has to exist is honest engineering. the same loop with no comment is a landmine for whoever reads it next, including me.

next in this series: the postgres service runs cap_drop: ALL with an empty cap_add, and it took me an embarrassingly long time to understand why running the container as uid 70 is what makes zero capabilities actually work. capabilities, gosu, and the entrypoint chown that never happens.