~/posts/self-contained-go-binary
go 2024-11-18 · 10 min

shipping a single self-contained go binary

embedded migrations and a self-writing config, so the service boots with zero external files

picture it: a fresh vps, your ssh key just accepted, a blank box that has never heard of your app. you scp up a binary. then you scp up a config.yaml. then you remember the database is empty, so you apt install the migrate cli, scp up the migrations/ folder too, point one at the other, and pray the versions line up. four artifacts and a small ceremony, just to watch the thing print a banner and die because the config key got renamed two commits ago.

i did that dance one too many times on a little learning project and thought: why does a single go binary need a goddamn support convoy to boot?

the project is mojodojo — a rest api for running a martial-arts gym that i built through 2024 to actually feel hexagonal architecture in go in my hands instead of nodding along to a diagram. that post was about the shape of the code. this one is about a much more boring, much more useful property i backed into along the way: it boots with zero external files. no config to ship, no migration step to run, no sidecar. you copy up one binary, you run it, it sorts itself out.

for a solo dev, or a two-person team with no platform group to lean on, that property is worth more than any amount of clever domain modelling. here’s how the three pieces fit.

the ritual i wanted to kill

the old convoy looked like this, every single deploy:

bash 
scp mojodojo            user@box:/opt/mojodojo/
scp config.yaml         user@box:/opt/mojodojo/
scp -r migrations/      user@box:/opt/mojodojo/
ssh user@box 'migrate -path migrations -database "$DSN" up'

three of those four lines are just moving files the binary already knows about. the migrations were written by me, for this binary. the config schema is defined in this binary. and yet i was treating them like third-party assets that had to be couriered in separately and kept in sync by hand. the whole thing is bullshit, and go has had the fix since 1.16.

everything it needs, it carries

go 1.16 gave us embed ↗ , and it quietly changed what “a binary” means. a //go:embed directive reads files off disk at compile time and bakes them into the executable. the entire surface for mojodojo is one file at the module root:

embed.go

go 
package mojodojocasahouse

import (
	"embed"
)

// DefaultConfig holds the default configuration.
//
//go:embed config/config.default.yaml
var DefaultConfig []byte

//go:embed migrations
var Migrations embed.FS

two directives, two flavours. config.default.yaml is a single file, so it lands in a plain []byte. migrations is a directory, so it becomes an embed.FS ↗ — a read-only, io/fs-shaped filesystem i can hand to anything that expects an fs.FS. the docs ↗ are strict about the rules: the directive has to sit immediately above the var, patterns use forward slashes even on windows, and dotfiles are skipped unless you ask for them with all:. mind those and the compiler does the rest.

what’s embedded is exactly the convoy i used to scp:

text 
migrations/
├── 000001_setup_database.up.sql
└── 000001_setup_database.down.sql

after go build, none of that exists on the target box as a file. the yaml and every .sql line live inside the executable.

a config that writes itself

embedding the default config is half the trick. the other half is doing something useful with it on first boot. config.Load tries to read the file; if — and only if — it’s genuinely absent, it writes the embedded default to disk and re-reads it:

internal/common/config/config.go

go 
func Load(v *viper.Viper, path string) *Config {
	v.SetConfigFile(path)

	if err := v.ReadInConfig(); err != nil {
		// a real read error (perms, malformed yaml) is fatal...
		if _, err := os.Stat(path); !os.IsNotExist(err) {
			logger.Fatal("[LoadConfiguration] - Couldn't read existing config file: %v", err)
		}

		// ...but "file not there" just means this is a first boot.
		log.Printf("[LoadConfiguration] - No config file. Creating default one...")

		if err := os.MkdirAll(filepath.Dir(path), 0770); err != nil {
			logger.Fatal("[LoadConfiguration] - Couldn't create default config path: %v", err)
		}

		f, err := os.Create(path)
		if err != nil {
			logger.Fatal("[LoadConfiguration] - Couldn't create default config file: %v", err)
		}
		defer f.Close()

		// write the embedded default straight to disk
		if _, err = f.Write(mojodojocasahouse.DefaultConfig); err != nil {
			logger.Fatal("[LoadConfiguration] - Couldn't write default configuration: %v", err)
		}

		// then re-read the file we just created
		if err := v.ReadInConfig(); err != nil {
			logger.Fatal("[LoadConfiguration] - Couldn't load created default config file: %v", err)
		}
	}

	config := &Config{}
	if err := v.Unmarshal(&config); err != nil {
		logger.Fatal("[LoadConfiguration] - Couldn't unmarshal config file: %v", err)
	}
	// ...
	return config
}

the detail i’m quietly proud of is the os.Stat / os.IsNotExist split. a failed ReadInConfig is ambiguous — the file could be missing, or it could be there and broken. “missing” is a normal first boot and should self-heal; “broken” is an operator error and should stop the world loudly. lumping those together is how you get a service that silently overwrites a config someone hand-edited. mojodojocasahouse.DefaultConfig there is the exact byte slice from embed.go — no disk read, it was compiled in.

so is it really zero files?

no, and i want to be straight about it. the config it writes is a template full of placeholders, not working settings:

config/config.default.yaml

yaml 
port: 3000

database:
  host: localhost
  port: 5432
  user: myUser
  password: myPassword
  dbName: databaseName
  ssl: disable
  schema: yourschema

auth:
  secret: your-super-duper-secure-secret
  auth0:
    domain: your-domain
    clientId: your-client-id
    clientSecret: your-client-secret
    callbackUrl: your-callback-url

log:
  format: json
  level: info
  path: path/to/file.log

so on a truly fresh box the service writes this, re-reads it, and cheerfully tries to reach localhost:5432 as myUser. “zero files to boot” is true. “zero files to do anything useful” is a lie — you still edit the dropped file and restart once. what embedding actually buys you is that the file’s shape is always correct and always present. no more “wait, which keys does this version want?” the schema and the binary can never drift, because the schema ships inside the binary.

migrations that run themselves

the config trick is cute. the migration trick is the one that actually removes a moving part from every deploy. the postgres client takes the embedded embed.FS, wraps it in golang-migrate’s iofs ↗ source, and runs anything pending — right there in the constructor:

internal/infra/postgre/client.go

go 
import (
	"github.com/golang-migrate/migrate/v4"
	psqlMigrate "github.com/golang-migrate/migrate/v4/database/postgres"
	"github.com/golang-migrate/migrate/v4/source/iofs"
	"gorm.io/driver/postgres"
	"gorm.io/gorm"

	mojodojocasahouse "mojodojo" // root pkg holding the embedded migrations
	// ...
)

iofs exists precisely for this: its docs say it “can accept various file systems (like embed.FS) implementing io/fs#FS.” the wiring is almost verbatim from their example:

internal/infra/postgre/client.go

go 
// build a golang-migrate source from the embedded FS
d, err := iofs.New(mojodojocasahouse.Migrations, "migrations")
if err != nil {
	return nil, ErrInitPostgreClient
}

m, err := migrate.NewWithInstance("iofs", d, "postgres", driver)
if err != nil {
	return nil, ErrInitPostgreClient
}

// apply everything pending; ErrNoChange just means we're already current
err = m.Up()
if err != nil && err == migrate.ErrNoChange {
	v, _, err := m.Version()
	// ... (bail on err)
	logger.Debug("[Postgre] - [NewClient] Database already at version: %d. No further migrations found", v)
	return db, nil
}
// ... (any other err fails client init)

newV, _, err := m.Version()
// ...
logger.Debug("[Postgre] - [NewClient] Database migrated to version: %d", newV)
return db, nil

m.Up() applies every migration the database hasn’t seen and reports the resulting schema version. migrate.ErrNoChange isn’t a failure — it’s the happy “you’re up to date” path, and it’s worth handling explicitly so a clean boot doesn’t look like an error. (confession: there’s still a blank _ "github.com/golang-migrate/migrate/v4/source/file" import loitering in that file from before i embedded anything. embedded means i don’t need the file source at all — and yet it sits there. warts.)

net result: to migrate the database, i deploy the binary. that’s the whole procedure.

a binary that knows its own name

if the binary is going to be the only artifact, it had better be able to tell me which build it is. no external version.txt, no git checkout on the box. the standard go answer is the linker’s -X flag, driven from the Makefile:

Makefile

makefile 
REVISION := $(shell git rev-parse --short=8 HEAD || echo unknown)
VERSION  ?= $(shell git describe --tags --always)
BUILT    := $(shell date +%Y-%m-%dT%H:%M:%S)
BRANCH   := $(shell git show-ref | grep "$(REVISION)" | ... )

GO_LDFLAGS ?= -X mojodojo/main.Version=$(VERSION) \
              -X mojodojo/main.Branch=$(BRANCH) \
              -X mojodojo/main.Revision=$(REVISION) \
              -X mojodojo/main.Built=$(BUILT) \
              -s \
              -w

-X importpath.name=value ↗ sets a string variable at link time. the linker docs are explicit about the one constraint that bites everyone: it only works if the variable is “declared in the source code either uninitialized or initialized to a constant string expression.” so the receiving vars are plain constant-string defaults, printed by a banner on startup:

cmd/server/main.go

go 
var (
	pathToConfig = flag.String("config", "config/config.yaml", "path to load config file")
	Version      = "dev"
	Branch       = "NA"
	Revision     = "NA"
	Built        = "NA"
)

func banner() {
	fmt.Printf("--------------------------------\n")
	fmt.Printf("       Mojo Dojo Casa House     \n")
	fmt.Printf("--------------------------------\n")
	fmt.Printf(" Version  : %s\n", Version)
	fmt.Printf(" Branch   : %s\n", Branch)
	fmt.Printf(" Revision : %s\n", Revision)
	fmt.Printf(" Built    : %s\n", Built)
	fmt.Printf("------------------------------\n\n")
}

and here is the most honest thing in this whole post: for a long time that banner lied to me. look closely at the Makefile — it targets mojodojo/main.Version. but the vars live in package main at mojodojo/cmd/server, so the import path is .../cmd/server, not .../main. -X matches on import path plus name, finds nothing at .../main, and — per those same linker docs — silently injects nothing. no error. every build i shipped proudly announced Version: dev. the footgun of -X isn’t the flag, it’s that a wrong path fails quiet.

the honest part: convenient now, a footgun at scale

running m.Up() inside the postgres constructor is lovely when there’s exactly one of you and exactly one process. it stops being lovely the moment there are two.

  • two replicas booting together race the same migration. golang-migrate takes a lock, so they won’t both run it — but the loser blocks on that lock, and if the migration is slow, your snappy rolling deploy stalls behind schema surgery nobody scheduled.
  • a new replica can migrate a schema the old replicas are still serving. v2 pod comes up, runs a migration that drops a column, and your still-live v1 pods start throwing 500s because they’re querying a shape that no longer exists. the migration and the code that assumes it went out of sync by design.
  • there is no gate. boot means migrate. you can’t ship code without also mutating the database in the same breath — exactly the coupling every “expand / contract” migration guide begs you to avoid.

for a single-instance learning project this is correct, and i’d do it again without blinking. for anything with replicas, the boring-but-right move is to lift migrations out of the boot path and run them as their own deliberate step — an init container, a k8s job, a migrate call in ci — before the new code rolls. same embedded embed.FS, same golang-migrate, just triggered on purpose instead of as a side effect of main(). i knew this when i wrote it. i’m writing it down so future-me doesn’t paste this file into something that sits behind a load balancer.

lessons learned

  • “zero external files to boot” is a real, shippable property, and embed basically hands it to you. migrations and config templates aren’t third-party assets — they’re your code. compile them in and a whole column of your deploy runbook disappears.
  • self-healing should be surgical, not eager. the os.IsNotExist check is the difference between “recreate a missing default” and “clobber a config an operator carefully edited.” one branch, big consequences.
  • convenience at n=1 is usually a coupling at n=2. auto-migrate-on-boot is the cleanest example i own: the exact line that makes solo ops delightful is the first line you rip out when you grow a second replica. know which line that is before you need to.
  • read the linker docs before you trust your own version string. -X matches the full import path, fails silent on a typo, and will happily let you deploy dev to a hundred boxes.

next, still on mojodojo: fourteen entities, six near-identical files each, and an 890-line router — the boilerplate tax, and whether a code generator should have been writing most of this for me.

mojodojo is a 2024 learning project and it lives, rough and unfinished, on my github ↗ — which is the point.

letters to the editor

no letters yet. the editor is patient.