testcontainers 102

alright. so the last post was a simple introduction to testcontainers: what are they, how to set it up, how to run a couple integration tests against a database… it was quite good for an introduction, but there were of course things missing.

basically, i’ve been playing around with testcontainers some more and i’ve decided to:

i. make some changes on the setup and the structure,

ii. add more scenarios to the test.

and now if i look back at my previous code - well, never look back at the code you’ve written. it’ll look like shit.

if you’re running rancher desktop it’s too late part 2

before anything else: running testcontainers on rancher desktop is starting to bother the shit out of me.

sometimes they spin up perfectly. other times they’re stuck in limbo, refusing to run, making me waste my time until they finally timeout after a couple of minutes.

i’ve read the issue multiple times back and forth and i still don’t understand what’s going on. moral of the story? why the fuck are we using macOS?

i’ve had no issue running them on my fedora laptop, so my approach for now is just to stop using the mac when i want to have some cool ass testcontainers on my project.

introducing: test helpers

i kind of gave the idea away in part 1, so let’s start with that.

container setup

First, let’s create a helper for our PostgreSQL container setup:

internal/common/testhelpers/containers.go

type PostgreSQLContainer struct {
	*postgres.PostgresContainer
	ConnectionString string
}

func CreatePostgreSQLContainer(ctx context.Context) (*PostgreSQLContainer, error) {
	pgContainer, err := postgres.Run(
		ctx,
		"postgres:16.4",
		postgres.WithDatabase("mojodojo-test"),
		postgres.WithUsername("postgres"),
		postgres.WithPassword("postgres"),
		testcontainers.WithWaitStrategy(
			wait.
				ForLog("database system is ready to accept connections").
				WithOccurrence(2).
				WithStartupTimeout(5*time.Second),
		),
	)
	if err != nil {
		return nil, err
	}

	connStr, err := pgContainer.ConnectionString(ctx, "sslmode=disable")
	if err != nil {
		return nil, err
	}

	return &PostgreSQLContainer{
		PostgresContainer: pgContainer,
		ConnectionString:  connStr,
	}, nil
}

database migrations

Next, let’s extract our migration helpers. First, the “up” migrations:

internal/common/testhelpers/migrations.go

func RunMigrationsUp(db *sql.DB) error {
	driver, err := psqlMigrate.WithInstance(db, &psqlMigrate.Config{})
	if err != nil {
		return err
	}

	d, err := iofs.New(mojodojo.Migrations, "migrations")
	if err != nil {
		return err
	}

	m, err := migrate.NewWithInstance(
		"iofs",
		d,
		"postgres",
		driver,
	)
	if err != nil {
		return err
	}

	err = m.Up()
	if err != nil {
		return err
	}

	return nil
}

And then the “down” migrations to clean up after tests:

func RunMigrationsDown(db *sql.DB) error {
	driver, err := psqlMigrate.WithInstance(db, &psqlMigrate.Config{})
	if err != nil {
		return err
	}

	d, err := iofs.New(mojodojo.Migrations, "migrations")
	if err != nil {
		return err
	}

	m, err := migrate.NewWithInstance(
		"iofs",
		d,
		"postgres",
		driver,
	)
	if err != nil {
		return err
	}

	err = m.Down()
	if err != nil {
		return err
	}

	return nil
}

this leaves our test suite leaner and allows us to reuse these methods across all other test suites we might add.

refactoring the test suite

Now our suite definition becomes much cleaner:

type DojoRepositoryTestSuite struct {
	suite.Suite
	db          *gorm.DB // to interact directly with db
	pgContainer *testhelpers.PostgreSQLContainer // now from the testhelpers package
	ctx         context.Context
}

And the setup/teardown methods are simplified:

func (suite *DojoRepositoryTestSuite) SetupSuite() {
	suite.ctx = context.Background()

	pgContainer, err := testhelpers.CreatePostgreSQLContainer(suite.ctx)
	suite.NoError(err)

	db, err := gorm.Open(pg.Open(pgContainer.ConnectionString), &gorm.Config{})
	suite.NoError(err)

	suite.pgContainer = pgContainer
	suite.db = db

	sqlDB, err := suite.db.DB()
	suite.NoError(err)

	err = sqlDB.Ping()
	suite.NoError(err)
}

func (suite *DojoRepositoryTestSuite) TearDownSuite() {
	err := suite.pgContainer.Terminate(suite.ctx)
	suite.NoError(err)
}

func (suite *DojoRepositoryTestSuite) SetupTest() {
	sqlDB, err := suite.db.DB()
	suite.NoError(err)

	err = testhelpers.RunMigrationsUp(sqlDB)
	suite.NoError(err)
}

func (suite *DojoRepositoryTestSuite) TearDownTest() {
	sqlDB, err := suite.db.DB()
	suite.NoError(err)

	err = testhelpers.RunMigrationsDown(sqlDB)
	suite.NoError(err)
}

Since we’re going to add more test scenarios, let’s include a method to clear the entity table between tests:

func (suite *DojoRepositoryTestSuite) Clear() error {
	r := suite.db.Delete(&domain.Dojo{}, "1=1")
	return r.Error
}

extending our test scenarios

Now that we have a cleaner setup, let’s add more comprehensive tests. Here are some scenarios we should cover:

testing validation errors

What happens when we try to create an entity with invalid data?

func (suite *DojoRepositoryTestSuite) TestCreateWithInvalidData() {
	repo := NewDojoRepository(suite.db)
	
	// Try to create with empty name
	entity, err := domain.NewDojo(
		"", // empty name, should fail validation
		"123 Main St",
		"555-1234",
		"[email protected]",
		"2023/01/01",
	)
	
	// NewDojo should fail with validation error
	suite.Error(err)
	suite.Equal("name: cannot be blank", err.Error())
	
	// Try to create with invalid date
	entity, err = domain.NewDojo(
		"Valid Dojo", 
		"123 Main St",
		"555-1234",
		"[email protected]",
		"not-a-date", // invalid date format
	)
	
	// Should fail with date validation error
	suite.Error(err)
}

testing with filters

Let’s make sure our List method works with filters:

func (suite *DojoRepositoryTestSuite) TestListWithFilters() {
	// Create a few test entities
	entities := []domain.Dojo{
		{
			ID:      uuid.NewString(),
			Name:    "Cobra Kai",
			Address: "Reseda, CA",
		},
		{
			ID:      uuid.NewString(),
			Name:    "Miyagi-Do",
			Address: "Encino, CA",
		},
		{
			ID:      uuid.NewString(),
			Name:    "Eagle Fang",
			Address: "Valley, CA",
		},
	}
	
	// Add them to the database
	for _, entity := range entities {
		result := suite.db.Create(&entity)
		suite.NoError(result.Error)
	}
	
	repo := NewDojoRepository(suite.db)
	
	// Test filter by name
	filter := &domain.DojoFilter{
		Name: "Cobra Kai",
	}
	
	results, err := repo.List(filter)
	suite.NoError(err)
	suite.Equal(1, len(results))
	suite.Equal("Cobra Kai", results[0].Name)
	
	// Test with no filter (should get all)
	filter = &domain.DojoFilter{}
	results, err = repo.List(filter)
	suite.NoError(err)
	suite.Equal(3, len(results))
}

testing edge cases

What about when an entity doesn’t exist?

func (suite *DojoRepositoryTestSuite) TestGetNonExistent() {
	repo := NewDojoRepository(suite.db)
	
	// Try to get an entity with a random UUID
	nonExistentID := uuid.NewString()
	_, err := repo.Get(nonExistentID)
	
	// Should get an error
	suite.Error(err)
	suite.Contains(err.Error(), domain.ErrGetDojo.Error())
}

testing concurrency

Testing concurrent operations is important for real-world scenarios:

func (suite *DojoRepositoryTestSuite) TestConcurrentCreates() {
	repo := NewDojoRepository(suite.db)
	
	var wg sync.WaitGroup
	errChan := make(chan error, 10)
	
	// Try to create 10 entities concurrently
	for i := 0; i < 10; i++ {
		wg.Add(1)
		go func(index int) {
			defer wg.Done()
			
			entity, err := domain.NewDojo(
				fmt.Sprintf("Concurrent Dojo %d", index),
				fmt.Sprintf("Address %d", index),
				"",
				"",
				"",
			)
			
			if err != nil {
				errChan <- err
				return
			}
			
			_, err = repo.Create(&entity)
			if err != nil {
				errChan <- err
			}
		}(i)
	}
	
	wg.Wait()
	close(errChan)
	
	// Check if any errors occurred
	for err := range errChan {
		suite.Fail("Error in concurrent creation:", err)
	}
	
	// Verify all 10 were created
	var count int64
	suite.db.Model(&domain.Dojo{}).Count(&count)
	suite.Equal(int64(10), count)
}

updating our test runner

Let’s make sure our test suite runs with the new tests:

func TestDojoRepository(t *testing.T) {
	if testing.Short() {
		t.Skip("skipping integration test in short mode")
	}
	
	suite.Run(t, new(DojoRepositoryTestSuite))
}

Notice I added a check for testing.Short() which allows you to skip integration tests when you’re just running unit tests with go test -short.

improving our test helpers even more

Let’s take it a step further and create a base test suite that all our repository tests can embed:

// BaseRepositoryTestSuite provides common functionality for all repository test suites
type BaseRepositoryTestSuite struct {
	suite.Suite
	db          *gorm.DB
	pgContainer *PostgreSQLContainer
	ctx         context.Context
}

func (suite *BaseRepositoryTestSuite) SetupSuite() {
	suite.ctx = context.Background()

	pgContainer, err := CreatePostgreSQLContainer(suite.ctx)
	suite.NoError(err)

	db, err := gorm.Open(pg.Open(pgContainer.ConnectionString), &gorm.Config{})
	suite.NoError(err)

	suite.pgContainer = pgContainer
	suite.db = db

	sqlDB, err := suite.db.DB()
	suite.NoError(err)

	err = sqlDB.Ping()
	suite.NoError(err)
}

func (suite *BaseRepositoryTestSuite) TearDownSuite() {
	err := suite.pgContainer.Terminate(suite.ctx)
	suite.NoError(err)
}

func (suite *BaseRepositoryTestSuite) SetupTest() {
	sqlDB, err := suite.db.DB()
	suite.NoError(err)

	err = RunMigrationsUp(sqlDB)
	suite.NoError(err)
}

func (suite *BaseRepositoryTestSuite) TearDownTest() {
	sqlDB, err := suite.db.DB()
	suite.NoError(err)

	err = RunMigrationsDown(sqlDB)
	suite.NoError(err)
}

// DB returns the database connection
func (suite *BaseRepositoryTestSuite) DB() *gorm.DB {
	return suite.db
}

// Context returns the test context
func (suite *BaseRepositoryTestSuite) Context() context.Context {
	return suite.ctx
}

Then our dojo test suite becomes super clean:

type DojoRepositoryTestSuite struct {
	testhelpers.BaseRepositoryTestSuite
}

func (suite *DojoRepositoryTestSuite) TestCreate() {
	// Our tests using suite.DB() instead of suite.db
	// ...
}

wrapping up

with these improvements, our test suite is now:

  1. more maintainable - common code is extracted to reusable helpers
  2. more comprehensive - we test more scenarios and edge cases
  3. more resilient - we handle concurrency and invalid inputs
  4. more flexible - we can easily create new test suites for other repositories

yes, it’s a bit more code upfront, but it pays off massively when your application grows and you need to test multiple repositories with similar patterns.

what’s next?

in part 3 (if i ever get around to writing it), we might explore:

  • testing with transactions
  • more complex database interactions
  • testing with multiple dependency containers (postgres + redis, anyone?)
  • performance testing

but for now, this should give you a solid foundation for testing your go applications with testcontainers. maybe one day i’ll actually write proper tests for my own projects before they blow up in production… but let’s not get carried away with unrealistic expectations.

remember: tests are like insurance - nobody wants to pay for them until after the accident happens.

krtffl.dev

physicist → chemist → dev

testing: the real deal??

2024-09-22

part of the testcontainers series

on this page: