Go’s synctest is amazing

We threw Go’s new “testing/synctest” package at a particularly gnarly part of our codebase and were pleasantly surprised by how effective it was. This post covers the synctest package, its nuances, and how it does much more than speed up your tests.

The main headline for Go 1.25’s synctest is its ability to magically advance time. Tests run in a “bubble” with a fake clock and calls to time.Sleep are virtualized, causing them to seem to run instantly:

// This test runs instantly!
func TestSleep(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		var t1, t2 time.Time
		go func() {
			time.Sleep(time.Second)
			t1 = time.Now()
		}()

		time.Sleep(time.Second * 2)
		t2 = time.Now()
		if t1.After(t2) {
			t.Errorf("Expected t2 (%s) to be after t1 (%s)", t2, t1)
		}
	})
}

Run on the Go Playground

But even more than speed, synctest’s real advantage is being able to deterministically reason about the ordering of events in a test. Let me explain.

Background loops

Our product has a large number of background routines. These can be as simple as deleting expired database rows, or more complex logic like leader election between instances.

These routines take the form of Run functions that call another function in a loop (plus some backoff on failures). This is an abbreviated sample of what that might look like:

type DB struct { /* */ }
func NewDB(ctx context.Context, path string) (*DB, error) { /* */ }
func (db *DB) Close() error { /* */ }
func (db *DB) CreateSession(ctx context.Context, id string, exp time.Time) error { /* */}
func (db *DB) GetSessionExpiry(ctx context.Context, id string) (time.Time, error) { /* */ }
func (db *DB) DeleteExpiredSessions(ctx context.Context, now time.Time) error { /* */ }

// Ummm... how do we test RunDeleteExpiredSessions?

func RunDeleteExpiredSessions(ctx context.Context, db *DB) {
	for {
		select {
		case <-ctx.Done():
			return
		case <-time.After(time.Minute):
			// Run every minute.
			if err := db.DeleteExpiredSessions(ctx, time.Now()); err != nil {
				log.Printf("Deleting expired sessions: %v", err)
			}
		}
	}
}

How do we write a test if we need to wait a minute for the logic to execute? This gets even more complicated with nested loops. Our leader election performs different sub-loops depending on if the current process is the leader or not. What do we do?

synctest is about blocking

synctest advances time when it considers all goroutines within the test “blocked”. There are a few rules, but this largely means channel operations, wait groups, and time methods.

https://pkg.go.dev/testing/synctest#hdr-Blocking

The following test demonstrates this by spawning three goroutines and blocking them on different conditions. The runtime sees that all goroutines are blocked, determines the smallest value passed to time.Sleep(), and advances time exactly enough to unblock that call:

func TestBlocking(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		ch := make(chan struct{})
		events := []string{}
		wg := &sync.WaitGroup{}
		wg.Add(1)
		
		doneCh := make(chan struct{})

		go func() {
			<-ch // Blocked on channel read
			events = append(events, "channel")
			close(doneCh)
		}()
		go func() {
			wg.Wait() // Blocked on wait group
			events = append(events, "waitgroup")
			close(ch)
		}()
		go func() {
			time.Sleep(time.Second) // Blocked on time.Sleep
			events = append(events, "sleep")
			wg.Done()
		}()

		<-doneCh      // Also blocked on a channel read
		t.Log(events) // Will print ["sleep", "waitgroup", "channel"]
	})
}

Run on the Go Playground

If there are multiple sleeps, each is unblocked in order until it blocks again or exits. This not only makes the test fast, but allows the use of time for synchronization in a way that would be ill-advised otherwise.

For example, the following loop always increments the counter exactly three times. In a “normal” program, this might occasionally only run twice based on when a goroutine happened to be scheduled.

func TestLoop(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		done := make(chan struct{})
		n := 0
		wg := &sync.WaitGroup{}
		defer wg.Wait()
		wg.Go(func() {
			for {
				select {
				case <-time.After(time.Minute): // Wait for a minute
					n++
				case <-done:
					return
				}
			}
		})

		// Wait for three minutes and a millisecond
		time.Sleep((time.Minute * 3) + time.Millisecond)

		t.Log(n) // The loop always runs exactly three times and this prints "3"
		close(done)
	})
}

Run on the Go Playground

Testing our database code

Putting this together, we can cause RunDeleteExpiredSessions to run its loop by sleeping longer than its call to time.After. Since synctest then waits for the loop to block again, we can also be sure that the call to DeleteExpiredSessions has completed when the test logic picks up again.

func TestRunDeleteExpiredSessions(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		// Create a database.
		ctx, cancel := context.WithCancel(t.Context())
		db, err := NewDB(ctx, filepath.Join(t.TempDir(), "test.db"))
		if err != nil {
			t.Fatalf("Creating test database: %v", err)
		}
		defer db.Close()
		
		// Start the deletion process in an external goroutine.
		wg := sync.WaitGroup{}
		defer wg.Wait()
		wg.Go(func() {
			RunDeleteExpiredSessions(ctx, db)
		})
		defer cancel() // Cause RunDeleteExpiredSessions to exit

		// Create a session that's valid for 30 seconds.
		exp := time.Now().Add(time.Second*30)
		if err := db.CreateSession(ctx, "test-session", exp); err != nil {
			t.Fatalf("Creating session: %v", err)
		}

		// RunDeleteExpiredSessions runs every 1 minute. Block until the loop
		// runs once.
		time.Sleep(time.Minute+time.Second)
		
		// Verify that the garbage collector deleted the session.
		if _, err := db.GetSessionExpiry(ctx, "test-session"); !errors.Is(err, sql.ErrNoRows) {
			t.Errorf("GetSessionExpiry returned unexpected error on expired session: got=%s, want=%s",
				err, sql.ErrNoRows)
		}
	})
}

And this runs in a fraction of a second:

% go test -v 
=== RUN   RunDeleteExpiredSessions
--- PASS: RunDeleteExpiredSessions (0.01s)
PASS
ok  	example	0.340s

We used synctest with our system’s core leader election logic that spawns a dozen of these loops, and it ran perfectly without any changes to the code. We were able to test leader election transitions where a new instance take over from the previous one, and pick arbitrary points in the process to stop and inspect the state of the database to verify.

There are definitely nuances to synctest, such as knowing what does or doesn't block, or ensuring that all the goroutines are cleaned up properly. You can read more about this on Go’s Blog, or the package docs.