~ % ./cyphar.sh_

« Generating Coverage Profiles for Golang Integration Tests

Aleksa Sarai

free software golang testing

12 April 2017

I think most developers agree that testing is quite important, and we all know that there is no single style of testing that “fits all”. In particular, many projects use a combination of unit tests and integration tests (as well as many additional tests such as conformance or end-to-end tests, but those are just large forms of integration tests at the end of the day). In different languages, the line between integration and unit tests shifts quite a bit, but in Go the line is quite clear.

To be clear, when I refer to “integration testing” I’m referring to taking a compiled Go binary and making sure it acts in a way that a user might expect. Some people call this “end-to-end testing” with integration testing referring to the testing of discrete modules rather than functions. While this might be a more accurate description, I’m more used to calling them “integration tests”.

If you want to create unit tests, use go test and the testing package available in the standard library. Because the concept of testing is built into the compiler, go test gives users many features they expect from unit testing frameworks such as coverage analysis. However, the integration testing story is much less full-featured. In the projects I maintain and contribute to, generic integration testing frameworks like bats are used to test binaries. So, how can we get some of the cool features from go test in our integration tests?

How go test Works

Before we get into the nuts and bolts of how to add coverage profiles to integration tests, it’s helpful to know what go test is actually doing. For anyone unfamiliar with how Go unit tests work, the general idea is that you call go test on a particular package and the unit tests in that package are executed. In effect go test is a modified compilation pipeline that adds all of the code that runs unit tests (and rather than executing your main.main function it runs the go test code). Unit tests are stored in source files that have names ending with _test.go (which are not compiled normally) and are filled with code that looks like this:

package thepackage

import "testing"

func TestThisIsATest(t *testing.t) {
    // Test contents here. @t has a bunch of methods that can be used to
    // control the test run and pass/fail the current test. All unit tests must
    // have a function name starting with "Test".
}

And you can get this special “test binary” if you pass -c -o <file> to go test, allowing you to run the tests separately (without recompiling the source code each time). As an aside, this is how we build our docker.test binaries for openSUSE so that we can run the integration tests on a separate system from where the source code and packages were built. Note that the test binary will only include the tests in the package that you explicitly state in the package line (imported packages won’t have their tests run implicitly).

While this is all pretty standard there doesn’t appear to be any clear benefit to using go test for integration here, since it seems to just be running a series of functions (that happen to not be main.main). However, this changes quite drastically when it comes to code coverage reports. For a full understanding of how coverage profiles are generated in Go, here’s the Go blog explaining it. The basic gist is that when you pass -cover to go test the compiler will add a bunch of instrumentation to your source code to count how often a line of code was executed. Unfortunately it’s not (easily) possible to get this cover tool to generate such instrumentation using go build. So if you want code coverage you need to use go test.

Turning main Into a Test

Since we can compile a test binary and execute it, the obvious solution to having coverage profiles for your “real binary” would be to just create a main_test.go file in your main package:

package main

import "testing"

func TestMain(t *testing.T) {
    main()
}

Simple, right? Blog post over, everyone! Well, not quite. There are quite a few problems with this naive solution that need to be handled in order for this hack to work properly. The first problem will hit you pretty quickly, as soon as you try to run your normal unit tests (assuming you’re doing the standard “recursive, run-all-tests” go test incantations):

% go test -v scm/your/project/...
[ the rest of your tests ]
=== RUN   TestProject
[ your help page ]
--- FAIL: TestProject (0.00s)
FAIL    scm/your/project/cmd/project     0.003s

This one is pretty easy to solve. You just need to do something like this, which will work for most usecases (the final version is more robust, this is just as an example).

package main

import (
    "os"
    "testing"
)

func TestMain(t *testing.T) {
    if os.Args[0] == "your-binary-name" {
        main()
    }
}

Effectively your main test will only run if argv[0] is "your-binary-name", which shouldn’t be true when you use go test. So now your normal unit tests should be unaffected.

The next issue you will probably run into is that if your program accepts Unix-style flags you’ll find that go test has its own set of flags and flag parsing code. And it definitely doesn’t like your flags. Luckily, we have a few saving graces:

• Go is kind enough to follow C and allow us to modify os.Args (it’s just a global slice, not some weird getter function that we couldn’t mess with).

• go tests flag parsing code will halt if it hits a non-flag. Unfortunately it will error our if it hits an unknown flag, but that’s easily resolved. Note that all go test flags are also prefixed as -test.

• In addition, go test does not modify os.Args when it does its flag parsing.

So in order to make go test binaries behave while also allowing our own flags to be passed un-touched to our un-modified main function, we can do the following:

func TestMain(t *testing.T) {
    var args []string
    for _, arg := range os.Args {
        if !strings.HasPrefix(arg, "-test") {
            args = append(args, arg)
        }
    }
    os.Args = args

    if os.Args[0] == "your-binary-name" {
        main()
    }
}

In order for this to run properly though, you’ll have to call your program like this:

% go test -c -o your-binary-name scm/your/project/cmd/project
% ./your-binary-name -test.v dummy-argument-to-end-parsing --your-flags
[ your program ]

Most programs wouldn’t like this style of interface (your main function would see the dummy-argument-to-end-parsing argument) and you shouldn’t have to modify your main to make it work. So to make it cleaner we can define a “flag” (though it can’t start with -) that specifies that we want to run the TestMain test.

func TestMain(t *testing.T) {
    var (
        args []string
        run  bool
    )

    for _, arg := range os.Args {
        switch {
        case arg == "__DEVEL--i-heard-you-like-tests":
            run = true
        case strings.HasPrefix(arg, "-test"):
        case strings.HasPrefix(arg, "__DEVEL"):
        default:
            args = append(args, arg)
        }
    }
    os.Args = args

    if run {
        main()
    }
}

Another issue is if your program calls os.Exit(1) on error – the go test instrumentation doesn’t write anything until after the test suite has finished executing (and os.Exit(1) kills the program before then). This means that you won’t be able to see the code coverage of cases where you’re testing for errors. So if you have something like the following in your main() function:

if err := someFunction(...); err != nil {
    log.Fprintf(os.Stderr, "%v", err)
    os.Exit(1)
}

You’ll have to adjust your program to have a single Main function that returns err and make your main look like the following:

func Main(args []string) error {
    // ...
}

func main() {
    if err := Main(os.Args); err != nil {
        log.Fprintf(os.Stderr, "%v", err)
        os.Exit(1)
    }
}

And this has the nice benefit that you can now avoid modifying os.Args and just pass the modified argument slice. In order to return a non-zero exit code you just need to use t.Fail (though unfortunately this doesn’t help if you need a specific non-zero return code). So now you end up with:

func TestMain(t *testing.T) {
    var (
        args []string
        run  bool
    )

    for _, arg := range os.Args {
        switch {
        case arg == "__DEVEL--i-heard-you-like-tests":
            run = true
        case strings.HasPrefix(arg, "-test"):
        case strings.HasPrefix(arg, "__DEVEL"):
        default:
            args = append(args, arg)
        }
    }

    if run {
        if err := Main(args); err != nil {
            // Output to stderr rather than the test log so that the
            // integration tests can properly handle cleaning up the output.
            fmt.Fprintf(os.Stderr, "%v\n", err)
            t.Fail()
        }
    }
}

This is the same function we use in umoci, and it works pretty well. However, if you want to actually profile your entire codebase and also accumulate multiple test runs you need some extra tricks.

Coverage Profiles

Now that we have a go test binary that actually works, it’s important to make sure that the cover instrumentation is added to all of the packages we care about – otherwise we’re only going to be instrumenting the main package (which isn’t very useful). Luckily there’s a flag for that that can be provided during building: -covermode=./.... You can also specify scm/your/project/... as the package list if you prefer to be explicit.

But now that you have all of this coverage instrumentation, how are you mean to make sense of it? A new coverage profile will be created for each execution of your go test binary. Unless you have some very weird development style, it’s unlikely that your program can test all of its code paths with a single invocation. So, you’ll need to collate the various coverage profiles so you get a cohesive picture of what the actual code coverage is in aggregate. Luckily this will also allow you to add the coverage profiles from your unit tests to the mix as well (giving you a full-picture view of what lines of code have been tested by some test).

First you need to specify -covermode=count when compiling your binary (this is the default but better explicit than sorry). Then for each invocation of your test binary you need to specify a unique path for the coverage profile with -test.coverprofile=path. I wrote the following awk script for umoci that will read a bunch of concatenated coverage profiles and output a “super profile” that combines everything.

# collate.awk allows you to collate a bunch of Go coverprofiles for a given
# binary (generated with -test.coverprofile), so that the statistics actually
# make sense. The input to this function is just the concatenated versions of
# the coverage reports, and the output is the combined coverage report.
#
# NOTE: This will _only_ work on coverage binaries compiles with
# -covermode=count. The other modes aren't supported.

{
    # Every coverage file in the set will start with a "mode:" header. Just make
    # sure they're all set to "count".
    if ($1 == "mode:") {
        if ($0 != "mode: count") {
            print "Invalid coverage mode", $2 > "/dev/stderr"
            exit 1
        }
        next
    }

    # The format of all other lines is as follows.
    #   <file>:<startline>.<startcol>,<endline>.<endcol> <numstmt> <count>
    # We only care about the first field and the count.
    statements[$1] = $2
    counts[$1] += $3
}

END {
    print "mode: count"
    for (block in statements) {
        print block, statements[block], counts[block]
    }
}

Conclusion

All-in-all with a few hacks and messing around with Go’s unit test builder you can create a special binary that will generate coverage profiles for normal execution of your binary. umoci has been using this for a while now, and it’s been working pretty well.

There are various tools you can use to actually understand the final collated coverage profile such as go tool cover which even allows you to generate a fancy static HTML page (with -html) that shows your codebase with text colouring indicating how many times a particular line of code was executed in your tests. Within umoci‘s extensive test suite we output a final coverage profile for umoci so we can keep track of the code coverage percentages. Hopefully you can use something similar for your own projects.

Hope you enjoyed and happy hacking!