~ % ./cyphar.sh_

« Four Years On, Umoci Celebrates A Long-Awaited Release

Aleksa Sarai

containers oci umoci

25 May 2025

To borrow a phrase, “hello there.”

Long time no see. I’m sure you know how the world has gone since my last blog post in 2019, but let’s just say my life definitely went in a different direction to the one I was expecting. (Though I’m sure that’s a very unoriginal observation.) I have a few more draft blog posts I’m planning to release soon, but for now let’s get down to brass tacks.

After a nearly four-year hiatus, there is a new umoci release!

Since I’ve only mentioned it in passing before, umoci is the reference implementation of the Open Container Initiative (OCI) image-spec, and provides users with the ability to create, manipulate, and otherwise interact with container images. It is designed to be as small and unopinionated as possible, so as to act as a foundation for larger systems to be built on top of.

I originally wrote umoci back in 2016 to allow SUSE to create container images natively with our existing OS image build system called Kiwi, which is in turn used by the Open Build Service (not to be confused with the other OBS) to automatically build all of openSUSE and SUSE’s base container images. At the time there was no real tool to do this^†, and so I sat down and wrote one to generate OCI images and then implemented support for converting OCI images to Docker images in a way that didn’t require a Docker daemon – something that we couldn’t require for OBS builds (not to mention it makes little sense to require Docker just to generate some tar archives).

I’m not sure that me from nearly 10 years ago (dear lord…) thought it would continue to be such a core part of our build infrastructure at SUSE, nor that it would end up becoming the reference implementation of the OCI image specification. It also bears mentioning that umoci has seen a fair amount of use outside of SUSE – the notably very-much-not-OCI-thank-you-very-much LXC and Incus container runtimes both use umoci to provide support for OCI images, and the fairly unique Stacker build system uses umoci for its own builds. The main reason that I think umoci has found such varied usage is because it is quite unopinionated and acts like a trusty multi-tool you can adapt for any situation involving container images.

†: This is actually a little bit more nuanced. There was a tool called oci-image-tool that it would be unfair to leave unmentioned, but I will talk about that a little later.

What’s New?

Despite the long wait, for most users there are only a handful of notable features and improvements added in this release (the most notable is support for zstd-compressed images – an image-spec v1.1 feature that is getting used more and more in the wild). There was also a fix for a performance regression in the last release (though it’s been so long that maybe it’s better to call it a “performance improvement” at this point). Unfortunately, full image-spec v1.1 support is not included yet, for reasons I will outline later.

The bulk of the changes were code modernisation changes (Go 1.13 errors were still somewhat new back in 2021), as well as some very large overhauls and improvements to overlayfs support (something that is used by Stacker but is definitely not widely used). You can find a more detailed list of changes in our changelog. There are still a fair number of things that I would like to get done in umoci, but a 4-year gap between releases means that a release was more than overdue.

So, why did it take so long? Well, there are quite a few reasons but the main technical one is that I had planned for image-spec v1.1 support to be included in this release but unfortunately there are a cacophony of issues that made this too big of a job that I kept putting it off. But first, we need to talk a little bit about the historical relationship between oci-image-tool and umoci.

oci-image-tool

Before umoci, there was a tool called oci-image-tool which was a collection of small helpers to do some limited operations on OCI images.

Now, I am going to talk about the shortcomings of oci-image-tool in this section, but I don’t want this to be interpreted as an attack on the folks who worked on it (myself included). There were a variety of factors that resulted in oci-image-tool ending up in the situation it is today, and I would never stoop to think that my own projects don’t have their own substantial shortcomings.

At the time, the problem I was trying to solve was that we needed a simple tool that Kiwi could use to create our OS container images directly without needing to use Docker itself. The system we were using then was based around generating a tar archive and then submitting a pull request to the docker-library/official-images repo which is used for building all of the “official library” Docker images, which were based on Dockerfiles. Our Dockerfile was just a single ADD statement to the rootfs.tar.xz archive generated by Kiwi, stored in a GitHub repo. Given that OBS was (and still is) a far superior build system for distributions, this kind of setup was quite unfortunate. We would ideally just push our images directly to a registry (or even better, host our own registry that is managed by OBS). And for that we needed a tool to generate Docker images (though with a bit of work we could generate OCI images and convert them using skopeo).

I was also a bit disappointed by the lack of an image-spec equivalent to the runtime-spec’s runc – that is, a relatively unopinionated and flexible reference implementation of the specification. Now, runc’s relationship to OCI is a little bit of a historical oddity (runc was donated to the OCI before) but I really wanted to be able to just run a few commands to download an image, unpack it, and spawn a container with runc.

It was clear from the outset that oci-image-tool would not be enough for everything we needed (the design was incomaptible with any kind of incremental build system where you progressively modify a rootfs and create new layers). But my first thought was “no matter, we could create a small wrapper around oci-image-tool‘s APIs”. And that was the first incarnation of umoci, it was a fairly minimal set of wrappers around oci-image-tool. The only real added feature was the usage of go-mtree to generate a manifest we could use to generate diff layers relatively efficiently.

And if oci-image-tool had been able to do what we needed, maybe those small extra features would’ve been folded into oci-image-tool or Kiwi and umoci would only have ended up being a small prototype that disappeared after a few months.

Unfortunately, it turns out that oci-image-tool was (in my view) basically only useful as a proof-of-concept and had several very severe bugs that made it unusable for our needs (there is a list of them in the project README). I initially sent patches to try to improve the situation but it became more and more clear that there was a deeper design issue, at which point I ended up reimplementing all of the parts of oci-image-tool we were using, with the intention of pushing the new API back into oci-image-tool. Unfortunately, it eventually seemed that there was a lack of interest in doing this (which I understand – I was basically suggesting that I replace most of the code with code that I wrote). I did become a maintainer of oci-image-tool but development behind oci-image-tool basically stalled soon after umoci was ready for use, and as a project we ended up pushing people to use umoci for image unpacking and repacking.

After being used in production for a few years, in 2020 I put forward a proposal that umoci be added to the OCI as a reference implementation, and the proposal was accepted. As part of the discussion of that proposal, the OCI Technical Oversight Board felt that oci-image-tool (which had not received any real development work for a few years at that point) should probably be retired in favour of umoci.

So, what about image-spec v1.1?

So far I have neglected to mention that there is one feature from oci-image-tool that umoci currently cannot do – validating that an image conforms to the specification.

For background, ever since I first started developing umoci, I always wanted to make sure we had strict validation of our generated images so that there would be no question that our project was producing valid images. Even today, most of the run time of our test suite is running image-spec validation against our images at every stage of our tests. And the only tool that currently exists for image-spec validation is oci-image-tool validate.

Unfortunately, the image-spec validation tools have not been actively developed for many years and cannot handle image-spec v1.1 (and the rest of the features of image-tools have been deprecated by the existence of umoci). The codebase is so old (it’s still using Godep, for the few old-timers who remember what that is…) that we have to hack around the repo in our CI in order to build it in a form that can actually validate images.

As part of the discussion to add umoci to the OCI as a reference implementation, there were talks about taking oci-image-tool validate and merging it into umoci so that umoci could do validation of images and so that the oci-image-tool repo could be archived.

However, in practice this will require rewriting most (if not all) of the validation code just for umoci’s test suite – and there is a decent argument to make that having the validation code live in the same repo as the implementation is not nearly as useful a check for bugs in our implementation. I would also want to be able to validate against multiple versions of the image-spec, which is something that oci-image-tool validate (nor the image-spec Go jsonschema APIs) support.

I’m still of two minds about what I should do. I think it’s important to have proper validation code for specifications, but at this point it seems overwhelmingly obvious that umoci is the only project actually validating their image-spec images and I don’t really want to take on even more maintainership burden for code that it seems nobody except us uses. But I also don’t want to just remove the validation we do in our tests – what if we subtly break something but in a way that umoci cannot detect? And not doing anything is what leads to 4-year gaps between releases.

A similar story is true for the runtime-spec validation, and is the reason why umoci isn’t using runtime-spec v1.2 (though this is less critical for umoci because we generate a fairly standard runtime-spec configuration).

I’m open to suggestions, but I am leaning towards just sitting down and rewriting the validation code so we can put this whole issue behind us. When will I have enough time for what (in my view) qualifies as pure busy work is a different question…

What’s Next?

As I said, there are a lot of things I would like to do with umoci, and I have recently felt more invigorated about working on it than I have for some time, but it never feels like I have enough time so knowing what to prioritise would be very useful.

If you are an umoci user, feel free to let me know what features or improvements you would prefer I focus on. While umoci is not a particularly exciting project, I’ve always felt that it was well positioned to be a dependable workhorse (and the users I know of seem to agree with that sentiment).