« umoci: a New Tool for OCI Images
29 November 2016
umoci is a free software command-line tool I’ve been working on that
allows someone to create and modify an OCI image without needing to
understand any aspect of how OCI images are structured. Currently
umoci is in
an alpha stage, with lots of features being actively developed. It will be
heading for a
0.0.0 beta release in the coming weeks.
In recent months, the OCI image and runtime
specifications have been slowly moving towards
1.0 status. However, a lot of
discussion has come up regarding users of the image
specification and how stable we can consider the specification if nobody has
really created new tooling around it or is using it in production.
To be fair, there is some work to get support for the format inside
Docker, but that hasn’t landed yet and it isn’t clear if Docker
will ever fully switch to the OCI specification. There are also plans to add
OCI support to
acbuild but I haven’t seen any actual code to that
So, I decided to stop waiting for someone else to implement what was necessary
and just decided to do it myself. In particular, I’m hoping that a lot of the
code I’ve written will be merged back into the OCI’s
codebase so that it can be maintained by the same body of people that are
writing the spec. While I am a contributor to the OCI specifications, I don’t
want to have to maintain a library for a specification that might drastically
change in the future.
The main reason why openSUSE and SUSE would be interested in this sort of
tooling is better explain in the applications section. But the
short version is that it’s related to integrating the automated building of OCI
and Docker images in our build infrastructure without needing
to run Docker inside our build infrastructure. This would be done by
umoci into KIWI and the Open Build Service.
While there has been a lot of discussion about various projects (such as
rkt and others) implementing OCI specification support, as far as
I’m aware there’s only two examples of “completed” tooling for OCI images:
skopeowhich is a tool for translating between different image formats.
skopeosupports translating to-and-from different image formats (thanks to the
containers/imagelibrary) including the Docker registry, Docker archive and OCI image formats. I’m personally very excited about
skopeobecause it means that we can use OCI-only tooling to create a Docker image without ever needing to run a Docker daemon in our build infrastructure!
And of course the official
oci-image-toolstooling by the OCI. The main issue with this tooling is that it isn’t full-featured enough to be able to fully interact with and manipulate OCI images. There are several pull requests to implement functionality that would make the
image-toolsmore useful but still would not make that tooling useful for anyone who is not intimately familiar with the OCI specification. In addition, those pull requests have been open for a long enough time that I’m convinced they won’t be merged any time soon (the discussion has stalled).
So, all in all, there isn’t really any OCI-native tooling that allows us to
modify an OCI image without understanding the internals of the spec. In
addition, because we want to add support for this tooling into KIWI, we need to
have a tool that doesn’t rely on using containers to modify the contents of an
image (which is a restriction of Docker as well as most other tooling), because
KIWI already knows how to install packages with
zypper and so on.
Now, with all of that background out of the way, what does
look like? Unlike most tools these days, the interface does not look like
git – though the concepts are fairly similar because of how the OCI
Looking at the help page, we can get an idea where to start from:
% umoci -h NAME: umoci - umoci modifies Open Container images USAGE: umoci [global options] command [command options] [arguments...] VERSION: 0.0.0~rc2 AUTHOR(S): Aleksa Sarai <email@example.com> COMMANDS: help, h Shows a list of commands or help for one command image: config modifies the image configuration of an OCI image unpack unpacks a reference into an OCI runtime bundle repack repacks an OCI runtime bundle into a reference new creates a blank tagged OCI image tag creates a new tag in an OCI image remove, rm removes a tag from an OCI image stat displays status information of an image manifest layout: gc garbage-collects an OCI image's blobs init create a new OCI layout list, ls lists the set of tags in an OCI image GLOBAL OPTIONS: --debug set log level to debug --help, -h show help --version, -v print the version
umoci doesn’t have any man pages, though I am currently working on
it, so I’ll give some more description about what the general
workflow is of
umoci and how we can use it to modify images.
Getting an Image
I would recommend getting an OCI image using
skopeo so that you can
mess around with an image that already has content. Here’s how you can pull an
skopeo (if you’re on openSUSE you can get
skopeo from this
repository maintained by us):
% # Download the latest openSUSE Leap 42.2 image from the Docker hub, convert # it to an OCI image and store the OCI image in a directory called "opensuse". # The tag of the image converted to the OCI image is "latest". % skopeo copy docker://opensuse/amd64:42.2 oci:opensuse:latest Getting image source manifest Getting image source signatures Getting image source configuration Uploading blob sha256:16724059119c810dd03218fd597625067dcf14d661c687a4396413345ced91c4 0 B / 1.73 KB [---------------------------------------------------------------] Uploading blob sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e 46.97 MB / 46.97 MB [=========================================================] Uploading manifest to image destination Storing signatures % # You can copy more than one Docker image into a single OCI image. % skopeo copy docker://opensuse/amd64:42.1 oci:opensuse:old Getting image source manifest Getting image source signatures Getting image source configuration Uploading blob sha256:f041be4a5bbe4e191ec9e323f30a7e82ec4a1781e3376a885e8e5218bce6400c 0 B / 1.73 KB [---------------------------------------------------------------] Uploading blob sha256:b5b3627caa3d91971b1d701feec88c16e621d9f28122facfe22dd6ab57476221 37.03 MB / 37.03 MB [=========================================================] Uploading manifest to image destination Storing signatures
It is also possible to create an image from scratch with
umoci new and
umoci init , but that’s not as cool as modifying a
Docker image using OCI tooling.
Now that we have an OCI image in a particular directory, we can play around
with the image using
umoci. Note that some of the modifications we do require
privileges, but that’s due to the nature of having to extract an image that has
files owned by more than one user.
Interacting with Tags
OCI images have a list of “named references”, which are the same as tags in Docker. Essentially inside an OCI image bundle, there can be any number of tagged images. An image inside an OCI image bundle is the combination of a configuration file and the layers that make up the root filesystem of the image.
umoci ls we can take a look at what we just pulled.
% umoci ls --layout opensuse latest old
The very first thing to note is the
--layout opensuse argument. Since every
OCI image is independent, we have to specify which one is the one we’re
operating on. All
umoci commands take either a
umoci -h you can see what commands fall into the respective
umoci supports local, directory-based OCI images so
--image) refers to a local directory (it’s unclear if the OCI will in
the forseeable future define any remote-registry API).
We can also create, inspect or delete tags using
umoci stat and
umoci rm respectively. A trivial example is swapping the
% umoci stat --image opensuse:latest LAYER CREATED CREATED BY SIZE COMMENT <none> 2016-12-14T00:17:31.334478162Z /bin/sh -c #(nop) MAINTAINER SUSE Containers Team <firstname.lastname@example.org> <none> sha256:d05d5d3c35088ecc4cf83c172c881cedc58df38ed127a9b0dc4adda9b291afa3 2016-12-14T00:17:44.456345827Z /bin/sh -c #(nop) ADD file:379aeb5188443ed46af74119ff2eb532b9280e72d137b1e1865ad41911c13e58 in / 49.3 MB % umoci stat --image opensuse:old LAYER CREATED CREATED BY SIZE COMMENT <none> 2016-12-14T00:13:28.318720201Z /bin/sh -c #(nop) MAINTAINER SUSE Containers Team <email@example.com> <none> sha256:8f0d2170f95bbad6858ad6432396bfebb990cd3396d841581d4b4c6b55c7d333 2016-12-14T00:13:37.494221488Z /bin/sh -c #(nop) ADD file:2f6306c949ad0e3316bf5afa7b1e9f598d88cedc31caa0abe616bfef470fade6 in / 38.85 MB % # Preform the swap. % umoci tag --image opensuse:latest tmp % umoci tag --image opensuse:old latest % umoci tag --image opensuse:tmp old % # Delete the tmp tag. % umoci rm --image opensuse:tmp % # Verify that :old is now :latest. % umoci stat --image opensuse:latest LAYER CREATED CREATED BY SIZE COMMENT <none> 2016-12-14T00:13:28.318720201Z /bin/sh -c #(nop) MAINTAINER SUSE Containers Team <firstname.lastname@example.org> <none> sha256:8f0d2170f95bbad6858ad6432396bfebb990cd3396d841581d4b4c6b55c7d333 2016-12-14T00:13:37.494221488Z /bin/sh -c #(nop) ADD file:2f6306c949ad0e3316bf5afa7b1e9f598d88cedc31caa0abe616bfef470fade6 in / 38.85 MB
As you can see,
umoci stat is in the
image category of commands and thus
--image flag. The exact meaning of
--image depends on the command,
--image is always of the format
layout:tag. If you don’t specify a
tag, the default is
latest (so I could’ve used
umoci stat --image
opensuse in the above example if I wanted to save characters).
Please note that the output of
umoci stat will change in future versions (use
--json if you want to script around
umoci stat) but the general idea will
always be the same – it gives you information about what a tag points to.
One of the most important features of
umoci is the fact that it allows us to
unpack an image into an OCI runtime bundle, which
includes an extracted root filesystem for the container as well as the OCI
runtime configuration file (which is generated from the OCI image configuration
for the extracted image). This means, if we wanted, we could go from an OCI
image to a running container with
runc in a few seconds:
% sudo umoci unpack --image opensuse:latest opensuse_bundle INFO parsed mappings map.gid= map.uid= INFO unpack manifest: unpacking layer sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e diffid="sha256:33e694f8e290bc896a8da5718854e81845fe79579f34704cf249ea457500134f" INFO unpack manifest: unpacking config config="sha256:16724059119c810dd03218fd597625067dcf14d661c687a4396413345ced91c4" % sudo runc run -b opensuse_bundle ctr sh-4.3# cat /etc/os-release NAME="openSUSE Leap" VERSION="42.2" ID=opensuse ID_LIKE="suse" VERSION_ID="42.2" PRETTY_NAME="openSUSE Leap 42.2" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:opensuse:leap:42.2" BUG_REPORT_URL="https://bugs.opensuse.org" HOME_URL="https://www.opensuse.org/" sh-4.3# exit
umoci unpack has other options related to mapping of user IDs (for
rootless and user namespaced containers). However, there’s probably enough
information in the help page (and in the man pages) that I don’t need to give
an example here.
It is true that this feature already exists within the
project (under the
oci-create-runtime-bundle command), but it will become
umoci is different in the next section. Also,
umoci unpack has
stronger guarantees about the reproducibility of extraction than
umoci has integration tests that make sure that
the same image will extract to have precisely the same root filesystem, every
time the same image is extracted).
While unpacking is all well and good, at the end of the day we want to have an
image that we can distribute (not a root filesystem that is essentially useless
to everyone else). The OCI image specification defines that a root filesystem
is made up of different diff layers, which is what we extracted
umoci unpack. But how do we create our own layers once we’ve
edited our root filesystem to our heart’s liking? The answer is, of course,
% # Starting where we left off in the previous example, we still have an # unpacked openSUSE Leap 42.2 bundle at opensuse_bundle, which we have not # touched. % # First, let's make some modifications inside a container. If the network # doesn't work for you, try removing the "network" namespace from # "linux.namespaces" and then add a bind-mount (or just copy) your host's # /etc/resolv.conf to the container. This shouldn't be necessary for most # people, but it's necessary on my machines (for some unholy reason). % sudo runc -b opensuse_bundle ctr-build sh-4.3# zypper lr # | Alias | Name | Enabled | GPG Check | Refresh --+----------------+----------------+---------+-----------+-------- 1 | non-oss | NON-OSS | Yes | ( p) Yes | Yes 2 | oss | OSS | Yes | ( p) Yes | Yes 3 | oss-update | OSS Update | Yes | ( p) Yes | Yes 4 | update-non-oss | Update Non-Oss | Yes | ( p) Yes | Yes sh-4.3# zypper rr 1 4 Removing repository 'NON-OSS' ...........................................[done] Repository 'NON-OSS' has been removed. Removing repository 'Update Non-Oss' ....................................[done] Repository 'Update Non-Oss' has been removed. sh-4.3# zypper ref Retrieving repository 'OSS' metadata ....................................[done] Building repository 'OSS' cache .........................................[done] Retrieving repository 'OSS Update' metadata .............................[done] Building repository 'OSS Update' cache ..................................[done] All repositories have been refreshed. sh-4.3# zypper in strace Loading repository data... Reading installed packages... Resolving package dependencies... The following 2 NEW packages are going to be installed: libunwind strace 2 new packages to install. Overall download size: 217.7 KiB. Already cached: 0 B. After the operation, additional 709.7 KiB will be used. Continue? [y/n/? shows all options] (y): Retrieving package libunwind-1.1-12.5.x86_64 (1/2), 47.4 KiB (137.4 KiB unpacked) Retrieving: libunwind-1.1-12.5.x86_64.rpm ...............................[done] Retrieving package strace-4.10-3.2.x86_64 (2/2), 170.3 KiB (572.3 KiB unpacked) Retrieving: strace-4.10-3.2.x86_64.rpm ..................................[done] Checking for file conflicts: ............................................[done] (1/2) Installing: libunwind-1.1-12.5.x86_64 .............................[done] (2/2) Installing: strace-4.10-3.2.x86_64 ................................[done] sh-4.3# strace -V strace -- version 4.10 sh-4.3# exit % # Now, let's edit the root filesystem manually, just for the fun of it. % sudo touch opensuse_bundle/rootfs/a_new_file % sudo rm -rf opensuse_bundle/rootfs/selinux % # The fun part! Let's repack the image into a new tag. % sudo umoci repack --image opensuse:new-latest opensuse_bundle INFO parsed mappings map.gid= map.uid= INFO created new image digest="sha256:e18f2438c89d9e6ae1e931448bcf525ba0ec4ebdc0888af836889e6ebcf70cd5" mediatype="application/vnd.oci.image.manifest.v1+json" size=586
Okay, so what just happened? Well, we first messed around with the root
filesystem of the image by creating a container (using runc, which you
can get from the official repositories if you’re on openSUSE – or from this
repo if you want the bleeding-edge version) and doing some
updates and installing
strace. Then we also went and manually modified the
root filesystem, without using a container. And finally we told
create a modified version of the
latest image (tagged
new-latest), with our
opensuse_bundle/rootfs being converted into a diff layer and added
to the set of diff layers for the new image.
You might be wondering how
umoci repack knew that the unpacked image came
opensuse:latest. The answer is that
umoci unpack stores some metadata
inside the bundle that allows it to infer what the extracted filesystem came
umoci repack creates a derivative image from a base image,
using the changes made between an
unpack and a
repack as the set of
differences to package inside the new diff layer.
umoci repack does not parse the
config.json in the extracted
bundle and thus won’t try to change the image configuration based on changes to
the runtime configuration. Use
umoci config to make changes to the image
We can now extract our new image and verify that our changes really were added
to the new image. Note that
umoci unpack now extracts one more layer than it
did in the
umoci unpack example.
% sudo umoci unpack --image opensuse:new-latest opensuse_bundle_updated INFO parsed mappings map.gid= map.uid= INFO unpack manifest: unpacking layer sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e diffid="sha256:33e694f8e290bc896a8da5718854e81845fe79579f34704cf249ea457500134f" INFO unpack manifest: unpacking layer sha256:9dab7d4b116241c5a3e30212f703a1f3ab786e05fae7555a4fbd63ab211df0ae diffid="sha256:1a7f92d610b1a4ebb11a81a6d9dd011f82598afa92be7f9f9f517c4b77708f68" INFO unpack manifest: unpacking config config="sha256:f622b0688a120097a594222961a465cefe1b05543db1507b6036873e563c24d2" % sudo runc run -b opensuse_bundle_updated ctr sh-4.3# ls -la /selinux /a_new_file ls: cannot access '/selinux': No such file or directory -rw-r--r-- 1 root root 0 Nov 23 05:07 /a_new_file sh-4.3# strace -V strace -- version 4.10 sh-4.3# zypper lr # | Alias | Name | Enabled | GPG Check | Refresh --+------------+------------+---------+-----------+-------- 1 | oss | OSS | Yes | (r ) Yes | Yes 2 | oss-update | OSS Update | Yes | (r ) Yes | Yes sh-4.3# exit
While this might seem a bit magical, the way this is implemented is described in more detail in the implementation section.
The final major component of
umoci is the ability to modify the configuration
of an image in a similar way to how
umoci repack allows us to modify the set
of diff layers of an image. The interface is also fairly similar (though the
overall interface [will almost certainly be refined before
released][umoci-ux]). The options are self-explanatory, and will look familiar
if you’ve used Dockerfiles or have looked at the output
docker inspect <some-image>.
It should be noted that changing the configuration of an image may cause the
runtime configuration generated by
umoci unpack to change. We can use the
oci-runtime-tools generate command to modify
the runtime configuration in order to avoid problems.
% umoci config --help NAME: umoci config - modifies the image configuration of an OCI image USAGE: umoci config [command options] --image <image-path>[:<tag>] [--tag <new-tag>] Where "<image-path>" is the path to the OCI image, and "<tag>" is the name of the tagged image from which the config modifications will be based (if not specified, it defaults to "latest"). "<new-tag>" is the new reference name to save the new image as, if this is not specified then umoci will replace the old image. CATEGORY: image OPTIONS: --config.user value --config.memory.limit value (default: 0) --config.memory.swap value (default: 0) --config.cpu.shares value (default: 0) --config.exposedports value --config.env value --config.entrypoint value --config.cmd value --config.volume value --config.label value --config.workingdir value --created value --author value --architecture value --os value --manifest.annotation value --clear value --tag value tag name --history.author value author value for the history entry --history.comment value comment for the history entry --history.created value created value for the history entry --history.created_by value created_by value for the history entry --image value OCI image URI of the form 'path[:tag]'
The flags are documented in a much better fashion if you look at the man pages,
but the names should sound familiar to anyone who has used Docker (or ever
looked at the output of
docker inspect). For example, here is an example of using
umoci config to change the default user and program that containers based on
this image will run:
% umoci config --image opensuse --config.user daemon:daemon --config.entrypoint="id" --config.cmd="-a" INFO created new image digest="sha256:7429ff32d9b769405620688d581d435a76608a187fe3cd3d8dcd11ed0c34379b" mediatype="application/vnd.oci.image.manifest.v1+json" size=426 % sudo umoci unpack --image opensuse opensuse_bundle_new INFO parsed mappings map.gid= map.uid= INFO unpack manifest: unpacking layer sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e diffid="sha256:33e694f8e290bc896a8da5718854e81845fe79579f34704cf249ea457500134f" INFO unpack manifest: unpacking config config="sha256:1383884f41c46931acfb180609109848556d545a501bb4190468c1b8d4649d9a" % sudo runc run -b opensuse_bundle_new ctr-build uid=2(daemon) gid=2(daemon) groups=2(daemon)
Note that if we don’t specify
umoci will overwrite the old tag.
--clear flag allows us to clear list-based configuration options.
Examples include the
Creating new Images
umoci init and
umoci new allow users to create an OCI
image from scratch. It allows us to create a brand-new OCI image layout
directory (without needing to pull a base image with
skopeo). In addition, it
lets us create an “empty image” that we can use
umoci unpack and
repack on. This can be considered something like the Docker
which contains no files or other metadata and is a blank slate for users to
craft an image from.
umoci init allows you to create a new OCI image layout (an image with no tags
or blobs inside it).
umoci new allows you to create a new tagged image (which
has no layers and has a dummy configuration) that you can then modify to your
% umoci init --layout newimage INFO created new OCI image layout path=newimage % umoci create --image newimage:new-tag INFO creating new manifest tag=new-tag INFO created new image digest="sha256:3b6ff1d61cd4d3ed25370607a48db9094c3bb1aa2de796e25fca51c08ef86a8b" mediatype="application/vnd.oci.image.manifest.v1+json" size=249 % sudo umoci unpack --image newimage:new-tag newbundle INFO parsed mappings map.gid= map.uid= INFO unpack manifest: unpacking config config="sha256:a6d0e1ce7500a4b80cfed9779255a5ce2eccc8508de10ffc1825b4217f8588e1" % ls -la newbundle/rootfs total 0 drwxr-xr-x 1 root root 0 Jan 1 1970 . drwxr-xr-x 1 root root 188 Nov 23 02:26 ..
From there, we can use
umoci repack to create a new OCI image. You could even
docker import by extracting a
tar archive to the root filesystem
directory and then using
umoci repack. The plan is that this is the first
step KIWI will use when creating an image.
And finally, for cleanliness reasons we have
umoci gc. This command is very
git gc command. It removes all unused blobs within an OCI
image. It will not modify the contents or layers inside an OCI image, and is
perfectly safe (and recommended) to be run on an image before pushing the
image to production or to a registry.
The command-line is very simple, and we can see what I mean by “unused blobs” if we delete a tag from the OCI image.
% umoci gc --layout opensuse INFO GC: garbage collected 0 blobs % umoci gc --layout opensuse INFO GC: garbage collected 0 blobs % umoci rm --image opensuse:old % umoci gc --layout opensuse INFO GC: garbage collecting blob digest="sha256:51caf3bdd5f3e395f81e5454276b2dc2e15cf7a3047cc991f14b77e158356a3b" INFO GC: garbage collecting blob digest="sha256:b5b3627caa3d91971b1d701feec88c16e621d9f28122facfe22dd6ab57476221" INFO GC: garbage collecting blob digest="sha256:f041be4a5bbe4e191ec9e323f30a7e82ec4a1781e3376a885e8e5218bce6400c" INFO GC: garbage collected 3 blobs
The reason a garbage collector is even required is because it is not safe for
umoci to start deleting blobs that we have replaced, because it is possible
for there to be multiple references to the same blob from different image tags.
To make life easy, rather than being clever about automatic garbage collection
the user just has to manually garbage collect blobs if they want to.
Converting Back to Docker
So, while all of the above is enough for OCI images, maybe you want to convert
your brand-new OCI image into a Docker image.
skopeo to the rescue again! You
can push an OCI image to a Docker registry or to a local Docker daemon with the
following commands. You need to have a skopeo version later than
0.1.17 (which you can get from
if you’re on openSUSE).
Also, note that pushing to a Docker registry requires you to have logged into
the registry with
docker login. This is not intended (and I’ve submitted a
bug report which is being fixed), but is necessary for the
% docker login Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username: cyphar Password: Login Succeeded % # Push to the Docker Hub. You can also push to a local registry by setting # some other options. This requires a patched version of skopeo in order to # be able to use an OCI image as a source in a copy. % skopeo copy oci:opensuse:new_latest docker://opensuse/amd64:latest Getting image source signatures Copying blob sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e 0 B / 46.97 MB [--------------------------------------------------------------] Copying blob sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09 0 B / 23.48 MB [--------------------------------------------------------------] Copying config sha256:a04dcb512d1d9fdd58667c4b2aaec916418f51130f9d8a9dd5b2dbed066e8c62 0 B / 1003 B [----------------------------------------------------------------] Writing manifest to image destination Storing signatures % # You can also push it to a local Docker daemon. % skopeo copy oci:opensuse:new_latest docker-daemon:opensuse/amd64:latest Getting image source signatures Copying blob sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e 17.16 MB / 46.97 MB [====================>------------------------------------] Copying blob sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09 46.97 MB / 46.97 MB [=========================================================] Copying config sha256:a04dcb512d1d9fdd58667c4b2aaec916418f51130f9d8a9dd5b2dbed066e8c62 0 B / 1003 B [----------------------------------------------------------------] Writing manifest to image destination Storing signatures 1003 B / 1003 B [=============================================================]
Unfortunately, both of these methods require you to either have a Docker daemon
or Docker registry in your build infrastructure. The other option is to just
push to the Docker Hub, but that doesn’t work as a generic build system. This
annoyed me too, so I’ve started work on a PR that will allow you
to convert an OCI image into a
docker load-friendly Docker image (without
needing a daemon). This is also quite important for my “master plan” with KIWI
integration, because we cannot run a Docker registry or Docker
daemon inside of the Open Build Service infrastructure. With that patch applied
(on top of the rest applied above), you can create
% skopeo copy oci:opensuse:new docker-archive:opensuse.tar:opensuse/amd64:42.42 Getting image source signatures Copying blob sha256:467db25190688bc9dc1d5fd6dbced1ac56f55d93a942987f448e62ad2614e46e 37.88 MB / 46.97 MB [=============================================>-----------] Copying blob sha256:eb9108dbdabef43ae694b955990153159c1cbd069c3e4ca5ac9ccdbdda0e4b09 0 B / 23.48 MB [--------------------------------------------------------------] Copying config sha256:a04dcb512d1d9fdd58667c4b2aaec916418f51130f9d8a9dd5b2dbed066e8c62 0 B / 1003 B [----------------------------------------------------------------] Writing manifest to image destination Storing signatures % docker load <opensuse.tar Loaded image: opensuse/amd64:42.42
So, hopefully soon all of this code will be merged upstream so that you can
just grab a copy of
skopeo from upstream to be able to do all of these
conversions from OCI images. I will patch the openSUSE version once these
patches get more review and are merged upstream.
There isn’t much magic within
umoci, but one of the cool features of
is the that
umoci repack knows what changes were made to the image root
filesystem after doing an
umoci unpack. The way this works is using manifests.
If you look at the extracted bundle, you can see that there’s an oddly-named
.mtree file in the bundle. And there is also an
umoci.json file as well.
% ls -l opensuse_bundle total 744 -rw-r--r-- 1 root root 24738 Dec 16 18:06 config.json drwxr-xr-x 1 root root 128 Jan 1 1970 rootfs -rw-r--r-- 1 root root 728474 Dec 16 18:06 sha256_80e063bbd4b40705b6d7d6e4d0b3f567376a9ac21ddf7aeb90ef9c7ca461c4e5.mtree -rw-r--r-- 1 root root 324 Dec 16 18:06 umoci.json
mtree is a fairly old FreeBSD utility that allows you to
generate a manifest for a directory hierarchy, so that you can verify that a
given directory matches the manifest. In particular, it means that we can see
what files have changed after the
.mtree manifest was generated. However,
mtree is a FreeBSD utility and although it has a port to
GNU/Linux, the port is not very widely packaged and is not very
Luckily though, Vincent Batts has been working on a reimplementation
mtree for GNU/Linux written in Go and organised as a Go
library. I’d also like to thank him for telling me about this project, because
it’s what inspired me to go solve this problem in the way I did. I’ve also been
go-mtree to fix bugs, make
it much more usable as a library, as well as adding some cool
features related to rootless containers.
So, when we call
umoci unpack it will use
go-mtree to figure out what files
need to be added to the diff layer of the image. You can see the changes
go-mtree will detect by using the command-line tool provided by the
project (which if you’re on openSUSE can be installed from my home
project on OBS):
% sudo gomtree -p opensuse_bundle/rootfs -f opensuse_bundle/sha256_80e063bbd4b40705b6d7d6e4d0b3f567376a9ac21ddf7aeb90ef9c7ca461c4e5.mtree "usr/bin": keyword "size": expected 5614; got 5682 "var/lib/rpm/Name": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 "var/log/zypper.log": keyword "size": expected 260934; got 549336 "var/tmp": keyword "tar_time": expected 1479529765.000000000; got 1479826841.000000000 "var/lib/rpm/Providename": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 "var/cache/zypp": keyword "size": expected 8; got 30 "var/cache/ldconfig": keyword "tar_time": expected 1479529762.000000000; got 1479826840.000000000 "tmp": keyword "tar_time": expected 1479529762.000000000; got 1479826829.000000000 "var/lib/zypp/AutoInstalled": keyword "size": expected 1400; got 1354 "var/lib/rpm/Packages": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 "lib64": keyword "tar_time": expected 1479529755.000000000; got 1479826840.000000000 "var/cache/zypp/solv": keyword "size": expected 14; got 40 "var/cache/zypp/solv/@System": keyword "tar_time": expected 1479529761.000000000; got 1479826841.000000000 "etc/ld.so.cache": keyword "size": expected 10989; got 11503 "etc/resolv.conf": keyword "size": expected 0; got 870 "var/lib/rpm/Sigmd5": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 "var/lib/zypp": keyword "size": expected 60; got 104 "var/log/zypp/history": keyword "tar_time": expected 1479529761.000000000; got 1479826841.000000000 "var/lib/rpm/Sha1header": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 "usr/lib64": keyword "size": expected 6024; got 6424 "var/lib/rpm/Requirename": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 "var/cache/zypp/solv/@System/cookie": keyword "tar_time": expected 1479529761.000000000; got 1479826841.000000000 "var/cache/zypp/solv/@System/solv.idx": keyword "size": expected 4097; got 3791 "var/lib/rpm/Basenames": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 "etc/zypp/repos.d": keyword "tar_time": expected 1447075872.000000000; got 1479826746.000000000 "run/zypp.pid": keyword "tar_time": expected 1479529761.000000000; got 1479826841.000000000 ".": keyword "size": expected 128; got 122 "root/.bash_history": keyword "size": expected 0; got 142 "var/cache/zypp/solv/@System/solv": keyword "sha256digest": expected 6f15a4f936cb17364dad0d920874415b60541cc2ea1e8fecf18a103d044f43f7; got 6aef5835e85ae4257bcd71c7d5238d97ddb8759b52019b3e1bd82e37c37f76a6 "var/lib/rpm/Installtid": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 "etc": keyword "tar_time": expected 1479529767.000000000; got 1479826840.000000000 "var/lib/rpm/Obsoletename": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 "var/lib/rpm/Dirnames": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 "var/cache/ldconfig/aux-cache": keyword "size": expected 8851; got 9196 "var/lib/rpm/Group": keyword "tar_time": expected 1479529762.000000000; got 1479826841.000000000 % echo $? 1
umoci then uses this manifest comparison output to generate the root
filesystem diff layers. Then
umoci modifies the image manifest and
configuration to include the new diff layer and creates the requested tag to
point to the new manifest.
umoci.json contains information about what
--image flag was
used to extract the image initially (so
umoci repack knows what image it has
to modify), as well as the
values. This is why
umoci repack doesn’t have any of those flags (they’re all
umoci repack and
umoci unpack would always
have be run with the same flags anyway).
All of this sounds well and good, but why bother? If we have
Docker can already create images using
docker build, why spend time
implementing an OCI-native tool? There are two main answers to this question.
First of all, it’s quite important to the container community that we have independent implementations of the Open Containers Initiative’s specifications. That way, we can “stress test” the specification and make sure that we avoid the mono-culture that you get in other specifications (such as TLS and OpenSSL).
The second reason is far more related to our requirements at SUSE. Put simply,
docker build is just not good enough. Sure, it works for most developers and
that’s fine, but if you need to distribute images that have been built
reproducibly and are automatically rebuilt if a dependency has been updated,
docker build doesn’t really help you. To be fair, this is a fairly hard
problem to solve and
umoci is a very small part of the solution (the rest of
the solution comes in the form of KIWI and the Open Build
umoci by itself is a fairly simple tool, and probably will only be
used by hardcore people like me that want to do crazy stuff like run rootless
containers on university computing clusters, there are a
lot of cool applications of
umoci on the horizon.
It should therefore be no surprise that the main integration that SUSE is
planning on doing is integrating
umoci into KIWI and the Open Build
Service. For those not familiar with these projects, I’ll give you a
KIWI is a free software Appliance building tool. It can build VM images, ISOs, raw disk images, and root filesystem archives based on a specification of what packages and other metadata should be set in the image. Currently we use the root filesystem archive functionality of KIWI to generate both openSUSE’s and SUSE’s Docker images (we have a Jenkins job that updates and builds the images from the root filesystem). The important thing to note is that the current system of building Docker images this way simply won’t work if you want to use KIWI to create different image profiles or multi-layered images.
The Open Build Service (OBS) is a free software build and distribution system used by openSUSE and SUSE for the building of packages for various operating systems (such as the openSUSE distributions, SUSE Linux Enterprise and others like Fedora, Arch Linux, etc). While mainly focused on packages, OBS also allows for the building of virtual machine images with KIWI and will publish said virtual machine images in repositories. And currently the way we provide Docker images as packages to customers is by using KIWI to create the image root filesystem, and using a fork of this project to generate an RPM that can be installed. OBS also includes dependency tracking, and will rebuild an image (or package) if one of its dependencies (which includes OS packages, and all of the dependencies of those packages) changes. This is something quite unique to images generated by KIWI inside OBS (Docker doesn’t provide for this kind of workflow).
The one thing you might’ve noticed that all of the above sounds quite hacky.
And it is. Combine that with the fact that Docker’s official library of
images comes from a repo full of
git repos and commit
IDs, and you have a fairly complicated build system with
quite a lot of moving parts. In addition, KIWI doesn’t provide us all of the
features we’d like to have in a Docker image (you can’t set Docker metadata in
KIWI because KIWI just generates the root filesystem).
umoci can solve all of that, and potentially make the build system we have
far more reproducible. First off, integrating
umoci support into KIWI means
that we get OCI image generation for free (something that KIWI could not
support otherwise). It also allows us to make our packaging of Docker images
(potentially) much simpler, with guarantees about the fact that all users will
get precisely the same image. And it could improve the state of the fracturing
opensuse/amd64 (which is a Docker repository
that we control exclusively) and
(which Docker, Inc. control).
The main concern with the fracturing between those two repositories is that
because of the
official-library‘s build system,
library/opensuse will be rebuilt at random which results in the hashes not
opensuse/amd64 (there’s also a separate issue about the source of
our images but that can also potentially be solved). Which is
quite bad. In my opinion, the best fix would be to switch to a setup where we
publish Docker images on OBS (that are signed with the official openSUSE PGP
keys), which are then just pushed to
opensuse/amd64 and then sourced in
FROM opensuse/amd64@sha256:... or similar). But
there’s a lot of work that needs to happen before we can start solving this
Overall, I’m quite excited for what the general containers community will do
umoci. Personally, I’m going to implement a
system that creates OCI images using runC. And I’m also currently implementing
rootless unpacking which should be quite useful for
Unless otherwise stated, all of the opinions in the above post are solely my own and do not necessary represent the views of anyone else. This post is released under the Creative Commons BY-SA 4.0 license.
Want to keep up to date with my posts?
You can subscribe to the Atom Feed.