Pushing an image index

[sjackman]

Does ORAS support pushing an image index application/vnd.oci.image.index.v1+json that contains multiple application/vnd.oci.image.manifest.v1+json?
See https://github.com/opencontainers/image-spec/blob/master/media-types.md#oci-image-media-types
and https://github.com/opencontainers/image-spec/blob/master/image-index.md

[sjackman]

• skopeo copy -a source dest works for OCI images and creates application/vnd.oci.image.index.v1+json https://github.com/containers/skopeo
• docker buildx imagetools create --tag dest src1 src2 works for OCI images and creates application/vnd.docker.distribution.manifest.list.v2+json

A solution for ORAS would be nice. Since my artifacts are tarballs, I'm just pretending they're Docker layers, which works.

[SteveLasker]

We were initially planning on adding support for oci.index, if it added a oci.index.config object to understand which type of artifact the index represented.
As index doesn't actually support blobs, as it's really an index to other manifests that have content, we're looking at other means to store content, with a collection of references.
Please see:

• OCI artifact manifest #29
• WIP generic object spec #37

These are the current artifact reference types we're hoping to evolve.

Can you help with a bit more details for what you're looking to generate to understand where it would fit in?

[sjackman]

Homebrew https://brew.sh is a package manager that supports both macOS and Linux. We have binary packages (called bottles), which are tarballs, for multiple versions of macOS and one universal Linux bottle that works on all distributions. We store each bottle in an ORAS artifact in an image manifest. We bundle these image manifests up into a single image index. We use the .manifests[].platform object, which includes architecture, variant, os, and os.version to select which bottle to download.
See https://github.com/opencontainers/image-spec/blob/master/image-index.md#image-index-property-descriptions

You can see examples of these ORAS image indexes at https://github.com/orgs/brewsci/packages and https://github.com/orgs/homebrew/packages.

I now use skopeo to upload the artifacts and either curl or oras or skopeo to download the artifacts.

You can close this issue now, but I'm happy to answer follow up questions if you think this would be an interesting use case for ORAS.

[deitch]

Homebrew stores its bottles as artifacts in an OCI registry? I didn't know that; been using it since earliest days, must be close to a decade now, switched over from, was it "macports"?. Which registry do you use, @sjackman?

@SteveLasker I think @sjackman is looking for something very similar to how container images work. One root index, with manifests property, which is a list of manifests, tagged by target platform (including OS version). Except that the actual manifests are not container images (application/vnd.oci.image.manifest.v1+json), nor is the root a container index (application/vnd.oci.image.index.v1+json), but rather they are appropriate types.

This seems like the right use case for oras.

[sjackman]

Homebrew stores its bottles as artifacts in an OCI registry?

As of this week!

Which registry do you use, @sjackman?

GitHub Packages, also known as GitHub Container Registry https://ghcr.io
See https://github.com/orgs/homebrew/packages

Except that the actual manifests are not container images (application/vnd.oci.image.manifest.v1+json), nor is the root a container index (application/vnd.oci.image.index.v1+json), but rather they are appropriate types.

We actually do use the media types application/vnd.oci.image.index.v1+json and application/vnd.oci.image.manifest.v1+json and even application/vnd.oci.image.layer.v1.tar+gzip for compatibility with oras, skopeo, even docker.

[sjackman]

For example https://github.com/orgs/Homebrew/packages/container/package/core/hello

$ oras pull ghcr.io/homebrew/core/hello:2.10
Downloaded 449de5ea35d0 hello-2.10.catalina.bottle.tar.gz
Downloaded b3b083db0807 hello-2.10.arm64_big_sur.bottle.tar.gz
Downloaded 54ac46b692fc hello-2.10.el_capitan.bottle.tar.gz
Downloaded 69489ae397e4 hello-2.10.big_sur.bottle.tar.gz
Downloaded f9d6285eafa4 hello-2.10.mojave.bottle.tar.gz
Downloaded 1b66790d4266 hello-2.10.high_sierra.bottle.tar.gz
Pulled ghcr.io/homebrew/core/hello:2.10
Digest: sha256:4f1d476de4e32f4a70a46df406fb917a0f1d990dcefd29be0a16c282a0a285d5
$ skopeo copy docker://ghcr.io/homebrew/core/hello:2.10 oci:hello:2.10
Getting image source signatures
Copying blob 449de5ea35d0 [--------------------------------------] 0.0b / 0.0b
Copying config bd012697f6 done  
Writing manifest to image destination
Storing signatures
$ docker pull --platform=darwin/amd64 ghcr.io/homebrew/core/hello:2.10
2.10: Pulling from homebrew/core/hello
operating system is not supported
$ docker pull ghcr.io/linuxbrew/core/hello:2.10
2.10: Pulling from linuxbrew/core/hello
f81d7c0a3eee: Pull complete 
Digest: sha256:b36a2cb7220504776fc540b67e9fe93947b97635d40fd98c62f27970be16d9ff
Status: Downloaded newer image for ghcr.io/linuxbrew/core/hello:2.10
ghcr.io/linuxbrew/core/hello:2.10

[deitch]

We actually do use the media types application/vnd.oci.image.index.v1+json and application/vnd.oci.image.manifest.v1+json and even application/vnd.oci.image.layer.v1.tar+gzip for compatibility with oras, skopeo, even docker

Then what is missing from oras for your support? It looks pretty solid.

[deitch]

For example https://github.com/orgs/Homebrew/packages/container/package/core/hello

I like that. I am a big advocate for using registries as a universal distribution mechanism. The combination of layers, manifests with annotations and stacking, and hashes (especially hashes), plus natively-attached config to each manifest, makes for a very powerful system.

I have my own tool for doing the same, mostly to inspect and pull down images.

[sjackman]

Then what is missing from oras for your support? It looks pretty solid.

Nothing. Using oras for download works out of the box, and that's great. I don't believe oras supports pushing a multi-architecture image index application/vnd.oci.image.index.v1+json. We're using skopeo for upload, and that solution is working just fine for us.

You can close this issue now, but I'm happy to answer follow up questions if you think this would be an interesting use case for ORAS.

[sagikazarmark]

This might be an interesting use case for pushing (and pulling) multi-arch binaries (multiple binaries for different architectures), but I'm not sure it's within the scope of oras.

[SteveLasker]

multi-arch binaries.
Now, that's an interesting angle I hadn't thought of. Since these are new artifacts, we'll need to think through whether the oci.index is the best solution, or possibly one of the oci artifact manifest proposals

• OCI artifact manifest, Phase 1-Reference Types opencontainers/artifacts#29
• WIP generic object spec opencontainers/artifacts#37

@justincormack for thoughts around multi-arch for other artifactTypes.

[sagikazarmark]

The upside of using image index is that it works with skopeo and other tools. It doesn't have to be a requirement though, if there are alternatives.

I'm not 100% sure that oras needs to be the tool that supports multi-arch artifacts as it tries to a be a generic tool as far as I can tell. But it could be, I don't know.

I think this is a topic that's worth standardizing though.

[SteveLasker]

I wouldn't say ORAS is trying to be the container image tool, as there are great tools for that. Rather, it's the non-container image tool for utilizing registries, without the constraints of the container image specific artifact type.
That said, the multi-arch pivot was the eye-opener (duhh moment).

[deitch]

Rather, it's the non-container image tool for utilizing registries, without the constraints of the container image specific artifact type

Yes, that is exactly how I look at it.