{"id":9020,"date":"2024-10-21T21:16:08","date_gmt":"2024-10-21T15:46:08","guid":{"rendered":"https:\/\/www.satup.xyz\/index.php\/2024\/10\/21\/inside-a-container-registry-the-mechanics-of-push-and-pull\/"},"modified":"2024-10-21T21:16:08","modified_gmt":"2024-10-21T15:46:08","slug":"inside-a-container-registry-the-mechanics-of-push-and-pull","status":"publish","type":"post","link":"https:\/\/www.satup.xyz\/index.php\/2024\/10\/21\/inside-a-container-registry-the-mechanics-of-push-and-pull\/","title":{"rendered":"Inside a container registry: The mechanics of push and pull"},"content":{"rendered":"


\n<\/p>\n

\n

If you have packaged your code and deployed it in a cloud or on-prem server, you must have come across containers. That\u2019s how ubiquitous containers have become today. You must have written that infamous Dockerfile, ran Docker build command to create the image and pushed it to a registry. Conversely, you must also have pulled that image (often using shell script or k8s manifests) and ran your image as a container. But have you ever thought about how the image is actually pushed or pulled from a registry? We are going to talk just about that in this blog. From here on out, \u201cclients\u201d will be used to refer to the cli tools which can pull images \u2013 like Docker, nerdctl, ctr etc. \u2013 and \u201cregistry\u201d to refer to the backend which serves these images. Also, we will take an example curlimages\/curl:8.9.1 image stored in DigitalOcean Container Registry. We will use nerdctl as the client to get a better understanding of the image internals.<\/p>\n

Use a Linux VM (Optional)<\/em><\/a><\/h3>\n

I have used an Ubuntu (24.04) AMD64 DigitalOcean Droplet for this blog. You can create such a Droplet by following the instructions<\/a> in our documentation. Using a Linux VM is optional and any platform such as Windows and MacOS can be used.<\/p>\n

If you have a DigitalOcean Managed Kubernetes cluster, you can use a debug pod on any one of the nodes as:<\/p>\n

><\/span> k debug node\/any node><\/span> -it<\/span> --image<\/span>=<\/span>zmmdv\/nerdctl:1.7.6 -- \/bin\/bash\n><\/span> alias<\/span> nerdctl<\/span>=<\/span>\"nerdctl -a \/host\/run\/containerd\/containerd.sock\"<\/span>\n<\/span><\/code><\/pre>\n
Also, you would have to use \/host\/var\/lib\/containerd<\/code> in upcoming sections, if you choose this path, as the node\u2019s root filesystem is mounted in \/host<\/code> path of the pod.<\/p>\n
Install Docker\/containerd and nerdctl<\/a><\/h3>\n
We can install Docker\/containerd by performing the steps<\/a> from Docker\u2019s official install page. For our blog, we can get by with installing just containerd since we will be using nerdctl as the image client. We can do so by adding the Docker\u2019s apt repository<\/a> and do sudo apt-get install containerd.io<\/code>.<\/p>\n
Nerdctl can be installed from its official GitHub release page. I have used version 1.7.6<\/a>. For this blog, the minimal nerdctl archive should be sufficient, but if you want to run containers, you will require the full nerdctl archive.<\/p>\n
Using DigitalOcean Container Registry<\/a><\/h3>\n
You can create a container registry in DigitalOcean by following the steps<\/a> or if you already have one you can follow the steps here<\/a> to push\/pull an image.<\/p>\n
It\u2019s important to talk about what an image consists of before finding out how it\u2019s transferred between the client and registry. First let\u2019s try to pull an image using nerdctl:<\/p>\n
><\/span> nerdctl image pull [<\/span>registry.digitalocean.com\/coolreg\/curlimages\/curl:8.9.1]<\/span>(<\/span>http:\/\/registry.digitalocean.com\/coolreg\/curlimages\/curl:8.9.1)<\/span>\n\nregistry.digitalocean.com\/coolreg\/curlimages\/curl:8.9.1:\nindex-sha256:4d3d08d1019a4b4507f18f5700f13dd7e106ed8214229b878417805094f21376:\ndone<\/span>\nmanifest-sha256:d795b5d334f78dc8dbe55ba4332213a937b86ca193f4091e60963517f32340c4:done\nconfig-sha256:5f48a11a4b51dd9f8eddd97396069a65d9c8bd1ba2dbc4dffe98954a5078ad51:   done<\/span>\nlayer-sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3:    done<\/span>\nlayer-sha256:c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6:    done<\/span>\nlayer-sha256:4007b551d63a39f6c3235cb5af643f633cf6e0bf5a161465b074eaf60ab43f44:    done<\/span> \n<\/code><\/pre>\n
Container images consist of index, manifest, configs and layers as can be seen from the output of above command. These are described in the next sections in a top down manner.<\/p>\n
Index<\/a><\/h3>\n
The index is the first component which is fetched from the registry when an image is pulled. It is a JSON file outlining the manifest digests for each platform in the format {os}\/{arch} such as linux\/amd64, linux\/arm64, and linux\/386 etc. For each of the platforms, the image contents would be different and this JSON file describes which manifest belongs to which platform. All the image contents are stored in \/var\/lib\/containerd\/io.containerd.content.v1.content\/blobs<\/code> path (for Docker it is different). We can examine the index JSON file as:<\/p>\n
\n><\/span> cat<\/span> \/var\/lib\/containerd\/io.containerd.content.v1.content\/blobs\/sha256\/4d3d08d1019a4b4507f18f5700f13dd7e106ed8214229b878417805094f21376 |<\/span> jq '.manifests'<\/span>\n\n[<\/span>\n  {<\/span>\n    \"mediaType\"<\/span>:<\/span> \"application\/vnd.oci.image.manifest.v1+json\"<\/span>,\n    \"digest\"<\/span>:<\/span> \"sha256:d795b5d334f78dc8dbe55ba4332213a937b86ca193f4091e60963517f32340c4\"<\/span>,\n    \"size\"<\/span>:<\/span> 860<\/span>,\n    \"platform\"<\/span>:<\/span> {<\/span>\n      \"architecture\"<\/span>:<\/span> \"amd64\"<\/span>,\n      \"os\"<\/span>:<\/span> \"linux\"<\/span>\n    }<\/span>\n  }<\/span>,\n  {<\/span>\n    \"mediaType\"<\/span>:<\/span> \"application\/vnd.oci.image.manifest.v1+json\"<\/span>,\n    \"digest\"<\/span>:<\/span> \"sha256:7d3aecfe1d39dda61edefcdcead38a4eaa7c292734df00e9381504c0d699cab1\"<\/span>,\n    \"size\"<\/span>:<\/span> 860<\/span>,\n    \"platform\"<\/span>:<\/span> {<\/span>\n      \"architecture\"<\/span>:<\/span> \"arm64\"<\/span>,\n      \"os\"<\/span>:<\/span> \"linux\"<\/span>\n    }<\/span>\n  }<\/span>\n]<\/span>\n<\/code><\/pre>\n
I am using a Droplet in DigitalOcean which is linux\/amd64, so the manifest to pull next is sha256:d795b5d334f78dc8dbe55ba4332213a937b86ca193f4091e60963517f32340c4<\/code> which can be seen from the output of the nerdctl pull command before.<\/p>\n
Having an index is optional and is applicable only for multi-platform images. It is not used while uploading images for only one platform. In that case, only manifest is used (discussed in next section).<\/p>\n
Manifests<\/a><\/h3>\n
Manifests are also JSON files which specify the layers (described next) and config comprising the image. Our manifest contains below contents:<\/p>\n
><\/span> cat<\/span> \/var\/lib\/containerd\/io.containerd.content.v1.content\/blobs\/sha256\/d795b5d334f78dc8dbe55ba4332213a937b86ca193f4091e60963517f32340c4 |<\/span> jq .<\/span>\n\n{<\/span>\n  \"schemaVersion\"<\/span>:<\/span> 2<\/span>,\n  \"mediaType\"<\/span>:<\/span> \"application\/vnd.oci.image.manifest.v1+json\"<\/span>,\n  \"config\"<\/span>:<\/span> {<\/span>\n    \"mediaType\"<\/span>:<\/span> \"application\/vnd.oci.image.config.v1+json\"<\/span>,\n    \"digest\"<\/span>:<\/span> \"sha256:5f48a11a4b51dd9f8eddd97396069a65d9c8bd1ba2dbc4dffe98954a5078ad51\"<\/span>,\n    \"size\"<\/span>:<\/span> 1486<\/span>\n  }<\/span>,\n  \"layers\"<\/span>:<\/span> [<\/span>\n    {<\/span>\n      \"mediaType\"<\/span>:<\/span> \"application\/vnd.oci.image.layer.v1.tar+gzip\"<\/span>,\n      \"digest\"<\/span>:<\/span> \"sha256:c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6\"<\/span>,\n      \"size\"<\/span>:<\/span> 3622892<\/span>\n    }<\/span>,\n    {<\/span>\n      \"mediaType\"<\/span>:<\/span> \"application\/vnd.oci.image.layer.v1.tar+gzip\"<\/span>,\n      \"digest\"<\/span>:<\/span> \"sha256:4007b551d63a39f6c3235cb5af643f633cf6e0bf5a161465b074eaf60ab43f44\"<\/span>,\n      \"size\"<\/span>:<\/span> 5758798<\/span>\n    }<\/span>,\n    {<\/span>\n      \"mediaType\"<\/span>:<\/span> \"application\/vnd.oci.image.layer.v1.tar+gzip\"<\/span>,\n      \"digest\"<\/span>:<\/span> \"sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3\"<\/span>,\n      \"size\"<\/span>:<\/span> 42<\/span>\n    }<\/span>\n  ]<\/span>\n}<\/span>\n<\/code><\/pre>\n
Config<\/a><\/h3>\n
The config is a JSON file describing how to run the image when we use commands like nerdctl run (or docker run). The CMD, ENTRYPOINT, ENV, USER<\/code> etc directives in a Dockerfile makes its way to this config JSON file:<\/p>\n
><\/span>cat \/var\/lib\/containerd\/io.containerd.content.v1.content\/blobs\/sha256\/5f48a11a4b51dd9f8eddd97396069a65d9c8bd1ba2dbc4dffe98954a5078ad51 |<\/span> jq '{config: .config, rootfs: .rootfs}'<\/span>\n\n{<\/span>\n  \"config\"<\/span>:<\/span> {<\/span>\n    \"User\"<\/span>:<\/span> \"curl_user\"<\/span>,\n    \"Env\"<\/span>:<\/span> [<\/span>\n      \"PATH=\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin\"<\/span>,\n      \"CURL_CA_BUNDLE=\/cacert.pem\"<\/span>\n    ]<\/span>,\n    \"Entrypoint\"<\/span>:<\/span> [<\/span>\n      \"\/entrypoint.sh\"<\/span>\n    ]<\/span>,\n    \"Cmd\"<\/span>:<\/span> [<\/span>\n      \"curl\"<\/span>\n    ]<\/span>,\n    \"WorkingDir\"<\/span>:<\/span> \"\/home\/curl_user\"<\/span>,\n    \"Labels\"<\/span>:<\/span> {<\/span>\n      ..<\/span>.\n    }<\/span>,\n    \"OnBuild\"<\/span>:<\/span> null\n  }<\/span>,\n  \"rootfs\"<\/span>:<\/span> {<\/span>\n    \"type\"<\/span>:<\/span> \"layers\"<\/span>,\n    \"diff_ids\"<\/span>:<\/span> [<\/span>\n     \"sha256:78561cef0761903dd2f7d09856150a6d4fb48967a8f113f3e33d79effbf59a07\"<\/span>,\n     \"sha256:b138805d27824c78d3d19b77cf24caf06839c64f54f70439f84c96160fb4d928\"<\/span>,\n     \"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef\"<\/span>\n    ]<\/span>\n  }<\/span>\n}<\/span>\n<\/code><\/pre>\n
Here, the layer digests are also somewhat different from what we saw in our manifest before and what the nerdctl client actually pulled. These are actually the uncompressed digest of the layers.<\/p>\n
Layers<\/a><\/h3>\n
A layer is a filesystem (or a directory structure in simple terms) containing the binaries and the required dependencies of the image such as system libs etc. in a compressed format (usually tar,gzip). An image might have one or many such layers. All of these layers are combined by the container runtime to create the final filesystem for the runnable container. The layers are referenced by their digests.<\/p>\n
Our example image curlimages\/curl:8.9.1 contains 3 layers: c6a83fedfae6<\/code>, 4007b551d63a<\/code> and 4ca545ee6d5d<\/code>. We said these layers are actually compressed filesystems. Let\u2019s find out:<\/p>\n
><\/span> rm<\/span> -rf<\/span> test<\/span> &&<\/span> mkdir<\/span> test<\/span>\n><\/span> tar<\/span> zxf \/var\/lib\/containerd\/io.containerd.content.v1.content\/blobs\/sha256\/c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6 -C<\/span> .\/test\n><\/span> ls<\/span> .\/test\nbin  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var\n<\/code><\/pre>\n
We can see that the layer c6a83fedfae6<\/code> is a compressed generic filesystem of a linux distro (alpine in our case).<\/p>\n
Bonus Content:<\/h4>\n
Layers are the compressed digests. To get the uncompressed digests we can use ctr content ls<\/code>  and check the containerd.io\/uncompressed<\/code> label for the layer:<\/p>\n
><\/span> ctr content ls<\/span> |<\/span> awk<\/span> '$1 ~ \/c6a83fedfae6|4007b551d63a|4ca545ee6d5d\/ { print }'<\/span> |<\/span> sed<\/span> \"s\/^.*containerd.io\\\/uncompressed=\/\/\"<\/span>\n\nsha256:b138805d27824c78d3d19b77cf24caf06839c64f54f70439f84c96160fb4d928\nsha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef\nsha256:78561cef0761903dd2f7d09856150a6d4fb48967a8f113f3e33d79effbf59a07\n<\/code><\/pre>\n
This matches with that of rootfs<\/code> entry in the config JSON file.<\/p>\n
Pulling an image consists of fetching the tag index digest, index itself, platform manifest, config and the layers in this order. But in the case of a private registry, where does the client get the authorization information (in other words, a token) from? There is a specific sequence to get each of the components of an image mentioned before, and it goes like this, when taking the example of fetching index digest for a tag:<\/p>\n
\"DOCR<\/p>\n
    \n
  1. Client tries to get the manifest digest without any authorization token by doing a HEAD request. The url is something like: https:\/\/registry.digitalocean.com\/v2\/coolreg\/curlimages\/curl\/manifests\/8.9.1<\/code><\/li>\n<\/ol>\n