Image Registry

Image Registry
Image Registry Operator
Registry options
Access the registry
Expose the registry
1. Expose a secure registry manually

The container image registry, runs as a workload on the cluster and is managed by an infrastructure operator. It is integrated into the cluster user authentication and authorization system. Access to create and retrieve images is controlled by defining user permissions on the image resources.

The registry is typically used as...

Publication target for images built on the cluster
Source of images for workloads running on the cluster

When a new image is pushed to the registry, the cluster is notified of the new image and other components can react to and consume the updated image.

Image data is stored in a configurable storage location such as cloud storage or a filesystem volume. Image metadata, which is exposed by the standard cluster APIs and is used to perform access control, is stored as standard API resources, specifically images and imagestreams.

Image Registry Operator

The Image Registry Operator installs a single instance of the registry, and it manages all configuration of the registry.

Note that Storage is only automatically configured when we install on Amazon Web Services.

After the control plane deploys, based on configuration detected in the cluster, the Operator will create a default resource instance of...

configs.imageregistry.operator.openshift.io If insufficient information is available to define a complete resource instance, the incomplete resource will be defined and the operator will update the resource status with information about what is missing.

The Image Registry Operator runs in the openshift-image-registry namespace, and manages the registry instance in that location as well. All configuration and workload resources for the registry reside in that namespace.

Image Registry Operator configuration parameters

The configs.imageregistry.operator.openshift.io resource offers the following configuration parameters.

Parameter Description
ManagementState

Managed The Operator updates the registry as configuration resources are updated.
Unmanaged The Operator ignores changes to the configuration resources.
Removed The Operator removes the registry instance and tear down any storage that the Operator provisioned.

Log Set loglevel of the registry instance.
HTTPSecret Value needed by the registry to secure uploads, generated by default.
Proxy Define the Proxy to be used when calling master API and upstream registries.
Storage Storagetype: Details for configuring registry storage, for example S3 bucket coordinates. Normally configured by default.
Requests API Request Limit details. Controls how many parallel requests a given registry instance will handle before queuing additional requests.
DefaultRoute Determine whether or not an external route is defined using the default hostname. If enabled, the route uses re-encrypt encryption. Defaults to false.
Routes Array of additional routes to create. You provide the hostname and certificate for the route.
Replicas Replica count for the registry.

Image Registry Operator configuration resources

In addition to the configs.imageregistry.operator.openshift.io resource, additional configuration is provided to the Operator by separate ConfigMap and Secret resources located within the openshift-image-registry namespace.

Prerequisites

The CAs must be PEM-encoded.

Procedure

We can create a ConfigMap in the openshift-config namespace and use its name in AdditionalTrustedCA in the image.config.openshift.io resource to provide additional CAs that should be trusted when contacting external registries. The key is the host name of a registry with the port for which this CA is to be trusted. The image-registry-private-configuration(Secret) provides credentials needed for storage access and management. It overrides the default credentials used by the Operator, if default credentials were found.

Image registry CA example

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-registry-ca
data:
  registry.example.com: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  registry-with-port.example.com..5000: | 1
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

1 If the registry has the port, such as registry-with-port.example.com:5000. ":" should be replaced with "..."

For S3 storage the ConfigMap is expected to contain two keys:

REGISTRY_STORAGE_S3_ACCESSKEY
REGISTRY_STORAGE_S3_SECRETKEY

We can configure additional CAs with the following procedure.

To configure an additional CA:

$ oc create configmap registry-config \
    --from-file=<external_registry_address>=ca.crt \
    -n openshift-config

$ oc edit image.config.openshift.io cluster
spec:
  additionalTrustedCA:
    name: registry-config

Check the image inside the image-registry pod:

Enable the Image Registry default route

The Registry Operator controls the registry feature. The Operator is defined by the Custom Resource Definition (CRD)

configs.imageregistry.operator.openshift.io

To automatically enable the Image Registry default route, patch the Image Registry Operator CRD.

Patch the Image Registry Operator CRD:

$ oc patch configs.imageregistry.operator.openshift.io/cluster --type merge -p '{"spec":{"defaultRoute":true}}'

Configure Image Registry Storage

Configure registry storage for AWS with user-provisioned infrastructure

During installation, your cloud credentials are sufficient to create an S3 bucket and the Registry Operator will automatically configure storage.

If the Registry Operator cannot create an S3 bucket, and automatically configure storage, we can create a S3 bucket and configure storage with the following procedure:

Set up a Bucket Lifecycle Policy to abort incomplete multipart uploads that are one day old.

Fill in the storage configuration in configs.imageregistry.operator.openshift.io/cluster:

$ oc edit configs.imageregistry.operator.openshift.io/cluster

storage:
  s3:
    bucket: <bucket-name>
    region: <region-name>

To secure the registry images in AWS, block public access to the S3 bucket.

Configure registry storage for bare metal

Prerequisites

Cluster administrator permissions.
A cluster on bare metal.
A provisioned persistent volume (PV) with ReadWriteMany access mode, such as NFS.
Must have "100Gi" capacity.

Procedure

Change the spec.storage.pvc in the configs.imageregistry/cluster resource.
Verify we do not have a registry pod:

If the storage type is emptyDIR, the replica number can not be greater than 1. If the storage type is NFS, and we want to scale up the registry Pod by setting replica>1 enable the no_wdelay mount option. For example:

Check the registry configuration:
Leave the claim field blank to allow the automatic creation of an image-registry-storage PVC.
Check the clusteroperator status:

Configure registry storage for VMware vSphere

As a cluster administrator, configure the registry to use storage.

Prerequisites

Cluster administrator permissions.
A cluster on VMware vSphere.
A provisioned persistent volume (PV) with ReadWriteMany access mode, such as NFS.
Must have "100Gi" capacity.

Procedure

Change the spec.storage.pvc in the configs.imageregistry/cluster resource.
Verify we do not have a registry pod:

# cat /etc/exports
/mnt/data *(rw,sync,no_wdelay,no_root_squash,insecure,fsid=0)
sh-4.3# exportfs -rv
exporting *:/mnt/data

Check the registry configuration:
Leave the claim field blank to allow the automatic creation of an image-registry-storage PVC.
Check the clusteroperator status:

Registry options

We can deploy an image registry locally to...

Build images from source code
Deploy images
Manage image lifecycle

Integrated registry

The container image registry runs as a standard workload on the cluster, and is managed by an infrastructure operator. The image registry can be scaled up or down like any other cluster workload, does not require specific infrastructure provisioning, and is integrated into the cluster user authentication and authorization system. When a new image is pushed to the registry, the cluster is notified of the new image and other components can react to and consume the updated image.

Image data is stored in a configurable storage location such as cloud storage or a filesystem volume.
Image metadata is exposed by cluster APIs (images and imagestreams) and is used to perform access control.

OpenShift can create containers using images from third party registries. OpenShift will fetch tags from the remote registry upon imagestream creation. These registries may not offer the same image notification support as the integrated registry.

To refresh fetched tags, run...

import-image

OpenShift can push and pull images to and from private repositories using credentials supplied by the user.

Red Hat Quay registries

Red Hat Quay is available both as a hosted service and as software we can install in our data center or cloud environment. Features include...

Geo-replication
Image scanning
Ability to roll back images

To set up a hosted Quay registry account, visit Quay.io, the follow the Quay Tutorial to log in to the Quay registry and start managing the images.

We can access the Red Hat Quay registry from OpenShift like any remote container image registry.

Authentication enabled Red Hat registry

We can configure our cluster to pull images from the Red Hat Container Catalog hosted on registry.redhat.io.

The new registry uses OAuth mechanisms for authentication, with the following methods:

Authentication token	Service accounts that authenticate against the container image registry. Service accounts are not affected by changes in user accounts, so the token authentication method is reliable and resilient. This is the only supported authentication option for production clusters.
Web username and password	Standard set of credentials used to log in to resources such as access.redhat.com. While it is possible to use this authentication method with OpenShift, it is not supported for production deployments. Restrict this authentication method to stand-alone projects outside OpenShift.

We can use podman login with your credentials, either username and password or authentication token, to access content on the new registry.

All imagestreams point to the new registry. Because the registry requires authentication for access, the Samples Operator creates the samples-registry-credentials secret.

Place credentials in two places:

OpenShift namespace.
Credentials must exist in the OpenShift namespace so that the imagestreams in the OpenShift namespace can import.
Your host.
Credentials must exist on your host because Kubernetes uses the credentials from your host when it goes to pull images.

Access the registry

We can...

View logs and metrics
Secure the registry
Expose the registry

We can access the registry directly to invoke podman commands. This allows us to push images to or pull them from the integrated registry directly using operations like podman push or podman pull. To do so, log in to the registry using the oc login command. The operations we can perform depend on user permissions.

Prerequisites

Configure an identity provider (IDP).
For pulling images, for example when using the podman pull command, the user must have the registry-viewer role. To add this role:
For writing or pushing images, for example when using the podman push command, the user must have the registry-editor role. To add this role:

Access registry from the cluster

We can access the registry from inside the cluster.

Procedure

Access the registry from the cluster by using internal routes:

Access the node by getting the node's address:

$ oc get nodes
$ oc debug nodes/<node_address>

Log in to the container image registry using our access token:
You should see a message confirming login, such as:
We can pass any value for the user name; the token contains all necessary information. Passing a user name that contains colons will result in a login failure.
Since the Image Registry Operator creates the route, it will likely be similar to default-route-openshift-image-registry.<cluster_name>.

Perform podman pull and podman push operations against the registry: Important

We can pull arbitrary images, but if we have the system:registry role added, we can only push images to the registry in our project.

In the following examples, use:

Component Value

<registry_ip> 172.30.124.220
<port> 5000
<project> openshift
<image> image
<tag> omitted (defaults to latest)

Component	Value
<registry_ip>	172.30.124.220
<port>	5000
<project>	openshift
<image>	image
<tag>	omitted (defaults to latest)

Pull an arbitrary image:
Tag the new image with the form <registry_ip>:<port>/<project>/<image>. The project name must appear in this pull specification for OpenShift to correctly place and later access the image in the registry:
You must have the system:image-builder role for the specified project, which allows the user to write or push an image. Otherwise, the podman push in the next step will fail. To test, we can create a new project to push the image.

Push the newly-tagged image to the registry:

$ podman push image-registry.openshift-image-registry.svc:5000/openshift/image

View the registry's contents

As an administrator, we can view the registry's contents.

Prerequisites

Procedure

Check the pods under project openshift-image-registry:

# oc get pods
NAME READY STATUS RESTARTS AGE
cluster-image-registry-operator-764bd7f846-qqtpb 1/1 Running 0 78m
image-registry-79fb4469f6-llrln 1/1 Running 0 77m
node-ca-hjksc 1/1 Running 0 73m
node-ca-tftj6 1/1 Running 0 77m
node-ca-wb6ht 1/1 Running 0 77m
node-ca-zvt9q 1/1 Running 0 74m

View registry logs

We can view the logs for the registry using the oc logs command.

Procedure

Use the oc logs command with the DeploymentConfig to view the logs for the container image registry:

$ oc logs deployments/image-registry
2015-05-01T19:48:36.300593110Z time="2015-05-01T19:48:36Z" level=info msg="version=v2.0.0+unknown"
2015-05-01T19:48:36.303294724Z time="2015-05-01T19:48:36Z" level=info msg="redis not configured" instance.id=9ed6c43d-23ee-453f-9a4b-031fea646002
2015-05-01T19:48:36.303422845Z time="2015-05-01T19:48:36Z" level=info msg="using inmemory layerinfo cache" instance.id=9ed6c43d-23ee-453f-9a4b-031fea646002
2015-05-01T19:48:36.303433991Z time="2015-05-01T19:48:36Z" level=info msg="Using OpenShift Auth handler"
2015-05-01T19:48:36.303439084Z time="2015-05-01T19:48:36Z" level=info msg="listening on :5000" instance.id=9ed6c43d-23ee-453f-9a4b-031fea646002

Access registry metrics

The OpenShift Container Registry provides an endpoint for Prometheus metrics. Prometheus is a stand-alone, open source systems monitoring and alerting toolkit.

The metrics are exposed at the /extensions/v2/metrics path of the registry endpoint.

Procedure

There are two ways in which we can access the metrics, running a metrics query or using the cluster role.

Metrics query

Run a metrics query, for example:

$ curl -s -u <user>:<secret> \ 1
    http://172.30.30.30:5000/extensions/v2/metrics | grep openshift | head -n 10

# HELP openshift_build_info A metric with a constant '1' value labeled by major, minor, git commit & git version from which OpenShift was built.
# TYPE openshift_build_info gauge
openshift_build_info{gitCommit="67275e1",gitVersion="v3.6.0-alpha.1+67275e1-803",major="3",minor="6+"} 1
# HELP openshift_registry_request_duration_seconds Request latency summary in microseconds for each operation
# TYPE openshift_registry_request_duration_seconds summary
openshift_registry_request_duration_seconds{name="test/origin-pod",operation="blobstore.create",quantile="0.5"} 0
openshift_registry_request_duration_seconds{name="test/origin-pod",operation="blobstore.create",quantile="0.9"} 0
openshift_registry_request_duration_seconds{name="test/origin-pod",operation="blobstore.create",quantile="0.99"} 0
openshift_registry_request_duration_seconds_sum{name="test/origin-pod",operation="blobstore.create"} 0
openshift_registry_request_duration_seconds_count{name="test/origin-pod",operation="blobstore.create"} 5

1 <user> can be arbitrary, but <secret> must match the value specified in the registry configuration.

Cluster role

Create a cluster role if we do not already have one to access the metrics:

$ cat <<EOF |
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-scraper
rules:
- apiGroups:
  - image.openshift.io
  resources:
  - registry/metrics
  verbs:
  - get
EOF
$ oc create -f -

Add this role to a user...
Access the metrics using cluster role. The part of the configuration file responsible for metrics should look like this:

Additional resources

A kubeadmin can access the registry until deleted. See Remove the kubeadmin user for more information.
For more information on configuring an identity provider, see Understand identity provider configuration.

Expose the registry

By default, the registry is secured during cluster installation so that it serves traffic through TLS. Unlike previous versions of OpenShift, the registry is not exposed outside of the cluster at the time of installation.

Expose a secure registry manually

Instead of logging in to the registry from within the cluster, we can gain external access to it by exposing it with a route. This allows us to log in to the registry from outside the cluster using the route address, and to tag and push images using the route host.

Prerequisites:

The following prerequisites are automatically performed:
- Deploy the Registry Operator.
- Deploy the Ingress Operator.

Procedure

We can expose the route by using DefaultRoute parameter in the configs.imageregistry.operator.openshift.io resource or by using custom routes.

To expose the registry using DefaultRoute:

Set DefaultRoute to True:

$ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge

Log in with Podman:
1 --tls-verify=false is needed if the cluster's default certificate for routes is untrusted. We can set a custom, trusted certificate as the default certificate with the Ingress Operator.

To expose the registry using custom routes:

Create a secret with with your route's TLS keys:
This step is optional. If we do not create a secret, the route uses the default TLS configuration from the Ingress Operator.

On the Registry Operator:

spec:
  routes:
    - name: public-routes
      hostname: myregistry.mycorp.organization
      secretName: public-route-tls
...

Only set secretName if we are providing a custom TLS configuration for the registry's route.

Colophon

Image Registry

Image Registry Operator

Image Registry Operator configuration parameters

Image Registry Operator configuration resources

Enable the Image Registry default route

Configure Image Registry Storage

Configure registry storage for AWS with user-provisioned infrastructure

Configure registry storage for bare metal

Configure registry storage for VMware vSphere

Registry options

Integrated registry

Red Hat Quay registries

Authentication enabled Red Hat registry

Access the registry

Access registry from the cluster

View the registry's contents

View registry logs

Access registry metrics

Expose the registry

Expose a secure registry manually

Quick Links

Help

Site Info

Related Sites

About