Search Tips   |   Advanced Search

Containers - OpenShift Container Platform

OpenShift Container Platform runs containerized applications. Containers use small, dedicated Linux operating systems without a kernel. Their file system, networking, cgroups, process tables, and namespaces are separate from the host Linux system.

Kubernetes is used to manage applications. OpenShift provides enhancements...

Red Hat Enterprise Linux CoreOS (RHCOS)

OpenShift uses the Red Hat Enterprise Linux CoreOS (RHCOS) operating system, which combines features from the CoreOS and Red Hat Atomic Host operating systems...

    Ignition Firstboot system configuration for bringing up nodes.
    cri-o Container engine developed in tandem with Kubernetes releases. Provides facilities for running, stopping, and restarting containers. Fully replaces the Docker Container Engine.
    Kubelet Primary node agent for Kubernetes. Responsible for launching and monitoring containers.

Use RHCOS for all control plane machines. We can use RHEL as the OS for compute, or worker, machines.

Installation and update process

We can deploy a production cluster in supported clouds by running a single command and providing a few values. We can also customize the cloud installation or install the cluster in the data center.

For clusters that use RHCOS for all machines, updating, or upgrading, OpenShift is a highly-automated process that completely controls the systems and services running on each machine, including the operating system itself, from a central control plane, upgrades are designed to become automatic events. If the cluster contains RHEL worker machines, the control plane benefits from the streamlined update process, but perform more tasks to upgrade the RHEL machines.

Other features

Operators deploy applications and software components for applications to use, replacing manual upgrades of operating systems and control plane applications. Operators such as the Cluster Version Operator and Machine Config Operator allow cluster-wide management of those components. Operator Lifecycle Manager (OLM) and the OperatorHub provide facilities for storing and distributing Operators to people developing and deploying applications.

The Red Hat Quay Container Registry is a Quay.io container registry that serves most of the container images and Operators to OpenShift clusters. Quay.io is a public registry version of Red Hat Quay that stores millions of images and tags.

OpenShift lifecycle

  1. Create an OpenShift cluster
  2. Manage the cluster
  3. Deploy applications
  4. Scale up applications

Internet and Telemetry access

The Telemetry component provides...

  • metrics about cluster health
  • success of updates
  • subscription management (including legally entitling purchase from Red Hat)

There is no disconnected subscription management. Machines must have direct internet access to install the cluster.

Internet access is required to:

Installation and update

The OpenShift 4.x Tested Integrations page contains details about integration testing for different platforms.

To pull container images and to provide telemetry data to Red Hat, all machines, including the computer running the installation process, need direct internet access. We cannot specify a proxy server for OpenShift.

The main assets generated by the installation program are the Ignition config files for the bootstrap, master, and worker machines.

The installation program uses a set of targets and dependencies. Because each target is only concerned with its own dependencies, the installation program can act to achieve multiple targets in parallel. The ultimate target is a running cluster. By meeting dependencies instead of running commands, the installation program is able to recognize and use existing components instead of running the commands to create them again.

After installation, each cluster machine uses Red Hat Enterprise Linux CoreOS (RHCOS) as the operating system. RHCOS is the immutable container host version of Red Hat Enterprise Linux (RHEL) and features a RHEL kernel with SELinux enabled by default.

RHCOS includes the kubelet, which is the Kubernetes node agent, and the CRI-O container runtime, which is optimized for Kubernetes.

Every control plane machine in a cluster must use RHCOS, which includes a critical first-boot provisioning tool called Ignition. This tool enables the cluster to configure the machines. Operating system updates are delivered as an Atomic OSTree repository embedded in a container image rolled out across the cluster by an Operator. Actual operating system changes are made in-place on each machine as an atomic operation rpm-ostree.

RHCOS is used as the operating system for all cluster machines, the cluster manages all aspects of its components and machines, including the operating system. Because of this, only the installation program and the Machine Config Operator can change machines. The installation program uses Ignition config files to set the exact state of each machine, and the Machine Config Operator completes more changes to the machines, such as the application of new certificates or keys, after installation.


Download the installation program from the OpenShift Infrastructure Providers page. This site manages:

  • REST API for accounts
  • Registry tokens, which are the pull secrets used to obtain the required components
  • Cluster registration, which associates the cluster identity to our Red Hat account to facilitate the gathering of usage metrics

The installation program is a Go binary file that performs a series of file transformations on a set of assets. The way we interact with the installation program differs depending on the installation type.

  • For clusters with installer-provisioned infrastructure, we delegate the infrastructure bootstrapping and provisioning to the installation program instead of doing it ourselves. The installation program creates all of the networking, machines, and operating systems required to support the cluster.

  • If we provision and manage the infrastructure for the cluster, provide all of the cluster infrastructure and resources, including the bootstrap machine, networking, load balancing, storage, and individual cluster machines.

We use three sets of files during installation:

It is possible to modify Kubernetes and the Ignition config files that control the underlying RHCOS operating system during installation. However, no validation is available to confirm the suitability of any modifications made to these objects. If we modify these objects, we might render the cluster non-functional. Because of this risk, modifying Kubernetes and Ignition config files is not supported unless we are following documented procedures or are instructed to do so by Red Hat support.

The installation configuration file is transformed into Kubernetes manifests, and then the manifests are wrapped into Ignition config files. The installation program uses these Ignition config files to create the cluster.

The installation configuration files are all pruned when we run the installation program, so be sure to back up all configuration files that we want to use again.

We cannot modify the parameters set during installation, but we can modify many cluster attributes after installation.

The installation process with installer-provisioned infrastructure

We can install a cluster that uses installer-provisioned infrastructure on only Amazon Web Services (AWS).

The default installation type uses installer-provisioned infrastructure on AWS. By default, the installation program acts as an installation wizard, prompting you for values that it cannot determine on its own and providing reasonable default values for the remaining parameters. We can also customize the installation process to support advanced infrastructure scenarios. The installation program provisions the underlying infrastructure for the cluster.

We can install either a standard cluster or a customized cluster. With a standard cluster, we provide minimum details required to install the cluster. With a customized cluster, we can specify more details about the platform, such as the number of machines that the control plane uses, the type of virtual machine that the cluster deploys, or the CIDR range for the Kubernetes service network.

If possible, use this feature to avoid having to provision and maintain the cluster infrastructure. In all other environments, we use the installation program to generate the assets required to provision the cluster infrastructure.

With installer-provisioned infrastructure clusters, OpenShift manages all aspects of the cluster, including the operating system itself. Each machine boots with a configuration that references resources hosted in the cluster that it joins. This configuration allows the cluster to manage itself as updates are applied.

The installation process with user-provisioned infrastructure

Use the installation program to generate the assets required to provision the cluster infrastructure, create the cluster infrastructure, and then deploy the cluster to the infrastructure that we provided.

If we do not use infrastructure that the installation program provisioned, manage and maintain the cluster resources ourselves, including:

  • The control plane and compute machines that make up the cluster
  • Load balancers
  • Cluster networking, including the DNS records and required subnets
  • Storage for the cluster infrastructure and applications

If the cluster uses user-provisioned infrastructure, we have the option of adding RHEL worker machines to the cluster.

Installation process details

Because each machine in the cluster requires information about the cluster when it is provisioned, OpenShift uses a temporary bootstrap machine during initial configuration to provide the required information to the permanent control plane. It boots by using an Ignition config file that describes how to create the cluster. The boostrap machine creates the master machines that make up the control plane. The control plane machines then create the compute, or worker, machines.

Creating bootstrap

After the cluster machines initialize, the bootstrap machine is destroyed. All clusters use the bootstrap process to initialize the cluster, but if we provision the infrastructure for the cluster, we must complete many of the steps manually.

The Ignition config files that the installation program generates contain certificates that expire after 24 hours. Complete the cluster installation and keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

Bootstrapping a cluster...

  1. The bootstrap machine boots and starts hosting the remote resources required for the master machines to boot.
  2. The master machines fetch the remote resources from the bootstrap machine and finish booting.
  3. The master machines use the bootstrap machine to form an etcd cluster.
  4. The bootstrap machine starts a temporary Kubernetes control plane using the new etcd cluster.
  5. The temporary control plane schedules the production control plane to the master machines.
  6. The temporary control plane shuts down and passes control to the production control plane.
  7. The bootstrap machine injects OpenShift components into the production control plane.
  8. The installation program shuts down the bootstrap machine.
  9. The control plane sets up the worker nodes.
  10. The control plane installs additional services in the form of a set of Operators.

The result of this bootstrapping process is a fully running OpenShift cluster. The cluster then downloads and configures remaining components needed for the day-to-day operation, including the creation of worker machines in supported environments.

Control plane

The control plane, which is composed of master machines, manages the OpenShift cluster. The control plane machines manage workloads on the compute, or worker, machines. The cluster itself manages all upgrades to the machines by the actions of the Cluster Version Operator, the Machine Config Operator, and set of individual Operators.

Machine roles

OpenShift assigns hosts different roles. These roles define the function of the machine within the cluster. The cluster contains definitions for the standard master and worker role types.

The cluster also contains the definition for the bootstrap role. Because the bootstrap machine is used only during cluster installation, its function is explained in the cluster installation documentation.

Cluster workers

In a Kubernetes cluster, the worker nodes are where the actual workloads requested by Kubernetes users run and are managed. The worker nodes advertise their capacity and the scheduler, which is part of the master services, determines on which nodes to start containers and Pods. Important services run on each worker node, including...

    Service Description
    CRI-O Container runtime. Optimized for Kubernetes.
    Kubelet Kubernetes node agent. Accepts and fulfills requests for running and stopping container workloads.
    Service proxy Manages communication for pods across workers.

MachineSets control the worker machines. Machines with the worker role drive compute workloads governed by a specific machine pool that autoscales them. Because OpenShift has the capacity to support multiple machine types, the worker machines are classed as compute machines.

In this release, the terms "worker machine" and "compute machine" are used interchangeably because the only default type of compute machine is the worker machine. In future versions of OpenShift, different types of compute machines, such as infrastructure machines, might be used by default.

Cluster masters

In a Kubernetes cluster, the master nodes run services required to control the Kubernetes cluster. In OpenShift, the master machines are the control plane and contain more than just the Kubernetes services. All of the machines with the control plane role are master machines. The terms "master" and "control plane" are used interchangeably to describe them. Instead of being grouped into a MachineSet, master machines are defined by a series of standalone machine API resources. Extra controls apply to master machines to prevent you from deleting all master machines and breaking the cluster.

Kubernetes services that run on the control plane...

    Component Description
    Kubernetes API Server Configures data for pods, services, and replication controllers. Provides a focal point for cluster's shared state.
    etcd etcd stores the persistent master state while other components watch etcd for changes to bring themselves into the specified state.
    Controller Manager Server Watches etcd for changes to objects such as replication, namespace, and serviceaccount controller objects, and then uses the API to enforce the specified state. Several such processes create a cluster with one active leader at a time.
    HAProxy services

Some of these services on the master machines run as systemd services, while others run as static Pods. Systemd services are appropriate for services we need to always come up on that particular system shortly after restart. For master machines it includes services such as:

    sshd Allows remote login
    CRI-O container engine (crio) Runs the containers. Used instead of the Docker Container Engine.
    Kubelet (kubelet) Accepts requests for managing containers on the machine from master services.

Because CRI-O and Kubelet need to be running before we can run other containers, we run them on the host as systemd services


Operators manage services on the control plane, integrating with Kubernetes APIs and CLI tools such as the kubectl and oc commands. Operators perform health checks, manage over-the-air updates, and ensure that the applications remain in a specified state.

Because CRI-O and the Kubelet run on every node, almost every other cluster function can be managed on the control plane by using Operators. Components added to the control plane by using Operators include critical networking and credential services.

OpenShift 4.1 uses different classes of Operators to perform cluster operations and run services on the cluster for applications to use.

The Cluster Version Operator manages the other Operators in a cluster.

Platform Operators

Cluster functions are divided into a series of Platform Operators, which manage a particular area of cluster functionality, such as cluster-wide application logging, management of the Kubernetes control plane, or the machine provisioning system.

Each Operator provides an API for determining cluster functionality. The Operator hides the details of managing the lifecycle of that component. Operators can manage a single component or tens of components, but the end goal is always to reduce operational burden by automating common actions. Operators also offer a more granular configuration experience. We configure each component by modifying the API that the Operator exposes instead of modifying a global configuration file.

Operators managed by Cluster Operator Lifecycle Management (OLM)

The Cluster Operator Lifecycle Management (OLM) component manages Operators available for use in applications. OLM does not manage the Operators that comprise OpenShift. OLM is a framework that manages Kubernetes-native applications as Operators. Instead of managing Kubernetes manifests, it manages Kubernetes Operators. OLM manages two classes of Operators, Red Hat Operators and certified Operators.

Some Red Hat Operators drive the cluster functions, like the scheduler and problem detectors. Others are provided to manage ourselves and use in applications, like etcd OpenShift also offers certified Operators, which the community built and maintains. These certified Operators provide an API layer to traditional applications so we can manage the application through Kubernetes constructs.

OpenShift update service

The update service provides over-the-air updates to both OpenShift and Red Hat Enterprise Linux CoreOS (RHCOS). It provides a graph, or diagram that contain vertices and the edges that connect them, of component Operators. The edges in the graph show which versions we can safely update to, and the vertices are update payloads that specify the intended state of the managed cluster components.

The Cluster Version Operator (CVO) in the cluster checks with the update service to see the valid updates and update paths based on current component versions and information in the graph. When requesting an update, the CVO uses the release image for that update to upgrade the cluster. The release artifacts are hosted in Quay as container images.

To provide only compatible updates, a release verification pipeline exists. Each release artifact is verified for compatibility with supported cloud platforms and system architectures as well as other component packages. After the pipeline confirms the suitability of a release, the update service notifies status on availability.

During continuous update mode, two controllers run. One continuously updates the payload manifests, applies them to the cluster, and outputs the status of the controlled rollout of the Operators, whether they are available, upgrading, or failed. The second controller polls the update service to determine if updates are available.

Reverting the cluster to a previous version, or a rollback, is not supported. Only upgrading to a newer version is supported.

Machine Config Operator

OpenShift integrates operating system and cluster management. Clusters manage their own updates, including updates to RHCOS on cluster nodes.

Three DaemonSets and controllers use Kubernetes-style constructs to manage nodes, orchestrating OS updates and configuration changes to the hosts.

    machine-config-controller Coordinates machine upgrades from the control plane. Monitors all of the cluster nodes and orchestrates their configuration updates.
    machine-config-daemon Runs on each node in the cluster. Updates machine to a configuration defined by MachineConfig as instructed by the MachineConfigController. When the node sees a change, it drains off its pods, applies the update, and reboots. These changes come in the form of Ignition configuration files that apply the specified machine configuration and control kubelet configuration. The update itself is delivered in a container.
    machine-config-server Provides the Ignition config files to master nodes as they join the cluster.

The machine configuration is a subset of the Ignition configuration. The machine-config-daemon reads the machine configuration to see if it needs to do an OSTree update or if it must apply a series of systemd kubelet file changes, configuration changes, or other changes to the operating system or OpenShift configuration.

When performing node management operations, we create KubeletConfig Custom Resource (CR).

Develop containerized applications

  1. Build a container and store it in a registry
  2. Create a Kubernetes manifest and save it to a Git repository
  3. Make an Operator to share application with others

Build a container

  1. Select an application to containerize.

  2. Acquire a tool for building a container, like buildah or docker.

  3. Create a file that describes what goes in the container. Typically a Dockerfile.

  4. Acquire a location to push the resulting container image (container registry).

  5. Create a containerized application and push it to the registry...

  6. Pull the application to run anywhere we want it to run.

For computers running Red Hat Enterprise Linux (RHEL), to create containerized application...

  1. Install container build tools

    RHEL contains a set of tools that includes podman, buildah, and skopeo used to build and manage containers.

  2. Create a Dockerfile to combine base image and software

    Information about building the container goes into a file named Dockerfile. In that file, we identify the base image we build from, the software packages we install, and the software we copy into the container. We also identify parameter values like network ports that we expose outside the container and volumes that we mount inside the container. Put the Dockerfile and the software we want to containerized in a directory on the RHEL system.

  3. Run buildah or docker build

    To pull your chosen base image to the local system and create a container image stored locally

      buildah build-using-dockerfile


      docker build

    We can also build container without a Dockerfile by using buildah.

  4. Tag and push to a registry

    Add a tag to the new container image that identifies the location of the registry in which we want to store and share the container. Then push that image to the registry by running the podman push or docker push command.

  5. Pull and run the image

    From any system that has a container client tool, such as podman or docker, run a command that identifies the new image. For example, run...

      podman run <image_name>


      docker run <image_name>

    Here <image_name> is the name of the new container image, which resembles...


    The registry might require credentials to push and pull images.

Container build tool options

When running containers in OpenShift, we use the CRI-O container engine. CRI-O runs on every worker and master machine in an OpenShift cluster. CRI-O is not yet supported as a standalone runtime outside of OpenShift.

Alternatives to the Docker Container Engine and docker command are available. Other container tools include podman, skopeo, and buildah. The buildah, podman, and skopeo commands are daemonless, and can be run without root privileges.

Base image options

The base image contains software that resembles a Linux system to applications. When building an image, software is placed into that file system.

Red Hat provides Red Hat Universal Base Images (UBI). These images are based on RHEL and are freely redistributable without a Red Hat subscription.

UBI images have standard, init, and minimal versions. We can also use the Red Hat Software Collections images as a foundation for applications that rely on specific runtime environments such as Node.js, Perl, or Python. Special versions of some of these runtime base images are referred to as Source-to-image (S2I) images.

S2I images are available to use directly from the OpenShift web UI by selecting...

    Catalog | Developer Catalog

Registry options

Container Registries store container images for sharing with others. Public container registries offer free accounts or a premium version that offer more storage and special features. We can install our own registry that can be exclusive to our organization or selectively shared with others.

To get Red Hat and certified partner images, draw from the Red Hat Registry...

    registry.access.redhat.com unauthenticated and deprecated
    registry.redhat.io requires authentication

The Red Hat Container Catalog lists Red Hat container images along with information about the contents and quality of those images, including health scores based on applied security updates.

Public registries include Docker Hub and Quay.io. The Quay.io registry is owned and managed by Red Hat. Many of the components used in OpenShift are stored in Quay.io, including container images and the Operators used to deploy OpenShift itself. Quay.io can store other types of content, including Helm Charts.

A private container registry is installed with OpenShift and runs on its cluster. Red Hat also offers a private version of the Quay.io registry called Red Hat Quay. Red Hat Quay includes geo replication, Git build triggers, and Clair image scanning.

Create a Kubernetes Manifest

Kubernetes Pods and services

Pods can contain one or more than one container.

We can deploy a container in a Pod, and include logging and monitoring in the Pod. When we run the Pod and need to scale up an additional instance, those other containers are scaled up with it. For namespaces, containers in a Pod share the same network interfaces, shared storage volumes, and resource limitations, such as memory and CPU, which makes it easier to manage the contents of the Pod as a single unit. Containers in a Pod can also communicate with each other by using standard inter-process communications, such as System V semaphores or POSIX shared memory.

While individual Pods represent a scalable unit in Kubernetes, a service provides a means of grouping together a set of Pods to create a complete, stable application that can complete tasks such as load balancing. A service is also more permanent than a Pod because the service remains available from the same IP address until you delete it. When the service is in use, it is requested by name and the OpenShift cluster resolves that name into the IP addresses and ports where we can reach the Pods that compose the service.

By their nature, containerized applications are separated from the operating systems where they run and, by extension, their users. Part of the Kubernetes manifest describes how to expose the application to internal and external networks by defining network policies that allow fine-grained control over communication with the containerized applications. To connect incoming requests for HTTP, HTTPS, and other services from outside the cluster to services inside the cluster, we can use an Ingress resource.

If the container requires on-disk storage instead of database storage, which might be provided through a service, we can add volumes to the manifests to make that storage available to the Pods. We can configure the manifests to create persistent volumes (PVs) or dynamically create volumes that are added to the Pod definitions.

After defining a group of Pods that compose our application, we can define those Pods in deployments and deploymentconfigs.

Application types

To determine the appropriate workload for our application, consider if the application is:

  • Meant to run to completion and be done.

    An example an application that starts up to produce a report and exits when the report is complete. The application might not run again then for a month. Suitable OpenShift objects for these types of applications include Jobs and CronJob objects.

  • Expected to run continuously.

    For long-running applications, we can write a Deployment or a DeploymentConfig.

  • Required to be highly available.

    If our application requires high availability, then we want to size our deployment to have more than one instance. A Deployment or DeploymentConfig can incorporate a ReplicaSet for that type of application. With ReplicaSets, Pods run across multiple nodes to make sure the application is always available, even if a worker goes down.

  • Need to run on every node.

    Some types of Kubernetes applications are intended to run in the cluster itself on every master or worker node. DNS and monitoring applications are examples of applications that need to run continuously on every node. We can run this type of application as a DaemonSet. We can also run a DaemonSet on a subset of nodes, based on node labels.

  • Require life-cycle management.

    To hand off our application so that others can use it, consider creating an Operator. Operators let you build in intelligence, so it can handle things like backups and upgrades automatically. Coupled with the Operator Lifecycle Manager (OLM), cluster managers can expose Operators to selected namespaces so that users in the cluster can run them.

  • Have identity or numbering requirements.

    A requirement to run exactly three instances of the application and to name the instances 0, 1, and 2. A StatefulSet is suitable for this application. StatefulSets are most useful for applications that require independent storage, such as databases and zookeeper clusters.

Available supporting components

Supporting components can include a database or logging. The required component may be available from the following Catalogs available via the web console:

OperatorHub Makes Operators available from Red Hat, certified Red Hat partners, and community members to the cluster operator. The cluster operator can make those Operators available in all or selected namespaces in the cluster, so developers can launch them and configure them with their applications.
Service Catalog Alternatives to Operators. Use to get supporting applications if we are an existing OpenShift 3 customer and are invested in Service Catalog applications, or if we have a Cloud Foundry environment from which we are interested in consuming brokers from other ecosystems.
Templates Useful for a one-off type of application, where the lifecycle of a component is not important after it is installed. A template provides a way to get started developing a Kubernetes application with minimal overhead. A template can be a list of resource definitions, which could be deployments, services, routes, or other objects. If to change names or resources, we can set these values as parameters in the template. The Template Service Broker Operator is a service broker that we can use to instantiate our own templates. We can also install templates directly from the command line.

We can configure the supporting Operators, Service Catalog applications, and templates to the specific needs of the development team and then make them available in the namespaces in which the developers work. Many people add shared templates to the openshift namespace because it is accessible from all other namespaces.

Apply the manifest

We write manifests as YAML files and deploy them by applying them to the cluster, for example, by running the oc apply command.


To automate the container development process we can create a CI pipeline that builds the images and pushes them to a registry. For example a GitOps pipeline to integrate the container development with the Git repositories used to store the software required to build our applications.

  • Write some YAML then run the oc apply command to apply that YAML to the cluster and test that it works.

  • Put the YAML container configuration file into the Git repository. From there, people who want to install or improve the app can pull down the YAML and apply it to the cluster.

  • Consider writing an Operator for our application.

Develop for Operators

Packaging and deploying applications as an Operator might be preferred if we make our application available for others to run. Operators add a lifecycle component to our application that acknowledges that the job of running an application is not complete as soon as it is installed.

When creating an application as an Operator, we can build in features for upgrading the application, backing it up, scaling it, or keeping track of its state. Maintenance tasks, like updating the Operator, can happen automatically and invisibly to the Operator's users. For example create an Operator to automatically back up data at particular times.

Any application maintenance that has traditionally been completed manually, like backing up data or rotating certificates, can be completed automatically with an Operator.

Red Hat Enterprise Linux CoreOS (RHCOS)

Red Hat Enterprise Linux CoreOS (RHCOS) is the only supported operating system for OpenShift control plane (master), machines. Compute (worker) machines can use RHEL as their operating system.

RHCOS images are downloaded to the target platform during installation. Ignition config files are used to deploy the machines.

RHCOS software is in RPM packages, and each RHCOS system starts up with a RHEL kernel and a set of services managed by the systemd init system.

Management is performed remotely from the cluster. When setting up the RHCOS machines, we can modify only a few system settings. This controlled immutability allows OpenShift to store the latest state of RHCOS systems in the cluster so it is always able to create additional machines and perform updates based on the latest RHCOS configurations.

RHCOS incorporates the CRI-O container engine instead of the Docker container engine. CRI-O offers a smaller footprint and reduced attack surface than is possible with container engines that offer a larger feature set. At the moment, CRI-O is only available as a container engine within OpenShift clusters.

RHCOS replaces the Docker CLI tool with a compatible set of container tools. The podman CLI tool supports container runtime features, such as running, starting, stopping, listing, and removing containers and container images. The skopeo CLI tool can copy, authenticate, and sign images. Use the crictl CLI tool to work with containers and pods from the CRI-O container engine. While direct use of these tools in RHCOS is discouraged, we can use them for debugging purposes.

RHCOS features transactional upgrades and rollbacks using the rpm-ostree upgrade system. Updates are delivered via container images and are part of the OpenShift update process. When deployed, the container image is pulled, extracted, and written to disk, then the bootloader is modified to boot into the new version. The machine will reboot into the update in a rolling manner to ensure cluster capacity is minimally impacted.

For RHCOS systems, the layout of the rpm-ostree file system has the following characteristics:

  • /usr is where the operating system binaries and libraries are stored and is read-only. We do not support altering this.
  • /etc, /boot, /var are writable on the system but only intended to be altered by the Machine Config Operator.
  • /var/lib/containers is the graph storage location for storing container images.

The Machine Config Operator handles operating system upgrades. Instead of upgrading individual packages, as is done with yum upgrades, rpm-ostree delivers upgrades as an atomic unit. The downloaded tree goes into effect on the next reboot. If something goes wrong with the upgrade, a single rollback and reboot returns the system to the previous state. RHCOS upgrades in OpenShift are performed during cluster updates.

Configure RHCOS

RHCOS images are set up initially with a feature called Ignition, which runs only on the system's first boot. After first boot, RHCOS systems are managed by the Machine Config Operator (MCO) that runs in the OpenShift cluster.

Because RHCOS systems in OpenShift are designed to be fully managed from the cluster, directly logging into a RHCOS machine is discouraged. Limited direct access to RHCOS machines in a cluster can be completed for debugging purposes.

Ignition utility

Ignition is the utility used by RHCOS to complete common disk tasks, including partitioning, formatting, writing files, and configuring users. On first boot, Ignition reads configuration from the installation media, or the location specified, and applies the configuration to the machines.

Ignition performs the initial configuration of the cluster machines. For each machine, Ignition takes the RHCOS image and boots the RHCOS kernel. Options on the kernel command line identify the type of deployment and the location of the Ignition-enabled initial Ram Disk (initramfs).

OpenShift uses Ignition version 2 and Ignition config version 2.3

Ignition config files

The OpenShift installation program creates the Ignition config files needed to deploy the cluster. These files are based on the information provided to the installation program directly or through an install-config.yaml file.

The way that Ignition configures machines is similar to how tools like cloud-init or Linux Anaconda kickstart configure systems, but with some important differences:

  • Ignition runs from an initial RAM disk separate from the system we are installing to.

    Because of that, Ignition can repartition disks, set up file systems, and perform other changes to the machine's permanent file system. In contrast, cloud-init runs as part of a machine's init system when the system boots, so making foundational changes to things like disk partitions cannot be done as easily. With cloud-init, it is also difficult to reconfigure the boot process while we are in the middle of the node's boot process.

  • Ignition is meant to initialize systems, not change existing systems.

    After a machine initializes and the kernel is running from the installed system, the Machine Config Operator from the OpenShift cluster completes all future machine configuration.

  • Instead of completing a defined set of actions, Ignition implements a declarative configuration. It checks that all partitions, files, services, and other items are in place before the new machine starts. It then makes the changes, like copying files to disk that are necessary for the new machine to meet the specified configuration.

  • After Ignition finishes configuring a machine, the kernel keeps running but discards the initial RAM disk and pivots to the installed system on disk. All of the new system services and other features start without requiring a system reboot.

  • Because Ignition confirms that all new machines meet the declared configuration, we cannot have a partially-configured machine.

    If a machine's setup fails, the initialization process does not finish, and Ignition does not start the new machine. The cluster will never contain partially-configured machines. If Ignition cannot complete, the machine is not added to the cluster. We must add a new machine instead. This behavior prevents the difficult case of debugging a machine when the results of a failed configuration task are not known until something that depended on it fails at a later date.

  • If there is a problem with an Ignition config that causes the setup of a machine to fail, Ignition will not try to use the same config to set up another machine. For example, a failure could result from an Ignition config made up of a parent and child config that both want to create the same file. A failure in such a case would prevent that Ignition config from being used again to set up an other machines, until the problem is resolved.

  • If we have multiple Ignition config files, we get a union of that set of configs. Because Ignition is declarative, conflicts between the configs could cause Ignition to fail to set up the machine. The order of information in those files doesn't matter. Ignition will sort and implement each setting in ways that make the most sense. For example, if a file needs a directory several levels deep, if another file needs a directory along that path, the later file is created first. Ignition sorts and creates all files, directories, and links by depth.

  • Because Ignition can start with a completely empty hard disk, it can do something cloud-init can't do: set up systems on bare metal from scratch (using features such as PXE boot). In the bare metal case, the Ignition config is injected into the boot partition so Ignition can find it and configure the system correctly.

The Ignition sequence

The Ignition process for an RHCOS machine in a OpenShift cluster involves the following steps:

  • The machine gets its Ignition config file. Master machines get their Ignition config files from the bootstrap machine, and worker machines get Ignition config files from a master.

  • Ignition creates disk partitions, file systems, directories, and links on the machine. It supports RAID arrays but does not support LVM volumes

  • Ignition mounts the root of the permanent file system to the /sysroot directory in the initramfs and starts working in that /sysroot directory.

  • Ignition configures all defined file systems and sets them up to mount appropriately at runtime.

  • Ignition runs systemd temporary files to populate required files in the /var directory.

  • Ignition runs the Ignition config files to set up users, systemd unit files, and other configuration files.

  • Ignition unmounts all components in the permanent system that were mounted in the initramfs.

  • Ignition starts up new machine's init process which, in turn, starts up all other services on the machine that run during system boot.

The machine is then ready to join the cluster and does not require a reboot.

View Ignition configuration files

To see the Ignition config file used to deploy the bootstrap machine, run:

    $ openshift-install create ignition-configs --dir $HOME/testconfig

After answering a few questions, the bootstrap.ign, master.ign, and worker.ign files appear in the directory entered.

To see the contents of the bootstrap.ign file, pipe it through the jq filter. Here's a snippet from that file:

    $ cat $HOME/testconfig/bootstrap.ign | jq
      "ignition": \\{
    "config": \\{},
      "storage": \\{
    "files": [
    "filesystem": "root",
    "path": "/etc/motd",
    "user": \\{
      "name": "root"
    "append": true,
    "contents": \\{
      "source": "data:text/plain;charset=utf-8;base64,VGhpcy...NlcnZpY2UK",

To decode the contents of a file listed in the bootstrap.ign file, pipe the base64-encoded data string representing the contents of that file to the base64 -d command. Here's an example using the contents of the /etc/motd file added to the bootstrap machine from the output shown above:

    $ echo VGhpcy...C1iIC1mIC11IGJvb3RrdWJlLnNlcnZpY2UK | base64 -d

This is the bootstrap machine; it will be destroyed when the master is fully up.

The primary service is "bootkube.service". To watch its status, run, e.g.:

    journalctl -b -f -u bootkube.service

Repeat those commands on the master.ign and worker.ign files to see the source of Ignition config files for each of those machine types. You should see a line like the following for the worker.ign, identifying how it gets its Ignition config from the bootstrap machine:

    "source": "https://api.myign.develcluster.example.com:22623/config/worker",

Here are a few things we can learn from the bootstrap.ign file:

  • Format

    The format of the file is defined in the Ignition config spec. Files of the same format are used later by the MCO to merge changes into a machine's configuration.

  • Contents

    Because the bootstrap machine serves the Ignition configs for other machines, both master and worker machine Ignition config information is stored in the bootstrap.ign, along with the bootstrap machine's configuration.

  • Size

    The file is more than 1300 lines long, with path to various types of resources.

  • The content of each file that will be copied to the machine is actually encoded into data URLs, which tends to make the content a bit clumsy to read. (Use the jq and base64 commands shown previously to make the content more readable.)

  • Configuration

    The different sections of the Ignition config file are generally meant to contain files that are just dropped into a machine's file system, rather than commands to modify existing files. For example, instead of having a section on NFS that configures that service, you would just add an NFS configuration file, which would then be started by the init process when the system comes up.

  • users

    A user named core is created, with your ssh key assigned to that user. This will allow you to log into the cluster with that user name and your credentials.

  • storage

    The storage section identifies files that are added to each machine. A few notable files include /root/.docker/config.json (which provides credentials the cluster needs to pull from container image registries) and a bunch of manifest files in /opt/openshift/manifests that are used to configure the cluster.

  • systemd

    The systemd section holds content used to create systemd unit files. Those files are used to start up services at boot time, as well as manage those services on running systems.

  • Primitives

    Ignition also exposes low-level primitives that other tools can build on.

Change Ignition Configs after installation

Machine Config Pools manage a cluster of nodes and their corresponding Machine Configs. Machine Configs contain configuration information for a cluster. To list all known Machine Config Pools:

    $ oc get machineconfigpools
    NAME   CONFIG                                  UPDATED UPDATING DEGRADED
    master master-1638c1aea398413bb918e76632f20799 False   False    False
    worker worker-2feef4f8288936489a5a832ca8efe953 False   False    False

To list all Machine Configs:

    $ oc get machineconfig
    00-master                                 4.0.0-   2.2.0             16m
    00-master-ssh                             4.0.0-                     16m
    00-worker                                 4.0.0-   2.2.0             16m
    00-worker-ssh                             4.0.0-                     16m
    01-master-kubelet                         4.0.0-   2.2.0             16m
    01-worker-kubelet                         4.0.0-   2.2.0             16m
    master-1638c1aea398413bb918e76632f20799   4.0.0-   2.2.0             16m
    worker-2feef4f8288936489a5a832ca8efe953   4.0.0-   2.2.0             16m

The Machine Config Operator acts somewhat differently than Ignition when it comes to applying these machineconfigs. The machineconfigs are read in order (from 00* to 99*). Labels inside the machineconfigs identify the type of node each is for (master or worker). If the same file appears in multiple machineconfig files, the last one wins. So, for example, any file that appears in a 99* file would replace the same file that appeared in a 00* file. The input machineconfig objects are unioned into a "rendered" machineconfig object, which will be used as a target by the operator and is the value we can see in the machineconfigpool.

To see what files are being managed from a machineconfig, look for "Path:" inside a particular machineconfig. For example:

    $ oc describe machineconfigs 01-worker-container-runtime | grep Path:
    Path:            /etc/containers/registries.conf
    Path:            /etc/containers/storage.conf
    Path:            /etc/crio/crio.conf

To change a setting in one of those files, for example to change pids_limit to 1500 (pids_limit = 1500) inside the crio.conf file, create a new machineconfig containing only the file to change.

Give the machineconfig a later name (such as 10-worker-container-runtime). The content of each file is in URL-style data. Then apply the new machineconfig to the cluster.

Quick Links


Site Info

Related Sites