+

Search Tips   |   Advanced Search

Move to Red Hat OpenShift on IBM Cloud


Automate infrastructure deployments

See also:

We can use the open source Terraform tool to automate the provisioning of IBM Cloud infrastructure, including Kubernetes clusters.

After creating a cluster, set up the autoscaler to scale worker pools up and down on worker nodes in response to the workload's.


Move existing apps or develop new apps

Containerized apps must be able to run on RHEL 7. Consider also the statefulness of the app.

We can migrate exting apps to OpenShift on IBM Cloud. To develop a new app, see guidelines for developing stateless, cloud-native apps


Administrator tasks

Cluster admins set up, operate, secure, and manage the IBM Cloud infrastructure of the cluster.

  • Size the cluster
  • Design a cluster to meet the high availability, disaster recovery, and compliance standards.
  • Secure the cluster by setting up user permissions and limiting actions within the cluster.
  • Plan and manage network communication between infrastructure components to ensure network security, segmentation, and compliance.
  • Plan persistent storage options to meet data residency and data protection requirements.


Sizing the OpenShift cluster

How many resources does my app require?

  1. Use a metrics tools to calculate workload average CPU and memory usage. For example, On Windows, view the Task Manager, or on Linux run the top command.

  2. Anticipate the number of requests that the workload must serve so that we can decide how many app replicas we want to handle the workload. For example, we might design an app instance to handle 1000 requests per minute and anticipate that the workload must serve 10000 requests per minute. If so, we might decide to make 12 app replicas, with 10 to handle the anticipated amount and an extra 2 for surge capacity.


Other resources used

  1. The worker nodes reserve certain amounts of CPU and memory resources.
  2. Consider whether the app pulls large or many images, which can take up local storage on the worker node.
  3. We can integrate services into the cluster, such as Helm or Prometheus.


Availability

  1. Plan strategy for highly available clusters
  2. Review highly available deployments


Worker nodes required for workload

  1. Estimate the max worker node capacity.

    • Single zone clusters

      Plan to have at least three worker nodes in the cluster. Further, we want one extra node's worth of CPU and memory capacity available within the cluster.

    • Multizone clusters

      Plan to have at least two worker nodes per zone, so six nodes across three zones in total. Additionally, plan for the total capacity of the cluster to be at least 150% of our total workload's required capacity, so that if one zone goes down, we have resources available to maintain the workload.

  2. Align the app size and worker node capacity with one of the available worker node flavors. To see available flavors in a zone, run ibmcloud oc flavors --zone <zone>.

    • Don't overload worker nodes

      To avoid the pods competing for CPU or running inefficiently, we must know what resources our apps require so that we can plan the number of worker nodes that we need. For example, if our apps require less resources than the resources that are available on the worker node, we can limit the number of pods that we deploy to one worker node. Keep the worker node at around 75% capacity to leave space for other pods that might need to be scheduled. If our apps require more resources than we have available on the worker node, use a different worker node flavor that can fulfill these requirements. Worker nodes are overloaded when they frequently report back a status of NotReady or evict pods due to the lack of memory or other resources.

    • Larger vs. smaller worker node flavors

      Larger nodes can be more cost efficient than smaller nodes, particularly for workloads that are designed to gain efficiency when they process on a high-performance machine. However, if a large worker node goes down, we need to be sure that the cluster has enough capacity to gracefully reschedule all the workload pods onto other worker nodes in the cluster. Smaller worker can help us scale more gracefully.

    • Replicas of the app

      To determine the number of worker nodes that we want, we can also consider how many replicas of the app that we want to run. For example, if we know that the workload requires 32 CPU cores, and we plan to run 16 replicas of the app, each replica pod needs 2 CPU cores. To run only one app pod per worker node, we can order an appropriate number of worker nodes for the cluster type to support this configuration.


Monitor resource usage and capacity

After estimating app size and worker nodes required, we can deploy the app and continuously monitor performance to maximize the compute resource usage.

  1. Create the cluster with the worker pool flavor and number of worker nodes that we previously estimated.

  2. Review what compute resources the cluster uses by default and calculate the remaining cluster capacity that we can use for the workloads.

    1. With the IBM Cloud IAM Manager service role for the cluster in all namespaces: Access the OpenShift cluster.

    2. Find the CPU and memory usage across all worker nodes. From the OpenShift clusters console, we can also click the cluster and review the Cluster Insights card.

      oc top nodes

      Example output:

      NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
10.xxx.xx.xxx   149m         7%     2402Mi          84%       
10.xxx.xx.xxx   122m         6%     2395Mi          83%       
10.xxx.xx.xxx   144m         7%     2376Mi          83%
    3. Add the CPU and memory usage amounts to the worker node resource reserves that are set for each worker node by default.

    4. Subtract the worker node reserved and default app usage amounts from the total worker node size. This amount represents the total compute resources of our worker nodes before we deploy any apps or other workloads.

  3. Deploy our apps to the cluster, and make sure to set resource requests and limits based on the app size previously estimated for our apps, to limit the amount of compute resources the apps can consume.

  4. Deploy any add-ons, plug-ins, or other cloud services that we want to use.

  5. Review what compute resources the workloads consume and calculate the remaining cluster capacity to deploy additional apps or scale existing apps.

    1. With at least the IBM Cloud IAM Reader service role for the cluster in all namespaces: Access the OpenShift cluster.

    2. List the pods that run in the cluster. 
      oc get pods --all-namespaces

    3. Get the details of a pod. Note the limits and request map of the CPU and memory. 
      oc get pod -n <project> <pod> -o=jsonpath='{range .spec.containers[*]}  {.name}:{.resources}{"\n"}{end}'

    4. Repeat the previous step for each pod in the cluster.

    5. Add up the resource requests and limits of the apps that we deployed by default.

  6. Subtract the sum of our workload resource limits that we estimated in Step 5 from the available compute resources of our worker nodes that we estimated in Step 2. The remaining amount is the extra compute resources that we have to run new workloads or to scale the existing workloads.

  7. For workloads that need to scale up and down in response to resource requests, set up the horizontal pod autoscaler and cluster worker pool autoscaler.

  8. Set up monitoring tools to continue reviewing CPU and memory usage across worker nodes in the cluster.

  9. Run performance tests to continue refining the number of worker nodes we need in the cluster, with representative latency, scalability, data set, and workload requirements.



Structuring the OpenShift environment

Your Red Hat OpenShift on IBM Cloud is linked to one IBM Cloud infrastructure portfolio only. Within your account, we can create clusters that are composed of a master and various worker nodes. IBM manages the master, and we can create a mix of worker pools that pool together individual machines of the same flavor, or memory and CPU specs. Within the cluster, we can further organize resources by using namespaces and labels. Choose the right mix of cluster, flavors, and organization strategies so that we can make sure that your teams and workloads get the resources that they need.


What type of cluster and flavors should I get?

Types of clusters:

Decide whether we want a single zone, multizone, or multiple cluster setup. Multizone clusters are available in all six worldwide IBM Cloud metro regions. Also keep in mind that worker nodes vary by zone.

Types of worker nodes

In general, intensive workloads are more suited to run on bare metal physical machines, whereas for cost-effective testing and development work, we might choose virtual machines on shared or dedicated hardware. With bare metal worker nodes, the cluster has a network speed of 10 Gbps and hyper-threaded cores that offer higher throughput. Virtual machines come with a network speed of 1 Gbps and regular cores that do not offer hyper-threading. Check out the machine isolation and flavors that are available.


Do I use multiple clusters, or just add more workers to an existing cluster?

The number of clusters that you create depends on the workload, company policies and regulations, and what we want to do with the computing resources. We can also review security information about this decision in Container isolation and security.

Multiple clusters

Set up a global load balancer and copy and apply the same configuration YAML files in each to balance workloads across the clusters. Therefore, multiple clusters are generally more complex to manage, but can help you achieve important goals such as the following.

  • Comply with security policies that require you to isolate workloads.
  • Test how the app runs in a different version of OpenShift or other cluster software such as Calico.
  • Create a cluster with the app in another region for higher performance for users in that geographical area.
  • Configure user access on the cluster-instance level instead of customizing and managing multiple RBAC policies to control access within a cluster at the namespace level.

Fewer or single clusters

Fewer clusters can help you to reduce operational effort and per-cluster costs for fixed resources. Instead of making more clusters, we can add worker pools for different flavors of computing resources available for the app and service components that we want to use. When you develop the app, the resources it uses are in the same zone, or otherwise closely connected in a multizone, so that we can make assumptions about latency, bandwidth, or correlated failures. However, it becomes even more important for you to organize the cluster by using namespaces, resource quotas, and labels.


How can I set up my resources within the cluster?

    Consider your worker node capacity.
    To get the most out of our worker node's performance, consider the following:

    • Keep up your core strength

      Each machine has a certain number of cores. Depending on the app's workload, set a limit for the number of pods per core, such as 10.

    • Avoid node overload

      Similarly, just because a node can contain more than 100 pods doesn't mean that we want it to. Depending on the app's workload, set a limit for the number of pods per node, such as 40.

    • Don't tap out the cluster bandwidth

      Keep in mind that network bandwidth on scaling virtual machines is around 1000 Mbps. For hundreds of worker nodes in a cluster, split it up into multiple clusters with fewer nodes, or order bare metal nodes.

    • Sorting out your services

      Plan out how many services that we need for the workload before we deploy. Networking and port forwarding rules are put into Iptables. If you anticipate a larger number of services, such as more than 5,000 services, split up the cluster into multiple clusters.

    Provision different types of machines for a mix of computing resources.
    Everyone likes choices, right? With Red Hat OpenShift on IBM Cloud, we have a mix of flavors that we can deploy: from bare metal for intensive workloads to virtual machines for rapid scaling. Use labels or namespaces to organize deployments to your machines. When you create a deployment, limit it so that the app's pod deploys only on machines with the right mix of resources. For example, we might want to limit a database application to a bare metal machine with a significant amount of local disk storage like the md1c.28x512.4x4tb.

    Set up multiple namespaces when we have multiple teams and projects that share the cluster.

    Namespaces are kind of like a cluster within the cluster. They are a way to divide up cluster resources by using resource quotas and default limits . When you make new namespaces, be sure to set up proper RBAC policies to control access. For more information, see Share a cluster with namespaces in the Kubernetes documentation.

    If we have a small cluster, a couple dozen users, and resources that are similar (such as different versions of the same software), you probably don't need multiple namespaces. We can use labels instead.

    Set resource quotas so that users in the cluster must use resource requests and limits
    To ensure that every team has the necessary resources to deploy services and run apps in the cluster, we must set up resource quotas for every namespace. Resource quotas determine the deployment constraints for a project, such as the number of Kubernetes resources that we can deploy, and the amount of CPU and memory that can be consumed by those resources. After you set a quota, users must include resource requests and limits in their deployments.

    Organize your Kubernetes objects with labels

    To organize and select your Kubernetes resources such as pods or nodes, use Kubernetes labels . By default, Red Hat OpenShift on IBM Cloud applies some labels, including arch, os, region, zone, and machine-type.

    Example use cases for labels include limiting network traffic to edge worker nodes and restricting the app workloads to run on worker nodes that meet certain flavor or SDS capabilities, such as bare metal worker nodes. To see what labels are already applied to a resource, use the oc get command with the --show-labels flag. For example:

    oc get node <node_ID> --show-labels

    To apply labels to worker nodes, create your worker pool with labels or update an existing worker pool.


How can I keep my cluster in a supported state?

Make sure that the cluster runs a supported OpenShift version at all times. When a new OpenShift minor version is released, an older version is shortly deprecated after and then becomes unsupported. For more information, see Updating the master and worker nodes.



Making your resources highly available

While no system is entirely failsafe, we can take steps to increase the high availability of our apps and services in Red Hat OpenShift on IBM Cloud.

Review more information about making resources highly available.