+

Search Tips   |   Advanced Search

Install OpenShift 4.1

1. Install on Amazon Web Services
   1. Configure Route 53
   2. AWS account limits
   3. Create an IAM user
   4. Required AWS permissions
   5. Supported AWS regions
   6. Install a cluster on AWS using default configuration options
   7. Install a cluster on AWS with customizations
   8. Install a cluster on AWS with network customizations
   9. Install on user-provisioned AWS
      1. Install using CloudFormation templates
      2. Internet and Telemetry access
      3. Required AWS infrastructure components
         • Cluster machines
         • Other infrastructure components
         • Required AWS permissions
      4. Obtain the installation program
      5. Generate an SSH private key and add it to the agent
      6. Create the installation files for AWS
      7. Extract the infrastructure name
      8. Create a VPC in AWS
         • CloudFormation template for the VPC
         • Create networking and load balancing components in AWS
         • CloudFormation template for the network and load balancers
      9. Create security group and roles in AWS
         • CloudFormation template for security objects
      10. RHCOS AMIs for the AWS infrastructure
      11. Create the bootstrap node in AWS
          • CloudFormation template for the bootstrap machine
      12. Create the control plane machines in AWS
          • CloudFormation template for control plane machines
      13. Initialize the bootstrap node on AWS with user-provisioned infrastructure
          • Create the worker nodes in AWS
          • CloudFormation template for worker machines
      14. Install the OpenShift Command-line Interface
      15. Log on to the cluster
      16. Approve the CSRs for the machines
      17. Initial Operator configuration
          • Image registry storage configuration
          • Configure registry storage for AWS with user-provisioned infrastructure
          • Configure storage for the image registry in non-production clusters
      18. Complete an AWS installation on user-provisioned infrastructure
2. Install on bare metal
   1. Internet and Telemetry access
   2. Machine requirements for a cluster with user-provisioned infrastructure
      • Required machines
      • Network connectivity requirements
      • Minimum resource requirements
      • Certificate signing requests management
      • Networking requirements for user-provisioned infrastructure
      • User-provisioned DNS requirements
   3. Generate an SSH private key and add it to the agent
   4. Obtain the installation program
   5. Install the OpenShift Command-line Interface
   6. Manually create the installation configuration file
      • Sample install-config.yaml file for bare metal
   7. Create the Ignition config files
   8. Create RHCOS machines
      • Create RHCOS machines using an ISO image
      • Create RHCOS machines by PXE or iPXE booting
   9. Create the cluster
   10. Log on to the cluster
   11. Approve the CSRs for the machines
   12. Initial Operator configuration
       • Image registry storage configuration
       • Configure registry storage for bare metal
       • Configure storage for the image registry in non-production clusters
   13. Complete installation on user-provisioned infrastructure
3. Install on vSphere
   1. Internet and Telemetry access
   2. VMware vSphere infrastructure requirements
   3. Machine requirements for a cluster with user-provisioned infrastructure
      • Required machines
      • Network connectivity requirements
      • Minimum resource requirements
      • Certificate signing requests management
      • Create the user-provisioned infrastructure
      • Networking requirements for user-provisioned infrastructure
      • User-provisioned DNS requirements
   4. Generate an SSH private key and add it to the agent
   5. Obtain the installation program
   6. Manually create the installation configuration file
      • Sample install-config.yaml file for VMware vSphere
   7. Create the Ignition config files
   8. Create RHCOS machines in vSphere
   9. Install the OpenShift Command-line Interface
   10. Create the cluster
   11. Log on to the cluster
   12. Approve the CSRs for the machines
   13. Initial Operator configuration
       • Image registry storage configuration
       • Configure registry storage for VMware vSphere
       • Configure storage for the image registry in non-production clusters
   14. Complete installation on user-provisioned infrastructure
4. Gather installation logs
   • Gather logs from a failed installation
5. Installation configuration
   1. Available cluster customizations
   2. Cluster configuration resources
   3. Operator configuration resources
   4. Additional configuration resources
   5. Informational Resources
6. Configure the firewall to access Red Hat Insights

Install on Amazon Web Services

Configure Route 53

To install OpenShift, an Amazon Web Services (AWS) account must have a dedicated, authoritative public hosted zone in the Route 53 service. This service provides cluster DNS resolution and name lookup for external connections to the cluster.

1. Register new domain names using AWS

   ...or...

   Migrate an existing domain and registrar's DNS to AWS.

2. Create a public hosted zone for the domain or subdomain.

   Use an appropriate root domain, such as example.com, or subdomain, such as clusters.example.com.

3. Extract the new authoritative name servers from the hosted zone records.

4. Update the registrar records for the AWS Route 53 name servers used by the domain.

   For example, if a domain is registered to a Route 53 service in a different account, add or change name servers or glue records.

5. Add subdomain delegation records to the parent domain.

AWS account limits

The OpenShift cluster uses default Service Limits. The following table summarizes the AWS components whose limits can impact installation.

Component	Default number of clusters	Default AWS limit	Description
Instance Limits	Varies	Varies	Default instances: One bootstrap machine Three master nodes Three worker nodes These instance type counts are within a new account's default limit. To deploy more worker nodes, enable autoscaling, deploy large workloads, or use a different instance type, review account limits to ensure the cluster can deploy the machines required. In most regions, the bootstrap and worker machines uses an m4.large machines and the master machines use m4.xlarge instances. In some regions, including all regions that do not support these instance types, m5.large and m5.xlarge instances are used instead.
Elastic IPs (EIPs)	0 to 1	5 EIPs per account	To provision the cluster in a highly available configuration, the installation program creates a public and private subnet for each availability zone within a region. Each private subnet requires a NAT Gateway, and each NAT gateway requires a separate elastic IP. Review the AWS region map to determine how many availability zones are in each region. We can install a single cluster in many regions without increasing the EIP limit, but to take advantage of the default high availability, install the cluster in a region with at least three availability zones. To use the us-east-1 region, increase the EIP limit for the account.
Virtual Private Clouds (VPCs)	5	5 VPCs per region	Each cluster creates its own VPC.
Elastic Load Balancing (ELB/NLB)	3	20 per region	By default, each cluster creates an internal and external network load balancers for the master API server and a single classic elastic load balancer for the router. Deploying more Kubernetes LoadBalancer Service objects will create additional load balancers.
NAT Gateways	5	5 per availability zone	The cluster deploys one NAT gateway in each availability zone.
Elastic Network Interfaces (ENIs)	At least 12	350 per region	The default installation creates 21 ENIs and an ENI for each availability zone in your region. For example, the us-east-1 region contains six availability zones, so a cluster that is deployed in that zone uses 27 ENIs. Review the AWS region map to determine how many availability zones are in each region. Additional ENIs are created for additional machines and elastic load balancers that are created by cluster usage and deployed workloads.
VPC Gateway	20	20 per account	Your AWS account uses VPC Gateways for S3 access. Each cluster creates a single VPC Gateway for S3 access.
S3 buckets	99	100 buckets per account	Because the installation process creates a temporary bucket and the registry component in each cluster creates a bucket, we can create only 99 OpenShift clusters per AWS account.
Security Groups	250	2,500 per account	Each cluster creates 10 distinct security groups.

Create an IAM user

Each Amazon Web Services (AWS) account contains a root user account based on the email address used to create the account. This is a highly-privileged account, and it is recommended to use it for only initial account and billing configuration, creating an initial set of users, and securing the account.

Before installing OpenShift, create a secondary IAM administrative user, then set the following options:

1. Specify the IAM user name and select Programmatic access.

2. Attach the AdministratorAccess policy to ensure the account has sufficient permission to create the cluster. This policy provides the cluster with the ability to grant credentials to each OpenShift component. The cluster grants the components only the credentials that they require.

   While it is possible to create a policy that grants the all of the required AWS permissions and attach it to the user, this is not the preferred option. The cluster will not have the ability to grant additional credentials to individual components, so the same credentials are used by all components.

3. Optional: Add metadata to the user by attaching tags.

4. Confirm the specified user name is granted the AdministratorAccess policy.

5. Record the access key ID and secret access key values.

   Use these values when configuring a local machine to run the installation program.

   We cannot use a temporary session token generated while using a multi-factor authentication device to authenticate to AWS when you deploy a cluster. The cluster continues to use our current AWS credentials to create AWS resources for the entire life of the cluster, so use key-based, long-lived credentials.

Required AWS permissions

When attaching the AdministratorAccess policy to the IAM user, grant that user the following permissions.

Required EC2 permissions for installation

• ec2:AllocateAddress
• ec2:AssociateAddress
• ec2:AssociateDhcpOptions
• ec2:AssociateRouteTable
• ec2:AttachInternetGateway
• ec2:AuthorizeSecurityGroupEgress
• ec2:AuthorizeSecurityGroupIngress
• ec2:CopyImage
• ec2:CreateDhcpOptions
• ec2:CreateInternetGateway
• ec2:CreateNatGateway
• ec2:CreateRoute
• ec2:CreateRouteTable
• ec2:CreateSecurityGroup
• ec2:CreateSubnet
• ec2:CreateTags
• ec2:CreateVpc
• ec2:CreateVpcEndpoint
• ec2:CreateVolume
• ec2:DescribeAccountAttributes
• ec2:DescribeAddresses
• ec2:DescribeAvailabilityZones
• ec2:DescribeDhcpOptions
• ec2:DescribeImages
• ec2:DescribeInstanceAttribute
• ec2:DescribeInstanceCreditSpecifications
• ec2:DescribeInstances
• ec2:DescribeInternetGateways
• ec2:DescribeKeyPairs
• ec2:DescribeNatGateways
• ec2:DescribeNetworkAcls
• ec2:DescribePrefixLists
• ec2:DescribeRegions
• ec2:DescribeRouteTables
• ec2:DescribeSecurityGroups
• ec2:DescribeSubnets
• ec2:DescribeTags
• ec2:DescribeVpcEndpoints
• ec2:DescribeVpcs
• ec2:DescribeVpcAttribute
• ec2:DescribeVolumes
• ec2:DescribeVpcClassicLink
• ec2:DescribeVpcClassicLinkDnsSupport
• ec2:ModifyInstanceAttribute
• ec2:ModifySubnetAttribute
• ec2:ModifyVpcAttribute
• ec2:RevokeSecurityGroupEgress
• ec2:RunInstances
• ec2:TerminateInstances
• ec2:RevokeSecurityGroupIngress
• ec2:ReplaceRouteTableAssociation
• ec2:DescribeNetworkInterfaces
• ec2:ModifyNetworkInterfaceAttribute

Required Elasticloadbalancing permissions for installation

• elasticloadbalancing:AddTags
• elasticloadbalancing:ApplySecurityGroupsToLoadBalancer
• elasticloadbalancing:AttachLoadBalancerToSubnets
• elasticloadbalancing:CreateListener
• elasticloadbalancing:CreateLoadBalancer
• elasticloadbalancing:CreateLoadBalancerListeners
• elasticloadbalancing:CreateTargetGroup
• elasticloadbalancing:ConfigureHealthCheck
• elasticloadbalancing:DeregisterInstancesFromLoadBalancer
• elasticloadbalancing:DeregisterTargets
• elasticloadbalancing:DescribeInstanceHealth
• elasticloadbalancing:DescribeListeners
• elasticloadbalancing:DescribeLoadBalancers
• elasticloadbalancing:DescribeLoadBalancerAttributes
• elasticloadbalancing:DescribeTags
• elasticloadbalancing:DescribeTargetGroupAttributes
• elasticloadbalancing:DescribeTargetHealth
• elasticloadbalancing:ModifyLoadBalancerAttributes
• elasticloadbalancing:ModifyTargetGroup
• elasticloadbalancing:ModifyTargetGroupAttributes
• elasticloadbalancing:RegisterTargets
• elasticloadbalancing:RegisterInstancesWithLoadBalancer
• elasticloadbalancing:SetLoadBalancerPoliciesOfListener

Required IAM permissions for installation

• iam:AddRoleToInstanceProfile
• iam:CreateInstanceProfile
• iam:CreateRole
• iam:DeleteInstanceProfile
• iam:DeleteRole
• iam:DeleteRolePolicy
• iam:GetInstanceProfile
• iam:GetRole
• iam:GetRolePolicy
• iam:GetUser
• iam:ListInstanceProfilesForRole
• iam:ListRoles
• iam:ListUsers
• iam:PassRole
• iam:PutRolePolicy
• iam:RemoveRoleFromInstanceProfile
• iam:SimulatePrincipalPolicy
• iam:TagRole

Required Route 53 permissions for installation

• route53:ChangeResourceRecordSets
• route53:ChangeTagsForResource
• route53:GetChange
• route53:GetHostedZone
• route53:CreateHostedZone
• route53:ListHostedZones
• route53:ListHostedZonesByName
• route53:ListResourceRecordSets
• route53:ListTagsForResource
• route53:UpdateHostedZoneComment

Required S3 permissions for installation

• s3:CreateBucket
• s3:DeleteBucket
• s3:GetAccelerateConfiguration
• s3:GetBucketCors
• s3:GetBucketLocation
• s3:GetBucketLogging
• s3:GetBucketObjectLockConfiguration
• s3:GetBucketReplication
• s3:GetBucketRequestPayment
• s3:GetBucketTagging
• s3:GetBucketVersioning
• s3:GetBucketWebsite
• s3:GetEncryptionConfiguration
• s3:GetLifecycleConfiguration
• s3:GetReplicationConfiguration
• s3:ListBucket
• s3:PutBucketAcl
• s3:PutBucketTagging
• s3:PutEncryptionConfiguration

S3 permissions that cluster Operators require

• s3:PutObject
• s3:PutObjectAcl
• s3:PutObjectTagging
• s3:GetObject
• s3:GetObjectAcl
• s3:GetObjectTagging
• s3:GetObjectVersion
• s3:DeleteObject

All additional permissions required to uninstall a cluster

• autoscaling:DescribeAutoScalingGroups
• ec2:DeleteDhcpOptions
• ec2:DeleteInternetGateway
• ec2:DeleteNatGateway
• ec2:DeleteNetworkInterface
• ec2:DeleteRoute
• ec2:DeleteRouteTable
• ec2:DeleteSnapshot
• ec2:DeleteSecurityGroup
• ec2:DeleteSubnet
• ec2:DeleteVolume
• ec2:DeleteVpc
• ec2:DeleteVpcEndpoints
• ec2:DeregisterImage
• ec2:DetachInternetGateway
• ec2:DisassociateRouteTable
• ec2:ReleaseAddress
• elasticloadbalancing:DescribeTargetGroups
• elasticloadbalancing:DeleteTargetGroup
• elasticloadbalancing:DeleteLoadBalancer
• iam:ListInstanceProfiles
• iam:ListRolePolicies
• iam:ListUserPolicies
• route53:DeleteHostedZone
• tag:GetResources

Supported AWS regions

We can deploy an OpenShift cluster to the following regions:

• ap-northeast-1 (Tokyo)
• ap-northeast-2 (Seoul)
• ap-south-1 (Mumbai)
• ap-southeast-1 (Singapore)
• ap-southeast-2 (Sydney)
• ca-central-1 (Central)
• eu-central-1 (Frankfurt)
• eu-west-1 (Ireland)
• eu-west-2 (London)
• eu-west-3 (Paris)
• sa-east-1 (SaŁo Paulo)
• us-east-1 (N. Virginia)
• us-east-2 (Ohio)
• us-west-1 (N. California)
• us-west-2 (Oregon)

Next steps

• Install an OpenShift cluster. We can install a customized cluster or quickly install a cluster with default options.

Install a cluster on AWS using default configuration options

• Review details about the OpenShift installation and update processes.

• Configure an AWS account to host the cluster.

  If we have an AWS profile stored on our computer, it must not use a temporary session token generated while using a multi-factor authentication device. The cluster continues to use our current AWS credentials to create AWS resources for the entire life of the cluster, so use key-based, long-lived credentials. To generate appropriate keys, see Manage Access Keys for IAM Users in the AWS documentation. We can supply the keys when we run the installation program.

• If a firewall is used, configure it to access Red Hat Insights.

Internet and Telemetry access

Telemetry provides metrics about cluster health and the success of updates. To perform subscription management, including legally entitling a purchase Red Hat, use the Telemetry service and access the OpenShift Infrastructure Providers page.

There is no disconnected subscription management. We cannot both opt out of sending data back to Red Hat and entitle a purchase.

Machines must have direct internet access to install the cluster.

Internet access is required to:

• Access the OpenShift Infrastructure Providers page to download the installation program
• Access quay.io to obtain the packages required to install the cluster
• Obtain the packages required to perform cluster updates
• Access Red Hat's software as a service page to perform subscription management

Generate an SSH private key and add it to the agent

For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key for the ssh-agent process.

We can use this key to SSH onto the master nodes as the user core. When we deploy the cluster, the key is added to the core user's ~/.ssh/authorized_keys list.

Use a local key, not one configured with platform-specific approaches such as AWS key pairs.

Procedure

1. If we do not have an SSH key configured for password-less authentication on the computer, create one. For example, on a computer that uses a Linux operating system, run:
   $ ssh-keygen -t rsa -b 4096 -N '' -f /path/to/.ssh/id_rsa

2. Start the ssh-agent process as a background task:
   $ eval "$(ssh-agent -s)"
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:
   $ ssh-add /path/to/.ssh/id_rsa
   Identity added: /path/to/.ssh/id_rsa (computer_name)

Next steps

When we install OpenShift, provide the SSH public key to the installer. If we install a cluster on infrastructure that we provision, provide this key to the cluster's machines.

Obtain the installation program

Before installing OpenShift, we download the installation file on a local computer.

Prerequisites

Install the cluster from a computer that uses Linux or macOS.

We need 300 MB of local disk space to download the installation program.

Procedure

1. Access the OpenShift Infrastructure Providers page.

2. Download the installation program for your operating system and place the file in the directory where you will store the installation configuration files.

   The installation program creates several files on the computer used to install the cluster. Keep both the installation program and the files the installation program creates after you finish installing the cluster.

3. Extract the installation program.
   $ tar xvf installation_program.tar.gz

4. From the OpenShift Infrastructure Providers page, download your installation pull secret.

   This pull secret allows us to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift components.

Deploy the cluster on a cloud

We can run the installation program only once, during initial installation.

Prerequisites

• Configure an account with the cloud platform that hosts the cluster.
• Obtain the OpenShift installation program and the pull secret for the cluster.

Procedure

1. Run the installation program:
   $ ./openshift-install create cluster --dir=/path/to/install/files --log-level debug

   Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so do not reuse an installation directory. To reuse individual files from another cluster installation, copy them into the directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift version.

   Provide values at the prompts:

   1. Optional: Select an SSH key to use to access the cluster machines.

      For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.

   2. Select AWS as the platform to target.
   3. If we do not have an Amazon Web Services (AWS) profile stored on our computer, enter the AWS access key ID and secret access key for the user configured to run the installation program.
   4. Select the AWS region to deploy the cluster to.
   5. Select the base domain for the Route 53 service configured for the cluster.
   6. Enter a descriptive name for the cluster.
   7. Paste the pull secret obtained from the OpenShift Infrastructure Providers page.

   If the cloud provider account configured on the host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed.

   When the cluster deployment completes, directions for accessing the cluster, including a link to its web console and credentials for the kubeadmin user, display in your terminal.

   Do not delete the installation program or the files that the installation program creates. Both are required to delete the cluster.

2. Optional: Remove or disable the AdministratorAccess policy from the IAM account used to install the cluster.

Install the OpenShift Command-line Interface (oc)

Earlier versions of oc cannot be used to complete all of the commands in OpenShift 4.1. Download and install the new version of oc.

1. From the OpenShift Infrastructure Providers page, click...
   Download Command-line Tools

2. Download the compressed file for your operating system.

   We can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Log on to the cluster

1. Deploy an OpenShift cluster.

2. Install the oc CLI.

3. Export the kubeadmin credentials:
   $ export KUBECONFIG=/path/to/install/files/auth/kubeconfig
   $ oc whoami
   system:admin

   The kubeconfig file contains information about the cluster used by the CLI to connect a client to the correct cluster and API server. The file is specific to a cluster and is created during OpenShift installation.

Next steps

• Customize the cluster.
• If necessary, we can opt out of telemetry.

Install a cluster on AWS with customizations

We can install a customized cluster on infrastructure that the installation program provisions on Amazon Web Services (AWS). To customize the installation, you modify some parameters in the install-config.yaml file before installing the cluster.

• Review details about the installation and update processes.

• Configure an AWS account to host the cluster.

  If we have an AWS profile stored on our computer, it must not use a temporary session token generated while using a multi-factor authentication device. The cluster continues to use our current AWS credentials to create AWS resources for the entire life of the cluster, so use long-lived credentials. To generate appropriate keys, see Manage Access Keys for IAM Users in the AWS documentation. We can supply the keys when we run the installation program.

• If a firewall is used, configure it to access Red Hat Insights.

Internet and Telemetry access

Telemetry provides metrics about cluster health and the success of updates. To perform subscription management, including legally entitling a purchase Red Hat, use the Telemetry service and access the OpenShift Infrastructure Providers page.

There is no disconnected subscription management. We cannot both opt out of sending data back to Red Hat and entitle a purchase.

Machines have direct internet access to install the cluster.

Internet access is required to:

• Access the OpenShift Infrastructure Providers page to download the installation program
• Access quay.io to obtain the packages required to install the cluster
• Obtain the packages required to perform cluster updates
• Access Red Hat's software as a service page to perform subscription management

Generate an SSH private key and add it to the agent

For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key for the ssh-agent process.

We can use this key to SSH onto the master nodes as the user core. phen you deploy the cluster, the key is added to the core user's ~/.ssh/authorized_keys list.

Use a local key, not one configured with platform-specific approaches such as AWS key pairs.

Procedure

If we do not have an SSH key configured for password-less authentication on the computer, create one. For example, on a computer that uses a Linux operating system, run:

$ ssh-keygen -t rsa -b 4096 -N '' -f /path/to/.ssh/id_rsa

Start the ssh-agent process as a background task:

$ eval "$(ssh-agent -s)"
Agent pid 31874

Add your SSH private key to the ssh-agent:

$ ssh-add /path/to/.ssh/id_rsa
Identity added: /path/to/.ssh/id_rsa (computer_name)

Next steps

When we install OpenShift, provide the SSH public key to the installer. If we install a cluster on infrastructure that we provision, provide this key to the cluster's machines.

Obtain the installation program

Before installing OpenShift, download the installation file on a local computer.

Prerequisites

• Install the cluster from a computer that uses Linux or macOS.
• You need 300 MB of local disk space to download the installation program.

Procedure

1. Access the OpenShift Infrastructure Providers page. If we have a Red Hat account, log in with your credentials. If we do not, create an account.

2. Download the installation program for your operating system and place the file in the directory where you will store the installation configuration files.

   The installation program creates several files on the computer used to install the cluster. Keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run:

   $ tar xvf installation_program.tar.gz
   

4. From the OpenShift Infrastructure Providers page, download your installation pull secret. This pull secret allows us to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift components.

Create the installation configuration file

We can customize your installation of OpenShift on a compatible cloud.

Prerequisites

Obtain the OpenShift installation program and the pull secret for the cluster.

Procedure

1. Create the install-config.yaml file.

   1. Run the following command:
      $ ./openshift-install create install-config --dir=/path/to/install/files

      Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so do not reuse an installation directory. To reuse individual files from another cluster installation, copy them into the directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift version.

   2. At the prompts, provide the configuration details for your cloud:

      1. Optional: Select an SSH key to use to access the cluster machines.

         For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.

      2. Select AWS as the platform to target.
      3. If we do not have an Amazon Web Services (AWS) profile stored on our computer, enter the AWS access key ID and secret access key for the user configured to run the installation program.
      4. Select the AWS region to deploy the cluster to.
      5. Select the base domain for the Route 53 service configured for the cluster.
      6. Enter a descriptive name for the cluster.
      7. Paste the pull secret obtained from the OpenShift Infrastructure Providers page.

2. Modify the install-config.yaml file. We can find more information about the available parameters in the Installation configuration parameters section and in the Go documentation.

3. Back up the install-config.yaml file so that we can use it to install multiple clusters.

   The install-config.yaml file is consumed during the installation process. To reuse the file, back it up now.

Installation configuration parameters

Before deploying an OpenShift cluster, we provide parameter values to describe your Amazon Web Services (AWS) account and optionally customize the cluster's platform. When creating the install-config.yaml installation configuration file, we provide values for the required parameters through the command line. If you customize the cluster, we can modify the install-config.yaml file to provide more details about the platform.

We cannot modify these parameters after installation.

Table 1.1. Required parameters

Parameter	Description	Values
baseDomain	The base domain of the cloud provider. This value is used to create routes to the OpenShift cluster components. The full DNS name for the cluster is a combination of the baseDomain and metadata.name parameter values using metadata.name.baseDomain format.	A fully-qualified domain or subdomain name, such as example.com.
controlPlane.platform	The cloud provider to host the control plane machines. This parameter value must match the compute.platform parameter value.	aws
compute.platform	The cloud provider to host the worker machines. This parameter value must match the controlPlane.platform parameter value.	aws
metadata.name	The name of the cluster.	A string that contains uppercase or lowercase letters, such as dev.
platform.aws.region	The region to deploy the cluster in.	A valid AWS region, such as us-east-1.
pullSecret	The pull secret obtained from the OpenShift Infrastructure Providers page. We use this pull secret to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift components.	{ "auths":{ "cloud.openshift.com":{ "auth":"b3Blb=", "email":"you@example.com" }, "quay.io":{ "auth":"b3Blb=", "email":"you@example.com" } } }

Table 1.2. Optional AWS platform parameters

Parameter	Description	Values
sshKey	The SSH key to use to access the cluster machines. For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.	A valid, local public SSH key that you added to the ssh-agent process.
compute.hyperthreading	Whether to enable or disable simultaneous multithreading, or hyperthreading on compute machines. By default, simultaneous multithreading is enabled to increase the performance of the machines' cores. If you disable simultaneous multithreading, ensure that your capacity planning accounts for the dramatically decreased machine performance.	Enabled or Disabled
compute.platform.aws.rootVolume.iops	The Input/Output Operations Per Second (IOPS) that is reserved for the root volume.	Integer, for example 4000.
compute.platform.aws.rootVolume.size	The size in GiB of the root volume.	Integer, for example 500.
compute.platform.aws.rootVolume.type	The instance type of the root volume.	Valid AWS EBS instance type, such as io1.
compute.platform.aws.type	The EC2 instance type for the compute machines.	Valid AWS instance type, such as c5.9xlarge.
compute.platform.aws.zones	The availability zones where the installation program creates machines for the compute MachinePool.	A list of valid AWS availability zones, such as us-east-1c, in a YAML sequence.
compute.aws.region	The AWS region that the installation program creates compute resources in.	Valid AWS region, such as us-east-1.
compute.replicas	The number of compute, or worker, machines to provision.	A positive integer greater than or equal to 2. The default value is 3.
controlPlane.hyperthreading	Whether to enable or disable simultaneous multithreading, or hyperthreading on control plane machines. By default, simultaneous multithreading is enabled to increase the performance of the machines' cores. If you disable simultaneous multithreading, ensure that your capacity planning accounts for the dramatically decreased machine performance.	Enabled or Disabled
controlPlane.platform.aws.type	The EC2 instance type for the control plane machines.	Valid AWS instance type, such as c5.9xlarge.
controlPlane.platform.aws.zones	The availability zones where the installation program creates machines for the control plane MachinePool.	A list of valid AWS availability zones, such as us-east-1c, in a YAML sequence.
controlPlane.aws.region	The AWS region that the installation program creates control plane resources in.	Valid AWS region, such as us-east-1.
controlPlane.replicas	The number of control plane machines to provision.	A positive integer greater than or equal to 3. The default value is 3.
platform.aws.userTags	A map of keys and values that the installation program adds as tags to all resources that it creates.	Any valid YAML map, such as key value pairs in the key: value format. For more information about AWS tags, see Tagging Your Amazon EC2 Resources in the AWS documentation.

Sample customized install-config.yaml file for AWS

We can customize the install-config.yaml file to specify more details about the OpenShift cluster's platform or modify the values of the required parameters.

This sample YAML file is provided for reference only. Obtain your install-config.yaml file using the installation program and modify it.

apiVersion: v1
baseDomain: example.com 1
controlPlane: 2
  hyperthreading: Enabled 3 4
  name: master
  platform:
    aws:
      zones:
      - us-west-2a
      - us-west-2b
      rootVolume:
iops: 4000
size: 500
type: io1
      type: m5.xlarge 5
  replicas: 3
compute: 6
- hyperthreading: Enabled 7
  name: worker
  platform:
    aws:
      rootVolume:
iops: 2000
size: 500
type: io1 8
      type: c5.4xlarge
      zones:
      - us-west-2c
  replicas: 3
metadata:
  name: test-cluster 9
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineCIDR: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  aws:
    region: us-west-2 10
    userTags:
      adminContact: jdoe
      costCenter: 7536
pullSecret: '{"auths": ...}' 11
sshKey: ssh-ed25519 AAAA... 12

1 9 10 11 Required. The installation program prompts you for this value.

2 6 If we do not provide these parameters and values, the installation program provides the default value.

3 7 The controlPlane section is a single mapping, but the compute section is a sequence of mappings. To meet the requirements of the different data structures, the first line of the compute section must begin with a hyphen, -, and the first line of the controlPlane section must not. Although both sections currently define a single machine pool, it is possible that future versions of OpenShift will support defining multiple compute pools during installation. Only one control plane pool is used.

4 5 Whether to enable or disable simultaneous multithreading, or hyperthreading. By default, simultaneous multithreading is enabled to increase the performance of the machines' cores. We can disable it by setting the parameter value to Disabled. If you disable simultanous multithreading in some cluster machines, disable it in all cluster machines.

If you disable simultaneous multithreading, ensure that your capacity planning accounts for the dramatically decreased machine performance. Use larger instance types, such as m4.2xlarge or m5.2xlarge, for the machines if you disable simultaneous multithreading.

8 To configure faster storage for etcd especially for larger clusters, set the storage type as io1 and set iops to 2000.

12 We can optionally provide the sshKey value used to access the machines in the cluster.

For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.

Deploy the cluster

We can install OpenShift on a compatible cloud.

We can run the installation program only once, during initial installation.

Prerequisites

• Configure an account with the cloud platform that hosts the cluster.
• Obtain the OpenShift installation program and the pull secret for the cluster.

Procedure

1. Run the installation program:
   $ ./openshift-install create cluster --dir=/path/to/install/files --log-level debug

   For /path/to/install/files, specify the location of the customized ./install-config.yaml file. For /path/to/install/files, specify the directory name to store the files that the installation program creates. Optionally, include the --log-level debug option to view installation details.

   Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so do not reuse an installation directory. To reuse individual files from another cluster installation, copy them into the directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift version.

   Provide values at the prompts:

   1. Optional: Select an SSH key to use to access the cluster machines.

      For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.

   2. Select AWS as the platform to target.
   3. If we do not have an Amazon Web Services (AWS) profile stored on our computer, enter the AWS access key ID and secret access key for the user configured to run the installation program.
   4. Select the AWS region to deploy the cluster to.
   5. Select the base domain for the Route 53 service configured for the cluster.
   6. Enter a descriptive name for the cluster.
   7. Paste the pull secret obtained from the OpenShift Infrastructure Providers page.

   If the cloud provider account configured on the host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed.

   When the cluster deployment completes, directions for accessing the cluster, including a link to its web console and credentials for the kubeadmin user, display in your terminal.

   Do not delete the installation program or the files that the installation program creates. Both are required to delete the cluster.

2. Optional: Remove or disable the AdministratorAccess policy from the IAM account used to install the cluster.

Installation command options

The installation program uses Semantic Versioning for its user-facing API to ensure that commands are stable across versions. While most of the openshift-install command options comply with semantic versioning practices, some do not.

Commands that comply with semantic versioning

The following commands are covered by the versioning:

openshift-install [options] create install-config

Creates the install-config.yaml file in the asset directory. The version of the generated install-config might change.

openshift-install [options] create ignition-configs

Creates the bootstrap.ign, master.ign, and worker.ign files in the asset directory,. The content of the generated files might change between versions.

openshift-install [options] create cluster

Launches a new cluster.

openshift-install [options] destroy bootstrap

Destroys any bootstrap resources created for the cluster.

openshift-install [options] destroy cluster

Destroys the cluster resources.

openshift-install [options] help

Shows help for the command. The available options and unstable commands might change between versions

openshift-install [options] version

Displays sufficient version information for maintainers to identify the installation program version. The format and content of its output might change between versions.

The install-config format itself

New versions of this format might be released, but within a minor version series, the openshift-install command can read previous versions.

Commands that do not comply with semantic versioning

The following commands are not covered by the versioning:

openshift-install [options] graph, openshift-install [options] create manifest-templates, openshift-install [options] create manifests

If we alter, add, or remove Kubernetes objects, perform those actions after installing the cluster..

Install the OpenShift Command-line Interface (oc)

Earlier versions of oc cannot be used to install OpenShift 4.1. Download and install the new version of oc.

1. From the OpenShift Infrastructure Providers page, click...
   Download Command-line Tools

2. Download the compressed file for your operating system.

   We can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Log on to the cluster

1. Deploy an OpenShift cluster.

2. Install the oc CLI.

3. Export the kubeadmin credentials:
   $ export KUBECONFIG=/path/to/install/files/auth/kubeconfig
   $ oc whoami
   system:admin

   The kubeconfig file contains information about the cluster used by the CLI to connect a client to the correct cluster and API server. The file is specific to a cluster and is created during OpenShift installation.

Next steps

• Customize the cluster.
• If necessary, we can opt out of telemetry.

Install a cluster on AWS with network customizations

We can install a cluster on Amazon Web Services (AWS) with customized network configuration options. By customizing your network configuration, the cluster can coexist with existing IP address allocations in your environment and integrate with existing MTU and VXLAN configurations.

Set most of the network configuration parameters during installation. We can modify only kubeProxy configuration parameters in a running cluster.

• Review details about the installation and update processes.

• Configure an AWS account to host the cluster.

  If we have an AWS profile stored on our computer, it must not use a temporary session token generated while using a multi-factor authentication device. The cluster continues to use our current AWS credentials to create AWS resources for the entire life of the cluster, so use key-based, long-lived credentials. To generate appropriate keys, see Manage Access Keys for IAM Users in the AWS documentation. We can supply the keys when we run the installation program.

• If a firewall is used, configure it to access Red Hat Insights.

Internet and Telemetry access

Telemetry provides metrics about cluster health and the success of updates. To perform subscription management, including legally entitling a purchase Red Hat, use the Telemetry service and access the OpenShift Infrastructure Providers page.

There is no disconnected subscription management. We cannot both opt out of sending data back to Red Hat and entitle a purchase.

Machines have direct internet access to install the cluster.

Internet access is required to:

• Access the OpenShift Infrastructure Providers page to download the installation program
• Access quay.io to obtain the packages required to install the cluster
• Obtain the packages required to perform cluster updates
• Access Red Hat's software as a service page to perform subscription management

Generate an SSH private key and add it to the agent

For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key for the ssh-agent process.

We can use this key to SSH onto the master nodes as the user core. When we deploy the cluster, the key is added to the core user's ~/.ssh/authorized_keys list.

Use a local key, not one configured with platform-specific approaches such as AWS key pairs.

Procedure

1. If we do not have an SSH key configured for password-less authentication on the computer, create one. For example, on a computer that uses a Linux operating system, run:
   $ ssh-keygen -t rsa -b 4096 -N '' -f /path/to/.ssh/id_rsa

2. Start the ssh-agent process as a background task:
   $ eval "$(ssh-agent -s)"
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:
   $ ssh-add /path/to/.ssh/id_rsa
   Identity added: /path/to/.ssh/id_rsa (computer_name)

Next steps

When we install OpenShift, provide the SSH public key to the installer. If we install a cluster on infrastructure that we provision, provide this key to the cluster's machines.

Obtain the installation program

Before installing OpenShift, download the installation file on a local computer.

Prerequisites

• Install the cluster from a computer that uses Linux or macOS.
• You need 300 MB of local disk space to download the installation program.

Procedure

1. Access the OpenShift Infrastructure Providers page. If we have a Red Hat account, log in with your credentials. If we do not, create an account.

2. Download the installation program for your operating system and place the file in the directory where you will store the installation configuration files.

   The installation program creates several files on the computer used to install the cluster. Keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run:

   $ tar xvf installation_program.tar.gz
   

4. From the OpenShift Infrastructure Providers page, download your installation pull secret. This pull secret allows us to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift components.

Create the installation configuration file

We can customize your installation of OpenShift on a compatible cloud.

Prerequisites

• Obtain the OpenShift installation program and the pull secret for the cluster.

Procedure

1. Create the install-config.yaml file.

   1. Run the following command:

      $ ./openshift-install create install-config --dir=/path/to/install/files 
      

      Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so do not reuse an installation directory. To reuse individual files from another cluster installation, copy them into the directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift version.

   2. At the prompts, provide the configuration details for your cloud:

      1. Optional: Select an SSH key to use to access the cluster machines.

         For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.

      2. Select AWS as the platform to target.
      3. If we do not have an Amazon Web Services (AWS) profile stored on our computer, enter the AWS access key ID and secret access key for the user configured to run the installation program.
      4. Select the AWS region to deploy the cluster to.
      5. Select the base domain for the Route 53 service configured for the cluster.
      6. Enter a descriptive name for the cluster.
      7. Paste the pull secret obtained from the OpenShift Infrastructure Providers page.
2. Modify the install-config.yaml file. We can find more information about the available parameters in the Installation configuration parameters section and in the Go documentation.

3. Back up the install-config.yaml file so that we can use it to install multiple clusters.

   The install-config.yaml file is consumed during the installation process. To reuse the file, back it up now.

Installation configuration parameters

Before deploying an OpenShift cluster, we provide parameter values to describe your Amazon Web Services (AWS) account and optionally customize the cluster's platform. When creating the install-config.yaml installation configuration file, we provide values for the required parameters through the command line. If you customize the cluster, we can modify the install-config.yaml file to provide more details about the platform.

We cannot modify these parameters after installation.

Table 1.3. Required parameters

Parameter	Description	Values
baseDomain	The base domain of the cloud provider. This value is used to create routes to the OpenShift cluster components. The full DNS name for the cluster is a combination of the baseDomain and metadata.name parameter values using metadata.name.baseDomain format.	A fully-qualified domain or subdomain name, such as example.com.
controlPlane.platform	The cloud provider to host the control plane machines. This parameter value must match the compute.platform parameter value.	aws
compute.platform	The cloud provider to host the worker machines. This parameter value must match the controlPlane.platform parameter value.	aws
metadata.name	The name of the cluster.	A string that contains uppercase or lowercase letters, such as dev.
platform.aws.region	The region to deploy the cluster in.	A valid AWS region, such as us-east-1.
pullSecret	The pull secret obtained from the OpenShift Infrastructure Providers page. We use this pull secret to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift components.	{ "auths":{ "cloud.openshift.com":{ "auth":"b3Blb=", "email":"you@example.com" }, "quay.io":{ "auth":"b3Blb=", "email":"you@example.com" } } }

Table 1.4. Optional AWS platform parameters

Parameter	Description	Values
sshKey	The SSH key to use to access the cluster machines. For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.	A valid, local public SSH key that you added to the ssh-agent process.
compute.hyperthreading	Whether to enable or disable simultaneous multithreading, or hyperthreading on compute machines. By default, simultaneous multithreading is enabled to increase the performance of the machines' cores. If you disable simultaneous multithreading, ensure that your capacity planning accounts for the dramatically decreased machine performance.	Enabled or Disabled
compute.platform.aws.rootVolume.iops	The Input/Output Operations Per Second (IOPS) that is reserved for the root volume.	Integer, for example 4000.
compute.platform.aws.rootVolume.size	The size in GiB of the root volume.	Integer, for example 500.
compute.platform.aws.rootVolume.type	The instance type of the root volume.	Valid AWS EBS instance type, such as io1.
compute.platform.aws.type	The EC2 instance type for the compute machines.	Valid AWS instance type, such as c5.9xlarge.
compute.platform.aws.zones	The availability zones where the installation program creates machines for the compute MachinePool.	A list of valid AWS availability zones, such as us-east-1c, in a YAML sequence.
compute.aws.region	The AWS region that the installation program creates compute resources in.	Valid AWS region, such as us-east-1.
compute.replicas	The number of compute, or worker, machines to provision.	A positive integer greater than or equal to 2. The default value is 3.
controlPlane.hyperthreading	Whether to enable or disable simultaneous multithreading, or hyperthreading on control plane machines. By default, simultaneous multithreading is enabled to increase the performance of the machines' cores. If you disable simultaneous multithreading, ensure that your capacity planning accounts for the dramatically decreased machine performance.	Enabled or Disabled
controlPlane.platform.aws.type	The EC2 instance type for the control plane machines.	Valid AWS instance type, such as c5.9xlarge.
controlPlane.platform.aws.zones	The availability zones where the installation program creates machines for the control plane MachinePool.	A list of valid AWS availability zones, such as us-east-1c, in a YAML sequence.
controlPlane.aws.region	The AWS region that the installation program creates control plane resources in.	Valid AWS region, such as us-east-1.
controlPlane.replicas	The number of control plane machines to provision.	A positive integer greater than or equal to 3. The default value is 3.
platform.aws.userTags	A map of keys and values that the installation program adds as tags to all resources that it creates.	Any valid YAML map, such as key value pairs in the key: value format. For more information about AWS tags, see Tagging Your Amazon EC2 Resources in the AWS documentation.

Network configuration parameters

We can modify the cluster network configuration parameters in the install-config.yaml configuration file. The following table describes the parameters.

We cannot modify these parameters after installation.

Table 1.5. Required network parameters

Parameter	Description	Values
networking.networkType	The network plug-in to deploy. OpenShiftSDN is the only plug-in supported in OpenShift 4.1.	OpenShiftSDN
networking.clusterNetwork.cidr	A block of IP addresses from which Pod IP addresses are allocated. The OpenShiftSDN network plug-in supports multiple cluster networks. The address blocks for multiple cluster networks must not overlap. Select address pools large enough to fit your anticipated workload.	An IP address allocation in CIDR format. The default value is 10.128.0.0/14.
networking.clusterNetwork.hostPrefix	The subnet prefix length to assign to each individual node. For example, if hostPrefix is set to 23, then each node is assigned a /23 subnet out of the given cidr, allowing for 510 (2^(32 - 23) - 2) Pod IP addresses.	A subnet prefix. The default value is 23.
networking.serviceNetwork	A block of IP addresses for services. OpenShiftSDN allows only one serviceNetwork block. The address block must not overlap with any other network block.	An IP address allocation in CIDR format. The default value is 172.30.0.0/16.
networking.machineCIDR	A block of IP addresses used by the OpenShift installation program while installing the cluster. The address block must not overlap with any other network block.	An IP address allocation in CIDR format. The default value is 10.0.0.0/16.

Sample customized install-config.yaml file for AWS

We can customize the install-config.yaml file to specify more details about the OpenShift cluster's platform or modify the values of the required parameters.

This sample YAML file is provided for reference only. Obtain your install-config.yaml file using the installation program and modify it.

apiVersion: v1
baseDomain: example.com 1
controlPlane: 2
  hyperthreading: Enabled 3 4
  name: master
  platform:
    aws:
      zones:
      - us-west-2a
      - us-west-2b
      rootVolume:
iops: 4000
size: 500
type: io1
      type: m5.xlarge 5
  replicas: 3
compute: 6
- hyperthreading: Enabled 7
  name: worker
  platform:
    aws:
      rootVolume:
iops: 2000
size: 500
type: io1 8
      type: c5.4xlarge
      zones:
      - us-west-2c
  replicas: 3
metadata:
  name: test-cluster 9
networking:
networking: 10
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineCIDR: 10.0.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  aws:
    region: us-west-2 11
    userTags:
      adminContact: jdoe
      costCenter: 7536
pullSecret: '{"auths": ...}' 12
sshKey: ssh-ed25519 AAAA... 13

1 9 11 12 Required. The installation program prompts you for this value.

2 6 10 If we do not provide these parameters and values, the installation program provides the default value.

3 7 The controlPlane section is a single mapping, but the compute section is a sequence of mappings. To meet the requirements of the different data structures, the first line of the compute section must begin with a hyphen, -, and the first line of the controlPlane section must not. Although both sections currently define a single machine pool, it is possible that future versions of OpenShift will support defining multiple compute pools during installation. Only one control plane pool is used.

4 5 Whether to enable or disable simultaneous multithreading, or hyperthreading. By default, simultaneous multithreading is enabled to increase the performance of the machines' cores. We can disable it by setting the parameter value to Disabled. If you disable simultanous multithreading in some cluster machines, disable it in all cluster machines.

If you disable simultaneous multithreading, ensure that your capacity planning accounts for the dramatically decreased machine performance. Use larger instance types, such as m4.2xlarge or m5.2xlarge, for the machines if you disable simultaneous multithreading.

8 To configure faster storage for etcd, especially for larger clusters, set the storage type as io1 and set iops to 2000.

13 We can optionally provide the sshKey value used to access the machines in the cluster.

For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.

Modify advanced network configuration parameters

We can modify the advanced network configuration parameters only before installing the cluster. Advanced configuration customization lets you integrate the cluster into the existing network environment by specifying an MTU or VXLAN port, by allowing customization of kube-proxy settings, and by specifying a different mode for the openshiftSDNConfig parameter.

Modifying the OpenShift manifest files directly is not supported.

Prerequisites

• Generate the install-config.yaml file and complete any modifications to it.

Procedure

1. Create manifests:

   $ ./openshift-install create manifests --dir=/path/to/install/files 
   

2. Create a file named cluster-network-03-config.yml in the /path/to/install/files/manifests/ directory:

   $ touch /path/to/install/files/manifests/cluster-network-03-config.yml 
   

   After creating the file, three network configuration files are in the manifests/ directory, as shown:

   $ ls /path/to/install/files/manifests/cluster-network-*
   cluster-network-01-crd.yml
   cluster-network-02-config.yml
   cluster-network-03-config.yml
   

3. Open the cluster-network-03-config.yml file in an editor and enter a CR that describes the Operator configuration we want:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec: 1
     defaultNetwork:
       type: OpenShiftSDN
       openshiftSDNConfig:
         mode: NetworkPolicy
         mtu: 1450
         vxlanPort: 4789
   

   1 The parameters for the spec field are only an example. Specify your configuration for the Network Operator in the CR.

   The Network Operator provides default values for the parameters in the CR, so specify only the parameters that to change in the Network.operator.openshift.io CR.

4. Save the cluster-network-03-config.yml file and quit the text editor.
5. Optional: Back up the manifests/cluster-network-03-config.yml file. The installation program deletes the manifests/ directory when creating the cluster.

Cluster Network Operator custom resource (CR)

The cluster network configuration in the Network.operator.openshift.io custom resource (CR) stores the configuration settings for the Cluster Network Operator (CNO).

The following CR displays the default configuration for the CNO and explains both the parameters we can configure and valid parameter values:

Cluster Network Operator CR

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  clusterNetwork: 1
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork: 2
  - 172.30.0.0/16
  defaultNetwork:
    type: OpenShiftSDN 3
    openshiftSDNConfig: 4
      mode: NetworkPolicy 5
      mtu: 1450 6
      vxlanPort: 4789 7
  kubeProxyConfig: 8
    iptablesSyncPeriod: 30s 9
    proxyArguments:
      iptables-min-sync-period: 10
      - 30s

1 2 3 Specified in the install-config.yaml file.

4 Specify only to override part of the SDN configuration.

5 Configures the isolation mode for OpenShiftSDN. The allowed values are Multitenant, Subnet, or NetworkPolicy. The default value is NetworkPolicy.

6 MTU for the VXLAN overlay network. This value is normally configured automatically, but if the nodes in the cluster do not all use the same MTU, then set this explicitly to 50 less than the smallest node MTU value.

7 The port to use for all VXLAN packets. The default value is 4789. If we are running in a virtualized environment with existing nodes that are part of another VXLAN network then we might be required to change this.

On Amazon Web Services (AWS), we can select an alternate port for the VXLAN between port 9000 and port 9999.

8 The kube-proxy configuration. If we do not specify the parameter values, the Network Operator applies the displayed default parameter values.

9 The refresh period for iptables rules. The default value is 30s. Valid suffixes include s, m, and h and are described in the Go time package documentation.

10 The minimum duration before refreshing iptables rules. Ensures the refresh does not happen too frequently. Valid suffixes include s, m, and h and are described in the Go time package

Deploy the cluster

We can install OpenShift on a compatible cloud.

We can run the installation program only once, during initial installation.

Prerequisites

• Configure an account with the cloud platform that hosts the cluster.
• Obtain the OpenShift installation program and the pull secret for the cluster.

Procedure

1. Run the installation program:
   $ ./openshift-install create cluster --dir=/path/to/install/files --log-level debug

   For /path/to/install/files, specify the location of the customized ./install-config.yaml file. For /path/to/install/files, specify the directory name to store the files that the installation program creates. Optionally, include the --log-level debug option to view installation details.

   Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so do not reuse an installation directory. To reuse individual files from another cluster installation, copy them into the directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift version.

   Provide values at the prompts:

   1. Optional: Select an SSH key to use to access the cluster machines.

      For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.

   2. Select AWS as the platform to target.
   3. If we do not have an Amazon Web Services (AWS) profile stored on our computer, enter the AWS access key ID and secret access key for the user configured to run the installation program.
   4. Select the AWS region to deploy the cluster to.
   5. Select the base domain for the Route 53 service configured for the cluster.
   6. Enter a descriptive name for the cluster.
   7. Paste the pull secret obtained from the OpenShift Infrastructure Providers page.

   If the cloud provider account configured on the host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed.

   When the cluster deployment completes, directions for accessing the cluster, including a link to its web console and credentials for the kubeadmin user, display in your terminal.

   Do not delete the installation program or the files that the installation program creates. Both are required to delete the cluster.

2. Optional: Remove or disable the AdministratorAccess policy from the IAM account used to install the cluster.

Install the OpenShift Command-line Interface (oc)

Earlier versions of oc cannot be used to complete all of the commands in OpenShift 4.1. Download and install the new version of oc.

1. From the OpenShift Infrastructure Providers page, click...
   Download Command-line Tools

2. Download the compressed file for your operating system.

   We can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Log on to the cluster

1. Deploy an OpenShift cluster.

2. Install the oc CLI.

3. Export the kubeadmin credentials:
   $ export KUBECONFIG=/path/to/install/files/auth/kubeconfig
   $ oc whoami
   system:admin

   The kubeconfig file contains information about the cluster used by the CLI to connect a client to the correct cluster and API server. The file is specific to a cluster and is created during OpenShift installation.

Next steps

• Customize the cluster.
• If necessary, we can opt out of telemetry.

Remove a cluster deployed to Amazon Web Services (AWS)

Prerequisites

• Have a copy of the installation program used to deploy the cluster.

Procedure

1. Optional: From the computer used to install the cluster, run and record the UUID that it outputs:
   $ oc get clusterversion -o jsonpath='{.spec.clusterID}{"\n"}' version

   If not all of the cluster resources are removed from AWS, we can use this UUID to locate them and remove them.

2. From the computer used to install the cluster, run:

   ./openshift-install destroy cluster 
                       --dir=/path/to/install/files 
                       --log-level=debug 
   
   

   Specify the directory containing the cluster definition files for the cluster. The installation program requires the metadata.json file in this directory to delete the cluster.

3. Optional: Delete the /path/to/install/files directory and the OpenShift installation program.

Install on user-provisioned AWS

Install using CloudFormation templates

We can install a cluster on Amazon Web Services (AWS) using infrastructure that we provide.

One way to create this infrastructure is to use the provided CloudFormation templates. We can modify the templates to customize our infrastructure or use the information that they contain to create AWS objects according to our company's policies.

Prerequisites

• Review details about the OpenShift installation and update processes.

• Configure an AWS account to host the cluster.

  If we have an AWS profile stored on our computer, it must not use a temporary session token generated while using a multi-factor authentication device. The cluster continues to use our current AWS credentials to create AWS resources for the entire life of the cluster, so use key-based, long-lived credentials. To generate appropriate keys, see Manage Access Keys for IAM Users in the AWS documentation. We can supply the keys when we run the installation program.

• Download the AWS CLI and install it on our computer. See Install the AWS CLI Using the Bundled Installer (Linux, macOS, or Unix) in the AWS documentation.
• If a firewall is used, configure it to access Red Hat Insights.

Internet and Telemetry access

Telemetry provides metrics about cluster health and the success of updates. To perform subscription management, including legally entitling a purchase Red Hat, use the Telemetry service and access the OpenShift Infrastructure Providers page.

There is no disconnected subscription management. We cannot both opt out of sending data back to Red Hat and entitle a purchase.

Machines have direct internet access to install the cluster.

Internet access is required to:

• Access the OpenShift Infrastructure Providers page to download the installation program
• Access quay.io to obtain the packages required to install the cluster
• Obtain the packages required to perform cluster updates
• Access Red Hat's software as a service page to perform subscription management

Required AWS infrastructure components

To install OpenShift on user-provisioned infrastructure in Amazon Web Services (AWS), manually create both the machines and their supporting infrastructure.

For more information about the integration testing for different platforms, see the OpenShift 4.x Tested Integrations page.

We can use the provided CloudFormation templates to create this infrastructure, we can manually create the components, or we can reuse existing infrastructure that meets the cluster requirements. Review the CloudFormation templates for more details about how the components interrelate.

Cluster machines

You need AWS::EC2::Instance objects for the following machines:

• A bootstrap machine.

  This machine is required during installation, but we can remove it after the cluster deploys.

• At least three control plane machines.

  The control plane machines are not governed by a MachineSet.

• Compute machines.

  Create at least two compute, or worker, machines during installation. These machines are not governed by a MachineSet.

We can use the following instance types for the cluster machines: Valid instance types for machines

If m4 instance types are not available in your region, such as with eu-west-3, use m5 types instead.

Instance type	Bootstrap	Control plane	Compute
i3.large	x
m4.large or m5.large			x
m4.xlarge or m5.xlarge		x	x
m4.2xlarge		x	x
m4.4xlarge		x	x
m4.8xlarge		x	x
m4.10xlarge		x	x
m4.16xlarge		x	x
c4.large			x
c4.xlarge			x
c4.2xlarge		x	x
c4.4xlarge		x	x
c4.8xlarge		x	x
r4.large			x
r4.xlarge		x	x
r4.2xlarge		x	x
r4.4xlarge		x	x
r4.8xlarge		x	x
r4.16xlarge		x	x

Other infrastructure components

• A VPC
• DNS entries
• Load balancers and listeners
• A Route 53 zone
• Security groups
• IAM roles
• S3 buckets

Required VPC components

Provide a suitable VPC and subnets that allow communication to your machines.

Component	AWS type	Description
VPC	AWS::EC2::VPC AWS::EC2::VPCEndpoint	Provide a public VPC for the cluster to use. The VPC requires an endpoint that references the route tables for each subnet.
Public subnets	AWS::EC2::Subnet AWS::EC2::SubnetNetworkAclAssociation	Your VPC must have public subnets for between 1 and 3 availability zones and associate them with appropriate Ingress rules.
Internet gateway	AWS::EC2::InternetGateway AWS::EC2::VPCGatewayAttachment AWS::EC2::RouteTable AWS::EC2::Route PublicSubnetRouteTableAssociation AWS::EC2::NatGateway AWS::EC2::EIP	We must have a public internet gateway, with public routes, attached to the VPC. Each public subnet must also be attached to the route and have a NAT gateway and EIP address.
Network access control	AWS::EC2::NetworkAcl AWS::EC2::NetworkAclEntry	We must allow the VPC to access the following ports:
		Port	Reason
		80	Inbound HTTP traffic
		443	Inbound HTTPS traffic
		22	Inbound SSH traffic
		1024 - 65535	Inbound ephemeral traffic
		0 - 65535	Outbound ephemeral traffic
Private subnets	AWS::EC2::Subnet AWS::EC2::RouteTable AWS::EC2::SubnetRouteTableAssociation	Your VPC can have a private subnets. The provided CloudFormation templates can create private subnets for between 1 and 3 availability zones. If you use private subnets, provide appropriate routes and tables for them.

Required DNS and load balancing components

Your DNS and load balancer configuration needs to use a public hosted zone and can use a private hosted zone similar to the one that the installation program uses if it provisions the cluster's infrastructure. Create a DNS entry that resolves to your load balancer. An entry for api.cluster_name.domain must point to the external load balancer, and an entry for api-int.cluster_name.domain must point to the internal load balancer.

The cluster also requires load balancers and listeners for port 6443, which are required for the Kubernetes API and its extensions, and port 22623, which are required for the Ignition config files for new machines. The targets will be the master nodes. Port 6443 must be accessible to both clients external to the cluster and nodes within the cluster. Port 22623 must be accessible to nodes within the cluster.

Component	AWS type	Description
DNS	AWS::Route53::HostedZone	The hosted zone for your internal DNS.
etcd record sets	AWS::Route53::RecordSet	The registration records for etcd for your control plane machines.
Public load balancer	AWS::ElasticLoadBalancingV2::LoadBalancer	The load balancer for your public subnets.
External API server record	AWS::Route53::RecordSetGroup	Alias records for the external API server.
External listener	AWS::ElasticLoadBalancingV2::Listener	A listener on port 6443 for the external load balancer.
External target group	AWS::ElasticLoadBalancingV2::TargetGroup	The target group for the external load balancer.
Private load balancer	AWS::ElasticLoadBalancingV2::LoadBalancer	The load balancer for your private subnets.
Internal API server record	AWS::Route53::RecordSetGroup	Alias records for the internal API server.
Internal listener	AWS::ElasticLoadBalancingV2::Listener	A listener on port 22623 for the internal load balancer.
Internal target group	AWS::ElasticLoadBalancingV2::TargetGroup	The target group for the Internal load balancer.
Internal listener	AWS::ElasticLoadBalancingV2::Listener	A listener on port 6443 for the internal load balancer.
Internal target group	AWS::ElasticLoadBalancingV2::TargetGroup	The target group for the Internal load balancer.

Security groups

The control plane and worker machines require access to the following ports:

Group	Type	IP Protocol	Port range
MasterSecurityGroup	AWS::EC2::SecurityGroup	icmp	0
		tcp	22
		tcp	6443
		tcp	22623
WorkerSecurityGroup	AWS::EC2::SecurityGroup	icmp	0
		tcp	22
BootstrapSecurityGroup	AWS::EC2::SecurityGroup	tcp	22
		tcp	19531

Control plane Ingress

The control plane machines require the following Ingress groups. Each Ingress group is a AWS::EC2::SecurityGroupIngress resource.

Ingress group	Description	IP protocol	Port range
MasterIngressEtcd	etcd	tcp	2379- 2380
MasterIngressVxlan	Vxlan packets	udp	4789
MasterIngressWorkerVxlan	Vxlan packets	udp	4789
MasterIngressInternal	Internal cluster communication	tcp	9000 - 9999
MasterIngressWorkerInternal	Internal cluster communication	tcp	9000 - 9999
MasterIngressKube	Kubernetes kubelet, scheduler and controller manager	tcp	10250 - 10259
MasterIngressWorkerKube	Kubernetes kubelet, scheduler and controller manager	tcp	10250 - 10259
MasterIngressIngressServices	Kubernetes Ingress services	tcp	30000 - 32767
MasterIngressWorkerIngressServices	Kubernetes Ingress services	tcp	30000 - 32767

Worker Ingress

The worker machines require the following Ingress groups. Each Ingress group is a AWS::EC2::SecurityGroupIngress resource.

Ingress group	Description	IP protocol	Port range
WorkerIngressVxlan	Vxlan packets	udp	4789
WorkerIngressWorkerVxlan	Vxlan packets	udp	4789
WorkerIngressInternal	Internal cluster communication	tcp	9000 - 9999
WorkerIngressWorkerInternal	Internal cluster communication	tcp	9000 - 9999
WorkerIngressKube	Kubernetes kubelet, scheduler and controller manager	tcp	10250
WorkerIngressWorkerKube	Kubernetes kubelet, scheduler and controller manager	tcp	10250
WorkerIngressIngressServices	Kubernetes Ingress services	tcp	30000 - 32767
WorkerIngressWorkerIngressServices	Kubernetes Ingress services	tcp	30000 - 32767

Roles and instance profiles

Grant the machines permissions in AWS. The provided CloudFormation templates grant the machines permission the following AWS::IAM::Role objects and provide a AWS::IAM::InstanceProfile for each set of roles. If we do not use the templates, we can grant the machines the following broad permissions or the following individual permissions.

Role	Effect	Action	Resource
Master	Allow	ec2:*	*
	Allow	elasticloadbalancing:*	*
	Allow	iam:PassRole	*
	Allow	s3:GetObject	*
Worker	Allow	ec2:Describe*	*
Bootstrap	Allow	ec2:Describe*	*
	Allow	ec2:AttachVolume	*
	Allow	ec2:DetachVolume	*

Required AWS permissions

When you attach the AdministratorAccess policy to the IAM user that created, you grant that user all of the required permissions. To deploy an OpenShift cluster, the IAM user requires the following permissions:

Required EC2 permissions for installation

• ec2:AllocateAddress
• ec2:AssociateAddress
• ec2:AssociateDhcpOptions
• ec2:AssociateRouteTable
• ec2:AttachInternetGateway
• ec2:AuthorizeSecurityGroupEgress
• ec2:AuthorizeSecurityGroupIngress
• ec2:CopyImage
• ec2:CreateDhcpOptions
• ec2:CreateInternetGateway
• ec2:CreateNatGateway
• ec2:CreateRoute
• ec2:CreateRouteTable
• ec2:CreateSecurityGroup
• ec2:CreateSubnet
• ec2:CreateTags
• ec2:CreateVpc
• ec2:CreateVpcEndpoint
• ec2:CreateVolume
• ec2:DescribeAccountAttributes
• ec2:DescribeAddresses
• ec2:DescribeAvailabilityZones
• ec2:DescribeDhcpOptions
• ec2:DescribeImages
• ec2:DescribeInstanceAttribute
• ec2:DescribeInstanceCreditSpecifications
• ec2:DescribeInstances
• ec2:DescribeInternetGateways
• ec2:DescribeKeyPairs
• ec2:DescribeNatGateways
• ec2:DescribeNetworkAcls
• ec2:DescribePrefixLists
• ec2:DescribeRegions
• ec2:DescribeRouteTables
• ec2:DescribeSecurityGroups
• ec2:DescribeSubnets
• ec2:DescribeTags
• ec2:DescribeVpcEndpoints
• ec2:DescribeVpcs
• ec2:DescribeVpcAttribute
• ec2:DescribeVolumes
• ec2:DescribeVpcClassicLink
• ec2:DescribeVpcClassicLinkDnsSupport
• ec2:ModifyInstanceAttribute
• ec2:ModifySubnetAttribute
• ec2:ModifyVpcAttribute
• ec2:RevokeSecurityGroupEgress
• ec2:RunInstances
• ec2:TerminateInstances
• ec2:RevokeSecurityGroupIngress
• ec2:ReplaceRouteTableAssociation
• ec2:DescribeNetworkInterfaces
• ec2:ModifyNetworkInterfaceAttribute

Required Elasticloadbalancing permissions for installation

• elasticloadbalancing:AddTags
• elasticloadbalancing:ApplySecurityGroupsToLoadBalancer
• elasticloadbalancing:AttachLoadBalancerToSubnets
• elasticloadbalancing:CreateListener
• elasticloadbalancing:CreateLoadBalancer
• elasticloadbalancing:CreateLoadBalancerListeners
• elasticloadbalancing:CreateTargetGroup
• elasticloadbalancing:ConfigureHealthCheck
• elasticloadbalancing:DeregisterInstancesFromLoadBalancer
• elasticloadbalancing:DeregisterTargets
• elasticloadbalancing:DescribeInstanceHealth
• elasticloadbalancing:DescribeListeners
• elasticloadbalancing:DescribeLoadBalancers
• elasticloadbalancing:DescribeLoadBalancerAttributes
• elasticloadbalancing:DescribeTags
• elasticloadbalancing:DescribeTargetGroupAttributes
• elasticloadbalancing:DescribeTargetHealth
• elasticloadbalancing:ModifyLoadBalancerAttributes
• elasticloadbalancing:ModifyTargetGroup
• elasticloadbalancing:ModifyTargetGroupAttributes
• elasticloadbalancing:RegisterTargets
• elasticloadbalancing:RegisterInstancesWithLoadBalancer
• elasticloadbalancing:SetLoadBalancerPoliciesOfListener

Required IAM permissions for installation

• iam:AddRoleToInstanceProfile
• iam:CreateInstanceProfile
• iam:CreateRole
• iam:DeleteInstanceProfile
• iam:DeleteRole
• iam:DeleteRolePolicy
• iam:GetInstanceProfile
• iam:GetRole
• iam:GetRolePolicy
• iam:GetUser
• iam:ListInstanceProfilesForRole
• iam:ListRoles
• iam:ListUsers
• iam:PassRole
• iam:PutRolePolicy
• iam:RemoveRoleFromInstanceProfile
• iam:SimulatePrincipalPolicy
• iam:TagRole

Required Route53 permissions for installation

• route53:ChangeResourceRecordSets
• route53:ChangeTagsForResource
• route53:GetChange
• route53:GetHostedZone
• route53:CreateHostedZone
• route53:ListHostedZones
• route53:ListHostedZonesByName
• route53:ListResourceRecordSets
• route53:ListTagsForResource
• route53:UpdateHostedZoneComment

Required S3 permissions for installation

• s3:CreateBucket
• s3:DeleteBucket
• s3:GetAccelerateConfiguration
• s3:GetBucketCors
• s3:GetBucketLocation
• s3:GetBucketLogging
• s3:GetBucketObjectLockConfiguration
• s3:GetBucketReplication
• s3:GetBucketRequestPayment
• s3:GetBucketTagging
• s3:GetBucketVersioning
• s3:GetBucketWebsite
• s3:GetEncryptionConfiguration
• s3:GetLifecycleConfiguration
• s3:GetReplicationConfiguration
• s3:ListBucket
• s3:PutBucketAcl
• s3:PutBucketTagging
• s3:PutEncryptionConfiguration

S3 permissions that cluster Operators require

• s3:PutObject
• s3:PutObjectAcl
• s3:PutObjectTagging
• s3:GetObject
• s3:GetObjectAcl
• s3:GetObjectTagging
• s3:GetObjectVersion
• s3:DeleteObject

All additional permissions required to uninstall a cluster

• autoscaling:DescribeAutoScalingGroups
• ec2:DeleteDhcpOptions
• ec2:DeleteInternetGateway
• ec2:DeleteNatGateway
• ec2:DeleteNetworkInterface
• ec2:DeleteRoute
• ec2:DeleteRouteTable
• ec2:DeleteSnapshot
• ec2:DeleteSecurityGroup
• ec2:DeleteSubnet
• ec2:DeleteVolume
• ec2:DeleteVpc
• ec2:DeleteVpcEndpoints
• ec2:DeregisterImage
• ec2:DetachInternetGateway
• ec2:DisassociateRouteTable
• ec2:ReleaseAddress
• elasticloadbalancing:DescribeTargetGroups
• elasticloadbalancing:DeleteTargetGroup
• elasticloadbalancing:DeleteLoadBalancer
• iam:ListInstanceProfiles
• iam:ListRolePolicies
• iam:ListUserPolicies
• route53:DeleteHostedZone
• tag:GetResources

Obtain the installation program

Before installing OpenShift, download the installation file on a local computer.

Prerequisites

• Install the cluster from a computer that uses Linux or macOS.
• You need 300 MB of local disk space to download the installation program.

Procedure

1. Access the OpenShift Infrastructure Providers page. If we have a Red Hat account, log in with your credentials. If we do not, create an account.

2. Download the installation program for your operating system and place the file in the directory where you will store the installation configuration files.

   The installation program creates several files on the computer used to install the cluster. Keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run...

   $ tar xvf installation_program.tar.gz
   

4. From the OpenShift Infrastructure Providers page, download your installation pull secret. This pull secret allows us to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift components.

Generate an SSH private key and add it to the agent

For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key for the ssh-agent process.

We can use this key to SSH onto the master nodes as the user core. When we deploy the cluster, the key is added to the core user's ~/.ssh/authorized_keys list.

Use a local key, not one configured with platform-specific approaches such as AWS key pairs.

Procedure

1. If we do not have an SSH key configured for password-less authentication on the computer, create one. For example, on a computer that uses a Linux operating system, run:
   $ ssh-keygen -t rsa -b 4096 -N '' -f /path/to/.ssh/id_rsa

2. Start the ssh-agent process as a background task:
   $ eval "$(ssh-agent -s)"
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:
   $ ssh-add /path/to/.ssh/id_rsa
   Identity added: /path/to/.ssh/id_rsa (computer_name)

Next steps

When we install OpenShift, provide the SSH public key to the installer. If we install a cluster on infrastructure that we provision, provide this key to the cluster's machines.

Create the installation files for AWS

To install OpenShift on Amazon Web Services (AWS) using user-provisioned infrastructure, generate the files that the installation program needs to deploy the cluster and modify them so that the cluster creates only the machines that it will use. You generate and customize the install_config.yaml file, Kubernetes manifests, and Ignition config files.

The Ignition config files that the installation program generates contain certificates that expire after 24 hours. Complete the cluster installation and keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

Prerequisites

• Obtain the OpenShift installation program and the pull secret for the cluster.

Procedure

1. Obtain the install-config.yaml file.

   1. Run the following command:

      $ ./openshift-install create install-config --dir=/path/to/install/files 
      

      Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so do not reuse an installation directory. To reuse individual files from another cluster installation, copy them into the directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift version.

   2. At the prompts, provide the configuration details for your cloud:

      1. Optional: Select an SSH key to use to access the cluster machines.

         For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.

      2. Select AWS as the platform to target.
      3. If we do not have an AWS profile stored on our computer, enter the AWS access key ID and secret access key for the user configured to run the installation program.
      4. Select the AWS region to deploy the cluster to.
      5. Select the base domain for the Route53 service configured for the cluster.
      6. Enter a descriptive name for the cluster.
      7. Paste the pull secret obtained from the OpenShift Infrastructure Providers page.

2. Edit the install-config.yaml file to set the number of compute, or worker, replicas to 0, as shown in the following compute stanza:

   compute:
   - hyperthreading: Enabled
     name: worker
     platform: {}
     replicas: 0
   

3. Optional: Back up the install-config.yaml file.

   The install-config.yaml file is consumed during the next step. To reuse the file, back it up now.

4. Remove the Kubernetes manifest files for the control plane machines. By removing these files, you prevent the cluster from automatically generating control plane machines.

   1. Generate the Kubernetes manifests for the cluster:

      $ ./openshift-install create manifests --dir=/path/to/install/files 
      
      WARNING There are no compute nodes specified. The cluster will not fully initialize without compute nodes.
      INFO Consuming "Install Config" from target directory
      

      Because create our own compute machines later in the installation process, we can safely ignore this warning.

   2. Remove the files that define the control plane machines:
      $ rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml

5. Remove the Kubernetes manifest files that define the worker machines:
   $ rm -f openshift/99_openshift-cluster-api_worker-machineset-*

   Because we create and manage the worker machines ourselves, we do not need to initialize these machines.

6. Obtain the Ignition config files:
   $ ./openshift-install create ignition-configs --dir=/path/to/install/files

   The following files are generated in the directory:

   .
   auth
         kubeadmin-password
         kubeconfig
   bootstrap.ign
   master.ign
   metadata.json
   worker.ign
   

Extract the infrastructure name

The Ignition configs contain a unique cluster identifier that we can use to uniquely identify the cluster in Amazon Web Services (AWS) tags. The provided CloudFormation templates contain references to this tag, so we must extract it.

Prerequisites

• Obtain the OpenShift installation program and the pull secret for the cluster.
• Generate the Ignition config files for the cluster.
• Install the jq package.

Procedure

• Extract the infrastructure name from the Ignition config file metadata:
  $ jq -r .infraID //path/to/install/files/metadata.json openshift-vw9j6

  You need the output of this command to configure the provided CloudFormation templates and can use it in other AWS tags.

Create a VPC in AWS

Create a VPC in Amazon Web Services (AWS) for the OpenShift cluster to use. We can customize the VPC to meet your requirements, including VPN and route tables. The easiest way to create the VPC is to modify the provided CloudFormation template.

If we do not use the provided CloudFormation template to create your AWS infrastructure, review the provided information and manually create the infrastructure. If the cluster does not initialize correctly, we might have to contact Red Hat support with your installation logs.

Prerequisites

• Configure an AWS account.
• Generate the Ignition config files for the cluster.

Procedure

1. Create a JSON file containing the parameter values that the template requires:

   [
     {
       "ParameterKey": "VpcCidr", 1
       "ParameterValue": "10.0.0.0/16" 2
     },
     {
       "ParameterKey": "AvailabilityZoneCount", 3
       "ParameterValue": "1" 4
     },
     {
       "ParameterKey": "SubnetBits", 5
       "ParameterValue": "12" 6
     }
   ]
   

   1 The CIDR block for the VPC.

   2 Specify a CIDR block in the format x.x.x.x/16-24.

   3 The number of availability zones to deploy the VPC in.

   4 Specify an integer between 1 and 3.

   5 The size of each subnet in each availability zone.

   6 Specify an integer between 5 and 13, where 5 is /27 and 13 is /19.

2. Copy the template from the CloudFormation template for the VPC section of this topic and save it as a YAML file on our computer. This template describes the VPC that the cluster requires.

3. Launch the template:

   Enter the command on a single line.

   $ aws cloudformation create-stack --stack-name name 1
        --template-body file://template.yaml 2
        --parameters file://parameters.json 3
   

   1 name is the name for the CloudFormation stack, such as cluster-vpc. You need the name of this stack if you remove the cluster.

   2 template is the relative path to and name of the CloudFormation template YAML file that you saved.

   3 parameters is the relative path to and name of the CloudFormation parameters JSON file.

4. Confirm that the template components exist:

   $ aws cloudformation describe-stacks --stack-name name
   

   After the StackStatus displays CREATE_COMPLETE, the output displays values for the following parameters. Provide these parameter values to the other CloudFormation templates that you run to create the cluster:

   VpcId	The ID of the VPC.
   PublicSubnetIds	The IDs of the new public subnets.
   PrivateSubnetIds	The IDs of the new private subnets.

CloudFormation template for the VPC

We can use the following CloudFormation template to deploy the VPC required you need for the OpenShift cluster.

AWSTemplateFormatVersion: 2010-09-09
Description: Template for Best Practice VPC with 1-3 AZs

Parameters:
  VpcCidr:
    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
    ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-24.
    Default: 10.0.0.0/16
    Description: CIDR block for VPC.
    Type: String
  AvailabilityZoneCount:
    ConstraintDescription: "The number of availability zones. (Min: 1, Max: 3)"
    MinValue: 1
    MaxValue: 3
    Default: 1
    Description: "How many AZs to create VPC subnets for. (Min: 1, Max: 3)"
    Type: Number
  SubnetBits:
    ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/19-27.
    MinValue: 5
    MaxValue: 13
    Default: 12
    Description: "Size of each subnet to create within the availability zones. (Min: 5 = /27, Max: 13 = /19)"
    Type: Number

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
    - Label:
default: "Network Configuration"
      Parameters:
      - VpcCidr
      - SubnetBits
    - Label:
default: "Availability Zones"
      Parameters:
      - AvailabilityZoneCount
    ParameterLabels:
      AvailabilityZoneCount:
default: "Availability Zone Count"
      VpcCidr:
default: "VPC CIDR"
      SubnetBits:
default: "Bits Per Subnet"

Conditions:
  DoAz3: !Equals [3, !Ref AvailabilityZoneCount]
  DoAz2: !Or [!Equals [2, !Ref AvailabilityZoneCount], Condition: DoAz3]

Resources:
  VPC:
    Type: "AWS::EC2::VPC"
    Properties:
      EnableDnsSupport: "true"
      EnableDnsHostnames: "true"
      CidrBlock: !Ref VpcCidr
  PublicSubnet:
    Type: "AWS::EC2::Subnet"
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !Select [0, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
      AvailabilityZone: !Select
      - 0
      - Fn::GetAZs: !Ref "AWS::Region"
  PublicSubnet2:
    Type: "AWS::EC2::Subnet"
    Condition: DoAz2
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !Select [1, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
      AvailabilityZone: !Select
      - 1
      - Fn::GetAZs: !Ref "AWS::Region"
  PublicSubnet3:
    Type: "AWS::EC2::Subnet"
    Condition: DoAz3
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !Select [2, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
      AvailabilityZone: !Select
      - 2
      - Fn::GetAZs: !Ref "AWS::Region"
  InternetGateway:
    Type: "AWS::EC2::InternetGateway"
  GatewayToInternet:
    Type: "AWS::EC2::VPCGatewayAttachment"
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref InternetGateway
  PublicRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VPC
  PublicRoute:
    Type: "AWS::EC2::Route"
    DependsOn: GatewayToInternet
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway
  PublicSubnetRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      SubnetId: !Ref PublicSubnet
      RouteTableId: !Ref PublicRouteTable
  PublicSubnetRouteTableAssociation2:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Condition: DoAz2
    Properties:
      SubnetId: !Ref PublicSubnet2
      RouteTableId: !Ref PublicRouteTable
  PublicSubnetRouteTableAssociation3:
    Condition: DoAz3
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      SubnetId: !Ref PublicSubnet3
      RouteTableId: !Ref PublicRouteTable
  PrivateSubnet:
    Type: "AWS::EC2::Subnet"
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !Select [3, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
      AvailabilityZone: !Select
      - 0
      - Fn::GetAZs: !Ref "AWS::Region"
  PrivateRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VPC
  PrivateSubnetRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      SubnetId: !Ref PrivateSubnet
      RouteTableId: !Ref PrivateRouteTable
  NAT:
    DependsOn:
    - GatewayToInternet
    Type: "AWS::EC2::NatGateway"
    Properties:
      AllocationId:
"Fn::GetAtt":
- EIP
- AllocationId
      SubnetId: !Ref PublicSubnet
  EIP:
    Type: "AWS::EC2::EIP"
    Properties:
      Domain: vpc
  Route:
    Type: "AWS::EC2::Route"
    Properties:
      RouteTableId:
Ref: PrivateRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId:
Ref: NAT
  PrivateSubnet2:
    Type: "AWS::EC2::Subnet"
    Condition: DoAz2
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !Select [4, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
      AvailabilityZone: !Select
      - 1
      - Fn::GetAZs: !Ref "AWS::Region"
  PrivateRouteTable2:
    Type: "AWS::EC2::RouteTable"
    Condition: DoAz2
    Properties:
      VpcId: !Ref VPC
  PrivateSubnetRouteTableAssociation2:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Condition: DoAz2
    Properties:
      SubnetId: !Ref PrivateSubnet2
      RouteTableId: !Ref PrivateRouteTable2
  NAT2:
    DependsOn:
    - GatewayToInternet
    Type: "AWS::EC2::NatGateway"
    Condition: DoAz2
    Properties:
      AllocationId:
"Fn::GetAtt":
- EIP2
- AllocationId
      SubnetId: !Ref PublicSubnet2
  EIP2:
    Type: "AWS::EC2::EIP"
    Condition: DoAz2
    Properties:
      Domain: vpc
  Route2:
    Type: "AWS::EC2::Route"
    Condition: DoAz2
    Properties:
      RouteTableId:
Ref: PrivateRouteTable2
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId:
Ref: NAT2
  PrivateSubnet3:
    Type: "AWS::EC2::Subnet"
    Condition: DoAz3
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !Select [5, !Cidr [!Ref VpcCidr, 6, !Ref SubnetBits]]
      AvailabilityZone: !Select
      - 2
      - Fn::GetAZs: !Ref "AWS::Region"
  PrivateRouteTable3:
    Type: "AWS::EC2::RouteTable"
    Condition: DoAz3
    Properties:
      VpcId: !Ref VPC
  PrivateSubnetRouteTableAssociation3:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Condition: DoAz3
    Properties:
      SubnetId: !Ref PrivateSubnet3
      RouteTableId: !Ref PrivateRouteTable3
  NAT3:
    DependsOn:
    - GatewayToInternet
    Type: "AWS::EC2::NatGateway"
    Condition: DoAz3
    Properties:
      AllocationId:
"Fn::GetAtt":
- EIP3
- AllocationId
      SubnetId: !Ref PublicSubnet3
  EIP3:
    Type: "AWS::EC2::EIP"
    Condition: DoAz3
    Properties:
      Domain: vpc
  Route3:
    Type: "AWS::EC2::Route"
    Condition: DoAz3
    Properties:
      RouteTableId:
Ref: PrivateRouteTable3
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId:
Ref: NAT3
  S3Endpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
  Principal: '*'
  Action:
  - '*'
  Resource:
  - '*'
      RouteTableIds:
      - !Ref PublicRouteTable
      - !Ref PrivateRouteTable
      - !If [DoAz2, !Ref PrivateRouteTable2, !Ref "AWS::NoValue"]
      - !If [DoAz3, !Ref PrivateRouteTable3, !Ref "AWS::NoValue"]
      ServiceName: !Join
      - ''
      - - com.amazonaws.
- !Ref 'AWS::Region'
- .s3
      VpcId: !Ref VPC

Outputs:
  VpcId:
    Description: ID of the new VPC.
    Value: !Ref VPC
  PublicSubnetIds:
    Description: Subnet IDs of the public subnets.
    Value:
      !Join [
",",
[!Ref PublicSubnet, !If [DoAz2, !Ref PublicSubnet2, !Ref "AWS::NoValue"], !If [DoAz3, !Ref PublicSubnet3, !Ref "AWS::NoValue"]]
      ]
  PrivateSubnetIds:
    Description: Subnet IDs of the private subnets.
    Value:
      !Join [
",",
[!Ref PrivateSubnet, !If [DoAz2, !Ref PrivateSubnet2, !Ref "AWS::NoValue"], !If [DoAz3, !Ref PrivateSubnet3, !Ref "AWS::NoValue"]]
      ]

Create networking and load balancing components in AWS

Configure networking and load balancing in Amazon Web Services (AWS) for the OpenShift cluster to use. The easiest way to create these components is to modify the provided CloudFormation template, which also creates a hosted zone and subnet tags.

We can run the template multiple times within a single VPC.

If we do not use the provided CloudFormation template to create your AWS infrastructure, review the provided information and manually create the infrastructure. If the cluster does not initialize correctly, we might have to contact Red Hat support with your installation logs.

Prerequisites

• Configure an AWS account.
• Generate the Ignition config files for the cluster.
• Create and configure a VPC and associated subnets in AWS.

Procedure

1. Obtain the Hosted Zone ID for the the Route53 zone that you specified in the install-config.yaml file for the cluster. We can obtain this ID from the AWS console or by running the following command:

   Enter the command on a single line.

   $ aws route53 list-hosted-zones-by-name |
        jq --arg name "route53_domain." \ 1
        -r '.HostedZones | .[] | select(.Name=="\($name)") | .Id'
   

   1 For the route53_domain, specify the Route53 base domain that you used when you generated the install-config.yaml file for the cluster.

2. Create a JSON file containing the parameter values that the template requires:

   [
     {
       "ParameterKey": "ClusterName", 1
       "ParameterValue": "mycluster" 2
     },
     {
       "ParameterKey": "InfrastructureName", 3
       "ParameterValue": "mycluster-random_string" 4
     },
     {
       "ParameterKey": "HostedZoneId", 5
       "ParameterValue": "random_string" 6
     },
     {
       "ParameterKey": "HostedZoneName", 7
       "ParameterValue": "example.com" 8
     },
     {
       "ParameterKey": "PublicSubnets", 9
       "ParameterValue": "subnet-random_string" 10
     },
     {
       "ParameterKey": "PrivateSubnets", 11
       "ParameterValue": "subnet-random_string" 12
     },
     {
       "ParameterKey": "VpcId", 13
       "ParameterValue": "vpc-random_string" 14
     }
   ]
   

   1 A short, representative cluster name to use for host names, etc.

   2 Specify the cluster name that you used when you generated the install-config.yaml file for the cluster.

   3 The name for the cluster infrastructure that is encoded in your Ignition config files for the cluster.

   4 Specify the infrastructure name that you extracted from the Ignition config file metadata, which has the format cluster-name-random-string.

   5 The Route53 public zone ID to register the targets with.

   6 Specify the Route53 public zone ID, which as a format similar to Z21IXYZABCZ2A4. We can obtain this value from the AWS console.

   7 The Route53 zone to register the targets with.

   8 Specify the Route53 base domain that you used when you generated the install-config.yaml file for the cluster. Do not include the trailing period (.) that is displayed in the AWS console.

   9 The public subnets that createdd for your VPC.

   10 Specify the PublicSubnetIds value from the output of the CloudFormation template for the VPC.

   11 The private subnets that createdd for your VPC.

   12 Specify the PrivateSubnetIds value from the output of the CloudFormation template for the VPC.

   13 The VPC that createdd for the cluster.

   14 Specify the VpcId value from the output of the CloudFormation template for the VPC.

3. Copy the template from the CloudFormation template for the network and load balancers section of this topic and save it as a YAML file on our computer. This template describes the networking and load balancing objects that the cluster requires.

4. Launch the template:

   Enter the command on a single line.

   $ aws cloudformation create-stack --stack-name name 1
        --template-body file://template.yaml 2
        --parameters file://parameters.json 3
        --capabilities CAPABILITY_NAMED_IAM
   

   1 name is the name for the CloudFormation stack, such as cluster-dns. You need the name of this stack if you remove the cluster.

   2 template is the relative path to and name of the CloudFormation template YAML file that you saved.

   3 parameters is the relative path to and name of the CloudFormation parameters JSON file.

5. Confirm that the template components exist:

   $ aws cloudformation describe-stacks --stack-name name
   

   After the StackStatus displays CREATE_COMPLETE, the output displays values for the following parameters. Provide these parameter values to the other CloudFormation templates that you run to create the cluster:

   PrivateHostedZoneId	Hosted zone ID for the private DNS.
   ExternalApiLoadBalancerName	Full name of the external API load balancer.
   InternalApiLoadBalancerName	Full name of the internal API load balancer.
   ApiServerDnsName	Full host name of the API server.
   RegisterNlbIpTargetsLambda	Lambda ARN useful to help register/deregister IP targets for these load balancers.
   ExternalApiTargetGroupArn	ARN of external API target group.
   InternalApiTargetGroupArn	ARN of internal API target group.
   InternalServiceTargetGroupArn	ARN of internal service target group.

CloudFormation template for the network and load balancers

We can use the following CloudFormation template to deploy the networking objects and load balancers required for the OpenShift cluster.

AWSTemplateFormatVersion: 2010-09-09
Description: Template for OpenShift Cluster Network Elements (Route53 & LBs)

Parameters:
  ClusterName:
    AllowedPattern: ^([a-zA-Z][a-zA-Z0-9\-]{0,26})$
    MaxLength: 27
    MinLength: 1
    ConstraintDescription: Cluster name must be alphanumeric, start with a letter, and have a maximum of 27 characters.
    Description: A short, representative cluster name to use for host names and other identifying names.
    Type: String
  InfrastructureName:
    AllowedPattern: ^([a-zA-Z][a-zA-Z0-9\-]{0,26})$
    MaxLength: 27
    MinLength: 1
    ConstraintDescription: Infrastructure name must be alphanumeric, start with a letter, and have a maximum of 27 characters.
    Description: A short, unique cluster ID used to tag cloud resources and identify items owned or used by the cluster.
    Type: String
  HostedZoneId:
    Description: The Route53 public zone ID to register the targets with, such as Z21IXYZABCZ2A4.
    Type: String
  HostedZoneName:
    Description: The Route53 zone to register the targets with, such as example.com. Omit the trailing period.
    Type: String
    Default: "example.com"
  PublicSubnets:
    Description: The internet-facing subnets.
    Type: ListAWS::EC2::Subnet::Id
  PrivateSubnets:
    Description: The internal subnets.
    Type: ListAWS::EC2::Subnet::Id
  VpcId:
    Description: The VPC-scoped resources will belong to this VPC.
    Type: AWS::EC2::VPC::Id

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
    - Label:
default: "Cluster Information"
      Parameters:
      - ClusterName
      - InfrastructureName
    - Label:
default: "Network Configuration"
      Parameters:
      - VpcId
      - PublicSubnets
      - PrivateSubnets
    - Label:
default: "DNS"
      Parameters:
      - HostedZoneName
      - HostedZoneId
    ParameterLabels:
      ClusterName:
default: "Cluster Name"
      InfrastructureName:
default: "Infrastructure Name"
      VpcId:
default: "VPC ID"
      PublicSubnets:
default: "Public Subnets"
      PrivateSubnets:
default: "Private Subnets"
      HostedZoneName:
default: "Public Hosted Zone Name"
      HostedZoneId:
default: "Public Hosted Zone ID"

Resources:
  ExtApiElb:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Join ["-", [!Ref InfrastructureName, "ext"]]
      IpAddressType: ipv4
      Subnets: !Ref PublicSubnets
      Type: network

  IntApiElb:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Join ["-", [!Ref InfrastructureName, "int"]]
      Scheme: internal
      IpAddressType: ipv4
      Subnets: !Ref PrivateSubnets
      Type: network

  IntDns:
    Type: "AWS::Route53::HostedZone"
    Properties:
      HostedZoneConfig:
Comment: "Managed by CloudFormation"
      Name: !Join [".", [!Ref ClusterName, !Ref HostedZoneName]]
      HostedZoneTags:
      - Key: Name
Value: !Join ["-", [!Ref InfrastructureName, "int"]]
      - Key: !Join ["", ["kubernetes.io/cluster/", !Ref InfrastructureName]]
Value: "owned"
      VPCs:
      - VPCId: !Ref VpcId
VPCRegion: !Ref "AWS::Region"

  ExternalApiServerRecord:
    Type: AWS::Route53::RecordSetGroup
    Properties:
      Comment: Alias record for the API server
      HostedZoneId: !Ref HostedZoneId
      RecordSets:
      - Name:
  !Join [
".",
["api", !Ref ClusterName, !Join ["", [!Ref HostedZoneName, "."]]],
  ]
Type: A
AliasTarget:
  HostedZoneId: !GetAtt ExtApiElb.CanonicalHostedZoneID
  DNSName: !GetAtt ExtApiElb.DNSName

  InternalApiServerRecord:
    Type: AWS::Route53::RecordSetGroup
    Properties:
      Comment: Alias record for the API server
      HostedZoneId: !Ref IntDns
      RecordSets:
      - Name:
  !Join [
".",
["api", !Ref ClusterName, !Join ["", [!Ref HostedZoneName, "."]]],
  ]
Type: A
AliasTarget:
  HostedZoneId: !GetAtt IntApiElb.CanonicalHostedZoneID
  DNSName: !GetAtt IntApiElb.DNSName
      - Name:
  !Join [
".",
["api-int", !Ref ClusterName, !Join ["", [!Ref HostedZoneName, "."]]],
  ]
Type: A
AliasTarget:
  HostedZoneId: !GetAtt IntApiElb.CanonicalHostedZoneID
  DNSName: !GetAtt IntApiElb.DNSName

  ExternalApiListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
      - Type: forward
TargetGroupArn:
  Ref: ExternalApiTargetGroup
      LoadBalancerArn:
Ref: ExtApiElb
      Port: 6443
      Protocol: TCP

  ExternalApiTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: 6443
      Protocol: TCP
      TargetType: ip
      VpcId:
Ref: VpcId
      TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
Value: 60

  InternalApiListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
      - Type: forward
TargetGroupArn:
  Ref: InternalApiTargetGroup
      LoadBalancerArn:
Ref: IntApiElb
      Port: 6443
      Protocol: TCP

  InternalApiTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: 6443
      Protocol: TCP
      TargetType: ip
      VpcId:
Ref: VpcId
      TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
Value: 60

  InternalServiceInternalListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
      - Type: forward
TargetGroupArn:
  Ref: InternalServiceTargetGroup
      LoadBalancerArn:
Ref: IntApiElb
      Port: 22623
      Protocol: TCP

  InternalServiceTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: 22623
      Protocol: TCP
      TargetType: ip
      VpcId:
Ref: VpcId
      TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
Value: 60

  RegisterTargetLambdaIamRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Join ["-", [!Ref InfrastructureName, "nlb", "lambda", "role"]]
      AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
  Principal:
Service:
- "lambda.amazonaws.com"
  Action:
  - "sts:AssumeRole"
      Path: "/"
      Policies:
      - PolicyName: !Join ["-", [!Ref InfrastructureName, "master", "policy"]]
PolicyDocument:
  Version: "2012-10-17"
  Statement:
  - Effect: "Allow"
Action:
  [
    "elasticloadbalancing:RegisterTargets",
    "elasticloadbalancing:DeregisterTargets",
  ]
Resource: !Ref InternalApiTargetGroup
  - Effect: "Allow"
Action:
  [
    "elasticloadbalancing:RegisterTargets",
    "elasticloadbalancing:DeregisterTargets",
  ]
Resource: !Ref InternalServiceTargetGroup
  - Effect: "Allow"
Action:
  [
    "elasticloadbalancing:RegisterTargets",
    "elasticloadbalancing:DeregisterTargets",
  ]
Resource: !Ref ExternalApiTargetGroup

  RegisterNlbIpTargets:
    Type: "AWS::Lambda::Function"
    Properties:
      Handler: "index.handler"
      Role:
Fn::GetAtt:
- "RegisterTargetLambdaIamRole"
- "Arn"
      Code:
ZipFile: |
  import json
  import boto3
  import cfnresponse
  def handler(event, context):
elb = boto3.client('elbv2')
if event['RequestType'] == 'Delete':
  elb.deregister_targets(TargetGroupArn=event['ResourceProperties']['TargetArn'],Targets=[{'Id': event['ResourceProperties']['TargetIp']}])
elif event['RequestType'] == 'Create':
  elb.register_targets(TargetGroupArn=event['ResourceProperties']['TargetArn'],Targets=[{'Id': event['ResourceProperties']['TargetIp']}])
responseData = {}
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, event['ResourceProperties']['TargetArn']+event['ResourceProperties']['TargetIp'])
      Runtime: "python3.7"
      Timeout: 120

  RegisterSubnetTagsLambdaIamRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Join ["-", [!Ref InfrastructureName, "subnet-tags-lambda-role"]]
      AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
  Principal:
Service:
- "lambda.amazonaws.com"
  Action:
  - "sts:AssumeRole"
      Path: "/"
      Policies:
      - PolicyName: !Join ["-", [!Ref InfrastructureName, "subnet-tagging-policy"]]
PolicyDocument:
  Version: "2012-10-17"
  Statement:
  - Effect: "Allow"
Action:
  [
    "ec2:DeleteTags",
    "ec2:CreateTags"
  ]
Resource: "arn:aws:ec2:*:*:subnet/*"
  - Effect: "Allow"
Action:
  [
    "ec2:DescribeSubnets",
    "ec2:DescribeTags"
  ]
Resource: "*"

  RegisterSubnetTags:
    Type: "AWS::Lambda::Function"
    Properties:
      Handler: "index.handler"
      Role:
Fn::GetAtt:
- "RegisterSubnetTagsLambdaIamRole"
- "Arn"
      Code:
ZipFile: |
  import json
  import boto3
  import cfnresponse
  def handler(event, context):
ec2_client = boto3.client('ec2')
if event['RequestType'] == 'Delete':
  for subnet_id in event['ResourceProperties']['Subnets']:
    ec2_client.delete_tags(Resources=[subnet_id], Tags=[{'Key': 'kubernetes.io/cluster/' + event['ResourceProperties']['InfrastructureName']}]);
elif event['RequestType'] == 'Create':
  for subnet_id in event['ResourceProperties']['Subnets']:
    ec2_client.create_tags(Resources=[subnet_id], Tags=[{'Key': 'kubernetes.io/cluster/' + event['ResourceProperties']['InfrastructureName'], 'Value': 'shared'}]);
responseData = {}
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, event['ResourceProperties']['InfrastructureName']+event['ResourceProperties']['Subnets'][0])
      Runtime: "python3.7"
      Timeout: 120

  RegisterPublicSubnetTags:
    Type: Custom::SubnetRegister
    Properties:
      ServiceToken: !GetAtt RegisterSubnetTags.Arn
      InfrastructureName: !Ref InfrastructureName
      Subnets: !Ref PublicSubnets

  RegisterPrivateSubnetTags:
    Type: Custom::SubnetRegister
    Properties:
      ServiceToken: !GetAtt RegisterSubnetTags.Arn
      InfrastructureName: !Ref InfrastructureName
      Subnets: !Ref PrivateSubnets

Outputs:
  PrivateHostedZoneId:
    Description: Hosted zone ID for the private DNS, which is required for private records.
    Value: !Ref IntDns
  ExternalApiLoadBalancerName:
    Description: Full name of the External API load balancer created.
    Value: !GetAtt ExtApiElb.LoadBalancerFullName
  InternalApiLoadBalancerName:
    Description: Full name of the Internal API load balancer created.
    Value: !GetAtt IntApiElb.LoadBalancerFullName
  ApiServerDnsName:
    Description: Full hostname of the API server, which is required for the Ignition config files.
    Value: !Join [".", ["api-int", !Ref ClusterName, !Ref HostedZoneName]]
  RegisterNlbIpTargetsLambda:
    Description: Lambda ARN useful to help register or deregister IP targets for these load balancers.
    Value: !GetAtt RegisterNlbIpTargets.Arn
  ExternalApiTargetGroupArn:
    Description: ARN of External API target group.
    Value: !Ref ExternalApiTargetGroup
  InternalApiTargetGroupArn:
    Description: ARN of Internal API target group.
    Value: !Ref InternalApiTargetGroup
  InternalServiceTargetGroupArn:
    Description: ARN of internal service target group.
    Value: !Ref InternalServiceTargetGroup

Create security group and roles in AWS

Create security groups and roles in Amazon Web Services (AWS) for the OpenShift cluster to use. The easiest way to create these components is to modify the provided CloudFormation template.

If we do not use the provided CloudFormation template to create your AWS infrastructure, review the provided information and manually create the infrastructure. If the cluster does not initialize correctly, we might have to contact Red Hat support with your installation logs.

Prerequisites

• Configure an AWS account.
• Generate the Ignition config files for the cluster.
• Create and configure a VPC and associated subnets in AWS.

Procedure

1. Create a JSON file containing the parameter values that the template requires:

   [
     {
       "ParameterKey": "InfrastructureName", 1
       "ParameterValue": "mycluster-random_string" 2
     },
     {
       "ParameterKey": "VpcCidr", 3
       "ParameterValue": "10.0.0.0/16" 4
     },
     {
       "ParameterKey": "PrivateSubnets", 5
       "ParameterValue": "subnet-random_string" 6
     },
     {
       "ParameterKey": "VpcId", 7
       "ParameterValue": "vpc-random_string" 8
     }
   ]
   

   1 The name for the cluster infrastructure that is encoded in your Ignition config files for the cluster.

   2 Specify the infrastructure name that you extracted from the Ignition config file metadata, which has the format cluster-name-random-string.

   3 The CIDR block for the VPC.

   4 Specify the CIDR block parameter that you used for the VPC that you defined in the form x.x.x.x/16-24.

   5 The private subnets that createdd for your VPC.

   6 Specify the PrivateSubnetIds value from the output of the CloudFormation template for the VPC.

   7 The VPC that createdd for the cluster.

   8 Specify the VpcId value from the output of the CloudFormation template for the VPC.

2. Copy the template from the CloudFormation template for security objects section of this topic and save it as a YAML file on our computer. This template describes the security groups and roles that the cluster requires.

3. Launch the template:

   Enter the command on a single line.

   $ aws cloudformation create-stack --stack-name name 1
        --template-body file://template.yaml 2
        --parameters file://parameters.json 3
        --capabilities CAPABILITY_NAMED_IAM
   

   1 name is the name for the CloudFormation stack, such as cluster-sec. You need the name of this stack if you remove the cluster.

   2 template is the relative path to and name of the CloudFormation template YAML file that you saved.

   3 parameters is the relative path to and name of the CloudFormation parameters JSON file.

4. Confirm that the template components exist:

   $ aws cloudformation describe-stacks --stack-name name
   

   After the StackStatus displays CREATE_COMPLETE, the output displays values for the following parameters. Provide these parameter values to the other CloudFormation templates that you run to create the cluster:

   MasterSecurityGroupId	Master Security Group ID
   WorkerSecurityGroupId	Worker Security Group ID
   MasterInstanceProfile	Master IAM Instance Profile
   WorkerInstanceProfile	Worker IAM Instance Profile

CloudFormation template for security objects

We can use the following CloudFormation template to deploy the security objects required for the OpenShift cluster.

AWSTemplateFormatVersion: 2010-09-09
Description: Template for OpenShift Cluster Security Elements (Security Groups & IAM)

Parameters:
  InfrastructureName:
    AllowedPattern: ^([a-zA-Z][a-zA-Z0-9\-]{0,26})$
    MaxLength: 27
    MinLength: 1
    ConstraintDescription: Infrastructure name must be alphanumeric, start with a letter, and have a maximum of 27 characters.
    Description: A short, unique cluster ID used to tag cloud resources and identify items owned or used by the cluster.
    Type: String
  VpcCidr:
    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
    ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-24.
    Default: 10.0.0.0/16
    Description: CIDR block for VPC.
    Type: String
  VpcId:
    Description: The VPC-scoped resources will belong to this VPC.
    Type: AWS::EC2::VPC::Id
  PrivateSubnets:
    Description: The internal subnets.
    Type: ListAWS::EC2::Subnet::Id

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
    - Label:
default: "Cluster Information"
      Parameters:
      - InfrastructureName
    - Label:
default: "Network Configuration"
      Parameters:
      - VpcId
      - VpcCidr
      - PrivateSubnets
    ParameterLabels:
      InfrastructureName:
default: "Infrastructure Name"
      VpcId:
default: "VPC ID"
      VpcCidr:
default: "VPC CIDR"
      PrivateSubnets:
default: "Private Subnets"

Resources:
  MasterSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Cluster Master Security Group
      SecurityGroupIngress:
      - IpProtocol: icmp
FromPort: 0
ToPort: 0
CidrIp: !Ref VpcCidr
      - IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref VpcCidr
      - IpProtocol: tcp
ToPort: 6443
FromPort: 6443
CidrIp: !Ref VpcCidr
      - IpProtocol: tcp
FromPort: 22623
ToPort: 22623
CidrIp: !Ref VpcCidr
      VpcId: !Ref VpcId

  WorkerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Cluster Worker Security Group
      SecurityGroupIngress:
      - IpProtocol: icmp
FromPort: 0
ToPort: 0
CidrIp: !Ref VpcCidr
      - IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref VpcCidr
      VpcId: !Ref VpcId

  MasterIngressEtcd:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt MasterSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
      Description: etcd
      FromPort: 2379
      ToPort: 2380
      IpProtocol: tcp

  MasterIngressVxlan:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt MasterSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
      Description: Vxlan packets
      FromPort: 4789
      ToPort: 4789
      IpProtocol: udp

  MasterIngressWorkerVxlan:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt MasterSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
      Description: Vxlan packets
      FromPort: 4789
      ToPort: 4789
      IpProtocol: udp

  MasterIngressInternal:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt MasterSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
      Description: Internal cluster communication
      FromPort: 9000
      ToPort: 9999
      IpProtocol: tcp

  MasterIngressWorkerInternal:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt MasterSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
      Description: Internal cluster communication
      FromPort: 9000
      ToPort: 9999
      IpProtocol: tcp

  MasterIngressKube:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt MasterSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
      Description: Kubernetes kubelet, scheduler and controller manager
      FromPort: 10250
      ToPort: 10259
      IpProtocol: tcp

  MasterIngressWorkerKube:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt MasterSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
      Description: Kubernetes kubelet, scheduler and controller manager
      FromPort: 10250
      ToPort: 10259
      IpProtocol: tcp

  MasterIngressIngressServices:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt MasterSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
      Description: Kubernetes ingress services
      FromPort: 30000
      ToPort: 32767
      IpProtocol: tcp

  MasterIngressWorkerIngressServices:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt MasterSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
      Description: Kubernetes ingress services
      FromPort: 30000
      ToPort: 32767
      IpProtocol: tcp

  WorkerIngressVxlan:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt WorkerSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
      Description: Vxlan packets
      FromPort: 4789
      ToPort: 4789
      IpProtocol: udp

  WorkerIngressWorkerVxlan:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt WorkerSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
      Description: Vxlan packets
      FromPort: 4789
      ToPort: 4789
      IpProtocol: udp

  WorkerIngressInternal:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt WorkerSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
      Description: Internal cluster communication
      FromPort: 9000
      ToPort: 9999
      IpProtocol: tcp

  WorkerIngressWorkerInternal:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt WorkerSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
      Description: Internal cluster communication
      FromPort: 9000
      ToPort: 9999
      IpProtocol: tcp

  WorkerIngressKube:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt WorkerSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
      Description: Kubernetes secure kubelet port
      FromPort: 10250
      ToPort: 10250
      IpProtocol: tcp

  WorkerIngressWorkerKube:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt WorkerSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
      Description: Internal Kubernetes communication
      FromPort: 10250
      ToPort: 10250
      IpProtocol: tcp

  WorkerIngressIngressServices:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt WorkerSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt WorkerSecurityGroup.GroupId
      Description: Kubernetes ingress services
      FromPort: 30000
      ToPort: 32767
      IpProtocol: tcp

  WorkerIngressWorkerIngressServices:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !GetAtt WorkerSecurityGroup.GroupId
      SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
      Description: Kubernetes ingress services
      FromPort: 30000
      ToPort: 32767
      IpProtocol: tcp

  MasterIamRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
  Principal:
Service:
- "ec2.amazonaws.com"
  Action:
  - "sts:AssumeRole"
      Policies:
      - PolicyName: !Join ["-", [!Ref InfrastructureName, "master", "policy"]]
PolicyDocument:
  Version: "2012-10-17"
  Statement:
  - Effect: "Allow"
Action: "ec2:*"
Resource: "*"
  - Effect: "Allow"
Action: "elasticloadbalancing:*"
Resource: "*"
  - Effect: "Allow"
Action: "iam:PassRole"
Resource: "*"
  - Effect: "Allow"
Action: "s3:GetObject"
Resource: "*"

  MasterInstanceProfile:
    Type: "AWS::IAM::InstanceProfile"
    Properties:
      Roles:
      - Ref: "MasterIamRole"

  WorkerIamRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
  Principal:
Service:
- "ec2.amazonaws.com"
  Action:
  - "sts:AssumeRole"
      Policies:
      - PolicyName: !Join ["-", [!Ref InfrastructureName, "worker", "policy"]]
PolicyDocument:
  Version: "2012-10-17"
  Statement:
  - Effect: "Allow"
Action: "ec2:Describe*"
Resource: "*"

  WorkerInstanceProfile:
    Type: "AWS::IAM::InstanceProfile"
    Properties:
      Roles:
      - Ref: "WorkerIamRole"

Outputs:
  MasterSecurityGroupId:
    Description: Master Security Group ID
    Value: !GetAtt MasterSecurityGroup.GroupId

  WorkerSecurityGroupId:
    Description: Worker Security Group ID
    Value: !GetAtt WorkerSecurityGroup.GroupId

  MasterInstanceProfile:
    Description: Master IAM Instance Profile
    Value: !Ref MasterInstanceProfile

  WorkerInstanceProfile:
    Description: Worker IAM Instance Profile
    Value: !Ref WorkerInstanceProfile

RHCOS AMIs for the AWS infrastructure

Use a valid Red Hat Enterprise Linux CoreOS (RHCOS) AMI for your Amazon Web Services (AWS) zone for the OpenShift nodes.

AWS zone	AWS AMI
ap-northeast-1	ami-0c63b39219b8123e5
ap-northeast-2	ami-073cba0913d2250a4
ap-south-1	ami-0270be11430101040
ap-southeast-1	ami-06eb9d35ede4f08a3
ap-southeast-2	ami-0d980796ce258b5d5
ca-central-1	ami-0f907257d1686e3f7
eu-central-1	ami-02fdd627029c0055b
eu-west-1	ami-0d4839574724ed3fa
eu-west-2	ami-053073b95aa285347
eu-west-3	ami-09deb5deb6567bcd5
sa-east-1	ami-068a2000546e1889d
us-east-1	ami-046fe691f52a953f9
us-east-2	ami-0649fd5d42859bdfc
us-west-1	ami-0c1d2b5606111ac8c
us-west-2	ami-00745fcbb14a863ed

Create the bootstrap node in AWS

Create the bootstrap node in Amazon Web Services (AWS) to use during OpenShift cluster initialization. The easiest way to create this node is to modify the provided CloudFormation template.

If we do not use the provided CloudFormation template to create your bootstrap node, review the provided information and manually create the infrastructure. If the cluster does not initialize correctly, we might have to contact Red Hat support with your installation logs.

Prerequisites

• Configure an AWS account.
• Generate the Ignition config files for the cluster.
• Create and configure a VPC and assocated subnets in AWS.
• Create and configure DNS, load balancers, and listeners in AWS.
• Create control plane and compute roles.

Procedure

1. Provide a location to serve the bootstrap.ign Ignition config file to the cluster. This file is located in your installation directory. One way to do this is to create an S3 bucket in the cluster's region and upload the Ignition config file to it.

   The provided CloudFormation Template assumes that the Ignition config files for the cluster are served from an S3 bucket. If we choose to serve the files from another location, we must modify the templates.

   The bootstrap Ignition config file does contain secrets, like X.509 keys. The following steps provide basic security for the S3 bucket. To provide additional security, we can enable an S3 bucket policy to allow only certain users, such as the OpenShift IAM user, to access objects that the bucket contains. We can avoid S3 entirely and serve your bootstrap Ignition config file from any address that the bootstrap machine can reach.

   1. Create the bucket:

      $ aws s3 mb s3://cluster-name-infra 1
      

      1 cluster-name-infra is the bucket name.

   2. Upload the bootstrap.ign Ignition config file to the bucket:

      $ aws s3 cp bootstrap.ign s3://cluster-name-infra/bootstrap.ign
      

   3. Verify that the file uploaded:

      $ aws s3 ls s3://cluster-name-infra/
      
      2019-04-03 16:15:16     314878 bootstrap.ign
      

2. Create a JSON file containing the parameter values that the template requires:

   [
     {
       "ParameterKey": "InfrastructureName", 1
       "ParameterValue": "mycluster-random_string" 2
     },
     {
       "ParameterKey": "RhcosAmi", 3
       "ParameterValue": "ami-random_string" 4
     },
     {
       "ParameterKey": "AllowedBootstrapSshCidr", 5
       "ParameterValue": "0.0.0.0/0" 6
     },
     {
       "ParameterKey": "PublicSubnet", 7
       "ParameterValue": "subnet-random_string" 8
     },
     {
       "ParameterKey": "MasterSecurityGroupId", 9
       "ParameterValue": "sg-random_string" 10
     },
     {
       "ParameterKey": "VpcId", 11
       "ParameterValue": "vpc-random_string" 12
     },
     {
       "ParameterKey": "BootstrapIgnitionLocation", 13
       "ParameterValue": "s3://bucket_name/bootstrap.ign" 14
     },
     {
       "ParameterKey": "AutoRegisterELB", 15
       "ParameterValue": "yes" 16
     },
     {
       "ParameterKey": "RegisterNlbIpTargetsLambdaArn", 17
       "ParameterValue": "arn:aws:lambda:region:account_number:function:dns_stack_name-RegisterNlbIpTargets-random_string" 18
     },
     {
       "ParameterKey": "ExternalApiTargetGroupArn", 19
       "ParameterValue": "arn:aws:elasticloadbalancing:region:account_number:targetgroup/dns_stack_name-Exter-random_string" 20
     },
     {
       "ParameterKey": "InternalApiTargetGroupArn", 21
       "ParameterValue": "arn:aws:elasticloadbalancing:region:account_number:targetgroup/dns_stack_name-Inter-random_string" 22
     },
     {
       "ParameterKey": "InternalServiceTargetGroupArn", 23
       "ParameterValue": "arn:aws:elasticloadbalancing:region:account_number:targetgroup/dns_stack_name-Inter-random_string" 24
     }
   ]
   

   1 The name for the cluster infrastructure that is encoded in your Ignition config files for the cluster.

   2 Specify the infrastructure name that you extracted from the Ignition config file metadata, which has the format cluster-name-random-string.

   3 Current RHCOS AMI to use for the boostrap node.

   4 Specify a valid AWS::EC2::Image::Id value.

   5 CIDR block to allow SSH access to the bootstrap node.

   6 Specify a CIDR block in the format x.x.x.x/16-24.

   7 The public subnet that is associated with your VPC to launch the bootstrap node into.

   8 Specify the PublicSubnetIds value from the output of the CloudFormation template for the VPC.

   9 The master security group ID (for registering temporary rules)

   10 Specify the MasterSecurityGroupId value from the output of the CloudFormation template for the security group and roles.

   11 The VPC created resources will belong to.

   12 Specify the VpcId value from the output of the CloudFormation template for the VPC.

   13 Location to fetch bootstrap Ignition config file from.

   14 Specify the S3 bucket and file name in the form s3://bucket_name/bootstrap.ign.

   15 Whether or not to register a network load balancer (NLB).

   16 Specify yes or no. If we specify yes, provide a Lambda Amazon Resource Name (ARN) value.

   17 The ARN for NLB IP target registration lambda group.

   18 Specify the RegisterNlbIpTargetsLambda value from the output of the CloudFormation template for DNS and load balancing.

   19 The ARN for external API load balancer target group.

   20 Specify the ExternalApiTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

   21 The ARN for internal API load balancer target group.

   22 Specify the InternalApiTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

   23 The ARN for internal service load balancer target group.

   24 Specify the InternalServiceTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

3. Copy the template from the CloudFormation template for the bootstrap machine section of this topic and save it as a YAML file on our computer. This template describes the bootstrap machine that the cluster requires.

4. Launch the template:

   Enter the command on a single line.

   $ aws cloudformation create-stack --stack-name name 1
        --template-body file://template.yaml 2
        --parameters file://parameters.json 3
        --capabilities CAPABILITY_NAMED_IAM
   

   1 name is the name for the CloudFormation stack, such as cluster-bootstrap. You need the name of this stack if you remove the cluster.

   2 template is the relative path to and name of the CloudFormation template YAML file that you saved.

   3 parameters is the relative path to and name of the CloudFormation parameters JSON file.

5. Confirm that the template components exist:

   $ aws cloudformation describe-stacks --stack-name name
   

   After the StackStatus displays CREATE_COMPLETE, the output displays values for the following parameters. Provide these parameter values to the other CloudFormation templates that you run to create the cluster:

   BootstrapInstanceId	The bootstrap Instance ID.
   BootstrapPublicIp	The bootstrap node public IP address.
   BootstrapPrivateIp	The bootstrap node private IP address.

Cl. oudFormation template for the bootstrap machine

We can use the following CloudFormation template to deploy the bootstrap machine required for the OpenShift cluster.

AWSTemplateFormatVersion: 2010-09-09
Description: Template for OpenShift Cluster Bootstrap (EC2 Instance, Security Groups and IAM)

Parameters:
  InfrastructureName:
    AllowedPattern: ^([a-zA-Z][a-zA-Z0-9\-]{0,26})$
    MaxLength: 27
    MinLength: 1
    ConstraintDescription: Infrastructure name must be alphanumeric, start with a letter, and have a maximum of 27 characters.
    Description: A short, unique cluster ID used to tag cloud resources and identify items owned or used by the cluster.
    Type: String
  RhcosAmi:
    Description: Current Red Hat Enterprise Linux CoreOS AMI to use for boostrap.
    Type: AWS::EC2::Image::Id
  AllowedBootstrapSshCidr:
    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|1[0-9]|2[0-9]|3[0-2]))$
    ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/0-32.
    Default: 0.0.0.0/0
    Description: CIDR block to allow SSH access to the bootstrap node.
    Type: String
  PublicSubnet:
    Description: The public subnet to launch the bootstrap node into.
    Type: AWS::EC2::Subnet::Id
  MasterSecurityGroupId:
    Description: The master security group ID for registering temporary rules.
    Type: AWS::EC2::SecurityGroup::Id
  VpcId:
    Description: The VPC-scoped resources will belong to this VPC.
    Type: AWS::EC2::VPC::Id
  BootstrapIgnitionLocation:
    Default: s3://my-s3-bucket/bootstrap.ign
    Description: Ignition config file location.
    Type: String
  AutoRegisterELB:
    Default: "yes"
    AllowedValues:
    - "yes"
    - "no"
    Description: Do we want to invoke NLB registration, which requires a Lambda ARN parameter?
    Type: String
  RegisterNlbIpTargetsLambdaArn:
    Description: ARN for NLB IP target registration lambda.
    Type: String
  ExternalApiTargetGroupArn:
    Description: ARN for external API load balancer target group.
    Type: String
  InternalApiTargetGroupArn:
    Description: ARN for internal API load balancer target group.
    Type: String
  InternalServiceTargetGroupArn:
    Description: ARN for internal service load balancer target group.
    Type: String

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
    - Label:
default: "Cluster Information"
      Parameters:
      - InfrastructureName
    - Label:
default: "Host Information"
      Parameters:
      - RhcosAmi
      - BootstrapIgnitionLocation
      - MasterSecurityGroupId
    - Label:
default: "Network Configuration"
      Parameters:
      - VpcId
      - AllowedBootstrapSshCidr
      - PublicSubnet
    - Label:
default: "Load Balancer Automation"
      Parameters:
      - AutoRegisterELB
      - RegisterNlbIpTargetsLambdaArn
      - ExternalApiTargetGroupArn
      - InternalApiTargetGroupArn
      - InternalServiceTargetGroupArn
    ParameterLabels:
      InfrastructureName:
default: "Infrastructure Name"
      VpcId:
default: "VPC ID"
      AllowedBootstrapSshCidr:
default: "Allowed SSH Source"
      PublicSubnet:
default: "Public Subnet"
      RhcosAmi:
default: "Red Hat Enterprise Linux CoreOS AMI ID"
      BootstrapIgnitionLocation:
default: "Bootstrap Ignition Source"
      MasterSecurityGroupId:
default: "Master Security Group ID"
      AutoRegisterELB:
default: "Use Provided ELB Automation"

Conditions:
  DoRegistration: !Equals ["yes", !Ref AutoRegisterELB]

Resources:
  BootstrapIamRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
  Principal:
Service:
- "ec2.amazonaws.com"
  Action:
  - "sts:AssumeRole"
      Path: "/"
      Policies:
      - PolicyName: !Join ["-", [!Ref InfrastructureName, "bootstrap", "policy"]]
PolicyDocument:
  Version: "2012-10-17"
  Statement:
  - Effect: "Allow"
Action: "ec2:Describe*"
Resource: "*"
  - Effect: "Allow"
Action: "ec2:AttachVolume"
Resource: "*"
  - Effect: "Allow"
Action: "ec2:DetachVolume"
Resource: "*"
  - Effect: "Allow"
Action: "s3:GetObject"
Resource: "*"

  BootstrapInstanceProfile:
    Type: "AWS::IAM::InstanceProfile"
    Properties:
      Path: "/"
      Roles:
      - Ref: "BootstrapIamRole"

  BootstrapSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Cluster Bootstrap Security Group
      SecurityGroupIngress:
      - IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref AllowedBootstrapSshCidr
      - IpProtocol: tcp
ToPort: 19531
FromPort: 19531
CidrIp: 0.0.0.0/0
      VpcId: !Ref VpcId

  BootstrapInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref RhcosAmi
      IamInstanceProfile: !Ref BootstrapInstanceProfile
      InstanceType: "i3.large"
      NetworkInterfaces:
      - AssociatePublicIpAddress: "true"
DeviceIndex: "0"
GroupSet:
- !Ref "BootstrapSecurityGroup"
- !Ref "MasterSecurityGroupId"
SubnetId: !Ref "PublicSubnet"
      UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"replace":{"source":"${S3Loc}","verification":{}}},"timeouts":{},"version":"2.1.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}'
- {
  S3Loc: !Ref BootstrapIgnitionLocation
}

  RegisterBootstrapApiTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref ExternalApiTargetGroupArn
      TargetIp: !GetAtt BootstrapInstance.PrivateIp

  RegisterBootstrapInternalApiTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalApiTargetGroupArn
      TargetIp: !GetAtt BootstrapInstance.PrivateIp

  RegisterBootstrapInternalServiceTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalServiceTargetGroupArn
      TargetIp: !GetAtt BootstrapInstance.PrivateIp

Outputs:
  BootstrapInstanceId:
    Description: Bootstrap Instance ID.
    Value: !Ref BootstrapInstance

  BootstrapPublicIp:
    Description: The bootstrap node public IP address.
    Value: !GetAtt BootstrapInstance.PublicIp

  BootstrapPrivateIp:
    Description: The bootstrap node private IP address.
    Value: !GetAtt BootstrapInstance.PrivateIp

Create the control plane machines in AWS

Create the control plane machines in Amazon Web Services (AWS) for the cluster to use. The easiest way to create these nodes is to modify the provided CloudFormation template.

If we do not use the provided CloudFormation template to create your control plane nodes, review the provided information and manually create the infrastructure. If the cluster does not initialize correctly, we might have to contact Red Hat support with your installation logs.

Prerequisites

• Configure an AWS account.
• Generate the Ignition config files for the cluster.
• Create and configure a VPC and assocated subnets in AWS.
• Create and configure DNS, load balancers, and listeners in AWS.
• Create control plane and compute roles.
• Create the bootstrap machine.

Procedure

1. Create a JSON file containing the parameter values that the template requires:

   [
     {
       "ParameterKey": "InfrastructureName", 1
       "ParameterValue": "mycluster-random_string" 2
     },
     {
       "ParameterKey": "RhcosAmi", 3
       "ParameterValue": "ami-random_string" 4
     },
     {
       "ParameterKey": "AutoRegisterDNS", 5
       "ParameterValue": "yes" 6
     },
     {
       "ParameterKey": "PrivateHostedZoneId", 7
       "ParameterValue": "random_string" 8
     },
     {
       "ParameterKey": "PrivateHostedZoneName", 9
       "ParameterValue": "mycluster.example.com" 10
     },
     {
       "ParameterKey": "Master0Subnet", 11
       "ParameterValue": "subnet-random_string" 12
     },
     {
       "ParameterKey": "Master1Subnet", 13
       "ParameterValue": "subnet-random_string" 14
     },
     {
       "ParameterKey": "Master2Subnet", 15
       "ParameterValue": "subnet-random_string" 16
     },
     {
       "ParameterKey": "MasterSecurityGroupId", 17
       "ParameterValue": "sg-random_string" 18
     },
     {
       "ParameterKey": "IgnitionLocation", 19
       "ParameterValue": "https://api-int.cluster_name.domain_name:22623/config/master" 20
     },
     {
       "ParameterKey": "CertificateAuthorities", 21
       "ParameterValue": "data:text/plain;charset=utf-8;base64,ABC...xYz==" 22
     },
     {
       "ParameterKey": "MasterInstanceProfileName", 23
       "ParameterValue": "roles_stack-MasterInstanceProfile-random_string" 24
     },
     {
       "ParameterKey": "MasterInstanceType", 25
       "ParameterValue": "m4.xlarge" 26
     },
     {
       "ParameterKey": "AutoRegisterELB", 27
       "ParameterValue": "yes" 28
     },
     {
       "ParameterKey": "RegisterNlbIpTargetsLambdaArn", 29
       "ParameterValue": "arn:aws:lambda:region:account_number:function:dns_stack_name-RegisterNlbIpTargets-random_string" 30
     },
     {
       "ParameterKey": "ExternalApiTargetGroupArn", 31
       "ParameterValue": "arn:aws:elasticloadbalancing:region:account_number:targetgroup/dns_stack_name-Exter-random_string" 32
     },
     {
       "ParameterKey": "InternalApiTargetGroupArn", 33
       "ParameterValue": "arn:aws:elasticloadbalancing:region:account_number:targetgroup/dns_stack_name-Inter-random_string" 34
     },
     {
       "ParameterKey": "InternalServiceTargetGroupArn", 35
       "ParameterValue": "arn:aws:elasticloadbalancing:region:account_number:targetgroup/dns_stack_name-Inter-random_string" 36
     }
   ]
   

   1 The name for the cluster infrastructure that is encoded in your Ignition config files for the cluster.

   2 Specify the infrastructure name that you extracted from the Ignition config file metadata, which has the format cluster-name-random-string.

   3 Current RHCOS AMI to use for the control plane machines.

   4 Specify an AWS::EC2::Image::Id value.

   5 Whether or not to perform DNS etcd registration.

   6 Specify yes or no. If we specify yes, provide Hosted Zone information.

   7 The Route53 private zone ID to register the etcd targets with.

   8 Specify the PrivateHostedZoneId value from the output of the CloudFormation template for DNS and load balancing.

   9 The Route53 zone to register the targets with.

   10 Specify cluster_name.domain_name where domain_name is the Route53 base domain that you used when you generated install-config.yaml file for the cluster. Do not include the trailing period (.) that is displayed in the AWS console.

   11 13 15 A subnet, preferably private, to launch the control plane machines on.

   12 14 16 Specify a subnet from the PrivateSubnets value from the output of the CloudFormation template for DNS and load balancing.

   17 The master security group ID to associate with master nodes.

   18 Specify the MasterSecurityGroupId value from the output of the CloudFormation template for the security group and roles.

   19 The location to fetch control plane Ignition config file from.

   20 Generated Ignition config file location, https://api-int.cluster_name.domain_name:22623/config/master.

   21 The base64 encoded certificate authority string to use.

   22 Specify the value from the master.ign file in the installation directory. This value is the long string with the format data:text/plain;charset=utf-8;base64,ABC…​xYz==.

   23 The IAM profile to associate with master nodes.

   24 Specify the MasterInstanceProfile parameter value from the output of the CloudFormation template for the security group and roles.

   25 The type of AWS instance to use for the control plane machines.

   26 Allowed values:

   • m4.xlarge
   • m4.2xlarge
   • m4.4xlarge
   • m4.8xlarge
   • m4.10xlarge
   • m4.16xlarge
   • c4.2xlarge
   • c4.4xlarge
   • c4.8xlarge
   • r4.xlarge
   • r4.2xlarge
   • r4.4xlarge
   • r4.8xlarge

   • r4.16xlarge

     If m4 instance types are not available in your region, such as with eu-west-3, specify an m5 type, such as m5.xlarge, instead.

   27 Whether or not to register a network load balancer (NLB).

   28 Specify yes or no. If we specify yes, provide a Lambda Amazon Resource Name (ARN) value.

   29 The ARN for NLB IP target registration lambda group.

   30 Specify the RegisterNlbIpTargetsLambda value from the output of the CloudFormation template for DNS and load balancing.

   31 The ARN for external API load balancer target group.

   32 Specify the ExternalApiTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

   33 The ARN for internal API load balancer target group.

   34 Specify the InternalApiTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

   35 The ARN for internal service load balancer target group.

   36 Specify the InternalServiceTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

2. Copy the template from the CloudFormation template for control plane machines section of this topic and save it as a YAML file on our computer. This template describes the control plane machines that the cluster requires.
3. If you specified an m5 instance type as the value for MasterInstanceType, add that instance type to the MasterInstanceType.AllowedValues parameter in the CloudFormation template.

4. Launch the template:

   Enter the command on a single line.

   $ aws cloudformation create-stack --stack-name name 1
        --template-body file://template.yaml 2
        --parameters file://parameters.json 3
   

   1 name is the name for the CloudFormation stack, such as cluster-control-plane. You need the name of this stack if you remove the cluster.

   2 template is the relative path to and name of the CloudFormation template YAML file that you saved.

   3 parameters is the relative path to and name of the CloudFormation parameters JSON file.

5. Confirm that the template components exist:

   $ aws cloudformation describe-stacks --stack-name name
   

CloudFormation template for control plane machines

We can use the following CloudFormation template to deploy the control plane machines required for the OpenShift cluster.

AWSTemplateFormatVersion: 2010-09-09
Description: Template for OpenShift Cluster Node Launch (EC2 master instances)

Parameters:
  InfrastructureName:
    AllowedPattern: ^([a-zA-Z][a-zA-Z0-9\-]{0,26})$
    MaxLength: 27
    MinLength: 1
    ConstraintDescription: Infrastructure name must be alphanumeric, start with a letter, and have a maximum of 27 characters.
    Description: A short, unique cluster ID used to tag nodes for the kubelet cloud provider.
    Type: String
  RhcosAmi:
    Description: Current Red Hat Enterprise Linux CoreOS AMI to use for boostrap.
    Type: AWS::EC2::Image::Id
  AutoRegisterDNS:
    Default: "yes"
    AllowedValues:
    - "yes"
    - "no"
    Description: Do we want to invoke DNS etcd registration, which requires Hosted Zone information?
    Type: String
  PrivateHostedZoneId:
    Description: The Route53 private zone ID to register the etcd targets with, such as Z21IXYZABCZ2A4.
    Type: String
  PrivateHostedZoneName:
    Description: The Route53 zone to register the targets with, such as cluster.example.com. Omit the trailing period.
    Type: String
  Master0Subnet:
    Description: The subnets, recommend private, to launch the master nodes into.
    Type: AWS::EC2::Subnet::Id
  Master1Subnet:
    Description: The subnets, recommend private, to launch the master nodes into.
    Type: AWS::EC2::Subnet::Id
  Master2Subnet:
    Description: The subnets, recommend private, to launch the master nodes into.
    Type: AWS::EC2::Subnet::Id
  MasterSecurityGroupId:
    Description: The master security group ID to associate with master nodes.
    Type: AWS::EC2::SecurityGroup::Id
  IgnitionLocation:
    Default: https://api-int.$CLUSTER_NAME.$DOMAIN:22623/config/master
    Description: Ignition config file location.
    Type: String
  CertificateAuthorities:
    Default: data:text/plain;charset=utf-8;base64,ABC...xYz==
    Description: Base64 encoded certificate authority string to use.
    Type: String
  MasterInstanceProfileName:
    Description: IAM profile to associate with master nodes.
    Type: String
  MasterInstanceType:
    Default: m4.xlarge
    Type: String
    AllowedValues:
    - "m4.xlarge"
    - "m4.2xlarge"
    - "m4.4xlarge"
    - "m4.8xlarge"
    - "m4.10xlarge"
    - "m4.16xlarge"
    - "c4.2xlarge"
    - "c4.4xlarge"
    - "c4.8xlarge"
    - "r4.xlarge"
    - "r4.2xlarge"
    - "r4.4xlarge"
    - "r4.8xlarge"
    - "r4.16xlarge"
  AutoRegisterELB:
    Default: "yes"
    AllowedValues:
    - "yes"
    - "no"
    Description: Do we want to invoke NLB registration, which requires a Lambda ARN parameter?
    Type: String
  RegisterNlbIpTargetsLambdaArn:
    Description: ARN for NLB IP target registration lambda. Supply the value from the cluster infrastructure or select "no" for AutoRegisterELB.
    Type: String
  ExternalApiTargetGroupArn:
    Description: ARN for external API load balancer target group. Supply the value from the cluster infrastructure or select "no" for AutoRegisterELB.
    Type: String
  InternalApiTargetGroupArn:
    Description: ARN for internal API load balancer target group. Supply the value from the cluster infrastructure or select "no" for AutoRegisterELB.
    Type: String
  InternalServiceTargetGroupArn:
    Description: ARN for internal service load balancer target group. Supply the value from the cluster infrastructure or select "no" for AutoRegisterELB.
    Type: String

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
    - Label:
default: "Cluster Information"
      Parameters:
      - InfrastructureName
    - Label:
default: "Host Information"
      Parameters:
      - MasterInstanceType
      - RhcosAmi
      - IgnitionLocation
      - CertificateAuthorities
      - MasterSecurityGroupId
      - MasterInstanceProfileName
    - Label:
default: "Network Configuration"
      Parameters:
      - VpcId
      - AllowedBootstrapSshCidr
      - Master0Subnet
      - Master1Subnet
      - Master2Subnet
    - Label:
default: "DNS"
      Parameters:
      - AutoRegisterDNS
      - PrivateHostedZoneName
      - PrivateHostedZoneId
    - Label:
default: "Load Balancer Automation"
      Parameters:
      - AutoRegisterELB
      - RegisterNlbIpTargetsLambdaArn
      - ExternalApiTargetGroupArn
      - InternalApiTargetGroupArn
      - InternalServiceTargetGroupArn
    ParameterLabels:
      InfrastructureName:
default: "Infrastructure Name"
      VpcId:
default: "VPC ID"
      Master0Subnet:
default: "Master-0 Subnet"
      Master1Subnet:
default: "Master-1 Subnet"
      Master2Subnet:
default: "Master-2 Subnet"
      MasterInstanceType:
default: "Master Instance Type"
      MasterInstanceProfileName:
default: "Master Instance Profile Name"
      RhcosAmi:
default: "Red Hat Enterprise Linux CoreOS AMI ID"
      BootstrapIgnitionLocation:
default: "Master Ignition Source"
      CertificateAuthorities:
default: "Ignition CA String"
      MasterSecurityGroupId:
default: "Master Security Group ID"
      AutoRegisterDNS:
default: "Use Provided DNS Automation"
      AutoRegisterELB:
default: "Use Provided ELB Automation"
      PrivateHostedZoneName:
default: "Private Hosted Zone Name"
      PrivateHostedZoneId:
default: "Private Hosted Zone ID"

Conditions:
  DoRegistration: !Equals ["yes", !Ref AutoRegisterELB]
  DoDns: !Equals ["yes", !Ref AutoRegisterDNS]

Resources:
  Master0:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref RhcosAmi
      BlockDeviceMappings:
      - DeviceName: /dev/xvda
Ebs:
  VolumeSize: "120"
  VolumeType: "gp2"
      IamInstanceProfile: !Ref MasterInstanceProfileName
      InstanceType: !Ref MasterInstanceType
      NetworkInterfaces:
      - AssociatePublicIpAddress: "false"
DeviceIndex: "0"
GroupSet:
- !Ref "MasterSecurityGroupId"
SubnetId: !Ref "Master0Subnet"
      UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"append":[{"source":"${SOURCE}","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}'
- {
  SOURCE: !Ref IgnitionLocation,
  CA_BUNDLE: !Ref CertificateAuthorities,
}
      Tags:
      - Key: !Join ["", ["kubernetes.io/cluster/", !Ref InfrastructureName]]
Value: "shared"

  RegisterMaster0:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref ExternalApiTargetGroupArn
      TargetIp: !GetAtt Master0.PrivateIp

  RegisterMaster0InternalApiTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalApiTargetGroupArn
      TargetIp: !GetAtt Master0.PrivateIp

  RegisterMaster0InternalServiceTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalServiceTargetGroupArn
      TargetIp: !GetAtt Master0.PrivateIp

  Master1:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref RhcosAmi
      BlockDeviceMappings:
      - DeviceName: /dev/xvda
Ebs:
  VolumeSize: "120"
  VolumeType: "gp2"
      IamInstanceProfile: !Ref MasterInstanceProfileName
      InstanceType: !Ref MasterInstanceType
      NetworkInterfaces:
      - AssociatePublicIpAddress: "false"
DeviceIndex: "0"
GroupSet:
- !Ref "MasterSecurityGroupId"
SubnetId: !Ref "Master1Subnet"
      UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"append":[{"source":"${SOURCE}","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}'
- {
  SOURCE: !Ref IgnitionLocation,
  CA_BUNDLE: !Ref CertificateAuthorities,
}
      Tags:
      - Key: !Join ["", ["kubernetes.io/cluster/", !Ref InfrastructureName]]
Value: "shared"

  RegisterMaster1:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref ExternalApiTargetGroupArn
      TargetIp: !GetAtt Master1.PrivateIp

  RegisterMaster1InternalApiTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalApiTargetGroupArn
      TargetIp: !GetAtt Master1.PrivateIp

  RegisterMaster1InternalServiceTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalServiceTargetGroupArn
      TargetIp: !GetAtt Master1.PrivateIp

  Master2:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref RhcosAmi
      BlockDeviceMappings:
      - DeviceName: /dev/xvda
Ebs:
  VolumeSize: "120"
  VolumeType: "gp2"
      IamInstanceProfile: !Ref MasterInstanceProfileName
      InstanceType: !Ref MasterInstanceType
      NetworkInterfaces:
      - AssociatePublicIpAddress: "false"
DeviceIndex: "0"
GroupSet:
- !Ref "MasterSecurityGroupId"
SubnetId: !Ref "Master2Subnet"
      UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"append":[{"source":"${SOURCE}","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}'
- {
  SOURCE: !Ref IgnitionLocation,
  CA_BUNDLE: !Ref CertificateAuthorities,
}
      Tags:
      - Key: !Join ["", ["kubernetes.io/cluster/", !Ref InfrastructureName]]
Value: "shared"

  RegisterMaster2:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref ExternalApiTargetGroupArn
      TargetIp: !GetAtt Master2.PrivateIp

  RegisterMaster2InternalApiTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalApiTargetGroupArn
      TargetIp: !GetAtt Master2.PrivateIp

  RegisterMaster2InternalServiceTarget:
    Condition: DoRegistration
    Type: Custom::NLBRegister
    Properties:
      ServiceToken: !Ref RegisterNlbIpTargetsLambdaArn
      TargetArn: !Ref InternalServiceTargetGroupArn
      TargetIp: !GetAtt Master2.PrivateIp

  EtcdSrvRecords:
    Condition: DoDns
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZoneId
      Name: !Join [".", ["_etcd-server-ssl._tcp", !Ref PrivateHostedZoneName]]
      ResourceRecords:
      - !Join [
" ",
["0 10 2380", !Join [".", ["etcd-0", !Ref PrivateHostedZoneName]]],
      ]
      - !Join [
" ",
["0 10 2380", !Join [".", ["etcd-1", !Ref PrivateHostedZoneName]]],
      ]
      - !Join [
" ",
["0 10 2380", !Join [".", ["etcd-2", !Ref PrivateHostedZoneName]]],
      ]
      TTL: 60
      Type: SRV

  Etcd0Record:
    Condition: DoDns
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZoneId
      Name: !Join [".", ["etcd-0", !Ref PrivateHostedZoneName]]
      ResourceRecords:
      - !GetAtt Master0.PrivateIp
      TTL: 60
      Type: A

  Etcd1Record:
    Condition: DoDns
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZoneId
      Name: !Join [".", ["etcd-1", !Ref PrivateHostedZoneName]]
      ResourceRecords:
      - !GetAtt Master1.PrivateIp
      TTL: 60
      Type: A

  Etcd2Record:
    Condition: DoDns
    Type: AWS::Route53::RecordSet
    Properties:
      HostedZoneId: !Ref PrivateHostedZoneId
      Name: !Join [".", ["etcd-2", !Ref PrivateHostedZoneName]]
      ResourceRecords:
      - !GetAtt Master2.PrivateIp
      TTL: 60
      Type: A

Outputs:
  PrivateIPs:
    Description: The control-plane node private IP addresses.
    Value:
      !Join [
",",
[!GetAtt Master0.PrivateIp, !GetAtt Master1.PrivateIp, !GetAtt Master2.PrivateIp]
      ]

Initializing the bootstrap node on AWS with user-provisioned infrastructure

After creating all of the required infrastructure in Amazon Web Services (AWS), we can install the cluster.

Prerequisites

• Configure an AWS account.
• Generate the Ignition config files for the cluster.
• Create and configure a VPC and assocated subnets in AWS.
• Create and configure DNS, load balancers, and listeners in AWS.
• Create control plane and compute roles.
• Create the bootstrap machine.
• Create the control plane machines.
• If you plan to manually manage the worker machines, create the worker machines.

Procedure

1. Change to the directory containing the installation program and run the following command:

   cd /path/to/install/files
   $ ./openshift-install wait-for bootstrap-complete --dir=/path/to/install/files --log-level debug 
   

   If the command exits without a FATAL warning, your production control plane has initialized.

Create the worker nodes in AWS

We can create worker nodes in Amazon Web Services (AWS) for the cluster to use. The easiest way to manually create these nodes is to modify the provided CloudFormation template.

The CloudFormation template creates a stack that represents one worker machine. Create a stack for each worker machine.

If we do not use the provided CloudFormation template to create your worker nodes, review the provided information and manually create the infrastructure. If the cluster does not initialize correctly, we might have to contact Red Hat support with your installation logs.

Prerequisites

• Configure an AWS account.
• Generate the Ignition config files for the cluster.
• Create and configure a VPC and assocated subnets in AWS.
• Create and configure DNS, load balancers, and listeners in AWS.
• Create control plane and compute roles.
• Create the bootstrap machine.
• Create the control plane machines.

Procedure

1. Create a JSON file containing the parameter values that the CloudFormation template requires:

   [
     {
       "ParameterKey": "InfrastructureName", 1
       "ParameterValue": "mycluster-random_string" 2
     },
     {
       "ParameterKey": "RhcosAmi", 3
       "ParameterValue": "ami-random_string" 4
     },
     {
       "ParameterKey": "Subnet", 5
       "ParameterValue": "subnet-random_string" 6
     },
     {
       "ParameterKey": "WorkerSecurityGroupId", 7
       "ParameterValue": "sg-random_string" 8
     },
     {
       "ParameterKey": "IgnitionLocation", 9
       "ParameterValue": "https://api-int.cluster_name.domain_name:22623/config/worker" 10
     },
     {
       "ParameterKey": "CertificateAuthorities", 11
       "ParameterValue": "" 12
     },
     {
       "ParameterKey": "WorkerInstanceProfileName", 13
       "ParameterValue": "" 14
     },
     {
       "ParameterKey": "WorkerInstanceType", 15
       "ParameterValue": "m4.large" 16
     }
   ]
   

   1 The name for the cluster infrastructure that is encoded in your Ignition config files for the cluster.

   2 Specify the infrastructure name that you extracted from the Ignition config file metadata, which has the format cluster-name-random-string.

   3 Current RHCOS AMI to use for the worker nodes.

   4 Specify an AWS::EC2::Image::Id value.

   5 A subnet, preferably private, to launch the worker nodes on.

   6 Specify a subnet from the PrivateSubnets value from the output of the CloudFormation template for DNS and load balancing.

   7 The worker security group ID to associate with worker nodes.

   8 Specify the WorkerSecurityGroupId value from the output of the CloudFormation template for the security group and roles.

   9 The location to fetch bootstrap Ignition config file from.

   10 Generated Ignition config location, https://api-int.cluster_name.domain_name:22623/config/worker.

   11 Base64 encoded certificate authority string to use.

   12 Specify the value from the worker.ign file in the installation directory. This value is the long string with the format data:text/plain;charset=utf-8;base64,ABC…​xYz==.

   13 The IAM profile to associate with worker nodes.

   14 Specify the WorkerInstanceProfile parameter value from the output of the CloudFormation template for the security group and roles.

   15 The type of AWS instance to use for the control plane machines.

   16 Allowed values:

   • m4.large
   • m4.xlarge
   • m4.2xlarge
   • m4.4xlarge
   • m4.8xlarge
   • m4.10xlarge
   • m4.16xlarge
   • c4.large
   • c4.xlarge
   • c4.2xlarge
   • c4.4xlarge
   • c4.8xlarge
   • r4.large
   • r4.xlarge
   • r4.2xlarge
   • r4.4xlarge
   • r4.8xlarge

   • r4.16xlarge

     If m4 instance types are not available in your region, such as with eu-west-3, use m5 types instead.

2. Copy the template from the CloudFormation template for worker machines section of this topic and save it as a YAML file on our computer. This template describes the networking objects and load balancers that the cluster requires.
3. If you specified an m5 instance type as the value for WorkerInstanceType, add that instance type to the WorkerInstanceType.AllowedValues parameter in the CloudFormation template.

4. Create a worker stack.

   1. Launch the template:

      Enter the command on a single line.

      $ aws cloudformation create-stack --stack-name name 1
           --template-body file://template.yaml \ 2
           --parameters file://parameters.json 3
      

      1 name is the name for the CloudFormation stack, such as cluster-workers. You need the name of this stack if you remove the cluster.

      2 template is the relative path to and name of the CloudFormation template YAML file that you saved.

      3 parameters is the relative path to and name of the CloudFormation parameters JSON file.

   2. Confirm that the template components exist:

      $ aws cloudformation describe-stacks --stack-name name
      

5. Continue to create worker stacks until we have created enough worker Machines for the cluster.

   Create at least two worker machines, so create at least two stacks that use this CloudFormation template.

CloudFormation template for worker machines

We can use the following CloudFormation template to deploy the worker machines required for the OpenShift cluster.

AWSTemplateFormatVersion: 2010-09-09
Description: Template for OpenShift Cluster Node Launch (EC2 worker instance)

Parameters:
  InfrastructureName:
    AllowedPattern: ^([a-zA-Z][a-zA-Z0-9\-]{0,26})$
    MaxLength: 27
    MinLength: 1
    ConstraintDescription: Infrastructure name must be alphanumeric, start with a letter, and have a maximum of 27 characters.
    Description: A short, unique cluster ID used to tag nodes for the kubelet cloud provider.
    Type: String
  RhcosAmi:
    Description: Current Red Hat Enterprise Linux CoreOS AMI to use for boostrap.
    Type: AWS::EC2::Image::Id
  Subnet:
    Description: The subnets, recommend private, to launch the master nodes into.
    Type: AWS::EC2::Subnet::Id
  WorkerSecurityGroupId:
    Description: The master security group ID to associate with master nodes.
    Type: AWS::EC2::SecurityGroup::Id
  IgnitionLocation:
    Default: https://api-int.$CLUSTER_NAME.$DOMAIN:22623/config/worker
    Description: Ignition config file location.
    Type: String
  CertificateAuthorities:
    Default: data:text/plain;charset=utf-8;base64,ABC...xYz==
    Description: Base64 encoded certificate authority string to use.
    Type: String
  WorkerInstanceProfileName:
    Description: IAM profile to associate with master nodes.
    Type: String
  WorkerInstanceType:
    Default: m4.large
    Type: String
    AllowedValues:
    - "m4.large"
    - "m4.xlarge"
    - "m4.2xlarge"
    - "m4.4xlarge"
    - "m4.8xlarge"
    - "m4.10xlarge"
    - "m4.16xlarge"
    - "c4.large"
    - "c4.xlarge"
    - "c4.2xlarge"
    - "c4.4xlarge"
    - "c4.8xlarge"
    - "r4.large"
    - "r4.xlarge"
    - "r4.2xlarge"
    - "r4.4xlarge"
    - "r4.8xlarge"
    - "r4.16xlarge"

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
    - Label:
default: "Cluster Information"
      Parameters:
      - InfrastructureName
    - Label:
default: "Host Information"
      Parameters:
      - WorkerInstanceType
      - RhcosAmi
      - IgnitionLocation
      - CertificateAuthorities
      - WorkerSecurityGroupId
      - WorkerInstanceProfileName
    - Label:
default: "Network Configuration"
      Parameters:
      - Subnet
    ParameterLabels:
      Subnet:
default: "Subnet"
      InfrastructureName:
default: "Infrastructure Name"
      WorkerInstanceType:
default: "Worker Instance Type"
      WorkerInstanceProfileName:
default: "Worker Instance Profile Name"
      RhcosAmi:
default: "Red Hat Enterprise Linux CoreOS AMI ID"
      IgnitionLocation:
default: "Worker Ignition Source"
      CertificateAuthorities:
default: "Ignition CA String"
      WorkerSecurityGroupId:
default: "Worker Security Group ID"

Resources:
  Worker0:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref RhcosAmi
      BlockDeviceMappings:
      - DeviceName: /dev/xvda
Ebs:
  VolumeSize: "120"
  VolumeType: "gp2"
      IamInstanceProfile: !Ref WorkerInstanceProfileName
      InstanceType: !Ref WorkerInstanceType
      NetworkInterfaces:
      - AssociatePublicIpAddress: "false"
DeviceIndex: "0"
GroupSet:
- !Ref "WorkerSecurityGroupId"
SubnetId: !Ref "Subnet"
      UserData:
Fn::Base64: !Sub
- '{"ignition":{"config":{"append":[{"source":"${SOURCE}","verification":{}}]},"security":{"tls":{"certificateAuthorities":[{"source":"${CA_BUNDLE}","verification":{}}]}},"timeouts":{},"version":"2.2.0"},"networkd":{},"passwd":{},"storage":{},"systemd":{}}'
- {
  SOURCE: !Ref IgnitionLocation,
  CA_BUNDLE: !Ref CertificateAuthorities,
}
      Tags:
      - Key: !Join ["", ["kubernetes.io/cluster/", !Ref InfrastructureName]]
Value: "shared"

Outputs:
  PrivateIP:
    Description: The compute node private IP address.
    Value: !GetAtt Worker0.PrivateIp

Install the OpenShift Command-line Interface (oc)

Earlier versions of oc cannot be used to complete all of the commands in OpenShift 4.1. Download and install the new version of oc.

1. From the OpenShift Infrastructure Providers page, click...
   Download Command-line Tools

2. Download the compressed file for your operating system.

   We can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Log on to the cluster

1. Deploy an OpenShift cluster.

2. Install the oc CLI.

3. Export the kubeadmin credentials:
   $ export KUBECONFIG=/path/to/install/files/auth/kubeconfig
   $ oc whoami
   system:admin

   The kubeconfig file contains information about the cluster used by the CLI to connect a client to the correct cluster and API server. The file is specific to a cluster and is created during OpenShift installation.

Next steps

• Customize the cluster.
• If necessary, we can opt out of telemetry.

Approve the CSRs for the machines

When you add machines to a cluster, two pending certificates signing request (CSRs) are generated for each machine that you added. Cofirm that these CSRs are approved or, if necessary, approve them ourselves.

Prerequisites

• You added machines to the cluster.
• Install the jq package.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes
   
   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.13.4+b626c2fe1
   master-1  Ready     master  63m  v1.13.4+b626c2fe1
   master-2  Ready     master  64m  v1.13.4+b626c2fe1
   worker-0  NotReady  worker  76s  v1.13.4+b626c2fe1
   worker-1  NotReady  worker  70s  v1.13.4+b626c2fe1
   

   The output lists all of the machines that createdd.

2. Review the pending certificate signing requests (CSRs) and ensure that you see a client and server request with Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr
   
   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 1
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending 2
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...
   

   • 1 A client request CSR.
   • 2 A server request CSR.

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for the cluster machines:

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If we do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. Approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. Implement a method of automatically approving the kubelet serving certificate requests.

   • To approve them individually, run for each valid CSR:

     $ oc adm certificate approve csr_name 1
     

     1 csr_name is the name of a CSR from the list of current CSRs.

   • If all the CSRs are valid, approve them all:
     $ oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

Initial Operator configuration

After the control plane initializes, immediately configure some Operators so that they all become available.

Prerequisites

• Your control plane has initialized.

Procedure

1. Watch the cluster components come online:

   $ watch -n5 oc get clusteroperators
   
   NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                       4.1.0     True        False         False      69s
   cloud-credential                     4.1.0     True        False         False      12m
   cluster-autoscaler                   4.1.0     True        False         False      11m
   console                              4.1.0     True        False         False      46s
   dns                                  4.1.0     True        False         False      11m
   image-registry                       4.1.0     False       True          False      5m26s
   ingress                              4.1.0     True        False         False      5m36s
   kube-apiserver                       4.1.0     True        False         False      8m53s
   kube-controller-manager              4.1.0     True        False         False      7m24s
   kube-scheduler                       4.1.0     True        False         False      12m
   machine-api                          4.1.0     True        False         False      12m
   machine-config                       4.1.0     True        False         False      7m36s
   marketplace                          4.1.0     True        False         False      7m54m
   monitoring                           4.1.0     True        False         False      7h44s
   network                              4.1.0     True        False         False      5m9s
   node-tuning                          4.1.0     True        False         False      11m
   openshift-apiserver                  4.1.0     True        False         False      11m
   openshift-controller-manager         4.1.0     True        False         False      5m943s
   openshift-samples                    4.1.0     True        False         False      3m55s
   operator-lifecycle-manager           4.1.0     True        False         False      11m
   operator-lifecycle-manager-catalog   4.1.0     True        False         False      11m
   service-ca                           4.1.0     True        False         False      11m
   service-catalog-apiserver            4.1.0     True        False         False      5m26s
   service-catalog-controller-manager   4.1.0     True        False         False      5m25s
   storage                              4.1.0     True        False         False      5m30s
   

2. Configure the Operators that are not available.

Image registry storage configuration

If the image-registry Operator is not available, configure storage for it. Instructions for both configuring a PersistentVolume, which is required for production clusters, and for configuring an empty directory as the storage location, which is available for only non-production clusters, are shown.

Configure registry storage for AWS with user-provisioned infrastructure

During installation, your cloud credentials are sufficient to create an S3 bucket and the Registry Operator will automatically configure storage.

If the Registry Operator cannot create an S3 bucket, and automatically configure storage, we can create a S3 bucket and configure storage with the following procedure.

Prerequisites

• A cluster on AWS with user-provisioned infrastructure.

Procedure

Use the following procedure if the Registry Operator cannot create an S3 bucket and automatically configure storage.

1. Set up a Bucket Lifecycle Policy to abort incomplete multipart uploads that are one day old.

2. Fill in the storage configuration in configs.imageregistry.operator.openshift.io/cluster:

   $ oc edit configs.imageregistry.operator.openshift.io/cluster
   
   storage:
     s3:
       bucket: bucket-name
       region: region-name
   

To secure the registry images in AWS, block public access to the S3 bucket.

Configure storage for the image registry in non-production clusters

Configure storage for the image registry Operator. For non-production clusters, we can set the image registry to an empty directory. If we do so, all images are lost if you restart the registry.

Procedure

• To set the image registry storage to an empty directory:

  $ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
  

  Warning

  Configure this option for only non-production clusters.

  If you run this command before the Image Registry Operator initializes its components, the oc patch command fails with the following error:

  Error from server (NotFound): configs.imageregistry.operator.openshift.io "cluster" not found
  

  Wait a few minutes and run the command again.

Complete an AWS installation on user-provisioned infrastructure

After you start the OpenShift installation on Amazon Web Service (AWS) user-provisioned infrastructure, remove the bootstrap node, reconcile the default Machine and MachineSet definitions, and delete unused nodes

Prerequisites

• Deploy the bootstrap node for an OpenShift cluster on user-provisioned AWS infrastructure.
• Install the oc CLI and log in.

Procedure

1. Delete the bootstrap resources. If you used the CloudFormation template, delete its stack:

   $ aws cloudformation delete-stack --stack-name name 1
   

   1 name is the name of the bootstrap stack.

2. Complete the cluster installation:

   $ ./openshift-install --dir=/path/to/install/files wait-for install-complete 
   
   INFO Waiting up to 30m0s for the cluster to initialize...
   

Next steps

Customize the cluster.

If necessary, we can opt out of telemetry.

Install on bare metal

We can install a cluster on bare metal infrastructure that we provision.

While we might be able to follow this procedure to deploy a cluster on virtualized or cloud environments, be aware of additional considerations for non-bare metal platforms. Review the information in the guidelines for deploying OpenShift on non-tested platforms before attempting to install an OpenShift cluster in such an environment.

Prerequisites

Provision persistent storage for the cluster. To deploy a private image registry, storage must provide ReadWriteMany access modes.

Review details about the installation and update processes.

If a firewall is used, configure it to access Red Hat Insights.

Internet and Telemetry access

Telemetry provides metrics about cluster health and the success of updates. To perform subscription management, including legally entitling a purchase Red Hat, use the Telemetry service and access the OpenShift Infrastructure Providers page.

There is no disconnected subscription management. We cannot both opt out of sending data back to Red Hat and entitle a purchase.

Machines have direct internet access to install the cluster.

Internet access is required to:

• Access the OpenShift Infrastructure Providers page to download the installation program
• Access quay.io to obtain the packages required to install the cluster
• Obtain the packages required to perform cluster updates
• Access Red Hat's software as a service page to perform subscription management

Machine requirements for a cluster with user-provisioned infrastructure

For a cluster that contains user-provisioned infrastructure, deploy all of the required machines.

Required machines

The smallest OpenShift clusters require the following hosts:

• One bootstrap machine
• Three control plane, or master, machines
• At least two compute, or worker, machines

The bootstrap machine deploys OpenShift cluster on the three control plane machines. We can remove the bootstrap machine after installing the cluster.

To maintain high availability of the cluster, use separate physical hosts for these cluster machines.

The bootstrap and control plane machines must use Red Hat Enterprise Linux CoreOS (RHCOS) as the operating system.

Network connectivity requirements

All the RHCOS machines require initramfs during boot to fetch Ignition config files from the Machine Config Server. During the initial boot, the machines require a DHCP server to establish a network connection to download Ignition config files. After the initial boot, the machines can be configured to use static IP addresses.

Minimum resource requirements

Each cluster machine must meet the following minimum requirements:

Machine	Operating System	vCPU	RAM	Storage
Bootstrap	RHCOS	4	16 GB	120 GB
Control plane	RHCOS	4	16 GB	120 GB
Compute	RHCOS or RHEL 7.6	2	8 GB	120 GB

Certificate signing requests management

Because the cluster has limited access to automatic machine management when using infrastructure that we provision, provide a mechanism for approving cluster certificate signing requests (CSRs) after installation. The kube-controller-manager only approves the kubelet client CSRs. The machine-approver cannot guarantee the validity of a serving certificate requested using kubelet credentials because it cannot confirm that the correct machine issued the request. Implement a method of verifying the validity of the kubelet serving certificate requests and approving them.

Create the user-provisioned infrastructure

Before deploying a OpenShift cluster that uses user-provisioned infrastructure, create the underlying infrastructure.

Prerequistes

• Review the OpenShift 4.x Tested Integrations page before creating the supporting infrastructure for the cluster.

Procedure

1. Configure DHCP.
2. Provision the required load balancers.
3. Configure the ports for the machines.
4. Configure DNS.
5. Ensure network connectivity.

Networking requirements for user-provisioned infrastructure

All the RHCOS machines require initramfs during boot to fetch Ignition config from the Machine Config Server.

During the initial boot, the machines require a DHCP server to establish a network connection to download Ignition config files.

It is recommended to use the DHCP server to manage the machines for the cluster long-term. Ensure that the DHCP server is configured to provide persistent IP addresses and host names to the cluster machines.

The Kubernetes API server must be able to resolve the node names of the cluster machines. If the API servers and worker nodes are in different zones, configure a default DNS search zone to allow the API server to resolve the node names. Another acceptable approach is to always refer to hosts by their fully-qualified domain names in both the node objects and all DNS requests.

Configure the network connectivity between machines to allow cluster components to communicate. Each machine must be able to resolve the host names of all other machines in the cluster.

Ports	Description
2379-2380	etcd server, peer, and metrics
6443	Kubernetes API
9000-9999	Host level services, including the node exporter on ports 9100-9101 and the Cluster Version Operator on port 9099.
10249-10259	The default ports that Kubernetes reserves
10256	openshift-sdn
30000-32767	Kubernetes NodePort

Network topology requirements

The infrastructure provisioned for the cluster must meet the following network topology requirements.

OpenShift requires all nodes to have internet access to pull images for platform containers and provide telemetry data to Red Hat.

Load balancers

Before installing OpenShift, provision two layer-4 load balancers.

Port	Machines	Internal	External	Description
6443	Bootstrap and control plane, remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane.	x	x	Kubernetes API server
22623	Bootstrap and control plane, remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane.	x		Machine Config server
443	The machines that run the Ingress router pods, compute, or worker, by default.	x	x	HTTPS traffic
80	The machines that run the Ingress router pods, compute, or worker by default.	x	x	HTTP traffic

A working configuration for the Ingress router is required for an OpenShift cluster. Configure the Ingress router after the control plane initializes.

User-provisioned DNS requirements

The following DNS records are required for a OpenShift cluster that uses user-provisioned infrastructure. In each record, cluster_name is the cluster name and base_domain is the cluster base domain specified in the install-config.yaml file.

Table 3.2. Required DNS records

Component	Record	Description
Kubernetes API	api.cluster_name.base_domain	This DNS record must point to the load balancer for the control plane machines. This record must be resolvable by both clients external to the cluster and from all the nodes within the cluster.
	api-int.cluster_name.base_domain	This DNS record must point to the load balancer for the control plane machines. This record must be resolvable from all the nodes within the cluster. The API server must be able to resolve the worker nodes by the host names that are recorded in Kubernetes. If it cannot resolve the node names, proxied API calls can fail, and we cannot retrieve logs from Pods.
Routes	*.apps.cluster_name.base_domain	A wildcard DNS record that points to the load balancer that targets the machines that run the Ingress router pods, which are the worker nodes by default. This record must be resolvable by both clients external to the cluster and from all the nodes within the cluster.
etcd	etcd-index.cluster_name.base_domain	OpenShift requires DNS records for each etcd instance to point to the control plane machines that host the instances. The etcd instances are differentiated by index values, which start with 0 and end with n-1, where n is the number of control plane machines in the cluster. The DNS record must resolve to an unicast IPV4 address for the control plane machine, and the records must be resolvable from all the nodes in the cluster.
	_etcd-server-ssl._tcp.cluster_name.base_domain	For each control plane machine, OpenShift also requires a SRV DNS record for etcd server on that machine with priority 0, weight 10 and port 2380. A cluster that uses three control plane machines requires the following records:

# _service._proto.name.                            TTL    class SRV priority weight port target.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-0.cluster_name.base_domain.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-1.cluster_name.base_domain.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-2.cluster_name.base_domain.

# _service._proto.name.                            TTL    class SRV priority weight port target.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-0.cluster_name.base_domain.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-1.cluster_name.base_domain.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-2.cluster_name.base_domain.

Generate an SSH private key and add it to the agent

For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key for the ssh-agent process.

We can use this key to SSH onto the master nodes as the user core. When we deploy the cluster, the key is added to the core user's ~/.ssh/authorized_keys list.

Use a local key, not one configured with platform-specific approaches such as AWS key pairs.

Procedure

1. If we do not have an SSH key configured for password-less authentication on the computer, create one. For example, on a computer that uses a Linux operating system, run:
   $ ssh-keygen -t rsa -b 4096 -N '' -f /path/to/.ssh/id_rsa

2. Start the ssh-agent process as a background task:
   $ eval "$(ssh-agent -s)"
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:
   $ ssh-add /path/to/.ssh/id_rsa
   Identity added: /path/to/.ssh/id_rsa (computer_name)

Next steps

When we install OpenShift, provide the SSH public key to the installer. If we install a cluster on infrastructure that we provision, provide this key to the cluster's machines.

Obtain the installation program

Before installing OpenShift, download the installation file on a local computer.

Prerequisites

• Install the cluster from a computer that uses Linux or macOS.
• You need 300 MB of local disk space to download the installation program.

Procedure

1. Access the OpenShift Infrastructure Providers page. If we have a Red Hat account, log in with your credentials. If we do not, create an account.

2. Download the installation program for your operating system and place the file in the directory where you will store the installation configuration files.

   The installation program creates several files on the computer used to install the cluster. Keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run:

   $ tar xvf installation_program.tar.gz
   

4. From the OpenShift Infrastructure Providers page, download your installation pull secret. This pull secret allows us to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift components.

Install the OpenShift Command-line Interface (oc)

Earlier versions of oc cannot be used to complete all of the commands in OpenShift 4.1. Download and install the new version of oc.

1. From the OpenShift Infrastructure Providers page, click...
   Download Command-line Tools

2. Download the compressed file for your operating system.

   We can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Manually create the installation configuration file

For installations of OpenShift that use user-provisioned infrastructure, manually generate the installation configuration file.

Prerequisites

• Obtain the OpenShift installation program and the access token for the cluster.

Procedure

1. Create an installation directory to store installation assets in:

   $ mkdir /path/to/install/files
   

   Create a directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so do not reuse an installation directory. To reuse individual files from another cluster installation, copy them into the directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift version.

2. Customize the following install-config.yaml file template and save it in the /path/to/install/files.

   Name this configuration file install-config.yaml.

3. Back up the install-config.yaml file so that we can use it to install multiple clusters.

   The install-config.yaml file is consumed during the next step of the installation process. Back it up now.

Sample install-config.yaml file for bare metal

We can customize the install-config.yaml file to specify more details about the OpenShift cluster's platform or modify the values of the required parameters.

apiVersion: v1
baseDomain: example.com 1
compute:
- hyperthreading: Enabled 2 3
  name: worker
  replicas: 0 4
controlPlane:
  hyperthreading: Enabled 5 6
  name: master 7
  replicas: 3 8
metadata:
  name: test 9
networking:
  clusterNetworks:
  - cidr: 10.128.0.0/14 10
    hostPrefix: 23 11
  networkType: OpenShiftSDN
  serviceNetwork: 12
  - 172.30.0.0/16
platform:
  none: {} 13
pullSecret: '{"auths": ...}' 14
sshKey: 'ssh-ed25519 AAAA...' 15

1 The base domain of the cluster. All DNS records must be sub-domains of this base and include the cluster name.

2 5 The controlPlane section is a single mapping, but the compute section is a sequence of mappings. To meet the requirements of the different data structures, the first line of the compute section must begin with a hyphen, -, and the first line of the controlPlane section must not. Although both sections currently define a single machine pool, it is possible that future versions of OpenShift will support defining multiple compute pools during installation. Only one control plane pool is used.

3 6 7 Whether to enable or disable simultaneous multithreading, or hyperthreading. By default, simultaneous multithreading is enabled to increase the performance of the machines' cores. We can disable it by setting the parameter value to Disabled. If you disable simultanous multithreading in some cluster machines, disable it in all cluster machines.

If you disable simultaneous multithreading, ensure that your capacity planning accounts for the dramatically decreased machine performance.

4 Set the value of the replicas parameter to 0. This parameter controls the number of workers that the cluster creates and manages for you, which are functions that the cluster does not perform when using user-provisioned infrastructure. Manually deploy worker machines for the cluster to use before you finish installing OpenShift.

8 The number of control plane machines that you add to the cluster. Because the cluster uses this values as the number of etcd endpoints in the cluster, the value must match the number of control plane machines that you deploy.

9 The cluster name that you specified in your DNS records.

10 A block of IP addresses from which Pod IP addresses are allocated. This block must not overlap with existing physical networks. These IP addresses are used for the Pod network, and if you need to access the Pods from an external network, configure load balancers and routers to manage the traffic.

11 The subnet prefix length to assign to each individual node. For example, if hostPrefix is set to 23, then each node is assigned a /23 subnet out of the given cidr, which allows for 510 (2^(32 - 23) - 2) pod IPs addresses. If we are required to provide access to nodes from an external network, configure load balancers and routers to manage the traffic.

12 The IP address pool to use for service IP addresses. We can enter only one IP address pool. To access the services from an external network, configure load balancers and routers to manage the traffic.

13 Set the platform to none. We cannot provide additional platform configuration variables for bare metal infrastructure.

14 The pull secret obtained from the OpenShift Infrastructure Providers page. This pull secret allows us to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift components.

15 The public portion of the default SSH key for the core user in Red Hat Enterprise Linux CoreOS (RHCOS).

For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installation program.

Create the Ignition config files

Because manually start the cluster machines, generate the Ignition config files that the cluster needs to make its machines.

The Ignition config files that the installation program generates contain certificates that expire after 24 hours. Complete the cluster installation and keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

Prerequisites

• Obtain the OpenShift installation program and the pull secret for the cluster.

Procedure

1. Obtain the Ignition config files:

   $ ./openshift-install create ignition-configs --dir=/path/to/install/files 
   

   If createdd an install-config.yaml file, specify the directory that contains it. Otherwise, specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so do not reuse an installation directory. To reuse individual files from another cluster installation, copy them into the directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift version.

   The following files are generated in the directory:

   .
   auth
        kubeadmin-password
        kubeconfig
        bootstrap.ign
       master.ign
   metadata.json
   worker.ign
   

Create Red Hat Enterprise Linux CoreOS (RHCOS) machines

Before installing a cluster on bare metal infrastructure that we provision, create RHCOS machines for it to use. Follow either the steps to use an ISO image or network PXE booting to create the machines.

Create Red Hat Enterprise Linux CoreOS (RHCOS) machines using an ISO image

Before installing a cluster on bare metal infrastructure that we provision, create RHCOS machines for it to use. We can use an ISO image to create the machines.

Prerequisites

• Obtain the Ignition config files for the cluster.
• Have access to an HTTP server that we can access from our computer and that the machines that created can access.

Procedure

1. Upload the control plane, compute, and bootstrap Ignition config files that the installation program created to your HTTP server. Note the URLs of these files.

2. Obtain the RHCOS images required for your preferred method of installing operating system instances from the Product Downloads page on the Red Hat customer portal or the RHCOS image mirror page.

   The RHCOS images might not change with every release of OpenShift. Download images with the highest version that is less than or equal to the OpenShift version that we install. Use the image versions that match the OpenShift version if they are available.

   Download the ISO file and either the BIOS or UEFI file. Those file names resemble the following examples:

   • ISO: rhcos-version-architecture-installer.iso
   • Compressed metal BIOS: rhcos-version-architecture-metal-bios.raw.gz
   • Compressed metal UEFI: rhcos-version-architecture-metal-uefi.raw.gz
3. Upload either the BIOS or UEFI RHCOS image file to your HTTP server and note its URL.

4. Use the ISO to start the RHCOS installation. Use one of the following installation options:
   • Burn the ISO image to a disk and boot it directly.
   • Use ISO redirection via a LOM interface.
5. After the instance boots, press the TAB or E key to edit the kernel command line.

6. Add the parameters to the kernel command line:

   coreos.inst=yes
   coreos.inst.install_dev=sda 1
   coreos.inst.image_url=bare_metal_image_URL 2
   coreos.inst.ignition_url=http://example.com/config.ign 3
   

   1 Specify the block device of the system to install to.

   2 Specify the URL of the UEFI or BIOS image that you uploaded to your server.

   3 Specify the URL of the Ignition config file for this machine type.

7. Press Enter to complete the installation. After RHCOS installs, the system reboots. After the system reboots, it applies the Ignition config file that you specified.

8. Continue to create the machines for the cluster.

   Create the boostrap and control plane machines at this time. Because some pods are deployed on compute machines by default, also create at least two compute machines before installing the cluster.

Create Red Hat Enterprise Linux CoreOS (RHCOS) machines by PXE or iPXE booting

Before installing a cluster on bare metal infrastructure that we provision, create RHCOS machines for it to use. We can use PXE or iPXE booting to create the machines.

Prerequisites

• Obtain the Ignition config files for the cluster.
• Configure suitable PXE or iPXE infrastructure.
• Have access to an HTTP server that we can access from our computer.

Procedure

1. Upload the master, worker, and bootstrap Ignition config files that the installation program created to your HTTP server. Note the URLs of these files.

2. Obtain the RHCOS ISO image, compressed metal BIOS, kernel and initramfs files from the Product Downloads page on the Red Hat customer portal or the RHCOS image mirror page.

   The RHCOS images might not change with every release of OpenShift. Download images with the highest version that is less than or equal to the OpenShift version that we install. Use the image versions that match the OpenShift version if they are available.

   The file names contain the OpenShift version number. They resemble the following examples:

   • ISO: rhcos-version-architecture-installer.iso
   • Compressed metal BIOS: rhcos-version-architecture-metal-bios.raw.gz
   • kernel: rhcos-version-architecture-installer-kernel
   • initframs: rhcos-version-architecture-installer-initramfs.img
3. Upload the compressed metal BIOS file and the kernel and initramfs files to your HTTP server.
4. Configure the network boot infrastructure so that the machines boot from their local disks after RHCOS is installed on them.

5. Configure PXE or iPXE installation for the RHCOS images.

   Modify one of the following example menu entries for your environment and verify that the image and Ignition files are properly accessible:

   • For PXE:

     DEFAULT pxeboot
     TIMEOUT 20
     PROMPT 0
     LABEL pxeboot
         KERNEL http://HTTP_server/rhcos-version-architecture-installer-kernel 1
         APPEND ip=dhcp rd.neednet=1 initrd=http://HTTP_server/rhcos-version-architecture-installer-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url=http://HTTP_server/rhcos-version-architecture-metal-bios.raw.gz coreos.inst.ignition_url=http://HTTP_server/bootstrap.ign 2 3
     

     1 Location of the kernel file that you uploaded to your HTTP server.

     2 If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC named eno1, set ip=eno1:dhcp.

     3 Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the initramfs file, the coreos.inst.image_url parameter value is the location of the compressed metal BIOS file, and the coreos.inst.ignition_url parameter value is the location of the bootstrap Ignition config file.

   • For iPXE:

     kernel  http://HTTP_server/rhcos-version-architecture-installer-kernel ip=dhcp rd.neednet=1 initrd=http://HTTP_server/rhcos-version-architecture-installer-initramfs.img console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url=http://HTTP_server/rhcos-version-architecture-metal-bios.raw.gz coreos.inst.ignition_url=http://HTTP_server/bootstrap.ign 1 2
     initrd http://HTTP_server/rhcos-version-architecture-installer-initramfs.img 3
     boot
     

     1 Specify locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd parameter value is the location of the initramfs file, the coreos.inst.image_url parameter value is the location of the compressed metal BIOS file, and the coreos.inst.ignition_url parameter value is the location of the bootstrap Ignition config file.

     2 If you use multiple NICs, specify a single interface in the ip option. For example, to use DHCP on a NIC named eno1, set ip=eno1:dhcp.

     3 Location of the initramfs file that you uploaded to your HTTP server.

6. If you use UEFI, edit the included grub.conf file that is included in the ISO that we downloaded to include the following installation options:

   menuentry 'Install Red Hat Enterprise Linux CoreOS' --class fedora --class gnu-linux --class gnu --class os {
       linux /images/vmlinuz nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=sda coreos.inst.image_url=http://HTTP_server/rhcos-version-architecture-metal-bios.raw.gz coreos.inst.ignition_url=http://HTTP_server/bootstrap.ign 1
       initrd http://HTTP_server/rhcos-version-architecture-installer-initramfs.img 2
   }
   

   1 For the coreos.inst.image_url parameter value, specify the location of the compressed metal UEFI file that you uploaded to your HTTP server. For the coreos.inst.ignition_url, specify the location of the bootstrap Ingition config file that you uploaded to your HTTP server.

   2 Location of the initramfs file that you uploaded to your HTTP server.

7. Continue to create the machines for the cluster.

   Create the boostrap and control plane machines at this time. Because some pods are deployed on compute machines by default, also create at least two compute machine before installing the cluster.

Create the cluster

To create the OpenShift cluster, we provision machines using the Ignition config files that you generated with the installation program.

Prerequisites

• Create the required infrastructure for the cluster.
• You obtained the installation program and generated the Ignition config files for the cluster.
• We used the Ignition config files to create RHCOS machines for the cluster.
• Your machines have direct internet access.

Procedure

1. Start and monitor the installation process:

   $ ./openshift-install --dir=/path/to/install/files wait-for bootstrap-complete --log-level debug 
   
   INFO Waiting up to 30m0s for the Kubernetes API at https://api.test.example.com:6443...
   INFO API v1.13.4+b626c2fe1 up
   INFO Waiting up to 30m0s for the bootstrap-complete event...
   

   The command succeeds when the Kubernetes API server signals that it has been bootstrapped on the control plane machines.

2. After bootstrap process is complete, remove the bootstrap machine from the load balancer.

   Remove the bootstrap machine from the load balancer at this point. We can also remove or reformat the machine itself.

Log on to the cluster

1. Deploy an OpenShift cluster.

2. Install the oc CLI.

3. Export the kubeadmin credentials:
   $ export KUBECONFIG=/path/to/install/files/auth/kubeconfig
   $ oc whoami
   system:admin

   The kubeconfig file contains information about the cluster used by the CLI to connect a client to the correct cluster and API server. The file is specific to a cluster and is created during OpenShift installation.

Next steps

• Customize the cluster.
• If necessary, we can opt out of telemetry.

Approve the CSRs for the machines

When you add machines to a cluster, two pending certificates signing request (CSRs) are generated for each machine that you added. Cofirm that these CSRs are approved or, if necessary, approve them ourselves.

Prerequisites

• You added machines to the cluster.
• Install the jq package.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes
   
   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.13.4+b626c2fe1
   master-1  Ready     master  63m  v1.13.4+b626c2fe1
   master-2  Ready     master  64m  v1.13.4+b626c2fe1
   worker-0  NotReady  worker  76s  v1.13.4+b626c2fe1
   worker-1  NotReady  worker  70s  v1.13.4+b626c2fe1
   

   The output lists all of the machines that createdd.

2. Review the pending certificate signing requests (CSRs) and ensure that you see a client and server request with Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr
   
   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 1
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending 2
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...
   

   1 A client request CSR.

   2 A server request CSR.

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for the cluster machines:

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If we do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. Approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. Implement a method of automatically approving the kubelet serving certificate requests.

   • To approve them individually, run for each valid CSR:
     $ oc adm certificate approve csr_name ¹

     1 csr_name is the name of a CSR from the list of current CSRs.

   • If all the CSRs are valid, approve them all:
     $ oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

Initial Operator configuration

After the control plane initializes, immediately configure some Operators so that they all become available.

Prerequisites

• Your control plane has initialized.

Procedure

1. Watch the cluster components come online:

   $ watch -n5 oc get clusteroperators
   
   NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                       4.1.0     True        False         False      69s
   cloud-credential                     4.1.0     True        False         False      12m
   cluster-autoscaler                   4.1.0     True        False         False      11m
   console                              4.1.0     True        False         False      46s
   dns                                  4.1.0     True        False         False      11m
   image-registry                       4.1.0     False       True          False      5m26s
   ingress                              4.1.0     True        False         False      5m36s
   kube-apiserver                       4.1.0     True        False         False      8m53s
   kube-controller-manager              4.1.0     True        False         False      7m24s
   kube-scheduler                       4.1.0     True        False         False      12m
   machine-api                          4.1.0     True        False         False      12m
   machine-config                       4.1.0     True        False         False      7m36s
   marketplace                          4.1.0     True        False         False      7m54m
   monitoring                           4.1.0     True        False         False      7h44s
   network                              4.1.0     True        False         False      5m9s
   node-tuning                          4.1.0     True        False         False      11m
   openshift-apiserver                  4.1.0     True        False         False      11m
   openshift-controller-manager         4.1.0     True        False         False      5m943s
   openshift-samples                    4.1.0     True        False         False      3m55s
   operator-lifecycle-manager           4.1.0     True        False         False      11m
   operator-lifecycle-manager-catalog   4.1.0     True        False         False      11m
   service-ca                           4.1.0     True        False         False      11m
   service-catalog-apiserver            4.1.0     True        False         False      5m26s
   service-catalog-controller-manager   4.1.0     True        False         False      5m25s
   storage                              4.1.0     True        False         False      5m30s
   

2. Configure the Operators that are not available.

Image registry storage configuration

If the image-registry Operator is not available, configure storage for it. Instructions for both configuring a PersistentVolume, which is required for production clusters, and for configuring an empty directory as the storage location, which is available for only non-production clusters, are shown.

Configure registry storage for bare metal

As a cluster administrator, following installation configure the registry to use storage.

Prerequisites

• Cluster administrator permissions.
• A cluster on bare metal.
• A provisioned persistent volume (PV) with ReadWriteMany access mode, such as NFS.
• Must have "100Gi" capacity.

Procedure

1. To configure the registry to use storage, change the spec.storage.pvc in the configs.imageregistry/cluster resource.

2. Verify we do not have a registry pod:

   $ oc get pod -n openshift-image-registry
   

If the storage type is emptyDIR, the replica number can not be greater than 1. If the storage type is NFS, and we want to scale up the registry Pod by setting replica1 enable the no_wdelay mount option. For example:

# cat /etc/exports
/mnt/data *(rw,sync,no_wdelay,no_root_squash,insecure,fsid=0)
sh-4.3# exportfs -rv
exporting *:/mnt/data

1. Check the registry configuration:

   $ oc edit configs.imageregistry.operator.openshift.io
   
   storage:
     pvc:
       claim:
   

   Leave the claim field blank to allow the automatic creation of an image-registry-storage PVC.

2. Check the clusteroperator status:

   $ oc get clusteroperator image-registry
   

Configure storage for the image registry in non-production clusters

Configure storage for the image registry Operator. For non-production clusters, we can set the image registry to an empty directory. If we do so, all images are lost if you restart the registry.

Procedure

• To set the image registry storage to an empty directory:

  $ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
  

  Warning

  Configure this option for only non-production clusters.

  If you run this command before the Image Registry Operator initializes its components, the oc patch command fails with the following error:

  Error from server (NotFound): configs.imageregistry.operator.openshift.io "cluster" not found
  

  Wait a few minutes and run the command again.

Complete installation on user-provisioned infrastructure

After you complete the Operator configuration, we can finish installing the cluster on infrastructure that we provide.

Prerequisites

• Your control plane has initialized.
• You have completed the initial Operator configuration.

Procedure

1. Confirm that all the cluster components are online:

   $ watch -n5 oc get clusteroperators
   
   NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                       4.1.0     True        False         False      10m
   cloud-credential                     4.1.0     True        False         False      22m
   cluster-autoscaler                   4.1.0     True        False         False      21m
   console                              4.1.0     True        False         False      10m
   dns                                  4.1.0     True        False         False      21m
   image-registry                       4.1.0     True        False         False      16m
   ingress                              4.1.0     True        False         False      16m
   kube-apiserver                       4.1.0     True        False         False      19m
   kube-controller-manager              4.1.0     True        False         False      18m
   kube-scheduler                       4.1.0     True        False         False      22m
   machine-api                          4.1.0     True        False         False      22m
   machine-config                       4.1.0     True        False         False      18m
   marketplace                          4.1.0     True        False         False      18m
   monitoring                           4.1.0     True        False         False      18m
   network                              4.1.0     True        False         False      16m
   node-tuning                          4.1.0     True        False         False      21m
   openshift-apiserver                  4.1.0     True        False         False      21m
   openshift-controller-manager         4.1.0     True        False         False      17m
   openshift-samples                    4.1.0     True        False         False      14m
   operator-lifecycle-manager           4.1.0     True        False         False      21m
   operator-lifecycle-manager-catalog   4.1.0     True        False         False      21m
   service-ca                           4.1.0     True        False         False      21m
   service-catalog-apiserver            4.1.0     True        False         False      16m
   service-catalog-controller-manager   4.1.0     True        False         False      16m
   storage                              4.1.0     True        False         False      16m
   

   When all of the cluster Operators are AVAILABLE, we can complete the installation.

2. Monitor for cluster completion:

   $ ./openshift-install --dir=/path/to/install/files wait-for install-complete 
   INFO Waiting up to 30m0s for the cluster to initialize...
   

   The command succeeds when the Cluster Version Operator finishes deploying the OpenShift cluster from Kubernetes API server.

3. Confirm that the Kubernetes API server is communicating with the Pods.

   1. To view a list of all Pods, use the following command:

      $ oc get pods --all-namespaces
      
      NAMESPACE                         NAME                                            READY   STATUS      RESTARTS   AGE
      openshift-apiserver-operator      openshift-apiserver-operator-85cb746d55-zqhs8   1/1     Running     1          9m
      openshift-apiserver               apiserver-67b9g                                 1/1     Running     0          3m
      openshift-apiserver               apiserver-ljcmx                                 1/1     Running     0          1m
      openshift-apiserver               apiserver-z25h4                                 1/1     Running     0          2m
      openshift-authentication-operator authentication-operator-69d5d8bf84-vh2n8        1/1     Running     0          5m
      ...
      

   2. View the logs for a Pod that is listed in the output of the previous command using the following command:

      $ oc logs pod_name -n namespace 1
      

      1 Specify the Pod name and namespace, as shown in the output of the previous command.

      If the Pod logs display, the Kubernetes API server can communicate with the cluster machines.

Next steps

• Customize the cluster.
• If necessary, we can opt out of telemetry.

Install on vSphere

We can install a cluster on VMware vSphere infrastructure that we provision.

Prerequisites

• Provision persistent storage for the cluster. To deploy a private image registry, your storage must provide ReadWriteMany access modes.
• Review details about the OpenShift installation and update processes.
• If a firewall is used, configure it to access Red Hat Insights.

Internet and Telemetry access

Telemetry provides metrics about cluster health and the success of updates. To perform subscription management, including legally entitling a purchase Red Hat, use the Telemetry service and access the OpenShift Infrastructure Providers page.

There is no disconnected subscription management. We cannot both opt out of sending data back to Red Hat and entitle a purchase.

Machines have direct internet access to install the cluster.

Internet access is required to:

• Access the OpenShift Infrastructure Providers page to download the installation program
• Access quay.io to obtain the packages required to install the cluster
• Obtain the packages required to perform cluster updates
• Access Red Hat's software as a service page to perform subscription management

VMware vSphere infrastructure requirements

Install the OpenShift cluster on a VMware vSphere version 6.5 or 6.7U2 or later instance.

VMware recommends using vSphere Version 6.7 U2 or later. vSphere 6.7U2 includes:

• Support for VMware NSX-T
• Support for vSAN, VMFS and NFS, using the in-tree VCP

While vSphere 6.5 with Hardware version 11 is supported, OpenShift clusters are subject to the following restrictions:

• NSX-T SDN is not supported.
• Use another SDN or storage provider that OpenShift supports.

For a vSphere version 6.5 instance, consider upgrading to 6.7U2 before installing OpenShift.

Machine requirements for a cluster with user-provisioned infrastructure

For a cluster that contains user-provisioned infrastructure, deploy all of the required machines.

Required machines

The smallest OpenShift clusters require the following hosts:

• One bootstrap machine
• Three control plane, or master, machines
• At least two compute, or worker, machines

The bootstrap machine deploys OpenShift cluster on the three control plane machines. We can remove the bootstrap machine after installing the cluster.

To maintain high availability of the cluster, use separate physical hosts for these cluster machines.

The bootstrap and control plane machines must use Red Hat Enterprise Linux CoreOS (RHCOS) as the operating system.

Network connectivity requirements

All the RHCOS machines require initramfs during boot to fetch Ignition config files from the Machine Config Server. During the initial boot, the machines require a DHCP server to establish a network connection to download Ignition config files. After the initial boot, the machines can be configured to use static IP addresses.

Minimum resource requirements

Each cluster machine must meet the following minimum requirements:

Machine	Operating System	vCPU	RAM	Storage
Bootstrap	RHCOS	4	16 GB	120 GB
Control plane	RHCOS	4	16 GB	120 GB
Compute	RHCOS or RHEL 7.6	2	8 GB	120 GB

Certificate signing requests management

Because the cluster has limited access to automatic machine management when using infrastructure that we provision, provide a mechanism for approving cluster certificate signing requests (CSRs) after installation. The kube-controller-manager only approves the kubelet client CSRs. The machine-approver cannot guarantee the validity of a serving certificate requested using kubelet credentials because it cannot confirm that the correct machine issued the request. Implement a method of verifying the validity of the kubelet serving certificate requests and approving them.

Create the user-provisioned infrastructure

Before deploying a OpenShift cluster that uses user-provisioned infrastructure, create the underlying infrastructure.

Prerequistes

• Review the OpenShift 4.x Tested Integrations page before creating the supporting infrastructure for the cluster.

Procedure

1. Configure DHCP.
2. Provision the required load balancers.
3. Configure the ports for the machines.
4. Configure DNS.
5. Ensure network connectivity.

Networking requirements for user-provisioned infrastructure

All the RHCOS machines require initramfs during boot to fetch Ignition config from the Machine Config Server.

During the initial boot, the machines require a DHCP server to establish a network connection to download Ignition config files.

It is recommended to use the DHCP server to manage the machines for the cluster long-term. Ensure that the DHCP server is configured to provide persistent IP addresses and host names to the cluster machines.

The Kubernetes API server must be able to resolve the node names of the cluster machines. If the API servers and worker nodes are in different zones, configure a default DNS search zone to allow the API server to resolve the node names. Another acceptable approach is to always refer to hosts by their fully-qualified domain names in both the node objects and all DNS requests.

Configure the network connectivity between machines to allow cluster components to communicate. Each machine must be able to resolve the host names of all other machines in the cluster.

Ports	Description
2379-2380	etcd server, peer, and metrics
6443	Kubernetes API
9000-9999	Host level services, including the node exporter on ports 9100-9101 and the Cluster Version Operator on port 9099.
10249-10259	The default ports that Kubernetes reserves
10256	openshift-sdn
30000-32767	Kubernetes NodePort

Network topology requirements

The infrastructure provisioned for the cluster must meet the following network topology requirements.

OpenShift requires all nodes to have internet access to pull images for platform containers and provide telemetry data to Red Hat.

Load balancers

Before installing OpenShift, provision two layer-4 load balancers.

Port	Machines	Internal	External	Description
6443	Bootstrap and control plane, remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane.	x	x	Kubernetes API server
22623	Bootstrap and control plane, remove the bootstrap machine from the load balancer after the bootstrap machine initializes the cluster control plane.	x		Machine Config server
443	The machines that run the Ingress router pods, compute, or worker, by default.	x	x	HTTPS traffic
80	The machines that run the Ingress router pods, compute, or worker by default.	x	x	HTTP traffic

A working configuration for the Ingress router is required for an OpenShift cluster. Configure the Ingress router after the control plane initializes.

User-provisioned DNS requirements

The following DNS records are required for a OpenShift cluster that uses user-provisioned infrastructure. In each record, cluster_name is the cluster name and base_domain is the cluster base domain specified in the install-config.yaml file.

Component	Record	Description
Kubernetes API	api.cluster_name.base_domain	This DNS record must point to the load balancer for the control plane machines. This record must be resolvable by both clients external to the cluster and from all the nodes within the cluster.
	api-int.cluster_name.base_domain	This DNS record must point to the load balancer for the control plane machines. This record must be resolvable from all the nodes within the cluster. The API server must be able to resolve the worker nodes by the host names that are recorded in Kubernetes. If it cannot resolve the node names, proxied API calls can fail, and we cannot retrieve logs from Pods.
Routes	*.apps.cluster_name.base_domain	A wildcard DNS record that points to the load balancer that targets the machines that run the Ingress router pods, which are the worker nodes by default. This record must be resolvable by both clients external to the cluster and from all the nodes within the cluster.
etcd	etcd-index.cluster_name.base_domain	OpenShift requires DNS records for each etcd instance to point to the control plane machines that host the instances. The etcd instances are differentiated by index values, which start with 0 and end with n-1, where n is the number of control plane machines in the cluster. The DNS record must resolve to an unicast IPV4 address for the control plane machine, and the records must be resolvable from all the nodes in the cluster.
	_etcd-server-ssl._tcp.cluster_name.base_domain	For each control plane machine, OpenShift also requires a SRV DNS record for etcd server on that machine with priority 0, weight 10 and port 2380. A cluster that uses three control plane machines requires the following records:

# _service._proto.name.                            TTL    class SRV priority weight port target.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-0.cluster_name.base_domain.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-1.cluster_name.base_domain.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-2.cluster_name.base_domain.

# _service._proto.name.                            TTL    class SRV priority weight port target.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-0.cluster_name.base_domain.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-1.cluster_name.base_domain.
_etcd-server-ssl._tcp.cluster_name.base_domain 86400 IN    SRV 0        10     2380 etcd-2.cluster_name.base_domain.

Generate an SSH private key and add it to the agent

For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key for the ssh-agent process

We can use this key to SSH onto the master nodes as the user core. When we deploy the cluster, the key is added to the core user's ~/.ssh/authorized_keys list.

Use a local key, not one configured with platform-specific approaches such as AWS key pairs.

Procedure

1. If we do not have an SSH key configured for password-less authentication on the computer, create one. For example, on a computer that uses a Linux operating system, run:
   $ ssh-keygen -t rsa -b 4096 -N '' -f /path/to/.ssh/id_rsa

2. Start the ssh-agent process as a background task:
   $ eval "$(ssh-agent -s)"
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:
   $ ssh-add /path/to/.ssh/id_rsa
   Identity added: /path/to/.ssh/id_rsa (computer_name)

Next steps

When we install OpenShift, provide the SSH public key to the installer. If we install a cluster on infrastructure that we provision, provide this key to the cluster's machines.

Obtain the installation program

Before installing OpenShift, download the installation file on a local computer.

Prerequisites

• Install the cluster from a computer that uses Linux or macOS.
• You need 300 MB of local disk space to download the installation program.

Procedure

1. Access the OpenShift Infrastructure Providers page. If we have a Red Hat account, log in with your credentials. If we do not, create an account.

2. Download the installation program for your operating system and place the file in the directory where you will store the installation configuration files.

   The installation program creates several files on the computer used to install the cluster. Keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run:

   $ tar xvf installation_program.tar.gz
   

4. From the OpenShift Infrastructure Providers page, download your installation pull secret. This pull secret allows us to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift components.

Manually create the installation configuration file

For installations of OpenShift using user-provisioned infrastructure, manually generate the installation configuration file.

Prerequisites

• Obtain the OpenShift installation program and the access token for the cluster.

Procedure

1. Create an installation directory to store installation assets in:

   $ mkdir /path/to/install/files
   

   Create a directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so do not reuse an installation directory. To reuse individual files from another cluster installation, copy them into the directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift version.

2. Customize the following install-config.yaml file template and save it in the /path/to/install/files.

   Name this configuration file install-config.yaml.

3. Back up the install-config.yaml file so that we can use it to install multiple clusters.

   The install-config.yaml file is consumed during the next step of the installation process. Back it up now.

Sample install-config.yaml file for VMware vSphere

We can customize the install-config.yaml file to specify more details about the OpenShift cluster's platform or modify the values of the required parameters.

apiVersion: v1
baseDomain: example.com 1
compute:
- hyperthreading: Enabled 2 3
  name: worker
  replicas: 0 4
controlPlane:
  hyperthreading: Enabled 5 6
  name: master
  replicas: 3 7
metadata:
  name: test 8
platform:
  vsphere:
    vcenter: your.vcenter.server 9
    username: username 10
    password: password 11
    datacenter: datacenter 12
    defaultDatastore: datastore 13
pullSecret: '{"auths": ...}' 14
sshKey: 'ssh-ed25519 AAAA...' 15

1 The base domain of the cluster. All DNS records must be sub-domains of this base and include the cluster name.

2 5 The controlPlane section is a single mapping, but the compute section is a sequence of mappings. To meet the requirements of the different data structures, the first line of the compute section must begin with a hyphen, -, and the first line of the controlPlane section must not. Although both sections currently define a single machine pool, it is possible that future versions of OpenShift will support defining multiple compute pools during installation. Only one control plane pool is used.

3 6 Whether to enable or disable simultaneous multithreading, or hyperthreading. By default, simultaneous multithreading is enabled to increase the performance of the machines' cores. We can disable it by setting the parameter value to Disabled. If you disable simultanous multithreading in some cluster machines, disable it in all cluster machines.

If you disable simultaneous multithreading, ensure that your capacity planning accounts for the dramatically decreased machine performance. Machines use at least 8 CPUs and 32 GB of RAM if you disable simultaneous multithreading.

4 Set the value of the replicas parameter to 0. This parameter controls the number of workers that the cluster creates and manages for you, which are functions that the cluster does not perform when using user-provisioned infrastructure. Manually deploy worker machines for the cluster to use before you finish installing OpenShift.

7 The number of control plane machines that you add to the cluster. Because the cluster uses this values as the number of etcd endpoints in the cluster, the value must match the number of control plane machines that you deploy.

8 The cluster name that you specified in your DNS records.

9 The fully-qualified host name or IP address of the vCenter server.

10 The name of the user for accessing the server. This user must have at least the roles and privileges required for dynamic persistent volume provisioning in vSphere.

11 The password associated with the vSphere user.

12 The vSphere datacenter.

13 The default vSphere datastore to use.

14 The pull secret obtained from the OpenShift Infrastructure Providers page. This pull secret allows us to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift components.

15 The public portion of the default SSH key for the core user in Red Hat Enterprise Linux CoreOS (RHCOS).

For production clusters on which we want to perform installation debugging or disaster recovery, provide an SSH key that your ssh-agent process uses to the installer.

Create the Ignition config files

Because manually start the cluster machines, generate the Ignition config files that the cluster needs to make its machines.

The Ignition config files that the installation program generates contain certificates that expire after 24 hours. Complete the cluster installation and keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

Prerequisites

• Obtain the OpenShift installation program and the pull secret for the cluster.

Procedure

1. Obtain the Ignition config files:

   $ ./openshift-install create ignition-configs --dir=/path/to/install/files 
   

   If createdd an install-config.yaml file, specify the directory that contains it. Otherwise, specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so do not reuse an installation directory. To reuse individual files from another cluster installation, copy them into the directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift version.

   The following files are generated in the directory:

   .
   auth
         kubeadmin-password
         kubeconfig
   bootstrap.ign
   master.ign
   metadata.json
   worker.ign
   

Create Red Hat Enterprise Linux CoreOS (RHCOS) machines in vSphere

Before installing a cluster that contains user-provisioned infrastructure on VMware vSphere, create RHCOS machines on vSphere hosts for it to use.

Prerequisites

• Obtain the Ignition config files for the cluster.
• Have access to an HTTP server that we can access from our computer and that the machines that created can access.
• Create a vSphere cluster.

Procedure

1. Upload the bootstrap Ignition config file, which is named /path/to/install/files/bootstrap.ign, that the installation program created to your HTTP server. Note the URL of this file.

   Host the bootstrap Ignition config file because it is too large to fit in a vApp property.

2. Save the following secondary Ignition config file for your bootstrap node to our computer as /path/to/install/files/append-bootstrap.ign.

   {
     "ignition": {
       "config": {
         "append": [
   {
     "source": "bootstrap_ignition_config_url", 1
     "verification": {}
   }
         ]
       },
       "timeouts": {},
       "version": "2.1.0"
     },
     "networkd": {},
     "passwd": {},
     "storage": {},
     "systemd": {}
   }
   

   1 Specify the URL of the bootstrap Ignition config file that you hosted.

   When creating the Virtual Machine (VM) for the bootstrap machine, you use this Ignition config file.

3. Convert the master, worker, and secondary bootstrap Ignition config files to Base64 encoding.

   For example, if we use a Linux operating system, we can use the base64 command to encode the files.

   $ base64 -w0 /path/to/install/files/master.ign  /path/to/install/files/master.64
   $ base64 -w0 /path/to/install/files/worker.ign  /path/to/install/files/worker.64
   $ base64 -w0 /path/to/install/files/append-bootstrap.ign  /path/to/install/files/append-bootstrap.64
   

4. Obtain the RHCOS OVA image from the Product Downloads page on the Red Hat customer portal or the RHCOS image mirror page.

   The RHCOS images might not change with every release of OpenShift. Download an image with the highest version that is less than or equal to the OpenShift version that we install. Use the image version that matches the OpenShift version if it is available.

   The file name contains the OpenShift version number in the format rhcos-version-architecture-vmware.ova.

5. In the vSphere Client, create a folder in your datacenter to store your VMs.
   1. Click the VMs and Templates view.
   2. Right-click the name of the datacenter.
   3. Click New Folder -New VM and Template Folder.
   4. In the window that is displayed, enter the folder name. The folder name must match the cluster name that you specified in the install-config.yaml file.

6. In the vSphere Client, create a template for the OVA image.

   In the following steps, we use the same template for all of the cluster machines and provide the location for the Ignition config file for that machine type when we provision the VMs.

   1. From the Hosts and Clusters tab, right-click the cluster's name and click Deploy OVF Template.
   2. On the Select an OVF tab, specify the name of the RHCOS OVA file that we downloaded.
   3. On the Select a name and folder tab, set a Virtual machine name, such as RHCOS, click the name of the vSphere cluster, and select the folder createdd in the previous step.
   4. On the Select a compute resource tab, click the name of the vSphere cluster.

   5. On the Select storage tab, configure the storage options for your VM.
      • Select Thin Provision.
      • Select the datastore that you specified in your install-config.yaml file.
   6. On the Select network tab, specify the the network configured for the cluster, if available.
   7. If you plan to use the same template for all cluster machine types, do not specify values on the Customize template tab.

7. After the template deploys, deploy a VM for a machine in the cluster.
   1. Right-click the template's name and click Clone -Clone to Virtual Machine.
   2. On the Select a name and folder tab, specify a name for the VM. You might include the machine type in the name, such as control-plane-0 or compute-1.
   3. On the Select a name and folder tab, select the name of the folder that createdd for the cluster.
   4. On the Select a compute resource tab, select the name of a host in your datacenter.
   5. Optional: On the Select storage tab, customize the storage options.
   6. On the Select clone options, select Customize this virtual machine's hardware.

   7. On the Customize hardware tab, click VM Options -Advanced.
      • From the Latency Sensitivity list, select High.

      • Click Edit Configuration, and on the Configuration Parameters window, click Add Configuration Params. Define the following parameter names and values:
        • guestinfo.ignition.config.data: Paste the contents of the base64-encoded Ignition config file for this machine type.
        • guestinfo.ignition.config.data.encoding: Specify base64.
        • disk.EnableUUID: Specify TRUE.
   8. In the Virtual Hardware panel of the Customize hardware tab, modify the specified values as required. Ensure that the amount of RAM, CPU, and disk storage meets the minimum requirements for the machine type.
   9. Complete the configuration and power on the VM.

8. Create the rest of the machines for the cluster by following the preceding steps for each machine.

   Create the boostrap and control plane machines at this time. Because some pods are deployed on compute machines by default, also create at least two compute machine before installing the cluster.

Install the OpenShift Command-line Interface (oc)

Earlier versions of oc cannot be used to complete all of the commands in OpenShift 4.1. Download and install the new version of oc.

1. From the OpenShift Infrastructure Providers page, click...
   Download Command-line Tools

2. Download the compressed file for your operating system.

   We can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Create the cluster

To create the OpenShift cluster, we provision machines using the Ignition config files that you generated with the installation program.

Prerequisites

• Create the required infrastructure for the cluster.
• You obtained the installation program and generated the Ignition config files for the cluster.
• We used the Ignition config files to create RHCOS machines for the cluster.
• Your machines have direct internet access.

Procedure

1. Start and monitor the installation process:

   $ ./openshift-install --dir=/path/to/install/files wait-for bootstrap-complete  --log-level debug 
   INFO Waiting up to 30m0s for the Kubernetes API at https://api.test.example.com:6443...
   INFO API v1.13.4+b626c2fe1 up
   INFO Waiting up to 30m0s for the bootstrap-complete event...
   

   The command succeeds when the Kubernetes API server signals that it has been bootstrapped on the control plane machines.

2. After bootstrap process is complete, remove the bootstrap machine from the load balancer.

   Remove the bootstrap machine from the load balancer at this point. We can also remove or reformat the machine itself.

Log on to the cluster

1. Deploy an OpenShift cluster.

2. Install the oc CLI.

3. Export the kubeadmin credentials:
   $ export KUBECONFIG=/path/to/install/files/auth/kubeconfig
   $ oc whoami
   system:admin

   The kubeconfig file contains information about the cluster used by the CLI to connect a client to the correct cluster and API server. The file is specific to a cluster and is created during OpenShift installation.

Next steps

• Customize the cluster.
• If necessary, we can opt out of telemetry.

Approve the CSRs for the machines

When you add machines to a cluster, two pending certificates signing request (CSRs) are generated for each machine that you added. Cofirm that these CSRs are approved or, if necessary, approve them ourselves.

Prerequisites

• You added machines to the cluster.
• Install the jq package.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes
   
   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.13.4+b626c2fe1
   master-1  Ready     master  63m  v1.13.4+b626c2fe1
   master-2  Ready     master  64m  v1.13.4+b626c2fe1
   worker-0  NotReady  worker  76s  v1.13.4+b626c2fe1
   worker-1  NotReady  worker  70s  v1.13.4+b626c2fe1
   

   The output lists all of the machines that createdd.

2. Review the pending certificate signing requests (CSRs) and ensure that you see a client and server request with Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr
   
   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 1
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending 2
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...
   

   1 A client request CSR.

   2 A server request CSR.

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for the cluster machines:

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If we do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. Approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. Implement a method of automatically approving the kubelet serving certificate requests.

   • To approve them individually, run for each valid CSR:
     $ oc adm certificate approve csr_name ¹

     1 csr_name is the name of a CSR from the list of current CSRs.

   • If all the CSRs are valid, approve them all:
     $ oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

Initial Operator configuration

After the control plane initializes, immediately configure some Operators so that they all become available.

Prerequisites

• Your control plane has initialized.

Procedure

1. Watch the cluster components come online:

   $ watch -n5 oc get clusteroperators
   
   NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                       4.1.0     True        False         False      69s
   cloud-credential                     4.1.0     True        False         False      12m
   cluster-autoscaler                   4.1.0     True        False         False      11m
   console                              4.1.0     True        False         False      46s
   dns                                  4.1.0     True        False         False      11m
   image-registry                       4.1.0     False       True          False      5m26s
   ingress                              4.1.0     True        False         False      5m36s
   kube-apiserver                       4.1.0     True        False         False      8m53s
   kube-controller-manager              4.1.0     True        False         False      7m24s
   kube-scheduler                       4.1.0     True        False         False      12m
   machine-api                          4.1.0     True        False         False      12m
   machine-config                       4.1.0     True        False         False      7m36s
   marketplace                          4.1.0     True        False         False      7m54m
   monitoring                           4.1.0     True        False         False      7h44s
   network                              4.1.0     True        False         False      5m9s
   node-tuning                          4.1.0     True        False         False      11m
   openshift-apiserver                  4.1.0     True        False         False      11m
   openshift-controller-manager         4.1.0     True        False         False      5m943s
   openshift-samples                    4.1.0     True        False         False      3m55s
   operator-lifecycle-manager           4.1.0     True        False         False      11m
   operator-lifecycle-manager-catalog   4.1.0     True        False         False      11m
   service-ca                           4.1.0     True        False         False      11m
   service-catalog-apiserver            4.1.0     True        False         False      5m26s
   service-catalog-controller-manager   4.1.0     True        False         False      5m25s
   storage                              4.1.0     True        False         False      5m30s
   

2. Configure the Operators that are not available.

Image registry storage configuration

If the image-registry Operator is not available, configure storage for it. Instructions for both configuring a PersistentVolume, which is required for production clusters, and for configuring an empty directory as the storage location, which is available for only non-production clusters, are shown.

Configure registry storage for VMware vSphere

As a cluster administrator, following installation configure the registry to use storage.

Prerequisites

• Cluster administrator permissions.
• A cluster on VMware vSphere.
• A provisioned persistent volume (PV) with ReadWriteMany access mode, such as NFS.
• Must have "100Gi" capacity.

Procedure

1. To configure the registry to use storage, change the spec.storage.pvc in the configs.imageregistry/cluster resource.

2. Verify we do not have a registry pod:

   $ oc get pod -n openshift-image-registry
   

If the storage type is emptyDIR, the replica number can not be greater than 1. If the storage type is NFS, and we want to scale up the registry Pod by setting replica1 enable the no_wdelay mount option. For example:

# cat /etc/exports
/mnt/data *(rw,sync,no_wdelay,no_root_squash,insecure,fsid=0)
sh-4.3# exportfs -rv
exporting *:/mnt/data

1. Check the registry configuration:

   $ oc edit configs.imageregistry.operator.openshift.io
   
   storage:
     pvc:
       claim:
   

   Leave the claim field blank to allow the automatic creation of an image-registry-storage PVC.

2. Check the clusteroperator status:

   $ oc get clusteroperator image-registry
   

Configure storage for the image registry in non-production clusters

Configure storage for the image registry Operator. For non-production clusters, we can set the image registry to an empty directory. If we do so, all images are lost if you restart the registry.

Procedure

• To set the image registry storage to an empty directory:

  $ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
  

  Warning

  Configure this option for only non-production clusters.

  If you run this command before the Image Registry Operator initializes its components, the oc patch command fails with the following error:

  Error from server (NotFound): configs.imageregistry.operator.openshift.io "cluster" not found
  

  Wait a few minutes and run the command again.

Complete installation on user-provisioned infrastructure

After you complete the Operator configuration, we can finish installing the cluster on infrastructure that we provide.

Prerequisites

• Your control plane has initialized.
• You have completed the initial Operator configuration.

Procedure

1. Confirm that all the cluster components are online:

   $ watch -n5 oc get clusteroperators
   
   NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                       4.1.0     True        False         False      10m
   cloud-credential                     4.1.0     True        False         False      22m
   cluster-autoscaler                   4.1.0     True        False         False      21m
   console                              4.1.0     True        False         False      10m
   dns                                  4.1.0     True        False         False      21m
   image-registry                       4.1.0     True        False         False      16m
   ingress                              4.1.0     True        False         False      16m
   kube-apiserver                       4.1.0     True        False         False      19m
   kube-controller-manager              4.1.0     True        False         False      18m
   kube-scheduler                       4.1.0     True        False         False      22m
   machine-api                          4.1.0     True        False         False      22m
   machine-config                       4.1.0     True        False         False      18m
   marketplace                          4.1.0     True        False         False      18m
   monitoring                           4.1.0     True        False         False      18m
   network                              4.1.0     True        False         False      16m
   node-tuning                          4.1.0     True        False         False      21m
   openshift-apiserver                  4.1.0     True        False         False      21m
   openshift-controller-manager         4.1.0     True        False         False      17m
   openshift-samples                    4.1.0     True        False         False      14m
   operator-lifecycle-manager           4.1.0     True        False         False      21m
   operator-lifecycle-manager-catalog   4.1.0     True        False         False      21m
   service-ca                           4.1.0     True        False         False      21m
   service-catalog-apiserver            4.1.0     True        False         False      16m
   service-catalog-controller-manager   4.1.0     True        False         False      16m
   storage                              4.1.0     True        False         False      16m
   

   When all of the cluster Operators are AVAILABLE, we can complete the installation.

2. Monitor for cluster completion:

   ./openshift-install --dir=/path/to/install/files wait-for install-complete 
   
    INFO Waiting up to 30m0s for the cluster to initialize...
   

   The command succeeds when the Cluster Version Operator finishes deploying the OpenShift cluster from Kubernetes API server.

3. Confirm that the Kubernetes API server is communicating with the Pods.

   1. To view a list of all Pods, use the following command:

      $ oc get pods --all-namespaces
      
      NAMESPACE                         NAME                                            READY   STATUS      RESTARTS   AGE
      openshift-apiserver-operator      openshift-apiserver-operator-85cb746d55-zqhs8   1/1     Running     1          9m
      openshift-apiserver               apiserver-67b9g                                 1/1     Running     0          3m
      openshift-apiserver               apiserver-ljcmx                                 1/1     Running     0          1m
      openshift-apiserver               apiserver-z25h4                                 1/1     Running     0          2m
      openshift-authentication-operator authentication-operator-69d5d8bf84-vh2n8        1/1     Running     0          5m
      ...
      

   2. View the logs for a Pod that is listed in the output of the previous command using the following command:

      $ oc logs pod_name -n namespace 1
      

      1 Specify the Pod name and namespace, as shown in the output of the previous command.

      If the Pod logs display, the Kubernetes API server can communicate with the cluster machines.

Next steps

• Customize the cluster.
• If necessary, we can opt out of telemetry.

Gather installation logs

To assist in troubleshooting a failed OpenShift installation, we can gather logs from the bootstrap and control plane, or master, machines.

• You attempted to install a OpenShift cluster, and installation failed.
• You provided an SSH key to the installation program, and that key is in your running ssh-agent process.

Gather logs from a failed installation

If you gave an SSH key to your installation program, we can gather data about your failed installation.

We use a different command to gather logs about an unsuccessful installation than to gather logs from a running cluster. If gather logs from a running cluster, use the oc adm must-gather command.

Prerequisites

• Your OpenShift installation failed before the bootstrap process finished. The bootstrap node must be running and accessible through SSH.
• The ssh-agent process is active on our computer, and we provided both the ssh-agent process and the installation program the same SSH key.
• If you tried to install a cluster on infrastructure that we provisioned, have the fully-qualified domain names of the control plane, or master, machines.

Procedure

1. Generate the commands required to obtain the installation logs from the bootstrap and control plane machines:

   • If you used installer-provisioned infrastructure, run:
     $ ./openshift-install gather bootstrap --dir=/path_to/install/files ¹

     For installer-provisioned infrastructure, the installation program stores information about the cluster, so we do not specify the host names or IP addresses

   • If you used infrastructure that we provisioned ourselves, run:

     ./openshift-install gather bootstrap 
                         --dir=directory \ 1
                         --bootstrap bootstrap_address \ 2
                         --master "master_address master_address master_address" 3
     

     2 bootstrap_address is the fully-qualified domain name or IP address of the cluster's bootstrap machine.

     3 master_address is the fully-qualified domain name or IP address of a control plane, or master, machine in the cluster. Specify a space-delimited list that contains all the control plane machines in the cluster.

   The command output resembles the following example:

   INFO Use the following commands to gather logs from the cluster
   INFO ssh -A core@bootstrap_address '/usr/local/bin/installer-gather.sh master_address master_address master_address'
   INFO scp core@bootstrap_address:~/log-bundle.tar.gz .
   

   We use both commands that are displayed to gather and download the logs.

2. Gather logs from the bootstrap and master machines:

   $ ssh -A core@bootstrap_address '/usr/local/bin/installer-gather.sh master_address master_address master_address'
   

   You SSH into the bootstrap machine and run the gather tool, which is designed to collect as much data as possible from the bootstrap and control plane machines in the cluster and compress all of the gathered files.

   It is normal to see errors in the command output. If the command output displays the instructions to download the compressed log files, log-bundle.tar.gz, then the command succeeded.

3. Download the compressed file containing the logs:

   $ scp core@bootstrap_address:~/log-bundle.tar.gz . 1
   

   1 bootstrap_address is the fully-qualified domain name or IP address of the bootstrap machine.

   The command to download the log files is included at the end of the gather command output.

   If you open a Red Hat support case about your installation failure, include the compressed logs in the case.

Installation configuration

Available cluster customizations

You complete most of the cluster configuration and customization after you deploy the OpenShift cluster. A number of configuration resources are available.

You modify the configuration resources to configure the major features of the cluster, such as the image registry, networking configuration, image build behavior, and the identity provider.

For current documentation of settings these resources expose, use the oc explain command, for example oc explain builds --api-version=config.openshift.io/v1

Cluster configuration resources

All cluster configuration resources are globally scoped (not namespaced) and named cluster.

Resource name	Description
apiserver.config.openshift.io	Provides api-server configuration such as certificates and certificate authorities.
authentication.config.openshift.io	Controls the identity providerand authentication configuration for the cluster.
build.config.openshift.io	Controls default and enforced configuration for all builds on the cluster.
console.config.openshift.io	Configures the behavior of the web console interface, including the logout behavior.
featuregate.config.openshift.io	Enables FeatureGates so that we can use Tech Preview features.
image.config.openshift.io	Configures how specific image registries should be treated (allowed, disallowed, insecure, CA details).
ingress.config.openshift.io	Configuration details related to routing such as the default domain for routes.
oauth.config.openshift.io	Configures identity providers and other behavior related to internal OAuth server flows.
project.config.openshift.io	Configures how projects are created including the project template.
proxy.config.openshift.io	Defines proxies to be used by components needing external network access. Not all components currently consume this value.
scheduler.config.openshift.io	Configures scheduler behavior such as policies and default nodeselectors.

Operator configuration resources

These configuration resources are cluster-scoped instances, named cluster, which control the behavior of a specific component as owned by a particular operator.

Resource name	Description
console.operator.openshift.io	Controls console appearance such as branding customizations
config.imageregistry.operator.openshift.io	Configures internal image registry settings such as public routing, log levels, proxy settings, resource constraints, replica counts, and storage type.
config.samples.operator.openshift.io	Configures the Samples Operator to control which example imagestreams and templates are installed on the cluster.

Additional configuration resources

These configuration resources represent a single instance of a particular component, in some cases multiple instances can be requested by creating multiple instances of the resource. In other cases only a specific resource instance name in a specific namespace will be consumed by the operator. Reference the component-specific documentation for details on how and when additional resource instances can be created.

Resource name	Instance name	Namespace	Description
alertmanager.monitoring.coreos.com	main	openshift-monitoring	Controls the alertmanager deployment parameters.
ingresscontroller.operator.openshift.io	default	openshift-ingress-operator	Configures Ingress Operator behavior such as domain, number of replicas, certificates, and controller placement.

Informational Resources

We use these resources to retrieve information about the cluster. You should not edit these resources directly.

Resource name	Instance name	Description
clusterversion.config.openshift.io	version	In OpenShift 4.1, we must not customize the ClusterVersion resource for production clusters. Instead, follow the process to update a cluster.
dns.config.openshift.io	cluster	We cannot modify the DNS settings for the cluster. We can view the DNS Operator status.
infrastructure.config.openshift.io	cluster	Configuration details allowing the cluster to interact with its cloud provider.
network.config.openshift.io	cluster	We cannot modify the cluster networking after installation. To customize your network, follow the process to customize networking during installation.

Configure the firewall to access Red Hat Insights

Set the firewall to allow the following host names and ports on the outgoing network firewall:

cert-api.access.redhat.com:443
api.access.redhat.com:443
infogw.api.openshift.com:443

Quick Links

• Downloads
• Subscriptions
• Support Cases
• Customer Service
• Product Documentation

Help

• Contact Us
• Customer Portal FAQ
• Log-in Assistance

Site Info

• Trust Red Hat
• Browser Support Policy
• Accessibility
• Awards and Recognition
• Colophon

Related Sites

• redhat.com
• openshift.com
• developers.redhat.com
• connect.redhat.com

About

• Red Hat Subscription Value
• About Red Hat
• Red Hat Jobs