Configuring VPC subnets

Change the pool of available portable public or private IP addresses by adding subnets to your Red Hat OpenShift on IBM Cloud VPC cluster.

Overview of VPC networking in Red Hat OpenShift on IBM Cloud

Understand the basic concepts of VPC networking in Red Hat OpenShift on IBM Cloud clusters.

Subnets

Before you create a VPC cluster for the first time, we must create a VPC subnet in each zone where we want to deploy worker nodes. A VPC subnet is a specified private IP address range (CIDR block) and configures a group of worker nodes and pods as if they are attached to the same physical wire.

When you create a cluster, you can specify only one existing VPC subnet for each zone. Each worker node that you add in a cluster is deployed with a private IP address from the VPC subnet in that zone. After the worker node is provisioned, the worker node IP address persists after a reboot operation, but the worker node IP address changes after replace and update operations.

Do not delete the subnets that you attach to your cluster during cluster creation or when you add worker nodes in a zone. If you delete a VPC subnet that your cluster used, any load balancers that use IP addresses from the subnet might experience issues, and we might be unable to create new load balancers.

How many IP addresses do I need for my VPC subnet?
When you create your VPC subnet, make sure to create a subnet with enough IP addresses for your cluster, such as 256. You cannot change the number of IP addresses that a VPC subnet has later.

Keep in mind the following IP address reservations.

5 IP addresses are reserved by VPC from each subnet by default.
1 IP address is required per worker node in your cluster.
1 IP address is required each time that you update or replace a worker node. These IP addresses are eventually reclaimed and available for reuse.
2 IP addresses are used each time that you create a public or private load balancer. If we have a multizone cluster, these 2 IP addresses are spread across zones, so the subnet might not have an IP address reserved.
Other networking resources that you set up for the cluster, such as a VPNaaS or LBaaS autoscaling, might require additional IP addresses or have other service limitations. For example, LBaaS autoscaling might scale up to 16 IP addresses per load balancer.

What IP ranges can I use for my VPC subnets?
The default IP address range for VPC subnets is 10.0.0.0 – 10.255.255.255. For a list of IP address ranges per VPC zone, see the VPC default address prefixes. If you enable your VPC with classic access, or access to classic infrastructure resources, the default IP ranges per VPC zone are different. For more information, see Classic access VPC default address prefixes.

For to create your cluster by using custom-range subnets, see the guidance for custom address prefixes. However, if you use custom-range subnets for your worker nodes, we must ensure that the IP range for the worker node subnets do not overlap with your cluster's pod subnet.

If you specified your own pod subnet in the --pod-subnet flag during cluster creation, your pods are assigned IP addresses from this range.
If you did not specify a custom pod subnet during cluster creation, your cluster uses the default pod subnet. In the first cluster that you create in a Gen 2 VPC, the default pod subnet is 172.17.0.0/18. In the second cluster that you create in that VPC, the default pod subnet is 172.17.64.0/18. In each subsequent cluster, the pod subnet range is the next available, non-overlapping /18 subnet.

Can I specify subnets for pods and services in my cluster?

If we plan to connect your cluster to on-premises networks through IBM Cloud Direct Link or a VPN service, you can avoid subnet conflicts by specifying a custom subnet CIDR that provides the private IP addresses for your pods, and a custom subnet CIDR to provide the private IP addresses for services.

To specify custom pod and service subnets during cluster creation, use the --pod-subnet and --service-subnet flags in the ibmcloud oc cluster create CLI command.

To see the pod and service subnets that your cluster uses, look for the Pod Subnet and Service Subnet fields in the output of ibmcloud oc cluster get.

Pods:

Default range: In the first cluster that you create in a Gen 2 VPC, the default pod subnet is 172.17.0.0/18. In the second cluster that you create in that VPC, the default pod subnet is 172.17.64.0/18. In each subsequent cluster, the pod subnet range is the next available, non-overlapping /18 subnet.
Size requirements: When you specify a custom subnet, consider the size of the cluster that we plan to create and the number of worker nodes that we might add in the future. The subnet must have a CIDR of at least /23, which provides enough pod IPs for a maximum of four worker nodes in a cluster. For larger clusters, use /22 to have enough pod IP addresses for eight worker nodes, /21 to have enough pod IP addresses for 16 worker nodes, and so on.
Range requirements: The pod and service subnets cannot overlap each other, and the pod subnet cannot overlap the VPC subnets for your worker nodes. The subnet that you choose must be within one of the following ranges:
- 172.17.0.0 - 172.17.255.255
- 172.21.0.0 - 172.31.255.255
- 192.168.0.0 - 192.168.254.255
- 198.18.0.0 - 198.19.255.255

Services:

Default range: All services that are deployed to the cluster are assigned a private IP address in the 172.21.0.0/16 range by default.
Size requirements: When you specify a custom subnet, the subnet must be specified in CIDR format with a size of at least /24, which allows a maximum of 255 services in the cluster, or larger.
Range requirements: The pod and service subnets cannot overlap each other. The subnet that you choose must be within one of the following ranges:
- 172.17.0.0 - 172.17.255.255
- 172.21.0.0 - 172.31.255.255
- 192.168.0.0 - 192.168.254.255
- 198.18.0.0 - 198.19.255.255

Public gateways

A public gateway enables a subnet and all worker nodes that are attached to the subnet to establish outbound connections to the internet. Enable a public gateway on the VPC subnets that worker nodes are deployed to so that they can run default OpenShift components.

For example, default components such as the web console and OperatorHub use a secure, public connection to complete actions such as pulling images from remote, private registries. Also, if an IBM Cloud service does not support private service endpoints, your worker nodes must be connected to a subnet that has a public gateway attached to it. The pods on those worker nodes can securely communicate with the services over the public network through the subnet's public gateway. Note that a public gateway is not required on your subnets to allow inbound network traffic from the internet to LoadBalancer services or ALBs.

Within one VPC, you can create only one public gateway per zone, but that public gateway can be attached to multiple subnets within the zone. For more information about public gateways, see the Networking for VPC documentation.

Network segmentation

Network segmentation describes the approach to divide a network into multiple sub-networks. Apps that run in one sub-network cannot see or access apps in another sub-network. For more information about network segmentation options for VPC subnets, see this cluster security topic.

Subnets provide a channel for connectivity among the worker nodes within the cluster. Additionally, any system that is connected to any of the private subnets in the same VPC can communicate with workers. For example, all subnets in one VPC can communicate through private layer 3 routing with a built-in VPC router.

If we have multiple clusters that must communicate with each other, you can create the clusters in the same VPC. However, if your clusters do not need to communicate, you can achieve better network segmentation by creating the clusters in separate VPCs. You can also create access control lists (ACLs) for your VPC subnets to mediate traffic on the private network. ACLs consist of inbound and outbound rules that define which ingress and egress is permitted for each VPC subnet.

VPC networking limitations

When you create VPC subnets for your clusters, keep in mind the following features and limitations.

The default CIDR size of each VPC subnet is /24, which can support up to 253 worker nodes. If we plan to deploy more than 250 worker nodes per zone in one cluster, consider creating a subnet of a larger size.
After you create a VPC subnet, you cannot resize it or change its IP range.
Multiple clusters in the same VPC can share subnets.
VPC subnets are bound to a single zone and cannot span multiple zones or regions.
After you create a subnet, you cannot move it to a different zone, region, or VPC.
If we have worker nodes that are attached to an existing subnet in a zone, you cannot change the subnet for that zone in the cluster.
The 172.16.0.0/16, 172.18.0.0/16, 172.19.0.0/16, and 172.20.0.0/16 ranges are prohibited.
Within one VPC, you can create only one public gateway per zone, but that public gateway can be attached to multiple subnets within the zone.

Creating a VPC subnet and attaching a public gateway

Create a VPC subnet for your cluster and optionally attach a public gateway to the subnet.

Creating a VPC subnet in the console

Use the IBM Cloud console to create a VPC subnet for your cluster and optionally attach a public gateway to the subnet.

From the VPC subnet dashboard, click New subnet.
Enter a name for your subnet and select the name of the VPC that you created.
Select the location and zone where we want to create the subnet.
Specify the number of IP addresses to create.
- VPC subnets provide IP addresses for your worker nodes and load balancer services in the cluster, so create a VPC subnet with enough IP addresses, such as 256. You cannot change the number of IPs that a VPC subnet has later.
- If you enter a specific IP range, do not use the following reserved ranges: 172.16.0.0/16, 172.18.0.0/16, 172.19.0.0/16, and 172.20.0.0/16.
To run default OpenShift components such as the web console or OperatorHub, and to allow your cluster to access public endpoints such as a public URL of another app or an IBM Cloud service that supports public service endpoints only, you must attach a public gateway to your subnet.
Click Create subnet.
Use the subnet to create a cluster, create a new worker pool, or add the subnet to an existing worker pool.
Do not delete the subnets that you attach to your cluster during cluster creation or when you add worker nodes in a zone. If you delete a VPC subnet that your cluster used, any load balancers that use IP addresses from the subnet might experience issues, and we might be unable to create new load balancers.

Creating a VPC subnet in the CLI

Use the IBM Cloud CLI to create a VPC subnet for your cluster and optionally attach a public gateway to the subnet.

Before beginning:

In your terminal, log in to your IBM Cloud account and target the IBM Cloud region and resource group where we want to create your VPC cluster. For supported regions, see Creating a VPC in a different region. The cluster's resource group can differ from the VPC resource group. Enter your IBM Cloud credentials when prompted. If we have a federated ID, use the --sso flag to log in.
```
ibmcloud login -r <region> [--sso]
```
Target the VPC compute generation for your cluster.
```
ibmcloud is target --gen 2
```
Create a VPC in the same region where we want to create the cluster.

To create a VPC subnet:

Get the ID of the VPC where we want to create the subnet.
```
ibmcloud oc vpcs
```
Create the subnet. For more information about the options in this command, see the CLI reference.
```
ibmcloud is subnet-create <subnet_name> <vpc_id> --zone <vpc_zone> --ipv4-address-count <number_of_ip_address>
```
- VPC subnets provide IP addresses for your worker nodes and load balancer services in the cluster, so create a VPC subnet with enough IP addresses, such as 256. You cannot change the number of IPs that a VPC subnet has later.
- Do not use the following reserved ranges: 172.16.0.0/16, 172.18.0.0/16, 172.19.0.0/16, and 172.20.0.0/16.

Check whether we have a public gateway in the zones where we want to create a cluster. Within one VPC, you can create only one public gateway per zone, but that public gateway can be attached to multiple subnets within the zone.

ibmcloud is public-gateways

Example output:

ID                                     Name                                       VPC                          Zone         Floating IP                  Created                     Status      Resource group
26426426-6065-4716-a90b-ac7ed7917c63   test-pgw                                   testvpc(36c8f522-.)          us-south-1   169.xx.xxx.xxx(26466378-.)   2019-09-20T16:27:32-05:00   available   -
2ba2ba2b-fffa-4b0c-bdca-7970f09f9b8a   pgw-73b62bc0-b53a-11e9-9838-f3f4efa02374   team3(ff537d43-.)            us-south-2   169.xx.xxx.xxx(2ba9a280-.)   2019-08-02T10:30:29-05:00   available   -

If you already have a public gateway in each zone, note the IDs of the public gateways.

If you do not have a public gateway in each zone, create a public gateway. Consider naming the public gateway in the format <cluster>-<zone>-gateway. In the output, note the public gateway's ID.

ibmcloud is public-gateway-create <gateway_name> <VPC_ID> <zone>

Example output:

ID               26466378-6065-4716-a90b-ac7ed7917c63
Name             mycluster-us-south-1-gateway
Floating IP      169.xx.xx.xxx(26466378-6065-4716-a90b-ac7ed7917c63)
Status           pending
Created          2019-09-20T16:27:32-05:00
Zone             us-south-1
VPC              myvpc(36c8f522-4f0d-400c-8226-299f0b8198cf)
Resource group   -

Using the IDs of the public gateway and the subnet, attach the public gateway to the subnet.

ibmcloud is subnet-update <subnet_ID> --public-gateway-id <gateway_ID>

Example output:

ID                  91e946b4-7094-46d0-9223-5c2dea2e5023
Name                mysubnet1
IPv4 CIDR           10.240.xx.xx/24
Address available   250
Address total       256
ACL                 allow-all-network-acl-36c8f522-4f0d-400c-8226-299f0b8198cf(585bc142-5392-45d4-afdd-d9b59ef2d906)
Gateway             mycluster-us-south-1-gateway(26466378-6065-4716-a90b-ac7ed7917c63)
Created             2019-08-21T09:43:11-05:00
Status              available
Zone                us-south-1
VPC                 myvpc(36c8f522-4f0d-400c-8226-299f0b8198cf)

Use the subnet to create a cluster, create a new worker pool, or add the subnet to an existing worker pool.
Do not delete the subnets that you attach to your cluster during cluster creation or when you add worker nodes in a zone. If you delete a VPC subnet that your cluster used, any load balancers that use IP addresses from the subnet might experience issues, and we might be unable to create new load balancers.