Storing data on Block Storage for VPC (Gen 1 and Gen 2 compute)

IBM® Cloud Block Storage for Virtual Private Cloud (Gen 1 and Gen 2 compute) provides hypervisor-mounted, high-performance data storage for your virtual server instances that you can provision within a VPC.

You can choose between predefined storage tiers with GB sizes and IOPS that meet the requirements of the workloads. To find out if Block Storage for VPC is the right storage option for you, see Choosing a storage solution. For pricing information, see Pricing for Block Storage for VPC.

The Block Storage for VPC add-on is installed and enabled by default on VPC clusters. You can enable or disable the add-on by using the addon enable or addon disable command in the CLI.

Quickstart for IBM Cloud Block Storage for VPC

In this quickstart guide, you create a 10Gi 5IOPS tier Block Storage for VPC volume in your cluster by creating a PVC to dynamically provision the volume. Then, you create an app deployment that mounts your PVC.

Create a file for your PVC and name it pvc.yaml.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: my-pvc
spec:
 accessModes:
 - ReadWriteOnce
 resources:
   requests:
     storage: 10Gi
 storageClassName: ibmc-vpc-block-5iops-tier

Create the PVC in your cluster.
```
oc apply -f pvc.yaml
```

After your PVC is bound, create an app deployment that uses your PVC. Create a file for your deployment and name it deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-deployment
 labels:
   app:
spec:
 selector:
   matchLabels:
     app: my-app
 template:
   metadata:
     labels:
       app: my-app
   spec:
     containers:
     - image: # Your contanerized app image.
       name: my-container
       volumeMounts:
       - name: my-volume
         mountPath: /mount-path
     volumes:
     - name: my-volume
       persistentVolumeClaim:
         claimName: my-pvc

Create the deployment in your cluster.
```
oc apply -f deployment.yaml
```

For more information, see:

Adding Block Storage for VPC to your apps

Choose your Block Storage for VPC profile and create a persistent volume claim to dynamically provision Block Storage for VPC for your cluster. Dynamic provisioning automatically creates the matching persistent volume and orders the physical storage device in your IBM Cloud account.

Decide on the Block Storage for VPC profile that best meets the capacity and performance requirements that we want.

Select the corresponding storage class for your Block Storage for VPC profile.

All IBM pre-defined storage classes set up Block Storage for VPC with an ext4 file system by default. If we want to use a different file system, such as xfs or ext3, create a customized storage class.

Block Storage for VPC profile	Corresponding storage class
10 IOPS/GB	ibmc-vpc-block-10iops-tier ibmc-vpc-block-retain-10iops-tier
5 IOPS/GB	ibmc-vpc-block-5iops-tier ibmc-vpc-block-retain-5iops-tier
3 IOPS/GB	ibmc-vpc-block-general-purpose ibmc-vpc-block-retain-general-purpose
Custom	ibmc-vpc-block-custom ibmc-vpc-block-retain-custom

Decide on your Block Storage for VPC configuration.
1. Choose a size for your storage. Make sure that the size is supported by the Block Storage for VPC profile that you chose.
2. Choose if we want to keep your data after the cluster or the persistent volume claim (PVC) is deleted.
  - If we want to keep your data, then choose a retain storage class. When you delete the PVC, only the PVC is deleted. The persistent volume (PV), the physical storage device in your IBM Cloud account, and your data still exist. To reclaim the storage and use it in your cluster again, we must remove the PV and follow the steps for using existing Block Storage for VPC.
  - If we want the PV, the data, and your physical Block Storage for VPC device to be deleted when you delete the PVC, choose a storage class without retain.

Create a configuration file to define your persistent volume claim and save the configuration as a YAML file.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <pvc_name>
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: <size>
  storageClassName: <storage_class>

Component	Description
name	Enter a name for your PVC.
accessMode	Block Storage for VPC supports a ReadWriteOnce access mode only. You can mount the PVC to one pod on one worker node in the cluster at a time.
storage	Enter the size of the Block Storage for VPC instance, in gigabytes (Gi). Make sure that the size is supported in the Block Storage for VPC profile that you chose. For example, if we want 10 gigabyte of Block Storage, enter 10Gi.
storageClassName	Enter the storage class name that you selected earlier.

Create the PVC in your cluster.
```
oc apply -f pvc.yaml
```

Verify that your PVC is created and bound to the PV. This process can take a few minutes.

oc describe pvc <pvc_name>

Example output:

Name:          mypvv
Namespace:     default
StorageClass:  ibmc-vpc-block-5iops-tier
Status:        Bound
Volume:        
Labels:        <none>
Annotations:   oc.kubernetes.io/last-applied-configuration:
     {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"csi-block-pvc-good","namespace":"default"},"spec":{...
   volume.beta.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity: 10Gi   
Access Modes:  
VolumeMode:    Filesystem
Events:
  Type       Reason                Age               From                         Message
  ----       ------                ----              ----                         -------
  Normal     ExternalProvisioning  9s (x3 over 18s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "vpc.block.csi.ibm.io" or manually created by system administrator
Mounted By:  <none>

Create a deployment configuration file for your app and mount the PVC to your app.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <deployment_name>
  labels:
    app: <deployment_label>
spec:
  selector:
    matchLabels:
      app: <app_name>
  template:
    metadata:
      labels:
        app: <app_name>
    spec:
      containers:
      - image: <image_name>
        name: <container_name>
        volumeMounts:
        - name: <volume_name>
 mountPath: /<file_path>
      volumes:
      - name: <volume_name>
        persistentVolumeClaim:
 claimName: <pvc_name>

Component	Description
labels.app	In the metadata section, enter a label for the deployment.
matchLabels.app labels.app	In the spec selector and template metadata sections, enter a label for your app.
image	Specify the name of the container image that we want to use. To list available images in your IBM Cloud Container Registry account, run ibmcloud cr image-list.
name	Specify the name of the container that we want to deploy in your pod.
mountPath	In the container volume mounts section, specify the absolute path of the directory to where the PVC is mounted inside the container.
name	In the container volume mounts section, enter the name of the volume to mount to your pod. You can enter any name that we want.
name	In the volumes section, enter the name of the volume to mount to your pod. Typically this name is the same as volumeMounts.name.
claimName	In the volumes persistent volume claim section, enter the name of the PVC that you created earlier.

Create the deployment in your cluster.
```
oc apply -f deployment.yaml
```
Verify that the PVC is successfully mounted to your app. It might take a few minutes for your pods to get into a Running state.

During the deployment of your app, we might see intermittent Unable to mount volumes errors in the Events section of your CLI output. The Block Storage for VPC add-on automatically retries mounting the storage to your apps. Wait a few more minutes for the storage to mount to your app.
```
oc describe deployment <deployment_name>
```
Example output:
```
...
Volumes:
myvol:
  Type:    PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
  ClaimName:    mypvc
  ReadOnly:    false
```

Using an existing Block Storage for VPC instance

If we have an existing physical Block Storage for VPC device that we want to use in your cluster, you can manually create the PV and PVC to statically provision the storage.

You can attach a volume to one worker node only. Make sure that the volume is in the same zone as the worker node for the attachment to succeed.

Determine the volume that we want to attach to a worker node in your VPC cluster. Note the volume ID.
```
ibmcloud is volumes
```
List the details of your volume. Note the Size, Zone, and IOPS. These values are used to create your PV.
```
ibmcloud is volume <volume_id>
```
Retrieve a list of worker nodes in your VPC cluster. Note the Zone of the worker node that is in the same zone as your storage volume.
```
ibmcloud oc worker ls <cluster_name>
```
Optional: If you provisioned your physical Block Storage for VPC instance by using a retain storage class, the PV and the physical storage is not removed when you remove the PVC. To use your physical Block Storage for VPC device in your cluster, we must remove the existing PV first.
1. List the PVs in your cluster and look for the PV that belongs to your Block Storage for VPC device. The PV is in a released state.
```
oc get pv
```
2. Remove the PV.
```
oc delete pv <pv_name>
```

Create a configuration file for your PV. Include the ID, Size, Zone, and IOPS that you retrieved earlier.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: <pv_name> # Example: my-persistent-volume
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: <vpc_block_storage_size> # Example: 20Gi
  csi:
    driver: vpc.block.csi.ibm.io
    fsType: ext4
    volumeAttributes:
      iops: "<vpc_block_storage_iops>" # Example: "3000"
      volumeId: <vpc_block_storage_ID> # Example: a1a11a1a-a111-1111-1a11-1111a11a1a11
      zone: "<vpc_block_zone>" # Example: "eu-de-3"
    volumeHandle: <vpc_block_storage_ID>
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: failure-domain.beta.kubernetes.io/zone
 operator: In
 values:
 - <worker_node_zone> # Example: eu-de-3
        - key: failure-domain.beta.kubernetes.io/region
 operator: In
 values:
 - <worker_node_region> # Example: eu-de
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  volumeMode: Filesystem

Scroll for moreScroll for more

Component	Description
name	In the metadata section, enter a name for your PV.
storage	In the spec capacity section, enter the size of your Block Storage volume in gigabytes (Gi) that you retrieved earlier. For example, if the size of your device is 100 GB, enter 100Gi.
iops	In the spec CSI volume attributes section, enter the Max IOPS of the Block Storage volume that you retrieved earlier.
zone	In the spec CSI volume attributes section, enter the VPC block zone that matches the location that you retrieved earlier. For example, if your location is Washington DC-1, then use us-east-1 as your zone. To list available zones, run ibmcloud is zones. To find an overview of available VPC zones and locations, see Creating a VPC in a different region.
volumeId spec.csi.volumeHandle	In the spec CSI volume attributes section, enter the ID of the Block Storage volume that you retrieved earlier.
storageClassName	For the spec storage class name, enter an empty string.
matchExpressions	In the spec node affinity section, enter the node selector terms to match the zone. For the key, enter failure-domain.beta.kubernetes.io/zone. For the value, enter the zone of your worker node where we want to attach storage.
matchExpressions	In the spec node affinity section, enter the node selector terms to match the region. For the key, enter failure-domain.beta.kubernetes.io/region. For the value, enter the region of the worker node where we want to attach storage.

Create the PV in your cluster.
```
oc apply -f pv.yaml
```
Verify that the PV is created in your cluster.
```
oc get pv
```
Create another configuration file for your PVC. In order for the PVC to match the PV that you created earlier, we must choose the same value for the storage size and access mode. In your storage class field, enter an empty string value to match your PV. If any of these fields do not match the PV, then a new PV and a Block Storage instance are created automatically via dynamic provisioning.
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <pvc_name>
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: <vpc_block_storage_size>
  storageClassName: ""
```
Create your PVC.
```
oc apply -f pvc.yaml
```
Verify that your PVC is created and bound to the PV that you created earlier. This process can take a few minutes.
```
oc describe pvc <pvc_name>
```
Create a deployment or a pod that uses your PVC.

Updating the Block Storage for VPC add-on

You can update the Block Storage for VPC add-on by disabling and re-enabling the add-on in your cluster.

Check to see if an update is available. If an update is available, the plug-in version is flagged with an asterisk and the latest version is shown. Note the latest version as this value is used later.
```
ibmcloud oc cluster addons --cluster <cluster_name_or_ID>
```
Example output:
```
Name                   Version                 Health State   Health Status   
vpc-block-csi-driver   1.0.0* (2.0.0 latest)   normal         Addon Ready
```

Disable the Block Storage for VPC add-on.

ibmcloud oc cluster addon disable vpc-block-csi-driver --cluster <cluster_name_or_ID> -f

Example Output

Disabling add-on vpc-block-csi-driver for cluster <cluster_name_or_ID>...
OK

Verify that the add-on is disabled. If the add-on is disabled it does not appear in the list of add-ons in your cluster. The add-on might still display in your list of add-ons for a few minutes after running the disable command.
```
ibmcloud oc cluster addon ls --cluster <cluster_name_or_ID>
```

Re-enable the add-on and specify the latest version that you retrieved earlier.

ibmcloud oc cluster addon enable vpc-block-csi-driver --version <version> --cluster <cluster_name_or_ID>

Example Output

Enabling add-on vpc-block-csi-driver(2.0.0) for cluster <cluster_name_or_ID>...
OK

Verify that the add-on is in the Addon Ready state. The add-on might take a few minutes to become ready.

ibmcloud oc cluster addon ls --cluster <cluster_name_or_ID>

Example Output

Name                   Version   Health State   Health Status   
vpc-block-csi-driver   2.0.0     normal         Addon Ready

Creating Block Storage for VPC with a different file system

You can create a customized storage class to provision Block Storage for VPC with a different file system, such as xfs or ext3. By default, all Block Storage for VPC instances are provisioned with an ext4 file system.

Follow the steps to create a customized storage class with the file system that we want to use.

Example storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <storage_class_name>
parameters:
  billingType: hourly
  classVersion: "1"
  csi.storage.k8s.io/fstype: <file_system_type>
  encrypted: "false"
  encryptionKey: ""
  generation: gc
  profile: general-purpose
  resourceGroup: ""
  sizeRange: '[10-2000]GiB'
  tags: ""
  zone: ""
provisioner: vpc.block.csi.ibm.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

Component	Description
name	Enter a name for your storage class.
cs.storage.k8s.io/fstype	In the parameters, enter the file system for your Block Storage for VPC instance. Choose xfs or ext3.

Follow step 4-9 in Adding Block Storage for VPC to your apps to create a PVC with your customized storage class to provision Block Storage with a different file system. Then, mount this storage to a sample app.

Your app might take a few minutes to mount the storage and get into a Running state.
Verify that your storage is mounted with the correct file system.
1. List the pods in your cluster and note the Name of the pod that you used to mount your storage.
```
oc get pod
```
2. Log in to your pod.
```
oc exec <pod_name> -it bash
```
3. List the mount paths inside your pod.
```
mount | grep /dev/xvdg
```
  Example output for xfs:
```
/dev/xvdg on /test type xfs (rw,relatime,attr2,inode64,noquota)
```
4. Exit your pod.
```
exit
```

Setting up encryption for Block Storage for VPC

Use IBM® Key Protect to create a private root key that you use in your Block Storage for VPC instance to encrypt data as it is written to the storage. After you create the private root key, create a custom storage class or a Kubernetes secret with your root key and then use this storage class or secret to provision your Block Storage for VPC instance.

Create a Key Protect service instance.
Create a root key. By default, the root key is created without an expiration date.
Retrieve the service CRN for your root key.
1. From the Key Protect details page, select the Manage tab to find the list of your keys.
2. Find the root key that you created and from the actions menu, click View CRN.
3. Note the CRN of your root key.
Authorize Block Storage for VPC to access IBM® Key Protect.
1. From the IBM Cloud menu, select Manage > Access (IAM).
2. From the menu, select Authorizations.
3. Click Create.
4. Select Cloud Block Storage as your source service.
5. Select Key Protect as your target service.
6. Select the Reader service access role and click Authorize.

Decide if we want to store the Key Protect root key CRN in a customized storage class or in a Kubernetes secret. Then, follow the steps to create a customized storage class or a Kubernetes secret.

Example customized storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <storage_class_name>
provisioner: vpc.block.csi.ibm.io
parameters:
  profile: "5iops-tier"
  sizeRange: "[10-2000]GiB"
  csi.storage.k8s.io/fstype: "ext4"
  billingType: "hourly"
  encrypted: "true"
  encryptionKey: "<encryption_key>"
  resourceGroup: ""
  zone: ""
  tags: ""
  generation: "gc"
  classVersion: "1"
reclaimPolicy: "Delete"

Component	Description
name	Enter a name for your storage class.
encrypted	In the parameters, enter true to create a storage class that sets up encryption for your Block Storage volumes. If you set this option to true, we must provide the root key CRN of your Key Protect service instance that we want to use in parameters.encryptionKey.
encryptionKey	In the parameters, enter the root key CRN that you retrieved earlier.

Example Kubernetes secret:

 apiVersion: v1
 kind: Secret
 type: vpc.block.csi.ibm.io
 metadata:
   name: <secret_name>
   namespace: <namespace_name>
 stringData:
   encrypted: <true_or_false>
 data
   encryptionKey: <encryption_key>

Component	Description
name	Enter a name for your secret.
namespace	Enter the namespace where we want to create your secret.
encrypted	In the parameters, enter true to set up encryption for your Block Storage volumes.
encryptionKey	In the parameters, enter the root key CRN of your Key Protect service instance that we want to use to encrypt your Block Storage volume. To use your root key CRN in a secret, we must first convert it to base64 by running echo -n "<root_key_CRN>" \| base64.

Follow step 4-9 in Adding Block Storage for VPC to your apps to create a PVC with your customized storage class to provision Block Storage for VPC that is configured for encryption with your Key Protect root key. Then, mount this storage to a sample app.

Your app might take a few minutes to mount the storage and get into a Running state.

Verify that your data is encrypted.

List your Block Storage volumes and note the ID of the instance that you created. The storage instance Name equals the name of the PV that was automatically created when you created the PVC.

ibmcloud is vols

Example output:

ID                                     Name                                       Status      Capacity   IOPS   Profile           Attachment type   Created                     Zone         Resource group
a395b603-74bf-4703-8fcb-b68e0b4d6960   pvc-479d590f-ca72-4df2-a30a-0941fceeca42   available   10         3000   5iops-tier        data              2019-08-17T12:29:18-05:00   us-south-1   a8a12accd63b437bbd6d58fb6a462ca7

Using the volume ID, list the details for your Block Storage instance to ensure that your Key Protect root key is stored in the storage instance. You can find the root key in the Encryption key field of your CLI output.

ibmcloud is vol <volume_ID>

Example output:

ID                                     a395b603-74bf-4703-8fcb-b68e0b4d6960   
Name                                   pvc-479d590f-ca72-4df2-a30a-0941fceeca42   
Status                                 available   
Capacity                               10   
IOPS                                   3000   
Profile                                5iops-tier   
Encryption key                         crn:v1:bluemix:public:kms:us-south:a/6ef045fd2b43266cfe8e6388dd2ec098:53369322-958b-421c-911a-c9ae8d5156d1:key:47a985d1-5f5e-4477-93fc-12ce9bae343f   
Encryption                             user_managed   
Resource group                         a8a12accd63b437bbd6d58fb6a462ca7
Created                                2019-08-17T12:29:18-05:00
Zone                                   us-south-1   
Volume Attachment Instance Reference

Customizing the default storage settings

You can change some of the default PVC settings by using a customized storage class or a Kubernetes secret to create Block Storage for VPC with your customized settings.

What is the benefit of using a secret and specifying my parameters in a customized storage class?
As a cluster admin, create a customized storage class when we want all of the PVCs that your cluster users create to be provisioned with a specific configuration and you don't want to enable your cluster users to override the default configuration.

However, when multiple configurations are required and you don't want to create a customized storage class for every possible PVC configuration, you can create one customized storage class with the default PVC settings and a reference to a generic Kubernetes secret. If your cluster users must override the default settings of your customized storage class, they can do so by creating a Kubernetes secret that holds their custom settings.

When we want to set up encryption for your Block Storage instance, you can also use a Kubernetes secret if we want to encode the Key Protect root key CRN to base64 instead of providing the key directly in the customized storage class.

Customizing a storage class

Use one of the IBM-provided storage classes as a basis to create your own customized storage class with the preferred settings for your Block Storage instance.

Review step 1 and 2 in Adding Block Storage for VPC to your apps to find the pre-defined storage class that best meets the performance and capacity requirements of your app. This storage class is used as the basis to create your own customized storage class.

Retrieve the YAML file for the storage class that we want to use as the basis to create your own customized storage class. For example, the following command retrieves the YAML file for the ibmc-vpc-block-5iops-tier storage class.

oc get storageclass ibmc-vpc-block-5iops-tier -o yaml

Example output:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    armada-service: addon-vpc-block-csi-driver
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"armada-service":"addon-vpc-block-csi-driver","version":"0.0.1_57"},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","app":"ibm-vpc-block-csi-driver"},"name":"ibmc-vpc-block-5iops-tier"},"parameters":{"billingType":"hourly","classVersion":"1","csi.storage.k8s.io/fstype":"ext4","encrypted":"false","encryptionKey":"","generation":"gc","profile":"5iops-tier","resourceGroup":"","sizeRange":"[10-2000]GiB","tags":"","zone":""},"provisioner":"vpc.block.csi.ibm.io","reclaimPolicy":"Delete"}
    version: 0.0.1_57
  creationTimestamp: "2019-08-02T20:29:29Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    app: ibm-vpc-block-csi-driver
  name: ibmc-vpc-block-5iops-tier
  resourceVersion: "458548"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/ibmc-vpc-block-5iops-tier
  uid: 94a920c6-cfda-4a57-9332-1f9b78881d50
parameters:
  billingType: hourly
  classVersion: "1"
  csi.storage.k8s.io/fstype: ext4
  encrypted: "false"
  encryptionKey: ""
  generation: gc
  profile: 5iops-tier
  resourceGroup: ""
  sizeRange: '[10-2000]GiB'
  tags: ""
  zone: ""
provisioner: vpc.block.csi.ibm.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

Create a customized storage class YAML file that is based on the YAML file that you retrieved. You can streamline your YAML file by removing all of the information from the metadata section, except for the name.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <storage_class_name>
provisioner: vpc.block.csi.ibm.io
parameters:
  profile: "5iops-tier"
  sizeRange: "<size_range>"
  csi.storage.k8s.io/fstype: "<file_system_type>"
  billingType: "hourly"
  encrypted: "<encrypted_true_false>"
  encryptionKey: "<encryption_key>"
  resourceGroup: ""
  zone: "<zone>"
  tags: "<tags>"
  generation: "gc"
  classVersion: "1"
reclaimPolicy: "<reclaim_policy>"
volumeBindingMode: <volume_binding_mode>

Scroll for moreScroll for more

Component	Description
name	Enter a name for your storage class.
sizeRange	In the parameters, enter the size range for your storage in gigabytes (GiB), such as [10-2000]GiB. The size range must match the Block Storage for VPC profile that you specify in parameters.profile. To find supported storage sizes for a specific profile, see Tiered IOPS profiles. Any PVC that uses this storage class must specify a size value that is within this range.
csi.storage.k8s.io/fstype	In the parameters, enter the file system for your Block Storage instance. Choose xfs, ext3, or ext4. The default value is ext4 and is used if you do not specify a file system.
encrypted	In the parameters, enter true to create a storage class that sets up encryption for your {{site.data.keyworblockstorageshort}} volume. If you set this option to true, we must provide the root key CRN of your Key Protect service instance that we want to use in parameterencryptionKey. For more information about encrypting your data, see Setting up encryption for your {{sitdata.keyword.block_storage_is_short}}.
encryptionKey	If you entered true for parameters.encrypted, then enter the root key CRN of your Key Protect service instance that we want to use to encrypt your Block Storage volume. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.
zone	In the parameters, enter the VPC zone where we want to create the Block Storage for VPC instance. Make sure that you use a zone that your worker nodes are connected to. To list VPC zones that your worker nodes use, run ibmcloud oc cluster get --cluster and look at the Worker Zones field in your CLI output. If you do not specify a zone, one of the worker node zones is automatically selected for your Block Storage for VPC instance.
tags	In the parameters, enter a comma-separated list of tags to apply to your {{site.data.keyworblock_storage_is_short}} instance. Tags can help we find instances more easily or group your instances based on common characteristics, such as the app or the environment that it is used for.
reclaimPolicy	Enter the reclaim policy for your storage class. If we want to keep the PV, the physical storage device and your data when you remove the PVC, enter Retain. If we want to delete the PV, the physical storage device and your data when you remove the PVC, enter Delete.
volumeBindingMode	Choose if we want to delay the creation of the Block Storage for VPC instance until the first pod that uses this storage is ready to be scheduled. To delay the creation, enter WaitForFirstConsumer. To create the instance when you create the PVC, enter Immediate.

Create the customized storage class in your cluster.
```
oc apply -f custom_storageclass.yaml
```

Verify that your storage class is available in the cluster.

oc get storageclasses

Example output:

NAME                                    PROVISIONER            AGE
ibmc-vpc-block-10iops-tier              vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-5iops-tier               vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-custom                   vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-general-purpose          vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-retain-10iops-tier       vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-retain-5iops-tier        vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-retain-custom            vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-retain-general-purpose   vpc.block.csi.ibm.io   4d21h
custom_storageclass                     vpc.block.csi.ibm.io   4m26s

Follow the steps in Adding Block Storage for VPC to your apps to create a PVC with your customized storage class to provision Block Storage for VPC. Then, mount this storage to a sample app.

Storing your custom PVC settings in a Kubernetes secret

Specify your PVC settings in a Kubernetes secret and reference this secret in a customized storage class. Then, use the customized storage class to create a PVC with the custom parameters that you set in your secret.

What options do I have to use the Kubernetes secret?
As a cluster admin, you can choose if we want to allow each cluster user to override the default settings of a storage class, or if we want to create one secret that everyone in your cluster must use and that enforces base64 encoding for your Key Protect root key CRN.

Every user can customize the default settings: In this scenario, the cluster admin creates one customized storage class with the default PVC settings and a reference to a generic Kubernetes secret. Cluster users can override the default settings of the storage class by creating a Kubernetes secret with the PVC settings that they want. In order for the customized settings in the secret to get applied to your Block Storage instance, we must create a PVC with the same name as your Kubernetes secret.
Enforce base64 encoding for the Key Protect root key: In this scenario, you create one customized storage class with the default PVC settings and a reference to a static Kubernetes secret that overrides or enhances the default settings of the customized storage class. Your cluster users cannot override the default settings by creating their own Kubernetes secret. Instead, cluster users must provision Block Storage for VPC with the configuration that you chose in your customized storage class and secret. The benefit of using this method over creating a customized storage class only is that you can enforce base64 encoding for the root key CRN of your Key Protect service instance when we want to encrypt the data in your Block Storage instance.

What do I need to be aware of before I start using the Kubernetes secret for my PVC settings?
Some of the PVC settings, such as the reclaimPolicy, fstype, or the volumeBindingMode cannot be set in the Kubernetes secret and must be set in the storage class. As the cluster admin, if we want to enable your cluster users to override your default settings, we must ensure that you set up enough customized storage classes that reference a generic Kubernetes secret so that your users can provision Block Storage for VPC with different reclaimPolicy, fstype, and volumeBindingMode settings.

Enabling every user to customize the default PVC settings

As the cluster admin, follow the steps to create a customized storage class. In the customized storage class YAML file, reference the Kubernetes secret in the metadata.annotation section as follows. Make sure to add the code as-is and not to change variables names.
```
csi.storage.k8s.io/provisioner-secret-name: ${pvc.name}
csi.storage.k8s.io/provisioner-secret-namespace: ${pvc.namespace}
```

As the cluster user, create a Kubernetes secret that customizes the default settings of the storage class.

apiVersion: v1
kind: Secret
type: vpc.block.csi.ibm.io
metadata:
  name: <secret_name>
  namespace: <namespace_name>
stringData:
  iops: "<IOPS_value>"
  zone: "<zone>"
  tags: "<tags>"
  encrypted: <true_or_false>
  resourceGroup: "<resource_group>"
data
  encryptionKey: <encryption_key>

Component	Description
name	Enter a name for your Kubernetes secret.
namespace	Enter the namespace where we want to create your secret. To reference the secret in your PVC, the PVC must be created in the same namespace.
iops	In the string data section, enter the range of IOPS that we want to allow for your Block Storage instance. The range that you enter must match the Block Storage for VPC tier that we plan to use.
zone	In the string data section, enter the VPC zone where we want to create the Block Storage instance. Make sure that you use a zone that your worker nodes are connected to. To list VPC zones that your worker nodes use, run ibmcloud oc cluster get --cluster <cluster_name_or_ID> and look at the Worker Zones field in your CLI output. If you do not specify a zone, one of the worker node zones is automatically selected for your Block Storage instance.
tags	In the string data section, enter a comma-separated list of tags to use when the PVC is created. Tags can help we find your storage instance more easily after it is created.
resourceGroup	In the string data section, enter the resource group that we want your Block Storage instance to get access to. If you do not enter a resource group, the instance is automatically authorized to access resources of the resource group that your cluster belongs to.
encrypted	In the string data section, enter true to create a secret that sets up encryption for Block Storage volumes. If you set this option to true, we must provide the root key CRN of your Key Protect service instance that we want to use in parameters.encryptionKey. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.
encryptionKey	In the data section, if you entered true for parameters.encrypted, then enter the root key CRN of your Key Protect service instance that we want to use to encrypt your Block Storage volumes. To use your root key CRN in a secret, we must first convert it to base64 by running echo -n "<root_key_CRN>" \| base64. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.

Create your Kubernetes secret.
```
oc apply -f secret.yaml
```
Follow the steps in Adding Block Storage for VPC to your apps to create a PVC with your custom settings. Make sure to create the PVC with the customized storage class that the cluster admin created and use the same name for your PVC that you used for your secret. Using the same name for the secret and the PVC triggers the storage provider to apply the settings of the secret in your PVC.

Enforcing base64 encoding for the Key Protect root key CRN

As the cluster admin, create a Kubernetes secret that includes the base64 encoded value for your Key Protect root key CRN. To retrieve the root key CRN, see Setting up encryption for your Block Storage for VPC.

apiVersion: v1
kind: Secret
type: vpc.block.csi.ibm.io
metadata:
  name: <secret_name>
  namespace: <namespace_name>
stringData:
  encrypted: <true_or_false>
  resourceGroup: "<resource_group>"
data
  encryptionKey: <encryption_key>

Component	Description
name	Enter a name for your Kubernetes secret.
namespace	Enter the namespace where we want to create your secret. To reference the secret in your PVC, the PVC must be created in the same namespace.
encrypted	In the string data section, enter true to create a secret that sets up encryption for Block Storage volumes. If you set this option to true, we must provide the root key CRN of your Key Protect service instance that we want to use in parameters.encryptionKey. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.
encryptionKey	In the data section, if you entered true for parameters.encrypted, then enter the root key CRN of your Key Protect service instance that we want to use to encrypt your Block Storage volume. To use your root key CRN in a secret, we must first convert it to base 64 by running echo -n "<root_key_CRN>" \| base64. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.

Create the Kubernetes secret.
```
oc apply -f secret.yaml
```
Follow the steps to create a customized storage class. In the customized storage class YAML file, reference the Kubernetes secret in the metadata.annotation section as follows. Make sure to enter the name of the Kubernetes secret that you created earlier and the namespace where you created the secret.
```
csi.storage.k8s.io/provisioner-secret-name: <secret_name>
csi.storage.k8s.io/provisioner-secret-namespace: <secret_namespace>
```
As the cluster user, follow the steps in Adding Block Storage for VPC to your apps to create a PVC from your customized storage class.

Backing up and restoring data

Data that is stored on Block Storage for VPC is secured across redundant fault zones in your region. To manually back up your data, use the Kubernetes oc cp command.

You can use the oc cp command to copy files and directories to and from pods or specific containers in your cluster

Before beginning: Access your OpenShift cluster.

To back up or restore data, choose between the following options:

Copy data from the local machine to a pod in your cluster:

oc cp <local_filepath>/<filename> <namespace>/<pod>:<pod_filepath>

Copy data from a pod in your cluster to the local machine:

oc cp <namespace>/<pod>:<pod_filepath>/<filename> <local_filepath>/<filename>

Copy data from the local machine to a specific container that runs in a pod in your cluster:
```
oc cp <local_filepath>/<filename> <namespace>/<pod>:<pod_filepath> -c <container>
```

Storage class reference

Storage classes that have retain in the title have a reclaim policy of Retain. Example: ibmc-file-retain-bronze. Storage classes that do not have retain in the title have a reclaim policy of Delete. Example: ibmc-file-bronze.

10 IOPS-tier5 IOPS-tierCustomScroll for moreScroll for more

Characteristics	Setting
Name	ibmc-vpc-block-10iops-tier ibmc-vpc-block-retain-10iops-tier
File system	ext4
Corresponding Block Storage for VPC tier	10 IOPS/GB
Reclaim policy	ibmc-vpc-block-10iops-tier: Delete ibmc-vpc-block-retain-10iops-tier: Retain
Billing	Hourly
Pricing	Pricing information

Scroll for moreScroll for more

Characteristics	Setting
Name	ibmc-vpc-block-5iops-tier ibmc-vpc-block-retain-5iops-tier
File system	ext4
Corresponding Block Storage for VPC tier	5 IOPS/GB
Reclaim policy	ibmc-vpc-block-5iops-tier: Delete ibmc-vpc-block-retain-5iops-tier: Retain
Billing	Hourly
Pricing	Pricing information

Scroll for moreScroll for more

Characteristics	Setting
Name	ibmc-vpc-block-custom ibmc-vpc-block-retain-custom
File system	ext4
Corresponding Block Storage for VPC tier	Custom
Reclaim policy	ibmc-vpc-block-custom: Delete ibmc-vpc-block-retain-custom: Retain
Billing	Hourly
Pricing	Pricing information

Scroll for moreScroll for more

Characteristics	Setting
Name	ibmc-vpc-block-general-purpose ibmc-vpc-block-retain-general-purpose
File system	ext4
Corresponding Block Storage for VPC tier	3 IOPS/GB
Reclaim policy	ibmc-vpc-block-general-purpose: Delete ibmc-vpc-block-retain-general-purpose: Retain
Billing	Hourly
Pricing	Pricing information

Removing persistent storage from a cluster

When you set up persistent storage in your cluster, we have three main components: the Kubernetes persistent volume claim (PVC) that requests storage, the Kubernetes persistent volume (PV) that is mounted to a pod and described in the PVC, and the IBM Cloud infrastructure instance, such as classic file or block storage. Depending on how you created your storage, we might need to delete all three components separately.

Understanding your storage removal options

Removing persistent storage from your IBM Cloud account varies depending on how you provisioned the storage and what components you already removed.

Is my persistent storage deleted when I delete my cluster?
During cluster deletion, we have the option to remove your persistent storage. However, depending on how your storage was provisioned, the removal of your storage might not include all storage components.

If you dynamically provisioned storage with a storage class that sets reclaimPolicy: Delete, your PVC, PV, and the storage instance are automatically deleted when you delete the cluster. For storage that was statically provisioned, or storage that you provisioned with a storage class that sets reclaimPolicy: Retain, the PVC and the PV are removed when you delete the cluster, but your storage instance and your data remain. You are still charged for your storage instance. Also, if you deleted your cluster in an unhealthy state, the storage might still exist even if you chose to remove it.

How do I delete the storage when I want to keep my cluster?
When you dynamically provisioned the storage with a storage class that sets reclaimPolicy: Delete, you can remove the PVC to start the deletion process of your persistent storage. Your PVC, PV, and storage instance are automatically removed.

For storage that was statically provisioned, or storage that you provisioned with a storage class that sets reclaimPolicy: Retain, we must manually remove the PVC, PV, and the storage instance to avoid further charges.

How does the billing stop after I delete my storage?
Depending on what storage components you delete and when, the billing cycle might not stop immediately. If you delete the PVC and PV, but not the storage instance in your IBM Cloud account, that instance still exists and we are charged for it.

If you delete the PVC, PV, and the storage instance, the billing cycle stops depending on the billingType that you chose when you provisioned your storage and how you chose to delete the storage.

When you manually cancel the persistent storage instance from the IBM Cloud console or the ibmcloud sl CLI, billing stops as follows:
- Hourly storage: Billing stops immediately. After your storage is canceled, we might still see your storage instance in the console for up to 72 hours.
- Monthly storage: You can choose between immediate cancellation or cancellation on the anniversary date. In both cases, we are billed until the end of the current billing cycle, and billing stops for the next billing cycle. After your storage is canceled, we might still see your storage instance in the console or the CLI for up to 72 hours.
  - Immediate cancellation: Choose this option to immediately remove your storage. Neither you nor your users can use the storage anymore or recover the data.
  - Anniversary date: Choose this option to cancel your storage on the next anniversary date. Your storage instances remain active until the next anniversary date and you can continue to use them until this date, such as to give your team time to make backups of your data.
When you dynamically provisioned the storage with a storage class that sets reclaimPolicy: Delete and you choose to remove the PVC, the PV and the storage instance are immediately removed. For hourly billed storage, billing stops immediately. For monthly billed storage, we are still charged for the remainder of the month. After your storage is removed and billing stops, we might still see your storage instance in the console or the CLI for up to 72 hours.

What do I need to be aware of before I delete persistent storage?
When you clean up persistent storage, you delete all the data that is stored in it. For a copy of the data, make a backup for file storage or block storage.

I deleted my storage instance. Why can I still see my instance?
After you remove persistent storage, it can take up to 72 hours for the removal to be fully processed and for the storage to disappear from your IBM Cloud console or CLI.

Cleaning up persistent storage

Remove the PVC, PV, and the storage instance from your IBM Cloud account to avoid further charges for your persistent storage.

Before beginning:

Make sure that you backed up any data that we want to keep.
Access your OpenShift cluster.

To clean up persistent data:

List the PVCs in your cluster and note the NAME of the PVC, the STORAGECLASS, and the name of the PV that is bound to the PVC and shown as VOLUME.

oc get pvc

Example output:

NAME                  STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS            AGE
claim1-block-bronze   Bound     pvc-06886b77-102b-11e8-968a-f6612bb731fb   20Gi       RWO           ibmc-block-bronze       78d
claim-file-bronze     Bound     pvc-457a2b96-fafc-11e7-8ff9-b6c8f770356c   4Gi        RWX           ibmc-file-bronze-retain 105d
claim-file-silve      Bound     pvc-1efef0ba-0c48-11e8-968a-f6612bb731fb   24Gi       RWX           ibmc-file-silver        83d

Review the ReclaimPolicy and billingType for the storage class.
```
oc describe storageclass <storageclass_name>
```
If the reclaim policy says Delete, your PV and the physical storage are removed when you remove the PVC. If the reclaim policy says Retain, or if you provisioned your storage without a storage class, then your PV and physical storage are not removed when you remove the PVC. You must remove the PVC, PV, and the physical storage separately.

If your storage is charged monthly, you still get charged for the entire month, even if you remove the storage before the end of the billing cycle.
Remove any pods that mount the PVC.
1. List the pods that mount the PVC.
```
oc get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.volumes[*]}{.persistentVolumeClaim.claimName}{" "}{end}{end}' | grep "<pvc_name>"
```
  Example output:
```
blockdepl-12345-prz7b:    claim1-block-bronze
```
  If no pod is returned in your CLI output, you do not have a pod that uses the PVC.
2. Remove the pod that uses the PVC. If the pod is part of a deployment, remove the deployment.
```
oc delete pod <pod_name>
```
3. Verify that the pod is removed.
```
oc get pods
```
Remove the PVC.
```
oc delete pvc <pvc_name>
```
Review the status of your PV. Use the name of the PV that you retrieved earlier as VOLUME.
```
oc get pv <pv_name>
```
When you remove the PVC, the PV that is bound to the PVC is released. Depending on how you provisioned your storage, your PV goes into a Deleting state if the PV is deleted automatically, or into a Released state, if we must manually delete the PV. Note: For PVs that are automatically deleted, the status might briefly say Released before it is deleted. Rerun the command after a few minutes to see whether the PV is removed.
If your PV is not deleted, manually remove the PV.
```
oc delete pv <pv_name>
```
Verify that the PV is removed.
```
oc get pv
```