Achieving cross zone HA in GKE – MayaData Help Center

Why a multi-zone cluster?

GKE has multiple zones within each region, when we set up a cluster by default all the nodes get deployed in a single zone. Though rare, GCP data centres of a zone may face an outage, which might cause a business disruption lasting for a few minutes or for hours so in order to increase resiliency we can deploy the nodes of a cluster in different zones of a region. To know about setting up multi-zone cluster you can follow Google Cloud Docs.

But there are certain limitations with GKE multi-zone cluster as well,

When a node goes down a new node will come up automatically (if autoscaling is enabled), but the problem lies with data availability in case of the attached disk is ephemeral. When a node goes down the data associated with the node is lost.
Secondly, even if your application is capable of performing replication of data on its own, it will be more time consuming thereby resulting in a negative impact on the performance.

Hence, in any of the cases, it is advisable to provision OpenEBS volumes.

How is replication Done with OpenEBS?

OpenEBS needs a minimum of 3 replicas to run the OpenEBS cluster with high availability. If a node fails, OpenEBS will manage the data to be replicated to a new disk. In the meantime, your workload can access data from one of the replicas thereby ensuring better performance and no downtime.

Getting started with OpenEBS Enterprise Platform is simple all you need is to follow the below-mentioned steps.

WORKFLOW:-

STEP 1

Getting Started with OpenEBS Enterprise Platform

Install OpenEBS on your cluster using the following command:

kubectl apply -f https://openebs.github.io/charts/openebs-operator-1.3.0.yaml

In order to verify whether the namespace is created and all pods are working as expected, run the following command:

kubectl get ns

The OpenEBS namespace should be in Active State. ( By default pods are created in OpenEBS namespace)

NAME         STATUS        AGE
default      Active        8m
kube-public  Active        8m
kube-system  Active        8m
openebs      Active        11s

Run the following command to check all OpenEBS pods are in running state using the following command: By default OpenEBS pods are created in openebs namespace.

kubectl get pod -n openebs

Example output:

NAME                                          READY   STATUS    RESTARTS   AGE   IP             NODE                                         
maya-apiserver-7867696cb4-56jrw               1/1     Running   2          38m   100.96.3.2     ip-10-5-66-142.us-east-2.compute.internal    
openebs-admission-server-6ffc599f77-vnsvh     1/1     Running   0          38m   100.96.3.3     ip-10-5-66-142.us-east-2.compute.internal    
openebs-localpv-provisioner-59796cd78-jwt9t   1/1     Running   0          38m   100.96.2.3     ip-10-5-55-83.us-east-2.compute.internal     
openebs-ndm-5wsk9                             1/1     Running   0          38m   10.5.113.122   ip-10-5-113-122.us-east-2.compute.internal   
openebs-ndm-gm7j7                             1/1     Running   0          38m   10.5.66.142    ip-10-5-66-142.us-east-2.compute.internal    
openebs-ndm-l2l8x                             1/1     Running   0          38m   10.5.55.83     ip-10-5-55-83.us-east-2.compute.internal     
openebs-ndm-operator-548c4b9d87-gmt9x         1/1     Running   1          38m   100.96.2.2     ip-10-5-55-83.us-east-2.compute.internal     
openebs-ndm-xn2cq                             1/1     Running   0          38m   10.5.51.65     ip-10-5-51-65.us-east-2.compute.internal     
openebs-provisioner-744b56bd77-p5vpj          1/1     Running   0          38m   100.96.4.2     ip-10-5-51-65.us-east-2.compute.internal     
openebs-snapshot-operator-69658bd85b-7dxvn    2/2     Running   0          38m   100.96.4.3     ip-10-5-51-65.us-east-2.compute.internal

An easier way to monitor pods and other components that will be used in further steps is with Director Online or Director onPrem.

STEP 2

Creating and Attaching Disks

Next, you need to create disk and attach them to the desired VMs, this can be done either from the console or using command line arguments.

In order to get the detailed steps to create and attach disks in GCP click the corresponding link.

1. Creation of disk

2. Attaching the disk

STEP 3

Provisioning OpenEBS volumes

Next, you have to provision OpenEBS volumes (cStor or Jiva or Local PV). Going through the following links can help you decide as to which will be best suited for your scenario.

Click here to know about cStor.

Click here to know about Jiva.

Click here to know about Local PV.

First, Create a cStorPool

For creation of a cStorPool you will need to specify the blockdevices, to view the disks attached in step 2 execute the following command

kubectl get blockdevice -n <openebs_namespace>

The output looks something like this:

NAME                                         NODENAME                                       SIZE        CLAIMSTATE STATUS AGE
blockdevice-66a74896b61c60dcdaf7c7a76fde0ebb gke-anupriya-az-doc-default-pool-152c1305-x7s8 42949672960 Claimed    Active 22h 
blockdevice-b34b3f97840872da9aa0bac1edc9578a gke-anupriya-az-doc-default-pool-c36a00d4-549p 42949672960 Claimed    Active 22h
blockdevice-ce41f8f5fa22acb79ec56292441dc207 gke-anupriya-az-doc-default-pool-25b9dd3a-kgq5 42949672960 Claimed    Active 22h

Now, copy the below snippet with your blockdevices specified in a file, say cstor-pool-config.yaml

apiVersion: openebs.io/v1alpha1
kind: StoragePoolClaim
metadata:
  name: cstor-disk-pool
  annotations:
    cas.openebs.io/config: |
      - name: PoolResourceRequests
        value: |-
            memory: 2Gi
      - name: PoolResourceLimits
        value: |-
            memory: 4Gi
spec:
  name: cstor-disk-pool
  type: disk
  poolSpec:
    poolType: striped
  blockDevices:
    blockDeviceList:
    ## Replace the following with actual blockDevice CRs from your cluster using "kubectl get bd -n openebs" .
    - blockdevice-66a74896b61c60dcdaf7c7a76fde0ebb
    - blockdevice-b34b3f97840872da9aa0bac1edc9578a
    - blockdevice-ce41f8f5fa22acb79ec56292441dc207
---

Now, apply the above yaml file using the following command:

kubectl apply -f cstor-pool-config.yaml

To verify execute :

kubectl get spc

The output should have a cStor-pool created

NAME              AGE
cstor-disk-pool   1m

Also, Verify csp status:

kubectl get csp

The output status of the csp must be Healthy.

NAME                  ALLOCATED  FREE   CAPACITY STATUS  TYPE     AGE 
cstor-disk-pool-c4qj  269K       29.7G  29.8G    Healthy striped  2m 
cstor-disk-pool-c4ql  206K       29.7G  29.8G    Healthy striped  2m
cstor-disk-pool-gke5  257K       29.7G  29.8G    Healthy striped  2m

For further confirmation that things are working as expected execute the following command:

kubectl get pods -n openebs | grep <poolname>

Here, the name of pool is cstor-disk-pool so the command to be executed is:

kubectl get pods -n openebs | grep cstor-disk-pool

The output should display pods in running state.

cstor-disk-pool-c4qj-664bd98889-pclpb          3/3     Running   0          42m
cstor-disk-pool-c4ql-5fcb7646d6-jlk6w          3/3     Running   0          42m
cstor-disk-pool-gke5-76d97dc449-8px9c          3/3     Running   0          42m

Next, You have to create a StorageClass and mention about the StoragePoolClaim and ReplicaCount as the number of cStor Volume Replica is getting created.
Copy the YAML file given below, say cstor-sc.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-sc-statefulset
  annotations:
    openebs.io/cas-type: cstor
    cas.openebs.io/config: |
      - name: StoragePoolClaim
        value: "cstor-disk-pool"
      - name: ReplicaCount
        value: "3"    
provisioner: openebs.io/provisioner-iscsi

To apply the above yaml execute:

kubectl apply -f cstor-sc.yaml

To verify execute:

kubectl get sc

The output must contain the StorageClass name specified in the yaml file.

NAME                            PROVISIONER                         AGE
openebs-sc-statefulset          openebs.io/provisioner-iscsi        1m

Next, you need a PVC spec or volumeClaimTemplate to use the StorageClass that is pointing to a pool with real disks.

Note: This spec will be included in the application yaml given in step 4, it is given here only for your understanding.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cstor-pvc-mysql-large
spec:
  storageClassName: openebs-sc-statefulset
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Gi

First, you need to mount your attached disks. If the disk attached is mapped to /dev/sdb you need to verify the size, format the disk and mount it.
SSH into each node and follow the below-mentioned steps:

Command to SSH into a node:

gcloud compute ssh <node_nme> --zone <zone_name>

Command to get the device name:

sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL

Now,Now, format the device with ext4 filesystem and then create new directory if required and then mount the formatted device to the directory.

sudo mkfs.ext4 /dev/<device-name>
sudo mkdir <new-directory> /home/openebs-gpd 
sudo mount /dev/<device-name> <new-directory> /home/openebs-gpd

Example:

sudo mkfs.ext4 /dev/sdb
sudo mkdir /home/openebs-gpd 
sudo mount /dev/sdb  /home/openebs-gpd

Next, Create a Jiva Pool using the mounted disk

Jiva pool requires mount path to be prepared and available on the Node.

Note: if the mount path is not pointing a real disk, then a local directory is created with this mount path and the replica data goes to the container image disk (similar to the case of default pool).

Copy the below content into a file called jiva-gpd-pool.yaml

apiVersion: openebs.io/v1alpha1
kind: StoragePool
metadata:
  name: gpdpool            
  type: hostdir
spec:
  path: "/home/openebs-gpd"

Create the pool using the following command.

kubectl apply -f jiva-gpd-pool.yaml

You can verify if the pool is created or not using the following command

kubectl get storagepool

The output would look something like this,

NAME      AGE
default   2d
gpdpool   14s

Next, you need to create a StorageClass

This StorageClass is mainly for using the Jiva Storagepool created with a mounted disk. Jiva volume can be provisioned using Jiva StorageClass named openebs-jiva-gpd-3repl in the corresponding PVC spec. The default StorageClass has replica count as 3. Specify the Jiva pool in the StoragePool annotation of StorageClass. Add ReplicaAntiAffinityTopoKey for ensuring that no two replica pods of the same volume will be co-located on the same availability zone Example StorageClass specification is given below.
Copy the below content into a file called jiva-gpd-3repl-sc.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-jiva-gpd-3repl
  annotations:
    openebs.io/cas-type: jiva
    cas.openebs.io/config: |
      - name: ReplicaCount
        value: "3"
      - name: StoragePool
        value: gpdpool
      - name: ReplicaAntiAffinityTopoKey
        value: failure-domain.beta.kubernetes.io/zone
provisioner: openebs.io/provisioner-iscsi

Create the StorageClass using the following command:

kubectl apply -f jiva-gpd-3repl-sc.yaml

To verify execute

kubectl get sc

The output looks like,

NAME                        PROVISIONER                                                AGE
openebs-jiva-gpd-3repl      openebs.io/provisioner-iscsi                               74s

Next you need a PVC spec

Note: This PVC spec has been included in the Application YAML given in Step 4, the below spec is just for a clearer understanding.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: demo-vol1-claim
spec:
  storageClassName: openebs-jiva-gpd-3repl
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4G

Local PVs are provisioned dynamically by OpenEBS Local PV provisioner. Local PV with default StorageClass can be provisioned with storage-type as device or hostpath. Local PV gets provisioned dynamically on the node after the application is provisioned. This is ensured by adding volumeBindingMode: WaitForFirstConsumer in the yaml file

As the Local PV gets provisioned on the node where the application is present, to have multi-zone cluster ensure the application get replicated on the desired nodes.
You can refer the following links to view the yaml file for provisioning Local PV

STEP 4

Provisioning an application.

Now, things are all set to deploy an application and get started.

You can refer to the below sample articles for provisioning an application.

1. Sample application for cStor.

2. Sample application for Jiva.

3. Sample application for Local PV.

STEP 5

Backing up Application

It is extremely important to have a data backup and restore mechanism so that the service continues flawlessly in every situation.

Backup and Restore for cStor can be done using DMaaS, a solution for data agility and data motion in Kubernetes platform. DMaaS being application aware ensures that the state of application remains consistent before and after migration, also the time taken to restore the operations back to normal mode is quite optimised.
DMaaS is an offering from MayaData that works with both Director Online as well as OnPrem. To know more about configuring DMaaS with your cluster click here.

For applications provisioned using Local PV and Jiva volume, restore and backup can be done using Velero. Velero provides tools to backup and restore Kubernetes cluster resources and persistent volume. Velero gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. Using Velero you can:

Take backups of your cluster and restore in case of loss.
Migrate cluster resources to other clusters.
Replicate your production cluster to development and testing clusters.

You can run Velero in clusters on a cloud provider or on-premises.
To know the steps to create and manage backups of jiva volume using Velero Click Here.
To know the steps to create and manage backups of Loacal PV using Velero Click Here.

STEP 6

Monitoring of stateful applications and OpenEBS volumes

Once the entire thing is set up, volumes are provisioned, the application is deployed it is necessary to monitor these components continuously. Director provides easy and efficient monitoring. To know about how to monitor, different monitoring views click here.

OpenEBS volumes can be deployed on various other Kubernetes Platform as well, to know about it Click Here.

Summary

This article gives you an overview of multi-zone clusters in GKE and deploying of OpenEBS volumes on these clusters. Multi-zone cluster with OpenEBS volumes(cStor or Jiva) provisioned on it ensures data availability and high resiliency to failures at all times.

Summary

Related articles

Articles in this section