In this article, we are going to use OpenEBS Enterprise Edition's cStor storage engine and kubernetes-retired/external-storage/nfs to create an RWX volume (ReadWriteMany accessmode).
Prerequisites
Prerequisites for the cStor-based NFS server are as follows:
- A Kubernetes cluster running Kubernetes v1.14 or above
- Block devices for storage -- At least 1 block device is required. You will need at least 3 nodes and block devices on each of those nodes for high-availablility.
- iSCSI initiator packages installed on all of your storage nodes. Use the
systemctl status iscsidcommand to check if the iSCSI initiator service is active and running. You can install the iSCSI initiator packages using one of the following commands:- For Ubuntu/Debian:
apt-get install -y open-iscsi - For CentOS/RHEL:
yum install -y iscsi-initiator-utils
- For Ubuntu/Debian:
- NFS common and NFS client packages must installed on the client nodes. You can install them using one of the commands below:
- For Ubuntu/Debian:
apt-get install -y nfs-common - For CentOS/RHEL:
yum install -y nfs-utils
- For Ubuntu/Debian:
Creating cStor StorageClass
STEP 1
Sign in to Kubera and connect your cluster at director.mayadata.io. Learn more about signing up and connecting your cluster here.
STEP 2
Install OpenEBS Enterprise Edition. For detailed instructions on installing OpenEBS Enterprise Edition, click here.
STEP 3
Create a cStor Pool Cluster. For detailed instructions on creating cStorPoolClusters, click here.
STEP 4
Create a cStor-CSI storage class. A minimum value of "3" is required in the parameters.replicaCount field for high-availability. Execute the following snippet in a terminal window with the kubectl tool configured:
CSPC_NAME=$(kubectl get cspc --sort-by='{.metadata.creationTimestamp}' -o=jsonpath='{.items[0].metadata.name}' -A)
CSPI_COUNT=$(kubectl get cspc --sort-by='{.metadata.creationTimestamp}' -o=jsonpath='{.items[0].status.provisionedInstances}' -A)
cat << EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: cstor-csi
provisioner: cstor.csi.openebs.io
allowVolumeExpansion: true
parameters:
cas-type: cstor
cstorPoolCluster: $CSPC_NAME
replicaCount: "$(if [ $CSPI_COUNT -lt 3 ];then echo -n "$CSPI_COUNT";else echo -n "3";fi)"
EOF
Creating NFS server
STEP 1
Create a namespace for the NFS server provisioner components. If you want to use an existing namespace for this, you may skip this step.
kubectl create ns nfs-server
STEP 2
Create the appropriate ServiceAccount, PodSecurityPolicy and RBAC objects. You can create them by using the following YAML spec. Copy the following snippet into a YAML file and use the kubectl create -f <file>.yaml command.
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-provisioner
# Namespace where the NFS server will be deployed
namespace: nfs-server
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: nfs-provisioner
spec:
fsGroup:
rule: RunAsAny
allowedCapabilities:
- DAC_READ_SEARCH
- SYS_RESOURCE
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- secret
- hostPath
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services", "endpoints"]
verbs: ["get"]
- apiGroups: ["extensions"]
resources: ["podsecuritypolicies"]
resourceNames: ["nfs-provisioner"]
verbs: ["use"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-provisioner
subjects:
- kind: ServiceAccount
name: nfs-provisioner
# Namespace where the NFS server will be deployed
namespace: nfs-server
roleRef:
kind: ClusterRole
name: nfs-provisioner-runner
apiGroup: rbac.authorization.k8s.io
STEP 3
Create the NFS server provisioner Service, StatefulSet and StorageClass. Use the following YAML spec.
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
# Namespace where the NFS server will be deployed
namespace: nfs-server
labels:
app: nfs-provisioner
spec:
ports:
- name: nfs
port: 2049
- name: nfs-udp
port: 2049
protocol: UDP
- name: nlockmgr
port: 32803
- name: nlockmgr-udp
port: 32803
protocol: UDP
- name: mountd
port: 20048
- name: mountd-udp
port: 20048
protocol: UDP
- name: rquotad
port: 875
- name: rquotad-udp
port: 875
protocol: UDP
- name: rpcbind
port: 111
- name: rpcbind-udp
port: 111
protocol: UDP
- name: statd
port: 662
- name: statd-udp
port: 662
protocol: UDP
selector:
app: nfs-provisioner
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: nfs-provisioner
# Namespace where the NFS server will be deployed
namespace: nfs-server
spec:
selector:
matchLabels:
app: nfs-provisioner
serviceName: "nfs-provisioner"
replicas: 1
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccount: nfs-provisioner
terminationGracePeriodSeconds: 10
containers:
- name: nfs-provisioner
image: quay.io/kubernetes_incubator/nfs-provisioner:v2.3.0
ports:
- name: nfs
containerPort: 2049
- name: nfs-udp
containerPort: 2049
protocol: UDP
- name: nlockmgr
containerPort: 32803
- name: nlockmgr-udp
containerPort: 32803
protocol: UDP
- name: mountd
containerPort: 20048
- name: mountd-udp
containerPort: 20048
protocol: UDP
- name: rquotad
containerPort: 875
- name: rquotad-udp
containerPort: 875
protocol: UDP
- name: rpcbind
containerPort: 111
- name: rpcbind-udp
containerPort: 111
protocol: UDP
- name: statd
containerPort: 662
- name: statd-udp
containerPort: 662
protocol: UDP
securityContext:
capabilities:
add:
- DAC_READ_SEARCH
- SYS_RESOURCE
args:
- "-provisioner=openebs.io/nfs"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: SERVICE_NAME
value: nfs-provisioner
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: export-volume
mountPath: /export
volumeClaimTemplates:
- metadata:
name: export-volume
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "cstor-csi"
resources:
requests:
# This is the size of the cStor volume.
# This storage space will become available to the NFS server.
storage: 10Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
# This is the name of the StorageClass we will use to create RWX volumes.
name: nfs-sc
provisioner: openebs.io/nfs
mountOptions:
- vers=4.1
STEP 4
Create ReadWriteMany PVCs using the NFS StorageClass to create RWX volumes. You can use these volumes in your application pods. Here's a sample PVC yaml:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nfs-claim
spec:
storageClassName: nfs-sc
accessModes: [ "ReadWriteMany" ]
resources:
requests:
# This is the storage capacity that will be available to the application pods.
storage: 9Gi