Mayastor is currently under development as a storage engine option for the Open Source CNCF project, OpenEBS. It is actively developed and maintained by MayaData, as part of the Kubera platform for data agility.
Mayastor has been developed keeping in mind the following design goals:
- High availability
- Durable persistence of data
- Readily deployable
- Low-overhead abstraction
- Easy management
It has been designed from the ground up to leverage the protocol and compute efficiency of NVMe-oF semantics, and the performance capabilities of the latest generation of solid state storage devices, in order to deliver a storage abstraction with performance overhead measured to be within the range of single digit percentages.
Mayastor is beta software. It is considered largely, feature complete and substantially without major known defects. Minor and unknown defects can be expected.
Prerequisites
Each Mayastor Node, that is each cluster worker node which will host an instance of a Mayastor pod, must have these resources free and available for use by Mayastor:
- Two x86-64 CPU cores with SSE4.2 instruction support:
- Intel Nehalem processor (march=nehalem) and newer
- AMD Bulldozer processor and newer
- 4GiB of RAM
- HugePage support
- A minimum of 1GiB of 2MiB-sized huge pages
- Instances of the Mayastor pod must be run in privileged mode.
- The iSCSI client should be installed and correctly configured as per this guide.
- Mayastor node candidates must be labeled with the OpenEBS engine type "mayastor". To add label, execute:
kubectl label node node1 openebs.io/engine=mayastor
Deploying Mayastor
We will be deploying Mayastor using the manifest files present in the deploy
folder of Mayastor GitHub repository. The repository is configured for the GitFlow release pattern, wherein the master branch contains official releases.
1. Create Mayastor Application Resources
- Namespace
kubectl create -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/namespace.yaml
- RBAC Resources
kubectl create -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/moac-rbac.yaml
- Custom Resource Definitions
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/csi/moac/crds/mayastorpool.yaml
2.Deploy Mayastor Dependencies
NATSkubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/nats-deployment.yamlTo verify the deployment of the NATS application to the cluster was successful, execute:
kubectl -n mayastor get pods --selector=app=natsSample Output:
NAME READY STATUS RESTARTS AGE
nats-b4cbb6c96-nbp75 1/1 Running 0 28s
3.Deploy Mayastor Components
1. CSI Node Pluginkubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/csi-daemonset.yamlTo verify that the CSI Node Plugin DaemonSet has been correctly deployed to all worker nodes in the cluster, execute:
kubectl -n mayastor get daemonset mayastor-csiSample Output:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
mayastor-csi 3 3 3 3 3 kubernetes.io/arch=amd64 26m
2. Control Plane
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/moac-deployment.yamlTo verify that the MOAC control plane pod is running, execute:
kubectl get pods -n mayastor --selector=app=moacSample Output:
NAME READY STATUS RESTARTS AGE
moac-7d487fd5b5-9hj62 3/3 Running 0 8m4s
3. Data Plane
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/mayastor-daemonset.yamlTo verify that the Mayastor DaemonSet has been correctly deployed, execute:
kubectl -n mayastor get daemonset mayastorSample Output:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
mayastor 3 3 3 3 3 kubernetes.io/arch=amd64,openebs.io/engine=mayastor 108s
For each resulting Mayastor pod instance, a Mayastor Node (MSN) custom resource definition should be created. List these definitions and verify that the count meets the expected number and that all nodes are reporting their State as online
kubectl -n mayastor get msnSample Output:
NAME STATE AGE
aks-agentpool-12194210-0 online 8m18s
aks-agentpool-12194210-1 online 8m19s
aks-agentpool-12194210-2 online 8m15s
Creating Storage Pool
We need at least one pool that has to be created and hosted by one of the Mayastor Nodes. While creating pools ensure that the number of pools configured should be no lower than the desired maximum replication factor of the PVs to be created, for example, the minimum viable configuration for a Mayastor deployment which is intended to test 3-way mirrored PVs must have three Mayastor Nodes, each having one Mayastor Pool, with each of those pools having one unique block device allocated to it.
For the current version of Mayastor, a pool may have only a single block device member, which constitutes the entire data persistence layer for that pool and thus determines its maximum capacity.
NOTE:Replace the values for the fields "name", "node" and "disks" as appropriate to your cluster's intended configuration.
To verify the status of the pool(s), execute:
kubectl -n mayastor get msp
Sample Output:
NAME NODE STATE AGE
pool-on-node-0 aks-agentpool-12194210-0 online 127m
pool-on-node-1 aks-agentpool-12194210-1 online 27s
pool-on-node-2 aks-agentpool-12194210-2 online 4s
Creating Storage Class
Mayastor dynamically provisions Persistent Volumes based on StorageClass definition. In the current version of Mayastor, StorageClass definitions are used to control both, which transport protocol is used to mount the PV to the worker node hosting the consuming application pod (iSCSI, or NVMe-oF TCP) and the level of data protection afforded to it (that is, the number of synchronous data replicas which are maintained, for purposes of redundancy). It is possible to create any number of custom StorageClass definitions to span this range of permutations. Execute the below example to deploy StorageClass
Set the value of protocol and name as per requirement. Permissible values for the field "protocol" are either "iscsi", or "nvmf"
Deploying a test application
Once StoragePool and StorageClass is deployed, you are all set to deploy an application and test Mayastor. To deploy an application, create a persistent volume claim.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ms-volume-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
storageClassName: mayastor-iscsi
Ensure the created PVC should is in bound state. To verify, execute:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ms-volume-claim Bound pvc-1dca40de-bc62-4da4-a85f-8cf6879849e8 2Gi RWO mayastor-iscsi 6d11h
In this example, we are deploying a MySQL application.
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql-test
spec:
replicas: 1
selector:
matchLabels:
app: mysql-test
template:
metadata:
labels:
app: mysql-test
spec:
containers:
- image: mysql:8.0.20
name: mysql
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
volumeMounts:
- mountPath: /var/lib/mysql
name: mysqlvol
volumes:
- name: mysqlvol
persistentVolumeClaim:
claimName: ms-volume-claim
Next, you need to create a secret for the above MySQL example,
kubectl create secret generic mysql-secret --from-literal=password=abc@123 -n mayastor
To deploy the application execute:
kubectl create -f mysql.yaml -n mayastor
To verify if the application is successfully deployed, execute:
kubectl get pods -n mayastor
Sample Output:
NAME READY STATUS RESTARTS AGE
mayastor-2xm4r 1/1 Running 13 50d
mayastor-bcwp7 1/1 Running 12 50d
mayastor-cr5w4 1/1 Running 13 50d
mayastor-csi-fqpm8 2/2 Running 12 50d
mayastor-csi-l28ws 2/2 Running 12 50d
mayastor-csi-s7455 2/2 Running 12 50d
moac-54d5db-mhtsg 3/3 Running 29 50d
mysql-test-59fd865cdd-5flgt 1/1 Running 0 6d11h
nats-6fdd6dfb4f-67xdc 1/1 Running 6 50d