Mayastor is currently under development as a storage engine option for the Open Source CNCF project, OpenEBS. It is actively developed and maintained by MayaData, as part of the Kubera platform for data agility.
Mayastor has been developed keeping in mind the following design goals:
- High availability
- Durable persistence of data
- Readily deployable
- Low-overhead abstraction
- Easy management
It has been designed from the ground up to leverage the protocol and compute efficiency of NVMe-oF semantics, and the performance capabilities of the latest generation of solid state storage devices, in order to deliver a storage abstraction with performance overhead measured to be within the range of single digit percentages.
Mayastor is beta software. It is considered largely, feature complete and substantially without major known defects. Minor and unknown defects can be expected.
Each Mayastor Node, that is each cluster worker node which will host an instance of a Mayastor pod, must have these resources free and available for use by Mayastor:
- Two x86-64 CPU cores with SSE4.2 instruction support:
- Intel Nehalem processor (march=nehalem) and newer
- AMD Bulldozer processor and newer
- 4GiB of RAM
- HugePage support
- A minimum of 1GiB of 2MiB-sized huge pages
- Instances of the Mayastor pod must be run in privileged mode.
- The iSCSI client should be installed and correctly configured as per this guide.
- Mayastor node candidates must be labeled with the OpenEBS engine type "mayastor". To add label, execute:
kubectl label node node1 openebs.io/engine=mayastor
We will be deploying Mayastor using the manifest files present in the
deploy folder of Mayastor GitHub repository. The repository is configured for the GitFlow release pattern, wherein the master branch contains official releases.
1. Create Mayastor Application Resources
kubectl create -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/namespace.yaml
- RBAC Resources
kubectl create -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/moac-rbac.yaml
- Custom Resource Definitions
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/csi/moac/crds/mayastorpool.yaml
2.Deploy Mayastor DependenciesNATS
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/nats-deployment.yamlTo verify the deployment of the NATS application to the cluster was successful, execute:
kubectl -n mayastor get pods --selector=app=natsSample Output:
NAME READY STATUS RESTARTS AGE nats-b4cbb6c96-nbp75 1/1 Running 0 28s
3.Deploy Mayastor Components1. CSI Node Plugin
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/csi-daemonset.yamlTo verify that the CSI Node Plugin DaemonSet has been correctly deployed to all worker nodes in the cluster, execute:
kubectl -n mayastor get daemonset mayastor-csiSample Output:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE mayastor-csi 3 3 3 3 3 kubernetes.io/arch=amd64 26m2. Control Plane
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/moac-deployment.yamlTo verify that the MOAC control plane pod is running, execute:
kubectl get pods -n mayastor --selector=app=moacSample Output:
NAME READY STATUS RESTARTS AGE moac-7d487fd5b5-9hj62 3/3 Running 0 8m4s3. Data Plane
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/mayastor-daemonset.yamlTo verify that the Mayastor DaemonSet has been correctly deployed, execute:
kubectl -n mayastor get daemonset mayastorSample Output:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE mayastor 3 3 3 3 3 kubernetes.io/arch=amd64,openebs.io/engine=mayastor 108sFor each resulting Mayastor pod instance, a Mayastor Node (MSN) custom resource definition should be created. List these definitions and verify that the count meets the expected number and that all nodes are reporting their State as
kubectl -n mayastor get msnSample Output:
NAME STATE AGE aks-agentpool-12194210-0 online 8m18s aks-agentpool-12194210-1 online 8m19s aks-agentpool-12194210-2 online 8m15s
Creating Storage Pool
We need at least one pool that has to be created and hosted by one of the Mayastor Nodes. While creating pools ensure that the number of pools configured should be no lower than the desired maximum replication factor of the PVs to be created, for example, the minimum viable configuration for a Mayastor deployment which is intended to test 3-way mirrored PVs must have three Mayastor Nodes, each having one Mayastor Pool, with each of those pools having one unique block device allocated to it.
For the current version of Mayastor, a pool may have only a single block device member, which constitutes the entire data persistence layer for that pool and thus determines its maximum capacity.
NOTE:Replace the values for the fields "name", "node" and "disks" as appropriate to your cluster's intended configuration.
To verify the status of the pool(s), execute:
kubectl -n mayastor get msp
NAME NODE STATE AGE pool-on-node-0 aks-agentpool-12194210-0 online 127m pool-on-node-1 aks-agentpool-12194210-1 online 27s pool-on-node-2 aks-agentpool-12194210-2 online 4s
Creating Storage Class
Mayastor dynamically provisions Persistent Volumes based on StorageClass definition. In the current version of Mayastor, StorageClass definitions are used to control both, which transport protocol is used to mount the PV to the worker node hosting the consuming application pod (iSCSI, or NVMe-oF TCP) and the level of data protection afforded to it (that is, the number of synchronous data replicas which are maintained, for purposes of redundancy). It is possible to create any number of custom StorageClass definitions to span this range of permutations. Execute the below example to deploy StorageClass
Set the value of protocol and name as per requirement. Permissible values for the field "protocol" are either "iscsi", or "nvmf"
Deploying a test application
Once StoragePool and StorageClass is deployed, you are all set to deploy an application and test Mayastor. To deploy an application, create a persistent volume claim.
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ms-volume-claim spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: mayastor-iscsi
Ensure the created PVC should is in bound state. To verify, execute:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ms-volume-claim Bound pvc-1dca40de-bc62-4da4-a85f-8cf6879849e8 2Gi RWO mayastor-iscsi 6d11h
In this example, we are deploying a MySQL application.
apiVersion: apps/v1 kind: Deployment metadata: name: mysql-test spec: replicas: 1 selector: matchLabels: app: mysql-test template: metadata: labels: app: mysql-test spec: containers: - image: mysql:8.0.20 name: mysql ports: - containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-secret key: password volumeMounts: - mountPath: /var/lib/mysql name: mysqlvol volumes: - name: mysqlvol persistentVolumeClaim: claimName: ms-volume-claim
Next, you need to create a secret for the above MySQL example,
kubectl create secret generic mysql-secret --from-literal=password=abc@123 -n mayastor
To deploy the application execute:
kubectl create -f mysql.yaml -n mayastor
To verify if the application is successfully deployed, execute:
kubectl get pods -n mayastor
NAME READY STATUS RESTARTS AGE mayastor-2xm4r 1/1 Running 13 50d mayastor-bcwp7 1/1 Running 12 50d mayastor-cr5w4 1/1 Running 13 50d mayastor-csi-fqpm8 2/2 Running 12 50d mayastor-csi-l28ws 2/2 Running 12 50d mayastor-csi-s7455 2/2 Running 12 50d moac-54d5db-mhtsg 3/3 Running 29 50d mysql-test-59fd865cdd-5flgt 1/1 Running 0 6d11h nats-6fdd6dfb4f-67xdc 1/1 Running 6 50d