Topics covered in this article:
Problem Statement
User is unable to create cStor storage pool on 1.0.0 OpenEBS version. The user was on 0.9 and deletion of old openebs cluster was performed and then installed openebs 1.0.0 version on the same Kubernetes cluster.
OpenEBS version
1.0.0
OpenEBS Storage engine
cStor
Symptoms
After applying StoragePoolClaim(SPC) YAML configuration, SPC has been created but pools are not created. Output of kubectl get csp is showing nothing.
SPC configuration details can be obtained by running the following command.
kubectl get spc
Example Output:
NAME AGE
cstor-pool-mirror 5s
cStor Storage pool details can be obtained by running following command.
kubectl get csp
Example output
No resources found.
Following are the logs from maya-apiserver pod.
Pool provisioning failed for 2/2 for storagepoolclaim cstor-pool-mirror: failed to build cas pool for spc cstor-pool-mirror: aborting storagepool create operation as no claimed block devices available: pending block device claim count 2 on node {ip-172-20-75-90.ec2.internal}
While checking BDC specification of one of the blockdevice, it was shown Status.Phase as empty.
Following is a snippet of BDC spec for one of the blockdevice.
Spec:
Block Device Name: blockdevice-4028d68186df0ccec5ccae123c0aa5ee
Device Claim Details:
Device Type:
Host Name: ip-172-20-75-90.ec2.internal
Resources:
Requests:
Storage: 1T
Status:
Phase:
Events: <none>
If the operator is running, it will be either marked as Pending or Bound. Since the phase was empty, the user needs to check the openebs-ndm-operator logs also.
Following is the log snippet from openebs-ndm-operator pod.
{"level":"info","ts":1563807730.730718,"logger":"ndm-operator","msg":"Go Version: go1.10.3"}
{"level":"info","ts":1563807730.7307622,"logger":"ndm-operator","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1563807730.7307794,"logger":"ndm-operator","msg":"operator-sdk Version: v0.5.0"}
{"level":"info","ts":1563807730.7317438,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1563807734.2607658,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1563807734.2981257,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1563807734.6273496,"logger":"ndm-operator","msg":"Registering Components"}
{"level":"info","ts":1563807734.6276774,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"blockdevice-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1563807734.627876,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"blockdeviceclaim-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1563807734.6280491,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"disk-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1563807734.628184,"logger":"ndm-operator","msg":"Starting the ndm-operator..."}
E0722 15:02:14.703718 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdd"},"stats":{"deviceUtilizat|...
E0722 15:02:15.798626 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdd"},"stats":{"deviceUtilizat|...
E0722 15:02:16.843295 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext4","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext4","path":"/dev/xvdbf"},"stats":{"deviceUtiliza|...
E0722 15:02:18.123415 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdc"},"stats":{"deviceUtilizat|...
E0722 15:02:19.207698 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdd"},"stats":{"deviceUtilizat|...
E0722 15:02:20.304375 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext4","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext4","path":"/dev/xvdbv"},"stats":{"deviceUtiliza|...
E0722 15:02:21.407459 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext4","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext4","path":"/dev/xvdbf"},"stats":{"deviceUtiliza|...
E0722 15:02:22.431901 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdd"},"stats":{"deviceUtilizat|...
E0722 15:02:23.476962 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext4","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext4","path":"/dev/xvdbf"},"stats":{"deviceUtiliza|...
E0722 15:02:24.563248 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdd"},"stats":{"deviceUtilizat|...
E0722 15:02:25.661309 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext4","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext4","path":"/dev/xvdbv"},"stats":{"deviceUtiliza|...
E0722 15:02:26.693158 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdd"},"stats":{"deviceUtilizat|...
E0722 15:02:27.766808 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdd"},"stats":{"deviceUtilizat|...
E0722 15:02:28.966949 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdd"},"stats":{"deviceUtilizat|...
E0722 15:02:30.042117 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdd"},"stats":{"deviceUtilizat|...
E0722 15:02:31.141305 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.Disk: v1alpha1.DiskList.Items: []v1alpha1.Disk: v1alpha1.Disk.Spec: v1alpha1.DiskSpec.FileSystem: readObjectStart: expect { or n, but found ", error found in #10 byte of ...|eSystem":"ext3","pat|..., bigger context ...|ionRate":0,"serial":"","vendor":""},"fileSystem":"ext3","path":"/dev/xvdc"},"stats":{"deviceUtilizat|...
In the above output, the NDM operator tried to parse the old disk CRs and it was failed continuously.
Possible Reason
There could a chance of improper deletion of the old cluster. Due to ungraceful deletion of 0.9.0 version, deletion of disk CRDs went unsuccessful. After installing 1.0.0, NDM operator tried to parse the old disk CRs and it failed continuously. In 1.0.0 version, Disk schema has been changed. It has introduced blockdevice CR.
Solution
User should delete all the disk CRs and delete all openebs-ndm damemonset pods and it will create new openebs-ndm daemonset pods which will create new Disk CRs and remove stale Disk CRs. This can be done in the following way.
Deletion of disk CRs can be done by the following command.
kubectl delete disk --all
Deletion of openebs-ndm dameonset pods can be done by the following command.
kubectl delete pod openebs-ndm-<pod_template_hash> -n <openebs_installed_namespace>
After running above command, new openebs-ndm daemonset pods will be created and cStor pools also will be created as per the required configuration.