OpenEBS storage engines cStor and Local PV Device make use of 'Active' and 'Unclaimed' BlockDevice custom resources to provision storage. Once the cStor Pool/Local PV Device PVs are removed, the cleanup-blockdevice-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx job pods restore the BlockDevices to 'Unclaimed' state. After the cleanup process, the BlockDevices are ready to be used.
Problem
- BlockDevice CLAIMSTATE stuck in 'Released' state.
root@node-0:~# kubectl get bd -n openebs NAME NODENAME SIZE CLAIMSTATE STATUS AGE blockdevice-1479e28ffc54c79d93646e4fc04a516c node-1 16002605645824 Released Active 5d21h blockdevice-3c18b8694a71607fc130d8359de7e952 node-2 16002605645824 Released Active 5d21h blockdevice-b75af7834b354b0eb3708dfe8250365a node-3 16002605645824 Unclaimed Active 5d21h - Newly created application pod stuck in 'Pending' state, because the PV and PVC were not created. This is because the the BlockDevice CR was not in 'Unclaimed' state.
root@node-0:~# kubectl get pods NAME READY STATUS RESTARTS AGE fio-pvc-statefulset-0 0/1 Pending 0 6m6s - BlockDeviceClaim custom resources from previously claimed BlockDevices not deleted, and in 'Pending' state.
root@node-0:~# kubectl get bdc -n openebs NAME BLOCKDEVICENAME PHASE AGE bdc-pvc-1ad7d91a-99b4-4194-9709-3a183f069727 Pending 22h bdc-pvc-a672f63a-49ff-4a67-a3fc-5616b30595c3 Pending 7m30s bdc-pvc-f2da37c9-a950-4bc0-bf56-e610a057bd3a Pending 21h cleanup-blockdevice-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxpods in the 'openebs' namespace inImagePullBackOffstate.root@node-0:~# kubectl get jobs -n openebs NAME COMPLETIONS DURATION AGE cleanup-blockdevice-1479e28ffc54c79d93646e4fc04a516c 0/1 39m 39m cleanup-blockdevice-3c18b8694a71607fc130d8359de7e952 0/1 39m 39mroot@node-0:~ # kubectl get pods -n openebs NAME READY STATUS RESTARTS AGE cleanup-blockdevice-1479e28ffc54c79d93646e4fc04a516c-rdh4f 0/1 ImagePullBackOff 0 38m cleanup-blockdevice-3c18b8694a71607fc130d8359de7e952-f64lc 0/1 ImagePullBackOff 0 38m
OpenEBS version
1.x, 2.x
Solution
The container image quay.io/openebs/linux-utils is necessary for the cleanup job pod to run to completion.
The image pull may be failing for your system because you may be using a private image registry, and the image is not available in the registry or image pulls from quay.io is restricted. To solve this problem, you will need to make this image available to your image registry.You can do this in one of two ways...
- ...manually pulling the image from quay.io and pushing it to your registry. E.g.: If you are using a docker registry you can use the following commands.
- Pull the image...
docker pull quay.io/openebs/linux-utils:<openebs-version>
root@node-0:~$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/openebs/linux-utils 2.1.0 73a52fde94a5 5 days ago 11MB - Tag the image using the IMAGE ID from the previous output. You may tag the image as you deem fit for your private image registry.
docker tag 73a52fde94a5 my-private-registry.io/openebs/linux-utils:<openebs-version>
- Push the image to you private registry...
docker push my-private-registry.io/openebs/linux-utils:<openebs-version>
- Pull the image...
- Alternatively, you may clone the repo at https://github.com/openebs/linux-utils and build the image yourself. You can then push this image to your private image registry.
After following these steps, the cleanup jobs will run to completion and the BlockDevices will be restored to 'Unclaimed' state.