nokmem

The node has sufficient disks, but it keeps reporting that the disk is insufficient to create a Pod.

Scope

When the node is used for a long time, it prompts an error that the remaining space is insufficient. The error message is as follows:

mkdir: cannot create directory '/sys/fs/cgroup/memory/8': No space left on device

The node disk is sufficient but reports this error, and the creation of Pod always fails. This is a potential “time bomb”.

All environments that use early-version kernels and Kubernetes versions before 1.21 will be affected. In runc 1.0.0-rc94 (opencontainers/runc#2840) it has been fixed (removed directly).

Prevention

• Upgrade the system kernel
• Kubernetes 1.14 to 1.20
  • Rebuild Kubelet with -tags=nokmem
• Kubernetes 1.14 or earlier
  • For hard coding, refer to nokmem.1.13.patch
• Kubernetes 1.21 or higher
  • not affected

Fixed by KLTS

• /docs/kubernetes/releases/v1.20/v1.20.15-lts.1/ nokmem.1.20.patch
• /docs/kubernetes/releases/v1.19/v1.19.16-lts.1/ nokmem.1.20.patch
• v1.18.20-lts.1 nokmem.1.20.patch
• v1.17.17-lts.1 nokmem.1.20.patch
• v1.16.15-lts.1 nokmem.1.20.patch
• v1.15.12-lts.1 nokmem.1.20.patch
• v1.14.10-lts.1 nokmem.1.20.patch
• v1.13.12-lts.1 nokmem.1.13.patch
• v1.12.10-lts.1 nokmem.1.13.patch
• v1.11.10-lts.1 nokmem.1.13.patch
• v1.10.13-lts.1 nokmem.1.13.patch