Dockershim removal feedback & issues

[mrbobbytables]

Please use this issue for feedback and issues regarding dockershim removal in the 1.24 release.

/triage accepted
/milestone v1.24
/sig docs node
/area docker

[afbjorklund]

Please use this issue for feedback and issues regarding dockershim removal in the 1.24 release.

This information is confusing: https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/

"This section presents information you need to know when migrating from dockershim to other container runtimes."

Shouldn't it be "when migrating from dockershim to other CRI implementations for Docker" ?

i.e. going from dockershim to cri-dockerd

"It is recommended to migrate from dockershim to alternative container runtimes."

Shouldn't it be "it is recommended to migrate from dockerd to alternative container runtimes" ?

i.e. going from dockerd to containerd

---

We saw the same thing happen with CRI-O, before. As the saying goes: "now you have two problems".

It is especially confusing when the docs still installs containerd from docker.com instead of containerd.io.

Moving from dockershim to cri-dockerd should be mandatory, since CRI will be required in 1.24 and up.

Moving from dockerd to containerd (or cri-o) could still be optional, and requires more user intervention.

[neolit123]

A transition to cri-dockerd first might be smoother, indeed. There is especially a concern for business with a lot of clusters, with a lot of docker nodes. If they cannot move to alternative runtimes due to bugs, these bugs will have to be reported and resolved. Requiring things like a host OS update with the CR update is not great.

cri-dockerd is currently a third party project with an undefined maintenance state. Because of that it is difficult to maintain a k8s.io guide for it. The new guide for migrating to containerd seems like the best option for the time being.

At the same time we don't want users to be stuck with a 1.23 kubelet. Note: the k8s skew policy allows nodes to continue to have a 1.23 kubelet against a 1.24 and 1.25 kube-apiserver, but that would only delay the required changes.

https://kubernetes.io/releases/version-skew-policy/#kubelet

[afbjorklund]

If I understand correctly, the plan is to remove all support (for docker) from the documentation and from the supporting tools like kubeadm and leave users to make their own adventure (or to pay for Docker or Mirantis)

• No documentation on kubernetes.io
• No testing in kubernetes regression
• No binary packages available
• No support in kubeadm init

For minikube it seems like continuing with docker runtime is doable, even if it means having to build our own CRI binaries and do our own CRI configuration. We do have containerd available as opt-in, since a long time back.

• Use cri-dockerd instead of dockershim where needed minikube#12102
• Add cri-dockerd (and crictl) to installation minikube#12103
• consider using minikube image build GoogleContainerTools/skaffold#5739
• use minikube image load for containerd GoogleContainerTools/skaffold#5773

[neolit123]

If I understand correctly, the plan is to remove all support (for docker) from the documentation and from the supporting tools like kubeadm and leave users to make their own adventure (or to pay for Docker or Mirantis)

if cri-dockerd becomes a maintained and documented project, i don't see a problem for k8s.io to link and mention it as part of CR setup. https://kubernetes.io/docs/setup/production-environment/container-runtimes/

No documentation on kubernetes.io

cross-linking from k8s.io to cri-dockerd guides still make sense.

No testing in kubernetes regression

testing is difficult if the CRI implementation is external and unmaintained. if cri-dockerd ramp up on the maintenance i can see kubeadm adding back e2e tests for docker. we switched all test jobs to containerd for the time being.

No binary packages available

cri-dockerd need to start making releases and have a support matrix against the kubelet especially WRT the CRI API version that is in use by both.

No support in kubeadm init

kubeadm init|join allows you to pass any custom CRI socket via Init|JoinConfiguration.nodeRegistration.criSocket.
the CR will be treated as "remote" and configured (via a kubelet flags) similarly to how containerd | cri-o users, use kubeadm today. if docker.sock is present on the host and if cri-dockerd setups the legacy dockershim.sock, kubeadm would autodetect docker as the CR and start using it (we still have to make one minor change for kubeadm 1.24).

For minikube it seems like continuing with docker runtime is doable, even if it means having to build our own CRI binaries and do our own CRI configuration. We do have containerd available as opt-in, since a long time back

i would not recommend for minikube and users of k8s to start using cri-dockerd until the project is actually maintained, has docs and releases. if Mirantis cannot staff the project, they'd have to grant community ownership.

[afbjorklund]

I think Mirantis had plenty of time for that, with the delay... Don't think moving to 1.25 would have helped ?

Was referring to the auto-detection and the recommendations, it does work to use kubeadm with params

if cri-dockerd setups the legacy dockershim.sock

cri-dockerd now sets up /var/run/cri-docker.sock

EDIT: path changed to /var/run/cri-dockerd.sock

---

Users of minikube are free to choose their kubernetes version and the container runtime, using parameters..

If they want to stick with k8s 1.23, or move to containerd, they are free to do so. Otherwise, it is inevitable.
i.e. moving to version 1.24 and sticking with docker, without using cri-dockerd, is not going to be doable.

But it doesn't look to be too much worse than cri-o... Tested in another product only, scarce releases, etc.

[rphillips]

Linking my issue about local-up-cluster.sh not working nicely due to the dockershim removal: #106972

[jyotimahapatra]

@mrbobbytables 👋 I'm in the release 1.24 bug triage team. Since this issue is marked as a release blocker , want to find out if this is a placeholder issue? Let me know how we could track the progress as we go deeper into the release timeline.

[afbjorklund]

Currently there are no (working) upstream instructions for how to install the CRI for containerd and for docker.

There are inline instructions for containerd (downloading from docker.com, not from containerd.io) and for CRI-O.
But for cri-dockerd, there are no inline instructions and the upstream instructions have a fatal typo (and no packages)

It's not blocking, but it's not ideal. You would want the Kubernetes page to either have good info or link to good info ?

---

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ -->

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-runtime -->

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

• https://github.com/containerd/containerd
  https://containerd.io/docs/getting-started/

• https://github.com/cri-o/cri-o
  https://cri-o.io/

• https://github.com/Mirantis/cri-dockerd
  https://docs.mirantis.com (no info)

EDIT: I think it is a real risk, trying to use a corporate website for project documentation. Should have "cri-dockerd.io"
We already saw it happening with docs removed from docs.docker.com, it could happen also to docs.mirantis.com ?

• Add a user-facing documentation home page, for installing and configuring Mirantis/cri-dockerd#37

[afbjorklund]

Adding a flag to minikube, to make it easier to test the new CRI with any version of Kubernetes (since 1.20 or so):

• Add cri-service flag for overriding the default minikube#13600

minikube start --kubernetes-version=v1.24.0-alpha.2 --container-runtime=docker \
               --cri-socket=/var/run/cri-dockerd.sock --cri-service=cri-docker.service

Currently we are bundling the pre-release cri-dockerd, until there can be an official v0.2.1 with the socket fixes.

cri-dockerd v0.2.0-11-ga4d1895
(Mirantis/cri-dockerd@a4d1895)

• Release 0.2.1 with binary packages (deb / rpm) for end-user installation ? Mirantis/cri-dockerd#41

---

The default values for --cri-socket and --cri-service should otherwise work, matching the k8s version:

Kubernetes 1.23 and earlier: dockershim (internal)
Kubernetes 1.24 and later: cri-dockerd (external)

• Explain how to tell dockershim apart from cri-dockerd website#30974 (comment)

And yes, you can choose to run --container-runtime=containerd or --container-runtime=cri-o if you prefer.

They always run the external (remote) socket, since there was no internal "shim" for these two container runtimes.

[reylejano]

Linking the Docs Dockershim Removal project board
https://github.com/orgs/kubernetes/projects/67

[afbjorklund]

Looks like "kubenet" is still available in cri-dockerd, maybe spoke too soon after reading those other issues...

Perhaps it wouldn't be a bad idea to start using CNI anyway, and use a "bridge" example like those other CR ?

[JamesLaverack]

Hey all.

As we approach the release for Kubernetes 1.24 (scheduled just under a month away on 19th April 2022), we're keeping track of this issue in the release team. Currently this is tracked as a release blocker, but after discussions with the SIG Docs team (cc @nate-double-u) we're under the impression that the docs are in a good enough ™️ state to no longer be blocking.

I'd like to get community feedback on removing the release-blocker label from this issue, in particular:

• Are the docs in a state in which we'd be happy to release with, even if not everything was complete for the day?
• Is there anything else that we need to do for the Dockershim removal that would be a blocker for 1.24 if we didn't do it?

If people are happy, then I'll go ahead and remove the label.

[dims]

@justaugustus you had marked this release-blocker way back in December. Looks like we are in much better shape now. So we should remove the release-blocker tag. please revisit this when you get a chance (if you still think this is one).

@JamesLaverack +1 to remove release-blocker

[mrbobbytables]

I would also +1 removing release-blocker. The dockershim docs team has been doing a great job at shoring up the needed docs :)

[neolit123]

the decision seems up to release, but there are a number of open PRs (maybe some are not tracked in https://github.com/orgs/kubernetes/projects/67):

kubernetes/website#30882
kubernetes/website#32131

first one is rather important and has been in review since December.
without these merging prior to release there is potential for a flow of tickets to be logged in k/website and k/k.
as someone who does triage every week, i would like to avoid that...

[afbjorklund]

I will probably revert my change to stop using "kubenet", since it is still available (unlike indications to the contrary).
There are a lot of other flags removed from the kubelet that affects it, so it will need some code changes anyway*...

* those other code changes are not until 1.27 though

Hopefully we will have time to fix minikube before the beta release, so that it works (i.e. with docker) without them ?
The old code was relying on --network-plugin and --cni-conf-dir, both available as alpha features in cri-dockerd.

I think we can live with the other issues such as no release and no documentation, at least for 1.24 (maybe not 1.25)

But it seems like all warnings about trying to use it were warranted, and we better hurry up with the nerdctl tooling.

cri-dockerd is currently a third party project with an undefined maintenance state.

"Docker cri networking managed by network plugin kubernetes.io/no-op".

---

Plan B is to stick with the current k8s releases (1.23 and 1.24) and utilizing the version skew policy to get past 1.25.

So if you still require "docker" container runtime, you will get an older version of k8s when compared to "containerd".

[vsxen]

In addition, remove kubeadm check for socat & netfilter ?

	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
	[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

[neolit123]

@vsxen are you sure these are dockershim only?
See the containerd docs https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

Also socat and iptables forwarding are different things, AFAIK. Happy to cleanup some if these checks but we need proof.

[vsxen]

I only find socat in there

https://github.com/kubernetes/kubernetes/blob/v1.23.0/pkg/kubelet/dockershim/docker_streaming_others.go#L44

for br_ netfilter it depends on which cni plugin you use, for example some cni plugin (cilium ovs) don't use bridge.

[dims]

removing release-blocker

[neolit123]

@vsxen

I only find socat in there

We could move all dockershim dependencies to preflight warnings instead of errors in 1.24. We can also remove them, but cri-dockerd must document its dependencies first.

for br_ netfilter it depends on which cni plugin you use, for example some cni plugin (cilium ovs) don't use bridge.

This one is trickier. If some CNI (which ones? are they 'popular'?) need it, it might be better to continue throwing errors. Preflight errors can be ignored with the CLI flag / config.

[afbjorklund]

It seems the developer documentation for cri-dockerd gives just enough information to get started, but not to complete...

https://github.com/Mirantis/cri-dockerd#build-and-install

Missing some requirements like os version and go version, and missing the version linker flags that are provided by make

• Issue on building the code Mirantis/cri-dockerd#44

[ehashman]

/milestone clear
as this is not actionable for the 1.24 release

/lifecycle frozen
as we will want to continue collecting feedback for some time :)

[afbjorklund]

The docker runtime has now changed the default to "kubenet", which means that user needs to install CNI plugins.

NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: could not locate kubenet required CNI plugins [bridge host-local loopback] at [\"/opt/cni/bin\" \"/opt/cni/bin\"]

https://github.com/Mirantis/cri-dockerd/releases/tag/v0.2.4 (previously it would use the Docker network, by default)

[afbjorklund]

And now it is up to "cni" by default, and "no-op" by choice.

https://github.com/Mirantis/cri-dockerd/releases/tag/v0.2.5

[afbjorklund]

Now the docker removal is complete, since it was not supported in time for the (initial) 1.26 release.

It will probably be done retroactively in 1.27 timeframe, like how CNI was added retroactively with 1.25.
But the skeptics were right, and it would be better to move to the container runtime that is supported ?

Will look to nerdctl for picking up the ball, I would rather not resort to having to use the proprietary tools.

[dims]

It's been exactly a year. I am gonna close this now. please open fresh issues as needed in appropriate repositories.

/close

[k8s-ci-robot]

@dims: Closing this issue.

In response to this:

It's been exactly a year. I am gonna close this now. please open fresh issues as needed in appropriate repositories.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.