-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sidecar Containers #753
Comments
@enisoc @dchen1107 @fejta @thockin @kow3ns @derekwaynecarr, opened this tracking issue so that we can discuss. |
/assign |
@derekwaynecarr I've done some scoping out of the kubelet changes required for next week's sig-node meeting, I believe that changes are only needed in the In In Let me know if you want me to investigate any further. |
@kow3ns The discussion makes more sense to me if maybe we can define a full description of containers sequence in Pod spec (sig-app), and how to handle the sequence in kubelet for start, restart and cascading consideration (sig-node). Let's catch the Feb 5 sig-node meeting to give more inputs. |
The proposal says that sidecars only run after the init containers run. But what if the use-case requires the sidecar to run while/before the init containers run. For example, if you'd like route the pod's traffic through a proxy running as a sidecar (as in Istio), you probably want that proxy to be in place while the init containers run in case the init container itself does network calls. |
@luksa I think there's the possibility of looking at having sidecars that run in init phase at some point but currently the proposal is not going to cover that use case. There is currently no way to have concurrent containers running in the init phase so that would be potentially a much larger/messier change than what is being suggested here. |
Update on this KEP: We still need to agree on the API, it seems there is consensus that a simple way of marking containers as sidecars is prefered over more in depth ordering flags. Having a bool is somewhat limiting though so perhaps something more along the lines of |
@Joseph-Irving Actually, neither the boolean nor the
as opposed to
Please forgive my bad naming - I hope the names convey the idea. But there is one problem with the approach where we introduce |
@luksa You could also solve the init problem by just allowing an init container to be marked as a sidecar and have that run alongside the init containers. As I understand it, the problem is that init containers sometimes need sidecars, which is different from needing a container that runs for the entire lifetime of the pod. The problem with |
@Joseph-Irving We could work with that yes. It's not ideal for the sidecar to shut down after the init containers run and then have the same sidecar start up again, but it's better than not having that option. The bigger problem is that older Kubelets wouldn't handle init-sidecar containers properly (as is the case with main-sidecar containers). I'd just like you to keep init-sidecars in mind when finalizing the proposal. In essence, you're introducing the concept of "sidecar" into k8s (previously, we basically only had a set of containers that were all equal). Now you're introducing actual sidecars, so IMHO, you really should think this out thoroughly and not dismiss a very important sidecar use-case. I'd be happy to help with implementing this. Without it, Istio can't provide its features to init containers (actually, in a properly secured Kubernetes cluster running Istio, init containers completely lose the ability to talk to any service). |
In relation to the implementation discussion in #841, I've opened a WIP PR containing a basic PoC for this proposal kubernetes/kubernetes#75099. It's just a first draft and obviously not perfect but the basic functionality works and gives you an idea of the amount of change required. cc @enisoc |
I put together a short video just showing how the PoC currently behaves https://youtu.be/4hC8t6_8bTs. Seeing it in action can be better than reading about it. |
I've opened two new PRs:
Any thoughts or suggestions will be much appreciated. |
this KEP will not progress in 1.30 and will stay in beta. As per per https://kubernetes.slack.com/archives/CPNHUMN74/p1706017979795519 @matthyx will fork out another KEP for improved Pods termination logic. @matthyx pls ping me when done so I can tag the forked KEP for 1.30 milestone |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
we're going GA for 1.32 |
We hit an issue with sidecars being a little undefined. Looking for guidence on what it the right behavior. When the normal containers exit, the sidecars are shot. In some cases, they then return exit codes != 0. At least one k8s client, https://k9scli.io/ considers a completed pod with exit codes other then 0 bad and marks it in its ui as RED. Its a bit unclear what the expectation around this is. Should it be documented that exit codes != 0 for sidecar containers are normal on main container exit and should be ignored? That would mean k9s's code should be adusted. If exit code 0 is expected, each sidecar that doesnt return 0 should be adjusted, and perhaps alternate shutdown signals should be supported to the sidecar for more "graceful shutdown" signaling? |
From kubernetes perspective, sidecars will receive the
Hiding or recommending to hide the bad exit of sidecar may hide real issues with it. We would really hope that sidecars will be implemented correctly and not get into this situation. So I want to make sure there are only a limited set of situations when sidecar will be marked as failed. Any additional comments on how sidecar was implemented, what graceful period was set, and how long container took to terminate, will help understand your use case better |
We already decided we're going to update the spiffe-helper to exit 0 on sigterm. it was exiting 2 on sigterm. But may be good to clarify this behavior for other users of the sidecar api so everyone knows what to expect. |
It makes sense. Please let us know if there are other cases when it will become failed unexpectedly. I wonder if this sidecar was used before the feature was implemented. I would think in the past, sidecar (implemented as a regular container) would also receive SIGTERM on |
Definitely we can benefit from updating this document mentioning this: https://kubernetes.io/docs/tutorials/configuration/pod-sidecar-containers/ |
Enhancement Description
k/enhancements
) update PR(s):k/k
) update PR(s):k/website
) update PR(s):k/enhancements
) update PR(s): KEP-753: add PRR questionnaire answers for beta #4255k/k
) update PR(s):k/website
) update(s): Document SidecarContainers graduation to beta website#43471/sig node
Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.
The text was updated successfully, but these errors were encountered: