-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
With non-Calico CNI, Calico networkpolicy enforcement does not allow Terminating pods to gracefully shut down #4518
With non-Calico CNI, Calico networkpolicy enforcement does not allow Terminating pods to gracefully shut down #4518
Comments
FYI when uninstalling calico and re-creating all worker nodes, thereby cleaning all iptables rules, the problem does not occur. |
@fasaxc just merged a patch through to fix this here: projectcalico/libcalico-go#1397 Will be released in Calico v3.19.0, due in a couple of weeks. |
Hi @caseydavenport , thanks for the quick reply. Great to hear that this is a known issue and that a fix is underway! However, this is hitting us very hard. Do you think that perhaps this fix can be backported to 3.18? And if not, is there perhaps a previous version of calico that we can rollback to? |
Should be fine to backport - here's a PR for it. projectcalico/libcalico-go#1402 I think we will likely end up doing a patch release of v3.18 in a similar timeframe as v3.19 though - a week or two. |
I don't understand how this is a workaround? You start by uninstalling calico and then running a script. From the looks of it, this will leave clusters without policy enforcement, whatsoever. |
Once we applied calico on our cluster, we got errors when tried to redeploy
deployment. Calico 3.17 has the bug.
…On Tue, Apr 20, 2021, 10:36 Mitch Hulscher ***@***.***> wrote:
I don't understand how this is a workaround? You start by uninstalling
calico and then running a script. From the looks of it, this will leave
clusters without policy enforcement, whatsoever.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#4518 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEV5QKH42OAO5HEKVQD3TZDTJUVJFANCNFSM42SXQRXQ>
.
|
@eligosfx deleting Calico in that way will leave your cluster broken. In particular, you'll delete the IPAM tracking data so you may find that new pods get assigned IPs that belong to previously-networked pods. |
Oh, you're on EKS, then Calico isn't doing IPAM so you should be OK. |
Yes, I talked about a workaround for aws eks cluster.
…On Tue, Apr 20, 2021 at 3:35 PM Shaun Crampton ***@***.***> wrote:
Oh, you're on EKS, then Calico isn't doing IPAM so you should be OK.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4518 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEV5QKBHMZJMYVJBUA36HGLTJVYJVANCNFSM42SXQRXQ>
.
|
We've had to revert projectcalico/libcalico-go#1397; it caused a downstream problem that we don't fully understand yet. |
@fasaxc that is very unfortunate. Is the goal still to fix this issue in the next minor or patch release? Or should we expect it in a release after that? |
Azure/AKS#2223 documents a pretty good repro of this in AKS. when chained onto the azure cni. I'm confused if the revert https://github.com/projectcalico/libcalico-go/pull/1406/files might be a fix? If so taking 1.19 might be a soltuion here? |
@paulgmiller sadly that was reverting a candidate fix. The candidate fix broke a bunch of system tests in a non-obvious way |
Is there perhaps an updated ETA for a fix to this issue? |
I would also be keen to know if there is an ETA? |
To add an additional data point, we're also experiencing this issue as well. Calico version: |
We are also having this issue with
|
We have been testing various versions of calico and it seems that |
Just to let you know what's going on here; in v3.16 we fixed a different bug and inadvertently caused this one. The other bug was that pods that terminate without being deleted, such as Jobs, would still appear to Calico to have an IP address even though the CNI plugin had been called to remove the pod from the network. That was a security problem because Calico would think that the IP still belonged to the old Job pod even though it was now free and could be re-used by a new pod with different permissions. We closed that loophole by changing Calico CNI to add an annotation to the pod to mark whether it has an IP or not. However, you're not using Calico CNI so you don't get part of the fix; instead, with EKS CNI, we're checking for the annotation and not finding it and treating that as "no IP". It's not easy to fix because the k8s API doesn't cleanly expose the fact that a pod is "terminating but may be in its grace period" rather than "terminated and no longer owns its IP". |
After a lot of investigation we came up with the same observation: that a route to a Pod whose state is switching to Terminating is getting deleted right away - not respecting the Pod lifecycle. It's a significant impact .. but I am glad that reverting to 3.15.1 works (3.15.5 seem to be okay too). Thank you very much for giving this issue the attention it deserves as ongoing support for EKS using AWS-VPC-CNI is an important feature of Calico project. |
Safe to upgrade to 3.15.5
@fasaxc is the pod s[ec patched to remove PodIP after termination for jobs? Looks like yes pod ip is left in for job pods. Assuming phanse and container states aren't enough.
|
This one is being worked on as a high-priority issue - if anyone needs guidance in the meantime, please feel free to get in touch with me. |
Not sure if this KEP will help This is to expose a |
@nikskiz sadly not but we've got the fix in hand. We're just going to restore the old behaviour for non-Calico CNIs. The old behaviour was to use the "Phase" of the Pod to decide if its IP was legit. That works almost always but there was a bug in older k8s versions that meant that a pod could get stuck in the wrong phase for too long when it was terminating. |
Hi, is there a plan to backport this fix to older versions of Calico? AWS CNI comes with tigera operator thanks! |
@primeroz this is an AWS issue as they should never have moved off the v3.15 release family as this issue was known before the new Calico pattern was adopted. See the comments at the end of aws/amazon-vpc-cni-k8s#1541. |
thank you @stevehipwell I see there as well was mentioned that a backport has been done so i will wait for a release of the 3.17 series with the fix in and stick to the 3.15 i am currently running for now until that is released. thanks |
@primeroz we've stuck with the legacy Calico |
@stevehipwell great work and i agree on the 3.20 , just had no time to get to test it yet ( sweet sweet holidays ) I am not sure what is holding back the aws repo on that version, even for 1.9.0 ( recently released ) they still use the same operator version which has the bug in it. If you could report on your testing that would be extremely useful. |
I will do @primeroz, you might also be interested in tigera/operator#1428. FYI I have a custom Tigera Operator Helm chart due to the official chart lacking idiomatic Helm functionality. When I get some time I'll look at seeing if there is any interest in getting the changes into this repo. |
Calico v3.19.2 has now been released, which also has the fix. |
* Update Calico to 3.19.2 to fix projectcalico/calico#4518 * Update k8s api version for PodDisruptionBudget
I see 3.17.5 has been released ( and so is its tiger operator version ) was this backported to it ? |
Hi @primeroz, yes v3.17.5 and v3.18.5 both have the pod termination fix backported. |
Expected Behavior
When calico for networkpolicy enforcement is installed; pods in Terminating-state should keep their network access until containers are actually removed by the container-runtime.
Current Behavior
Pods that are in state Terminating immediately lose all network connectivity. Applications that are still handling in-flight network connections or applications that might want to reach out to the network for a graceful shutdown can not do so.
In our case, this is causing at least the following issues:
Possible Solution
Postpone iptables cleanup until pod/containers are actually removed.
Steps to Reproduce (for bugs)
kubectl delete pod <pod>
Below are the logs of aws-node running on the same node that hosted one of my debug pods. Of particular interest is the following line, which seems to suggest that all iptables rules for my pod are removed, even though the pod is still in Terminating state and cleaning up after itself.
Your Environment
v3.18.1
1.18
v20210302
v1.7.9
1.4.1
Context
debug.yaml
kubectl -n kube-system get ds aws-node -o yaml
kubectl -n kube-system get ds calico-node -o yaml
The text was updated successfully, but these errors were encountered: