Problem with accessibility to gke control plane via private IP

[Antuu]

Hi,
We are switching from using Public Control Plane Endpoint to private one. From time to time there is issue with timeout. We are connecting through VPN solution but there is also issue with connectivity from another clusters(argo and spinnaker) with the same VPC. Also Connectivity Test show error:

Even if Control plane global access is enabled.

[Antuu]

Also there is a lot of errors in Kubernetes Control Plane logs:

[Marvin_Lucero]

Hi @Antuu ,

I am not sure of the details of your setup, but, you can check the following factors affecting connectivity with your GKE clusters:

1. If you're using VPC peering to connect clusters in different regions, make sure that the peering connections are properly set up and functioning. You can check the VPC Peering status in your Cloud Console.

2. Check if there are any firewall rules that might be blocking traffic to the private control plane endpoint. Make sure the ports needed are opened. 

3. Check the routes of your Cloud Router if they are propagated, especially if you are using it for dynamic routing. 

These basic troubleshooting steps or checking that I have provided to you are just a general approach. I suggest to coordinate the case with Google Cloud Support since they can further check your project via creating a support case.

[Antuu]

Hi @Marvin_Lucero,
1. I am not using peering- both clusters as in the same VPC.

2. There is no firewall blocking network traffic. As I checked I can access cluster A from cluster B pod but issue occurs when there is a lot of requests(ArgoCD). also sometimes even single requests are getting timeout.

3. Everything looks fine here.

Issue looks like sometime there is too much traffic to this private endpoint and control plane cannot handle it correctly because it is overwhelmed.
Looks similar to:
https://www.googlecloudcommunity.com/gc/Google-Kubernetes-Engine-GKE/Problem-with-accessibility-to-g...