Hi all,
below is a small engineering puzzle that I had to solve recently. The essential components:
- a Linux Laptop (in my case, running the excellent ClearLinux distribution)
- aws-iam-authenticator
- Outline client (A shadowSocks client)
The setup was the following. A Kubernetes cluster, a bastion host using Outline as the means to connect and access the cluster. In the ~/.kube/config you can see the following stanza:
name: k8s.dev.SNIPPED
user:
exec: apiVersion: client.authentication.k8s.io/v1alpha1 args: [“token”, “–cache”, “-i”, “k8s.dev.SNIPPED”] command: aws-iam-authenticator
[SNIPPED]
Issuing commands such as kubectl get pods would fail, with a DNS resolution error Outline Client was enabled. The root cause for this was that our setup, UDP traffic was disabled over Outline. However, Outline would take over /etc/resolv.conf and add a options use-vc line, indicating that ALL DNS resolutions should happen over TCP.
aws-iam-authenticator communicates under the hood with https://sts.amazonaws.com and attempts to resolve this hostname using UDP. This does not play well with the existing Outline Client setup and eventually will fail with an i/o timeout along the lines of 10.0.85.1:highport -> 1.1.1.1:53:udp.
The easiest way I have found to fix this was the following: modify the routing table AFTER Outline client takes over. For my home network this can look along the lines of:
sudo route add -host 1.1.1.1 gw fritz.box wlp2s0
and Presto! DNS resolution works again for aws-iam-authenticator and kubectl workflow can proceed as normal. I tried experimenting with
export GODEBUG=netdns=cgo
export GODEBUG=netdns=go
but with both flavors of the resolver, it did not honor the options-vc.
Hope this is helpful to other people! Until next time!
Athanasios Kostopoulos 