AWS EKS cluster authentication Unauthorized

Scenario: After you created a EKS cluster with worker nodes using Terraform. You are not able to connect to the cluster endpoint. The cluster was created from an account which has admin rights to assume roles in other accounts. We confirmed the authorized roles and policies are correct. This terraform was used multiple times to create a dev cluster, however this time we are creating the cluster in a different account.
It seems like the EKS is not able to validate the user, or we are not able to send the right user/credential to the cluster. So its not an authorization issue but an authentication problem.
So, we have explore more into authentication modules rather then EKS role based access control (RBAC)

One of the first thing you can do is enable cluster auth logs and see which user cloudwatch is capturing as access denied.

Errors:

pankaj@PGautam-X280:~/terraform/eks3-thar$ kubectl versionClient Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
error: You must be logged in to the server (the server has asked for the client to provide credentials)


pankaj@PGautam-X280:~/terraform/eks3-thar$ kubectl get nodes
error: the server doesn't have a resource type "nodes"

pankaj@PGautam-X280:~/terraform/eks3-thar$ kubectl version --v=7
I1030 15:55:05.651723   23213 loader.go:359] Config loaded from file /home/pankaj/.kube/config
I1030 15:55:05.653152   23213 round_trippers.go:383] GET https://A4A4CXXX0XXXXXXXE.gr7.us-west-2.eks.amazonaws.com/version?timeout=32s
I1030 15:55:05.653363   23213 round_trippers.go:390] Request Headers:
I1030 15:55:05.653585   23213 round_trippers.go:393]     Accept: application/json, */*
I1030 15:55:05.653855   23213 round_trippers.go:393]     User-Agent: kubectl/v1.11.2 (linux/amd64) kubernetes/bb9ffb1
I1030 15:55:06.023642   23213 round_trippers.go:408] Response Status: 401 Unauthorized in 369 milliseconds
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
I1030 15:55:06.035173   23213 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server has asked for the client to provide credentials",
  "reason": "Unauthorized",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "unknown"
      }
    ]
  },
  "code": 401
}]
F1030 15:55:06.036149   23213 helpers.go:119]
error: You must be logged in to the server (the server has asked for the client to provide credentials)

=========================================
Solution:

Cluster is active
pankaj@PGautam-X280:~/terraform/eks3-thar$ aws eks --profile ops --region us-west-2 describe-cluster --name eks-dev-cluster-1 --query cluster.status
"ACTIVE"

Heptio token works:
pankaj@PGautam-X280:~/terraform/eks3-thar$ aws-iam-authenticator token -i eks-dev-cluster-1
{"kind":"ExecCredential","apiVersion":"client.authentication.k8s.io/v1alpha1","spec":{},"status":{"token":"k8s-aws-v1.aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhbGxlcklkZ
W50aXR5JlZlcnNpb249MjAxMS0wNi0xNSZYLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNI
QTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFaUkhORE9GVzJPM0ZXUjdIJTJGMjAxOTEw
N0JTNCeC1rOHMtYXdzLWlkJlgtQW16LVNpZ25hdHVyZT1jZjk4NTg5MDI2ZjA2MjMyM2Y5Njh
iNDljMzI2YWZmYmI5MDcxNzBlMTUxZDNlN2QxYzA0NGFiOWE0MDcxNTc3"}}

Update .kube config with env:
pankaj@PGautam-X280:~/terraform/eks3-thar$ echo $KUBECONFIG
/home/pankaj/terraform/eks3-thar/config

- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      env:
      - name: AWS_PROFILE
        value: opssbx
      args:
        - "token"
        - "-i"
        - "eks-dev-cluster-1"


pankaj@PGautam-X280:~/terraform/eks3-thar$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-eks-5047ed", GitCommit:"5047edce664593832e9b889e447ac75ab104f527", GitTreeState:"clean", BuildDate:"2019-08-21T22:32:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}