How do I resolve permission issues for Amazon EKS add-on manager when I install the EBS CSI driver?

Short description

If the Amazon EKS add-on manager permissions are misconfigured, then you might receive the following error message when you try to install the Amazon EBS CSI driver:

"Namespaces "kube-system" is forbidden: User "eks:addon-manager" cannot patch resource "namespaces" in API group "" in the namespace "kube-system"."

You receive this error message when the eks:addon-cluster-admin ClusterRoleBinding is missing. Without this binding, your Amazon EKS cluster continues to operate but  Amazon EKS can't manage any add-ons. For more information, see Considerations.

Resolution

It's a best practice to check all associated role-based access control (RBAC) roles and bindings.

The following are the cluster roles and their permissions.

For ClusterRole cluster-admin:

PolicyRule:

Resources  Non-Resource URLs Resource Names Verbs
 --------- ----------------- -------------- -----
    *.*            []               []       [*]
                   [*]              []       [*]

For ClusterRole eks-addon-manager:

PolicyRule:

  Resources                                              Non-Resource URLs  Resource Names                      Verbs
  ---------                                              -----------------  --------------                      -----
  customresourcedefinitions.apiextensions.k8s.io         []                 [eniconfigs.crd.k8s.amazonaws.com]  [create delete get list patch update watch]
  clusterrolebindings.rbac.authorization.k8s.io          []                 [aws-node]                          [create delete get list patch update watch]
  clusterrolebindings.rbac.authorization.k8s.io          []                 [ebs-csi-attacher-binding]          [create delete get list patch update watch]
  clusterrolebindings.rbac.authorization.k8s.io          []                 [ebs-csi-node-binding]              [create delete get list patch update watch]
  clusterrolebindings.rbac.authorization.k8s.io          []                 [ebs-csi-node-getter-binding]       [create delete get list patch update watch]
  clusterrolebindings.rbac.authorization.k8s.io          []                 [ebs-csi-provisioner-binding]       [create delete get list patch update watch]
  clusterrolebindings.rbac.authorization.k8s.io          []                 [ebs-csi-resizer-binding]           [create delete get list patch update watch]
  clusterrolebindings.rbac.authorization.k8s.io          []                 [ebs-csi-snapshotter-binding]       [create delete get list patch update watch]
  clusterrolebindings.rbac.authorization.k8s.io          []                 [eks:kube-proxy]                    [create delete get list patch update watch]
  clusterrolebindings.rbac.authorization.k8s.io          []                 [system:coredns]                    [create delete get list patch update watch]
  clusterroles.rbac.authorization.k8s.io                 []                 [aws-node]                          [create delete get list patch update watch]
  clusterroles.rbac.authorization.k8s.io                 []                 [ebs-csi-node-role]                 [create delete get list patch update watch]
  clusterroles.rbac.authorization.k8s.io                 []                 [ebs-external-attacher-role]        [create delete get list patch update watch]
  clusterroles.rbac.authorization.k8s.io                 []                 [ebs-external-provisioner-role]     [create delete get list patch update watch]
  clusterroles.rbac.authorization.k8s.io                 []                 [ebs-external-resizer-role]         [create delete get list patch update watch]
  clusterroles.rbac.authorization.k8s.io                 []                 [ebs-external-snapshotter-role]     [create delete get list patch update watch]
  clusterroles.rbac.authorization.k8s.io                 []                 [system:coredns]                    [create delete get list patch update watch]
  csidrivers.storage.k8s.io                              []                 [ebs.csi.aws.com]                   [create delete get list patch update watch]
  persistentvolumes                                      []                 []                                  [create delete get list watch update patch]
  volumesnapshotcontents.snapshot.storage.k8s.io         []                 []                                  [create get list watch update delete patch]
  nodes                                                  []                 []                                  [get list update watch]
  volumeattachments.storage.k8s.io                       []                 []                                  [get list watch update patch]
  persistentvolumeclaims                                 []                 []                                  [get list watch update]
  namespaces                                             []                 []                                  [get list watch]
  pods                                                   []                 []                                  [get list watch]
  replicasets.apps                                       []                 []                                  [get list watch]
  eniconfigs.crd.k8s.amazonaws.com                       []                 []                                  [get list watch]
  csinodeinfos.csi.storage.k8s.io                        []                 []                                  [get list watch]
  volumesnapshotclasses.snapshot.storage.k8s.io          []                 []                                  [get list watch]
  csinodes.storage.k8s.io                                []                 []                                  [get list watch]
  storageclasses.storage.k8s.io                          []                 []                                  [get list watch]
  secrets                                                []                 []                                  [get list]
  volumesnapshots.snapshot.storage.k8s.io                []                 []                                  [get list]
  leases.coordination.k8s.io                             []                 []                                  [get watch list delete update create]
  events                                                 []                 []                                  [list watch create patch update]
  events.events.k8s.io                                   []                 []                                  [list watch create patch update]
  endpoints                                              []                 []                                  [list watch]
  services                                               []                 []                                  [list watch]
  endpointslices.discovery.k8s.io                        []                 []                                  [list watch]
  *.extensions                                           []                 []                                  [list watch]
  volumeattachments.storage.k8s.io/status                []                 []                                  [patch]
  persistentvolumeclaims/status                          []                 []                                  [update patch]
  volumesnapshotcontents.snapshot.storage.k8s.io/status  []                 []                                  [update]

Complete the following steps:

1. Confirm whether the required ClusterRole and ClusterRoleBinding are present. Run the following command:

   kubectl get clusterrole,clusterrolebinding | grep addon

   Output:

   clusterrole.rbac.authorization.k8s.io/eks:addon-manager
   clusterrolebinding.rbac.authorization.k8s.io/eks:addon-cluster-admin
   clusterrolebinding.rbac.authorization.k8s.io/eks:addon-manager

2. If the eks:addon-cluster-admin role is missing, use the following .yaml manifest to create it:

   cat << EOF > ClusterRoleBinding.yaml
   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: eks:addon-cluster-admin
   subjects:
   - kind: User
     name: eks:addon-manager
     apiGroup: rbac.authorization.k8s.io
   roleRef:
     kind: ClusterRole
     name: cluster-admin
     apiGroup: rbac.authorization.k8s.io
   EOF
   
   kubectl apply -f ClusterRoleBinding.yaml