I wrote about the AWS resources created by OpenShift installer in my earlier post. In this post, let's take a closer look at IAM credentials used by OpenShift on AWS.

installation credentials

The simplest way to install OpenShift on AWS is to make use of the IPI (Installer Provisioned Infrastructure) method. With IPI, the installation process will provision a new VPC and other infrastructure resources, and then install OpenShift on top of it. It means that we need to use an account with the necessary privileges to create VPC, load balancers, etc for installation. We can use an adminstrator account for this, or, we can refer to the documentation for more specific permission requirements.

I also have a ready to use policy file here (for OpenShift 4.7).

Note that the credential used during installation is stored in your cluster as a Kubernetes secret named aws-creds in the kube-system namespace:

aws creds

For OpenShift on AWS, we can delete this secret post installation. However, it will need to be reinstated when we need to perform cluster upgrades, as documented here. Alternatively, we can go into the AWS console and deactivate the corresponding access key and re-activate it when you want to perform a cluster upgrade.

openshift created roles

As part of installation, OpenShift create roles for the bootstrap, master and worker nodes. The bootstrap role, and associated policy is only used during the installation, and are deleted when the installation completes.

The master role’s policy allows master nodes to configure load balancers as workload gets deployed/removed from the cluster, while the worker policy only allows the worker nodes to retrieve it’s own metadata, and region information.

These roles do not have any IAM permissions, and therefore are not able to create or modify access rights.

cloud credentials operator (cco)

Post installation, OpenShift interacts with the underlying cloud infrastructure provider to perform operations such as provisioning additional worker nodes, or configure load balancers as workloads get provisioned and orchestrated on the platform. To do this, OpenShift needs the necessary perimissions on the underlying infrastructure. The Cloud Credential Operator (CCO) is responsible for managing these credentials. The CCO operates on CredentialsRequest custom resource, and behaves differently based on the credentialMode setting.

cco modes

By default, CCO operates in the Mint mode. In this mode, CCO creates (mints) new credentials for components in the cluster. This allows it to create credentials with very specific permissions for the requesting component. However, this also means you need to provide an account with rights to create IAM users and roles to the CCO, which often raise concerns with enterprise security teams.

An alternative is the Manual mode, where the CCO takes a backseat and user has to manage all the different credentials used by OpenShift. For version 4.7 of OpenShift, there are 5 credentials to manage as described in my previous post.

A third, Passthrough mode, is available for OpenShift on AWS since version 4.5.8. In this mode, a single IAM credential with permissions to perform the different cluster functions is used for all components. Crucially, there are no permission to create or modify IAM configuration when using Passthrough mode, which should set security team at ease.

credentials request

CredentialsRequest is a custom Kubernetes resource (CRD) that describes the desired state of cloud credentials. Using the OpenShift command line client, we can obtain the credentialsRequest for the target cloud like so (version 4.7.2 on AWS):

oc adm release extract quay.io/openshift-release-dev/ocp-release:4.7.2-x86_64 --credentials-requests --cloud=aws

The result should be as follows:

---
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  annotations:
    exclude.release.openshift.io/internal-openshift-hosted: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
  name: cloud-credential-operator-iam-ro
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - action:
      - iam:GetUser
      - iam:GetUserPolicy
      - iam:ListAccessKeys
      effect: Allow
      resource: '*'
  secretRef:
    name: cloud-credential-operator-iam-ro-creds
    namespace: openshift-cloud-credential-operator
---
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
  labels:
    controller-tools.k8s.io: "1.0"
  name: openshift-image-registry
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - action:
      - s3:CreateBucket
      - s3:DeleteBucket
      - s3:PutBucketTagging
      - s3:GetBucketTagging
      - s3:PutBucketPublicAccessBlock
      - s3:GetBucketPublicAccessBlock
      - s3:PutEncryptionConfiguration
      - s3:GetEncryptionConfiguration
      - s3:PutLifecycleConfiguration
      - s3:GetLifecycleConfiguration
      - s3:GetBucketLocation
      - s3:ListBucket
      - s3:GetObject
      - s3:PutObject
      - s3:DeleteObject
      - s3:ListBucketMultipartUploads
      - s3:AbortMultipartUpload
      - s3:ListMultipartUploadParts
      effect: Allow
      resource: '*'
  secretRef:
    name: installer-cloud-credentials
    namespace: openshift-image-registry
---
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
  labels:
    controller-tools.k8s.io: "1.0"
  name: openshift-ingress
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - action:
      - elasticloadbalancing:DescribeLoadBalancers
      - route53:ListHostedZones
      - route53:ChangeResourceRecordSets
      - tag:GetResources
      effect: Allow
      resource: '*'
  secretRef:
    name: cloud-credentials
    namespace: openshift-ingress-operator
---
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
  name: aws-ebs-csi-driver-operator
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - action:
      - ec2:AttachVolume
      - ec2:CreateSnapshot
      - ec2:CreateTags
      - ec2:CreateVolume
      - ec2:DeleteSnapshot
      - ec2:DeleteTags
      - ec2:DeleteVolume
      - ec2:DescribeInstances
      - ec2:DescribeSnapshots
      - ec2:DescribeTags
      - ec2:DescribeVolumes
      - ec2:DescribeVolumesModifications
      - ec2:DetachVolume
      - ec2:ModifyVolume
      effect: Allow
      resource: '*'
  secretRef:
    name: ebs-cloud-credentials
    namespace: openshift-cluster-csi-drivers
---
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  annotations:
    exclude.release.openshift.io/internal-openshift-hosted: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
  labels:
    controller-tools.k8s.io: "1.0"
  name: openshift-machine-api-aws
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - action:
      - ec2:CreateTags
      - ec2:DescribeAvailabilityZones
      - ec2:DescribeDhcpOptions
      - ec2:DescribeImages
      - ec2:DescribeInstances
      - ec2:DescribeSecurityGroups
      - ec2:DescribeSubnets
      - ec2:DescribeVpcs
      - ec2:RunInstances
      - ec2:TerminateInstances
      - elasticloadbalancing:DescribeLoadBalancers
      - elasticloadbalancing:DescribeTargetGroups
      - elasticloadbalancing:RegisterInstancesWithLoadBalancer
      - elasticloadbalancing:RegisterTargets
      - iam:PassRole
      - iam:CreateServiceLinkedRole
      effect: Allow
      resource: '*'
    - action:
      - kms:Decrypt
      - kms:Encrypt
      - kms:GenerateDataKey
      - kms:GenerateDataKeyWithoutPlainText
      - kms:DescribeKey
      effect: Allow
      resource: '*'
    - action:
      - kms:RevokeGrant
      - kms:CreateGrant
      - kms:ListGrants
      effect: Allow
      policyCondition:
        Bool:
          kms:GrantIsForAWSResource: true
      resource: '*'
  secretRef:
    name: aws-cloud-credentials
    namespace: openshift-machine-api

You’ll see that five(5) credentialsRequest will be created for OpenShift on AWS.

mint mode

When operating in Mint (the default) mode, the Cloud Credentials Operator creates an IAM user of each credentialRequest.

In addition, a Kubernetes secret containing the AWS access keys will be created and managed by the operator based on the spec/secretRef section of the credentialsRequest.

To illustrate, the following users are created when I install OpenShift using Mint mode:

users

Let’s examine the credentialsRequest for the ingress operator:

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
  labels:
    controller-tools.k8s.io: "1.0"
  name: openshift-ingress
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries: (1)
    - action:
      - elasticloadbalancing:DescribeLoadBalancers
      - route53:ListHostedZones
      - route53:ChangeResourceRecordSets
      - tag:GetResources
      effect: Allow
      resource: '*'
  secretRef: (2)
    name: cloud-credentials
    namespace: openshift-ingress-operator
1 specifies the required permission, this will be defined as inline policy for the IAM user
2 specifies the namespace and name of the corresponding secret that will contain the access keys to this user

manual mode

In Manual mode, the user manages the cloud credentials instead of the Cloud Credentials Operator. This means we need to create the IAM users with the permissions as specified in the credentialsRequest above, along with the access keys and secrets during installation by following the instructions here.

We also should reconcile the permissions with credentialsRequests of target versions before performing cluster upgrades.

passthrough mode

In Passthrough mode, the Cloud Credentials Operator does not create IAM users. Instead, the secrets contains the same AWS access keys used during installation. To rotate the access key, we update the aws-creds secret in the kube-system namespace. The CCO will they sync the other credential secrets with the same access keys specified in the aws-creds.

This implies that all the different components will be using the same credentials. Hence, the supplied credentials needs to have all the permissions requested by all the credentialsRequest.

Note also that by default, kube-system/aws-creds contains the access keys used during installation. This contains permissions that are not required during normal operations in Passthrough mode, such as creating IAM users. To mitigate this, we can create 2 separate policies for installation and operations.

Attach both policies to the user during installation like so:

dual policy

Once installation is complete, remove the openshift-install-policy from the user.

Alternatively, we can create a separate IAM user for operation, and change the kube-system/aws-creds secret to use the operation account post installation.

In my (albeit limited) testing, we can perform cluster upgrades with just the operations policy attached with Passthrough mode, although it is prudent to review the credentialsRequest for the target version for any change in permissions.

which option’s for me?

So which option should we take?

Mint mode offers the most convenience from operations point of view. However, I recommend that you delete/deactivate the kube-system/aws-cred access keys post installation, and only re-instating them when performing cluster upgrades.

Manual and Passthrough modes are suitable for organizations that have an established process of managing IAM credentials. In this case, I suggest extending that process to include creating/rotating of credentials secrets. When using Passthrough mode, it’s a good idea to remove the IAM creation permissions after installation, as described above.

Finally, do consider configuring alerts to detect any unplanned changes to IAM configuration.

what’s next?

We’ll be able to use short lived tokens instead of access keys soon with manual mode using STS (Secure Token Service). This should improve the security posture and is currently in Technology Preview. I’ll look into this operating mode in a future article.