An example of ClusterAuthorizationRule
apiVersion: deckhouse.io/v1
kind: ClusterAuthorizationRule
metadata:
name: test-rule
spec:
subjects:
- kind: User
name: some@example.com
- kind: ServiceAccount
name: gitlab-runner-deploy
namespace: d8-service-accounts
- kind: Group
name: some-group-name
accessLevel: PrivilegedUser
portForwarding: true
# This option is only available if the enableMultiTenancy parameter is set (Enterprise Edition version)
allowAccessToSystemNamespaces: false
# This option is only available if the enableMultiTenancy parameter is set (Enterprise Edition version)
namespaceSelector:
labelSelector:
matchExpressions:
- key: stage
operator: In
vaules:
- test
- review
matchLabels:
team: frontend
Creating a user
There are two types of users in Kubernetes:
- Service accounts managed by Kubernetes via the API;
- Regular users managed by some external tool that the cluster administrator configures. There are many authentication mechanisms and, accordingly, many ways to create users. Currently, two authentication methods are supported:
- Via the user-authn module.
- Via the certificates.
When issuing the authentication certificate, you need to specify the name (CN=<name>
), the required number of groups (O=<group>
), and sign it using the root CA of the cluster. It is this mechanism that authenticates you in the cluster when, for example, you use kubectl on a bastion node.
Creating a ServiceAccount for a machine and granting it access
It may be required to give your machine static access to the Kubernetes API, e.g., for deploying applications from a CI system runner.
-
Create a
ServiceAccount
in thed8-service-accounts
namespace (you can change the name):kubectl create -f - <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: gitlab-runner-deploy namespace: d8-service-accounts --- apiVersion: v1 kind: Secret metadata: name: gitlab-runner-deploy-token namespace: d8-service-accounts annotations: kubernetes.io/service-account.name: gitlab-runner-deploy type: kubernetes.io/service-account-token EOF
-
Grant the necessary rights to the
ServiceAccount
(using the ClusterAuthorizationRule custom resource):kubectl create -f - <<EOF apiVersion: deckhouse.io/v1 kind: ClusterAuthorizationRule metadata: name: gitlab-runner-deploy spec: subjects: - kind: ServiceAccount name: gitlab-runner-deploy namespace: d8-service-accounts accessLevel: SuperAdmin # This option is only available if the enableMultiTenancy parameter is set (Enterprise Edition version) allowAccessToSystemNamespaces: true EOF
If the multitenancy mode is enabled in the Deckhouse configuration, you need to specify the
allowAccessToSystemNamespaces: true
parameter to give the ServiceAccount access to the system namespaces. -
Generate a
kube-config
(don’t forget to substitute your values):cluster_name=my-cluster user_name=gitlab-runner-deploy.my-cluster context_name=${cluster_name}-${user_name} file_name=kube.config
- The
cluster
section:- If there is direct access to the API server, then use its IP address:
-
Get the CA of our Kubernetes cluster:
kubectl get cm kube-root-ca.crt -o jsonpath='{ .data.ca\.crt }' > /tmp/ca.crt
-
Generate a section using the API server’s IP:
kubectl config set-cluster $cluster_name --embed-certs=true \ --server=https://$(kubectl get ep kubernetes -o json | jq -rc '.subsets[0] | "\(.addresses[0].ip):\(.ports[0].port)"') \ --certificate-authority=/tmp/ca.crt \ --kubeconfig=$file_name
-
-
If there is no direct access to the API server, enable the
publishAPI
parameter containing thewhitelistSourceRanges
array. Or you can do that via a separate Ingress-controller using theingressClass
option with the finiteSourceRange
. That is, specify the requests’ source addresses in theacceptRequestsFrom
controller parameter. -
If the CA is non-public:
-
Get the CA from the secret containing the
api.%s
domain’s certificate:kubectl -n d8-user-authn get secrets -o json \ $(kubectl -n d8-user-authn get ing kubernetes-api -o jsonpath="{.spec.tls[0].secretName}") \ | jq -rc '.data."ca.crt" // .data."tls.crt"' \ | base64 -d > /tmp/ca.crt
-
Generate a section with the external domain and CA:
kubectl config set-cluster $cluster_name --embed-certs=true \ --server=https://$(kubectl -n d8-user-authn get ing kubernetes-api -ojson | jq '.spec.rules[].host' -r) \ --certificate-authority=/tmp/ca.crt \ --kubeconfig=$file_name
-
-
If the CA is public, generate a section with just the external domain:
kubectl config set-cluster $cluster_name \ --server=https://$(kubectl -n d8-user-authn get ing kubernetes-api -ojson | jq '.spec.rules[].host' -r) \ --kubeconfig=$file_name
- If there is direct access to the API server, then use its IP address:
-
Generate the
user
section using the token from theServiceAccount
secret:kubectl config set-credentials $user_name \ --token=$(kubectl -n d8-service-accounts get secret gitlab-runner-deploy-token -o json |jq -r '.data["token"]' | base64 -d) \ --kubeconfig=$file_name
-
Generate the
context
to bind it all together:kubectl config set-context $context_name \ --cluster=$cluster_name --user=$user_name \ --kubeconfig=$file_name
-
Set default context of your newly created kubeconfig file:
kubectl config use-context $context_name --kubeconfig=$file_name
- The
How to create a user using a client certificate
Creating a user
- Get the cluster’s root certificate (ca.crt and ca.key).
-
Generate the user key:
openssl genrsa -out myuser.key 2048
-
Create a CSR file and specify in it the username (
myuser
) and groups to which this user belongs (mygroup1
&mygroup2
):openssl req -new -key myuser.key -out myuser.csr -subj "/CN=myuser/O=mygroup1/O=mygroup2"
-
Sign the CSR using the cluster root certificate:
openssl x509 -req -in myuser.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out myuser.crt -days 10
-
Now you can use the certificate issued in the config file:
cat << EOF apiVersion: v1 clusters: - cluster: certificate-authority-data: $(cat ca.crt | base64 -w0) server: https://<cluster_host>:6443 name: kubernetes contexts: - context: cluster: kubernetes user: myuser name: myuser@kubernetes current-context: myuser@kubernetes kind: Config preferences: {} users: - name: myuser user: client-certificate-data: $(cat myuser.crt | base64 -w0) client-key-data: $(cat myuser.key | base64 -w0) EOF
Granting access to the created user
To grant access to the created user, create a `ClusterAuthorizationRule’.
Example of a ClusterAuthorizationRule
:
apiVersion: deckhouse.io/v1
kind: ClusterAuthorizationRule
metadata:
name: myuser
spec:
subjects:
- kind: User
name: myuser
accessLevel: PrivilegedUser
portForwarding: true
Configuring kube-apiserver for multi-tenancy mode
The multi-tenancy mode, which allows you to restrict access to namespaces, is enabled by the enableMultiTenancy module’s parameter.
Working in multi-tenancy mode requires enabling the Webhook authorization plugin and configuring a kube-apiserver.
All actions necessary for the multi-tenancy mode are performed automatically by the control-plane-manager module; no additional steps are required.
Changes to the kube-apiserver
manifest that will occur after enabling multi-tenancy mode:
- The
--authorization-mode
argument will be modified: the Webhook method will be added in front of the RBAC method (e.g.,--authorization-mode=Node,Webhook,RBAC
); - The
--authorization-webhook-config-file=/etc/kubernetes/authorization-webhook-config.yaml
will be added; -
The
volumeMounts
parameter will be added:- name: authorization-webhook-config mountPath: /etc/kubernetes/authorization-webhook-config.yaml readOnly: true
-
The
volumes
parameter will be added:- name: authorization-webhook-config hostPath: path: /etc/kubernetes/authorization-webhook-config.yaml type: FileOrCreate
How do I check that a user has access?
Execute the command below with the following parameters:
resourceAttributes
(the same as in RBAC) - target resources;user
- the name of the user;groups
- user groups;
You can use Dex logs to find out groups and a username if this module is used together with the
user-authn
module (kubectl -n d8-user-authn logs -l app=dex
); logs available only if the user is authorized).
cat <<EOF | 2>&1 kubectl create --raw /apis/authorization.k8s.io/v1/subjectaccessreviews -f - | jq .status
{
"apiVersion": "authorization.k8s.io/v1",
"kind": "SubjectAccessReview",
"spec": {
"resourceAttributes": {
"namespace": "",
"verb": "watch",
"version": "v1",
"resource": "pods"
},
"user": "system:kube-controller-manager",
"groups": [
"Admins"
]
}
}
EOF
You will see if access is allowed and what role is used:
{
"allowed": true,
"reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""
}
If the multitenancy mode is enabled in your cluster, you need to perform another check to be sure that the user has access to the namespace:
cat <<EOF | 2>&1 kubectl --kubeconfig /etc/kubernetes/deckhouse/extra-files/webhook-config.yaml create --raw / -f - | jq .status
{
"apiVersion": "authorization.k8s.io/v1",
"kind": "SubjectAccessReview",
"spec": {
"resourceAttributes": {
"namespace": "",
"verb": "watch",
"version": "v1",
"resource": "pods"
},
"user": "system:kube-controller-manager",
"groups": [
"Admins"
]
}
}
EOF
{
"allowed": false
}
The allowed: false
message means that the webhook doesn’t block access. In case of webhook denying the request, you will see, e.g., the following message:
{
"allowed": false,
"denied": true,
"reason": "making cluster scoped requests for namespaced resources are not allowed"
}
Customizing rights of high-level roles
If you want to grant more privileges to a specific high-level role, you only need to create a ClusterRole with the user-authz.deckhouse.io/access-level: <AccessLevel>
annotation.
An example:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
user-authz.deckhouse.io/access-level: Editor
name: user-editor
rules:
- apiGroups:
- kuma.io
resources:
- trafficroutes
- trafficroutes/finalizers
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- flagger.app
resources:
- canaries
- canaries/status
- metrictemplates
- metrictemplates/status
- alertproviders
- alertproviders/status
verbs:
- get
- list
- watch
- create
- update
- patch
- delete