The module lifecycle stage: Preview
Default project templates
Deckhouse Platform includes the following project templates. They are cumulative: each one includes the capabilities of the previous one and adds its own. Parameter values are set in the .spec.parameters field of a Project:
-
simple: Minimal template that creates only the project namespace. Use it when you only need an isolated namespace managed as a project and configure access and limits through the standard fields and project role bindings.Parameters:
namespace.labelsandnamespace.annotations: Extra labels and annotations for the project namespace.requiredRequests(defaultfalse): Whentrue, the pods in the project must specify CPU and memory requests (an OperationPolicy in theDenymode).
-
default: Template that covers basic project use cases. On top of the namespace, it sets up network isolation, a pod security profile, extended monitoring and log shipping.Parameters (besides the ones listed for
simple):networkPolicy:Isolated(default) allows incoming traffic only from the project namespaces, Prometheus, the ingress-nginx controller and the service-with-healthchecks agent, and outgoing traffic only to the project namespaces and to DNS inkube-system.NotRestrictedallows all traffic.podSecurityProfile: The Pod Security Standards profile for the project namespaces:Baseline(default) prevents known privilege escalations,Restrictedapplies the strictest hardening practices,Privilegedrestricts nothing. An empty string sets no profile. The project puts nosecurity.deckhouse.io/pod-policylabel on its namespaces, and the default policy of theadmission-policy-enginemodule applies to them.extendedMonitoringEnabled(defaulttrue): Alerts on controller outages and restarts, 5xx errors in ingress-nginx and low free space on the project’s persistent volumes.clusterLogDestinationName: The name of the ClusterLogDestination to ship the project logs to. Left unset, the project logs are not shipped anywhere.requiredRequests(defaulttruein this template): Whentrue, the pods in the project must specify CPU and memory requests (an OperationPolicy in theDenymode). When an existing namespace becomes a project, the parameter is set tofalse, so running workloads are not blocked.
-
secure: Includes all the capabilities of thedefaulttemplate, and additionally restricts the users and groups inside containers, audits their calls to the kernel and scans images for vulnerabilities.Parameters (besides the ones listed for
default):allowedUIDsandallowedGIDs: The ranges (min,max) of IDs permitted for users and groups inside the project containers. See Security Context.runtimeAuditEnabled(defaultfalse): Audit rules that catch calls to the kernel and find malicious activity. They work only if the UID/GID ranges are set.securityScanningEnabled(defaulttrue): Periodic scanning of the launched images for known vulnerabilities (CVE) with Trivy, every 24 hours.
-
secure-with-dedicated-nodes: Includes all the capabilities of thesecuretemplate, and additionally places the pods of the main namespace of the project on dedicated nodes.Parameters (besides the ones listed for
secure). IfdedicatedNodesis set, at least one of its two fields must be set. WithoutdedicatedNodes, the pods are not placed on dedicated nodes:dedicatedNodes.nodeSelector: The node selector of the main namespace. It is merged into the node selector of every pod there, and a pod whose selector conflicts with it is rejected.dedicatedNodes.defaultTolerations: Tolerations in the format of the pod’sspec.tolerations. They are added to the pods created in the main namespace.
The default, secure, and secure-with-dedicated-nodes templates are described in the structured form (deckhouse.io/v1alpha2). The simple template is a minimal structured template that creates the namespace, sets its labels and annotations from the project parameters and, with requiredRequests: true, creates the OperationPolicy.
The exact set of parameters is in the template installed in the cluster. To view the parameter schema of a template, use the command:
d8 k get projecttemplates <PROJECT_TEMPLATE_NAME> -o jsonpath='{.spec.parametersSchema.openAPIV3Schema}' | jq
To view the whole template, use the command:
d8 k get projecttemplates <PROJECT_TEMPLATE_NAME> -o yaml
Creating a project
To create a project, follow these steps:
- Create a Project, specifying the name of the project template in .spec.projectTemplateName field.
- Set the standard fields .spec.administrators and .spec.quota. They work with any template.
-
In the .spec.parameters field of the Project resource, specify the parameter values suitable for the ProjectTemplate .spec.parametersSchema.openAPIV3Schema.
Example of a Project created from the
defaultProjectTemplate:apiVersion: deckhouse.io/v1alpha3 kind: Project metadata: name: my-project spec: description: This is an example from the Deckhouse documentation. projectTemplateName: default # Standard fields, managed by the Project resource itself. administrators: - kind: Group name: k8s-admins quota: requests.cpu: "5" requests.memory: 5Gi requests.storage: 1Gi limits.cpu: "5" limits.memory: 5Gi # Template-specific parameters. parameters: networkPolicy: Isolated podSecurityProfile: Restricted extendedMonitoringEnabled: trueThe Project API versions are described in Project API versions.
-
To check the status of the project, execute the command:
d8 k get projects my-projectA successfully created project should be in the
Deployedstate. If the state equalsError, add the-o yamlargument to the command (e.g.,d8 k get projects my-project -o yaml) to get more detailed information about the error.
Project API versions
The Project API is served as deckhouse.io/v1alpha3. v1alpha2 manifests keep working: a conversion webhook lifts parameters.administrators and parameters.resourceQuota into the .spec.administrators and .spec.quota standard fields automatically. If one administrator does not fit .spec.administrators (for example, it has accessLevel), the whole list stays in parameters and .spec.administrators is not filled. The same holds for a quota with anything other than quantities (for example, scopes), for a quota with the number 0 anywhere in it, and for an empty list or quota. A template that checks a value with with skips the number 0 and sets no limit, while in .spec.quota it would be a hard limit of zero. The string "0" is lifted into .spec.quota as such a limit. A v1alpha2 client reads a 0 from .spec.quota as the string "0".
A project cannot set .spec.administrators together with parameters.administrators, or .spec.quota together with parameters.resourceQuota.
The spec.grants field exists only in deckhouse.io/v1alpha3: a change of a project that has it through deckhouse.io/v1alpha2 is refused, see Granting cluster-wide resources to one project.
The deckhouse.io/v1alpha1 version is no longer served, and the API server refuses a manifest with apiVersion: deckhouse.io/v1alpha1. Before applying such a manifest, change apiVersion to deckhouse.io/v1alpha3, rename spec.projectTypeName to spec.projectTemplateName (a ProjectTemplate with that name must exist) and spec.template to spec.parameters, and move administrators and resourceQuota from the parameters to .spec.administrators and .spec.quota.
Creating a project without specifying a template
The projectTemplateName field is optional: when omitted, the simple template is used. Such a project consists only of the namespace and the standard fields (administrators, quota) — no template policies are created in it. Use it when the namespace needs no template settings or they are managed by other means:
apiVersion: deckhouse.io/v1alpha3
kind: Project
metadata:
name: my-plain-project
spec:
administrators:
- kind: Group
name: k8s-admins
quota:
requests.cpu: "2"
A template can be assigned later by setting it in .spec.projectTemplateName.
Project naming rules
The project name is also the name of its main namespace, so the following rules are checked when a project is created:
- The name cannot start with
d8-orkube-: these prefixes are reserved for system namespaces. - The name cannot be
defaultordeckhouse: these names belong to the virtual projects. - The name cannot be longer than 61 characters.
- The name cannot be the name of an existing namespace, or of a namespace that a ProjectNamespace has claimed. The resulting name
<project>-<spec.name>is taken from the moment the ProjectNamespace is created, before the controller creates the namespace. A namespace that belongs to no project becomes a project automatically. - The name cannot be the name of a namespace whose ServiceAccounts are still named among the subjects of a ClusterProjectRoleBinding, or of a ProjectRoleBinding of a project that the name extends with a dash and a suffix (
fooforfoo-extra). Such subjects are often left from a deleted additional namespace, and without this rule the ServiceAccounts of the new project would get the rights of that binding. Remove these subjects from the binding first. The refusal does not name the binding.
A project can be named after another project with a dash and a suffix (foo-bar beside foo). It is a separate project with a namespace of its own, not an additional namespace of foo, and the template, administrators, quota and bindings of foo do not apply to it. The webhook allows such a project with a warning, and the controller sends the NamedLikeAdditionalNamespace Warning event on the new project. The MultitenancyManagerProjectNamedLikeAdditionalNamespace alert lists every such pair of projects, including the ones made from namespaces that existed before. To add a namespace to a project, create a ProjectNamespace.
Project status and diagnostics
The .status.state field of a project is either Deployed (all project resources are in sync, or the objects of manifests wait for their binding) or Error. The cause of an error is described in the conditions (.status.conditions):
d8 k get project my-project -o jsonpath='{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}'
| Condition | False means |
|---|---|
ProjectTemplateFound |
The template referenced in .spec.projectTemplateName was not found. |
Validated |
The project parameters failed validation against the template schema (parametersSchema). |
ResourcesUpgraded |
The project resources could not be created or updated from the template (details in message). |
StandardFieldsApplied |
The standard fields (quota or administrators) could not be applied. |
TemplateRolesAllowed |
The template creates a binding to a role forbidden for granting in projects — the project switches to Error, the role is named in message. |
ManifestsApplied |
The objects of the template’s manifests are not applied. The message says what they wait for, why none of them is applied, which existing objects they leave alone, or which objects the API server refused, with the code of each refusal (for example, 403 Forbidden). |
HelmOwnership |
The namespace of the project belongs to another Helm release. The message names the release (see Creating a project automatically for a namespace). |
NamespaceDeleted |
The project is being deleted, and its namespace is still terminating. The message says since when and what remains in it. |
QuotaWithinLimits |
The usage of the project is above its quota; the message names the limits. A project without a quota does not have the condition. |
The HandoverPending condition is read the other way round. It is True while AuthorizationRule and ResourceQuota objects kept from a Helm text wait for the standard fields that replace them, and it is removed once they are deleted (see Switching a project from a Helm text).
Other useful status fields:
.status.namespaces: All namespaces of the project with their kind (MainorAdditional)..status.usage: The usage of the project summed over all of its namespaces, taken from its ProjectQuotaStatus..status.resources: State of the individual objects created from the template..status.keptObjects: Objects of a Helm text and of the release of the structured fields that the release left in place instead of deleting them, untilmanifeststake them over..status.manifestObjects: Objects applied frommanifestsof the template.
Service objects of a project
The controller creates service objects in the project namespaces. They are managed automatically — editing them manually is not possible (the attempt is rejected):
| Object | Where | Comes from |
|---|---|---|
ProjectQuotaStatus/<PROJECT_NAME> |
Cluster-scoped | The .spec.quota field of the project (see Project quota). |
ResourceQuota/d8-project-quota |
Every namespace of the project | The .spec.quota field of the project. |
ProjectRoleBinding/d8-administrators |
The main namespace | The .spec.administrators field of the project. |
ProjectRoleBinding/d8-manifests |
The main namespace | The manifests field of the project template. It grants the d8:project:admin role to the user that applies these objects. |
RoleBinding/d8:prb:<name> |
Every namespace of the project | The fan-out of the ProjectRoleBinding named <name>. |
RoleBinding/d8:cprb:<name> |
Every namespace of every project | The fan-out of the ClusterProjectRoleBinding named <name>. |
When the source object (a binding, the quota field, etc.) is removed, the corresponding service objects are removed automatically.
Virtual projects
Besides the user-created projects, the d8 k get projects list always contains two virtual projects (labeled projects.deckhouse.io/virtual-project: "true"):
deckhouse— groups system namespaces (d8-*,kube-*,upmeter-*,heritage: deckhouse/heritage: upmeter);default— groups remaining namespaces that do not belong to any project (thedefaultnamespace itself).
Virtual-project status is rebuilt from the live namespace list: a deleted namespace disappears from that list. Virtual projects do not recreate namespaces.
Virtual projects exist for completeness: with them, every namespace of the cluster belongs to some project. They cannot be managed: they are not editable, ProjectNamespace and ProjectRoleBinding resources cannot be created in them, and ClusterProjectRoleBinding does not extend to them. Their project template, virtual, is reserved for them: a user project with projectTemplateName: virtual is refused by the webhook, since such a project would get no namespace of its own.
The projects.deckhouse.io/virtual-project: "true" label is reserved too: the webhook refuses a project that is created with it or gets it added. A project is virtual by its template, not by the label, so a user project that already carries the label is an ordinary project; it keeps the label with a warning until the label is removed.
No ClusterResourceGrantPolicy applies to their namespaces either, even one whose selectors match a virtual Project, because the cluster resource availability rules apply only in namespaces with the projects.deckhouse.io/project label, and the namespaces of virtual projects do not carry it.
No ProjectTemplate object named virtual exists, and the webhook refuses to create one. A template with this name that is already in the cluster is not used by any project and can be deleted.
Additional project namespaces
In Deckhouse Platform Open (CE), a project has only its main namespace.
If an application needs several namespaces (for example, a separate one for a cache or queues), add them to the project with the ProjectNamespace resource. A ProjectNamespace is created in the main namespace of the project; the resulting namespace is named <project name>-<spec.name>:
apiVersion: deckhouse.io/v1alpha3
kind: ProjectNamespace
metadata:
name: cache
namespace: my-project
spec:
name: cache # The my-project-cache namespace will be created.
You can check the project composition in its status:
d8 k get project my-project -o jsonpath='{.status.namespaces}'
The rules for working with ProjectNamespace:
- The
spec.namefield is immutable: to rename a namespace, delete the ProjectNamespace and create a new one. - The resulting name
<project name>-<spec.name>cannot be longer than 63 characters (the Kubernetes limit on namespace names). - A ProjectNamespace can only be created in the main namespace of a project. It cannot be “nested” into an additional namespace or a foreign project.
- The resulting name must be free. The request is rejected when a namespace with that name exists and is not a namespace of this project, when a project has that name, or when another ProjectNamespace already claims it (the ProjectNamespace
a-backendof projectteamand the ProjectNamespacebackendof projectteam-aboth claimteam-a-backend). - The request is also rejected when the ServiceAccounts of a namespace with the resulting name are still named among the subjects of a ClusterProjectRoleBinding, or of a ProjectRoleBinding of another project that the resulting name extends with a dash and a suffix (
teamforteam-a-backendof projectteam-a). Remove these subjects from the binding first. A ProjectRoleBinding of the same project is no obstacle, and the ServiceAccounts it names get the rights back once the namespace is created again. - Deleting a ProjectNamespace deletes its namespace. Deleting the project deletes all of its namespaces.
- A namespace with the resulting name that is not a namespace of this project is never taken into it, even when it appears after the ProjectNamespace was created. The ProjectNamespace then does not become ready, and deleting it leaves the namespace as it is.
- A project can be deleted only by a user allowed to delete each of its namespaces (the
deleteverb onnamespaces): the main namespace and every namespace labeledprojects.deckhouse.io/project: <project name>. Otherwise the request is refused, and the refusal names the namespace the user may not delete. A namespace that does not exist is not checked. The identities the Project webhook skips (see Who bypasses admission) delete projects without this check.
What applies to the additional namespaces
The following automatically applies in all namespaces of the project (the main and the additional ones alike):
- Access: the ProjectRoleBinding and ClusterProjectRoleBinding bindings, including the automatic access of the project administrators. When a new namespace is added, all existing bindings fan out into it without any user action.
- Namespaced template objects: the network policy (
networkPolicy.mode: Isolated) and the log collection setup (logShipping) are created in every namespace of the project. The network isolation allows traffic between the namespaces of one project. A HelmresourcesTemplatecreates its objects in the main namespace unless an object names another namespace of the project. So doesmanifests, whose objects are not applied while one of them names a namespace outside the project. - Cluster-scoped template policies (
OperationPolicy, theSecurityPolicyfromallowedUIDs/allowedGIDs): they select namespaces by theprojects.deckhouse.io/projectlabel, that is, they cover the whole project. - The project quota: the
d8-project-quotaResourceQuota with the limits of the project quota is created in every namespace of the project. - Inherited labels: the pod security profile (
security.deckhouse.io/pod-policy), extended monitoring (extended-monitoring.deckhouse.io/enabled), vulnerability scanning (security-scanning.deckhouse.io/enabled), the template label (projects.deckhouse.io/project-template) and the labels the template sets throughnamespaceMetadata.labelsare synced from the main namespace to the additional ones. For a template withresourcesTemplate, these are the labels its Helm text puts on the main namespace. The sync is complete: if a feature is turned off or a label is removed in the template, the label is removed from the additional namespaces as well. The labels of the template are listed in theprojects.deckhouse.io/template-labelsannotation of each project namespace; they are changed through the Project or its ProjectTemplate, and a direct edit of one on an additional namespace is reverted. Thanks to the template label, the cluster resource availability rules also apply in all namespaces of the project.
The following stays in the main namespace only:
- The annotations from the template’s
namespaceMetadata - The labels and annotations set on the main namespace directly or by a GitOps tool, including the ones the namespace had before it became a project, other than the pod security, monitoring and scanning labels above
- The GitOps tracking labels (
app.kubernetes.io/instance,argocd.argoproj.io/*,kustomize.toolkit.fluxcd.io/*,helm.toolkit.fluxcd.io/*,kapp.k14s.io/*), even when the template sets them - The node placement annotations (from the template’s
nodeSelectorandtolerationsfields)
GitOps tracking labels are not inherited because a GitOps tool would treat an additional namespace that carries them as its own and could delete it. The module recognizes only the keys listed above. A custom tracking key that the template sets, for example the one configured with application.instanceLabelKey in Argo CD, is inherited.
When a namespace becomes a project automatically, its own labels and annotations stay on it and are not copied into the Project, so the additional namespaces of the project do not inherit them. The pod security, monitoring and scanning labels are the exception. Adoption turns them into parameters of the project, and they are inherited as on any project. To put another label of such a namespace on the additional namespaces too, declare it in spec.parameters.namespace.labels of the Project, or set it on each additional namespace directly.
The built-in templates of adopted projects are rewritten by the controller, so a label added to their namespaceMetadata.labels does not stay there. A grant policy whose namespaceSelector matches such a label does not cover the new additional namespaces until the label is declared in the Project. To cover the whole project, select it with projectSelector.
Labels of the project namespaces
| Label | Main | Additional | Purpose |
|---|---|---|---|
projects.deckhouse.io/project: <PROJECT_NAME> |
✓ | ✓ | Project ownership — the common label of all namespaces of the project |
projects.deckhouse.io/project-namespace: <spec.name> |
— | ✓ | Marks an additional namespace (the name of the ProjectNamespace resource) |
projects.deckhouse.io/project-template: <PROJECT_TEMPLATE_NAME> |
✓ | ✓ | The project template; the cluster resource availability rules match by it |
heritage: multitenancy-manager |
✓ | ✓ | The namespace is managed by the project controller: its spec, finalizers and the labels listed in this table are changed through the Project; other labels and annotations may be changed directly |
projects.deckhouse.io/quota: pending, projects.deckhouse.io/quota: enforced |
✓ | ✓ | The namespace belongs to a project with a quota; only the controller sets the label |
quota.projects.deckhouse.io/vm: enforced, quota.projects.deckhouse.io/lb: enforced |
✓ | ✓ | The quota of the project has keys of virtual machines or of load balancers |
security.deckhouse.io/pod-policy, extended-monitoring.deckhouse.io/enabled, security-scanning.deckhouse.io/enabled |
✓ | ✓ (inherited) | Policies and features from the project template |
Labels from the template’s namespaceMetadata.labels |
✓ | ✓ (inherited) | Labels of the project template. They are listed in the projects.deckhouse.io/template-labels annotation of the namespace and changed through the Project or its ProjectTemplate |
| Any other label set on a namespace directly or by a GitOps tool, including the labels a namespace had before it became a project | — | — | Not set by the module. The project release neither reverts nor restores it, and the additional namespaces do not inherit it |
The common projects.deckhouse.io/project label makes it possible to select the project namespaces with a plain get ns. To get all namespaces of the project (main and additional), use the command:
d8 k get ns -l projects.deckhouse.io/project=my-project
To get additional namespaces only:
d8 k get ns -l 'projects.deckhouse.io/project=my-project,projects.deckhouse.io/project-namespace'
To get the main namespace only:
d8 k get ns -l 'projects.deckhouse.io/project=my-project,!projects.deckhouse.io/project-namespace'
Creating a project automatically for a namespace
A namespace created directly (for example, d8 k create ns my-app) becomes a project with the same name:
- the template is picked from what the namespace already carries:
secureif it has thesecurity-scanning.deckhouse.io/enabledlabel,defaultif it hassecurity.deckhouse.io/pod-policyorextended-monitoring.deckhouse.io/enabled, andsimpleotherwise; - the project parameters are filled in from the current state of the namespace, so nothing inside it changes: the network policy stays unrestricted and the Pod Security Standard keeps the value the namespace already had;
- from then on the project is the source of truth and is edited like any other project. The namespace of the project cannot be deleted directly: the request is refused with the
Manual modification is forbiddenmessage. To remove the environment, delete the Project:d8 k delete project <NAME>deletes the namespace with it and returns once the namespace is gone. Only a user allowed to delete the namespace can delete its Project (see Additional project namespaces).
Users can create a namespace directly only while the allowNamespacesWithoutProjects parameter is true (the default). With false, only cluster administrators (system:masters, system:sudouser), Deckhouse, the module controller and system components create namespaces directly (see Who bypasses admission). Other users create a Project or a ProjectNamespace instead.
Every such namespace becomes a project of its own, whatever its name. Of the namespaces app and app-staging, app-staging becomes a separate project and not an additional namespace of app. The controller creates the project, so no admission warning reaches the author of the namespace. Instead, the controller sends the NamedLikeAdditionalNamespace Warning event on the new project and on the namespace. The events of cluster-scoped objects are kept in the default namespace, so reading them needs access to the events there: d8 k -n default get events --field-selector reason=NamedLikeAdditionalNamespace. The MultitenancyManagerProjectNamedLikeAdditionalNamespace alert lists such pairs, see Project naming rules.
System namespaces (d8-*, kube-*, upmeter-*, default, and anything labeled heritage: deckhouse or heritage: upmeter) are never adopted: they are listed on the virtual deckhouse project (except default, which stays on the virtual default project).
No label leaves a new user namespace without a project: users cannot put the heritage: upmeter label on a namespace, and only the module sets the projects.deckhouse.io/project label.
A namespace whose name is longer than 61 characters (the Project name limit) is also skipped, and so is a namespace named deckhouse: a Project of that name is the virtual project.
So that no namespace stays outside a project, the d8-multitenancy-manager-namespace-adoption ValidatingAdmissionPolicy refuses to create a namespace whose name is longer than 61 characters, starts with upmeter- or is deckhouse, and to put the heritage: upmeter label on a namespace.
The policy works whatever allowNamespacesWithoutProjects says and lets through the users and groups listed for the d8-multitenancy-manager-namespace-creation policy in Who bypasses admission.
A namespace that stays without a project for 10 minutes, for example while no replica of the module controller leads, raises the D8NamespaceNotAdopted alert.
The controller pods export the metric of the alert, so the alert is not raised while none of the pods is Ready.
The policy does not change the namespaces that exist already, so a namespace that was created before it, with one of these names or with the heritage: upmeter label, stays outside every project. To list the namespaces that belong to no project and are not system ones, run:
d8 k get namespaces -l '!projects.deckhouse.io/project,heritage!=deckhouse' -o name | grep -Ev '^namespace/(d8-|kube-|default$|upmeter-probe-namespace-)'
The list holds the namespaces that get no project: the names longer than 61 characters, deckhouse, the other upmeter-* names and the namespaces with the heritage: upmeter label. The namespaces of the upmeter agent can be among them, and they must not be changed.
To bring a namespace under a project, remove the heritage: upmeter label from it (if its name does not start with upmeter-), and the controller creates the project. Otherwise, move the workloads of the namespace to a namespace under another name. A namespace with an empty projects.deckhouse.io/project label is not in the list, d8 k get namespaces -l 'projects.deckhouse.io/project=' lists such namespaces.
On adoption, a namespace that gets the default or secure template because of its monitoring or scanning label, and has no security.deckhouse.io/pod-policy label, gets podSecurityProfile: "". The project puts no such label on it, and the default policy of the admission-policy-engine module keeps applying to it. A namespace whose label has a value other than baseline, restricted or privileged gets podSecurityProfile: "" too, and the label stays on it as it is. A namespace with none of these labels gets the simple template, which sets no profile and has no podSecurityProfile parameter.
To change the profile later, set podSecurityProfile in the project. For a project on simple, first move it to default with networkPolicy: NotRestricted, requiredRequests: false and extendedMonitoringEnabled: false, so that nothing else changes.
A namespace that already belongs to another Helm release (its meta.helm.sh/release-name annotation names a release other than the project’s) is not taken over: the project gets the HelmOwnership condition with the name of the foreign release and retries, and the namespace is left as it is. Remove the foreign release or its ownership annotations to let the project proceed.
Existing RoleBinding and AuthorizationRule objects inside the namespace keep working after adoption. The namespace Admin is not copied into .spec.administrators and does not become d8:project:admin: that role additionally manages ProjectRoleBinding resources, which is a wider contract than in-namespace Admin. To make the team lead a project administrator, a platform operator adds them to .spec.administrators or creates a ProjectRoleBinding. The namespace Admin role allows only get, list and watch on the Namespace object.
The labels and annotations the module sets — heritage, projects.deckhouse.io/*, quota.projects.deckhouse.io/vm, quota.projects.deckhouse.io/lb, security.deckhouse.io/pod-policy, extended-monitoring.deckhouse.io/enabled, security-scanning.deckhouse.io/enabled, app.kubernetes.io/managed-by, meta.helm.sh/*, the node-selector and tolerations annotations — are changed through the Project and its template;
any other label or annotation (istio-injection, a Pod Security label, a GitOps tracking annotation) can still be set directly on the Namespace by anyone whose RBAC allows it, and the controller does not touch it. The spec and finalizers of the Namespace, and its deletion, stay with the controller.
This includes the labels and annotations the namespace had before it became a project.
Adoption does not copy them into the project, so the project release does not own them,
and the controller neither reverts a change to one of them nor brings one back once it is removed.
They stay on the main namespace, and the additional namespaces of the project do not inherit them.
Where such a label is needed on an additional namespace too, declare it in spec.parameters.namespace.labels of the Project or set it on that namespace directly.
The node-selector and tolerations annotations are the exception.
They belong to the module, so adoption copies them into the namespace.annotations parameter of the project, where they are changed or removed.
The pod security, monitoring and scanning labels are turned into parameters of the project, and the additional namespaces inherit them. The extended-monitoring.deckhouse.io/enabled and security-scanning.deckhouse.io/enabled labels keep the value "true" if the namespace had it. Any other value, "false" included, is replaced with an empty one, because adoption turns the feature on whenever the namespace carries the label, whatever its value.
The projects.deckhouse.io/project label itself is set only by the controller: a namespace created by hand with that label is refused, an empty value included, so a namespace cannot be made to look owned by a project it does not belong to. A namespace belongs to the project its label names and to no other: one without the label, or with an empty value, is not counted towards the project of the same name.
A namespace that already carries the empty value is not adopted either: the empty value used to keep a namespace out of adoption, and it still does until the label is removed.
For example:
-
Create a new namespace:
d8 k create ns test -
Make sure that the project was created:
d8 k get projectsA new project corresponding to the namespace will appear in the project list:
NAME STATE PROJECT TEMPLATE DESCRIPTION AGE deckhouse Deployed virtual This is a virtual project 181d default Deployed virtual This is a virtual project 181d test Deployed simple 1m
To change the template of such a project, set another template in .spec.projectTemplateName.
If the namespace already has an object of the same kind and name as one that the new template creates, for example the isolated NetworkPolicy, the project switches to Error, and the ResourcesUpgraded condition names the object. Delete or rename the object first.
Standard project fields
Project administrators and the resource quota are set in fields of the Project and work with any template (including simple):
.spec.administrators— a list of subjects (kind: Userorkind: Groupandname) that receive administrative access to the project. The controller manages this access as an auto-generated ProjectRoleBinding in the project namespace. A name cannot contain control characters or line breaks and cannot start withsystem:multitenancy-manager:, the prefix of the users that the controller acts as. An update that keeps such a name as it was is accepted with a warning..spec.quota— a map of ResourceQuota hard limits (for example,requests.cpu,limits.memory) for the whole project. The controller turns it into the ProjectQuotaStatus of the project and reports the usage summed over the namespaces of the project in.status.usage; clearing the field deletes that ProjectQuotaStatus. Changing the field needs the right to set the quota of the project, and its keys follow the rules of the project quota. Formemoryandstorage, a unit suffix is required (for example2Gi). Numbers without a unit mean bytes, and all of them except 0 are rejected. An update that leaves such a value as it was is accepted with a warning, so a project whose quota came from bare numbers inparameters.resourceQuotastays editable and can still be applied unchanged. A create throughdeckhouse.io/v1alpha2with such numbers inparameters.resourceQuotais accepted with a warning as well, so such a manifest can create the project again, for example from a backup.
apiVersion: deckhouse.io/v1alpha3
kind: Project
metadata:
name: my-project
spec:
projectTemplateName: simple
administrators:
- kind: Group
name: k8s-admins
quota:
requests.cpu: "5"
requests.memory: 5Gi
limits.cpu: "10"
limits.memory: 10Gi
A Helm resourcesTemplate that renders a ResourceQuota or an AuthorizationRule keeps rendering it, next to the ResourceQuota and the ProjectRoleBinding that the standard fields create. A pod in such a project has to fit every ResourceQuota of its namespace, and the access such AuthorizationRules grant adds to the access of the standard fields.
Project quota
The project quota is one limit on the sum of resource usage over all namespaces of a project, the additional namespaces included. It is set in the .spec.quota field of the project:
apiVersion: deckhouse.io/v1alpha3
kind: Project
metadata:
name: my-project
spec:
projectTemplateName: default
quota:
requests.cpu: "20"
requests.memory: 40Gi
projectNamespaces: "3"
From the field the controller makes the ProjectQuotaStatus object named after the project and keeps it up to date. The object shows the limits and the usage of the project and is read-only: only the controller creates, changes and deletes it. Clearing the field deletes it.
How the quota applies:
- The controller creates the
d8-project-quotaResourceQuota with the limits ofspec.hardin every namespace of the project, the ones added later included. This ResourceQuota counts the usage of its namespace, so no namespace alone uses more than the whole project may. - A namespace that the quota applies in carries the
projects.deckhouse.io/quota: enforcedlabel once itsd8-project-quotais calculated by kube-controller-manager. A namespace created for a ProjectNamespace carriesprojects.deckhouse.io/quota: pendingfrom the start until then, and the main namespace of a project that has not been deployed yet gets it when the quota of the project is applied, so the manifests of the template wait for the quota too, however long the first attempts to deploy the project fail. The main namespace of a project that has been deployed (in theDeployedstate), as on an upgrade of the module or when a quota is set on a running project, is not held. The label is set by the controller only. - In a namespace labelled
projects.deckhouse.io/quota: pending, thed8-multitenancy-manager-quota-pendingadmission policy refuses the objects of users untild8-project-quotais calculated, and the ProjectNamespace reportsReady=False. That takes seconds after a namespace is created. In a namespace labelledprojects.deckhouse.io/quota: enforced, thed8-multitenancy-manager-quota-guardadmission policy refuses the objects the quota counts whiled8-project-quotahas no limits calculated at all or has scopes. A limit added to a quota that is calculated already does not hold anything back: it applies once kube-controller-manager has calculated it, andEnforcedisFalsewith theKeyRolloutreason until then. - A ProjectQuotaStatus is deleted in two steps: its
projects.deckhouse.io/project-quotafinalizer holds it until the controller has taken the label off the namespaces, and only then thed8-project-quotaResourceQuota objects go with it. A namespace that is being deleted loses the label at once, so that the objects waiting for a finalizer there, such as the volume claims protected from deletion, can be updated. If the module is disabled before the finalizer is removed, remove it by hand. -
The status of the ProjectQuotaStatus shows the usage summed over the namespaces (
status.used), the number of namespaces and the 50 namespaces closest to a limit. The usage of every namespace is in itsd8-project-quotaResourceQuota:d8 k get resourcequota -A -l projects.deckhouse.io/project=my-project -
In a namespace labelled
projects.deckhouse.io/quota: enforced, creating an object that takes the usage of the whole project over a limit is denied with HTTP 403:admission webhook "quota-ledger.multitenancy-manager.deckhouse.io" denied the request: exceeded quota: d8-project-quota, project my-project: requested: requests.cpu=500m, used: requests.cpu=19800m, limited: requests.cpu=20 (3 namespaces)The webhook, called the quota ledger, keeps the sum of every project in the memory of its leader in the
quota-ledgercontainer of the module pod. While no replica answers (for a few seconds when the leader changes), a counted create in a namespace of a quota project fails withproject quota ledger unavailable (<CAUSE>), retry; deletions, running objects and namespaces without the quota are not affected. A Deployment or a Job retries such a create by itself. The same answer comes at once, with thetoo many approvals of the project are not confirmed yetcause, for a write that would wait for the earlier approvals of its project to be confirmed while two writes from the same API server already wait: the third waiting write of a project from one API server gets a 503, and its controller retries it. - The sum is not checked for a request with
--dry-run=server: the API server does not write the usage of a dry run. The quota of the namespace is still checked. - The
Enforcedcondition says whether the quota applies in every namespace, and theExceededcondition whether the usage is above a limit. A limit may be lowered below the current usage: the change is accepted with a warning, nothing running is evicted, andExceededisTruewith theHardLoweredreason. TheD8ProjectQuotaExceededalert fires then.
Keys of a project quota
.spec.quota accepts the following keys:
cpu,memory,ephemeral-storage,pods,requests.*andlimits.*of CPU, memory and ephemeral storage,requests.storage,requests.hugepages-<SIZE>, the requests of extended resources (for example,requests.nvidia.com/gpu);services,services.loadbalancers,services.nodeports,persistentvolumeclaims,configmaps,secrets,replicationcontrollers;<CLASS>.storageclass.storage.k8s.io/requests.storageand<CLASS>.storageclass.storage.k8s.io/persistentvolumeclaims,<CLASS>.deviceclass.resource.k8s.io/devices;count/<RESOURCE>[.<GROUP>]for a namespaced resource the API server serves, for examplecount/ingresses.networking.k8s.io;projectNamespaces, an alias ofcount/projectnamespaces.deckhouse.io: the number of ProjectNamespace objects of the project, that is, of its additional namespaces. The alias is expanded in the ResourceQuota objects, and the object keeps it as written.- the keys of virtual machines and of load balancers, which no ResourceQuota counts.
Refused are resourcequotas and count/resourcequotas (d8-project-quota takes one in every namespace by itself), the count of a cluster-scoped resource such as count/namespaces, a resource misspelled in a served group, a count/ key of a resource that lacks one of the verbs create, list, watch and delete, and the keys with the reserved projectquota.deckhouse.io/ prefix.
A count/ key of a group the API server does not serve yet is accepted with a warning: nothing counts it until the module that provides the group is enabled, and Enforced is False with the KeyNotServed reason until then.
A change that keeps a key the object already has is accepted with a warning even when the key would be refused now.
A new key in a live quota starts to apply once d8-project-quota of every namespace has it calculated; until then Enforced is False with the KeyRollout reason.
Pods and volume claims with resource-quota-overrides.deckhouse.io/* labels set by platform controllers are counted as a ResourceQuota counts them.
Virtual machines in a project quota
Where the virtualization module is enabled, a project quota limits the virtual machines of the project by their specification. No ResourceQuota counts these keys, so they do not go into d8-project-quota: the quota ledger checks every create of a VirtualMachine and every change of its spec in the namespaces of a project whose quota has them, and counts the virtual machines itself in these keys:
virtualmachines.virtualization.deckhouse.io/cores: Sum ofspec.cpu.cores.virtualmachines.virtualization.deckhouse.io/cpu: Guaranteed CPU, the sum of cores ×spec.cpu.coreFraction. An emptyspec.cpu.coreFractionorAutocounts as whole cores.virtualmachines.virtualization.deckhouse.io/memory: Sum ofspec.memory.size.virtualmachines.virtualization.deckhouse.io/gpus: Number of GPUs inspec.gpus.<GPU_CLASS>.gpuclass.virtualization.deckhouse.io/gpus: Number of GPUs of the<GPU_CLASS>GPUClass.
The number of virtual machines is limited by the ordinary count/virtualmachines.virtualization.deckhouse.io key.
Every virtual machine counts in the keys above, a stopped one too: a machine stopped by hand may be started at any moment. Only a machine with runPolicy: AlwaysOff, which is kept switched off on purpose, is not counted in them; the count/virtualmachines.virtualization.deckhouse.io key counts such machines too.
A change that does not raise a key, such as making a machine smaller or stopping it, is never refused over the quota and passes even while the quota ledger does not answer; every other create and change of spec needs the ledger to answer. A change that touches only the metadata of a machine does not call the ledger.
A smaller spec frees the quota only once the machine runs with it: until a restart applies it, the machine counts with the cores and memory it runs with (status.resources) and with the GPUs that the changes in status.restartAwaitingChanges show. Until the virtualization module has processed a change of the spec, the machine keeps what it counted before.
The namespaces of such a project carry the quota.projects.deckhouse.io/vm: enforced label, which the controller sets.
While the virtualization module is disabled, the keys count nothing, and Enforced is False with the KeyNotServed reason. The quota is accepted with a warning when the API server does not serve VirtualMachine.
The pod of a virtual machine counts in requests.* and limits.* as well, twice while the machine migrates. A quota that has both the keys of virtual machines and the keys of pods (cpu, memory, requests.*, limits.*) is accepted with a warning. Limit virtual machines with their keys and containers with requests.* and limits.*.
Load balancers in a project quota
A project quota limits the Services of type LoadBalancer of the project by their class, by their pool and by the external addresses they take.
No ResourceQuota counts these keys, so they do not go into d8-project-quota: the quota ledger checks every create of a Service that is or was of type LoadBalancer, and every change of its spec or of the network.deckhouse.io/metal-load-balancer-class, network.deckhouse.io/l2-load-balancer-external-ips-count, network.deckhouse.io/load-balancer-ips and network.deckhouse.io/load-balancer-pool annotations, in the namespaces of a project whose quota has them, and counts the Services itself in these keys:
<CLASS>.loadbalancerclass.network.deckhouse.io/services: Number of load balancers of the<CLASS>class.<CLASS>.loadbalancerclass.network.deckhouse.io/addresses: External addresses of the<CLASS>class.<POOL>.loadbalancerpool.network.deckhouse.io/addresses: Addresses taken from the<POOL>pool of the BGP mode.loadbalancers.network.deckhouse.io/addresses: All external addresses of the project.
The number of load balancers of any class is limited by the ordinary services.loadbalancers key.
The class of a Service is the one the metallb module serves it with. It comes from the first source that applies:
- The class the
metallbmodule has bound the Service to in its status, even if no such MetalLoadBalancerClass exists spec.loadBalancerClass, when such a MetalLoadBalancerClass exists- The
network.deckhouse.io/metal-load-balancer-classannotation, when such a MetalLoadBalancerClass exists - The MetalLoadBalancerClass with
isDefault: true spec.loadBalancerClasswithout a check, the class of another load balancer controllerdefault, the load balancers of a cloud provider
Where the API server does not serve MetalLoadBalancerClass, no class is checked: the class is spec.loadBalancerClass, else default, and the annotation, which only the metallb module reads, does not count.
The quota ledger notices within 30 seconds that the metallb module has been enabled or disabled, and counts the load balancers again without a restart.
A Service with the network.deckhouse.io/load-balancer-pool annotation takes one address of that pool. Any other Service takes the number of addresses from its network.deckhouse.io/l2-load-balancer-external-ips-count annotation when the value is a number above one, and one address when the annotation has any other value. The network.deckhouse.io/load-balancer-ips annotation only picks which addresses the Service takes, as the metallb module assigns them.
Every Service of type LoadBalancer counts its own addresses, also when it shares an address with other Services under the network.deckhouse.io/load-balancer-shared-ip-key annotation: two Services that share one address take two addresses of the quota. The namespaces of such a project carry the quota.projects.deckhouse.io/lb: enforced label, which the controller sets.
Clusters where Deckhouse Platform does not manage the control plane
The sum over the namespaces of a project is enforced only where Deckhouse Platform manages the control plane of the cluster (the control-plane-manager module is enabled).
Elsewhere the d8-multitenancy-manager-quota-pending and d8-multitenancy-manager-quota-guard admission policies are not rendered, every ProjectQuotaStatus reports Enforced=False with the ControlPlaneNotManaged reason, and the D8ProjectQuotaNotSupported alert says so.
Each namespace still keeps its d8-project-quota ResourceQuota, so no namespace alone uses more than the quota of the whole project.
Dividing a project quota between namespaces
A project administrator may create ResourceQuota objects of their own in the namespaces of the project. Such a quota only narrows the project quota: an object has to fit every ResourceQuota of its namespace. A quota of a project administrator cannot be named d8-project-quota and cannot limit resourcequotas or count/resourcequotas. A d8-project-quota that the controller did not create, or one with scopes, is deleted and created again by the controller.
For example, to keep the my-project-db namespace within a quarter of the CPU of the project:
apiVersion: v1
kind: ResourceQuota
metadata:
name: db-share
namespace: my-project-db
spec:
hard:
requests.cpu: "5"
Who manages the quotas
Changing .spec.quota of a Project needs the update verb on projects/quota in the deckhouse.io group for that project, which the d8:system-capability:multitenancy-manager:manage_quotas capability grants; the IAM manager role and the roles above it include the capability. The Project webhook checks the right, so whoever may create and change projects does not choose their quota. A ProjectQuotaStatus is written by the controller only; everyone else, cluster administrators included, can only read it. The usage in the status of d8-project-quota is written by Kubernetes; it is protected from other writers where Deckhouse Platform manages the control plane.
When the project quota refuses
What the response says:
| Response | What it means |
|---|---|
403 ... exceeded quota: d8-project-quota, project <PROJECT_NAME>: requested: ..., used: ..., limited: ... (<N> namespaces) |
The sum of project <PROJECT_NAME> would go over the limits of its quota. Raise the limits in .spec.quota of the project or delete what the project does not need; status.topNamespaces of its ProjectQuotaStatus shows where the usage is. |
403 ... exceeded quota: d8-project-quota, requested: ... (no project) |
The usage of this namespace alone, with the object, would go over a limit of the project, which every d8-project-quota ResourceQuota contains. The quota plugin of the API server refuses it before the sum of the project is checked. |
503 ... project quota ledger unavailable (<CAUSE>), retry |
The ledger does not answer for a moment, for example while its leader changes. Retry; if it lasts longer than a minute, see the D8ProjectQuotaLedgerDown alert. |
500 ... failed calling webhook "quota-ledger.multitenancy-manager.deckhouse.io": ..., or the same with quota-vm.multitenancy-manager.deckhouse.io or quota-lb.multitenancy-manager.deckhouse.io |
The API server does not reach the ledger: no replica runs, the network, or the certificate. While the Service points at a frozen leader, the quota plugin of the API server retries the call, so each such create waits up to about 8 seconds until the address of the leader is taken out of the Service: in about 3.5 seconds by the other replica, in up to about 15 seconds by Deckhouse when no replica answers. |
403 ... ValidatingAdmissionPolicy 'd8-multitenancy-manager-quota-pending' ... |
The d8-project-quota of a new namespace is not calculated yet. Retry in a few seconds; if it lasts, see the Enforced condition of the ProjectQuotaStatus of the project and check the controller and kube-controller-manager. |
403 ... ValidatingAdmissionPolicy 'd8-multitenancy-manager-quota-guard' ... |
The d8-project-quota of the namespace is missing, has scopes or is not calculated at all yet. Retry in a few seconds; otherwise check the controller and kube-controller-manager. |
500 ... resource quota evaluation timed out |
The quota admission of the API server is overloaded; see the D8ResourceQuotaAdmissionDegraded alert. |
A new additional namespace refuses the objects of users for the few seconds until its d8-project-quota is calculated. A namespace where the quota applies and whose d8-project-quota is gone does the same until the controller makes it anew, and refuses the objects of system:sudouser too. Of the users, only system:sudouser can delete d8-project-quota.
If the ledger keeps the objects of projects from being created and the cause cannot be removed quickly, take these steps in order until the creates work again:
-
If the creates fail with
503 ... project quota ledger unavailable, a replica answers, and suspending the ledger helps. Within a second every running replica lets the writes through uncounted, and every ProjectQuotaStatus reportsEnforced=Falsewith theSuspendedreason. Deleting the ConfigMap resumes the check; theD8ProjectQuotaSuspendedalert reminds of it after 15 minutes.d8 k -n d8-multitenancy-manager create configmap quota-ledger-suspend --from-literal=reason="<REASON>" d8 k -n d8-multitenancy-manager delete configmap quota-ledger-suspendReplace
<REASON>with why the check is suspended. -
If the creates fail with
500 ... failed calling webhook "quota-ledger.multitenancy-manager.deckhouse.io", the API servers do not reach a replica that answers, as while theD8ProjectQuotaLedgerDownalert fires, and the suspension does not help: delete the webhook of the ledger assystem:sudouser. The check stops within a second, and every ProjectQuotaStatus reportsEnforced=Falsewith theWebhookMissingreason. Deckhouse Platform finds the webhook of the module missing and puts it back by itself within about 4 minutes, so this buys time to fix the cause rather than switching the check off.d8 k --as=system:sudouser delete validatingwebhookconfiguration d8-multitenancy-manager-quota-ledgerCheck that the webhook is back:
d8 k get validatingwebhookconfiguration d8-multitenancy-manager-quota-ledger. -
Disable the
multitenancy-managermodule. The check goes away together with everything else the module does for projects.
Before draining the worker nodes of a cluster with project quotas (evicting the pods from the nodes with d8 k drain or on a node update), check that the ledger has a leader (d8_multitenancy_quota_ledger_leader) and that the D8ProjectQuotaLedgerCanaryFailing alert is not firing: while the ledger does not answer, the pods that replace the evicted ones are not created in the namespaces of projects with a quota.
Granting access within a project
To grant access to project namespaces for users beyond the project administrators, use role bindings that reference cluster-wide roles and fan out into the appropriate project namespaces automatically:
- ProjectRoleBinding (namespaced, short name
prb) — grants a role within a single project. It must be created in the project’s main namespace (the namespace whose name equals the project name). The controller creates a RoleBinding in every namespace of that project. - ClusterProjectRoleBinding (cluster-scoped, short name
cprb) — grants a role across all non-virtual projects. The controller creates a RoleBinding in every namespace of every project and reports the number of bound projects in.status.boundProjects.
roleRef must reference a ClusterRole whose name starts with one of the allowed prefixes (d8:project:, d8:namespace:, d8:project-capability:, d8:namespace-capability:, d8:custom:). The roles are described in the user-authz module documentation.
The following checks apply when bindings are created:
- Privilege escalation protection: a binding can only be created by a user who has the right to bind (
bind) the referenced role, checked by name. A project administrator (d8:project:admin) holdsbindon exactly eight roles:d8:project:viewer,d8:project:user,d8:project:manager,d8:project:admin,d8:namespace:viewer,d8:namespace:user,d8:namespace:managerandd8:namespace:admin. A binding to any other role — a customd8:custom:*role, a capability,d8:project:superadmin— is refused for the project administrator even when the role is narrower than their own permissions, because nothing grants thembindon that name. Such bindings are created by a cluster administrator, or the cluster administrator grantsbindon the custom role to the project administrators with a ClusterRole of their own. - The role must exist: a binding to a non-existent role is rejected.
- A ServiceAccount used as a subject of a ProjectRoleBinding must belong to a namespace of that same project. Its main namespace, its additional namespaces and a namespace a ProjectNamespace of the project has claimed qualify. A namespace that is only named like an additional namespace (
<project>-*) but belongs to a separate project does not. When the binding is updated, a ServiceAccount the stored binding already has is not refused again. If its namespace is no longer a namespace of the project, the response carries a warning that this ServiceAccount gets no rights from the binding. - System and subsystem roles (
d8:system:*,d8:subsystem:*), as well as arbitrary roles outside the listed prefixes, cannot be granted via project bindings. - Roles with the
rbac.deckhouse.io/disabled-for-direct-use-in-projects: "true"annotation are forbidden for granting in projects. A cluster administrator can put this annotation on a role to phase it out: existing bindings keep working, but new ones cannot be created. If such a role is used by a project template, the project switches to theErrorstate with an explanation in theTemplateRolesAllowedcondition.
The controller checks the ServiceAccounts by the same rule every time it updates the RoleBindings of a ProjectRoleBinding, and also when a namespace appears, is deleted or moves to another project. A subject that names ServiceAccounts goes into the RoleBindings only while their namespace is the main namespace of the project, or exists, is not being deleted and carries the projects.deckhouse.io/project label with the project name. Besides subjects of the ServiceAccount kind, this applies to the User subject named system:serviceaccount:<namespace>:<name> and the Group subject named system:serviceaccounts:<namespace>, which RBAC matches to the same ServiceAccounts. Other users and groups are not affected. So a ServiceAccount of a deleted additional namespace gets no rights when a separate project takes the name later.
The left-out subjects stay in .spec.subjects. The SubjectsBound condition of the binding is False and lists them, and the SubjectsLeftOut Warning event is sent when the list changes. The Ready condition stays True, as the binding keeps working for the other subjects. Once the namespace is a namespace of the project again, for example after the deleted ProjectNamespace is created again, the subjects get the rights back without a change to the binding. You can check the condition in the status of the binding:
d8 k -n my-project get projectrolebinding ci -o jsonpath='{.status.conditions[?(@.type=="SubjectsBound")]}'
The d8-administrators binding created by the controller from the .spec.administrators field is managed by the controller only — it cannot be edited manually. To change the set of administrators, change the .spec.administrators field of the project.
The ServiceAccount rule above does not apply to this binding. An administrator given as the User system:serviceaccount:<namespace>:<name> or the Group system:serviceaccounts:<namespace> gets the rights in every namespace of the project, whichever project that namespace belongs to, because only those allowed to change the Project set its administrators. Remove such an administrator from the project before its namespace is deleted, as the ServiceAccounts of a namespace created later under the same name would get these rights.
The following bindings give the viewer@example.com user the d8:project:viewer role in every namespace of my-project, and the platform group the same role in every project except the virtual ones:
---
apiVersion: deckhouse.io/v1alpha3
kind: ProjectRoleBinding
metadata:
name: viewers
namespace: my-project
spec:
subjects:
- kind: User
name: viewer@example.com
roleRef:
kind: ClusterRole
name: d8:project:viewer
---
apiVersion: deckhouse.io/v1alpha3
kind: ClusterProjectRoleBinding
metadata:
name: platform-viewers
spec:
subjects:
- kind: Group
name: platform
roleRef:
kind: ClusterRole
name: d8:project:viewer
Roles available in a RoleBinding inside a project
Besides the project bindings, a plain RoleBinding can also be used inside a project namespace — the role then applies in that single namespace only. However, in projects the set of roles available to a plain RoleBinding is restricted: only cluster roles carrying the rbac.deckhouse.io/delegatable: "true" label are allowed. Among the built-in ones these are the d8:namespace:* and d8:project:* roles, as well as the access-level roles of the basic role model (user-authz:user, user-authz:privileged-user, user-authz:editor, user-authz:admin).
A RoleBinding to any other cluster role (for example, cluster-admin, system roles, or capabilities) is rejected in a project with the message references "<role>" which is not available to project. This protects the project isolation from being bypassed by binding to an overly broad role.
To use a custom role in projects, add the rbac.deckhouse.io/delegatable: "true" label to it:
d8 k label clusterrole d8:custom:namespace:developer rbac.deckhouse.io/delegatable=true
The restriction applies in the namespaces of every project, including the ones adopted from a standalone namespace.
Structured templates
A deckhouse.io/v1alpha2 project template is described by structured fields. You declaratively specify which settings the project namespaces get. The controller itself creates the corresponding objects (network policies, security policies, log collection settings, etc.) from these fields in every namespace of the project and keeps them up to date. Other objects of the project namespaces are described in manifests.
Available fields (all optional; the complete reference is in the ProjectTemplate resource description):
| Field | What it configures |
|---|---|
podSecurityStandard |
Pod security profile: Privileged, Baseline, or Restricted. A parameter with an empty value sets no profile |
networkPolicy.mode |
Network isolation: Isolated (traffic is only allowed within the project and from the platform system components) or NotRestricted |
features.monitoring |
Extended monitoring of the project namespaces |
features.vulnerabilityScanning |
Scanning of container images for vulnerabilities |
logShipping.clusterDestinationRef |
Collecting the logs of the project pods into the given destination (ClusterLogDestination) |
nodeSelector, tolerations |
Placing the pods of the main namespace on dedicated nodes |
allowedUIDs, allowedGIDs |
The allowed UID/GID ranges of the project containers |
runtimeAudit.enabled |
Auditing the project processes’ access to the Linux kernel |
namespaceMetadata.labels, namespaceMetadata.annotations |
Extra labels and annotations of the project namespaces. The labels the module manages cannot be set here. A template or a Project that sets one is refused. One that already has such a label can still be changed while it keeps the label as it is. The label is not applied, and the response warns about it |
resources, grantPolicies |
Granting cluster-wide resources through a project template |
manifests |
Other objects in the project namespaces, applied with the rights of a project administrator. For details, see Describing other objects in manifests |
parametersSchema.openAPIV3Schema |
The schema of parameters set when creating a project |
An example of a structured template:
apiVersion: deckhouse.io/v1alpha2
kind: ProjectTemplate
metadata:
name: my-template
spec:
title: "Team template"
description: "Isolated project with monitoring"
podSecurityStandard: Baseline
networkPolicy:
mode: Isolated
features:
monitoring: true
vulnerabilityScanning: true
Template parametrization
The following fields of the template can take a parameter: podSecurityStandard, networkPolicy.mode, features.monitoring, features.vulnerabilityScanning, logShipping.clusterDestinationRef, nodeSelector, tolerations, allowedUIDs, allowedGIDs, runtimeAudit.enabled, namespaceMetadata.labels and namespaceMetadata.annotations. Instead of a concrete value, specify {fromParam: <parameter name>} and declare the parameter in parametersSchema. The value does not have to be a scalar: a map (nodeSelector, namespaceMetadata.labels), a list (tolerations) or an object (allowedUIDs) works just as well. The other fields — title, description, resources, grantPolicies and parametersSchema itself — take literal values only. Each project then sets its own value in .spec.parameters; if the value is not set, the default from the schema is used.
apiVersion: deckhouse.io/v1alpha2
kind: ProjectTemplate
metadata:
name: my-parametrized-template
spec:
podSecurityStandard:
fromParam: securityProfile
networkPolicy:
mode:
fromParam: networkMode
parametersSchema:
openAPIV3Schema:
type: object
properties:
securityProfile:
type: string
enum: [Baseline, Restricted]
default: Baseline
networkMode:
type: string
enum: [Isolated, NotRestricted]
default: Isolated
A project using such a template:
apiVersion: deckhouse.io/v1alpha3
kind: Project
metadata:
name: my-project
spec:
projectTemplateName: my-parametrized-template
parameters:
securityProfile: Restricted
A template whose parametersSchema still declares the administrators and resourceQuota parameters keeps accepting projects. A schema declares them by name, or through an additionalProperties that takes every parameter it does not name, as long as it takes these two in the form a v1alpha2 Project holds them. The project parameters are validated against such a schema together with .spec.administrators and .spec.quota, laid out again as parameters.administrators and parameters.resourceQuota the way the conversion webhook lays them out for v1alpha2. A quota value is checked in the first form the schema takes among its canonical spelling (for example 1Gi for 1024Mi), a number (for example 1000 for 1k) and a spelling with another unit (for example 2000m for 2 when the schema has pattern: ^[0-9]+m$), in that order. A required field that the project leaves empty is checked in its empty form ([] for the administrators, {} for the quota).
A Helm resourcesTemplate and manifests receive the parameters laid out the same way in .parameters, so a text written for v1alpha2 projects keeps rendering its objects from them. A quota value reaches the text in the form it is checked in, so for a schema that takes strings a zero is the string "0", which with and if take as set. The structured fields do not receive these values as parameters.
A schema that neither names the administrators and resourceQuota parameters nor takes unnamed parameters through additionalProperties does not declare them, and the schemas of the built-in templates do not declare them either. On a template with such a schema, these parameters grant no access and set no quota. A project that holds them, for example an empty list or a quota with the number 0 that the conversion webhook leaves in parameters, keeps working and can be edited.
An update that keeps such a value as it was, including on a move to another template that does not declare them either, is accepted with a warning, and so are an empty list and a quota with no value other than the number 0. Any other new or changed value is rejected. Set the administrators in .spec.administrators and the quota in .spec.quota instead. In a v1alpha2 manifest, write them in a form that the conversion webhook lifts into these fields. Any other parameter that the schema does not declare is still rejected.
The fromParam references are checked when a template is created or changed: a reference to an undeclared parameter or to a parameter of an incompatible type (for example, a string parameter for a boolean field) is rejected.
A declared parameter is type-checked even where the schema allows undeclared keys with additionalProperties or x-kubernetes-preserve-unknown-fields. A reference to an undeclared key is accepted only in such places, and the type of such a key is not checked.
The requiredRequests parameter is read by name, without fromParam. When the template declares it in parametersSchema with type: boolean and its value for the project is true, the controller renders an OperationPolicy in the Deny mode that requires CPU and memory requests from the pods in every namespace of the project. A template that does not declare the parameter renders no such policy, even if its schema allows additionalProperties and the project sets requiredRequests as an undeclared key. To keep the policy on unless a project turns it off, declare the parameter with default: true, as the default, secure and secure-with-dedicated-nodes templates do:
spec:
parametersSchema:
openAPIV3Schema:
type: object
properties:
requiredRequests:
type: boolean
default: true
Template checks
The following rules apply to template operations:
- A template cannot be deleted while at least one project uses it or a project namespace is still rendered from it. After a project switches to another template, the old template stays until the new one has rendered.
- A change to a template is automatically applied to all projects created from it.
- Instead of structured fields, a template can describe its objects with a Helm text in
resourcesTemplate. Such a template is rendered through Helm with the project name (.projectName), the template name (.projectTemplateName) and the project parameters (.parameters), andresourcesTemplatecannot be set together with the fields that render namespace objects or withmanifests. A template that projects use keeps a non-emptyresourcesTemplateuntil a change sets such fields ormanifestsin its place. What such a change, or a move of a project to another template, does with the objects the Helm text rendered is described in Switching a project from a Helm text. A project whose parameters turn into objects or keys of the rendered manifests instead of values is refused, so quote every substitution. This form is deprecated, andresourcesTemplategoes away fromdeckhouse.io/v1alpha2together withdeckhouse.io/v1alpha1in a future release. TheMultitenancyManagerHelmProjectTemplatealert names every template that still renders from a Helm text, except the built-in ones. Describe new templates with structured fields andmanifests, and move the existing ones as Moving a template from a Helm text describes. - A template cannot get a new
resourcesTemplatethrough either version. A create with a non-emptyresourcesTemplateand an update that sets it on a template whoseresourcesTemplateis empty are refused. A template that has a Helm text keeps it, and the text can be changed through either version whiledeckhouse.io/v1alpha1can describe the whole template. So a template with a Helm text cannot be created again, for example from a manifest exported from another cluster, from a backup, or from Git after the template was deleted. Move such a template to structured fields andmanifestsfirst. - The
deckhouse.io/v1alpha1version of ProjectTemplate is served and deprecated, and the API server returns a warning on every request through it. A template is created throughdeckhouse.io/v1alpha2only, and a create throughdeckhouse.io/v1alpha1is refused. The templates created throughdeckhouse.io/v1alpha1earlier are read, changed and deleted through it as before. This version has thedescription,parametersSchemaandresourcesTemplatefields, anddeckhouse.io/v1alpha2keeps all three, so a template written asv1alpha1reads back unchanged. A change throughv1alpha1to a template that has fieldsv1alpha1cannot describe, such astitle,grantPolicies,podSecurityStandardormanifests, is refused, because it would erase them. Change such a template throughdeckhouse.io/v1alpha2.
Describing other objects in manifests
The structured fields cover the settings that most projects need.
Other objects of the project namespaces, such as workloads, ConfigMap, RoleBinding or LimitRange objects, are described in manifests.
It is a text in the go template format with the helm functions, like resourcesTemplate, and a template can set it together with the structured fields.
manifests and resourcesTemplate differ in whose rights create the objects.
The controller installs a Helm text from resourcesTemplate as a Helm release with its own rights, which cover the whole cluster.
So such a text can create any cluster-scoped object other than a namespace, and any object in the namespaces of the project.
The controller applies the objects of manifests as a user of the project that holds the d8:project:admin role in the namespaces of the project.
A template thus gets the rights of an administrator of the project, not the rights of the controller.
The text gets the project name in .projectName, the template name in .projectTemplateName and the project parameters completed with the defaults of parametersSchema in .parameters, the same values that resourcesTemplate gets.
A project whose parameters turn into objects or keys of the rendered manifests instead of values is refused, so quote every substitution.
The text is rendered without access to the cluster, so the lookup function finds nothing.
A project template can be read by users who have no access to its projects, the ones with the User access level included, so do not put secret values into manifests.
The objects are applied as follows:
- The controller creates the
d8-manifestsProjectRoleBinding in the main namespace of the project. It grants thed8:project:adminrole to thesystem:multitenancy-manager:project:<PROJECT_NAME>user in every namespace of the project. - The objects are applied once this binding is
Readyfor its current generation and thed8:project:adminClusterRole has its aggregated rules. Until then theManifestsAppliedcondition of the project says what the objects wait for, and the project staysDeployed. - Each object is applied with server-side apply as this user, with the
multitenancy-manager/manifestsfield manager. An object withoutmetadata.namespacegoes to the main namespace of the project. An object can also go to an additional namespace of the project, including one that a ProjectNamespace of the samemanifestsmakes, as Additional namespaces in manifests describes. - Each object gets the
projects.deckhouse.io/project: <PROJECT_NAME>andprojects.deckhouse.io/source: manifestslabels, and.status.manifestObjectsof the project lists the objects. - An object that exists already is applied only when it belongs to the project, that is, when
.status.manifestObjectslists it, when a switch from a Helm text left it in.status.keptObjects, or when it has both labels. Any other object is left alone, as Limitations of manifests describes. - The objects do not get the
heritage: multitenancy-managerlabel. They get theprojects.deckhouse.io/source: manifestslabel instead, and like the objects of the structured fields and of a Helm text, they are changed through the template only. Thed8-multitenancy-manager-manifests-objectsadmission policy refuses to create, change or delete such an object to everyone, project and cluster administrators included. Only the user themanifestsof the project are applied as, the controller, the service accounts of thed8-*namespaces and the cluster components can do it. Anyone who may update a workload may restart its rollout, when thekubectl.kubernetes.io/restartedAtannotation of its pod template is the only change.metadata.resourceVersion,metadata.generationandmetadata.managedFieldsare not compared, because the API server sets them on the write. Within 10 minutes the controller applies the objects again, which puts back the fields that the template sets if something else changed them. - An object that the template no longer renders is deleted as the same user after the other objects are applied, and only when none of them was refused or left alone. It is deleted through the version that the API server serves its kind at, also when it was applied through another one. An object that loses one of the two labels, and an object whose namespace left the project, leave the list and stay in place. Only the service accounts of the
d8-*namespaces, the controller among them, and the cluster components can take a label off. The user themanifestsare applied as can take off only theprojects.deckhouse.io/sourcelabel. - When one change makes the project release render an object that
manifestsapplied, under the same kind, namespace and name, the release takes the object over, and the object stays in the cluster. Until the upgrade finishes,.status.handingOverlists the object and.status.manifestObjectsdoes not. If the upgrade does not deploy the object, it goes back tomanifests, and.status.manifestObjectslists it again. - When the same change makes the release render another object that exists without the ownership marks of the release and is not an object of
manifests, Helm cannot take that object over. The ownership marks are theapp.kubernetes.io/managed-by: Helmlabel and themeta.helm.sh/release-nameannotation with the name of the project release. The controller does not run the upgrade then and leaves the objects ofmanifestsunchanged, and theResourcesUpgradedcondition of the project names the object that blocks the upgrade. The handover goes on once that object is deleted or gets the ownership marks. - When the template drops
manifestsor the project moves to a template without it, the objects are deleted, and then thed8-manifestsbinding is deleted. When the project is deleted, the binding is deleted first, and the objects go with the namespaces.
Only the controller binds the users whose names start with system:multitenancy-manager:.
A RoleBinding, ClusterRoleBinding, ProjectRoleBinding, ClusterProjectRoleBinding, AuthorizationRule or ClusterAuthorizationRule that names such a user or group is refused for everyone else, except the garbage collector and system:sudouser (see Who bypasses admission).
A Helm text in resourcesTemplate can also bind such a user, because the controller installs it with its own rights.
Apart from that, the objects of a template get the rights of the d8-manifests binding and the rights that the cluster grants to every authenticated user (the system:authenticated group).
For the same reason a project cannot name such a user or group among its administrators, in .spec.administrators or in the administrators parameter. A name that the project already has is accepted with a warning, so the project stays editable and the name can be removed.
The following template sets the pod security profile and the network isolation with structured fields and creates two objects in manifests, a LimitRange with the default limits from the project parameters and a ConfigMap with the names of the project and of its template:
apiVersion: deckhouse.io/v1alpha2
kind: ProjectTemplate
metadata:
name: team-with-limits
spec:
podSecurityStandard: Baseline
networkPolicy:
mode: Isolated
parametersSchema:
openAPIV3Schema:
type: object
properties:
defaultCPU:
type: string
default: 500m
defaultMemory:
type: string
default: 512Mi
manifests: |
apiVersion: v1
kind: LimitRange
metadata:
name: container-defaults
spec:
limits:
- type: Container
default:
cpu: {{ .parameters.defaultCPU | quote }}
memory: {{ .parameters.defaultMemory | quote }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: project-info
data:
project: {{ .projectName | quote }}
template: {{ .projectTemplateName | quote }}
Once the ManifestsApplied condition is True, the main namespace of every project of the template has the container-defaults LimitRange and the project-info ConfigMap.
To check the condition and to list the applied objects, use the commands:
d8 k get project <PROJECT_NAME> -o jsonpath='{.status.conditions[?(@.type=="ManifestsApplied")]}'
d8 k get project <PROJECT_NAME> -o jsonpath='{range .status.manifestObjects[*]}{.kind} {.namespace}/{.name}{"\n"}{end}'
Additional namespaces in manifests
manifests can make additional namespaces of the project with ProjectNamespace objects and put objects into them in the same text. A ProjectNamespace without metadata.namespace goes to the main namespace of the project, and its namespace is named <PROJECT_NAME>-<spec.name>, so an object in it names the namespace as {{ .projectName }}-<spec.name>.
The namespace does not exist on the first render, so the objects are applied in two steps:
- The ProjectNamespace objects and the objects of the namespaces that exist are applied at once.
- The objects of a namespace that a ProjectNamespace of the same
manifestsmakes wait until the namespace exists and thed8-manifestsbinding has reached it. Until then theManifestsAppliedcondition isFalseand names what the objects wait for, and the project staysDeployed. A namespace that is being deleted is reported as terminating, and one that is not there yet as still to be created. When the ProjectNamespace is notReady, the condition includes itsReady=Falsemessage. Objects that wait are kept. Objects thatmanifestsno longer renders are deleted. The project is reconciled when the namespace is created and when the binding reaches it, so the objects are applied without any action. - When
manifestsdrops a ProjectNamespace, the ProjectNamespace is deleted, and its namespace goes with everything in it. The objects ofmanifestsin that namespace leave.status.manifestObjectswithout a delete of their own.
An object in a namespace that is neither a namespace of the project nor made by a ProjectNamespace of the same manifests is refused, as Limitations of manifests describes. A create or a change of a project whose manifests render such an object is admitted with a warning that names the namespace.
The following manifests make the backend namespace and put a ConfigMap into it:
manifests: |
apiVersion: deckhouse.io/v1alpha3
kind: ProjectNamespace
metadata:
name: backend
spec:
name: backend
---
apiVersion: v1
kind: ConfigMap
metadata:
name: backend-settings
namespace: "{{ .projectName }}-backend"
data:
team: {{ .parameters.team | quote }}
Limitations of manifests
manifests covers only what an administrator of the project can do in its namespaces.
The following limitations apply:
- A text that does not render for a project, for example because of a syntax error or a
requiredvalue that the project does not set, is accepted when the template is written, because the template is not rendered at that point. Such a project switches toError,ManifestsAppliednames the error, and a create or a change of the project is refused until the text renders. A change ofmanifestsreaches every project of the template at once, so try it first on a copy of the template with one project. - A cluster-scoped object, an object in a namespace that is neither a namespace of the project nor made by a ProjectNamespace of the same
manifests, an object with theheritage: multitenancy-managerorheritage: deckhouselabel, an object withoutmetadata.nameand an object that the text renders twice are refused. While the render holds such an object, none of its objects is applied, the project switches toError, and theManifestsAppliedcondition names the object. - An object that exists already and does not belong to the project, for example one that a project administrator created by hand, is not changed, listed or deleted later. The other objects are applied, but nothing is deleted, the project switches to
Error,ManifestsAppliednames the object, and the controller looks again every minute. To hand the object over tomanifests, give it theprojects.deckhouse.io/project: <PROJECT_NAME>andprojects.deckhouse.io/source: manifestslabels, or delete it so thatmanifestscreates it. Only a user who may update objects of its kind in every namespace, such as a cluster administrator, may give it the labels. Do not delete an AuthorizationRule of theAdminaccess level this way, becausemanifestsdoes not create one. Give it the labels instead. - An object that the
d8:project:adminrole does not allow is refused by the API server with403 Forbidden. Examples are a RoleBinding to a role that a project administrator cannot bind and an object of a kind that the role does not cover. The other objects are applied, the project switches toError,ManifestsAppliednames the refused object, and the controller tries again with a growing interval. - An AuthorizationRule of the
Adminaccess level is refused whenmanifestscreates it, raises an existing one to this level or adds subjects to an existing one of this level, although thed8:project:adminrole allows it. The administrators of a project are set in.spec.administratorsof the Project. An AuthorizationRule of theUser,PrivilegedUserorEditorlevel is applied as other objects are. The project switches toErroras for an object that the role does not allow, and a create or a change of a project whosemanifestsrender a refused AuthorizationRule is admitted with a warning that names it. A create of a template whosemanifestsdescribe an AuthorizationRule, and a change of suchmanifests, are admitted with a warning about this refusal. An AuthorizationRule that exists already, for example one that a switch from a Helm text left in place, is taken over and updated as other objects are, unlessmanifestsraises it to theAdminlevel or adds subjects to it at this level. - An object of a kind that the API server does not serve keeps all the objects from being applied. The project switches to
Error, and the controller tries again every minute, for example while the module that brings the kind is being enabled. manifestscannot be set together withresourcesTemplate.deckhouse.io/v1alpha1has no place formanifests, so a write throughdeckhouse.io/v1alpha1to a template that has it is refused.- The
projects.deckhouse.io/skip-heritage-labelandprojects.deckhouse.io/unmanagedlabels of a Helm text have no effect on the objects ofmanifests.
manifests does not create cluster-scoped objects.
To give the projects of a template access to cluster-wide resources, use resources and grantPolicies.
A cluster administrator creates other cluster-scoped objects outside the template.
Moving a template from a Helm text
A template with a Helm text in resourcesTemplate keeps working, but this form is deprecated.
resourcesTemplate goes away from deckhouse.io/v1alpha2 together with deckhouse.io/v1alpha1 in a future release, and no template can get a new one (see Template checks).
The MultitenancyManagerHelmProjectTemplate alert names every such template, except the built-in ones.
Nothing in the cluster rewrites such a template, so move it to structured fields and manifests where it is applied from, for example in Git or in CI.
The built-in templates default, secure and secure-with-dedicated-nodes are structured and accept the same parameters as their Helm texts did, except administrators and resourceQuota, so they show how a text maps to the fields.
A field takes the value of a parameter with {fromParam: <PARAMETER_NAME>}, so the projects keep their parameters.
The following table shows where the objects of such a text go:
| Object of the Helm text | Replacement |
|---|---|
The security.deckhouse.io/pod-policy label of the namespace from the podSecurityProfile parameter |
podSecurityStandard: {fromParam: podSecurityProfile} |
The NetworkPolicy objects of the networkPolicy parameter |
networkPolicy.mode: {fromParam: networkPolicy} |
The extended-monitoring.deckhouse.io/enabled label of the extendedMonitoringEnabled parameter |
features.monitoring: {fromParam: extendedMonitoringEnabled} |
The security-scanning.deckhouse.io/enabled label of the securityScanningEnabled parameter |
features.vulnerabilityScanning: {fromParam: securityScanningEnabled} |
The PodLoggingConfig of the clusterLogDestinationName parameter |
logShipping.clusterDestinationRef: {fromParam: clusterLogDestinationName} |
The SecurityPolicy of the allowedUIDs and allowedGIDs parameters |
allowedUIDs: {fromParam: allowedUIDs} and allowedGIDs: {fromParam: allowedGIDs} |
The FalcoAuditRules of the runtimeAuditEnabled parameter |
runtimeAudit.enabled: {fromParam: runtimeAuditEnabled} |
| Other labels and annotations of the namespace | namespaceMetadata.labels and namespaceMetadata.annotations, in the built-in templates from the namespace.labels and namespace.annotations parameters |
| The placement on dedicated nodes | nodeSelector and tolerations, in the built-in templates from the dedicatedNodes.nodeSelector and dedicatedNodes.defaultTolerations parameters |
The AuthorizationRule objects of the administrators parameter |
.spec.administrators of each Project |
Other AuthorizationRule objects of the Admin access level |
.spec.administrators of each Project, because manifests does not create an AuthorizationRule of this level |
The all-pods ResourceQuota of the resourceQuota parameter |
.spec.quota of each Project |
The required-requests-* OperationPolicy, which refuses pods without CPU and memory requests |
The requiredRequests parameter, declared with type: boolean and default: true to keep the policy or with default: false to leave it off |
| Any other object in the namespaces of the project, such as a workload, a ConfigMap, a RoleBinding, a LimitRange or an AuthorizationRule of another access level | manifests |
| A cluster-scoped object | No replacement in a template. Access to cluster-wide resources is set with resources and grantPolicies |
For example, an existing template team has the following Helm text, which sets the pod security profile from a parameter and creates a LimitRange in the main namespace of each project. Such a template cannot be created again, so the manifest only shows the starting point:
apiVersion: deckhouse.io/v1alpha1
kind: ProjectTemplate
metadata:
name: team
spec:
parametersSchema:
openAPIV3Schema:
type: object
properties:
podSecurityProfile:
type: string
enum: [Baseline, Restricted]
default: Baseline
resourcesTemplate: |
apiVersion: v1
kind: Namespace
metadata:
name: {{ .projectName | quote }}
labels:
security.deckhouse.io/pod-policy: {{ .parameters.podSecurityProfile | lower | quote }}
---
apiVersion: v1
kind: LimitRange
metadata:
name: container-defaults
namespace: {{ .projectName | quote }}
spec:
limits:
- type: Container
default:
cpu: 500m
memory: 512Mi
In the structured form, the profile goes to podSecurityStandard and the LimitRange goes to manifests.
The text did not render the required-requests-* OperationPolicy, so the template declares the requiredRequests parameter with default: false:
apiVersion: deckhouse.io/v1alpha2
kind: ProjectTemplate
metadata:
name: team
spec:
podSecurityStandard:
fromParam: podSecurityProfile
parametersSchema:
openAPIV3Schema:
type: object
properties:
podSecurityProfile:
type: string
enum: [Baseline, Restricted]
default: Baseline
requiredRequests:
type: boolean
default: false
manifests: |
apiVersion: v1
kind: LimitRange
metadata:
name: container-defaults
spec:
limits:
- type: Container
default:
cpu: 500m
memory: 512Mi
After the switch, the namespace of each project keeps its label, the LimitRange stays in place, and manifests takes it over and removes the annotations of Helm from it.
A template that no project uses can be deleted, and the alert goes with it.
If the template is an unchanged copy of a built-in template, move its projects to that template in .spec.projectTemplateName instead, and then delete the copy.
To move a template that projects use, follow these steps:
-
Find the projects of the template that still set
administratorsorresourceQuotain.spec.parameters:d8 k get projects -l projects.deckhouse.io/project-template=<PROJECT_TEMPLATE_NAME> -o json | jq -r '.items[] | select((.spec.parameters // {}) | has("administrators") or has("resourceQuota")) | .metadata.name'These values get no standard fields and are not handed over. Move them to
.spec.administratorsand.spec.quotafirst, and only as far as they mean the same there..spec.administratorsgives users and groups thed8:project:adminrole, so an administrator with another access level or another kind of subject needs a ProjectRoleBinding of its own..spec.quotaenforces every limit it holds,0included, so leave out a limit that the text did not render, such as a0that it skipped. -
Make every tool that writes the template, such as a GitOps tool or a generator, write it as
deckhouse.io/v1alpha2. Once the template has a field thatdeckhouse.io/v1alpha1cannot describe, such asmanifests, a write of the template throughdeckhouse.io/v1alpha1is refused, because it would erase that field. -
Write the template as
deckhouse.io/v1alpha2with the fields from the table and withmanifests, withoutresourcesTemplate, in one change. Keep inparametersSchemathe parameters that the projects set. Declare therequiredRequestsparameter withtype: boolean, withdefault: trueif the text rendered therequired-requests-*OperationPolicy and withdefault: falseif it did not, so the projects keep refusing or admitting pods without CPU and memory requests as before. A template that projects use cannot loseresourcesTemplatewithout structured fields ormanifestsin its place. -
Check each project after the switch. Its
HandoverPendingcondition is removed once the replaced objects are deleted, itsManifestsAppliedcondition says whether the objects ofmanifestsare applied, and.status.keptObjectslists the objects left in place:d8 k get project <PROJECT_NAME> -o jsonpath='{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}{range .status.keptObjects[*]}{.kind} {.namespace}/{.name}{"\n"}{end}' -
Decide on each object that stays in
.status.keptObjects, as Switching a project from a Helm text describes. An object thatmanifestsrenders under the same kind, namespace and name leaves the list oncemanifeststakes it over, and from then on it has whatmanifestsrenders.
Switching a project from a Helm text
A project release rendered from a Helm text in resourcesTemplate holds whatever the text rendered. The project switches to structured fields when its template gets structured fields or manifests in place of resourcesTemplate, when the project moves to a structured template, and when a built-in template that was a Helm text is rewritten with structured fields. The release then holds only what the structured fields render. An object of the Helm text that the structured fields render under the same kind and name stays in the release and is updated in place. The other objects of the Helm text are handled as follows, and of them only an AuthorizationRule that names nobody from .spec.administrators any more is deleted at once.
The AuthorizationRule and ResourceQuota objects that the standard fields replace go through a handover, so the administrators never lose access and the namespace never loses its quota. These are the AuthorizationRule objects the release created that grant the Admin access level and nothing else to users and groups who are all administrators of the project, and the ResourceQuota objects of the main namespace without scopes whose every limit .spec.quota sets to the same value or a lower one. The handover goes in the following order:
- The Project gets the
HandoverPendingcondition set toTrue. Then the objects get thehelm.sh/resource-policy: keepannotation and theprojects.deckhouse.io/kept-by-handover: "true"annotation, so the release upgrade does not delete them. - The
d8-administratorsProjectRoleBinding and thed8-project-quotaResourceQuota are created before the release is upgraded. - The kept objects are deleted once
d8-administratorsnames the administrators from.spec.administratorsand isReadyfor its currentmetadata.generation, thed8:project:adminClusterRole has its aggregated rules, andd8-project-quotahas the limits from.spec.quotaand is calculated, and not earlier than 2 minutes afterd8-administratorsbecameReadyand after the handover began. Until then the administrators hold both roles, and the tighter of the quotas applies. TheHandoverPendingcondition says what the handover waits for, and it is removed once the kept objects are deleted.
A kept AuthorizationRule that names nobody from .spec.administrators any more grants only what was revoked, and it is deleted at once. A subject removed from .spec.administrators is removed at once from a kept AuthorizationRule that names other administrators too. A kept quota that .spec.quota no longer covers when the handover finishes, because a limit in .spec.quota was raised above it or removed, is left in place like the objects below. The HandoverPending condition is kept in the status of the Project, so a replace of the Project (for example, kubectl replace or Argo CD with the Replace=true option) does not lose it. If the kept objects are still there after 30 minutes, the MultitenancyManagerProjectHandoverStuck alert names the project and says what to check.
Every other object of the Helm text that the structured fields do not render is left in place instead of being deleted. Examples are AuthorizationRule objects of other subjects or of another access level, such as the ones a text renders for roles from its own parameters, a quota that .spec.quota does not cover, RoleBinding objects, network policies, and workloads with their volumes.
- Such an object keeps working as before. A rule or a binding keeps granting access, a quota keeps applying, a workload keeps running.
- The project no longer manages it. The object gets the
helm.sh/resource-policy: keepannotation and theprojects.deckhouse.io/kept-from-helm-template: "true"label, and it loses theheritage: multitenancy-managerlabel. While it carries the label, only a requester who may make the same change to objects of its kind across the cluster may change or delete it, for example a cluster administrator. The users of the project may not, so a network policy or a quota of the text keeps holding them. The protection does not cover thescaleandstatussubresources: whoever may scale objects of the kind or change their status in the namespace may still do so for a kept object, which matters only for a kept workload. - The Project lists it in
.status.keptObjects, and theMultitenancyManagerProjectKeepsHelmTemplateObjectsalert names the project. An object leaves the list once it is deleted or the label is removed from it. When the template renders an object of the same kind and name again, the project release takes the object over. The controller then removes the label, puts theheritage: multitenancy-managerlabel back and removes thehelm.sh/resource-policy: keepannotation, unless the template sets it itself. - When
manifestsof the template renders an object of the same kind, namespace and name, the controller applies it as the user of the project and takes the object over. The object then has whatmanifestsrenders. A field that the Helm text set andmanifestsdoes not set is removed, the annotations of Helm included, such ashelm.sh/resource-policy: keepandmeta.helm.sh/*, and so is theprojects.deckhouse.io/kept-from-helm-templatelabel. A field that somebody else set on the object stays. The object leaves.status.keptObjects. - When the project is deleted, the object goes with it. An object in a namespace of the project goes with the namespace, and the controller deletes a cluster-scoped object that still carries the label.
Describe what is still needed in the template or in the project, for example with a ProjectRoleBinding or with a limit in .spec.quota, and delete the left object, or remove the label to hand the object over to the project, after which the users of the project may change and delete it as well. An object in a namespace of the project can instead be described in manifests of the template, which takes it over. A cluster-scoped object without the label also stays when the project is deleted. An access that a left object grants ends only when the object is deleted. To list the left objects of a project, use the command:
d8 k get project <PROJECT_NAME> -o jsonpath='{range .status.keptObjects[*]}{.kind} {.namespace}/{.name}{"\n"}{end}'
An edit of a Helm text, and a move of a project to another template with a Helm text, delete the objects the new text does not render, except the AuthorizationRule and ResourceQuota objects that the standard fields replace. These go through the same handover.
Administrators and a quota that are still set in .spec.parameters, because they could not be moved to .spec.administrators and .spec.quota, get no standard fields. Their AuthorizationRule objects and their quota are not handed over, and a switch leaves them in place. Move such values to the standard fields before the switch.
After the handover the administrators have the d8:project:admin role in place of the Admin access level. The two roles do not grant the same permissions. In particular, the access that custom ClusterRoles with the user-authz.deckhouse.io/access-level annotation added to the Admin access level is not part of d8:project:admin. To grant such access, create a custom role and bind it with a ProjectRoleBinding.
Moving an object of the release to manifests
A structured template renders its objects into the project release. When one change makes manifests render an object the release contained, under the same kind, namespace and name, the upgrade would delete the object and manifests would create it again. The project would be without the object in between. An example is the NetworkPolicy named isolated once networkPolicy.mode is removed and manifests render that NetworkPolicy.
The controller leaves the object in place instead. Before the upgrade it sets the helm.sh/resource-policy: keep annotation and the projects.deckhouse.io/kept-from-helm-template: "true" label, and lists the object in .status.keptObjects, the same way a switch from a Helm text leaves an object. In the same reconcile manifests apply the object as the user of the project, remove the label and take the object off the list. The object then has what manifests render. A field that the release set and manifests do not set is removed. A field that somebody else set stays.
If manifests cannot apply the object, for example because the API server refuses the kind or returns 403 Forbidden, the object stays in .status.keptObjects. The MultitenancyManagerProjectKeepsHelmTemplateObjects alert names the project. The object keeps working, and only a requester who may make the same change to objects of its kind across the cluster may change or delete it, until manifests take it over or the label is removed.
An object the release drops and that manifests do not render is deleted by the upgrade.
Creating your own project template
To create your own template:
- Take one of the default templates as a basis, for example,
default. -
Copy it to a separate file, for example,
my-project-template.yamlusing the command:d8 k get projecttemplates default -o yaml > my-project-template.yaml - Edit the
my-project-template.yamlfile and adjust the structured fields, the objects inmanifestsand the input parameters schema to your needs. - Change the template name in the
.metadata.namefield. - Remove the
heritagelabel, themetadata.resourceVersion,metadata.uid,metadata.creationTimestamp,metadata.generationandmetadata.managedFieldsfields and thestatussection from the file. -
Apply your new template with the command:
d8 k apply -f my-project-template.yaml -
Check the availability of the new template with the command:
d8 k get projecttemplates <NEW_TEMPLATE_NAME>
Labels for the objects of a Helm text
The following labels change how the module handles the objects of a Helm text in resourcesTemplate. Only existing templates have such a text, because no template can get a new resourcesTemplate. The objects of manifests never get the heritage: multitenancy-manager label, and the module does not read these labels on them.
Skipping the heritage label
By default, every object of the Helm text gets the heritage: multitenancy-manager label. With this label, only the identities listed in Who bypasses admission can change the object.
To allow changes to the object (for example, for compatibility with other systems, or to control the created objects yourself), add the projects.deckhouse.io/skip-heritage-label label to it.
Example:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config
namespace: {{ .projectName }}
labels:
projects.deckhouse.io/skip-heritage-label: "true"
app: my-app
data:
key: value
In this case, the object gets the projects.deckhouse.io/project and projects.deckhouse.io/project-template labels, but not the heritage: multitenancy-manager label.
Excluding objects from management
To exclude an object from management by the module (for example, when it is managed manually or by another controller), add the projects.deckhouse.io/unmanaged label to it.
Example:
---
apiVersion: v1
kind: Secret
metadata:
name: external-secret
namespace: {{ .projectName }}
labels:
projects.deckhouse.io/unmanaged: "true"
type: Opaque
data:
token: <base64-encoded-value>
Objects with the projects.deckhouse.io/unmanaged label:
- are created only once, when the project release is first installed, so an object added to the template later is not created in existing projects;
- are not updated when the template changes;
- are not tracked in the project status;
- get the
projects.deckhouse.io/projectandprojects.deckhouse.io/project-templatelabels, but not theheritage: multitenancy-managerlabel; - get the
helm.sh/resource-policy: keepannotation, so the project release does not delete them.
After it is created, an unmanaged object does not depend on the template and must be managed manually.
Implementing validation of object changes with a custom label
The multitenancy-manager module uses a ValidatingAdmissionPolicy to protect objects labeled heritage: multitenancy-manager from manual changes.
You can implement similar validation for objects with any label.
How validation works in multitenancy-manager
Validation occurs for objects labeled heritage: multitenancy-manager.
The following components are used for this:
- ValidatingAdmissionPolicy: Defines validation rules:
- Operations:
CREATE,UPDATEandDELETE.system:mastersis not exempt. - Check: only the module’s controller and the identities this policy lets through (for details, see Who bypasses admission) may change such an object. For a project namespace, an update that changes only labels and annotations outside the keys the module owns is allowed as well (see Creating a project automatically for a namespace).
- Applies to all resources and API groups.
- Operations:
- ValidatingAdmissionPolicyBinding: Defines which objects the validation applies to:
- Uses
namespaceSelectorandobjectSelectorto select resources by the labelheritage: multitenancy-manager.
- Uses
Who bypasses admission
The webhooks and policies of the module have different exclusion lists:
- The Project and ProjectTemplate webhooks skip requests from platform components — the API server (
system:apiserver), the service accounts of Deckhouse (d8-system:deckhouse), of this module (d8-multitenancy-manager:multitenancy-manager) and ofuser-authz(d8-user-authz:controller), every service account of thed8-system,kube-systemandd8-user-authznamespaces, kubelets (system:nodes) — and from thesystem:sudouseridentity. Everyone else is validated,system:mastersincluded. - The ProjectRoleBinding, ProjectNamespace and ClusterProjectRoleBinding webhooks skip nobody and validate every request. Inside them only the controller and Deckhouse service accounts are privileged;
system:sudouserandsystem:mastersare ordinary users there. - The ValidatingAdmissionPolicy above lets through this module’s controller; the users
system:apiserver,system:kube-controller-manager,system:kube-scheduler,system:volume-scheduler,dhctl,observabilityandsystem:sudouser; the groupssystem:nodes,system:serviceaccounts:kube-systemandsystem:serviceaccounts:d8-system(notd8-user-authz); and a rollout restart of a project workload that changes only thekubectl.kubernetes.io/restartedAtannotation of the pod template. Everyone else is refused,system:mastersincluded. - The
d8-multitenancy-manager-manifests-objectsValidatingAdmissionPolicy guards objects labelledprojects.deckhouse.io/source: manifests. It lets through this module’s controller; the userssystem:apiserver,system:kube-controller-manager,system:kube-scheduler,system:volume-scheduler,dhctl,observabilityandsystem:sudouser; and the groupssystem:nodes,system:serviceaccounts:kube-systemandsystem:serviceaccounts:d8-system. Every service account whose name starts withsystem:serviceaccount:d8-may create, change and delete such objects. The usersystem:multitenancy-manager:project:<PROJECT_NAME>may create, change and delete the objects of that project. A rollout restart that changes only thekubectl.kubernetes.io/restartedAtannotation of the pod template is allowed. An update that puts theprojects.deckhouse.io/source: manifestslabel, together with theprojects.deckhouse.io/projectlabel, on an object that does not have theprojects.deckhouse.io/source: manifestslabel yet is allowed to a requester who may update that resource across the cluster. Everyone else is refused,system:mastersincluded. - The ValidatingAdmissionPolicy that reserves the users the controller acts as (the names that start with
system:multitenancy-manager:, see Describing other objects in manifests) lets through this module’s controller, the garbage collector, which runs assystem:serviceaccount:kube-system:generic-garbage-collectoror assystem:kube-controller-manager, andsystem:sudouser. Everyone else who names such a user or group in a binding is refused, the other service accounts ofkube-systemandsystem:mastersincluded. - The
d8-multitenancy-manager-namespace-creationValidatingAdmissionPolicy checks namespace creation and is in force only whileallowNamespacesWithoutProjectsisfalse. It lets through the API server (system:apiserver), the service accounts of Deckhouse (d8-system:deckhouse), of this module (d8-multitenancy-manager:multitenancy-manager) and of theupmeteragent (d8-upmeter:upmeter-agent), every service account of thed8-systemandkube-systemnamespaces, the groupssystem:mastersandsystem:nodes, andsystem:sudouser. Unlike the lists above, it includessystem:masters, so cluster administrators still create namespaces directly. Everyone else can create only thedefaultnamespace. - The
d8-multitenancy-manager-namespace-adoptionValidatingAdmissionPolicy refuses to create a namespace whose name is longer than 61 characters, starts withupmeter-or isdeckhouse, and to put theheritage: upmeterlabel on a namespace, whateverallowNamespacesWithoutProjectssays. It lets through the same users and groups as thed8-multitenancy-manager-namespace-creationpolicy. - The
d8-multitenancy-manager-manifestsValidatingAdmissionPolicy checks only the users that the controller appliesmanifestsas, whose names start withsystem:multitenancy-manager:project:, and refuses them to create an AuthorizationRule of theAdminaccess level, to raise one to this level or to add subjects to one of this level. Nobody else is checked by it. - The
d8-multitenancy-manager-v1alpha1-template-createValidatingAdmissionPolicy refuses every create of a ProjectTemplate throughdeckhouse.io/v1alpha1and lets nobody through,system:sudouser,system:mastersand the platform components included. The refusal of a newresourcesTemplatethroughdeckhouse.io/v1alpha2belongs to the ProjectTemplate webhook, so the requesters that the webhook skips,system:sudouseramong them, can still create a template with a Helm text that way. - The
d8-multitenancy-manager-namespace-ownershipValidatingAdmissionPolicy keeps theprojects.deckhouse.io/projectlabel and the labels and annotations that the module sets on a project namespace. It lets through only this module’s controller andd8-system:deckhouse. Everyone else is refused,system:sudouserandsystem:mastersincluded. - The
d8-multitenancy-manager-project-grantsValidatingAdmissionPolicy keeps the ClusterResourceGrantPolicy objects that the controller makes fromspec.grantsof a Project. It refuses to create an object whose name starts withproject-, and to create, change or delete an object that has themultitenancy.deckhouse.io/managed-by-projectlabel. It lets through this module’s controller,d8-system:deckhouse,system:sudouserand the service accounts of thekube-systemnamespace, the garbage collector among them. Everyone else is refused,system:mastersincluded. - The
d8-multitenancy-manager-kept-objectsValidatingAdmissionPolicy guards the objects listed in.status.keptObjectsof a project. It lets through this module’s controller, the users whose names start withsystem:multitenancy-manager:,system:sudouserand the system users and groups of the policy that guards theheritage: multitenancy-managerobjects. Anyone else may change such an object only with the right to make the same change across the cluster.
The /is-granted and /defaults webhooks, which check writes of objects that reference cluster-wide resources in project namespaces (see Managing access to cluster-wide resources), skip system:masters together with the platform components. The /protect webhook keeps AvailableClusterResource read-only. It lets through the controller, the platform components and system:masters.
An administrator who needs to bypass the Project or ProjectTemplate webhook, the policy that guards the heritage: multitenancy-manager objects, the d8-multitenancy-manager-manifests-objects policy, or the policy that reserves the users the controller acts as, does it deliberately as system:sudouser, for example with d8 k --as system:sudouser …, and the bypass is then visible in the audit log. There is no similar option for ProjectRoleBinding, ProjectNamespace and ClusterProjectRoleBinding, for a create of a ProjectTemplate through deckhouse.io/v1alpha1, or for the namespace labels and annotations that the d8-multitenancy-manager-namespace-ownership policy keeps.
Creating your own validation
To implement validation for resources with a different label (for example, heritage: my-custom-label):
-
Create a file with the ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding resource manifests:
apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: my-custom-label-validation spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["*"] apiVersions: ["*"] operations: ["CREATE", "UPDATE", "DELETE"] resources: ["*"] scope: "*" matchConditions: # Let the cluster components through, as the policy of the module does. - name: exclude-system-groups expression: '!(["system:nodes", "system:serviceaccounts:kube-system", "system:serviceaccounts:d8-system"].exists(e, (e in request.userInfo.groups)))' - name: exclude-system-users expression: '!(["system:apiserver", "system:kube-controller-manager", "system:kube-scheduler"].exists(e, (e == request.userInfo.username)))' variables: # A DELETE request carries no object, only oldObject. - name: subject expression: 'object == null ? oldObject : object' validations: - expression: 'request.userInfo.username == "system:serviceaccount:my-namespace:my-service-account"' # Replace with your ServiceAccount. reason: Forbidden messageExpression: 'variables.subject.kind == ''Namespace'' ? ''This resource is managed by '' + variables.subject.metadata.name + '' system. Manual modification is forbidden.'' : ''This resource is managed by '' + variables.subject.metadata.namespace + '' system. Manual modification is forbidden.''' --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: my-custom-label-validation spec: policyName: my-custom-label-validation validationActions: [Deny, Audit] matchResources: objectSelector: matchLabels: heritage: my-custom-label -
Configure the validation parameters:
policyName: Unique policy name (must match in Policy and Binding).request.userInfo.username: The name of the ServiceAccount allowed to change objects (replace with your ServiceAccount).heritage: my-custom-label: The value of theheritagelabel for your resources (replace with your value). The use of the valuesmultitenancy-manager,deckhouseis prohibited.failurePolicy: What happens when the policy cannot be evaluated, for example when an expression fails:Failrejects the request,Ignoreadmits it. A validation that returnsfalserejects the request with theDenyaction in any case.matchConditions: Requests that the policy does not check. Keep the cluster components in this list, otherwise the policy blocks the deletion of namespaces and the garbage collection of the labeled objects.validationActions: Validation actions:Deny: Deny unauthorized operations.Audit: Record operations in the audit log.
-
Apply the policy:
d8 k apply -f my-validation-policy.yaml -
Ensure your resources have the corresponding
heritagelabel:apiVersion: v1 kind: ConfigMap metadata: name: my-resource labels: heritage: my-custom-label
Managing access to cluster-wide resources
The module allows you to manage project access to cluster-wide resources such as StorageClass, ClusterIssuer, ClusterRole, and others.
For a description of the mechanism, the resources it uses, and the cluster-wide resources registered by the platform, refer to the module description.
The following sections provide common scenarios for configuring and using the mechanism.
How a policy selects namespaces
A ClusterResourceGrantPolicy (multitenancy.deckhouse.io/v1alpha2) has two selectors, projectSelector and namespaceSelector. The policy applies to a namespace of a project when every selector it sets matches:
projectSelectoris matched against the labels of the Project object. It selects every namespace of a project at once, including an additional namespace created later, so use it to select a project as a whole. The module puts theprojects.deckhouse.io/project: <PROJECT_NAME>andprojects.deckhouse.io/project-template: <PROJECT_TEMPLATE_NAME>labels on every Project, so a project can also be selected by its name or template. The virtual projects carry only the first of these labels. The two labels cannot be set to another project or template. A manifest that replaces a whole Project may leave them out. The request is admitted with a warning, and the module puts the labels back. The template label of a Project changes as soon asspec.projectTemplateNamechanges. The same label on the namespaces shows the template they were rendered from, and the service policies of a template select namespaces by it.namespaceSelectoris matched against the labels of the Namespace. It narrows the choice to some namespaces of the selected projects, or of all projects whenprojectSelectoris not set. For example, it can select only the main namespace (theprojects.deckhouse.io/project-namespacelabel does not exist) or only the namespaces labeled by a template.- A policy with neither selector matches no project on its own. Such a policy is a library policy, applied to a project through
grantPoliciesof its ProjectTemplate or of the Project itself (spec.grants.grantPolicies). An explicit empty selector ({}) matches everything.
The namespaces of the virtual projects deckhouse and default are never covered, even by {}.
An additional namespace is created by a project user and carries only the labels the module sets and the labels of the project template. A policy that restricts a resource through namespaceSelector may therefore not cover a new additional namespace. This happens when the selector relies on a label set on a namespace by hand, on a template label whose value was changed by hand on the main namespace, or on projects.deckhouse.io/project-namespace, whose value is the name the user chooses. If the GrantableClusterResourceDefinition of the resource has defaultAvailability: All (the default), such a namespace gets access to every resource of that type.
The policy reports this in its NewNamespacesCovered condition. The condition checks namespaceSelector against the labels that a new additional namespace of each covered project would get. Denied names count as a restriction for any defaultAvailability value. An allow-list or availabilityDefault: None counts only when defaultAvailability is All.
Do not select namespaces by the projects.deckhouse.io/project-namespace label without a projectSelector. A user of any project can create a ProjectNamespace with the name that such a namespaceSelector expects. The policy then applies in that project and makes the resources it allows available there.
The multitenancy.deckhouse.io/v1alpha1 version is still served. Its projectSelector is matched against the labels of the namespace and corresponds to namespaceSelector of v1alpha2. v1alpha1 has no field for the v1alpha2 projectSelector, so a v1alpha1 client sees it as JSON in the multitenancy.deckhouse.io/v1alpha2-project-selector annotation. A policy with this annotation selects projects by it, even though its v1alpha1 projectSelector is empty.
To move a policy manifest from v1alpha1 to v1alpha2, rename its projectSelector to namespaceSelector. Do not change only apiVersion. The same projectSelector is then matched against the labels of the Project, so a selector by a label that only the namespace has, such as kubernetes.io/metadata.name or a label from namespaceMetadata, matches nothing, and the policy silently stops applying to those namespaces.
When you change such a policy through v1alpha1, keep the annotation. d8 k apply and d8 k edit keep it, while d8 k replace with a manifest that lacks the annotation removes the projectSelector. To return to an old v1alpha1 manifest after you set the v1alpha2 projectSelector, remove the annotation explicitly, because d8 k apply of the old manifest leaves it in place.
For cluster administrators
The following examples show how to configure project access to cluster-wide resources using ClusterResourceGrantPolicy. They select projects with projectSelector by a label on the Project, which covers every namespace of the project.
Because the labels of a Project and of its namespaces take part in the match, the right to change these labels decides which policies apply. The labels of a Project are changed with the d8:system-capability:multitenancy-manager:edit capability, which the iam subsystem manager has and which also allows creating and changing the policies. The labels of a namespace are changed with the d8:subsystem-capability:cluster:manage_resources capability, which the cluster subsystem manager has. The project roles d8:project:* and d8:namespace:* only read namespaces.
Restricting StorageClass for a project
To allow projects to use only the fast-ssd and standard StorageClasses and use fast-ssd by default, create the following ClusterResourceGrantPolicy:
apiVersion: multitenancy.deckhouse.io/v1alpha2
kind: ClusterResourceGrantPolicy
metadata:
name: production-storage
spec:
projectSelector:
matchLabels:
environment: production
resources:
- resourceName: storageclasses
default: fast-ssd
allowed:
- fast-ssd
- standard
When a PersistentVolumeClaim is created without a value in spec.storageClassName, fast-ssd is automatically assigned to this field. If a StorageClass that is not in the allowed list is specified, it is replaced with fast-ssd, because StorageClass uses the Coerce default assignment mode. This also covers a class that the built-in Kubernetes admission controller has filled in. The replacement is reported as an admission warning in the response to the request, so the d8 k output shows both the original value and the substituted one. If the project has no default StorageClass, such a PersistentVolumeClaim is rejected.
A PersistentVolumeClaim that restores a volume from a snapshot or clones another volume (spec.dataSource or spec.dataSourceRef is set) keeps its StorageClass: the class must be served by the CSI driver of the source, so it is not replaced with the project default. If such a StorageClass is not available to the project, the PersistentVolumeClaim is rejected with an explanation; specify a StorageClass that is available to the project.
To check which StorageClasses are available to the project, run the following command:
d8 k get available storageclasses -n <PROJECT_NAME> -o yaml
While the storage-foundation module of version 1.0.3 or later is enabled, the storageclasses definition and the storageclasses-pvc GrantableClusterResourceReference are registered by that module and can differ from the ones described here, see the documentation of that module.
Restricting ClusterIssuer for a project
To allow projects to use only the letsencrypt-prod and vault-issuer ClusterIssuers and use letsencrypt-prod by default, create the following ClusterResourceGrantPolicy:
apiVersion: multitenancy.deckhouse.io/v1alpha2
kind: ClusterResourceGrantPolicy
metadata:
name: production-issuers
spec:
projectSelector:
matchLabels:
environment: production
resources:
- resourceName: clusterissuers
default: letsencrypt-prod
allowed:
- letsencrypt-prod
- vault-issuer
The policy applies to a ClusterIssuer specified in either of the following ways:
- In the
.spec.issuerRef.namefield of a Certificate if.spec.issuerRef.kindis set to ClusterIssuer. - In the
cert-manager.io/cluster-issuerannotation of an Ingress.
When a Certificate with a ClusterIssuer is created without a value in .spec.issuerRef.name, letsencrypt-prod is automatically assigned to this field. If a ClusterIssuer that is not in the allowed list is specified, the Certificate is rejected.
No default value is assigned to the cert-manager.io/cluster-issuer annotation. If the annotation is specified, its value is checked against the same list of allowed ClusterIssuers.
Access management for ClusterIssuer is available only when the cert-manager module is enabled: the clusterissuers definition is shipped by that module.
Restricting ClusterRoles for a project
By default, a RoleBinding in a project namespace can reference every ClusterRole with the rbac.deckhouse.io/delegatable: "true" label, and no other ClusterRole. A policy cannot make a ClusterRole without this label available. To use such a ClusterRole in projects, add the label to it (see Roles available in a RoleBinding inside a project).
To allow the projects of the payments team to use only the my-custom-role ClusterRole and the ClusterRoles with the shared: "true" label, create the following ClusterResourceGrantPolicy:
apiVersion: multitenancy.deckhouse.io/v1alpha2
kind: ClusterResourceGrantPolicy
metadata:
name: extra-roles
spec:
projectSelector:
matchLabels:
team: payments
resources:
- resourceName: clusterroles
allowed:
- my-custom-role
allowedSelector:
matchLabels:
shared: "true"
In RoleBindings of these projects, only these ClusterRoles can be used, and only the ones with the rbac.deckhouse.io/delegatable: "true" label. The other delegatable ClusterRoles, such as d8:project:* and d8:namespace:*, become unavailable in them, including the RoleBindings that the manifests of the template create. So list in the policy every role that these projects bind. ProjectRoleBinding and ClusterProjectRoleBinding are not affected, because the controller creates their RoleBindings.
The rbac.deckhouse.io/delegatable: "true" label is required on every ClusterRole the policy names: the clusterroles registration excludes roles without it from all projects, and an exclusion takes precedence over allowed and allowedSelector. A policy entry that names a role without the label grants nothing. The controller reports this in the AllowedEffective condition of the ClusterResourceGrantPolicy status, with the role name in the message:
d8 k get clusterresourcegrantpolicy extra-roles -o jsonpath='{.status.conditions[?(@.type=="AllowedEffective")]}'
When a RoleBinding is created or modified, the ClusterRole specified in it is checked for availability to the project. No ClusterRole value is assigned automatically.
Restricting LoadBalancerClass for a Service
To allow projects to use only the internal-lb and edge-lb LoadBalancerClass values and use internal-lb by default, create the following ClusterResourceGrantPolicy:
apiVersion: multitenancy.deckhouse.io/v1alpha2
kind: ClusterResourceGrantPolicy
metadata:
name: lb-classes
spec:
projectSelector:
matchLabels:
environment: staging
resources:
- resourceName: loadbalancerclasses
default: internal-lb
allowed:
- internal-lb
- edge-lb
Unlike StorageClass, ClusterIssuer, and ClusterRole, LoadBalancerClass is not a separate Kubernetes resource. The policy defines the allowed values of the .spec.loadBalancerClass field for Services of the LoadBalancer type.
When a Service of the LoadBalancer type is created without a value in .spec.loadBalancerClass, internal-lb is automatically assigned to this field. If a value that is not in the allowed list is specified, the Service is rejected.
The policy does not apply to Services of other types.
Granting access to all resources of a specific type
To allow specific projects to use all resources of a selected type without listing them explicitly, set availabilityDefault: All.
The following policy allows all projects with the environment: sandbox label to use any StorageClass:
apiVersion: multitenancy.deckhouse.io/v1alpha2
kind: ClusterResourceGrantPolicy
metadata:
name: open-storage-for-sandbox
spec:
projectSelector:
matchLabels:
environment: sandbox
resources:
- resourceName: storageclasses
availabilityDefault: All
The registered resources are available to all projects by default, so availabilityDefault: All changes something only where another policy restricts the resource. There it overrides the allowed and allowedSelector lists of every policy that applies to the project, the policies of its template included. denied, deniedSelector and the excluded filters of the registration still apply. The resources that match the reserved filters of the registration stay unavailable, see “Granting a reserved resource”.
Usually, explicitly specifying allowed resources using allowed or allowedSelector is sufficient for access management. Use availabilityDefault: All when the selected projects need access to all resources of the specified type.
Granting a reserved resource
A GrantableClusterResourceDefinition can mark some resources as reserved with reserved. A reserved resource is unavailable to projects by default, even when the other resources of its type are available. It becomes available to a project only if a policy that applies to the project lists it in allowed or matches it with allowedSelector. availabilityDefault: All and defaultAvailability do not make it available. denied, deniedSelector and the excluded filters take precedence over allowed and allowedSelector here as well.
An allowedSelector opens every reserved resource it matches. A broad or negative selector (Exists, NotIn, DoesNotExist) written for ordinary resources therefore also grants the reserved resources that happen to match it. To grant a reserved resource, name it in allowed or use a selector that targets it explicitly.
In the following examples, the myclusterresources registration reserves the premium and internal resources.
To grant a reserved resource without narrowing the other resources of the type, list it in allowed and set availabilityDefault: All:
apiVersion: multitenancy.deckhouse.io/v1alpha2
kind: ClusterResourceGrantPolicy
metadata:
name: grant-premium
spec:
projectSelector:
matchLabels:
tier: critical
resources:
- resourceName: myclusterresources
allowed:
- premium
availabilityDefault: All
The selected projects get every resource of the type that is not reserved, and premium. The internal resource stays unavailable, because availabilityDefault: All does not open reserved resources. availabilityDefault: All also overrides the allowed and allowedSelector lists of the other policies that apply to the project (see “Granting access to all resources of a specific type”).
allowed without availabilityDefault makes every resource not on its list unavailable, as in any other policy. If a project must stay restricted to a list, for example by the policy of its template, grant the reserved resource with allowed alone: the allowed lists of all policies that apply to the project add up. Alternatively, list in one policy every resource the project needs, the reserved one included.
To make a reserved resource the project default, specify it in both allowed and default of the same entry. A default that the project may not use is ignored. This also applies to a reserved resource marked as the default by defaultFrom: it becomes the project default only where a policy allows it.
The table shows what a project gets for typical combinations of policies. The registration reserves premium and internal, excludes system with excluded, and has defaultAvailability: All. The premium resource carries the tier: critical label. “Policies of the project” lists the entries of all policies that apply to the project.
| Policies of the project | standard |
premium (reserved) |
internal (reserved) |
system (excluded) |
|---|---|---|---|---|
| None | available | — | — | — |
availabilityDefault: All |
available | — | — | — |
allowed: [premium] |
— | available | — | — |
allowed: [premium] and availabilityDefault: All |
available | available | — | — |
allowedSelector matching tier: critical |
— | available | — | — |
allowed: [standard] in the template policy and allowed: [premium] in another policy |
available | available | — | — |
allowed: [premium] and, in another policy, denied: [premium] |
— | — | — | — |
allowed: [system] |
— | — | — | — |
allowed: [system] and availabilityDefault: All |
available | — | — | — |
Denying individual resources
To prevent projects from using individual resources while keeping the others available, use denied or deniedSelector.
The following policy prevents projects with the environment: dev label from using the expensive-nvme and archived-hdd StorageClasses:
apiVersion: multitenancy.deckhouse.io/v1alpha2
kind: ClusterResourceGrantPolicy
metadata:
name: deny-expensive-storage
spec:
projectSelector:
matchLabels:
environment: dev
resources:
- resourceName: storageclasses
denied:
- expensive-nvme
- archived-hdd
Deny rules take precedence over allow rules. If a resource matches both denied or deniedSelector and allowed or allowedSelector, it is considered unavailable.
Managing access using label selectors
To manage access to resources without listing their names, use allowedSelector and deniedSelector. Selectors allow or deny resources based on their labels.
The following policy allows projects with the tier: shared label to use StorageClasses with the shared: "true" label, except for StorageClasses with the deprecated: "true" label:
apiVersion: multitenancy.deckhouse.io/v1alpha2
kind: ClusterResourceGrantPolicy
metadata:
name: shared-storage-only
spec:
projectSelector:
matchLabels:
tier: shared
resources:
- resourceName: storageclasses
allowedSelector:
matchLabels:
shared: "true"
deniedSelector:
matchLabels:
deprecated: "true"
Granting cluster-wide resources through a project template
The availability rules for cluster-wide resources can be set directly in a structured template — they then automatically apply to all projects created from that template:
spec.resources: Rules inside the template, in the same format asresourcesin a ClusterResourceGrantPolicy (resource name,allowed/allowedSelector,default).spec.grantPolicies: List of names of library ClusterResourceGrantPolicy objects. A library policy describes a reusable set of rules and sets neitherprojectSelectornornamespaceSelector, because the referencing template determines which projects it applies to. This way, for example, a “corporate StorageClasses” policy can be maintained by one administrator and used by several templates.
The following template restricts StorageClass for its projects and connects a library policy:
apiVersion: deckhouse.io/v1alpha2
kind: ProjectTemplate
metadata:
name: my-template
spec:
resources:
- resourceName: storageclasses
allowed: ["standard"]
default: standard
grantPolicies:
# A library ClusterResourceGrantPolicy without selectors.
- corporate-issuers
For each source, the controller creates a service policy named template-<template>-<source> (for spec.resources — template-<template>-inline). A template that references a policy named inline is rejected, and so is a template whose service policy names would match those of another template. A reference to a non-existent policy or to a policy with either selector is rejected when the template is created or updated. A service policy selects the namespaces of the template’s projects with namespaceSelector by their projects.deckhouse.io/project-template label. The label is set when the template renders, so after a project switches to another template, the rules of the old template apply until the new one has rendered.
Granting cluster-wide resources to one project
A project can be given cluster-wide resources of its own, on top of the ones its template grants, in spec.grants of the Project. The field has the format of the template fields: resources with the rules themselves and grantPolicies with the names of library ClusterResourceGrantPolicy objects.
apiVersion: deckhouse.io/v1alpha3
kind: Project
metadata:
name: my-project
spec:
projectTemplateName: my-template
grants:
resources:
- resourceName: storageclasses
allowed: ["fast"]
grantPolicies:
- corporate-issuers
For each source, the controller creates a service policy named project-<PROJECT_NAME>-<SOURCE> (for resources, project-<PROJECT_NAME>-inline), owned by the project and selecting it with projectSelector by its projects.deckhouse.io/project label, so the policy applies to every namespace of the project, the ones created later included.
Deleting the project deletes its service policies.
The rules of the project and of its template combine as the rules of any two policies do: the allowed resources add up, and a resource that one of them denies stays unavailable.
The field has these rules:
- Changing
spec.grantsneeds the right to create ClusterResourceGrantPolicy objects, the same right as creating such a policy directly. A user who may only change a project cannot grant it cluster-wide resources. - The references and the names are checked like those of a template.
A reference to a policy named
inline, to a missing policy or to a policy with a selector is rejected. An empty or invalidallowedSelectorordeniedSelectorinresourcesis rejected. Aspec.grantswhose service policy names would match a policy that is not the project’s, or the service policies of another project, is rejected, and so is aspec.grantswhose service policy name is not a valid object name, which a very long name of a referenced policy causes. - If a service policy cannot be applied, or a referenced policy is deleted later, the controller records a Warning event on the Project. The other sources stay applied, and a source removed from
spec.grantsis revoked all the same. A policy that is created again is picked up without waiting. - The names that start with
project-are reserved for the service policies, and a service policy is changed only throughspec.grantsof its project. deckhouse.io/v1alpha2has no place for the field, so a change of a project that hasspec.grantsthrough that version, a write of its status included, is rejected: it would erase the field. Change such a project throughdeckhouse.io/v1alpha3.
For project users
Project users can view the cluster-wide resources available to them and check the values used by default.
Viewing available cluster-wide resources
An AvailableClusterResource is automatically created in each project namespace for every registered resource. These resources show which cluster-wide resources are available to the project and which resource is used by default.
To view all cluster-wide resources available to a project, run the following command:
d8 k get available -n <PROJECT_NAME>
Example output:
NAME KIND DEFAULT AVAILABLE AGE
storageclasses StorageClass fast-ssd 2 5m
clusterissuers ClusterIssuer letsencrypt 2 5m
To view detailed information about resources of a specific type, such as StorageClass, run the following command:
d8 k get available storageclasses -n <PROJECT_NAME> -o yaml
Example output (abridged):
status:
grantedResourceKind: StorageClass
default: fast-ssd
availableCount: 2
available:
- name: fast-ssd
default: true
fields:
provisioner: rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
- name: hdd
fields:
provisioner: local.csi.storage.deckhouse.io
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
The status.available[].fields field contains the values of some fields of each available resource, so that you can choose a resource without access to the resource itself. The fields to show are selected by the module that registers the resource, in the spec.catalogFields parameter of its GrantableClusterResourceDefinition. For StorageClass, these are provisioner, reclaimPolicy, volumeBindingMode and allowVolumeExpansion. A value keeps its JSON type (for example, allowVolumeExpansion is a boolean). A field whose value is missing in the resource is not shown: in the example above, hdd has no allowVolumeExpansion. A change in a cluster-wide resource usually appears in AvailableClusterResource within 30 seconds.
An AvailableClusterResource exists as long as the corresponding GrantableClusterResourceDefinition: when the GrantableClusterResourceDefinition is deleted, its AvailableClusterResource is deleted from the namespaces of all projects.
Rejection due to an unavailable cluster-wide resource
If the cluster-wide resource is not available to the project when an object is created, or when an update sets a new value, the request is rejected with the following message:
[multitenancy] <KIND> "<OBJECT_NAME>" references "<RESOURCE_NAME>" which is not available to project "<PROJECT_NAME>". Ask the cluster administrator to grant it.
In this case, check the available resources using AvailableClusterResource. Use a resource from the list of available resources or ask the cluster administrator to grant access to the required resource.
Assigning default values automatically
For some cluster-wide resources, the administrator can configure a default value. If the corresponding value is not specified when an object is created, it is assigned automatically.
For example, if fast-ssd is configured as the default StorageClass, the fast-ssd value can be automatically assigned to .spec.storageClassName when a PersistentVolumeClaim is created without this field.
You can specify a value explicitly by selecting any resource available to the project from the corresponding AvailableClusterResource.
Projects without a default value
AvailableClusterResource reports the project default in status.default and marks it among the names in status.available with default: true. Both are absent in the following cases:
- the grant policy sets no
default, and the registration has nodefaultFrom; - the resource is granted as a list of values rather than as objects of the cluster, and
defaultFromdoes not apply to it; defaultFromis set, but not exactly one object has the annotation set totrue;- a default is set, but the project may not use it, for example the default StorageClass of the cluster is not among the names granted to the project, or the default is a reserved resource that no policy allows for the project.
Then the field stays empty, and the object is created or rejected by the usual rules.
For module developers
Configuring validation of a cluster-wide resource reference
If a module resource contains a field that references a cluster-wide resource already registered using GrantableClusterResourceDefinition, create a GrantableClusterResourceReference. It defines which resources and fields use the cluster-wide resource and allows you to configure availability checks and automatic default value assignment.
For example, to configure validation of the StorageClass specified in the .spec.storageClassName field of a PostgresDatabase resource, add the following object to the module’s Helm chart:
apiVersion: multitenancy.deckhouse.io/v1alpha1
kind: GrantableClusterResourceReference
metadata:
name: postgresdatabases-storageclasses
labels:
heritage: deckhouse
module: postgres
spec:
grantableClusterResourceName: storageclasses
rule:
apiGroups: ["postgres.example.com"]
apiVersions: ["*"]
resources: ["postgresdatabases"]
fieldPaths:
- path: $.spec.storageClassName
defaulting: Coerce
In this example, storageclasses is the name of an existing GrantableClusterResourceDefinition, while Coerce allows the default StorageClass available to the project to be assigned when a PostgresDatabase is created if the value is missing or unavailable to the project.
If the resources listed in rule store the reference at different paths, limit fieldPaths items to specific resources using the resources parameter. For example, a Job and a CronJob store the PriorityClass name at different paths. In this case, when PriorityClass is registered as priorityclasses, use the following example:
apiVersion: multitenancy.deckhouse.io/v1alpha1
kind: GrantableClusterResourceReference
metadata:
name: jobs-priorityclasses
labels:
heritage: deckhouse
module: my-module
spec:
grantableClusterResourceName: priorityclasses
rule:
apiGroups: ["batch"]
apiVersions: ["v1"]
resources: ["jobs", "cronjobs"]
fieldPaths:
- resources: ["jobs"]
path: $.spec.template.spec.priorityClassName
- resources: ["cronjobs"]
path: $.spec.jobTemplate.spec.template.spec.priorityClassName
For each object, the most specific of the matching fieldPaths items is used: an item limited by resources takes precedence over items limited only by apiGroups or apiVersions and over an item without limits, which applies to all other resources from rule. * in resources does not limit the item: it has the same effect as omitting the parameter.
An item whose match condition does not hold for the object is skipped, and of equally limited items, an item with match takes precedence over an item without it. So several items with the same path can handle different objects of one resource, each with its own default assignment mode: the storageclasses-pvc reference checks the StorageClass of a PersistentVolumeClaim with spec.dataSource or spec.dataSourceRef in the None mode and of any other PersistentVolumeClaim in the Coerce mode. The text of the message parameter is appended to the rejection of an object for that item.
For descriptions of GrantableClusterResourceReference parameters, default assignment modes, match conditions, and configuration for resources with multiple API versions, refer to the resource description.
Registering a new cluster-wide resource
To add access management for a new cluster-wide resource:
- Create a GrantableClusterResourceDefinition to register the cluster-wide resource with the access management mechanism.
-
Create one or more GrantableClusterResourceReference objects to define the fields that reference it.
For example, to register the MyClusterResource resource, add the following object to the module’s Helm chart:
apiVersion: multitenancy.deckhouse.io/v1alpha1 kind: GrantableClusterResourceDefinition metadata: name: myclusterresources labels: heritage: deckhouse module: my-module spec: grantedResource: apiGroup: my.example.com kind: MyClusterResource enforcement: Managed defaultAvailability: All excluded: - matchExpressions: - key: my.example.com/internal operator: ExistsIn this example, the registered resources are available to projects by default. Resources with the
my.example.com/internallabel are excluded from the available resources.To keep some resources unavailable by default but let an administrator grant them to individual projects, list them in
reservedinstead ofexcluded. An administrator grants such a resource as described in “Granting a reserved resource”.Only a cluster-scoped resource can be specified in
grantedResource. A definition of a namespaced resource (for example, Secret) is refused when it is created or updated. If such a definition is stored anyway, it is not processed: no project gets its AvailableClusterResource, and theGrantedResourceValidcondition isFalse(read more in section “Checking resource registration status”). A resource whose CRD is not installed yet is accepted: its scope is checked once the resource is served.For descriptions of GrantableClusterResourceDefinition parameters and available access management modes, refer to the resource description.
- After registering the cluster-wide resource, configure references to it using GrantableClusterResourceReference as described in “Configuring validation of a cluster-wide resource reference”.
Showing resource fields to projects
Project users see the available cluster-wide resources in AvailableClusterResource (read more in section “Viewing available cluster-wide resources”), but usually cannot read the resources themselves. To help them choose, list the fields to show in spec.catalogFields of the GrantableClusterResourceDefinition. The controller copies the values of these fields from every available resource into status.available[].fields of AvailableClusterResource.
For example:
apiVersion: multitenancy.deckhouse.io/v1alpha1
kind: GrantableClusterResourceDefinition
metadata:
name: myclusterresources
spec:
grantedResource:
apiGroup: my.example.com
kind: MyClusterResource
catalogFields:
- name: tier
path: $.spec.tier
- name: region
path: $.metadata.labels['my.example.com/region']
Rules:
nameis the key infields, in lowerCamelCase;pathis an RFC 9535 JSONPath singular query that selects one value: member names and array indexes only, without wildcards (*), descendant segments (..), slices or filters.- Paths that read
metadata.managedFieldsor thekubectl.kubernetes.io/last-applied-configurationannotation, or a parent that contains them (for example,$.metadata), are not allowed: these fields hold a copy of the whole resource. - At most 10 fields can be listed.
- A value longer than 512 bytes in JSON serialization is not shown; a field whose value is missing in the resource is not shown either.
- If the
status.availablelist of an AvailableClusterResource with the fields of all resources would take more than 512 KiB in JSON serialization, no fields are shown in it; the names and the default stay. catalogFieldsworks only for definitions withgrantedResource, which must be cluster-scoped.- The values are visible to all users of every project the resource is available to. List only non-secret data.
Invalid paths and catalogFields without grantedResource are refused when the GrantableClusterResourceDefinition is created or updated, and are reported in its CatalogFieldsValid condition. Values left out for their size depend on the resources and the project, so they are reported by CatalogFieldsSkipped warning events on the GrantableClusterResourceDefinition:
d8 k describe grantableclusterresourcedefinition myclusterresources
Using x-deckhouse-grantable-resource in Deckhouse Platform application settings
To manage access to cluster-wide resources in Deckhouse Platform application settings, use the x-deckhouse-grantable-resource OpenAPI extension. In this case, deckhouse-controller automatically checks the availability of the specified resource and assigns the default value when necessary. You do not need to create GrantableClusterResourceReference manually.
For a description of the extension and usage examples, refer to “Application settings”.
Checking resource registration status
You can check the status of GrantableClusterResourceDefinition and its associated GrantableClusterResourceReference resources in the status field:
GrantableClusterResourceDefinition.status.references: Contains a list of associated GrantableClusterResourceReference resources and information about the resources to which they apply.GrantableClusterResourceReference.status.bound: Indicates whether the corresponding GrantableClusterResourceDefinition was found.GrantableClusterResourceReference.status.conditions[Bound]:Truewith the reasonResolvedif the definition was found,Falsewith the reasonUnknownResourceif it is missing.UnknownResourcecan indicate an incorrect GrantableClusterResourceDefinition name or a missing registration.GrantableClusterResourceReference.status.conditions[FieldPathsValid]: Whetherspec.fieldPathspass the checks applied when the object is created or updated (reasonsValid,InvalidFieldPaths; the message lists all problems).Falsemeans the object was stored bypassing these checks; the invalid paths are skipped during availability checks.GrantableClusterResourceDefinition.status.conditions[GrantedResourceValid]: Whetherspec.grantedResourcecan be processed.Truewith the reasonClusterScopedfor a cluster-scoped resource orValueBackedfor a definition withoutgrantedResource;Falsewith the reasonNamespacedfor a namespaced resource, which is not processed;Unknownwith the reasonKindNotServedwhile the resource is not served (for example, its CRD is not installed yet) orMappingFailedif its scope could not be determined.GrantableClusterResourceDefinition.status.conditions[CatalogFieldsValid]: Whetherspec.catalogFieldspass the checks applied when the object is created or updated (reasonsValid,InvalidCatalogFields; the message lists all problems). A definition withoutcatalogFieldshas the conditionTrue. Values left out for their size are reported byCatalogFieldsSkippedevents (read more in section “Showing resource fields to projects”).
For example:
d8 k get grantableclusterresourcedefinition myclusterresources -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}): {.message}{"\n"}{end}'