Deckhouse Platform includes a set of templates for creating projects. They are cumulative: each one includes the capabilities of the previous one and adds its own. Parameter values are set in the .spec.parameters field of a Project:
-
simpleis a minimal template that creates only the project namespace. Use it when you only need an isolated namespace managed as a project, while access and restrictions are configured through the standard project fields and project role bindings.Parameters:
namespace.labelsandnamespace.annotations— extra labels and annotations for the project namespace.requiredRequests(defaultfalse) — rendered only if the parameter is declared in the template schema and istrue. Iftrue, the project pods must specify CPU and memory requests (an OperationPolicy in theDenymode).
-
defaultis a template for basic project use cases. On top of the namespace, it sets up network isolation, a pod security profile, extended monitoring, and log shipping.Parameters (in addition to the ones for
simple):networkPolicy—Isolated(default) allows incoming traffic only from the project namespaces, Prometheus, the ingress-nginx controller and the service-with-healthchecks agent, and outgoing traffic only to the project namespaces and to DNS inkube-system;NotRestrictedallows all traffic.podSecurityProfile— the Pod Security Standards profile for the project namespaces:Baseline(default) prevents known privilege escalations,Restrictedapplies the strictest hardening practices,Privilegedrestricts nothing. An empty string sets no profile: the project puts nosecurity.deckhouse.io/pod-policylabel on its namespaces, and the default policy of theadmission-policy-enginemodule applies to them.extendedMonitoringEnabled(defaulttrue) — alerts on controller outages and restarts, 5xx errors in ingress-nginx, and low free space on the project’s persistent volumes.clusterLogDestinationName— the name of the ClusterLogDestination to ship the project logs to. Left unset, the project logs are not shipped anywhere.requiredRequests(defaulttruein this template) — iftrue, the project pods must specify CPU and memory requests (an OperationPolicy in theDenymode). When an existing namespace is adopted, the parameter is set tofalseso that the workloads already running are not blocked.
-
secureincludes all the capabilities of thedefaulttemplate, and additionally restricts the users and groups inside containers, audits their calls to the kernel, and scans images for vulnerabilities.Parameters (in addition to the ones for
default):allowedUIDsandallowedGIDs— the ranges (min,max) of IDs permitted for users and groups inside the project containers. See Security Context.runtimeAuditEnabled(defaultfalse) — audit rules that catch calls to the kernel and find malicious activity. They work only if the UID/GID ranges are set.securityScanningEnabled(defaulttrue) — periodic scanning of the launched images for known vulnerabilities (CVE) with Trivy, every 24 hours.
-
secure-with-dedicated-nodesincludes all the capabilities of thesecuretemplate, and additionally places the pods of the main namespace of the project on dedicated nodes.Parameters (in addition to the ones for
secure). IfdedicatedNodesis set, at least one of its two fields must be set. WithoutdedicatedNodes, the pods are not placed on dedicated nodes:dedicatedNodes.nodeSelector— the node selector of the main namespace. It is merged into the node selector of every pod there, and a pod whose selector conflicts with it is rejected.dedicatedNodes.defaultTolerations— tolerations in the format of the pod’sspec.tolerations. They are added to the pods created in the main namespace.
The default, secure, and secure-with-dedicated-nodes templates are described in structured form (deckhouse.io/v1alpha2). The simple template is a minimal structured template that creates the namespace, sets its labels and annotations from the project parameters and, with requiredRequests: true, creates the OperationPolicy.
For the exact set of parameters, check the template installed in your cluster — it matches your platform version.
To view the parameter schema of a template, run the following command:
d8 k get projecttemplates <PROJECT_TEMPLATE_NAME> -o jsonpath='{.spec.parametersSchema.openAPIV3Schema}' | jq
To view the whole template, run the following command:
d8 k get projecttemplates <PROJECT_TEMPLATE_NAME> -o yaml
Project creation
To create a project, follow these steps:
- Create a custom resource Project with the project template name specified in the .spec.projectTemplateName field.
- Set the standard fields — .spec.administrators and .spec.quota — which are managed directly by the Project resource regardless of the template.
-
In the .spec.parameters field, specify the values for the .spec.parametersSchema.openAPIV3Schema section of the custom resource ProjectTemplate.
An example of creating a project using Project from the
defaultProjectTemplate is shown below:apiVersion: deckhouse.io/v1alpha3 kind: Project metadata: name: my-project spec: description: This is an example from the Deckhouse documentation. projectTemplateName: default # Standard fields, managed by the Project resource itself, independently of the template. administrators: - kind: Group name: k8s-admins quota: requests.cpu: "5" requests.memory: 5Gi requests.storage: 1Gi limits.cpu: "5" limits.memory: 5Gi # Template-specific parameters. parameters: networkPolicy: Isolated podSecurityProfile: Restricted extendedMonitoringEnabled: trueThe Project API is served as
deckhouse.io/v1alpha3.v1alpha2manifests keep working: a conversion webhook automatically liftsparameters.administratorsandparameters.resourceQuotainto the.spec.administratorsand.spec.quotastandard fields. If one administrator does not fit.spec.administrators(for example, it hasaccessLevel), the whole list stays inparameters, and.spec.administratorsis not filled. The same holds for a quota with anything other than quantities (for example,scopes), for a quota with the number0anywhere in it, and for an empty list or quota. Thedeckhouse.io/v1alpha1version is no longer served, and the API server refuses a manifest withapiVersion: deckhouse.io/v1alpha1. Before applying such a manifest, changeapiVersiontodeckhouse.io/v1alpha3, renamespec.projectTypeNametospec.projectTemplateName(a ProjectTemplate with that name must exist) andspec.templatetospec.parameters, and moveadministratorsandresourceQuotafrom the parameters to.spec.administratorsand.spec.quota. -
To check the project status, run the following command:
d8 k get projects my-projectA successfully created project should have the
Deployed(synchronized) status. If theErrorstatus is displayed, add the-o yamlflag to the command (for example,d8 k get projects my-project -o yaml) to get more details about the cause of the error.
Automatic project creation for a namespace
A namespace created directly (for example, d8 k create ns test) automatically becomes a project with the same name — no annotation is required:
- the template is picked from what the namespace already carries:
secureif it has thesecurity-scanning.deckhouse.io/enabledlabel,defaultif it hassecurity.deckhouse.io/pod-policyorextended-monitoring.deckhouse.io/enabled, andsimpleotherwise; - the project parameters are filled in from the current state of the namespace, so nothing inside it changes: the network policy stays unrestricted and the Pod Security Standard keeps the value the namespace already had;
- from then on the project is the source of truth and is edited like any other project. The namespace of the project cannot be deleted directly: the request is refused with the
Manual modification is forbiddenmessage. To remove the environment, delete the Project:d8 k delete project <NAME>deletes the namespace with it and returns once the namespace is gone. Only a user allowed to delete the namespace can delete its Project (see Additional project namespaces).
Users can create a namespace directly only while the allowNamespacesWithoutProjects parameter is true (the default). With false, only cluster administrators (system:masters, system:sudouser), Deckhouse, the module controller and system components create namespaces directly. Other users create a Project or a ProjectNamespace instead.
System namespaces (d8-*, kube-*, upmeter-*, default, and anything labeled heritage: deckhouse or heritage: upmeter) are never adopted this way: they belong to the virtual deckhouse/default projects (see Virtual projects). A namespace whose name is longer than 61 characters is not adopted either: that is the Project name limit.
A namespace that gets the default or secure template because of its monitoring or scanning label and has no security.deckhouse.io/pod-policy label gets podSecurityProfile: "". The project puts no such label on it, and the default policy of the admission-policy-engine module keeps applying to it.
A namespace that already belongs to another Helm release (its meta.helm.sh/release-name annotation names a release other than the project’s) is not taken over: the project gets the HelmOwnership condition with the name of that release and retries, and the namespace is left as it is. Remove the foreign release or its ownership annotations to let the project proceed.
Existing RoleBinding and AuthorizationRule objects inside the namespace keep working after adoption. The namespace Admin is not copied into .spec.administrators and does not become d8:project:admin. To make them a project administrator, add them to .spec.administrators or create a ProjectRoleBinding.
For example:
-
Create a new namespace:
d8 k create ns test -
Make sure the project has been created:
d8 k get projectsA new project matching the namespace will appear in the list of projects:
NAME STATE PROJECT TEMPLATE DESCRIPTION AGE deckhouse Deployed virtual This is a virtual project 181d default Deployed virtual This is a virtual project 181d test Deployed simple 1m
To change the template of such a project, set another template in .spec.projectTemplateName.
If the namespace already has an object of the same kind and name as one that the new template creates, for example the isolated NetworkPolicy, the project switches to Error, and the ResourcesUpgraded condition names the object. Delete or rename the object first.
Creating a project without specifying a template
The projectTemplateName field is optional: when omitted, the simple template is used. Such a project consists only of the namespace and the standard fields (administrators, quota) — no template policies are created in it. This is convenient when no settings are needed or they are managed by other means:
apiVersion: deckhouse.io/v1alpha3
kind: Project
metadata:
name: my-plain-project
spec:
administrators:
- kind: Group
name: k8s-admins
quota:
requests.cpu: "2"
A template can be assigned later by setting it in .spec.projectTemplateName.
Project naming rules
The project name is also the name of its main namespace, so the following rules are checked when a project is created:
- the name cannot start with
d8-orkube-— these prefixes are reserved for system namespaces; - the name cannot be
defaultordeckhouse— these names belong to the virtual projects; - the name cannot be longer than 61 characters;
- the name cannot be the name of an existing namespace, or of a namespace that a ProjectNamespace has claimed (the name is taken from the moment the ProjectNamespace is created, before the controller creates the namespace);
- the name cannot be the name of a namespace whose ServiceAccounts are still named among the subjects of a ClusterProjectRoleBinding, or of a ProjectRoleBinding of a project that the name extends with a dash and a suffix (
fooforfoo-extra). Such subjects are often left from a deleted additional namespace. Remove them from the binding first.
A project can be named after another project with a dash and a suffix (foo-bar beside foo). It is a separate project with a namespace of its own, not an additional namespace of foo, and the template, administrators, quota and bindings of foo do not apply to it. The webhook allows such a project with a warning, and the controller sends the NamedLikeAdditionalNamespace Warning event on the new project. The MultitenancyManagerProjectNamedLikeAdditionalNamespace alert lists every such pair of projects. To add a namespace to a project, create a ProjectNamespace.
Project status and diagnostics
The .status.state field of a project is either Deployed (all project resources are in sync, or the objects of manifests wait for their binding) or Error. The cause of an error is described in the conditions (.status.conditions):
d8 k get project my-project -o jsonpath='{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}'
| Condition | False means |
|---|---|
ProjectTemplateFound |
The template referenced in .spec.projectTemplateName was not found. |
Validated |
The project parameters failed validation against the template schema (parametersSchema). |
ResourcesUpgraded |
The project resources could not be created or updated from the template (details in message). |
StandardFieldsApplied |
The standard fields (quota or administrators) could not be applied. |
TemplateRolesAllowed |
The template creates a binding to a role forbidden for granting in projects — the project switches to Error, the role is named in message. |
ManifestsApplied |
The objects of the template’s manifests are not applied. The message says what they wait for, why none of them is applied, which existing objects they leave alone, or which objects the API server refused, with the code of each refusal (for example, 403 Forbidden). |
HelmOwnership |
The namespace of the project belongs to another Helm release. The message names the release (see Automatic project creation for a namespace). |
NamespaceDeleted |
The project is being deleted, and its namespace is still terminating. The message says since when and what remains in it. |
The HandoverPending condition is read the other way round. It is True while AuthorizationRule and ResourceQuota objects kept from a Helm text wait for the standard fields that replace them, and it is removed once they are deleted (see Switching a project from a Helm text).
Other useful status fields:
.status.namespaces— all namespaces of the project with their kind (Main/Additional);.status.usage— the current quota usage (populated when.spec.quotais set);.status.resources— the state of the individual resources created from the template;.status.keptObjects— the objects of a Helm text that were left in place instead of being deleted;.status.manifestObjects— the objects applied from the template’smanifests.
Service objects of a project
The controller creates service objects in the project namespaces. They are managed automatically — editing them manually is not possible (the attempt is rejected):
| Object | Where | Comes from |
|---|---|---|
ResourceQuota/d8-project-quota |
The main namespace | The .spec.quota field of the project. |
ProjectRoleBinding/d8-administrators |
The main namespace | The .spec.administrators field of the project. |
ProjectRoleBinding/d8-manifests |
The main namespace | The manifests field of the project template. It grants the d8:project:admin role to the user that applies these objects. |
RoleBinding/d8:prb:<name> |
Every namespace of the project | The fan-out of the ProjectRoleBinding named <name>. |
RoleBinding/d8:cprb:<name> |
Every namespace of every project | The fan-out of the ClusterProjectRoleBinding named <name>. |
When the source object (a binding, the quota field, etc.) is removed, the corresponding service objects are removed automatically.
Virtual projects
Besides the user-created projects, the d8 k get projects list always contains two virtual projects (labeled projects.deckhouse.io/virtual-project: "true"):
deckhouse— groups system namespaces (d8-*,kube-*,upmeter-*,heritage: deckhouse/heritage: upmeter);default— groups remaining namespaces that do not belong to any project (thedefaultnamespace itself).
Virtual-project status is rebuilt from the live namespace list: a deleted namespace disappears from that list. Virtual projects do not recreate namespaces.
Virtual projects exist for completeness: with them, every namespace of the cluster belongs to some project. They cannot be managed: they are not editable, ProjectNamespace and ProjectRoleBinding resources cannot be created in them, and ClusterProjectRoleBinding does not extend to them. Their project template, virtual, is reserved for them: a user project with projectTemplateName: virtual is refused by the webhook. No ClusterResourceGrantPolicy applies to their namespaces either, even one whose selectors match a virtual Project, because the cluster resource availability rules apply only in namespaces with the projects.deckhouse.io/project label, and the namespaces of virtual projects do not carry it.
No ProjectTemplate object named virtual exists, and the webhook refuses to create one. A template with this name that is already in the cluster is not used by any project and can be deleted.
Additional project namespaces
If an application needs several namespaces (for example, a separate one for a cache or a queue), add them to the project with the ProjectNamespace resource. It is created in the main namespace of the project; the resulting namespace is named <project name>-<spec.name>:
apiVersion: deckhouse.io/v1alpha3
kind: ProjectNamespace
metadata:
name: cache
namespace: my-project
spec:
name: cache # The my-project-cache namespace will be created.
You can check the project composition in its status:
d8 k get project my-project -o jsonpath='{.status.namespaces}'
The rules for working with ProjectNamespace:
- The
spec.namefield is immutable: to rename a namespace, delete the resource and create a new one. - The resulting name
<project name>-<spec.name>cannot be longer than 63 characters (the Kubernetes limit on namespace names). - A ProjectNamespace can only be created in the main namespace of a project — it cannot be “nested” into an additional namespace or a foreign project.
- The resulting name must be free. The request is rejected when a namespace with that name exists and is not a namespace of this project, when a project has that name, or when another ProjectNamespace already claims it (the ProjectNamespace
a-backendof projectteamand the ProjectNamespacebackendof projectteam-aboth claimteam-a-backend). - The request is also rejected when the ServiceAccounts of a namespace with the resulting name are still named among the subjects of a ClusterProjectRoleBinding, or of a ProjectRoleBinding of another project that the resulting name extends with a dash and a suffix. Remove these subjects from the binding first.
- Deleting a ProjectNamespace resource deletes its namespace. Deleting the project deletes all of its namespaces.
- A project can be deleted only by a user allowed to delete each of its namespaces (the
deleteverb onnamespaces). Otherwise the request is refused, and the refusal names the namespace the user may not delete.
What applies to the additional namespaces
The following automatically applies in all namespaces of the project (the main and the additional ones alike):
- Access: the ProjectRoleBinding and ClusterProjectRoleBinding bindings, including the automatic access of the project administrators. When a new namespace is added, all existing bindings fan out into it without any user action.
- Namespaced template objects: the network policy (
networkPolicy.mode: Isolated) and the log collection setup (logShipping) are created in every namespace of the project. The network isolation allows traffic between the namespaces of one project. A HelmresourcesTemplatecreates its objects in the main namespace unless an object names another namespace of the project. So doesmanifests, whose objects are not applied while one of them names a namespace outside the project. - Cluster-scoped template policies (
OperationPolicy, theSecurityPolicyfromallowedUIDs/allowedGIDs): they select namespaces by theprojects.deckhouse.io/projectlabel, that is, they cover the whole project. - Inherited labels: the pod security profile (
security.deckhouse.io/pod-policy), extended monitoring (extended-monitoring.deckhouse.io/enabled), vulnerability scanning (security-scanning.deckhouse.io/enabled), the template label (projects.deckhouse.io/project-template) and the labels the template sets throughnamespaceMetadata.labelsare synced from the main namespace to the additional ones. The sync is complete: if a feature is turned off or a label is removed in the template, the label is removed from the additional namespaces as well. The labels of the template are listed in theprojects.deckhouse.io/template-labelsannotation of each project namespace; they are changed through the Project or its ProjectTemplate, and a direct edit of one on an additional namespace is reverted. Thanks to the template label, the cluster resource availability rules also apply in all namespaces of the project.
The following stays in the main namespace only:
- the project quota (the
ResourceQuotafrom.spec.quota); - the annotations from the template’s
namespaceMetadata; - the labels and annotations set on the main namespace directly or by a GitOps tool, including the ones the namespace had before it became a project, other than the pod security, monitoring and scanning labels above;
- the GitOps tracking labels (
app.kubernetes.io/instance,argocd.argoproj.io/*,kustomize.toolkit.fluxcd.io/*,helm.toolkit.fluxcd.io/*,kapp.k14s.io/*), even when the template sets them; - the node placement annotations (from the template’s
nodeSelectorandtolerationsfields).
When a namespace becomes a project automatically, its own labels and annotations are not copied into the Project, so the additional namespaces of the project do not inherit them. The pod security, monitoring and scanning labels are the exception: adoption turns them into parameters of the project. To put another label of such a namespace on the additional namespaces too, declare it in spec.parameters.namespace.labels of the Project, or set it on each additional namespace directly.
Labels of the project namespaces
| Label | Main | Additional | Purpose |
|---|---|---|---|
projects.deckhouse.io/project: <project name> |
✓ | ✓ | Project ownership — the common label of all namespaces of the project. |
projects.deckhouse.io/project-namespace: <spec.name> |
— | ✓ | Marks an additional namespace (the name of the ProjectNamespace resource). |
projects.deckhouse.io/project-template: <template name> |
✓ | ✓ | The project template; the cluster resource availability rules match by it. |
heritage: multitenancy-manager |
✓ | ✓ | The namespace is managed by the project controller: its spec, finalizers and the labels listed in this table are changed through the Project; other labels and annotations may be changed directly. |
security.deckhouse.io/pod-policy, extended-monitoring.deckhouse.io/enabled, security-scanning.deckhouse.io/enabled |
✓ | ✓ (inherited) | Policies and features from the project template. |
Labels from the template’s namespaceMetadata.labels |
✓ | ✓ (inherited) | Labels of the project template. They are listed in the projects.deckhouse.io/template-labels annotation of the namespace and changed through the Project or its ProjectTemplate. |
| Any other label set on a namespace directly or by a GitOps tool | — | — | Not set by the module. The project release neither reverts nor restores it, and the additional namespaces do not inherit it. |
The common projects.deckhouse.io/project label makes it possible to select the project namespaces with a plain get ns.
To get all namespaces of the project (main + additional):
d8 k get ns -l projects.deckhouse.io/project=my-project
To get additional namespaces only:
d8 k get ns -l 'projects.deckhouse.io/project=my-project,projects.deckhouse.io/project-namespace'
To get the main namespace only:
d8 k get ns -l 'projects.deckhouse.io/project=my-project,!projects.deckhouse.io/project-namespace'
Standard project fields
Project administrators and resource quotas are not project template parameters — they are top-level fields of the Project resource and work with any template (including simple):
.spec.administrators— a list of subjects (kind: Userorkind: Groupandname) that receive administrative access to the project. The controller implements this access through an auto-generated ProjectRoleBinding namedd8-administratorsin the project’s main namespace. A name cannot start withsystem:multitenancy-manager:, the prefix of the users that the controller acts as..spec.quota— a set of hard ResourceQuota limits (for example,requests.cpu,limits.memory). The controller maintains aResourceQuotain the project’s main namespace and reports current usage in.status.usage. Formemoryandstorage, a unit is required (for example,2Gi). Numbers without a unit mean bytes, and all of them except 0 are rejected. An update that leaves such a value as it was is accepted with a warning, and so is a create throughdeckhouse.io/v1alpha2with such numbers inparameters.resourceQuota.
A Helm resourcesTemplate that renders a ResourceQuota or an AuthorizationRule keeps rendering it, next to the ResourceQuota and the ProjectRoleBinding that the standard fields create. A pod in such a project has to fit every ResourceQuota of its namespace, and the access such AuthorizationRules grant adds to the access of the standard fields.
Granting access within a project
To grant access to project namespaces for users beyond the project administrators, use role bindings that reference cluster-wide roles and fan out into the appropriate project namespaces automatically:
- ProjectRoleBinding (namespaced, short name
prb) — grants a role within a single project. It must be created in the project’s main namespace. The controller creates a RoleBinding in every namespace of that project. - ClusterProjectRoleBinding (cluster-scoped, short name
cprb) — grants a role across all non-virtual projects.
roleRef must reference a ClusterRole whose name starts with one of the allowed prefixes (d8:project:, d8:namespace:, d8:project-capability:, d8:namespace-capability:, d8:custom:). See the user-authz module documentation for the description of the roles.
The following checks apply when a binding is created:
- Privilege escalation protection: a binding can only be created by a user who has the right to bind (
bind) the referenced role; the right is checked by the role’s name. A project administrator (d8:project:admin) hasbindon exactly eight roles:d8:project:viewer,d8:project:user,d8:project:manager,d8:project:admin,d8:namespace:viewer,d8:namespace:user,d8:namespace:manager, andd8:namespace:admin. A binding to any other role — a customd8:custom:*role, a capability role, ord8:project:superadmin— is rejected for a project administrator, even if it is within their own permissions, because nobody granted thembindon that name. Such bindings are created by a cluster administrator, or the cluster administrator grantsbindon that role to project administrators separately via a dedicated ClusterRole. - The role must exist: a binding to a non-existent role is rejected.
- A ServiceAccount used as a ProjectRoleBinding subject must belong to a namespace of that same project: its main namespace, its additional namespaces or a namespace a ProjectNamespace of the project has claimed. A namespace that is only named like an additional namespace (
<project>-*) but belongs to a separate project does not qualify. - System and subsystem roles (
d8:system:*,d8:subsystem:*), as well as arbitrary roles outside the listed prefixes, cannot be granted through project bindings. - Roles with the
rbac.deckhouse.io/disabled-for-direct-use-in-projects: "true"annotation are forbidden for granting in projects. A cluster administrator can put this annotation on a role to stop new grants of it: existing bindings keep working, but no new ones are created. If a project template uses such a role, the project switches toErrorwith the reason in theTemplateRolesAllowedcondition.
The controller checks the ServiceAccounts by the same rule every time it updates the RoleBindings of a ProjectRoleBinding, and also when a namespace appears, is deleted or moves to another project. The subjects that do not pass stay in .spec.subjects but get no rights: the SubjectsBound condition of the binding is False and lists them. Once the namespace is a namespace of the project again, the subjects get the rights back without a change to the binding.
The d8-administrators binding, created by the controller from the .spec.administrators field, is managed by the controller only — it cannot be edited manually. To change the set of administrators, change the project’s .spec.administrators field instead. The ServiceAccount rule above does not apply to this binding.
The following bindings give the viewer@example.com user the d8:project:viewer role in every namespace of my-project, and the platform group the same role in every project except the virtual ones:
---
apiVersion: deckhouse.io/v1alpha3
kind: ProjectRoleBinding
metadata:
name: viewers
namespace: my-project
spec:
subjects:
- kind: User
name: viewer@example.com
roleRef:
kind: ClusterRole
name: d8:project:viewer
---
apiVersion: deckhouse.io/v1alpha3
kind: ClusterProjectRoleBinding
metadata:
name: platform-viewers
spec:
subjects:
- kind: Group
name: platform
roleRef:
kind: ClusterRole
name: d8:project:viewer
Roles available in a RoleBinding inside a project
Besides the project bindings, a plain RoleBinding can also be used inside a project namespace — the role then applies in that single namespace only. However, in projects the set of roles available to a plain RoleBinding is restricted: only cluster roles carrying the rbac.deckhouse.io/delegatable: "true" label are allowed. Among the built-in ones these are the d8:namespace:* and d8:project:* roles, as well as the access-level roles of the basic role model (user-authz:user, user-authz:privileged-user, user-authz:editor, user-authz:admin).
A RoleBinding to any other cluster role (for example, cluster-admin, system roles, or capabilities) is rejected in a project with the message references "<role>" which is not available to project. This protects the project isolation from being bypassed by binding to an overly broad role.
To use a custom role in projects, add the rbac.deckhouse.io/delegatable: "true" label to it:
d8 k label clusterrole d8:custom:namespace:developer rbac.deckhouse.io/delegatable=true
The restriction applies in the namespaces of every project, including the ones adopted from a standalone namespace.
Structured templates
Starting with the deckhouse.io/v1alpha2 API version, a project template is described by structured fields — instead of a text Helm template, you declaratively specify which settings the project namespaces get. The controller itself creates the corresponding objects (network policies, security policies, log collection settings, etc.) from these fields in every namespace of the project and keeps them up to date. Other objects of the project namespaces are described in manifests.
Available fields (all optional; the complete reference is in the ProjectTemplate resource description):
| Field | What it configures |
|---|---|
podSecurityStandard |
Pod security profile: Privileged, Baseline, or Restricted. |
networkPolicy.mode |
Network isolation: Isolated (traffic is only allowed within the project and from the platform system components) or NotRestricted. |
features.monitoring |
Extended monitoring of the project namespaces. |
features.vulnerabilityScanning |
Scanning of container images for vulnerabilities. |
logShipping.clusterDestinationRef |
Collecting the logs of the project pods into the given destination (ClusterLogDestination). |
nodeSelector, tolerations |
Placing the pods of the main namespace on dedicated nodes. |
allowedUIDs, allowedGIDs |
The allowed UID/GID ranges of the project containers. |
runtimeAudit.enabled |
Auditing the project processes’ access to the Linux kernel. |
namespaceMetadata.labels, namespaceMetadata.annotations |
Extra labels and annotations of the project namespaces. The labels the module manages cannot be set here: a template or a Project that sets one is refused. |
resources, grantPolicies |
Granting cluster-scoped resources through a project template — see Managing access to cluster-wide resources. |
manifests |
Other objects in the project namespaces, applied with the rights of a project administrator. See Describing other objects in manifests. |
parametersSchema.openAPIV3Schema |
The schema of parameters set when creating a project. |
An example of a structured template:
apiVersion: deckhouse.io/v1alpha2
kind: ProjectTemplate
metadata:
name: my-template
spec:
title: "Team template"
description: "Isolated project with monitoring"
podSecurityStandard: Baseline
networkPolicy:
mode: Isolated
features:
monitoring: true
vulnerabilityScanning: true
Template parametrization
Twelve fields of the template can be turned into a parameter: podSecurityStandard, networkPolicy.mode, features.monitoring, features.vulnerabilityScanning, logShipping.clusterDestinationRef, nodeSelector, tolerations, allowedUIDs, allowedGIDs, runtimeAudit.enabled, namespaceMetadata.labels and namespaceMetadata.annotations. Instead of a concrete value, specify {fromParam: <parameter name>} and declare the parameter in parametersSchema. The value does not have to be a scalar: a map (nodeSelector, namespaceMetadata.labels), a list (tolerations) or an object (allowedUIDs) works just as well. The other fields — title, description, resources, grantPolicies and parametersSchema itself — take literal values only. Each project then sets its own value in .spec.parameters; if the value is not set, the default from the schema is used.
apiVersion: deckhouse.io/v1alpha2
kind: ProjectTemplate
metadata:
name: my-parametrized-template
spec:
podSecurityStandard:
fromParam: securityProfile
networkPolicy:
mode:
fromParam: networkMode
parametersSchema:
openAPIV3Schema:
type: object
properties:
securityProfile:
type: string
enum: [Baseline, Restricted]
default: Baseline
networkMode:
type: string
enum: [Isolated, NotRestricted]
default: Isolated
A project using such a template:
apiVersion: deckhouse.io/v1alpha3
kind: Project
metadata:
name: my-project
spec:
projectTemplateName: my-parametrized-template
parameters:
securityProfile: Restricted
The fromParam references are checked when a template is created or changed: a reference to an undeclared parameter or to a parameter of an incompatible type (for example, a string parameter for a boolean field) is rejected.
The requiredRequests parameter is read by name, without fromParam. The controller renders an OperationPolicy in the Deny mode that requires CPU and memory requests from the pods in every namespace of the project only when the template declares the parameter in parametersSchema with type: boolean and its value for the project is true. A template that does not declare the parameter renders no such policy, even if the project sets requiredRequests as an undeclared key.
Template checks
The following rules apply to template operations:
- A template cannot be deleted while at least one project uses it or a project namespace is still rendered from it. After a project switches to another template, the old template stays until the new one has rendered.
- A change to a template is automatically applied to all projects created from it.
- An existing template can describe its objects with a Helm text in
resourcesTemplateinstead of structured fields. This form is deprecated, and theresourcesTemplatefield will be removed together withdeckhouse.io/v1alpha1in one of the upcoming releases. TheMultitenancyManagerHelmProjectTemplatealert names every template that still renders from a Helm text, except the built-in ones. Describe new templates with structured fields andmanifests, and move the existing ones as Moving a template from a Helm text describes. - A template cannot get a new
resourcesTemplatethrough either version. A create with a non-emptyresourcesTemplateand an update that sets it on a template whoseresourcesTemplateis empty are refused. A template that has a Helm text keeps it, and the text can be changed through either version. So a template with a Helm text cannot be created again, for example from a manifest exported from another cluster, from a backup, or from Git after the template was deleted. Move such a template to structured fields andmanifestsfirst. - The
deckhouse.io/v1alpha1version of ProjectTemplate is served and deprecated, and the API server returns a warning on every request through it. A template is created throughdeckhouse.io/v1alpha2only, and a create throughdeckhouse.io/v1alpha1is refused. The templates created throughdeckhouse.io/v1alpha1earlier are read, changed and deleted through it as before. A change throughv1alpha1to a template that has fieldsv1alpha1cannot describe, such astitle,grantPolicies,podSecurityStandardormanifests, is refused, because it would erase them. Change such a template throughdeckhouse.io/v1alpha2.
Creating a custom project template
The default project templates cover common baseline scenarios and also serve as examples of what templates can do.
To create your own template:
- Use one of the default templates as a starting point, for example,
default. -
Export it to a separate file, for example,
my-project-template.yaml, using the following command:d8 k get projecttemplates default -o yaml > my-project-template.yaml - Edit the
my-project-template.yamlfile: adjust the structured fields and the input parameters schema to your needs. - Change the template name in the
.metadata.namefield. - Remove the
heritagelabel, themetadata.resourceVersion,metadata.uid,metadata.creationTimestamp,metadata.generationandmetadata.managedFieldsfields and thestatussection from the file. -
Apply the resulting template with the following command:
d8 k apply -f my-project-template.yaml -
Check that the new template is available by running:
d8 k get projecttemplates <NEW_TEMPLATE_NAME>
Labels for the objects of a Helm text
The following labels change how the module handles the objects of a Helm text in resourcesTemplate. Only existing templates have such a text, because no template can get a new resourcesTemplate. The objects of manifests never get the heritage: multitenancy-manager label, and the module does not read these labels on them.
Skipping the heritage label
By default, every object of the Helm text gets the heritage: multitenancy-manager label.
This label prohibits changes to the object by users or any other controller except multitenancy-manager.
To allow changes to the object (for example, for compatibility with other systems, or to control the created objects yourself), add the projects.deckhouse.io/skip-heritage-label label to it.
Example:
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config
namespace: {{ .projectName }}
labels:
projects.deckhouse.io/skip-heritage-label: "true"
app: my-app
data:
key: value
In this case, the object gets the projects.deckhouse.io/project and projects.deckhouse.io/project-template labels, but does not get the heritage: multitenancy-manager label.
Excluding objects from management
To exclude an object from management by the module (for example, when it is managed manually or by another controller), add the projects.deckhouse.io/unmanaged label to it.
Example:
apiVersion: v1
kind: Secret
metadata:
name: external-secret
namespace: {{ .projectName }}
labels:
projects.deckhouse.io/unmanaged: "true"
type: Opaque
data:
token: <base64-encoded-value>
Objects with the projects.deckhouse.io/unmanaged label:
- are created only once, when the project release is first installed, so an object added to the template later is not created in existing projects;
- are not updated when the template changes;
- are not tracked in the project status;
- get the
projects.deckhouse.io/projectandprojects.deckhouse.io/project-templatelabels, but not theheritage: multitenancy-managerlabel; - get the
helm.sh/resource-policy: keepannotation, so the project release does not delete them.
After it is created, an unmanaged object does not depend on the template and must be managed manually.
Implementing validation of object changes with a custom label
The multitenancy-manager module uses ValidatingAdmissionPolicy to protect resources labeled heritage: multitenancy-manager from manual changes.
You can implement similar validation for resources with any label.
How validation works in multitenancy-manager
multitenancy-manager validates objects labeled heritage: multitenancy-manager.
The following resources are used for this:
- ValidatingAdmissionPolicy: Defines validation rules:
- Operations:
CREATE,UPDATEandDELETE. - Exclusions: requests from cluster components are not checked, that is, from system groups (
system:nodes, thekube-systemandd8-systemservice accounts) and system users (system:apiserver,system:kube-controller-manager,system:kube-schedulerand others). - Check: only operations on behalf of the controller’s service account are allowed.
- Applies to all resources and API groups.
- Operations:
- ValidatingAdmissionPolicyBinding: Defines which objects the validation applies to:
- Uses
namespaceSelectorandobjectSelectorto select resources by the labelheritage: multitenancy-manager.
- Uses
Creating your own validation
To implement validation for resources with a different label (for example, heritage: my-custom-label):
-
Create a file with the ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding resource manifests:
apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: my-custom-label-validation spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["*"] apiVersions: ["*"] operations: ["CREATE", "UPDATE", "DELETE"] resources: ["*"] scope: "*" matchConditions: # Let the cluster components through, as the policy of the module does. - name: exclude-system-groups expression: '!(["system:nodes", "system:serviceaccounts:kube-system", "system:serviceaccounts:d8-system"].exists(e, (e in request.userInfo.groups)))' - name: exclude-system-users expression: '!(["system:apiserver", "system:kube-controller-manager", "system:kube-scheduler"].exists(e, (e == request.userInfo.username)))' variables: # A DELETE request carries no object, only oldObject. - name: subject expression: 'object == null ? oldObject : object' validations: - expression: 'request.userInfo.username == "system:serviceaccount:my-namespace:my-service-account"' # Replace with your ServiceAccount. reason: Forbidden messageExpression: 'variables.subject.kind == ''Namespace'' ? ''This resource is managed by '' + variables.subject.metadata.name + '' system. Manual modification is forbidden.'' : ''This resource is managed by '' + variables.subject.metadata.namespace + '' system. Manual modification is forbidden.''' --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: my-custom-label-validation spec: policyName: my-custom-label-validation validationActions: [Deny, Audit] matchResources: objectSelector: matchLabels: heritage: my-custom-label -
Configure the validation parameters:
policyName: Unique policy name (must match in Policy and Binding).request.userInfo.username: The name of the ServiceAccount allowed to change objects (replace with your ServiceAccount).heritage: my-custom-label: The value of theheritagelabel for your resources (replace with your value). The use of the valuesmultitenancy-manager,deckhouseis prohibited.failurePolicy: What happens when the policy cannot be evaluated, for example when an expression fails:Failrejects the request,Ignoreadmits it. A validation that returnsfalserejects the request with theDenyaction in any case.matchConditions: Requests that the policy does not check. Keep the cluster components in this list, otherwise the policy blocks the deletion of namespaces and the garbage collection of the labeled objects.validationActions: Validation actions:Deny: Deny unauthorized operations.Audit: Record operations in the audit log.
-
Apply the policy:
d8 k apply -f my-validation-policy.yaml -
Ensure your resources have the corresponding
heritagelabel:apiVersion: v1 kind: ConfigMap metadata: name: my-resource labels: heritage: my-custom-label
Managing access to cluster-wide resources
The multitenancy-manager module allows cluster administrators to define, for each project, which cluster-wide resources (such as StorageClass, ClusterIssuer, ClusterRole, and LoadBalancerClass) can be used from project namespaces and which values are used by default.
This mechanism works independently of RBAC. RBAC determines who can create and modify objects, while the cluster-wide resource access mechanism determines which resources those objects can use.
The mechanism uses four custom resources:
- GrantableClusterResourceDefinition registers a type of cluster-wide resource whose access can be managed. These resources are provided by DP or module developers.
- GrantableClusterResourceReference defines where a registered cluster-wide resource is used, for example, which resource field contains a reference to it. These resources are provided by modules.
- ClusterResourceGrantPolicy defines access rules: the allowed and denied resources, as well as the resource used by default. A cluster administrator selects the namespaces to which the policy applies with two selectors:
projectSelectormatches the labels of the Project, andnamespaceSelectormatches the labels of the Namespace. A policy with neither selector is a library policy: it applies to a project only throughgrantPoliciesof its ProjectTemplate. - AvailableClusterResource is a read-only list of cluster-wide resources available to the project under the policies that apply. The controller keeps one in every namespace of every project for each GrantableClusterResourceDefinition.
Until the administrator creates a ClusterResourceGrantPolicy, resource availability is determined by their registration: resources are available to all projects if defaultAvailability: All (the default value) is set in GrantableClusterResourceDefinition and the resource does not match the excluded or reserved filters.
Access checks are performed only for objects in project namespaces. If an access policy changes, cluster-wide resources already used by existing objects remain available to those objects.
The webhooks reject requests while the module controller is unavailable. Until it is back, objects of the registered kinds (for example, PersistentVolumeClaim, Service or RoleBinding) cannot be created or changed in project namespaces, except by the platform components and members of the system:masters group. To keep the controller available, enable highAvailability.
For a detailed description of the access management mechanism, refer to the multitenancy-manager module documentation.