Configuration

Requirements

To the Deckhouse Platform version: 1.72 and above.

Conversions

The module is configured using the ModuleConfig resource, the schema of which contains a version number. When you apply an old version of the ModuleConfig schema in a cluster, automatic transformations are performed. To manually update the ModuleConfig schema version, the following steps must be completed sequentially for each version:

• Updates from version 1 to 2:

  Replace storage.singleReplica: true with highAvailability: false and delete storage.singleReplica and storage.recreateConfirmation. The replica count now follows the platform’s high-availability mode, and the recreate acknowledgement is gone: switching the mode is a migration between two Garage clusters and loses no data.

• Updates from version 2 to 3:

  Delete the storage section: the size of the system storage volumes is no longer a setting — it is the sum of the SystemBuckets’ spec.size and grows with it.

Parameters

Schema version: 3

• settings

  object
  • settings.controller

    object

    Settings of the system-s3-controller (the reconciler for SystemBucket resources and for the storage data plane).

    Default: {}

    • settings.controller.resourcesManagement

      system-s3-controller resources management options.

      Default: {}

      Example:

      mode: VPA
      vpa:
        cpu:
          limitRatio: 1.5
          max: 200m
          min: 50m
        memory:
          limitRatio: 1.5
          max: 256Mi
          min: 64Mi
        mode: Auto
      

      • settings.controller.resourcesManagement.mode

        string

        The mode for managing resource requests. Classical Static requests/limits or VPA.

        Default: VPA

        Allowed values: VPA, Static

      • settings.controller.resourcesManagement.static

        object

        Static resource management settings.
        • settings.controller.resourcesManagement.static.limits

          object

          Resource limits for the controller pod.
          • settings.controller.resourcesManagement.static.limits.cpu

            CPU limit.
          • settings.controller.resourcesManagement.static.limits.memory

            Memory limit.
        • settings.controller.resourcesManagement.static.requests

          object

          Resource requests for the controller pod.
          • settings.controller.resourcesManagement.static.requests.cpu

            CPU request.
          • settings.controller.resourcesManagement.static.requests.memory

            Memory request.
      • settings.controller.resourcesManagement.vpa

        object

        Parameters of the VPA mode.

        Default: {}

        • settings.controller.resourcesManagement.vpa.cpu

          object

          CPU-related parameters.

          Default: {}

          • settings.controller.resourcesManagement.vpa.cpu.limitRatio

            number

            Coefficient used to calculate the initial CPU limit from the request.

            VPA preserves the resulting limits/requests ratio during further resource tuning.

          • settings.controller.resourcesManagement.vpa.cpu.max

            Maximum allowed CPU requests.

            Default: 200m

          • settings.controller.resourcesManagement.vpa.cpu.min

            Minimum allowed CPU requests.

            Default: 50m

        • settings.controller.resourcesManagement.vpa.memory

          object

          Memory-related parameters.

          Default: {}

          • settings.controller.resourcesManagement.vpa.memory.limitRatio

            number

            Coefficient used to calculate the initial memory limit from the request.

            VPA preserves the resulting limits/requests ratio during further resource tuning.

          • settings.controller.resourcesManagement.vpa.memory.max

            Maximum allowed memory requests.

            Default: 256Mi

          • settings.controller.resourcesManagement.vpa.memory.min

            Minimum allowed memory requests.

            Default: 64Mi

        • settings.controller.resourcesManagement.vpa.mode

          string

          The VPA usage mode.

          Default: Auto

          Allowed values: Initial, Auto

  • settings.highAvailability

    boolean

    Overrides the platform’s high-availability mode for the system object storage.

    Left unset it follows global.highAvailability, and with that unset too, whether the control plane itself is highly available (more than one master).

    The mode decides the shape of the data plane: highly available means three Garage replicas with a replication factor of 3, and not highly available means a single replica with a replication factor of 1 — a third of the footprint, no redundancy, and the store is unavailable while its pod is down.

    Left unset, the mode is DERIVED from the cluster, and one direction of that is not acted on: a control plane that loses masters starts reading as not highly available, and the store keeps its three replicas anyway. Dropping to one is a loss of redundancy, and losing masters is not a request for it — it happens when the cluster can least afford it. The store moves to a single replica only when this setting says so out loud. The other direction needs no asking: masters coming back grows the store on its own.

    Changing the mode on an installation that already holds data is done on the same Garage cluster, on the same volumes. Garage cannot change its replication factor while it runs, so the module stops the data plane, removes the saved cluster layout and starts it again with the other replica count; no objects are copied and no second cluster is built.

    Going from one replica to three, the store is unavailable from the stop until the three replicas are up — usually under a minute, longer if the new volumes are slow to be provisioned. The new replicas are filled in the background while the store serves. Going from three to one, writes are refused first, until every replica holds everything (reads keep working); if that does not happen within 10 minutes, writes are given back and the change is tried again later. Then the store is unavailable for about a minute while it restarts on one replica, and the volumes of the two replicas that are gone are released.

    The endpoint keeps its address and every key keeps working, so no credentials Secret has to be re-read. Progress is reported in the ReplicationReady condition of Module/system-s3.

    Examples:

    highAvailability: true
    

    highAvailability: false
    

  • settings.logLevel

    string

    Module log verbosity level.

    Default: INFO

    Allowed values: ERROR, WARN, INFO, DEBUG, TRACE

  • settings.storageClass

    string

    StorageClass the system object storage orders its volumes from — one PersistentVolumeClaim per Garage replica.

    Left unset it follows global.modules.storageClass, and with that unset too, the cluster’s default StorageClass.

    A StorageClass is required: the module has no storage of its own to fall back to, so an installation where none of the three resolves brings up no data plane at all and says so in the StorageReady condition of Module/system-s3.

    Which class to choose is a durability decision. A class whose volumes are pinned to a node (a local class) ties each replica’s data to the node it first landed on: the replica’s copy is lost with that node and is rebuilt from the others, which the highly available mode survives and the single replica does not. A class whose volumes follow the pod (a network class) keeps the data across a node’s replacement.

    On a local class, where the replicas run decides where their data lives. They run on nodes with the node-role.deckhouse.io/system-s3 role if the cluster has any, on system nodes otherwise, and on the masters when it has neither. Giving the role to a NodeGroup made for the store, with disks sized for it, keeps the data off nodes that a NodeGroup grows, shrinks and replaces for other workloads — each such replacement costs a replica its copy on a local class.

    Changing it on an installation that already holds data is performed as a migration, because the volumes of a running store cannot be re-ordered from another class in place: the module builds a second Garage cluster on the new class, copies the objects into it and moves the endpoint onto it. Both clusters exist at once for the length of it, so the storage it needs is up to twice the usual. What consumers see is a few minutes, during the switch itself, in which writes are refused. In the highly available mode the store keeps serving reads, though a read can stall for up to about a minute while a source replica restarts; a single-replica store is restarted once during the switch, and reads are unavailable for that minute or two. The cluster migrated away from is kept read-only only until every replica of the new one holds every block, and is then removed. Putting the setting back before the switch happens calls the migration off and removes the half-built cluster. A change of highAvailability made together with it is carried out by the same migration.

    Examples:

    storageClass: ceph-csi-rbd
    

    storageClass: localpath-system