Compute resource requirements

Deckhouse Development Portal (DDP) is installed by the development-platform module in a Deckhouse Platform (DP) cluster.

Allocate the following CPU, memory and disk for the installation scenarios:

The DDP budget includes the API, the web interface, the platform AI agent and workers. The managed mode is recommended for databases in the demo installation; the external mode is recommended for the other profiles.

The budget for each external database is given per server. It is sized according to the scenarios and assumptions below. CPU is sized by the rate and cost of operations, memory by the working set and concurrency, disk by the accumulated data and the space for maintenance. Additionally, account for the resources of all replicas, backups and service components.

The sizing covers the first 180 days of operation. How the values are obtained is described in the database CPU and memory sizing and the disk space sizing.

The portal budget is the free resources for DDP pods. Additionally, account for the consumption of DP, the operating system, other applications, temporary pods during updates and the margin for node failures.

DP version 1.72 or newer with the development-platform module available, the user-authn module and persistent storage for the databases in the cluster are required. The installation steps and the database connection are described in the installation guide.

Demo installation

The demo installation scenario assumes one user, a small catalog and one or two processes running at the same time. For the demo installation, use PostgreSQL, Redis and ClickHouse in the managed mode. Enable the DP modules managed-postgres, managed-valkey and managed-clickhouse before the installation. In this mode, Redis is replaced with the compatible Valkey. ClickHouse keeps the property history, which is available to demonstrate the portal capabilities.

For DDP together with the managed database instances, allocate the following resources in a running DP cluster:

The portal component budget for the demo is 4 CPU and 8 GB of memory. The managed instances add 5 CPU by the sum of their default limits and 14 GB of memory. The total is 9 CPU and 22 GB of memory. With coreFraction: 50 for PostgreSQL and coreFraction: "50%" for Valkey and ClickHouse, the guaranteed CPU request of the managed instances totals 2.5 CPU. The budget uses the full CPU limits.

With two processes that have two active steps each, the worker runs four process tasks and one additional background task at the same time. The resources of DP itself, container images and temporary disk are added to this budget.

Add the following settings to the ModuleConfig development-platform. Keep the other installation parameters unchanged:

spec:
  settings:
    postgres:
      mode: managed
    redis:
      mode: managed
    clickhouse:
      mode: managed
    workers:
      replicas: 1
      maxTasks: 5

The example uses the default managed instance resources. To change them, set the CPU in instance.cpu, memory in instance.memory and volume sizes in instance.persistentVolumeClaim.size, then recalculate the total budget. The class of the managed instance must allow the selected CPU, memory and volume sizes. The instance parameters are described in the installation guide for managed instances.

When the catalog, the number of tasks running at the same time or the number of active users grows, switch to the profiles with user load.

Load scenarios

Select a profile by the expected number of active users and the catalog size. In the examples, one running process has two steps active at the same time, each of which takes one worker slot. Background tasks include data source synchronizations, status checks and other operations from the shared queue:

Select a profile by all scenario parameters together. The number of registered users affects the accumulation of process runs, and the number of active users affects the request rate to the API and the databases. An installation for 10 users with frequent synchronizations of a large catalog may need more workers than the 100-user profile. The number of entities alone does not define the required memory either: consider the size of the parameters and the volume of one synchronization.

The API, the web interface and the platform AI agent run with one replica in every profile. The number of workers increases the concurrency of background tasks. For the 1000-user profile, check the performance of the API and the databases separately. Increasing the number of workers does not increase the performance of all components at once.

Sizing assumptions

The profiles are sized for the following data volume and activity. If these parameters change, recalculate the corresponding part of the budget:

The record sizes are profile parameters and include the payload. The sizing covers indexes, service structures, caches and temporary copies with the additional factors below. Process and action records keep accumulating after the selected horizon. Plan further disk expansion.

DDP component resources

For the 10–1000-user profiles, the budget is made up of the following component resources. The table lists approximate values per instance before rounding:

For N workers, the sum of the resources of the API, the web interface, the AI agent and the workers is 6 + 2 × N CPU and about 11.81 + 8.59 × N GB of memory. For example, for two workers the sum for the portal is 6 + 2 × 2 = 10 CPU and approximately 11.81 + 8.59 × 2 = 28.99 GB of memory. After rounding up, the budget is 16 CPU and 32 GB of memory without the databases.

Database CPU and memory sizing

The CPU sizing is based on the peak operation rate and their cost in CPU time. The sizing includes a twofold margin for CPU differences and a target load of up to 50%. Therefore, the consumption Σ(operations per second × ms of CPU per operation) / 1000 is multiplied by 4 and then rounded up to a power of two.

The cost of typical operations is taken from measurements of queries and commands with synthetic data. For history queries, the history of one entity with 900 records over 180 days was used:

Large responses, other kinds of search and more frequent requests change the cost of operations. Use SSD for the disks. Read and write latency also affects the response time.

PostgreSQL CPU and memory

The CPU sizing accounts for user reads, search and the peak change flow from synchronizations. The minimum budget in the model is 2 CPU. For example, in the 1000-user profile there are 200 active users and up to 579 entity changes per second arrive at peak. The estimated consumption is about 2.1 CPU. After multiplying by 4 and rounding, the budget is 16 CPU.

The memory working set includes the current catalog, 5 days of events and the run data for the last 30 calendar days, approximately 22 working days. The working set volume is multiplied by 1.5 for indexes and caching. The sizing additionally adds a service margin of 2 GB, 32 MB for the reads of each active user and 8 MB for each of the 10 parallel updates of one background task. The minimum memory budget in the model is 8 GB. Account for parallel operations and connections in the PostgreSQL settings as well. PostgreSQL consumes the memory for sorts and hash operations per operation, as described in the PostgreSQL documentation.

Redis CPU and memory

The CPU sizing includes the following Redis operations:

• Publishing events.
• Reading events and acknowledging them by automations.
• Enqueueing tasks.
• Reading new and pending tasks.
• Acknowledging and deleting completed tasks.
• Setting and renewing locks.
• Publishing and receiving results.

Each event includes a write, a retention extension and a read with acknowledgement by two recipients. The sizing additionally accounts for up to 35 short commands per second per worker and up to 2 requests per second per active user. In the 1000-user profile, this is about 4500 commands per second. The minimum budget in the model is 2 CPU, including a margin for persistence. For the given rates, the CPU sizing stays within this budget.

For memory, the sizing accounts for full streams: number of catalog resources × 10,000 events. The limit applies to each resource stream separately. Redis maintains it approximately. A new event extends the lifetime of the whole stream. The event retention in PostgreSQL does not limit the age of events in an active Redis stream. If you change events.streamMaxLen, recalculate the stream volume. With a value of 0 or less, the length limit is disabled.

The sizing adds a queue of 10,000 tasks and the results of tasks completed in the last 3 minutes to the events. The total volume is multiplied by 1.3 for Redis structures and by another 2 for the temporary copy during persistence. The sizing adds 0.5 GB of service margin and 1 MB of buffers per active user. The minimum memory budget in the model is 4 GB.

In the 1000-user profile, the payload takes about 8.54 GB. The sizing with margins gives about 23 GB and is rounded up to 32 GB. The effect of persistence is described in the Redis documentation.

ClickHouse CPU and memory

The CPU sizing accounts for history queries, restoring properties on a date, inserting changes and background merges. With 20 history queries per second and a peak of 579 inserts per second, the estimated consumption is about 0.32 CPU. After multiplying by 4, the consumption stays below the minimum budget of the model, 4 CPU. Therefore, the CPU is the same in the listed profiles.

The memory sizing is made up of 8 GB of service margin, 0.5 GB per user who views the history at the same time and a cache of 25% of the history volume for the last month. The sizing additionally accounts for four copies of a batch of 10,000 records during insert processing. The minimum memory budget in the model is 16 GB. For the 1000-user profile, the sizing gives about 22 GB and is rounded up to 32 GB. The memory consumption of reads and merges depends on the query, not only on the number of returned rows. The dependency of resources on queries is described in the ClickHouse recommendations.

For several database servers, multiply the budget by their number. If a server serves other applications, add the resources for DDP to the budgets of those applications.

What drives consumption

The main sources of load differ by component. Before selecting a profile, collect the following parameters:

During synchronization, DDP processes the collection of fetched objects and loads the entities of the corresponding catalog resource. A large fetch increases memory consumption even with a small number of users. Reducing the number of parallel synchronizations reduces their concurrent memory consumption.

Account separately for the resources of the AI model service and GPU when you host the model yourself. For workspaces, separately sum up the CPU, memory and storage requests from the templates of all workspaces running at the same time. Add the resources of the preparation tasks. The profile budgets include the permanent portal components. Workspaces, development environments, CI jobs and applications that are created through DDP are added to the profile budgets.

Worker count sizing

One worker processes up to workers.maxTasks tasks from the queue at the same time. The total concurrency limit is workers.replicas × workers.maxTasks. The profiles use workers.maxTasks: 10, so for 10, 20, 40, 60 and 80 tasks the profiles use 1, 2, 4, 6 and 8 workers.

The number of running processes and the number of occupied worker slots differ. One process with parallel branches can occupy several slots. A process does not occupy a worker slot while it waits for user confirmation. Also include synchronizations, status checks, MCP calls and AI agent runs in the total sizing. An AI agent run started from a process occupies a slot until the task completes. The transfer of the history to ClickHouse and its cleanup run in addition to the queue tasks and also consume worker resources.

For a constant task flow, estimate the average number of occupied slots as the sum of λ × t over all task types. Here λ is the number of incoming tasks per second and t is the average task duration in seconds. For example, 2 tasks per second with a duration of 5 seconds occupy 10 slots on average. With workers.maxTasks: 10, such a flow fully loads one worker. Add a concurrency margin for peaks and long tasks. The concurrency limits of the profiles do not define the throughput in tasks per second.

Increasing workers.maxTasks increases the concurrent memory consumption of one pod and the number of requests to PostgreSQL and external systems. For example, one synchronization can update the parameters of up to 10 entities in parallel. With ten such tasks for different catalog resources in one worker, up to 100 parallel parameter update operations are possible. Before increasing the concurrency, check the PostgreSQL connections and the limits of external APIs.

Disk space sizing

The demo installation uses the following persistent volume sizes of the managed instances:

These sizes are the managed instance defaults. They do not grow with the number of users or workers. When the data grows, increase the volumes or switch to a profile with dedicated databases. When the databases are hosted in the cluster, additionally account for storage replication and backups.

In the 10–1000-user profiles, each database disk must be at least as large as its volume in the demo installation. After rounding up to a power of two, the minimum budgets are 32 GB for PostgreSQL, 16 GB for Redis and 64 GB for ClickHouse. For each profile, choose the larger of the minimum budget and the budget calculated below.

PostgreSQL disk

The payload volume D is made up of:

• The current catalog: number of entities × 2 KB.
• Events: number of entities × 5 changes × 5 days × 8 KB.
• Process and action runs: number of users × 2 runs × 129 working days × 256 KB.
• The daily history transfer queue: number of entities × 5 changes × 1 KB.

In the model, D is doubled for indexes and the space that record updates consume. For the write-ahead log (WAL), the sizing adds the larger of two margins: 8 GB or twice the daily volume of written data. The daily write volume is made up of catalog changes number of entities × 5 × (2 + 8 + 1) KB and run data number of users × 2 × 256 KB per working day. The disk budget is (2 × D + WAL margin) / 0.7, rounded up to a power of two. The minimum budget is 32 GB, covering the 20 GB volume from the demo installation. Dividing by 0.7 leaves 30% of free space.

For 1000 users, the payload takes about 88.83 GB. With indexes, updates, WAL and free space, about 271 GB are required. After rounding, the budget is 512 GB.

This sizing covers the data of the given scenario. AI dialogs, large process contexts, additional relations and more detailed logs increase the volume.

The event retention events.ttl, 96h by default, does not define the retention of all DDP data. The 180 days for process runs are a planning horizon, not an automatic deletion period.

If the transfer of the history to ClickHouse is delayed by more than a day, increase the space margin for the queue in PostgreSQL proportionally to the delay. For WAL, also consider the actual settings and possible replica lag.

Redis disk

The payload volume B includes the event streams, the task queue and the temporary results. How to size their volume is described in the Redis memory section. The disk budget provides three copies of the volume 1.3 × B: the persisted data and the space for the snapshot, the AOF log and its rewrite. The sizing additionally adds one hour of writes at the peak rate of events and task completions.

With p events and q task completions per second, the hourly write volume is (p × 8 KB + q × (16 + 64) KB) × 3600. The disk budget is (3 × 1.3 × B + hourly write volume) / 0.7, rounded up to a power of two, with a minimum of 16 GB, covering the 10 GB volume from the demo installation. For the 1000-user profile, the sizing gives about 79 GB and is rounded up to 128 GB.

The budget is sized for enabled persistence with regular AOF log rewrites. With a different RDB/AOF policy, rare rewrites or long writes at the peak rate, recalculate the space for the AOF log and temporary files. The Redis disk does not replace memory: streams, queues and results must fit in RAM.

ClickHouse disk

The history volume is E × Δ × T × S, where:

• E is the number of entities.
• Δ is the number of changes of individual properties of one entity per day.
• T is the retention period in days.
• S is the average size of a change record.

A repeated synchronization of a value that has not changed does not count as a property change. By default, the history is kept for 180 days. The clickhouse.propertyHistory.retentionDays parameter sets the period.

The profiles assume 5 property changes per entity per day and a record size of 1 KB before compression. The compression gain is not taken into account in the sizing. The disk budget includes 16 GB of base margin, the space for the history and its temporary copy during maintenance, and 30% of free space. The resulting budget is (16 + 2 × history volume) / 0.7 GB, rounded up to a power of two, with a minimum of 64 GB, covering the 50 GB volume from the demo installation:

Select a volume no smaller than the calculated budget. If the change rate, the record size or the retention period changes, recalculate the disk budget. Compression reduces the actual data volume. Part merges and cleanup lag increase the temporary disk consumption.

DDP temporary disk

Allocate temporary disk space on the nodes separately. The API and the workers use emptyDir temporary volumes, including for the cache and for working with repositories. The module request for ephemeral-storage is about 0.052 GB per container and covers only logs. Cloned repositories, template packs and container images are not included in this request.