Module storage-foundation: advanced usage

HTTP client authentication

To transfer data over HTTP, first create a DataExport or DataImport in the user’s namespace as described in Exporting and importing volume data. Use status.url from the cluster network or status.publicURL for access through the published Ingress. The URLs use different TLS trust chains.

For the internal status.url, the server CA is in status.ca (Base64). Example for DataExport my-export in namespace project:

d8 k -n project get dataexports.storage-foundation.deckhouse.io my-export -o jsonpath='{.status.ca}' | base64 -d > internal-ca.pem
INTERNAL_URL=$(d8 k -n project get dataexports.storage-foundation.deckhouse.io my-export -o jsonpath='{.status.url}')

The published status.publicURL uses the same TLS certificate as the published Kubernetes API (api.<publicDomain>), not the certificate from status.ca. No --cacert is needed if the certificate is trusted by the system. Otherwise use the CA for the published API, for example the certificate-authority-data from a kubeconfig that connects to api.<publicDomain>.

# Run this only when the system does not trust the CA of the published API.
d8 k config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > public-api-ca.pem

The d8 client combines system CAs, kubeconfig CAs, and the internal status.ca. With curl, select a CA that matches the endpoint. Do not use status.ca as the CA for status.publicURL.

Client certificate from kubeconfig (internal URL only)

If your kubeconfig contains an authenticated client certificate and key, you can use them only with status.url. The published Ingress terminates TLS, so the client certificate is not forwarded to the transfer server. Protect the extracted private key and delete the temporary files after use.

umask 077
d8 k config view --raw --minify --flatten -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d > client.crt
d8 k config view --raw --minify --flatten -o jsonpath='{.users[0].user.client-key-data}' | base64 -d > client.key
curl --cacert internal-ca.pem --cert client.crt --key client.key "${INTERNAL_URL%/}/api/v1/files/"

Token authentication and permissions

For published access, use a Kubernetes bearer token. Create a ServiceAccount, bind the built-in data transfer role, and issue a time-limited token:

d8 k -n project create serviceaccount data-transfer
d8 k -n project create rolebinding data-transfer --clusterrole=d8:namespace-capability:storage-foundation:transfer_volume_data --serviceaccount=project:data-transfer
TOKEN=$(d8 k -n project create token data-transfer --duration=24h)

You can configure narrower permissions manually instead of using the built-in data transfer role. For Deckhouse Platform versions earlier than 1.78, use the legacy role name from the access permissions table.

Create a DataExport with published access enabled and wait until status.publicURL is populated:

d8 data export create -n project my-export pvc/data --ttl 10m --publish=true

d8 k -n project wait dataexports.storage-foundation.deckhouse.io/my-export \
  --for=jsonpath='{.status.publicURL}' --timeout=5m

Changing spec.publish on an existing DataExport requires the patch permission for DataExport objects. The built-in data transfer role does not grant this permission.

Then request data with the bearer token:

PUBLIC_URL=$(d8 k -n project get dataexports.storage-foundation.deckhouse.io my-export -o jsonpath='{.status.publicURL}')
curl -H "Authorization: Bearer ${TOKEN}" "${PUBLIC_URL%/}/api/v1/files/"

If the published API uses a certificate signed by a CA that the system does not trust, add --cacert public-api-ca.pem to this command. Use the internal URL and --cacert internal-ca.pem when connected to the cluster network.

Manual RBAC configuration

Data endpoint requests use SubjectAccessReview with the create verb on the dataexports/download subresource for exporting or dataimports/download for importing in the storage-foundation.deckhouse.io API group. Import also uses a download subresource despite the upload direction. No upload subresource exists.

An example minimal ClusterRole that permits reading exported data:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: volume-export-http-client
rules:
  - apiGroups: ["storage-foundation.deckhouse.io"]
    resources: ["dataexports/download"]
    verbs: ["create"]

Add dataimports/download to resources to permit imports as well. This role authorizes only HTTP data transfer; creating the DataExport or DataImport object requires separate permissions. Bind the role to a ServiceAccount in the target namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: volume-export-http-client
  namespace: project
subjects:
  - kind: ServiceAccount
    name: data-transfer
    namespace: project
roleRef:
  kind: ClusterRole
  name: volume-export-http-client
  apiGroup: rbac.authorization.k8s.io

When auditing permissions, account for the namespace admin level and bindings of d8:state-snapshotter:backup-agent as well. For details, see Backup integration.

Export and import HTTP API

Data endpoints support the short paths /api/v1/files or /api/v1/block and the full paths /<NAMESPACE>/<KIND_SHORT>/<NAME>/api/v1/files or /<NAMESPACE>/<KIND_SHORT>/<NAME>/api/v1/block. For exports, <KIND_SHORT> identifies the source kind, such as pvc. For imports, use the URL belonging to your DataImport. Do not mix URLs issued by different modules.

HTTP API: data export

HTTP export supports GET for reading, HEAD for metadata, directory listings, and resume through HTTP Range. This API does not apply compression. Directory listing file sizes are computed per request rather than cached.

GET /api/v1/files/<FILE_PATH>?attribute=hash.md5 HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: <SIZE>
Content-Disposition: attachment; filename=<BASE_NAME>
X-Attribute-Uid: <UID>
X-Attribute-Gid: <GID>
X-Attribute-Permissions: <PERMISSIONS>
X-Attribute-Modtime: <RFC3339_TIMESTAMP>
X-Attribute-Hash-Md5: <MD5>
X-Type: file

<FILE_CONTENT>

HTTP/1.1 200 OK
Content-Type: application/x-symlink
Content-Length: 0
X-Attribute-Uid: <UID>
X-Attribute-Gid: <GID>
X-Attribute-Permissions: <PERMISSIONS>
X-Attribute-Modtime: <RFC3339_TIMESTAMP>
X-LinkTarget: /dir/file
X-Type: link

GET /api/v1/files/<DIR_PATH>/ HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
X-Attribute-Uid: <UID>
X-Attribute-Gid: <GID>
X-Attribute-Permissions: <PERMISSIONS>
X-Attribute-Modtime: <RFC3339_TIMESTAMP>
X-Type: dir

{
  "apiVersion": "v1",
  "items": [
    {
      "name": "1.txt",
      "type": "file",
      "uri": "1.txt",
      "attributes": {
        "gid": 1000,
        "modtime": "2025-04-01T08:02:00Z",
        "permissions": "0644",
        "size": 12,
        "uid": 1000
      }
    }
  ]
}

HEAD /api/v1/files/<FILE_PATH> HTTP/1.1

GET /api/v1/block HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: <DEVICE_SIZE>
Content-Disposition: attachment; filename=data.img

<RAW_BLOCK_CONTENT>

HEAD /api/v1/block HTTP/1.1

HTTP API: data import

Use PUT with length and offset headers to transfer content, HEAD to query the offset, and POST /api/v1/finished to complete the import. Unlike d8 data import upload, a direct HTTP client must send the completion request explicitly.

A POST to the data endpoint itself (/api/v1/files/<FILE_PATH> or /api/v1/block) completes the import as well and answers 200 OK. Both routes do the same thing; /api/v1/finished is the one to prefer, because it does not require a path.

PUT /api/v1/files/<FILE_PATH> HTTP/1.1
Content-Length: 5
X-Offset: 0
X-Content-Length: 10
X-Attribute-Permissions: 0644
X-Attribute-Uid: 1000
X-Attribute-Gid: 1000

hello

HTTP/1.1 204 No Content
X-Next-Offset: 5

PUT /api/v1/files/<FILE_PATH> HTTP/1.1
Content-Length: 5
X-Offset: 5
X-Content-Length: 10
X-Attribute-Permissions: 0644
X-Attribute-Uid: 1000
X-Attribute-Gid: 1000

world

HTTP/1.1 201 Created

HEAD /api/v1/files/<FILE_PATH> HTTP/1.1
Content-Length: 0

HTTP/1.1 200 OK
X-Next-Offset: 5

POST /api/v1/finished HTTP/1.1
Content-Length: 0

HTTP/1.1 204 No Content

PUT /api/v1/block HTTP/1.1
Content-Length: 5
X-Offset: 0
X-Content-Length: 10
X-Attribute-Permissions: 0644
X-Attribute-Uid: 1000
X-Attribute-Gid: 1000

hello

HTTP/1.1 204 No Content
X-Next-Offset: 5

PUT /api/v1/block HTTP/1.1
Content-Length: 5
X-Offset: 5
X-Content-Length: 10
X-Attribute-Permissions: 0644
X-Attribute-Uid: 1000
X-Attribute-Gid: 1000

world

HTTP/1.1 201 Created
X-Next-Offset: 10

HEAD /api/v1/block HTTP/1.1
Content-Length: 0

HTTP/1.1 200 OK
X-Next-Offset: 10
X-Device-Size: 1024

POST /api/v1/finished HTTP/1.1
Content-Length: 0

HTTP/1.1 204 No Content

HTTP response codes

The table lists HTTP error codes by operation. Successful responses (200, 201, 204) are shown in the examples above. Authentication and authorization errors can also be returned before a request reaches the data handler.