The module lifecycle stage: General Availability
The module has requirements for installation

How can I export and download a volume without the d8 utility?

You can create DataExport resources directly through a YAML manifest and access the exporter via HTTP. In the example below, environment variables are used for easier configuration; replace their values with the required ones:

export NAMESPACE="d8-storage-volume-data-manager"
export DATA_EXPORT_RESOURCE_NAME="example-dataexport"
export TARGET_TYPE="PersistentVolumeClaim"
export TARGET_NAME="fs-pvc-data-exporter-fs-0"
d8 k apply -f -<<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: DataExport
metadata:
  name: ${DATA_EXPORT_RESOURCE_NAME}
  namespace: ${NAMESPACE}
spec:
  ttl: 10h
  targetRef:
    kind: ${TARGET_TYPE}
    name: ${TARGET_NAME}
EOF

After creating the resource, extract the CA certificate by executing the following command:

d8 k -n $NAMESPACE get dataexport $DATA_EXPORT_RESOURCE_NAME  -o jsonpath='{.status.ca}' | base64 -d > ca.pem

Check the certificate:

openssl x509 -in ca.pem -noout -text | head

Example output:

Issuer: CN = data-exporter-CA
Signature Algorithm: ecdsa-with-SHA256

Extract the URL from the DataExport resource and verify the export:

export POD_URL=$(d8 k -n $NAMESPACE get dataexport $DATA_EXPORT_RESOURCE_NAME  -o jsonpath='{.status.url}')
echo "POD_URL: $POD_URL"

After creating the DataExport resource and extracting the necessary data, you can connect to the exporter using one of the following methods:

Authentication using certificate and key from local kubeconfig

This method uses existing credentials from your local kubeconfig file. Extract the keys from the configuration by executing the following commands:

cat ~/.kube/config | grep "client-certificate-data" | awk '{print $2}' | base64 -d > client.crt
cat ~/.kube/config | grep "client-key-data" | awk '{print $2}' | base64 -d > client.key

Check the contents of the target PVC using the command:

curl -v --cacert ca.pem ${POD_URL}api/v1/files/ --key client.key --cert client.crt

Example output:

..
..
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{"apiVersion": "v1", "items": [{"name":"4.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"hello","size":5,"modTime":"2025-03-03 10:53:06.895434814 +0000 UTC","type":"file"}
,{"name":"7.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"lost+found","modTime":"2025-03-03 10:29:31 +0000 UTC","type":"dir"}
,{"name":"8.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"10.txt","size":13,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"9.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"3.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"2.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"1.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"6.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"5.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
]}

Authentication using token and roles

This method involves creating a separate ServiceAccount with appropriate access permissions. Create a ServiceAccount using the command:

d8 k -n $NAMESPACE create serviceaccount data-exporter-test

Create a ClusterRole by executing the following command:

d8 k create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: data-exporter-test-role
rules:
- apiGroups: ["storage.deckhouse.io"]
  resources: ["dataexports/download"]
  verbs: ["create"]
EOF

Create a token by running the commands:

export TOKEN=$(d8 k -n $NAMESPACE create token data-exporter-test --duration=24h)
echo $TOKEN

Create a RoleBinding by applying the manifest:

d8 k create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: data-exporter-test-role-binding
  namespace: ${NAMESPACE}
subjects:
- kind: ServiceAccount
  name: data-exporter-test
  namespace: ${NAMESPACE}
roleRef:
  kind: ClusterRole
  name: data-exporter-test-role
  apiGroup: rbac.authorization.k8s.io
EOF

Check the contents of the target PVC by executing the request:

curl -H "Authorization: Bearer $TOKEN" \
-v --cacert ca.pem ${POD_URL}api/v1/files/

Example output:

..
..
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{"apiVersion": "v1", "items": [{"name":"4.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"hello","size":5,"modTime":"2025-03-03 10:53:06.895434814 +0000 UTC","type":"file"}
,{"name":"7.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"lost+found","modTime":"2025-03-03 10:29:31 +0000 UTC","type":"dir"}
,{"name":"8.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"10.txt","size":13,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"9.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"3.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"2.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"1.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"6.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"5.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
]}

How can I import and upload a volume without the d8 utility?

Create a DataImport resource with a PVC template via manifest, extract the access URL and CA certificate from the resource status, then upload data using the HTTP import API.

Example: create a DataImport resource:

d8 k apply -f -<<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: DataImport
metadata:
  name: example-dataimport
  namespace: d8-storage-volume-data-manager
spec:
  ttl: 10h
  publish: true
  targetRef:
    kind: PersistentVolumeClaim
    pvcTemplate:
      metadata:
        name: imported-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: local-path
        volumeMode: Filesystem
EOF

Extract the URL and CA certificate:

export POD_URL=$(d8 k -n d8-storage-volume-data-manager get dataimport example-dataimport -o jsonpath='{.status.url}')
d8 k -n d8-storage-volume-data-manager get dataimport example-dataimport -o jsonpath='{.status.ca}' | base64 -d > ca.pem

Upload a file using curl (see the HTTP API reference below for required headers):

curl --cacert ca.pem \
  --cert client.crt --key client.key \
  -X PUT "${POD_URL}api/v1/files/myfile" \
  -H "Content-Length: $(stat -c%s ./local-file)" \
  -H "X-Content-Length: $(stat -c%s ./local-file)" \
  -H "X-Attribute-Permissions: 0644" \
  -H "X-Attribute-Uid: 0" \
  -H "X-Attribute-Gid: 0" \
  -H "X-Offset: 0" \
  --data-binary @./local-file

Signal import completion:

curl --cacert ca.pem \
  --cert client.crt --key client.key \
  -X POST "${POD_URL}api/v1/finished" \
  -H "Content-Length: 0"

HTTP API reference

All endpoints are available via the short path /api/v1/files|block and the full path /{namespace}/{kindShort}/{name}/api/v1/files|block.

Definitions: - kindShort — short name of the exported resource (persistentVolumeClaim => pvc) - name — the exported resource name

Export: Filesystem mode

  • Get file
    GET /api/v1/files/{path} HTTP/1.1
    
    Query (optional, for extra attributes):
      attribute=stat      — return static attributes (uid, gid, permissions, modtime)
      attribute=hash.md5  — return MD5 for regular files

Response (file):

    HTTP/1.1 200 OK
    Content-Type: application/octet-stream
    Content-Length: <size>
    Content-Disposition: attachment; filename=<basename>
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Attribute-Hash-Md5: <md5>  # if attribute=hash.md5 is requested
    X-Type: dir

    <file content>

Response (symbolic link):

    HTTP/1.1 200 OK
    Content-Type: application/x-symlink
    Content-Length: 0 - constant value for a symbolic link
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Link-Target: /dir/file
    X-Type: link
  • Directory listing
    GET /api/v1/files/{path/} HTTP/1.1  # directory path must end with '/'
    
    Query (optional): attribute=stat | attribute=hash.md5

Response:

    HTTP/1.1 200 OK
    Content-Type: application/json
    Transfer-Encoding: chunked
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Type: dir
    
    {"apiVersion":"v1","items":[{...}]}
  • Metadata without body
    HEAD /api/v1/files/{path|path/} HTTP/1.1

Response (file):

    HTTP/1.1 200 OK
    Content-Type: application/octet-stream
    Content-Length: <size>
    Content-Disposition: attachment; filename=<basename>
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Type: dir
    X-Attribute-Hash-Md5: <md5>  # if attribute=hash.md5 is requested

Response (directory):

    HTTP/1.1 200 OK
    Content-Type: application/json
    Transfer-Encoding: chunked
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Type: dir

Response (symbolic link):

    HTTP/1.1 200 OK
    Content-Type: application/x-symlink
    Content-Length: 0 - constant value for a symbolic link
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Link-Target: /dir/file
    X-Type: link

Filesystem errors (main):

    400 Bad Request  — type mismatch (file/directory), forbidden links, etc.
    404 Not Found    — path not found
    405 Method Not Allowed — methods other than GET/HEAD
    500 Internal Server Error — internal error

Export: Block mode

  • Download block device (raw image)
    GET /api/v1/block HTTP/1.1

Response:

    HTTP/1.1 200 OK
    Content-Type: application/octet-stream
    Content-Length: <device_size>
    Content-Disposition: attachment; filename=data.img
    
    <raw block content>
  • Metadata without body
    HEAD /api/v1/block HTTP/1.1

Response:

    HTTP/1.1 200 OK
    Content-Type: application/octet-stream
    Content-Length: <device_size>
    Content-Disposition: attachment; filename=data.img

Import: Filesystem mode

  • Upload file
    PUT /api/v1/files/{path} HTTP/1.1
    Content-Length: <bytes> — required header
    X-Attribute-Permissions: 0775 — required header (octal 0000–0777)
    X-Attribute-Uid: <uid> — required header
    X-Attribute-Gid: <gid> — required header
    X-Offset: 0 — offset from the file start (non-negative integer)
    X-Content-Length: 10 — required header, expected total file size (non-negative integer)
    X-Attribute-ModTime: RFC3339 timestamp (e.g., 2024-12-27T10:16:53.4537715+03:00)

Partial upload response:

    HTTP/1.1 204 No Content
    X-Next-Offset: 10 — current offset (non-negative integer)

Completed upload response:

    HTTP/1.1 201 Created
  • Query current upload progress
    HEAD /api/v1/files/{path} HTTP/1.1
    Content-Length: 0
    HTTP/1.1 200 OK
    X-Next-Offset: 10 — current offset (non-negative integer)
  • Finish import (signal to stop the file server)
    POST /api/v1/finished HTTP/1.1
    Content-Length: 0
    HTTP/1.1 200 OK

Import: Block mode

  • Upload raw block images
    PUT /api/v1/block HTTP/1.1
    Content-Length: <bytes> — required header
    X-Attribute-Permissions: 0775 — required header (octal 0000–0777)
    X-Attribute-Uid: <uid> — required header
    X-Attribute-Gid: <gid> — required header
    X-Offset: 0 — optional; offset from the device start (non-negative integer)
    X-Content-Length: 10 — optional; expected total size (non-negative integer)

Partial upload response:

    HTTP/1.1 204 No Content
    X-Next-Offset: 10 — current offset (non-negative integer)

Completed upload response:

    HTTP/1.1 201 Created
    X-Next-Offset: 10 — final offset (equals X-Content-Length)
  • Query current upload progress
    HEAD /api/v1/block HTTP/1.1
    Content-Length: 0
    HTTP/1.1 200 OK
    X-Next-Offset - 10 — current offset (non-negative integer)
    X-Device-Size - 1024 — current device size in bytes
  • Finish import (signal to stop the file server)
    POST /api/v1/finished HTTP/1.1
    Content-Length: 0
    HTTP/1.1 200 OK

Errors (Filesystem)

400 Bad Request           — invalid/missing required headers (Missing required headers, Invalid Content-Length), invalid X-Offset / X-Content-Length / X-Attribute-ModTime
405 Method Not Allowed    — method not supported (except PUT/HEAD/POST)
403 Forbidden             — insufficient permissions to write to the filesystem
404 Not Found             — HEAD for a non-existent file
409 Conflict              — concurrent write; offset mismatch (X-Expected-Offset returned); file already fully uploaded (current size == X-Content-Length)
500 Internal Server Error — internal filesystem errors (creating directories, open/seek/copy/close), setting attributes, etc.

Errors (Block)

400 Bad Request           — invalid path (for block only "/" within the prefix is allowed); invalid headers (X-Offset / X-Content-Length)
405 Method Not Allowed    — method not supported (except PUT/HEAD/POST)
403 Forbidden             — insufficient permissions to write to the block device
409 Conflict              — concurrent write; offset mismatch (X-Expected-Offset returned)
416 Requested Range Not Satisfiable — offset < 0 or > device size; Content-Length exceeds remaining size
422 Unprocessable Entity  — written amount exceeded the declared X-Content-Length
500 Internal Server Error — internal errors (opening device, getting size, seek/copy, etc.)

Important notes on working with the API

When working with exported data through the HTTP API, consider the following features:

  • File downloads: Files are downloaded using standard GET requests containing the file path in the URL: GET /api/v1/files/largeimage.iso, GET /api/v1/files/directory/largeimage.iso. The file path should not end with /. This download method is supported by standard tools (browsers, curl, etc.). File resumption is supported, but compression is not.
  • Directory browsing: Accessing a directory is carried out with a similar GET request, where the directory path should end with /: GET /api/v1/files/ — path to root, GET /api/v1/files/directory/ — path to directory.
  • File listing: When accessing a directory, a file listing in this directory is provided: a JSON string containing the list of files is sent in the response body, including the name, type, and size of the files. File sizes are not cached and are recalculated on each directory request.