The storage-volume-data-manager module: configuration examples

The module lifecycle stage: General Availability

Alternative ways to create and use DataExport and DataImport resources without the d8 utility

Besides using the d8 utility, you can create DataExport and DataImport resources directly through a YAML manifest. In the example below, environment variables are used for easier configuration, replace their values with the required ones:

export NAMESPACE="d8-storage-volume-data-manager"
export DATA_EXPORT_RESOURCE_NAME="example-dataexport"
export TARGET_TYPE="PersistentVolumeClaim"
export TARGET_NAME="fs-pvc-data-exporter-fs-0"

kubectl apply -f -<<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: DataExport
metadata:
  name: ${DATA_EXPORT_RESOURCE_NAME}
  namespace: ${NAMESPACE}
spec:
  ttl: 10h
  targetRef:
    kind: ${TARGET_TYPE}
    name: ${TARGET_NAME}
EOF

After creating the resource, extract the CA certificate by executing the following command:

kubectl -n $NAMESPACE get dataexport $DATA_EXPORT_RESOURCE_NAME  -o jsonpath='{.status.ca}' | base64 -d > ca.pem

Check the certificate:

openssl x509 -in ca.pem -noout -text | head

Example output:

Issuer: CN = data-exporter-CA
Signature Algorithm: ecdsa-with-SHA256

Extract the URL from the DataExport resource and verify the export:

export POD_URL=$(kubectl -n $NAMESPACE get dataexport $DATA_EXPORT_RESOURCE_NAME  -o jsonpath='{.status.url}')
echo "POD_URL: $POD_URL"

After creating the DataExport resource and extracting the necessary data, you can connect to the exporter using one of the following methods:

1. Authentication using certificate and key from local kubeconfig

This method uses existing credentials from your local kubeconfig file. Extract the keys from the configuration by executing the following commands:

cat ~/.kube/config | grep "client-certificate-data" | awk '{print $2}' | base64 -d > client.crt
cat ~/.kube/config | grep "client-key-data" | awk '{print $2}' | base64 -d > client.key

Check the contents of the target PVC using the command:

curl -v --cacert ca.pem ${POD_URL}api/v1/files/ --key client.key --cert client.crt

Example output:

..
..
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{"apiVersion": "v1", "items": [{"name":"4.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"hello","size":5,"modTime":"2025-03-03 10:53:06.895434814 +0000 UTC","type":"file"}
,{"name":"7.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"lost+found","modTime":"2025-03-03 10:29:31 +0000 UTC","type":"dir"}
,{"name":"8.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"10.txt","size":13,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"9.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"3.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"2.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"1.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"6.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"5.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
]}

2. Authentication using token and roles

This method involves creating a separate ServiceAccount with appropriate access permissions. Create a ServiceAccount using the command:

kubectl -n $NAMESPACE create serviceaccount data-exporter-test

Create a ClusterRole by executing the following command:

kubectl create -f - <<EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: data-exporter-test-role
rules:
- apiGroups: ["storage.deckhouse.io"]
  resources: ["dataexports/download"]
  verbs: ["create"]
EOF

Create a token by running the commands:

export TOKEN=$(kubectl create token data-exporter-test --duration=24h)
echo $TOKEN

Create a ClusterRoleBinding by applying the manifest:

kubectl create -f - <<EOF
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: data-exporter-test-role-binding
namespace: ${NAMESPACE}
subjects:
- kind: ServiceAccount
  name: data-exporter-test
  namespace: ${NAMESPACE}
  roleRef:
  kind: ClusterRole
  name: data-exporter-test-role
  apiGroup: rbac.authorization.k8s.io
  EOF

Check the contents of the target PVC by executing the request:

curl -H "Authorization: Bearer $TOKEN" \
-v --cacert ca.pem ${POD_URL}api/v1/files/

Example output:

..
..
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{"apiVersion": "v1", "items": [{"name":"4.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"hello","size":5,"modTime":"2025-03-03 10:53:06.895434814 +0000 UTC","type":"file"}
,{"name":"7.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"lost+found","modTime":"2025-03-03 10:29:31 +0000 UTC","type":"dir"}
,{"name":"8.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"10.txt","size":13,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"9.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"3.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"2.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"1.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"6.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
,{"name":"5.txt","size":12,"modTime":"2025-04-01 08:02:00.228156524 +0000 UTC","type":"file"}
]}

API data export

All endpoints are available via the short path /api/v1/files|block and the full path /{namespace}/{kindShort}/{name}/api/v1/files|block.

Definitions: - kindShort — short name of the exported resource (persistentVolumeClaim => pvc) - name — the exported resource name

Filesystem mode

Get file

    GET /api/v1/files/{path} HTTP/1.1
    
    Query (optional, for extra attributes):
      attribute=stat      — return static attributes (uid, gid, permissions, modtime)
      attribute=hash.md5  — return MD5 for regular files

Response (file):

    HTTP/1.1 200 OK
    Content-Type: application/octet-stream
    Content-Length: <size>
    Content-Disposition: attachment; filename=<basename>
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Attribute-Hash-Md5: <md5>  # if attribute=hash.md5 is requested
    X-Type: dir

    <file content>

Response (symbolic link):

    HTTP/1.1 200 OK
    Content-Type: application/x-symlink
    Content-Length: 0 - constant value for a symbolic link
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Link-Target: /dir/file
    X-Type: link

Directory listing

    GET /api/v1/files/{path/} HTTP/1.1  # directory path must end with '/'
    
    Query (optional): attribute=stat | attribute=hash.md5

Response:

    HTTP/1.1 200 OK
    Content-Type: application/json
    Transfer-Encoding: chunked
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Type: dir
    
    {"apiVersion":"v1","items":[{...}]}

Metadata without body

    HEAD /api/v1/files/{path|path/} HTTP/1.1

Response (file):

    HTTP/1.1 200 OK
    Content-Type: application/octet-stream
    Content-Length: <size>
    Content-Disposition: attachment; filename=<basename>
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Type: dir
    X-Attribute-Hash-Md5: <md5>  # if attribute=hash.md5 is requested

Response (directory):

    HTTP/1.1 200 OK
    Content-Type: application/json
    Transfer-Encoding: chunked
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Type: dir

Response (symbolic link):

    HTTP/1.1 200 OK
    Content-Type: application/x-symlink
    Content-Length: 0 - constant value for a symbolic link
    X-Attribute-Uid: <uid>
    X-Attribute-Gid: <gid>
    X-Attribute-Permissions: <perm>
    X-Attribute-Modtime: <RFC3339>
    X-Link-Target: /dir/file
    X-Type: link

Filesystem errors (main):

    400 Bad Request  — type mismatch (file/directory), forbidden links, etc.
    404 Not Found    — path not found
    405 Method Not Allowed — methods other than GET/HEAD
    500 Internal Server Error — internal error

Block mode

Download block device (raw image)

    GET /api/v1/block HTTP/1.1

Response:

    HTTP/1.1 200 OK
    Content-Type: application/octet-stream
    Content-Length: <device_size>
    Content-Disposition: attachment; filename=data.img
    
    <raw block content>

Metadata without body

    HEAD /api/v1/block HTTP/1.1

Response:

    HTTP/1.1 200 OK
    Content-Type: application/octet-stream
    Content-Length: <device_size>
    Content-Disposition: attachment; filename=data.img

API data import

All endpoints are available via the short path /api/v1/files|block and the full path /{namespace}/{kindShort}/{name}/api/v1/files|block

Definitions: - kindShort — short name of the imported resource (persistentVolumeClaim => pvc) - name — the resource name

Filesystem mode

Upload file

    PUT /api/v1/files/{path} HTTP/1.1
    Content-Length: <bytes> — required header
    X-Attribute-Permissions: 0775 — required header (octal 0000–0777)
    X-Attribute-Uid: <uid> — required header
    X-Attribute-Gid: <gid> — required header
    X-Offset: 0 — offset from the file start (non-negative integer)
    X-Content-Length: 10 — required header, expected total file size (non-negative integer)
    X-Attribute-ModTime: RFC3339 timestamp (e.g., 2024-12-27T10:16:53.4537715+03:00)

Partial upload response:

    HTTP/1.1 204 No Content
    X-Next-Offset: 10 — current offset (non-negative integer)

Completed upload response:

    HTTP/1.1 201 Created

Query current upload progress

    HEAD /api/v1/files/{path} HTTP/1.1
    Content-Length: 0

    HTTP/1.1 200 OK
    X-Next-Offset: 10 — current offset (non-negative integer)

Finish import (signal to stop the file server)

    POST /api/v1/finished HTTP/1.1
    Content-Length: 0

    HTTP/1.1 200 OK

Upload raw block images

    PUT /api/v1/block HTTP/1.1
    Content-Length: <bytes> — required header
    X-Attribute-Permissions: 0775 — required header (octal 0000–0777)
    X-Attribute-Uid: <uid> — required header
    X-Attribute-Gid: <gid> — required header
    X-Offset: 0 — optional; offset from the device start (non-negative integer)
    X-Content-Length: 10 — optional; expected total size (non-negative integer)

Partial upload response:

    HTTP/1.1 204 No Content
    X-Next-Offset: 10 — current offset (non-negative integer)

Completed upload response:

    HTTP/1.1 201 Created
    X-Next-Offset: 10 — final offset (equals X-Content-Length)

Query current upload progress

    HEAD /api/v1/block HTTP/1.1
    Content-Length: 0

    HTTP/1.1 200 OK
    X-Next-Offset - 10 — current offset (non-negative integer)
    X-Device-Size - 1024 — current device size in bytes

Finish import (signal to stop the file server)

    POST /api/v1/finished HTTP/1.1
    Content-Length: 0

    HTTP/1.1 200 OK

Errors (Filesystem)

400 Bad Request           — invalid/missing required headers (Missing required headers, Invalid Content-Length), invalid X-Offset / X-Content-Length / X-Attribute-ModTime
405 Method Not Allowed    — method not supported (except PUT/HEAD/POST)
403 Forbidden             — insufficient permissions to write to the filesystem
404 Not Found             — HEAD for a non-existent file
409 Conflict              — concurrent write; offset mismatch (X-Expected-Offset returned); file already fully uploaded (current size == X-Content-Length)
500 Internal Server Error — internal filesystem errors (creating directories, open/seek/copy/close), setting attributes, etc.

Errors (Block)

400 Bad Request           — invalid path (for block only "/" within the prefix is allowed); invalid headers (X-Offset / X-Content-Length)
405 Method Not Allowed    — method not supported (except PUT/HEAD/POST)
403 Forbidden             — insufficient permissions to write to the block device
409 Conflict              — concurrent write; offset mismatch (X-Expected-Offset returned)
416 Requested Range Not Satisfiable — offset < 0 or > device size; Content-Length exceeds remaining size
422 Unprocessable Entity  — written amount exceeded the declared X-Content-Length
500 Internal Server Error — internal errors (opening device, getting size, seek/copy, etc.)

Important notes on working with the API

When working with exported data through the HTTP API, consider the following features:

File downloads: Files are downloaded using standard GET requests containing the file path in the URL: GET /api/v1/files/largeimage.iso, GET /api/v1/files/directory/largeimage.iso. The file path should not end with /. This download method is supported by standard tools (browsers, curl, etc.). File resumption is supported, but compression is not.
Directory browsing: Accessing a directory is carried out with a similar GET request, where the directory path should end with /: GET /api/v1/files/ — path to root, GET /api/v1/files/directory/ — path to directory.
File listing: When accessing a directory, a file listing in this directory is provided: a JSON string containing the list of files is sent in the response body, including the name, type, and size of the files. File sizes are not cached and are recalculated on each directory request.

The storage-volume-data-manager module: configuration examples

Alternative ways to create and use DataExport and DataImport resources without the d8 utility

1. Authentication using certificate and key from local kubeconfig

2. Authentication using token and roles

API data export

Filesystem mode

Block mode

API data import

Filesystem mode

Errors (Filesystem)

Errors (Block)

Important notes on working with the API

An error has occurred

Tell us what you didn’t like.

The storage-volume-data-manager module: configuration examples

Alternative ways to create and use DataExport and DataImport resources without the d8 utility

1. Authentication using certificate and key from local kubeconfig

2. Authentication using token and roles

API data export

Filesystem mode

Block mode

API data import

Filesystem mode

Errors (Filesystem)

Errors (Block)

Important notes on working with the API

An error has occurred

Tell us what you didn’t like.

Request trial access

Thank you

Error

Request callback

Thank you

Something went wrong

Book your sessions

Thank you

Error

Request demo

Thank you

Error

Get the PCI SSC Compliance Report

Thank you

Error