How do I set up LoadBalancer?
Note! Load Balancer must support Proxy Protocol to determine the client IP correctly.
An example of IngressNginxController
Below is a simple example of the `IngressNginxController’ configuration:
apiVersion: deckhouse.io/v1
kind: IngressNginxController
metadata:
name: main
spec:
ingressClass: nginx
inlet: LoadBalancerWithProxyProtocol
loadBalancerWithProxyProtocol:
annotations:
loadbalancer.openstack.org/proxy-protocol: "true"
loadbalancer.openstack.org/timeout-member-connect: "2000"
nodeSelector:
node-role.deckhouse.io/frontend: ""
tolerations:
- effect: NoExecute
key: dedicated.deckhouse.io
operator: Equal
value: frontend
How do I set up security policies on cluster nodes?
There may be many reasons why you may need to restrict or expand incoming/outgoing traffic on cluster VMs in OpenStack:
- Allow VMs on a different subnet to connect to cluster nodes.
- Allow connecting to the ports of the static node so that the application can work.
- Restrict access to external resources or other VMs in the cloud for security reasons.
For all this, additional security groups should be used. You can only use security groups that are created in the cloud tentatively.
Enabling additional security groups on static and master nodes
This parameter can be set either in an existing cluster or when creating one. In both cases, additional security groups are declared in the OpenStackClusterConfiguration:
- for master nodes, in the
additionalSecurityGroupsof themasterNodeGroupsection; - for static nodes, in the
additionalSecurityGroupsfield of thenodeGroupssubsection that corresponds to the target nodeGroup.
The additionalSecurityGroups field contains an array of strings with security group names.
Enabling additional security groups on ephemeral nodes
You have to set the additionalSecurityGroups parameter for all OpenStackInstanceClasses in the cluster that require additional security groups. See the parameters of the cloud-provider-openstack module.
How do I create a hybrid cluster?
A hybrid cluster combines bare metal and OpenStack nodes. To create such a cluster, you need an L2 network between all nodes of the cluster.
To set up a hybrid cluster, follow these steps:
- Delete flannel from kube-system:
kubectl -n kube-system delete ds flannel-ds. - Enable and configure the module.
- Create one or more OpenStackInstanceClass custom resources.
- Create one or more NodeManager custom resources for specifying the number of machines and managing the provisioning process in the cloud.
Caution! Cloud-controller-manager synchronizes OpenStack and Kubernetes states by deleting Kubernetes nodes that are not in OpenStack. In a hybrid cluster, such behavior does not always make sense. That is why cloud-controller-manager automatically skips Kubernetes nodes that do not have the
--cloud-provider=externalparameter (Deckhouse insertsstatic://into nodes in.spec.providerID, and cloud-controller-manager ignores them).
Attaching storage devices to instances in a hybrid cluster
To use PersistentVolumes on OpenStack nodes, you must create StorageClass with the appropriate OpenStack volume type. The openstack volume type list command lists all available types.
Here is the example config for the ceph-ssd volume type:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-ssd
provisioner: csi-cinderplugin # have to be like this
parameters:
type: ceph-ssd
volumeBindingMode: WaitForFirstConsumer
How do I create an image in OpenStack?
-
Download the latest stable Ubuntu 18.04 image:
curl -L https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img --output ~/ubuntu-18-04-cloud-amd64 -
Prepare an OpenStack RC (openrc) file containing credentials for accessing the OpenStack API:
The interface for getting an openrc file may differ depending on the OpenStack provider. If the provider has a standard interface for OpenStack, you can download the openrc file using the following instruction.
-
Otherwise, install the OpenStack client using this instruction.
Also, you can run the container and mount an openrc file and a downloaded Ubuntu image in it:
docker run -ti --rm -v ~/ubuntu-18-04-cloud-amd64:/ubuntu-18-04-cloud-amd64 -v ~/.openrc:/openrc jmcvea/openstack-client -
Initialize the environment variables from the openrc file:
source /openrc -
Get a list of available disk types:
/ # openstack volume type list +--------------------------------------+---------------+-----------+ | ID | Name | Is Public | +--------------------------------------+---------------+-----------+ | 8d39c9db-0293-48c0-8d44-015a2f6788ff | ko1-high-iops | True | | bf800b7c-9ae0-4cda-b9c5-fae283b3e9fd | dp1-high-iops | True | | 74101409-a462-4f03-872a-7de727a178b8 | ko1-ssd | True | | eadd8860-f5a4-45e1-ae27-8c58094257e0 | dp1-ssd | True | | 48372c05-c842-4f6e-89ca-09af3868b2c4 | ssd | True | | a75c3502-4de6-4876-a457-a6c4594c067a | ms1 | True | | ebf5922e-42af-4f97-8f23-716340290de2 | dp1 | True | | a6e853c1-78ad-4c18-93f9-2bba317a1d13 | ceph | True | +--------------------------------------+---------------+-----------+ -
Create an image, pass the disk format to use (if OpenStack does not support local disks or these disks don’t fit):
openstack image create --private --disk-format qcow2 --container-format bare \ --file /ubuntu-18-04-cloud-amd64 --property cinder_img_volume_type=dp1-high-iops ubuntu-18-04-cloud-amd64 -
Check that the image was created successfully:
/ # openstack image show ubuntu-18-04-cloud-amd64 +------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | checksum | 3443a1fd810f4af9593d56e0e144d07d | | container_format | bare | | created_at | 2020-01-10T07:23:48Z | | disk_format | qcow2 | | file | /v2/images/01998f40-57cc-4ce3-9642-c8654a6d14fc/file | | id | 01998f40-57cc-4ce3-9642-c8654a6d14fc | | min_disk | 0 | | min_ram | 0 | | name | ubuntu-18-04-cloud-amd64 | | owner | bbf506e3ece54e21b2acf1bf9db4f62c | | properties | cinder_img_volume_type='dp1-high-iops', direct_url='rbd://b0e441fc-c317-4acf-a606-cf74683978d2/images/01998f40-57cc-4ce3-9642-c8654a6d14fc/snap', locations='[{u'url': u'rbd://b0e441fc-c317-4acf-a606-cf74683978d2/images/01998f40-57cc-4ce3-9642-c8654a6d14fc/snap', u'metadata': {}}]' | | protected | False | | schema | /v2/schemas/image | | size | 343277568 | | status | active | | tags | | | updated_at | 2020-05-01T17:18:34Z | | virtual_size | None | | visibility | private | +------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
How to check whether the provider supports SecurityGroups?
Run the following command: openstack security group list. If there are no errors in the output, then Security Groups are supported.
How to set up online disk resize
The OpenStack API states that the resize is completed successfully. However, Nova does not get any information about the resize from Cinder. As a result, the size of the disk in the guest OS remains the same.
To get rid of this problem, you need to insert the Nova API access parameters into the cinder.conf file, e.g., as follows:
[nova]
interface = admin
insecure = {{ keystone_service_internaluri_insecure | bool }}
auth_type = {{ cinder_keystone_auth_plugin }}
auth_url = {{ keystone_service_internaluri }}/v3
password = {{ nova_service_password }}
project_domain_id = default
project_name = service
region_name = {{ nova_service_region }}
user_domain_id = default
username = {{ nova_service_user_name }}
How to use rootDiskSize and when it is preferred?
Disks in OpenStack
The node disk can be local or network. A local disk in OpenStack, is an ephemeral disk, and a network disk is a persistent disk (cinder storage). The first one is deleted along with the VM, and the second one remains in the cloud when the VM is deleted.
- A network disk is preferred for the master node so that the node can migrate between hypervisors.
- A local disk is preffered for the ephemeral node to save on cost. Not all cloud providers support the use of local disks. If local disks are not supported, you have to use network disks for ephemeral nodes.
| Local disk (ephemeral) | Network disk (persistent) |
|---|---|
| Is removed along with the VM | Stays in the cloud and can be reused |
| Cheaper | More expensive |
| Suitable for ephemeral nodes | Suitable for master nodes |
The rootDiskSize parameter
The OpenStackInstanceClass has a rootDiskSize parameter, and OpenStack flavor has a disk size parameter. Which disk will be ordered depending on the combination of parameters is shown in the table:
| flavor disk size = 0 | flavor disk size > 0 | |
|---|---|---|
rootDiskSize is not specified |
❗️You need to set the size. Without specifying the size, there will be an error creating a VM. | Local disk with size according to the flavor |
rootDiskSize is specified |
Network disk with the rootDiskSize size |
❗ Network disk (rootDiskSize) and local disk (according to the flavor). Avoid using this option, as the cloud provider will charge for both disks. |
Network disk is recommended for master nodes and bastion host
- Use flavor with a zero disk size.
- Set the
rootDiskSizein theOpenStackInstanceClass. - Check the disk type. The disk type will be taken from the OS image if it is set. If it is not set, the disk type will be taken from volumeTypeMap.
Local disk is recommended for ephemeral nodes
- Use flavor with the specified disk size.
- Do not use the
rootDiskSizeparameter in theOpenStackInstanceClass. - Check the disk type. The disk type will be taken from the OS image if it is set. If it is not set, the default disk type of the cloud provider will be used.
How do I check the disk volume in a flavor?
# openstack flavor show m1.medium-50g -c disk
+-------+-------+
| Field | Value |
+-------+-------+
| disk | 50 |
+-------+-------+
How to override a default volume type of cloud provider?
If there are several types of disks in a cloud provider, you can set a default disk type for the image in order to select a specific VM’s disk type. To do this, specify the name of a disk type in the image metadata.
Also, you may need to create a custom OpenStack image; the “How do I create an image in OpenStack” section describes how to do it
Example:
openstack volume type list
openstack image set ubuntu-18-04-cloud-amd64 --property cinder_img_volume_type=VOLUME_NAME
OFFLINE disk resize
Some cloud providers may not support ONLINE disk resizing. If you get the following error, then you need to reduce the number of StatefulSet replicas to 0, wait for disk resizing and return the number of replicas that was before the start of the operation.
Warning VolumeResizeFailed 5s (x11 over 41s) external-resizer cinder.csi.openstack.org
resize volume "pvc-555555-ab66-4f8d-947c-296520bae4c1" by resizer "cinder.csi.openstack.org" failed:
rpc error: code = Internal desc = Could not resize volume "bb5a275b-3f30-4916-9480-9efe4b6dfba5" to size 2:
Expected HTTP response code [202] when accessing
[POST https://public.infra.myfavourite-cloud-provider.ru:8776/v3/555555555555/volumes/bb5a275b-3f30-4916-9480-9efe4b6dfba5/action], but got 406 instead
{"computeFault": {"message": "Version 3.42 is not supported by the API. Minimum is 3.0 and maximum is 3.27.", "code": 406}}