Clusters

If you are looking to install Qovery on your Kubernetes cluster, please refer to this guide.

Managing your Clusters with Qovery

From the Qovery Console, you can manage the settings of the clusters you want to run on your infrastructure. The clusters are then created (or updated) by the cloud provider that hosts them.

Creating a Cluster

To install a cluster, Qovery needs a set of credentials to access your cloud provider account (example: AWS secret_access_key and access_key_id). If this is the first time you are installing a cluster with Qovery, have a look at this guide on how to get the credentials: here for AWS, here for Scaleway, here for GCP.

To create a cluster:

1. Open your Qovery Console.

2. On the left menu bar, click on the Cluster page:

   

3. Click Add Cluster:

   

4. In the Create Cluster window enter:

   • Cluster name: enter the name of your choice for your cluster.
   • Description: enter a description to identify better your cluster.
   • Production cluster: select this option if your cluster will be used for production.
   • Cloud provider: select your cloud provider.
   • Region: select the geographical area in which you want your cluster to be hosted.
   • Credentials: select one of the existing cloud provider credentials or add a new one by clicking on New Credentials. In the New credentials window, add the credentials that you have generated on your cloud provider console (Procedure for AWS account, Procedure for Scaleway account, Procedure for GCP account). Added credentials can be used later to create and manage additional cluster.

   To confirm, click Next.

5. In the Set Resources window, select:

   • Cluster: select the cluster type to use. Please refer to this section for more information.
   • Disk size: select the size of the disks to be attached to your cluster instances (to locally store container images etc..). Setting available only on AWS.
   • Instance type: select the type of worker nodes you want to deploy to your cluster:
   • Node auto-scaling: define the minimum and the maximum number of worker nodes that your cluster can run. The lowest number is the number of worker nodes running on your infrastructure at any time, while the highest number is the maximum number of worker nodes that can automatically be deployed as traffic grows. Please note that a minimum of 3 worker nodes is required to deploy your EKS cluster.
   
   
   Instance type selection from your Qovery Console has direct consequences on your cloud provider’s bill. While Qovery allows you to switch to a different instance type whenever you want, it is your sole responsibility to keep an eye on your infrastructure costs, especially when you want to upsize.

   Please be aware that changing the instance type or disk size might cause a downtime for your service.

   For more information on the instance types provided by each cloud provider and their associated pricing, see What are the different instance types available when creating a cluster?

   Also, before downsizing, you need to ensure that your applications will still have enough resources to run correctly.

   
   

   At the bottom of the window, you can see an estimate of the cost associated with the selected instance type.

   For AWS EKS clusters, you have the possibility to enable Karpenter autoscaler to improve the efficiency and cost of running workloads on your cluster. You can check the official documentation for more information.

   

   Today, only new non-production clusters are supported. It means you won't be able to enable it on your already existing cluster. It will be supported soon.

   By activating Karpenter, you have to set:

   • Disk size: select the size of the disks to be attached to your cluster instances (to locally store container images etc..).
   • Default node architecture: If you build your application with the Qovery CI, your application will be built using this architecture by default.
   • Spot instances: In order to reduce even more your costs, you can also enable the spot instances on your clusters. Spot instances cost up to 90% less compared to On-Demand prices. But keep in mind that spot instances can be terminated by the cloud provider at any time. Check this documentation for more information. Even if this flag is enabled, the statefulsets won't run on spot instances.
   
   

   A SQS queue will be created. Before deploying your cluster, update the IAM permissions of the Qovery user, make sure to use the latest version here to add the permission on SQS.

   To confirm, click Next.

6. (Only for AWS K8S Clusters) In the Features window, select the features you want to enable on your cluster.

   For more information, see Features.

7. In the Ready to install your cluster window, check that the services needed to install your cluster are correct.

   You can now press the Create and Install button.

   Your cluster is now displayed in your organization settings, featuring the Installing... status (orange status). Once your cluster is properly installed, its status turns to green and you will be able to deploy your applications on it.

How was it? Did this tutorial work?  Yes  No

Managing your Cluster Settings

To manage the settings of an existing cluster:

1. Open your Qovery Console.

2. On the left menu bar, click on the Cluster page:

   

3. To access your cluster settings, click on the wheel button:

   

How was it? Did this tutorial work?  Yes  No

Below you can find a description of each section

General

The General tab allows you to define high-level information on your cluster:

Item	Description
Cluster Name	To edit the name of your cluster.
Description	To enter or edit the description of your cluster.
Production Cluster	To enter or edit the production flag of your cluster.

Credentials

Here you can manage here the cloud provider credentials associated with your cluster.

If you need to change the credentials:

• generate a new set of credentials on your cloud provider(Procedure for AWS account, Procedure for Scaleway account, Procedure for GCP account)
• create the new credential on the Qovery by opening the drop-down and selecting "New Credentials"

In the dedicated fields, enter the credentials you created on your cloud provider account:

Account Provider	Field Labels
AWS	Access Key and Secret Access Key
Scaleway	Scaleway Access Key, Scaleway Secret Key, Scaleway Project ID and Scaleway Organization ID
GCP	GCP JSON key

Once created and associated, you need to updating your cluster to apply the change.

Resources

Qovery allows you to modify the resources allocated for your cluster:

• In the Instance type dropdown menu, select the type of worker node(s) you want to deploy to your cluster.
• (AWS users only) In the Node disk size (GB) field, enter the disk capacity you want to allocate to your worker node(s) (meaning how much data, in gigabytes, you want each worker node to be able to hold).
• (EKS users only) On the Nodes auto-scaling, define the range of worker nodes you want to deploy to your cluster.

Instance type selection from your Qovery Console has direct consequences on your cloud provider’s bill. While Qovery allows you to switch to a different instance type whenever you want, it is your sole responsibility to keep an eye on your infrastructure costs, especially when you want to upsize.

For more information on the instance types provided by each cloud provider and their associated pricing, see What are the different instance types available when creating a cluster?

The lowest number is the number of worker nodes running on your infrastructure at any time, while the highest number is the maximum number of worker nodes that can automatically be deployed as traffic grows.

Please note that a minimum of 3 worker nodes is required to deploy your EKS cluster.

Image registry

In this tab, you will see that a container registry already exist (called registry-{$UIID}). This is your cloud provider container registry used by Qovery to manage the deployment of your applications by mirroring the docker images.

The credentials configured on this registry are the one used to create the cluster. But you can still update them if you prefer to manage them separately (dedicated pair of creds just to access the registry).

Check this link for more information.

Features

The Features tab in your cluster settings allows you to check if the Static IP, Custom VPC subnet, Deploy on existing VPC features are enabled on your cluster. The enabled features cannot be changed after the creation of the cluster.

Static IP

The Static IP feature is currently only available to clusters deployed on AWS and GCP with a VPC managed by Qovery and can only be enabled at cluster creation.

By default, when your cluster is created, its worker nodes are allocated public IP addresses, which are used for external communication. For improved security and control, the Static IP feature allows you to ensure that outbound traffic from your cluster uses specific IP addresses.

Here is what will be deployed on AWS:

• Nat Gateways
• Elastic IPs
• Private subnets

Here is what will be deployed on GCP:

• Cloud Nats
• Static IPs
• Routers

Once set up, here is the procedure to find your static IP addresses on AWS:

• On your AWS account, select the VPC service.
• On the left menu, you’ll find Elastic IP addresses. Once on it, in the Allocated IPv4 address column, you’ll have your public IPs.

Once set up, here is the procedure to find your static IP addresses on GCP:

• On your GCP account, select the IP addresses service.
• In the list you will find your static IP used by your cluster router.

If you work in a sensitive business area such as financial technology, enabling the Static IP feature can help fulfil the security requirements of some of the external services you use, therefore making it easier for you to get whitelisted by them.

This feature has been activated by default. Since February 1, 2024, AWS charge public IPv4 Addresses. Disabling it may cost you more, depending on the number of nodes in your cluster. Check this link for more information.

Custom VPC Subnet

The VPC feature is currently only available to clusters deployed on AWS with a VPC managed by Qovery and can only be enabled at cluster creation.

Virtual Private Cloud (VPC) peering allows you to set up a connection between your Qovery VPC and another VPC on your AWS account. This way, you can access resources stored on your AWS VPC directly from your Qovery applications.

A VPC can only be used if it has at least one range of IP addresses called a subnet. When you create a cluster, Qovery automatically picks a default subnet for it. However, to perform VPC peering, you may want to define which specific VPC subnet you want to use, so that you can avoid any conflicting settings. To do so, you can enable the Custom VPC Subnet feature on your cluster. For more information on how to set up VPC peering, see our dedicated tutorial.

Use existing VPC

The Deploy on existing VPC feature is currently only available to clusters deployed on AWS and GCP when you select Deploy on my existing VPC VPC mode and can only be enabled at cluster creation.

Use existing VPC - AWS:

You have to specify the VPC id (1) and ensure that in your VPC settings you have enabled the DNS hostnames (2):

Then you have to specify the different subnets ids:

EKS:

The EKS subnets are mandatory, you have to specify at least one subnet id per zone (1) and ensure you have enabled the auto-assign public IPv4 address setting on your subnets (2).

If you have activated Karpenter, you will have to specify at least one subnet id per zone. These subnets have to be private and connected to internet through a NAT Gateway. They will be used for AWS Fargate profile.

Managed databases:

This section is exclusively for enabling managed databases (container databases will be enabled by default).

Depending on the managed databases you want to you use (MongoDB, RDS:MySQL/PostgreSQL and Redis), specify at least one subnet id per zone.

Use existing VPC - GCP:

In GCP you have two VPC modes: Automatic or Custom.

If you are using an automatic or a custom VPC, you have to set:

• Your VPC Name
• External project id (optional): by default, the project id used is the one specified in the credentials file. But if your VPC is defined in another GCP project, you have to specify the Project id.

In addition if you are using a custom VPC, you have to set:

• Your Subnet range name (https://console.cloud.google.com/networking/networks/details/<your-vpc>?project=<your-project>&pageTab=SUBNETS)

You can also specify (optional):

• Pod ipv4 address range name
• Additional cluster pod ipv4 ranges names (separated with a comma)
• Ipv4 service range name

For these ranges, you have to create Secondary IPv4 ranges inside your subnet.

Please keep in mind that enabling them later may not be possible.

Network

The Network tab in your cluster settings allows you to update your Qovery VPC route table so that you can perform VPC peering. For step-by-step guidelines on how to set up VPC peering, see our dedicated tutorial.

Performing Actions on your Clusters

Qovery allows you to update, stop, restart or delete your clusters at organization level.

Action	Description
Updating a cluster	To redeploy your cluster after a change has been made to it.
Stopping a cluster	To temporarily stop your cluster. Some services you have subscribed to via your cloud provider may still be active and incur costs when your cluster is stopped. For more information, see Stopping a cluster.
Restarting a cluster	To restart your cluster after it has been temporarily stopped.
Deleting a cluster	To delete your cluster. This is final and needs to be done properly to ensure all the services deployed by Qovery on your cloud provider's account are disabled, with no leftover cloud-related costs. For more information, see Deleting a cluster.

To access these actions:

1. Open your Qovery Console.

2. On the left menu bar, click on the Cluster page:

   

3. To view your cluster actions, click Play button:

   

   A dropdown menu unfolds, featuring all the actions available on your cluster.

How was it? Did this tutorial work?  Yes  No

You can follow the execution of the action via the cluster status and/or by accessing the Cluster Logs

Updating a Cluster

If you made a change on your cluster, you need to run an update on your cluster to propagate remotely the new configuration.

To update your cluster, select the action Update from the drop-down menu.

A confirmation pop-up window opens before triggering the action.

Once confirmed, the status of your cluster turns Updating... (orange status).

Once the update is complete, the status dot next to your cluster turns green.

Stopping a Cluster

Qovery allows you to temporarily stop your cluster instead of deleting it.

When you stop a cluster from the Qovery console, only the workers nodes managed by Qovery are stopped. If you have subscribed to services via your cloud provider (load balancing, storage system, or any other managed services), they will remain active and you will be charged for them. For more information, please contact your cloud provider. To permanently delete a cluster and all its associated costs, see Deleting a Cluster.

To temporarily stop a cluster, select the Stop action from the drop-down menu. A confirmation pop-up window opens before triggering the action.

Once confirmed, the status of your cluster turns to Pausing... (orange status).

Once the stop is complete, the status dot next to your cluster turns to grey, and the status of your cluster turns to Paused (gray status).

Restarting a Cluster

You can restart a cluster after it has been temporarily stopped.

To restart your cluster, select the action Resume from the drop-down menu.

A confirmation pop-up window opens before triggering the action.

Once confirmed, the status of your cluster turns to Updating... (orange status).

Once your cluster has restarted, the status dot next to your cluster turns to green.

Deleting a Cluster

Deleting a cluster from the Qovery console is final and cannot be reverted.

To only temporarily stop a cluster, see Stopping a Cluster.

To delete a cluster, open the ... section and press Delete Cluster.

3 options can be chosen to delete a cluster:

1) Default This is the default behaviour, this option shall be chosen every time you want to delete properly a cluster from the Qovery console AND your cloud provider account.

This operation will delete:

• Cloud provider: any resource created by Qovery on your cloud provider account to run this cluster will be deleted, including any application running on it.
• Qovery organization: the configuration of this cluster and any linked environment.

Please note that you will have to manually delete on your cloud account:

• the S3 bucket created at cluster installation
• the image registry linked to this cluster
• any resource created by a lifecycle job that will not be properly deleted during the environment deletion event.

Check this section to find these elements and delete them.

2) Delete Cluster on cloud provider and Qovery configuration

This option shall be chosen when the cluster delete operation with the Default option fails since you have manually modified/deleted the RDS instances created by Qovery on your cloud provider account.

This operation will delete:

• Cloud provider: any resource created by Qovery on your cloud provider account to run this cluster will be deleted, including any application running on it.
• Qovery organization: the configuration of this cluster and any linked environment.

Please note that you will have to manually delete on your cloud account:

• the S3 bucket created at cluster installation
• the image registry linked to this cluster
• any managed database that was created via Qovery
• any resource created by a lifecycle job that will not be properly deleted during the environment deletion event.

Check this section to find these elements and delete them.

3) Delete Qovery config only

This option shall be chosen when you have already deleted any Qovery resource on your cloud account and you want to delete the cluster object from your Qovery console.

This operation will delete:

• Cloud provider: nothing will be removed from your cloud account. You will have to manually delete any resource created by Qovery directly from your cloud provider console.
• Qovery organization: the configuration of this cluster and any linked environment.

Check this section to find these elements and delete them.

Once confirmed, the cluster status turns to Deleting... (red status) and once the deletion is complete, the cluster is removed from your organization settings.

Audit logs

To get the cluster filtered audit logs, open the ... section and press See audit logs.

You will be redirected to the audit logs section. A filter on the dedicated cluster will be applied. You only see the audit logs regarding cluster operations.

Get your cluster id

To get your Qovery cluster id, open the ... section and press Copy identifier.

The cluster id in Qovery will be in your clipboard.

Get your cluster kubeconfig file

If you need to get your kubeconfig file, open the ... section and press Get Kubeconfig.

Then the kubeconfig yaml file will be automatically downloaded.

Logs

Qovery allows you to access the logs of your cluster in order to follow its installation or investigate any issue happening on it.

To access the logs you need to open the cluster, click the log button

A new window is opened, displaying the logs of the cluster.

The tab system on the right allows you to access the cluster information and, if an error occurs, the detail of the error.

The error message should provide you enough information to solve the issue. If that's not the case, feel free to ask for support on our forum

You can add the generated public SSH key at cluster creation (see Creating a Cluster), or later from your cluster settings.

To do so:

• on your Qovery Console, access your cluster settings.
• In the Remote Access tab, enter your SSH key and click Save.
• Launch the Update Cluster action to propagate the new key.

Use custom domain and wildcard TLS for the whole cluster (beta)

By default, Qovery provides a domain (ex bool.sh) on every deployed cluster. It is used to provide a DNS and TLS certificate to every application requiring external access on a cluster.

You can customize the domain for every application. However, when it comes to having more than 100 custom domains with the same domain you will hit Let's Encrypt quotas.

To overcome this issue, you can use a wildcard TLS certificate for the whole cluster. It will allow you to have as many DNS records for a single domain as you want on the same cluster with a single TLS certificate.

At the moment, Qovery only supports wildcard TLS certificates with Cloudflare. To use it, you need to have a Cloudflare account and a domain name managed by Cloudflare. If you don't have one, you can create a free account and transfer your domain to Cloudflare.

Once you have a Cloudflare account and a domain name managed by Cloudflare, you need to create a Cloudflare API token. Go into your Cloudflare account, click on your profile picture, then My Profile. In the API Tokens section, click on Create Token. In the Create Custom Token section, select the following permissions:

• API token a descriptive name: Qovery domain your domain name
• Permissions:
  • Zone - DNS - Edit
  • Zone - Zone - Read
• Zone Resources:
  • Include - Specific zone - your domain name

To finish, click on Continue to Summary and Create Token. Save the token somewhere safe, you will need it later.

Prepare the Token, the Cloudflare account email and the domain to be set on your cluster. Now contact Qovery and request to use your domain.

Cleaning up a Cluster from your AWS Account

The following troubleshooting procedure is intended for AWS users who did not properly delete their cluster before revoking Qovery's access to their platform.

To properly delete your clusters and avoid any unexpected issues or costs, see Deleting a Cluster.

To clean up a Qovery cluster from your cloud provider account, go to AWS Console>Services>Management & Governance>Resource Groups & Tag Editor> Create Resource Group:

Step	Description
1	In the Group type area, select Tag based.
2	In the Tags field of the Grouping criteria area, enter ClusterId.
3	Click Add.
4	Click Preview Resources. All your Qovery clusters are now displayed in the Group resources table, and you can delete them by hand.

FAQ

What is a cluster?

Please refer to the basic concepts section

Why do I need a cluster?

Qovery is built on top of Kubernetes, which means we need Kubernetes clusters to be able to deploy and run your applications.

Thanks to clusters, you can easily deploy several (and many) instances of the same application, so that if one fails, the others can instantly take over. Also, clusters can auto-scale, meaning that the number of worker nodes in a cluster can automatically go up or down as traffic fluctuates on your application(s), thus ensuring high availability and performance. Clusters are also extremely useful to isolate your production environment from your staging environment.

In short, through the use of clusters, Kubernetes provides you with a resilient, flexible and powerful infrastructure, fit for production environment needs and requirements. And with the help of Qovery, setting up and maintaining your Kubernetes clusters has never been easier.

What are the different instance types available when creating a cluster?

The range of instance types available at cluster creation depends on your cloud provider:

• AWS offers over 400 instance types. You can view their details on the official AWS website, as well as their pricing.
• Scaleway also offers a wide range of instance types, whose details and pricing you can view on the official Scaleway website.
• GCP clusters are deployed in auto-pilot mode so you will have access by default to every instance type available

Qovery supports only instance types having an x86_64 or ARM architecture.

How does Qovery handle Kubernetes version upgrades?

As far as cluster upgrades to a newer version of Kubernetes are concerned, our Qovery engineering team handles everything in due time, so you don’t even need to think about it!

You may notice that Qovery does not provide you with the latest Kubernetes version offered by your cloud provider. This is due to the high amount of testing we need to perform to ensure smooth upgrades with no interruptions for your applications. Our priority is always to guarantee you maximum uptime.

Please DO NOT upgrade the cluster version by yourself from the cloud provider console.

Proceeding with a cluster version upgrade outside of Qovery will prevent any future update on this cluster and might be irreversible preventing Qovery from properly deploying on this cluster. Most importantly will expose you to some unknown / untested areas which can put your application stability at risks.

If you did update by mistake, then you need to reach to Qovery team in order to get some help.

Usually, we work on a given upgrade for one month of intensive testing on our end in order to make sure everything will be smooth for you. Once we are pretty confident our stack is stable, we move on with the following steps which last approximately 3 weeks:

1. Notify users about new version coming in approximatively 1 month before
2. Upgrade clusters for a handful of beta-tester customers (1 week)
3. Make available the new version for all clusters (new or existing)
4. Upgrade all non-production flagged clusters (1 week)
5. Upgrade all clusters the production clusters (1 week)

If, somehow the planning or timeframe for the upgrade is clashing with your business needs, you will be able to trigger the upgrade of your cluster manually via the "Upgrade to XX.XX" action available from the Play menu of your cluster. This action will be available on your cluster once we will make the new version available globally (step 3), you will notice that the Play button of your cluster will be highlighted in yellow.

What do you do when a vulnerability is found?

Security is our main concern. When a vulnerability is found, here are the actions that we take:

1. We quickly identify how significant is the impact of the vulnerability.
2. We look at how we can solve or mitigate the vulnerability.
3. We transparently communicate with our customers about the vulnerability to help them take the right actions.