Maintenance

This guide will provide inputs about maintenance with Qovery. Qovery provides automatic and silent updates as much as possible. With and without cloud providers.

Kubernetes and components, patches, and upgrades

Qovery manages Kubernetes updates through the Cloud provider update mechanism and ensures full compatibility with all deployed infrastructure components (Nginx ingress, cert-manager, CNI, CSI, etc.) inside the Kubernetes cluster.

To avoid as maximum as possible small downtimes, Qovery is using the rolling update strategy for the Kubernetes cluster and components. This strategy is the default strategy for Kubernetes and is the most reliable one. You may need to adapt some liveness and readiness probes for some long-running applications to avoid downtimes.

Security patches and minor updates are applied automatically and silently by the cloud provider. Kubernetes major updates are applied automatically by Qovery to ensure compatibility between every deployed components inside the cluster.

1. While Qovery allows customers to access Kubernetes cluster and manually deploy resources, Qovery is not responsible for any issues that may occur on those deployed resources.
2. Qovery support can be canceled by Qovery, if the customer is manually updating or upgrading the Kubernetes cluster or components managed by Qovery.

Managed services patches and upgrades

By default, every managed service deployed by Qovery is configured with automatic patches and upgrades proposed by the cloud provider.

Major version upgrades are up to the end user to decide when it's the right time to upgrade.

Cloud providers' limits

Cloud providers are using quotas for various reasons. Some of them are to prevent abuse, some others are to prevent overloading the infrastructure, and others are to prevent an excessive bill.

It occurs that some customers are reaching the limits of their cloud provider. In this case, Qovery gives the information in the infrastructure or applications logs.

It is up to the customer to contact the Cloud provider via ticketing support to increase the limits.

Rotating system credentials

Some customers want to rotate their system credentials because on legal requests, security requirements, or other reasons. Qovery provides makes it simple to rotate credentials.

Here is the way we recommend to avoid any downtime on your cluster and for your application deployments. Open your AWS console and open the Qovery user in the IAM service.

Click on the Security credentials tab, you will see one access key present:

For a single account, we can create up to two access keys. So we can create a new one, request a cluster deployment, wait for the deployment to be done, and then delete the old one.

This key is also used to connect to the Kubernetes cluster and make deployments. You may encounter deployment failures if the key is deleted before the deployment is done. We advise your to wait 1h or 2h before deleting the old key.

You can now 2 ways to rotate your credentials, select the one you prefer:

• Manual: you update manually credentials from the Qovery interface
• Automatic: you update automatically credentials with a script

Manual rotation

You can update or rotate manually credentials on your AWS account this way:

1. Click on the Create access key button
2. Save the access key and secret access Key in a safe place
3. Go to your Qovery dashboard to update the credentials on Qovery console.
4. Deploy the cluster once again to apply changes
5. Once the cluster is fully updated, wait 2h (to ensure all ongoing deployments are done)
6. Delete the old access key from the AWS console:

If you have a doubt about the old credentials deletion, you can simply deactivate the old access key for a while and delete it later.

Automatic rotation

Another way to do it more programmatically. Here is a script to perform those actions, adapt it to your needs if you need and add it to your

#!/bin/bash

############## VARIABLES AND INSTRUCTIONS ##############

# Ensure you have jq and awscli installed

# 1. Ensure your AWS environment variables are set: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
# 2. AWS username to perform the rotation
aws_iam_username="Qovery"
# 3. Use qovery CLI to generate a dedicated token
qovery_token="xxx"
# 4. Organization ID can be retrieved inside the Qovery console URL, where your cluster is located
organization_id="xxx"
# 5. Get your credentials: curl -s -X GET -H "Content-type: application/json' -H 'Authorization: Token $qovery_token" "https://api.qovery.com/organization/$organization_id/aws/credentials"
credentials_id="xxx"
# 6. Name of the credentials to update
credentials_name="My organization credentials"
# 7. Cluster ID can be retrieved inside the Qovery console URL, where your cluster is located
cluster_id="xxx"
# 8. Set the delay to wait before deleting the old key in seconds (do not go below 7200)
delay_before_delete_old_key=7200

############## DO NOT EDIT ##############

set -e

echo "[+] Ensure there is only one Access Key"
old_aws_access_key=$(aws iam list-access-keys --user-name $aws_iam_username | jq -r '.AccessKeyMetadata[].AccessKeyId')
if [ $(echo $old_aws_access_key | grep -c ' ') -ne 0 ]; then
  echo "ERROR: more than one access key found, please delete the one not used by Qovery"
  exit 1
fi
if [ "$old_aws_access_key" == "" ] ; then
  echo "ERROR: no access key found, are you sure it's the correct user?"
  exit 1
fi
echo " -> Current (future old) key detected: $old_aws_access_key"

current_time=$(date +"%s")
max_time=$((current_time + delay_before_delete_old_key))
cluster_status=""

echo "[+] Create a new Access Key"
new_aws_access_key_json=$(aws iam create-access-key --user-name $aws_iam_username)
new_aws_access_key=$(echo $new_aws_access_key_json | jq -r '.AccessKey.AccessKeyId')
new_aws_secret_key=$(echo $new_aws_access_key_json | jq -r '.AccessKey.SecretAccessKey')
echo " -> Successfully created a new access key: $new_aws_access_key"

echo "[+] Update Qovery credentials"
curl -s -X PUT -H "Content-type: application/json" -H "Authorization: Token $qovery_token" -d "{\"name\": \"$credentials_name\", \"access_key\": \"$new_aws_access_key\", \"secret_key\": \"$new_aws_secret_key\"}" "https://api.qovery.com/organization/$organization_id/aws/credentials/$credentials_id" 1>/dev/null

echo "[+] Deploy the cluster with the new credentials"
curl -s -X POST -H "Content-type: application/json" -H "Authorization: Token $qovery_token" "https://api.qovery.com/organization/$organization_id/cluster/$cluster_id/deploy" 1>/dev/null

echo "[+] Wait for the cluster deployment to be done"
sleep 15
while [ "$cluster_status" != "RUNNING" ]; do
  sleep 60
  cluster_status=$(curl -s -X GET -H "Content-type: application/json" -H "Authorization: Token $qovery_token" "https://api.qovery.com/organization/$organization_id/cluster/$cluster_id/status" | jq -r '.status')
  echo " -> $(date "+%H:%M") Waiting for the cluster deployment to be done. Current status: $cluster_status..."
  # Ensure the cluster is in a valid state
  if [ "$cluster_status" != "DEPLOYMENT_QUEUED" ] && [ "$cluster_status" != "DEPLOYING" ] && [ "$cluster_status" != "DEPLOYED" ] && [ "$cluster_status" != "RUNNING" ]; then
    echo "ERROR: the cluster does not have a correct status, please check cluster logs and fix the issue. Then delete the key $old_aws_access_key and retry"
    exit 1
  fi
  if [ $(date +"%s") -gt $max_time ]; then
    echo "ERROR: timeout reached, the cluster is not deployed yet, please check cluster logs and fix the cluster issue. Then delete the key $new_aws_access_key and retry"
    exit 1
  fi
done

echo "[+] Waiting up to 2h to ensure all ongoing deployments are done ($(date -d @$max_time))"
while [ $(date +"%s") -lt $max_time ]; do
  sleep 10
done

echo "[+] Delete the old Access Key"
aws iam delete-access-key --access-key-id $old_aws_access_key --user-name $aws_iam_username

echo "[+] Done"

You will see the following output:

[+] Ensure there is only one Access Key
 -> Current (future old) key detected: xxx
[+] Create a new Access Key
 -> Successfully created a new access key: yyy
[+] Update Qovery credentials
[+] Deploy the cluster with the new credentials
[+] Wait for the cluster deployment to be done
 -> 15:04 Waiting for the cluster deployment to be done. Current status: DEPLOYING...
 -> 15:05 Waiting for the cluster deployment to be done. Current status: DEPLOYING...
 -> 15:06 Waiting for the cluster deployment to be done. Current status: DEPLOYING...
 -> 15:07 Waiting for the cluster deployment to be done. Current status: RUNNING...
[+] Waiting up to 2h to ensure all ongoing deployments are done (Fri Nov 11 03:22:57 PM CET 2022)
[+] Delete the old Access Key
[+] Done