Integrate your application logs to Cloudwatch

Add Kubernetes pod logs into Cloudwatch to perform full text search

Qovery provides by default an easy way to get access to your logs through the Console or the CLI. For statistics, debugging or security reasons, you may want to access all logs and perform a full-text search inside them.

Qovery implementation is based on Loki for performance and cost-effective reasons. However, Loki is not a full-text search engine. It is a log aggregation system. It is not designed to be queried directly.

Serveral solutions exists, with and without 3rd parties. We will cover here a solution without a third party. But if you're interrested, you can take a look at Datadog integration.

Note: in this tutorial, we are using Fluent-bit with proposed solutions above. However, if none of those solutions suits your needs, feel free to look at supported solution on the official website.

AWS permissions for Cloudwatch

We will create a dedicated service account (note: STS account can be used, but for simplicity reasons, we will use a dedicated service account).

On IAM create a policy with the following permissions, and name this policy fluent-bit-write-policy:

policy content

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CloudWatchLogs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutRetentionPolicy",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
}
]
}

policy create

Once done, let's create a user and attach the policy to it:

User create

User permissions

Finish the user creation and keep credentials for the coming section.

Helm

Ensure you have the following elements before going ahead:

  1. Helm should be installed on your machine. If you don't have it, you can follow the official documentation.
  2. You need Kubeconfig configuration file and permissions to access the cluster. You can use the same documentation as kubectl to get the kubeconfig file.

We will use AWS fluent-bit Helm Chart to setup logs streaming:

helm repo add eks https://aws.github.io/eks-charts

Create on your workstation a values.yaml file to setup your custom configuration and adapt required fields:

cloudWatch:
enabled: true
region: "<enter your cluster region>"
logGroupName: "/aws/eks/fluentbit-<enter your cluster region>/logs"
logRetentionDays: 7
env:
- name: "AWS_ACCESS_KEY_ID"
value: "<your access key>"
- name: "AWS_SECRET_ACCESS_KEY"
value: "<your secret access key>"
firehose:
enabled: false
kinesis:
enabled: false
elasticsearch:
enabled: false

You can take a look at additional configuration options on the AWS provided chart

Then deploy fluent-bit with the following command:

helm upgrade --install aws-for-fluent-bit -f values.yaml --namespace fluent-bit --create-namespace eks/aws-for-fluent-bit --version 0.1.21

You should start seeing fluent-bit pods. Take a look at the logs to ensure there is no configuration issue.

Cloudwatch usage

You can now use Cloudwatch to look at your logs. Connect to Cloudwatch, go into the Logs insight section, then you can perform queries:

cloudwatch search

  1. Select the fluent-bit group of logs
  2. Create a query (syntax examples)
  3. Run your query
  4. See the result and expand to filter on other elements