Integrate your application logs to Cloudwatch

Add Kubernetes pod logs into Cloudwatch to perform full text search

Qovery provides by default an easy way to get access to your logs through the Console or the CLI. For statistics, debugging or security reasons, you may want to access all logs and perform a full-text search inside them.

Qovery implementation is based on Loki for performance and cost-effective reasons. However, Loki is not a full-text search engine. It is a log aggregation system. It is not designed to be queried directly.

Serveral solutions exists, with and without 3rd parties. We will cover here a solution without a third party. But if you're interrested, you can take a look at Datadog integration.

Note: in this tutorial, we are using Fluent-bit with proposed solutions above. However, if none of those solutions suits your needs, feel free to look at supported solution on the official website.

AWS permissions for Cloudwatch

We will create a dedicated service account (note: STS account can be used, but for simplicity reasons, we will use a dedicated service account).

On IAM create a policy with the following permissions, and name this policy fluent-bit-write-policy:

policy content

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CloudWatchLogs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutRetentionPolicy",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
}
]
}

policy create

Once done, let's create a user and attach the policy to it:

User create

User permissions

Finish the user creation and keep credentials for the coming section.

Helm

We will use AWS fluent-bit Helm Chart to setup logs streaming and deploy it with Qovery.

Add the AWS EKS helm repository

Add the AWS EKS helm repository in your Qovery settings by following this documentation

  • Repository name: eks
  • Kind: HTTPS
  • Repository URL: https://aws.github.io/eks-charts

Create and deploy the helm chart within Qovery

Create a helm service in the Qovery environment of your choice (preferrably within a dedicated Tooling project) by following this documentation and these values:

  • General:
    • Application name: fluent-bit
    • Source:
      • Helm source: Helm repository
      • Repository: eks (the name given during the AWS EKS helm repository added in the previous step)
      • Chart name: aws-for-fluent-bit
      • Version: 0.1.21 (this is the version we used for this setup, update it based on the chosen version)
      • Allow cluster-wide resources ✔️
  • Values
    • Values override as file:
    • File source: Raw YAML
    • Raw YAML:
priorityClassName: system-node-critical
cloudWatch:
enabled: true
region: "<enter your cluster region>"
logGroupName: "/aws/eks/fluentbit-<enter your cluster region>/logs"
logRetentionDays: 7
env:
- name: "AWS_ACCESS_KEY_ID"
value: qovery.env.AWS_ACCESS_KEY
- name: "AWS_SECRET_ACCESS_KEY"
value: qovery.env.AWS_SECRET_ACCESS_KEY
firehose:
enabled: false
kinesis:
enabled: false
elasticsearch:
enabled: false

You can take a look at additional configuration options on the AWS provided chart

Now get to the last step and just Create the service on Qovery.

Store the AWS Secrets as Qovery secrets

In the previous step we have assigned the macro qovery.env.qovery.env.AWS_ACCESS_KEY and qovery.env.AWS_SECRET_ACCESS_KEY to the AWS secrets. In this step we will create these secrets within the Qovery console.

  • Open the service overview of the created Datadog service
  • Enter the Variables section
  • Add a new Variable with:
    • Variable = AWS_SECRET_ACCESS_KEY
    • Value = <your_SECRET_ACCESS_KEY>
    • Scope = Service (so that it is accessible only to this service)
    • Secret variable ✔️
  • Add a new Variable with:
    • Variable = AWS_ACCESS_KEY
    • Value = <your_ACCESS_KEY>
    • Scope = Service (so that it is accessible only to this service)
    • Secret variable ✔️

If you need more information on how to manage your environment variables, have a look at this documentation

Deploy your chart

Open the Play button and trigger the deployment of your chart.

Cloudwatch usage

You can now use Cloudwatch to look at your logs. Connect to Cloudwatch, go into the Logs insight section, then you can perform queries:

cloudwatch search

  1. Select the fluent-bit group of logs
  2. Create a query (syntax examples)
  3. Run your query
  4. See the result and expand to filter on other elements