Qovery provides by default an easy way to get access to your logs through the Console or the CLI. For statistics, debugging or security reasons, you may want to access all logs and perform a full-text search inside them.

Qovery implementation is based on Loki for performance and cost-effective reasons. However, Loki is not a full-text search engine. It is a log aggregation system. It is not designed to be queried directly.

Why Qovery does not provides current Loki access?

1. As mentioned Loki is not a full-text search and results may not reflect what you are looking for.
2. Loki is configured to answer usage from Qovery Console and CLI. Using it directly may impact Qovery Console and CLI performances or worst, lose logs and make it irresponsive.

Serveral solutions exists, with and without 3rd parties. We will cover here a solution without a third party. But if you're interrested, you can take a look at Datadog integration.

Note: in this tutorial, we are using Fluent-bit with proposed solutions above. However, if none of those solutions suits your needs, feel free to look at supported solution on the official website.

AWS permissions for Cloudwatch

We will create a dedicated service account (note: STS account can be used, but for simplicity reasons, we will use a dedicated service account).

On IAM create a policy with the following permissions, and name this policy fluent-bit-write-policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchLogs",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutRetentionPolicy",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]
}

You can enforce this policy by cluster if you need, by updating the Resource content. But we want to keep it simple in this tutorial, so we will apply it to all clusters (so you can reuse the same service account if you want for other clusters).

Once done, let's create a user and attach the policy to it:

Finish the user creation and keep credentials for the coming section.

Helm

Ensure you have the following elements before going ahead:

1. Helm should be installed on your machine. If you don't have it, you can follow the official documentation.
2. You need Kubeconfig configuration file and permissions to access the cluster. You can use the same documentation as kubectl to get the kubeconfig file.

We will use AWS fluent-bit Helm Chart to setup logs streaming:

helm repo add eks https://aws.github.io/eks-charts

Create on your workstation a values.yaml file to setup your custom configuration and adapt required fields:

cloudWatch:
  enabled: true
  region: "<enter your cluster region>"
  logGroupName: "/aws/eks/fluentbit-<enter your cluster region>/logs"
  logRetentionDays: 7

env:
  - name: "AWS_ACCESS_KEY_ID"
    value: "<your access key>"
  - name: "AWS_SECRET_ACCESS_KEY"
    value: "<your secret access key>"

firehose:
  enabled: false

kinesis:
  enabled: false

elasticsearch:
  enabled: false

You can take a look at additional configuration options on the AWS provided chart

Then deploy fluent-bit with the following command:

helm upgrade --install aws-for-fluent-bit -f values.yaml --namespace fluent-bit --create-namespace eks/aws-for-fluent-bit --version 0.1.21

You should start seeing fluent-bit pods. Take a look at the logs to ensure there is no configuration issue.

Cloudwatch usage

You can now use Cloudwatch to look at your logs. Connect to Cloudwatch, go into the Logs insight section, then you can perform queries:

1. Select the fluent-bit group of logs
2. Create a query (syntax examples)
3. Run your query
4. See the result and expand to filter on other elements