Qovery provides by default an easy way to get access to your logs through the Console or the CLI. For statistics, debugging or security reasons, you may want to access all logs and perform a full-text search inside them.
Qovery implementation is based on Loki for performance and cost-effective reasons. However, Loki is not a full-text search engine. It is a log aggregation system. It is not designed to be queried directly.
Serveral solutions exists, with and without 3rd parties. We will cover here a solution without a third party. But if you're interrested, you can take a look at Datadog integration.
Note: in this tutorial, we are using Fluent-bit with proposed solutions above. However, if none of those solutions suits your needs, feel free to look at supported solution on the official website.
AWS permissions for Cloudwatch
We will create a dedicated service account (note: STS account can be used, but for simplicity reasons, we will use a dedicated service account).
On IAM create a policy with the following permissions, and name this policy fluent-bit-write-policy
:
{"Version": "2012-10-17","Statement": [{"Sid": "CloudWatchLogs","Effect": "Allow","Action": ["logs:CreateLogGroup","logs:CreateLogStream","logs:PutRetentionPolicy","logs:PutLogEvents"],"Resource": "arn:aws:logs:*:*:*"}]}
Once done, let's create a user and attach the policy to it:
Finish the user creation and keep credentials for the coming section.
Helm
Ensure you have the following elements before going ahead:
- Helm should be installed on your machine. If you don't have it, you can follow the official documentation.
- You need Kubeconfig configuration file and permissions to access the cluster. You can use the same documentation as kubectl to get the kubeconfig file.
We will use AWS fluent-bit Helm Chart to setup logs streaming:
helm repo add eks https://aws.github.io/eks-charts
Create on your workstation a values.yaml
file to setup your custom configuration and adapt required fields:
cloudWatch:enabled: trueregion: "<enter your cluster region>"logGroupName: "/aws/eks/fluentbit-<enter your cluster region>/logs"logRetentionDays: 7env:- name: "AWS_ACCESS_KEY_ID"value: "<your access key>"- name: "AWS_SECRET_ACCESS_KEY"value: "<your secret access key>"firehose:enabled: falsekinesis:enabled: falseelasticsearch:enabled: false
You can take a look at additional configuration options on the AWS provided chart
Then deploy fluent-bit with the following command:
helm upgrade --install aws-for-fluent-bit -f values.yaml --namespace fluent-bit --create-namespace eks/aws-for-fluent-bit --version 0.1.21
You should start seeing fluent-bit pods. Take a look at the logs to ensure there is no configuration issue.
Cloudwatch usage
You can now use Cloudwatch to look at your logs. Connect to Cloudwatch, go into the Logs insight
section, then you can perform queries:
- Select the fluent-bit group of logs
- Create a query (syntax examples)
- Run your query
- See the result and expand to filter on other elements