AWS Firelens functions as a log routing agent exclusively designed for containers that operate within Amazon Elastic Container Service (ECS). As a reminder, applications on ECS are executed as Docker containers. Additionally, with the Fargate launch type, ECS manages a serverless infrastructure that can run these containers seamlessly.

You can effortlessly route your container logs to Fluentd or Fluent Bit With FireLens for Amazon ECS. This enables you to transmit them directly to AWS services and APN destinations, in addition to facilitating log delivery to CloudWatch Logs.

Moreover, AWS FireLens streamlines data aggregation across your entire ECS cluster, and its flexible architecture allows you to easily incorporate additional data sources, parsers, filters/buffering, and output plugins.

In this article, we’ll see how to forward AWS ECS logs to Apica using AWS FireLens

Configuring AWS FireLens to forward logs

One of the key benefits of AWS FireLens is its remarkable versatility – it can be linked to virtually any service endpoint, provided that the data sinks can handle general-purpose JSON over HTTP, FluentForward, or TCP protocols.

That being said, FireLens takes your ECS container log outputs, performs the necessary routing configuration, and delivers the logs to the desired logging service. To make this happen, you can leverage the AWS FireLens log driver to set up Fluentd or Fluent Bit output plugin configurations.

Your container logs are then directed to a sidecar or standalone FireLens container within your ECS cluster, which in turn dispatches them to their intended destination as defined in your task logConfiguration. Plus, for even more advanced use cases, you can utilize the options field of the FireLensConfiguration object in your task definition.

To set up log forwarding on AWS ECS, follow the steps given below:

  • Navigate to the task definition that you need logs to be sent to Apica
  • Click on Edit container
  • Navigate to the STORAGE AND LOGGING section
  • By default, the container will be sending logs to AWS CloudWatch. Uncheck Auto configure to CloudWatch Logs
  • Next, choose the Log driver as awsfirelens
  • Add the following variables in the Log options section
AWS Firelens
  • The following are the mandatory fields that need to be passed in order to forward logs to Apica
				
					Name     http
Match    *
Host     
Port       443 
URI        /v1/json_batch
Format   json
tls            on
tls.verify  off
net.keepalive  off
compress      gzip
Header Authorization Bearer
  • After saving the container definition, in the Log router integration section, you will see that Enable FireLens integration Type checkbox is checked along with the image which will be used (might differ as per region)

If you are using the Configure via JSON option, in the container definition section, add the following:

				
					 {
            "dnsSearchDomains": null,
            "environmentFiles": null,
            "logConfiguration": {
                "logDriver": "awsfirelens",
                "secretOptions": null,
                "options": {
                    "tls.verify": "off",
                    "net.keepalive": "off",
                    "Format": "json",
                    "compress": "gzip",
                    "Port": "443",
                    "match": "*",
                    "Header Authorization Bearer": "<Apica-TOKEN>",
                    "Host": "<Apica-ENDPOINT>",
                    "tls": "on",
                    "URI": "/v1/json_batch",
                    "Name": "http"
                }
            }
    }
  • Once you are done with the setup, you should see a logrouter container along with your container (for example, the wordpress container below).
FarGate

This way your ECS container logs will now be forwarded to your Apica instance and will be visible on the Logs page on the Apica UI. The Namespace will be mapped to your ECS cluster name and the Application is mapped to the task definition name.

Creating a custom AWS Fluent Bit image

To include custom parsers in your Fluent Bit configuration, you’ll need to create a custom Fluent Bit image by following these steps:

1. Clone the repository available at https://github.com/Apicaai/Apica-firelens-fluentbit and go to the custom-fluent-bit-image folder.

2. Use the command ‘docker build -t Apica-config’ to build a new Docker image with the fluent-Apica configuration.

3. The image must be pushed into the private Docker registry to use it later in the container. Tag the image using the command-

‘docker tag image-id Username.dkr.region.amazon.com/Apicaconfiguration’.

This is necessary to match the repository on AWS ECR to push the image successfully. For more information, refer to: https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html.

4. Use the command ‘aws ecr get-login’ to get the Docker login credentials to AWS ECR. Use these credentials to log in.

5. Once logged in, push the image to the private ECR registry using the command ‘docker push Username.dkr.region.amazon.com/Apicaconfiguration’.

6. Finally, reference the configuration file path in the FireLens configuration using the code below:

				
					"firelensConfiguration": {
"type": "fluentbit",
"options": {
"config-file-type": "file",
"config-file-value": "/firelens.conf"
  }
}
By following these steps, you can set up a custom Fluent Bit image that includes custom parsers in the Fluent Bit configuration.

Configuring AWS FireLens to forward your container logs

To forward your ECS Fargate container logs to Apica, you can either configure log forwarding globally for the entire Fargate cluster or configure it individually for each container. It’s better to configure log forwarding globally on the cluster unless you need individual container configurations.

To configure log forwarding globally:

  • First, go to the Log Router Integration section in your Fargate cluster
  • From the AWS Console, enable FireLens integration
  • Select fluentbit from the Type dropdown menu
  • Provide the path to your custom AWS Fluent Bit image with the Apica configuration in the Image field
  • Click on Apply
  • Then click Configure via JSON and paste the logConfiguration code
				
					logConfiguration": {
"logDriver": "awsfirelens",
"secretOptions": null,
"options": {
"tls.verify": "off",
"net.keepalive": "off",
"Format": "json",
"Header Authorization Bearer ": "",
"compress": "gzip",
"Port": "443",
"Host": "your Apica hostname",
"tls": "on",
"URI": "/v1/json_batch",
"Name": "http"
  }
}

After configuring log forwarding, you can access your container logs on Apica by heading to the Logs tab.

To identify your cluster, edit the environment variable on the log_router container and add a name for the CLUSTER_ID parameter.

Thus, you can easily forward your Fargate container logs to Apica by using Apica’s custom AWS FireLens Fluent Bit image and analyzing them alongside other logs and metrics.

If you’re not already using apica.io, sign up for a trial today.