Amazon Web Services (AWS) Fargate is a serverless compute engine for containers that works on Amazon ECS and EKS. AWS FireLens is a log routing agent built for ECS containers and enables administrators to use task definition attributes to route logs to external log aggregators. AWS FireLens unifies data collection across the ECS cluster, and its pluggable architecture allows adding data sources, parsers, filters/buffering, and output plugins.

In a previous article, we took you through how AWS FireLens works and configuring AWS FireLens to forward AWS Fargate logs for unification and analysis on Apica. While that guide still holds good for users of AWS FireLens versions 1.3 and below, this guide takes you through how to forward container logs for Fargate 1.4 and above to Apica using AWS FireLens. 

But first, for the uninitiated, let us tell you why you should use Apica as your only log data convergence, management, and analytics platform. 

What is Apica?

Apica is a unified data platform for monitoring, observability, log aggregation, and analytics with an infinite storage scale to bring simple and powerful logging to the masses. It uses object storage for data at rest and enables you to centralize logging for your entire stack – from applications and APIs to traditional and serverless infrastructure components. The use of object storage also means that neither do we dictate how much log data you can store and for how long, nor do we force you to favor logging specific components of your environment over others – you get to log everything.

The Apica platform includes:

There are plenty of deployment options available for those who haven’t installed Apica yet. With Apica, you can take either the SaaS route or the PaaS route. For a cloud instance of Apica SaaS, head over to our website and register for a 14-day free trial of Apica SaaS. We’ll have your instance set up in no time. 

On the other hand, there are plenty of deployment options to choose from if you’d like to try out Apica PaaS. You could go for the free-forever Apica PaaS Community Edition or try out our straightforward deployments on Kubernetes, AWS, MicroK8s, or Docker. 

If you’d like to get up and running quickly, try out our Ansible playbook for installing Apica on Ubuntu using MicroK8s. 

Configuring AWS FireLens to forward logs

Possibly the most significant advantage with AWS FireLens is that you can connect it to almost any service endpoint as long as data sinks can process general-purpose JSON over HTTP, FluentForward, or TCP protocols. FireLens magically transforms log outputs of ECS containers and generates the routing configuration needed for sending logs to the logging service. 

The “awsfirelens” log driver allows you to specify Fluentd or Fluent Bit output plugin configurations. Your application container logs are routed to a sidecar or independent Firelens container inside your cluster, which further routes your container logs to their destination as defined in your task “logConfiguration“. Additionally, you can use the options field of the `FireLensConfiguration` object in the task definition to serve any advanced use case.

"firelensConfiguration" : {
     "type" : "fluentbit",
     "essential":true,
     "options" : {
        "config-file-value" : "arn:aws:s3:::mybucket/myFile.conf",
        "config-file-type" : "s3"
     }
  }

The configuration above holds good for anyone using Fargate 1.3 or lower. For those on Fargate 1.4 or above, we’ve taken the AWS-provided Fluent Bit image and added the Apica cluster-id configuration in firelens.conf so that it’s packaged inside the AWS Fluent Bit image. Tasks hosted on Fargate only support the file configuration file type. So, instead of specifying it as s3, we’ll use the AWS Fluent Bit image packaged with our custom configuration.  

"firelensConfiguration": {
   "type": "fluentbit",
   "options": {
       "config-file-type": "file",
       "config-file-value": "/firelens.conf"
   }
}

Creating a custom AWS Fluent Bit image

To package the Apica custom configuration with the AWS Fluent Bit image, do the following.

  • Run the following command to build a new docker image with the fluent-Apica configuration:
    docker build -t Apica-config
  • This image should now be pushed into the private Docker registry for us to be able to use it in the container later on. In order to push the image to the AWS Private registry, tag the image by running the following command: docker tag image-id Username.dkr.region.amazon.com/Apicaconfiguration

Doing this should match the repository on AWS ECR in order to push the image successfully. For more information, read: https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html).

  • Run the following command to get the Docker login credentials to AWS ECR. Use the same credentials to login: aws ecr get-login
  • Once logged in, push the image to the private ECR registry using the command: docker push Username.dkr.region.amazon.com/Apicaconfiguration
  • Reference the configuration file path in the FireLens configuration, as shown below.
"firelensConfiguration": {
   "type": "fluentbit",
   "options": {
       "config-file-type": "file",
       "config-file-value": "/firelens.conf"
   }
}

Configuring AWS FireLens to forward your container logs

There are two ways to configure log forwarding from your ECS Fargate containers to Apica. You could either configure log forwarding globally for an entire Fargate cluster, or you can access and configure each individual container. We recommend configuring log forwarding globally on the cluster unless your use case specifically requires individual container configurations. 

To configure log forwarding on your Fargate cluster globally, do the following. 

  • Access your Fargate cluster from your AWS Console. 
  • Scroll down to Log Router Integration
  • Select Enable FireLens integration.
  • Select fluentbit from the Type dropdown menu.
  • Provide the path to your custom AWS Fluent Bit image that contains the Apica configuration in the Image field. 
  • Click Apply.
  • Next, click Configure via JSON.
  • Edit and paste the following logConfiguration. 
"logConfiguration": {
                "logDriver": "awsfirelens",
                "secretOptions": null,
                "options": {
                    "tls.verify": "off",
                    "net.keepalive": "off",
                    "Format": "json",
                    "Header Authorization Bearer ": "<Apica token>",
                    "compress": "gzip",
                    "Port": "443",
                    "Host": "your Apica hostname",
                    "tls": "on",
                    "URI": "/v1/json_batch",
                    "Name": "http"
			}
		}

Your ECS Fargate logs are now configured to be sent to Apica using AWS FireLens. 

Another way to configure log forwarding using AWS FireLens to Apica is to configure it on each container. In this method, you will access each container definition and configure log forwarding for each container. 

To set up log forwarding on an ECS container, do the following. 

  • Access a container definition and scroll down to the Log configuration section. 
  • From the Log driver dropdown menu, select awsfirelens. 
  • Under Log options, update the log configuration options to reflect the configuration we’ve set for Apica.
  • Click Apply. 
  • Repeat these steps for all the applicable containers in your Fargate cluster. 

Your container logs are now configured to be forwarded to Apica using AWS FireLens. 

After completing either of the above configuration steps, you’ll see a new container named log_router figuring in your list of containers for the cluster. 

Since containers expect the CLUSTER_ID as a parameter, let’s edit the environment variable on the log_router container and add a name to identify your cluster, as shown in the following image. 

Accessing your Fargate container logs on Apica

Now that your Fargate container logs have been configured to be forwarded to Apica, you can head over to your Apica UI and access the Logs tab. 

You’ll now see your Fargate container logs being ingested into Apica.

Conclusion

If you’re not monitoring your ECS Fargate container logs on Apica yet, now’s the time to get started. The flexibility that Apica’s custom AWS FireLens Fluent Bit image provides enables you to seamlessly forward your Fargate container logs to Apica andunify them with logs and metrics across all of your services and infrastructure. Apica enables you to set up a unified log analysis stack for your engineering teams and lets them analyse logs and metrics side-by-side. If you’re not using Apica yet, sign up for a free trial today.