Using Datadog with ECS Fargate | Datadog
Back to Architecture Center
Architecture Center Using Datadog with ECS Fargate

Using Datadog with ECS Fargate

August 26, 2024

Introduction

As modern applications become increasingly distributed and containerized, ensuring observability across your infrastructure is crucial for maintaining system health, optimizing performance, and delivering a seamless user experience. AWS Elastic Container Service (ECS) with Fargate has emerged as a popular choice for deploying and managing containerized workloads, offering an experience that abstracts away the complexities of provisioning and scaling compute resources. However, the ephemeral and isolated nature of Fargate tasks can pose challenges when it comes to monitoring, logging, and tracing your applications. Without direct access to the underlying hosts or the ability to run privileged containers, special approaches are required to provide robust observability of applications running in this environment. In this reference architecture, Datadog provides comprehensive observability of an application running In an ECS Fargate environment.

Using Datadog with ECS Fargate

Explanation of the architecture

The following architecture provides complete observability of ECS Fargate workloads including APM tracing, metrics, and logs:

  • Step 1

    The application container with Datadog tracing libraries: By instrumenting the application code with Datadog's tracing libraries, application traces are collected and sent to Datadog. Datadog APM leverages distributed tracing to follow requests across containers, tasks, and hosts in your ECS infrastructure.

  • Step 2

    The Datadog agent container is deployed alongside the main application container in the ECS task definition. This "sidecar" pattern allows the agent to collect metrics and traces from the application container without being tightly coupled to it.

  • Step 3

    Datadog agent collects ECS Fargate metrics from the ECS Metadata API. The metadata collected is applied to metrics and traces emitted through the Datadog Agent sidecar.

  • Step 4

    The FireLens log router in ECS Fargate is a Fluent Bit based container that provides flexible log routing capabilities. The STDOUT and STDERR from the application container are to Datadog. Optionally, FireLens can route logs to Amazon Elasticsearch, Amazon S3, Kinesis Data Firehose, and other third-party log management tools.

  • Step 5

    The Firelens log router forwards the logs to the Datadog platform via HTTPS on port 443.

  • Step 6

    The Datadog agent sends metrics and traces to the Datadog platform via HTTPS on port 443.

  • Step 7

    The Datadog platform receives metrics, logs, and traces from ECS Fargate to their respective endpoints:

    • https://< VERSION >-app.agent.datadoghq.com
    • https://http-intake.logs.datadoghq.com
    • https://trace.agent.datadoghq.com

Authors

Kennon Kwok - Product Solutions Architect

References

Inspiration and reference documents or existing solutions: