All articles

Infra11 min read

Datadog from scratch: set it up and read the dashboard

A Datadog guide for beginners: the Agent, dd-trace in a Node.js API, service tags, logs linked to traces, and how to read p95 latency, traces and monitors.

Also in Português, Español, Français

I had to study Datadog, and the first time I opened the dashboard it was a shock: charts of every kind, acronyms everywhere, and no idea where to start.

This post is the guide I wish I had read that day. It covers both halves of the job: getting the data in (the Agent and the library in your application) and understanding what shows up on screen (latency, errors, traces, logs and alerts).

An honest warning: the code examples follow the official Datadog documentation, linked in each part, and I didn't run them against an account. The screen images are illustrations I drew to explain each part, not Datadog screenshots. The product changes its look often, but the concepts stay the same.

TL;DR#

  • Datadog brings together three kinds of data: metrics (numbers over time), logs (text events) and traces (the path of a request). What ties the three together are tags.
  • The Agent runs next to your application (in a container, for example), receives the data and sends it to Datadog. Your application talks to the Agent, not to the internet.
  • In a Node.js API, APM comes in with dd-trace, loaded before everything else: node --require dd-trace/init app.js.
  • Three tags are gold: env, service and version. With them, you can filter everything by environment and compare one deploy with the previous one.
  • On a service's dashboard, read three numbers: requests, errors and latency. And latency is read as p95 and p99, not as an average.
  • A trace shows where the time went. The widest span is the first suspect.
  • Watch the cost: too many logs, tags with many values (like a user id) on custom metrics and traces without sampling get expensive.

The three pieces: metrics, logs and traces#

Before any configuration, it helps to separate what each thing is:

  • Metrics are numbers over time: CPU usage, requests per second, queue size. They're cheap to store and good for charts and alerts.
  • Logs are text events (ideally JSON) that your application writes: "order 123 created", "failed to call the payment gateway".
  • Traces tell the story of one request: it hit the endpoint, queried the database, called another API, responded. Each step is a span, with a start, an end and a duration.

What makes Datadog worth it is the link between the three. You see a latency spike on the chart, click, land on the traces from that moment, open a slow trace and, from inside it, see the logs that request wrote. This only works if all three arrive with the same tags.

The architecture: your application talks to the Agent#

The Datadog Agent is a program that runs close to your application: on the same server, as a container next to yours or as a DaemonSet in Kubernetes. It:

  • collects host and container metrics;
  • receives the traces your application sends (port 8126, TCP);
  • receives custom metrics over the DogStatsD protocol (port 8125, UDP);
  • reads the containers' logs;
  • and sends everything to Datadog, with your API key.
The application doesn't talk to the internet: it sends everything to the Agent, and the Agent sends it to Datadog

Two settings show up everywhere:

  • DD_API_KEY: your organization's key. It lives only in the Agent, never in the application code.
  • DD_SITE: which region your account is in (datadoghq.com, datadoghq.eu, us5.datadoghq.com and others). If the site is wrong, the data simply doesn't show up.

Integrating a Node.js API#

1. The Agent in a container#

With Docker Compose, the Agent sits next to the API. The variables below turn on APM, container log collection and DogStatsD:

docker-compose.yml
services:
  datadog-agent:
    image: registry.datadoghq.com/agent:7
    environment:
      - DD_API_KEY=${DD_API_KEY}
      - DD_SITE=datadoghq.com
      - DD_APM_ENABLED=true
      - DD_APM_NON_LOCAL_TRAFFIC=true        # accept traces from other containers
      - DD_LOGS_ENABLED=true
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
      - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true  # accept custom metrics from other containers
      - DD_CONTAINER_EXCLUDE=name:datadog-agent
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
 
  api:
    build: .
    environment:
      - DD_AGENT_HOST=datadog-agent          # where to send traces and metrics
      - DD_ENV=staging
      - DD_SERVICE=orders-api
      - DD_VERSION=1.4.2
      - DD_LOGS_INJECTION=true
    depends_on:
      - datadog-agent

DD_APM_NON_LOCAL_TRAFFIC=true is the detail that trips people up the most: by default, the Agent only accepts traces from its own machine. With the API in another container and without this variable, the traces never arrive.

2. dd-trace in the application#

Node.js APM is the dd-trace library:

npm install dd-trace

It works by intercepting the application's require calls to instrument Express, pg, http and other libraries on its own. That's why it has to load before everything else. The safest way is from the command line, without touching the code:

node --require dd-trace/init app.js

If the application uses ES modules, the documentation points to --import:

node --import dd-trace/initialize.mjs app.js

With that, every HTTP request already becomes a trace, with spans for the database queries and the calls to other APIs, without a single line of instrumentation code.

3. The three tags that tie everything together#

Datadog calls this unified service tagging: three reserved tags, the same across metrics, logs and traces.

  • env: the environment (production, staging).
  • service: the service name (orders-api).
  • version: the code version. It should change on every deploy: that's how Datadog compares the new version with the previous one and shows whether the deploy made latency or errors worse.

In the application, they come from the DD_ENV, DD_SERVICE and DD_VERSION variables, as in the docker-compose.yml above. A good practice is to use the commit hash or the release tag as the version.

4. Logs linked to traces#

With DD_LOGS_INJECTION=true, dd-trace adds the trace id and the span id to every log line. That way, from inside a trace, you see the logs of that request, and from a log you open the trace.

Two things to watch:

  • It only works with JSON logs. Use a logger like Winston, Pino or Bunyan configured for JSON.
  • dd-trace has to load before the logger, again.

A log line looks like this (the documentation's format, with the names from our example):

{"dd":{"service":"orders-api","trace_id":"68c2114800000000669b6b6b2aaf59c9","span_id":"8985025821692657638","version":"1.4.2"},"level":"info","message":"order created"}

5. Custom metrics (carefully)#

To count things in your business, like orders created, you can send metrics through DogStatsD, which the Agent receives on port 8125:

orders.js
const tracer = require('dd-trace')
 
function onOrderCreated(order) {
  // Counts orders. Tags with few possible values only: never an order or user id.
  tracer.dogstatsd.increment('orders.created', 1, { payment_method: order.paymentMethod })
}

The comment in the code is the most important part: each combination of tag values becomes a separate series, and custom metrics are billed per series. A payment_method tag with three values creates three series. A user_id tag with 100 thousand users creates 100 thousand.

Reading a service's dashboard#

With the data coming in, the first screen to open is the service page (in APM, Services, click yours). It looks crowded, but the essentials fit in three charts, the famous RED: Requests, Errors and Duration.

An illustration, not a screenshot: the essentials of a service page are three charts and the list of endpoints

Requests#

How many requests per second the service receives. On its own, this chart says little. It's context for the other two: did latency go up along with traffic? Then it may be a lack of capacity. Did it go up while traffic stayed flat? Then something changed in the code or in a dependency.

Errors#

The error count and, more useful, the error rate: the percentage of requests that failed. 50 errors per minute on a service with 10 thousand requests is 0.5%. On a service with 100 requests, it's half of everything.

Latency: why p95 and not the average#

Latency shows up as percentiles: p50, p75, p90, p95, p99 and the max. An example to make it clear (made-up numbers):

  • 100 requests: 95 take 80 ms, 5 take 3 seconds.
  • The average comes out around 230 ms. It looks like just "a bit slow", and it describes nobody: no request took 230 ms.
  • The p50 (the median) is 80 ms: the typical request is great.
  • The p99 is 3 seconds: some people are waiting three seconds.

The average hides exactly who is suffering. That's why:

  • p50 shows the typical experience;
  • p95 and p99 show the tail, the users who wait the most (and they're often the most important ones, like whoever has the biggest cart);
  • latency alerts usually use p95 or p99.

The resources list#

Below the charts is the list of resources: each endpoint (GET /orders/:id, POST /orders) with its own requests, latency and error rate. That's where you find out which endpoint pulled the chart up. Sort by p95 latency or by error rate and start from the top.

Reading a trace#

Click a point on the chart or a resource and you get to the traces. Open one of the slow ones. It shows up as a flame graph: horizontal bars, one per span, with the width proportional to the time.

Illustration: the widest span shows where the time went

How to read it:

  1. The top bar is the whole request. Its duration is what the user felt.
  2. Look for the widest span. In the example, the call to the payment gateway takes 1.6 of the 1.8 seconds. Optimizing the 40 ms database query would barely change anything.
  3. Sequential spans that could run in parallel are another classic sign: three independent calls, one after the other.
  4. Lots of identical, short spans (hundreds of similar SELECTs) point to the N+1 problem: one query per item, instead of one query for all of them.
  5. Open the span's tabs: the request's logs (if injection is on), the tags and, on spans with an error, the message and the stack.

Searching the logs#

The Log Explorer has its own search syntax. The ones I use most:

service:orders-api status:error
service:orders-api @http.status_code:[500 TO 599]
env:production "payment declined"
service:orders-api -@http.status_code:404
  • Without @ you search by tag (service, env, status, host).
  • With @ you search by an attribute of the log's JSON (@http.status_code, @order.id).
  • In quotes it's a search on the message text.
  • A - in front excludes.

A tip: attribute search is case sensitive. Text search isn't.

Monitors: find out before the user does#

A monitor is an alert. The most useful one to start with is one per service, watching the same signals as the service page:

  • error rate above a threshold (for example, 2% for 5 minutes);
  • p95 or p99 latency above what the user tolerates;
  • no data: the service stopped sending requests. Sometimes it went down in a way that throws no error at all, just silence.

Each monitor has a warning level and an alert level, and a message with where the notification goes, like a Slack channel. Write the message thinking of whoever reads it at three in the morning: what broke, the link to the dashboard and the first step to investigate.

Start with a few. An alert that fires all the time turns into noise, and noise gets ignored exactly on the day it matters.

Watching the cost#

Datadog bills for several different things, and it's easy to get scared by the invoice. The three classic villains:

  • Logs: billed by the volume that comes in and by what stays indexed. Debug logs in production, or one log per item in a loop, blow up the volume. You can filter in the Agent or in pipelines before indexing.
  • Custom metrics with many tag values: like in the user_id example, each combination becomes a billed series.
  • Traces: on a high traffic service, keeping 100% of traces is expensive and rarely needed. Set up sampling.

Prices change and depend on the contract, so check Datadog's pricing page instead of trusting a number from a blog.

Checklist#

  • Agent running with DD_API_KEY and the right DD_SITE.
  • DD_APM_NON_LOCAL_TRAFFIC=true if the application is in another container.
  • dd-trace loaded before everything else (node --require dd-trace/init).
  • DD_ENV, DD_SERVICE and DD_VERSION on every service, with the version changing on every deploy.
  • JSON logs with DD_LOGS_INJECTION=true.
  • Custom metrics only with tags that have few values.
  • Service page: requests, error rate, p95 and p99, and the resources list.
  • Trace: start with the widest span.
  • Monitors for error rate, latency and no data, with a useful message.
  • An eye on the invoice: log volume, metric series and trace sampling.

References#

  • Infra
  • datadog
  • observability
  • apm
  • node.js
  • docker
  • logs

Written by

Software engineer and designer. I write about what I run into while building apps, APIs and products.