Datadog monitoring baseline

This document is the minimum monitoring baseline to check on a server deployment. The metrics the platform emits on a Kubernetes (Helm) install are in Platform metrics. The app does not include the Datadog SDK. It exports session traces over standard OTLP only when you configure it (off by default, no particular collector required); the keys are in the API server operations doc. The Datadog Agent described here does not handle traces; it watches logs, /readyz, the database and the host. The Datadog Agent collects Docker's JSON stdout logs, /readyz, read-only PostgreSQL queries and host metrics. The definitions are version-controlled only in infra/datadog/definitions, and no tool or CI in this repository calls the Datadog API.

Every threshold is an initial value to tune during the first 7 days after the server deployment. Do not treat a monitor tagged threshold_stage:first-week-tuning as a long-term baseline without the first week's data. Gaps already filled are marked None, and items outside that scope whose priority will be set again after the server deployment are marked Decide after deployment.

Startup and secret handlingCopy Anchor Link

The Agent is created only by the optional infra/compose.datadog.yml. The default stack, which does not name this file, gets no Agent, mount, label or required environment variable at all. The Agent image is pinned to gcr.io/datadoghq/agent:7.83.3.

Do not put DD_API_KEY or the PostgreSQL password for Datadog in the repository or in shell history. On the server, the operator creates /etc/werft/datadog.env and checks that its mode is 600. Change it with DATADOG_ENV_FILE only when you must use a different absolute path. The values are example placeholders, so do not copy real values into documents, tickets or logs.

DD_API_KEY=<operator-supplied>
DD_SITE=datadoghq.com
DD_ENV=alpha
DD_VERSION=<release-sha>
DD_POSTGRES_PASSWORD=<read-only-role-password>

sudo chown root:root /etc/werft/datadog.env
sudo chmod 600 /etc/werft/datadog.env
stat -c '%a %U:%G' /etc/werft/datadog.env

The Agent service reads this absolute path only as an env_file. The compose YAML has no interpolation of the name or value of DD_API_KEY, so an ambient DD_API_KEY cannot bypass it. DD_ENV and DD_VERSION are read from the same file as compose's --env-file and used for the unified tags of the product containers.

The PostgreSQL role is allowed only SELECT on queue_messages, sessions, turns, attempts and provider_usage, and its default transaction is fixed to read-only. The server operator runs the commands below once on an admin connection.

CREATE ROLE datadog LOGIN PASSWORD :'datadog_password';
ALTER ROLE datadog SET default_transaction_read_only = on;
GRANT CONNECT ON DATABASE sessions TO datadog;
GRANT USAGE ON SCHEMA public TO datadog;
GRANT SELECT ON TABLE queue_messages, sessions, turns, attempts, provider_usage TO datadog;

Keep the layers of the existing deploy command as they are, add only this overlay as the last -f, and give the file above as one more --env-file. Below is the example on the local stack's entry file infra/compose.yml. On a server install, list that install's own layers in its place.

docker compose \
  --env-file /etc/werft/datadog.env \
  -f infra/compose.yml \
  -f infra/compose.datadog.yml \
  --profile apps up -d

This form was checked as far as docker compose config with the repository's compose files. Adding the overlay adds one service, datadog-agent, to the service list and puts Datadog labels on api, scheduler, reconciler, egress-proxy and postgres. The service list was checked by rendering; the labels are read from infra/compose.datadog.yml and were not checked in the rendered output. Without DD_ENV or DD_VERSION, the render is refused. Running up on a real server and the Agent reporting to Datadog were not checked (NOT VERIFIED).

The Agent mounts the Docker socket and the container log directory read-only and reads the host PID and cgroup namespaces. With DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true, it also collects dynamic workers. Compose services get the service, env and version labels. For dynamic workers, service is derived from the container short-image, env comes from the Agent's global DD_ENV, and version from the global DD_TAGS. session_id and turn_id are kept only as log attributes for investigation and are not made into metric tags or facets.

Signals and initial monitorsCopy Anchor Link

The original record of the structured logger is timestamp, level, the machine-readable event, the human-readable message, the optional fields, and context (session_id and so on). The log pipeline parses the JSON and uses message as a fallback only when an older record has no event. Detailed values are under fields.*. The metric names in the queries are the names made by log-metrics.json and the PostgreSQL custom queries.

DB check query contractCopy Anchor Link

The custom_queries in infra/datadog/conf.d/postgres.d/conf.yaml are the source of truth. Every query is a single SELECT and does not change any table.

• queue oldest age: compares the oldest created_at of messages not yet claimed and already visible with the DB clock.
• checkpoint pending age: uses the oldest updated_at among sessions with checkpoint_pending_reason IS NOT NULL. Updates to the same session unrelated to the checkpoint can move this time, so it is a lower-bound proxy, not an exact SLI.
• outcome_unknown: the number of turns.outcome_unknown = true rows created in the last 5 minutes.
• stale lease: the number of attempts that are not exited or lost with lease_expires_at < clock_timestamp().
• provider cost: the total cost_usd over the last hour and the number of priced_by = 'fallback' rows. werft.provider.unmetered_tokens_1h is the token total (input, output, cache write, cache read) of priced_by = 'unmetered' rows over the same hour. A billing: none profile leaves cost at 0, so these token counts are the only place where a wrongly declared paid upstream shows up. No monitor is defined, because the normal value depends on the installation's local model usage.

No query returns session_id. The group-by in log-metrics.json also uses only service, env and low-cardinality states. session_id is used only to narrow down the original logs or the DB during an incident.

Applying and verifying the definitionsCopy Anchor Link

The definitions are not applied automatically. After reviewing the JSON, the operator applies the requests entries one by one to the endpoint of each file. log-facets.json is the list of facets to create in the UI and the high-cardinality attributes not to create. dashboard.json is the dashboard create body. Do not create monitors, dashboards or log pipelines that are already applied again; edit the names, queries and filters of the existing ones to match the new definitions. Creating them again leaves two copies of the same definition.

On the server deployment, check the following.

1. When the default stack is rendered and started without the overlay, the service list and settings are the same as before.
2. docker compose config with the overlay refuses a missing DD_ENV or DD_VERSION, the Agent's Docker socket mount is ro, and it reads only /etc/werft/datadog.env.
3. In the Agent status, the Docker logs, http_check:werft-readyz and the PostgreSQL custom queries all succeed.
4. On the server, make /readyz fail briefly and check that the service check changes and recovers. Local verification evidence does not replace an actual long-running stack run or monitor notification delivery.