AI observability and ops in 2026: what the LLM-in-production stack actually needs.

The procurement question is not whether you need LLM observability. It is whether an APM extension covers your production risk, or whether the workload needs a purpose-built tool with production evals, cost attribution, and drift detection baked in.

APM was designed for deterministic services. Request came in, service did work, response went out, latency and error rate were the two numbers that mattered. An LLM application is non-deterministic on purpose. Two identical requests produce two different outputs. The failure modes are new: a prompt template edited by a product manager accidentally doubles token consumption in production; a retrieval index quietly stops surfacing recent documents; a hallucination-rate signal creeps up two weeks after a model provider rolls a minor version. None of those show up on an APM dashboard because none of them are latency or error-rate events.

What LLM observability needs to capture

Six categories, at minimum. Prompt and response payloads, per call, with retention long enough for eval and audit. Token counts, per call, decomposed by model and by feature. Tool-call traces, including tool arguments and returned values, when the application calls out. Retrieval traces — the query issued to the vector store, the chunks returned, the relevance scores — for RAG applications. Eval scores, run online against a held-out sample, on quality and safety dimensions. Cost attribution rolled up per feature and per end user, not just per API key. The trace shape that unifies these is emerging around OpenTelemetry's GenAI semantic conventions, and every serious vendor now emits or ingests against it.

Purpose-built vs. APM-bundled — where each wins

The APM incumbents added LLM modules through 2024 and 2025. Datadog LLM Observability reached GA in mid-2024 and now sits inside the same platform enterprises already run for infrastructure monitoring. New Relic AI Monitoring and Coralogix AI Observability follow the same shape. When the LLM application is one workload among many and the observability team is already trained on the APM tool, the bundled module wins on operational cost. It loses when the workload needs production evals, prompt-versioned experiment tracking, or dataset-driven regression tests — capabilities the purpose-built vendors built the platform around.

The purpose-built pool sorts into two lanes. Framework-native platforms: LangSmith for LangChain and LangGraph shops, with tight LangSmith-to-LangGraph tracing and a mature eval product; Arize AI (with open-source Phoenix) for framework-agnostic OpenTelemetry-based tracing across LangChain, LlamaIndex, CrewAI, and native SDK code. Category specialists: Weights & Biases Weave and Traces for teams already doing ML experiment tracking; Fiddler AI for shops that need model-risk-management-grade governance; Helicone as the proxy-first lightweight entrant with strong cost tracking; TruLens for open-source eval-heavy workflows; Braintrust for teams that treat evals as CI.

Prompt and response logging with PII redaction

The compliance floor is redaction at the point of ingestion, not at query time. Every vendor in the pool now offers pre-ingestion redaction for common PII categories (name, email, phone, SSN, credit card), and the mature vendors let you extend the redaction dictionary to domain-specific patterns — member IDs, case numbers, internal identifiers. The RFP question is where the redaction runs (in the SDK before send, or in the vendor's ingestion pipeline), and what happens if the redactor fails. Silent leakage of unredacted payloads to a third-party observability vendor is the failure mode compliance cares about most.

Evals in production — who actually does this well

Offline eval on a fixed test set is not the hard problem. Online eval — running a quality or safety judge on a sample of live traffic and alerting on drift — is. LangSmith, Arize, Braintrust, and TruLens sit at the front of the pack on production eval workflows. Weights & Biases Weave is closing in. Datadog's LLM Observability offers eval hooks but the tuning surface is younger than the specialists. The pattern that survives audit is a judge-model eval running on 1 to 5 percent of production traffic, scores aggregated per feature, and an alert wired into the same on-call rotation the APM tool already feeds.

Cost attribution and pricing models

Every vendor prices differently. Datadog LLM Observability rolls into its existing per-host and per-ingested-span pricing. Arize prices per-trace and per-eval-run with an enterprise seat overlay. LangSmith and Braintrust price per-seat with generous trace allowances. Helicone charges per request with a proxy tier. Weights & Biases prices by user seat and stored traces. The pricing unit that matters at scale is cost-per-trace-retained-90-days, because trace volume grows with product adoption faster than any other line. Contract a not-to-exceed at 2× projected peak trace volume, in the MSA.

The buyer-side RFP snippet

Sources

• Datadog, LLM Observability product page. datadoghq.com
• Arize AI, agent observability and Phoenix. arize.com
• LangChain, LangSmith documentation. docs.langchain.com/langsmith
• Weights & Biases, Traces product page. wandb.ai
• Fiddler AI, product page. fiddler.ai
• Helicone, product page. helicone.ai
• Braintrust, product page. braintrust.dev
• Coralogix, AI Observability. coralogix.com
• OpenTelemetry, GenAI semantic conventions. opentelemetry.io

All linked sources were live at time of publish (September 2026). Verify before quoting in a procurement document.