Best LLM Observability Tools in 2026: 12 Platforms for Tracing, Evaluating, and Monitoring Production LLM Apps

著者: Camellia · 更新日: 2026年11月2日

If your LLM app runs on LangChain, LangSmith covers the most ground with the least setup. If you want the full feature set for free and no vendor lock-in, Langfuse self-hosts at no cost and is now backed by ClickHouse's infrastructure. If traces need to sit next to the APM and infra dashboards you already pay for, Datadog LLM Observability is built for exactly that. And if evaluation, not a pretty dashboard, is the actual job you're hiring for, Braintrust and Galileo (now sold by Cisco's Splunk) lead on testing an application's output systematically instead of eyeballing a log after a customer complains. This guide compares 12 platforms built to trace, evaluate, and monitor LLM applications in production, priced from each vendor's own pricing page.

The thing that makes this category hard to shop is that three different jobs hide under one label. Most vendors claim to do all three and are genuinely strong at one. So instead of one overall ranking, this guide scores each tool per job, prices every option at a real volume instead of the free-tier headline, and treats "speaks OpenTelemetry" as the lock-in question it actually is rather than a checkbox.

Key Facts

• The LLM observability platform market is on pace to grow from $1.97 billion in 2025 to $2.69 billion in 2026, a 36.3% jump, and reach $9.26 billion by 2030 at a 36.2% CAGR, per a Business Research Company report distributed through ResearchAndMarkets.
• 2% of all LLM-call spans logged in production returned an error as of March 2026, and 60% of February 2026's logged errors came from exceeded rate limits rather than model failures, per Datadog's State of AI Engineering report.
• Only 28% of LLM call spans show any cached-read input tokens even though most frontier models support prompt caching, per the same Datadog report, which is exactly the kind of waste a real trace catches and a transcript never will.
• Every gen_ai span, event, metric, and attribute defined in OpenTelemetry's dedicated GenAI semantic-conventions repository is still marked "development" stability, with none marked Stable, so every vendor in this guide claiming full OpenTelemetry support is building on a convention that is still moving.
• Six of the 12 vendors in this guide changed owners or announced a deal in 2026 alone: ClickHouse acquired Langfuse in January alongside its $400 million Series D, Mintlify acquired Helicone and Traceloop announced it's joining ServiceNow in March, Cisco completed its Galileo acquisition and Palo Alto Networks closed its Portkey acquisition in May, and Dynatrace, which agreed in August to buy Arize for $915 million, completed the deal in October.

What "LLM Observability" Actually Covers

Three separate jobs get sold under this one label, and almost nobody is equally good at all three:

1. Tracing and debugging. Capturing every prompt, completion, retrieval step, and tool call in a single run so you can see exactly where something went wrong, not just that it did.
2. Evaluation and scoring. Running a test set against a new prompt or model version before you ship it (offline), or scoring live production traffic after it ships (online), usually with an LLM-as-judge or a custom scorer.
3. Production monitoring and alerting. Watching latency, cost, error rate, and quality drift over time and paging someone when a number moves the wrong way.

A tool built for the first job can look thin on the third, and vice versa. Datadog's whole pitch is job three: correlating LLM spans with the APM and infrastructure alerts a platform team already watches. Braintrust's whole pitch is job two: treating evaluation as the release gate rather than a dashboard you check after the fact. Buying on "it has observability" without asking which of the three you're actually buying is how teams end up with a clean-looking dashboard and no idea whether the app is right.

This guide also covers single-call and RAG-style LLM applications, not just autonomous agents. General AI observability is the broadest version of this idea (logs, metrics, traces, and evals for any AI system). AI agent observability is the narrower case: tracing a multi-step loop where an agent calls tools and decides its own next action across five, ten, or fifty steps. If that loop, not a single prompt-response pair, is what you're instrumenting, read our dedicated agent observability roundup alongside this one. The vendor lists overlap heavily, but an agent buyer cares most about step-level tool-call tracing and evaluating multi-step behavior, while an LLM-app buyer (a chatbot, a RAG pipeline, a classification endpoint) cares more about raw trace economics at volume.

Quick Comparison Table

Score by Job: Tracing, Evaluation, or Monitoring

Rate each candidate on the three jobs separately before you shortlist on a single "best overall" score, because that score almost never exists in this category.

A "Good" in monitoring usually means the dashboards exist but alerting isn't the product's reason for being. A "Limited" in evaluation usually means you'll pair the tool with a specialist (Braintrust or Galileo) rather than trust its built-in scoring to gate a release.

The Real Cost at 1 Million Traces a Month

Free-tier limits are where most of this category's marketing lives, and they're close to useless for budgeting because nobody runs a free-tier volume in production. Here's what a genuinely real month, 1 million billable units, actually costs on each vendor's own published overage math. The billed unit differs by vendor (traces, spans, observations, logs, or GB of data), so read the unit column before comparing the dollar figure.

The flip worth sitting with: the two cheapest headline free tiers in this list (Opik and Langfuse) are also the two cheapest at real volume, because their overage math is public and linear. Datadog, attractive on the free tier, gets expensive fast once volume is real. Five vendors (Arize AX, Helicone, HoneyHive, Traceloop, and Galileo) don't publish a number you can extend past their included volume at all, which is itself a data point about who they're built for.

OpenTelemetry, Self-Hosting, and Licensing

OpenTelemetry support is the actual lock-in question in this category: a tool that ingests standard OTLP traces lets you swap the backend later without re-instrumenting a single line of application code. A tool that only works through its own proprietary SDK is cheaper to adopt today and more expensive to leave. Self-hosting matters here for a reason that's easy to forget: an LLM trace routinely contains the full prompt and completion, which means customer PII, health information, or anything else that can't leave your VPC travels with every single trace, not just with your database.

The licensing line is worth slowing down on. Four of the five open-core names here (Langfuse, Opik, Helicone, and Portkey's gateway) are genuinely permissive open source. The fifth, Arize Phoenix, is not: its LICENSE file names the Elastic License 2.0, a source-available license that lets you read, run, and modify the code but restricts offering Phoenix itself as a competing hosted service. That's a different legal position than Apache or MIT, so a procurement or legal team should check the actual LICENSE file, not a marketing page that just says "open source."

What Changed

This category moved fast enough in 2026 that pricing isn't the only thing worth re-checking before you buy. Half the vendors in this guide changed owners this year, and one older name shut down.

Langfuse is now owned by ClickHouse. ClickHouse announced the acquisition on January 16, 2026, alongside a $400 million Series D round. ClickHouse has stated that Langfuse remains open source under its existing MIT license for core features, that self-hosting at production scale continues, and that Langfuse Cloud keeps operating as a standalone service with the same SLAs and support, per ClickHouse's announcement.

Arize is now owned by Dynatrace. Dynatrace agreed to acquire Arize on August 13, 2026 for $915 million (about $815 million in cash plus replacement equity awards for Arize employees) and completed the acquisition in October. Dynatrace says Arize AX and the Dynatrace platform remain available as standalone offerings and Phoenix remains available as an open-source project, with a longer-term plan to connect Arize's workflows into Dynatrace's broader platform. Pricing continuity wasn't addressed, so re-confirm AX pricing before committing budget past a pilot.

Galileo is now Splunk Agent Observability. Cisco announced its intent to acquire Galileo in April 2026 and completed the acquisition in May, adding it to the Splunk Observability portfolio. Splunk now sells it as Splunk Agent Observability, Galileo's old pricing URL redirects to Splunk's pricing page, and the Free and Pro trace tiers gave way to annual pricing per span volume.

Portkey is now Palo Alto Networks' Prisma AIRS AI Gateway. Palo Alto Networks closed its acquisition of Portkey on May 29, 2026, making it the core AI gateway for its Prisma AIRS security platform. Portkey's own pricing page still lists its open-source, Developer, Production, and Enterprise plans, under a banner announcing the Prisma AIRS AI Gateway.

Helicone is in maintenance mode. Mintlify acquired Helicone in March 2026 and moved it into maintenance mode: security patches, bug fixes, and new model support continue, and Mintlify offered customers help migrating to another platform. Helicone's own site now says it's no longer part of Mintlify, with details still to come, so treat its roadmap as unsettled.

Traceloop is joining ServiceNow. Traceloop announced the move in March 2026, with its technology becoming part of ServiceNow's AI Control Tower. It says OpenLLMetry will remain open source, that it intends to keep selling Traceloop and honor customer obligations, and that roadmap changes will be announced in advance.

Weights and Biases has belonged to CoreWeave since 2025. CoreWeave completed the acquisition in May 2025, which is why Weave's pricing now sits on CoreWeave's Forge pages.

Literal AI is dead. If you've seen it recommended in an older roundup, it shut down. Its own migration guide states the hosted service was "fully operational until October 31st 2025" and that "all data will be permanently deleted" after that date; the main literalai.com domain now 404s. That guide pointed departing users toward LangSmith or Langfuse, two of the stronger picks on this list anyway. Check for this kind of dead-product risk before you shortlist anything from a 2024 or 2025 source.

Self-Hosting and the Compliance Angle

An LLM trace is not a generic log line. It usually carries the full prompt sent to the model and the full completion that came back, so any customer name, contract clause, or health detail that passed through your app sits inside the observability vendor's database too, not just your own. For a team handling regulated data, that turns "which tool is cheapest" into a secondary question behind "which of these can legally run inside our own VPC."

Three platforms here answer that cleanly with a genuinely free self-hosted path for the full observability product: Langfuse and Comet Opik under permissive licenses, and Arize Phoenix under the source-available Elastic License 2.0. Traceloop's OpenLLMetry SDK adds a fourth option at the instrumentation layer, since the SDK itself is free and portable even though Traceloop's own hosted dashboard gates self-hosting behind Enterprise, and Portkey's MIT-licensed gateway self-hosts for free if gateway-level logs are enough. Everything else here (LangSmith, Braintrust, Datadog, W&B Weave, Helicone, HoneyHive, Galileo) either requires an Enterprise contract or a production license to self-host, or doesn't offer it at any price. If your AI agent's data privacy posture rules out sending prompt content to a third party's SaaS by policy, confirm that with legal before, not after, a pilot generates the first trace.

1. LangSmith: Deepest Tracing for LangChain and LangGraph Teams

LangSmith is LangChain's own tracing and evaluation platform, and the setup reflects it: point an environment variable at a LangChain or LangGraph app and every chain, tool call, and retry shows up as a run in a full execution tree with no extra instrumentation. It's not LangChain-only traffic, though. LangSmith exposes a real OTLP endpoint and maps OpenTelemetry's GenAI attributes (gen_ai.system, gen_ai.prompt, and related fields) to its own schema, so a mixed stack can still land everything in one place. Evaluation is built in rather than bolted on: online and offline evals, dataset creation, and human annotation queues ship on every tier. LangChain vs LlamaIndex works through LangSmith's pricing for a five-person team at 50,000 traces a month.

Pricing: Developer $0/seat (up to 5,000 base traces/month, 1 seat max). Plus $39/seat/month, unlimited seats (up to 10,000 base traces/month, one free small serverless deployment). Enterprise custom, with SaaS, hybrid, or self-hosted deployment. Overage beyond base traces bills in LangChain Standard Units at $1.00/LSU, blended with feature-specific metering. Source: langchain.com/pricing.

Best for: Teams already building on LangChain or LangGraph who want tracing that understands the framework with zero setup. Not ideal for: teams that want a predictable flat per-trace bill, since LSU-based overage is genuinely hard to model in advance.

2. Langfuse: The Open-Source Standard, Now Owned by ClickHouse

Langfuse remains the open-source default in this category: the full product (tracing, evaluation, prompt management, datasets) is free to self-host under the MIT license, with Docker Compose for local setup and Kubernetes templates for production. ClickHouse's January 2026 acquisition didn't change that; ClickHouse committed in writing to keeping Langfuse's core MIT-licensed, to continued self-hosting at production scale, and to running Langfuse Cloud as a standalone service. OpenTelemetry support runs across every tier, cloud or self-hosted, and evaluation (LLM-as-judge, datasets, experiments) ships even on the free Hobby plan rather than gated behind a paid tier.

Pricing: Hobby free (50,000 units/month, 2 users, 30-day data access). Core $29/month (100,000 units/month, unlimited users, 90-day access). Pro $199/month (100,000 units/month, 3-year access, unlimited annotation queues; +$300/month Teams add-on). Enterprise $2,499/month (audit logs, SCIM, custom rate limits). Overage runs $8 per 100,000 units from 100K to 1M, $7 from 1M to 10M, $6.50 from 10M to 50M, and $6 above 50M. Self-hosting is free at any scale. Source: langfuse.com/pricing.

Best for: Teams that want the complete feature set for free by self-hosting, with a real exit path if they ever want to leave. Not ideal for: teams that specifically want to avoid acquired-and-absorbed risk in a vendor, though ClickHouse's public commitments here are unusually concrete.

3. Arize (Phoenix and AX): Open Source Core, Now Under Dynatrace

Arize covers both ends of this category under one brand, and that brand now sits inside Dynatrace, which completed its $915 million acquisition in October 2026. Phoenix is the self-hosted half: built on OpenTelemetry and Arize's own OpenInference instrumentation, it runs on Docker or Kubernetes for free, though its license is the Elastic License 2.0, source-available rather than permissive open source, so check that distinction before assuming it behaves like an Apache-licensed project. Arize AX is the commercial evolution, and both AX and Phoenix remain standalone offerings per Dynatrace's own acquisition announcement. AX Free and Pro both include unlimited evaluations, unusually generous for a paid entry tier in this category.

Pricing: Phoenix free and self-hosted under the Elastic License 2.0. AX Free $0/month (25,000 spans/month, 1GB storage, 15-day retention, unlimited users). AX Pro $50/month (50,000 spans/month, 10GB storage, 30-day retention, unlimited users); no overage rate is published. AX Enterprise custom, SaaS or self-hosted. Sources: arize.com/pricing; Phoenix's license is published in Arize-ai/phoenix's GitHub LICENSE file; acquisition details are in Dynatrace's announcement.

Best for: Teams that want a free, OpenTelemetry-native open-source starting point with a real enterprise upgrade path. Not ideal for: teams that assumed Phoenix was Apache or MIT licensed; verify the Elastic License 2.0's restrictions apply acceptably to your use case first.

4. Braintrust: Built Around Evaluation, Not Dashboards

Braintrust leads with evaluation as the primary product rather than a feature bolted onto a tracing tool. Every plan includes unlimited projects, datasets, playgrounds, and experiments, and its "Loop" feature runs autonomous evaluation iteration, generating test cases and refining scorers without a human writing every rubric from scratch. That makes it a natural fit for a CI-gated release process: run the eval suite against a new prompt or model version before it ships, not after a customer notices something's off.

Pricing: Starter free ($10 in model credits/month, 1GB processed data/month, 10,000 scores/month, 14-day retention). Pro $249/month ($100 in model credits/month, 5GB processed data/month, 50,000 scores/month, 30-day retention). Enterprise custom, with on-prem or hosted-private deployment. Overage: processed data $4/GB (Starter) or $3/GB (Pro); scores $2.50 per 1,000 (Starter) or $1.50 per 1,000 (Pro). Source: braintrust.dev/pricing.

Best for: Teams that want evaluation, not monitoring, to be the center of how they ship changes. Not ideal for: a team whose primary need is deep step-level tracing of a running agent; pair Braintrust with a tracing-first tool if that's also a requirement.

5. Datadog LLM Observability: Agent Traces Next to the Infra You Already Watch

Datadog's entry (branded LLM Observability, increasingly folded into a broader "Agent Observability" line) wins through consolidation, not depth: if your infrastructure, APM, and log data already live in Datadog, LLM traces show up correlated with the same services and hosts you already monitor, instead of a sixth tab to check separately. Tracing covers the full execution path (prompts, retrieval, tool calls, agent decisions) with latency, token usage, and errors captured at each step, and it natively supports the OpenTelemetry GenAI semantic conventions. Billing is scoped tightly: only spans that hit an LLM provider count against the metered limit, so tool, workflow, and retrieval spans trace for free.

Pricing: Free up to 40,000 LLM spans/month, 15-day retention. Pro $160/month on an annual commitment ($200 month-to-month, $240 on-demand) for up to 100,000 LLM spans/month; additional LLM spans cost $3.50 per 10,000 billed annually ($4.20 month-to-month, $5 on-demand). Extended retention is available, and the product can be bought without any other Datadog subscription. Source: datadoghq.com/product/llm-observability.

Best for: Teams already running Datadog for APM and infrastructure who want LLM traces in the same place rather than standing up a new vendor. Not ideal for: teams without an existing Datadog relationship, or anyone who needs a self-hosted option for compliance reasons.

6. W&B Weave: Now Living Inside CoreWeave Forge

Weights and Biases, the experiment-tracking platform many ML teams already use for training runs, was acquired by CoreWeave, which completed the deal in May 2025. Weave, its LLM and agent observability product, now ships under the "CoreWeave Forge" brand rather than wandb.ai. The pitch is still the same: if your ML team already lives in Weave or W&B for training history, agent traces and evaluations land in the same registry instead of a separate tool with a separate login. It traces at the operation level and includes LLM-as-a-judge metrics and PII redaction on every tier, including free. It also accepts OpenTelemetry traces through an OTLP endpoint that interprets OpenTelemetry GenAI and OpenInference attributes, so an OTel-instrumented app can send traces without the Weave SDK.

Pricing: Free $0/month (up to 5 seats, 5GB storage/month, 1GB Weave data ingestion/month). Pro from $60/month billed monthly (up to 10 seats, 100GB storage/month, 1.5GB ingestion/month, for companies under 50 employees; overage $0.03/GB storage, $0.10/MB ingestion). Enterprise custom (ingestion overage $0.20/MB). An Academic tier is free. Source: coreweave.com/forge-pricing (the legacy wandb.ai/site/pricing URL redirects there).

Best for: ML teams already running experiments in Weights and Biases who want agent and LLM tracing in the same platform. Not ideal for: teams that want trace-count pricing; ingestion is billed by data volume, so payload size drives the bill.

7. Helicone: The Fastest Proxy-Based Setup

Helicone's original design is a proxy: point API calls at Helicone instead of directly at OpenAI or Anthropic, and logging happens automatically with essentially no code change. That's still its fastest path to value for teams that mostly want cost, latency, and request-level visibility without touching instrumentation. Multi-step visibility comes through Sessions, which groups related calls (LLM calls, vector queries, tool calls) into one view of a full run. An async OpenLLMetry-based integration exists for teams that want OTel-style logging without routing traffic through the proxy, though that's a partial path, not full-stack OTel support.

The roadmap is the open question. Mintlify acquired Helicone in March 2026 and moved it into maintenance mode, with security patches, bug fixes, and new model support continuing and migration help offered to customers. Helicone's own site now says it's no longer part of Mintlify, with details still to come. It still works as a fast proxy, but don't plan a multi-year build around features it doesn't have today.

Pricing: Hobby free (10,000 requests/month, 1GB storage, 1 seat, 7-day retention). Pro $79/month (unlimited seats, usage-based overage on requests and storage, 1-month retention). Team $799/month (5 organizations, 3-month retention, SOC 2 and HIPAA compliance). Enterprise custom, with SAML SSO and on-prem deployment. Source: helicone.ai/pricing; the page doesn't publish a flat per-request overage rate.

Best for: Teams that want request-level cost and latency visibility live in minutes, with evaluation handled by a separate tool. Not ideal for: anyone trying to budget a specific monthly number past the free tier without first getting a quote, or anyone who needs an actively developed roadmap.

8. HoneyHive: OpenTelemetry-Native With No Self-Serve Middle Tier

HoneyHive's technical foundation is genuinely strong: it's natively built on OpenTelemetry, which the company describes as making it "fully agnostic across models, frameworks, and clouds," with native support for LangChain, CrewAI, and several agent frameworks out of the box. Both evaluation types ship even on the free tier: code-based checks and LLM-as-judge scoring for automated evaluation, plus human annotation queues and offline experiments.

The catch is the pricing ladder itself: HoneyHive's public pricing page lists only a Free developer tier (10,000 events/month, up to 5 users, 30-day retention) and a custom-quoted Enterprise tier. There's no visible self-serve paid plan in between, so a team that outgrows the free allotment goes straight into a sales conversation rather than an upgrade button.

Pricing: Free/Developer $0/month (10,000 events/month, up to 5 users, 30-day retention, multi-tenant SaaS only). Enterprise custom (unlimited users, customizable retention, SAML/SSO, multi-tenant, single-tenant, hybrid, or self-hosted deployment). Source: honeyhive.ai/pricing.

Best for: Teams that want auto-instrumented OpenTelemetry tracing out of the box and are comfortable moving straight to a sales conversation once they outgrow the free tier. Not ideal for: a self-serve-only buying process; there isn't one past 10,000 events a month.

9. Portkey (Prisma AIRS AI Gateway): Observability Bundled Into an AI Gateway

Portkey's core product is an AI gateway (routing, fallbacks, caching, and retries across model providers), and observability rides along with that gateway traffic rather than existing as a separate purchase. Palo Alto Networks closed its acquisition of Portkey on May 29, 2026, and Portkey now serves as the core AI gateway for Palo Alto's Prisma AIRS security platform, marketed as the Prisma AIRS AI Gateway. Its self-serve plans are still listed on Portkey's pricing page. The docs describe the suite as OpenTelemetry-compliant with a dedicated OTel export feature, and the open-source gateway is MIT-licensed and free to self-host for teams that want the gateway without a vendor relationship at all. Guardrails exist for policy enforcement, but there's no dedicated evaluation or scoring product the way Braintrust or Galileo offer one.

Pricing: Open Source (self-hosted) free. Developer free forever (10,000 recorded logs/month, 3-day log retention, 30-day metrics retention). Production $49/month (100,000 recorded logs/month, 30-day log/90-day metrics retention; overage $9 per additional 100,000 requests, up to 3M). Enterprise custom (10M+ requests, VPC hosting, SSO, SOC2/GDPR/HIPAA). Source: portkey.ai/pricing.

Best for: Teams that want an AI gateway and its observability in a single tool and a single bill. Not ideal for: teams that need deep, dedicated evaluation tooling; pair Portkey with a specialist if systematic scoring is a hard requirement.

10. Traceloop (OpenLLMetry): The OpenTelemetry Instrumentation Layer Itself

Traceloop built OpenLLMetry, an Apache 2.0-licensed, fully open-source instrumentation library that extends OpenTelemetry specifically for LLM and agent applications. Where most tools in this list added OTel support to an existing product, OpenLLMetry's semantic conventions for LLM calls were built alongside, and helped shape, the broader OTel GenAI semantic-convention effort the rest of the category is converging toward. The SDK itself works with any OTel-compatible backend, including several competitors on this list, which makes it the purest "instrumentation, not destination" option here.

Traceloop announced in March 2026 that it's joining ServiceNow, where its technology becomes part of ServiceNow's AI Control Tower. The company says OpenLLMetry will remain open source, that it intends to keep selling Traceloop and honor all customer obligations, and that it will announce roadmap changes in advance. The SDK's portability is the hedge either way.

Pricing: OpenLLMetry SDK free and open source (Apache 2.0), usable with any OTel-compatible backend. Traceloop Free Forever $0/month (up to 50,000 spans/month, up to 5 seats, 24-hour retention). Enterprise custom (unlimited seats, custom retention, SOC 2, on-prem deployment). Source: traceloop.com/pricing.

Best for: Teams that want the OpenTelemetry instrumentation itself to be portable, viewing traces in Traceloop or pointing them anywhere else that speaks OTel. Not ideal for: a team that wants a self-serve paid tier between free and Enterprise; that gap doesn't exist here.

11. Comet Opik: Open Source Plus the Cheapest Real Paid Tier

Comet, an established ML experiment-tracking platform and a direct competitor to Weights and Biases, built Opik as its answer to LLM observability and open-sourced it under Apache 2.0. That gives Opik the same "free forever if you self-host" option as Langfuse, plus a hosted Free Cloud tier and a genuinely cheap Pro Cloud tier at $19/month, the lowest paid entry price in this comparison by a wide margin, and (per the cost table above) still the cheapest at a real 1M-span month.

OpenTelemetry support is native across every tier, including languages beyond Python and JavaScript, and LLM-as-judge evaluation, custom metrics, and test suites ship at every paid level, not gated to Enterprise.

Pricing: Open Source free, self-hosted, Apache 2.0. Free Cloud $0/month (up to 10 team members, 25,000 spans/month, 60-day retention). Pro Cloud $19/month (up to 50 members, 100,000 spans/month, 60-day retention; overage $5 per 100,000 spans). Enterprise custom, flexible deployment including on-premises. Source: comet.com/site/pricing; license confirmed on comet-ml/opik's GitHub LICENSE file.

Best for: Budget-conscious teams that still want real OpenTelemetry support and LLM-as-judge evaluation rather than a stripped-down free tier. Not ideal for: teams that weight brand recognition or ecosystem size heavily in a vendor decision.

12. Galileo (Splunk Agent Observability): Purpose-Built Judge Models, Now Sold by Splunk

Galileo built an observability, evaluation, and guardrail platform, and its evaluation approach is the clearest differentiator: rather than relying only on a general-purpose LLM prompted to act as a judge, Galileo built its own Luna family of smaller, purpose-tuned evaluator models aimed at faster, cheaper, more consistent scoring than a frontier-model judge call every time. Runtime guardrails sit on top, aimed at catching a bad output before it reaches a customer rather than logging it afterward.

The way you buy it changed in 2026. Cisco completed its acquisition of Galileo in May, and Splunk now sells the product as Splunk Agent Observability, priced per span volume on an annual contract instead of the old Free and Pro trace tiers. OpenTelemetry portability is no longer a question mark: Galileo's docs describe an OTLP traces endpoint plus OpenInference instrumentors for LangGraph, LangChain, and OpenAI.

Pricing: A free edition is offered. Paid pricing starts at $100/month per 1.2 million spans a year, billed annually, including 500 million Luna tokens and 5 Luna fine-tuning runs per year. Extra Luna tokens and extended data retention are available by quote. Deploys in Splunk Observability Cloud or self-hosted. Source: splunk.com/en_us/products/pricing.html.

Best for: Enterprise teams, especially those already on Splunk or Cisco, that want purpose-built judge models and runtime guardrails rather than a general LLM prompted to grade its own category. Not ideal for: a team that wants self-serve monthly billing or a published rate at high span volume.

Stage and Sizing Fit

Buying Mistakes to Avoid

• Buying tracing and calling it done. A clean dashboard with no failed run in months usually means nobody's grading correctness, not that nothing's wrong. Budget for evaluation from day one, not as a phase-two project.
• Pricing off the free tier. The cost table above shows why: Datadog's attractive free tier gets expensive fast at real volume, Arize AX publishes no overage rate at all, and Opik and Langfuse stay cheap because their overage math is public and linear.
• Assuming "OpenTelemetry support" means full portability. Check whether evaluation and datasets travel with the traces or stay locked to the vendor; ingestion-only OTel support is common and easy to miss.
• Trusting a vendor's "open source" claim without opening the LICENSE file. Arize Phoenix is the clearest example here: it's source-available under the Elastic License 2.0, not Apache or MIT, and that distinction matters for anyone building on top of it commercially.
• Not re-checking who owns the vendor. Langfuse (ClickHouse), Helicone (Mintlify), Traceloop (ServiceNow), Galileo (Cisco), Portkey (Palo Alto Networks), and Arize (Dynatrace) all changed owners or announced a deal in 2026. Some kept standalone availability, per their own statements, while Galileo moved to Splunk pricing and Helicone went into maintenance mode, so a procurement team should ask the question at purchase and again at renewal.
• Treating LLM-as-judge as ground truth. It scales further than human review alone, but a model grading another model's work still needs a human-reviewed rubric and periodic spot checks, especially before a release that actually matters.

How to Choose: Decision Framework

What to Do Next

Pick one LLM application you already have running, ideally the one with the highest traffic or the most customer exposure, and instrument it with your shortlisted tool's free tier for two weeks before signing anything. Confirm you can actually see the full prompt and completion for a failed run, not just a success/failure flag, and run at least one offline evaluation against a small set of real cases before you trust the dashboard. Then price it at your real monthly volume using the cost table above, not the free-tier number, since that's where several of these vendors' actual cost picture changes completely. If that application is a RAG pipeline, the RAG tools roundup covers the retrieval stack those traces will be recording.

If you're specifically instrumenting a multi-step agent rather than a single-call or RAG-style app, read our dedicated agent observability roundup next; the buying questions around tool-call tracing and evaluating multi-step behavior are more specific than what a general LLM observability tool is built to answer. And if cost is the open question once you've picked a tool, AI agent cost optimization covers the token-spend side of the same budget this guide's pricing tables are built to inform.