LangSmith vs Langfuse vs Phoenix: LLM Observability

TL;DR: Shortlist LangSmith for managed tracing and evaluation, Langfuse when operating the trace backend yourself is a priority, and Phoenix for a local or hosted evaluation workflow. Test one representative trace and its evaluation in each candidate; OpenTelemetry transport does not guarantee identical fields or migration behavior.

Key Takeaways

Choose from required trace fields, evaluation workflow, data residency, and operating responsibility before comparing dashboards.
LangSmith documents self-hosting as an Enterprise add-on. Langfuse documents its application and storage stack for self-hosting.
Phoenix's repository uses Elastic License 2.0; do not assume that every free-to-run observability project has the same license terms.
Phoenix server startup and tracing registration are separate steps. `register()` configures instrumentation/export rather than starting a server.
Use the same fixture, expected result, evaluator rubric, and model configuration when comparing tools. No comparative benchmark was run for this article.

Why Does Your Agent Fail Silently in Production?

Imagine a support agent that retrieves an obsolete refund policy and returns a fluent but incorrect answer. Its HTTP response can still be successful. Request latency and error logs alone do not explain which policy it saw, which model call used it, or whether an evaluator caught the mistake.

A useful trace connects the request, retrieval, model generation, tool calls, and final output. Evaluation then asks whether the output satisfied the task. Capturing an entire prompt is not always necessary or acceptable: configure redaction and retention before enabling production instrumentation.

What Is LLM Observability, and How Does It Differ from Traditional APM?

LLM observability adds application-level evidence to infrastructure monitoring: model and prompt versions, retrieved context, tool inputs and outcomes, token usage, and quality scores. These fields help explain an answer rather than merely establish that a request completed.

The official LangSmith observability documentation, Langfuse observability documentation, and Phoenix documentation describe tracing and evaluation capabilities. This comparison was checked on September 11, 2026, and the head-to-head sections below were re-checked against the vendors' documentation and pricing pages on October 20, 2026. It does not establish a universal quality ranking or a procurement quote.

How Do LangSmith, Langfuse, and Phoenix Compare at a Glance?

Free software does not make storage or LLM-as-judge calls free. Read the selected release's license and feature terms separately from its hosting documentation.

What Is the Practical Difference Between LangSmith, Langfuse, and Phoenix?

For a team already using LangChain or LangGraph, inspect LangSmith's integrated tracing and evaluation workflow first. If self-hosting is a requirement, assess Langfuse's deployment stack and Phoenix's deployment options against the actual retention and access-control requirements. For a local debugging session, Phoenix provides a documented local server path.

The decision depends on what the team can operate. A backend that receives OTLP spans may still require vendor-specific attributes to display token usage, retrieved documents, and evaluation results correctly. Treat portability as something to demonstrate, not as an automatic consequence of OpenTelemetry support.

LangSmith vs Langfuse: how do they compare head-to-head?

LangSmith and Langfuse answer the same debugging question with different operating and commercial models. The table uses each vendor's own documentation and pricing page as read on October 20, 2026.

The structural difference is who runs the backend and how you pay for it. LangSmith is a managed service first: the Developer and Plus plans are cloud-only, every user costs a seat, and traces are billed per trace with a 14-day base retention that you can extend to 180 days for a fee. Langfuse ships the same code for its cloud and for self-hosters, so the self-hosting decision is about operating Postgres, ClickHouse, Redis/Valkey and object storage rather than about licensing, and the paid tiers add history, support and compliance reports rather than seats. A team of ten engineers therefore pays seat fees on LangSmith before the first trace and none on Langfuse, while a team sending very deep traces pays per trace on LangSmith and per observation on Langfuse; which comes out cheaper depends on your trace shape, not on the vendor.

On tracing and evaluation the two overlap more than the marketing suggests. For a LangGraph application, LangSmith's run tree arrives with almost no instrumentation work once LANGSMITH_TRACING is set; everything outside the framework needs @traceable or an OpenTelemetry exporter. Langfuse asks you to define observations at useful application boundaries but treats every framework the same way. Both attach scores to traces, run datasets through experiments and support LLM-as-a-judge evaluators on production traffic; Langfuse adds versioned prompt management as a first-class object. Run the support-policy fixture described later in this article through both before deciding, because the field mapping for token usage and retrieved documents is where migrations stall.

Langfuse vs Phoenix: how do they compare head-to-head?

Langfuse and Phoenix are the two candidates teams shortlist when they want to run the backend themselves, but their licenses, hosting paths and commercial models differ.

Phoenix is the lowest-friction way to see a trace: one command starts a local server, OpenInference instrumentors cover the common frameworks, and self-hosting carries no license fee or usage cap. The cost is operational and legal rather than financial. You own the database, retention and access control, and the Elastic License 2.0 is not the MIT license that Langfuse uses, which matters if you plan to redistribute Phoenix or run it as a service for others. Langfuse's self-hosting path is heavier, with ClickHouse and object storage in the stack, but it is the same deployment the vendor runs, and its cloud tiers give a team a managed exit when operating the stack stops being worth it.

Choose Phoenix when a single engineer or small team needs evaluation-driven debugging in notebooks and local environments, or when Arize AX is the eventual destination. Choose Langfuse when several teams share prompts, scores and datasets in production and want online evaluation and retention management without building them. If your question is pre-deployment testing versus production tracing rather than which tracing backend to run, read Langfuse vs Promptfoo, which covers the testing side of that split.

What Is LangSmith and Who Is It For?

LangSmith provides tracing, datasets, experiments, online evaluation, and human feedback workflows. It can instrument applications beyond LangChain. Its OpenTelemetry guide documents ingestion and supported attribute mappings; configure the endpoint and headers for the chosen hosted region or self-hosted instance.

LangSmith's self-hosting guide describes an Enterprise add-on and a license requirement. Confirm the deployment and commercial terms before designing around an on-premises instance. The public docs do not justify describing one product's UI as objectively the most polished.

What Is Langfuse and What Does Self-Hosting Require?

Langfuse provides observability, datasets, experiments, and evaluation workflows with a documented self-hosting path. Its architecture guide lists application services and storage dependencies including Postgres, ClickHouse, Redis/Valkey, and object storage.

Before choosing it, assign ownership for backups, retention, upgrades, queue failures, and access control. A local demonstration does not validate those production responsibilities. Self-hosting the trace backend also does not keep evaluator inputs inside the network if the evaluator calls an external model API.

Use the SDK documentation corresponding to the installed release. Current instrumentation examples support manual observations and nested generations; avoid copying decorators or experiment-runner signatures across SDK generations without checking them.

What Is Phoenix and How Do You Start Tracing?

Phoenix supports tracing, experiments, and evaluation. The repository identifies Elastic License 2.0; consult its terms if redistribution or a hosted service is part of your plan.

The Phoenix overview documents a local server command:

Tracing registration is separate. The Phoenix OTEL setup guide configures the collector endpoint and register(). A running collector and the relevant instrumentation packages are prerequisites. A successful register() call alone does not prove a trace was delivered or stored.

For a pilot, send a non-sensitive fixture trace and inspect it in the server. Pin the package versions used and verify the exporter flushes before a short-lived process exits. No server was started or live trace submitted for this article.

How Does Tracing Setup Differ Across the Three?

The original distinction between instrumentation and a backend remains useful. Instrumentation describes what happened in application code; the exporter sends it somewhere; the backend stores, displays, and evaluates it. A fault at any one of those boundaries can produce a misleadingly empty dashboard.

For LangSmith, inspect whether the framework integration already creates the desired parent and child runs before adding manual tracing. For Langfuse, define observations at useful application boundaries and nest model generations under the correct parent. For Phoenix, select an OpenInference integration for the framework or manually instrument the missing application steps.

Auto-instrumentation helps with supported library calls, but it cannot infer every business distinction. A successful HTTP call to a refund service and an accepted refund may be different outcomes. Add the application outcome to the trace rather than assuming a low-level span says enough.

Which fields make a trace useful for debugging?

Decide what may be collected before recording prompts or user data. Token counts and model identifiers alone can support cost analysis, but they do not explain a factual error; raw prompt capture may help debugging while introducing a separate retention and access responsibility.

Why are model cost and latency separate measurements?

A long task can spend time waiting on a tool, starting a browser, or retrying a request. Summing model generation durations does not necessarily equal end-to-end latency, particularly when work runs concurrently. Likewise, a backend's estimated model cost depends on correctly mapped usage fields and the rates it applies.

Keep provider-returned usage where available and identify estimates explicitly. For comparisons, include failures and retries, use the same workload and retention window, and separate trace-storage costs from model/evaluator costs. This is the operational reason to inspect a representative trace before comparing dashboard totals.

What does manual application tracing look like?

The LangSmith manual instrumentation guide still documents @traceable, LANGSMITH_TRACING, and LANGSMITH_API_KEY. Unexecuted instrumentation example: install a compatible langsmith SDK, provision a project/API key, and configure any required regional endpoint. The key below is a placeholder, not a credential:

This fixture sends a trace if you run it with valid credentials. It does not call an LLM or access customer data. The Python Client.flush() method is synchronous; call it before process exit:

Pin the SDK version whose documented flush API you use and inspect the resulting trace in the selected project. Flushing queued events is necessary for short-lived processes, but it is not proof that credentials, mapping, or retention are correct. Nested application functions can be traced to expose retrieval and generation separately.

Langfuse's SDK guide documents explicit observations. This fixture illustrates a span with a recorded output; it does not call a model or prove that export succeeded.

These are documentation adaptations, not runs performed for this article. For Phoenix, the server/collector setup above and the relevant OpenInference instrumentation fill the same collection role. Inspect the resulting trace in every case; installing an SDK is not the acceptance criterion.

How Can You Compare the Same Trace Across All Three?

Use this conceptual support-policy fixture to expose semantic gaps without inventing performance results. It is a test design, not an exported trace or a vendor-specific schema.

Run the same fixture through each candidate's supported instrumentation. Keep the input, policy revisions, model settings, and rubric fixed. Inspect these fields in the UI and export:

Use a deterministic check for known fixture facts and a separate human rubric for ambiguous answers. A judge-model score is a measurement with limitations, not ground truth. The Context Engineering guide explains why retrieved context and cache usage should be inspected alongside the answer.

What Should You Verify Before Migrating a Trace Backend?

Export a bounded set containing a successful task, a tool failure, and an evaluation result. Record the source schema and instrumentation versions.
Send the same fixture to the candidate backend using supported attribute mappings. Verify root/child IDs, errors, usage, document references, and timestamps.
Check which datasets, prompt versions, annotations, and scores require a separate migration. OTLP trace ingestion does not establish that these objects moved.
Test retention and deletion against a non-sensitive fixture. Verify who can read it in each deployment.
Retain the original dataset and rollback route until the candidate passes the acceptance checks. If dual-exporting, account for duplicate ingestion and additional data copies.

This migration checklist is an editorial recommendation derived from the documented integration boundaries. It is not a report that a migration was performed.

How Do You Run Evaluations on Each Platform?

LangSmith evaluation documents offline dataset evaluation and online evaluation of production traces. Langfuse evaluation likewise connects scores, datasets, experiments, and live traces. Phoenix LLM evaluations supports code-based and model-based evaluators.

Compare a concrete workflow: create a dataset item from a failure, attach an expected result or rubric, run the evaluator, and inspect disagreements. Confirm judge inputs, model configuration, and costs. Do not rank evaluation libraries by adjectives such as “strongest” without a defined evaluation task and results.

What production agent evaluation loop should you run?

Use three connected loops. An offline release gate runs a versioned dataset before deployment. The sampled online evaluation loop examines a bounded slice of real traces, while human adjudication resolves disputed or high-impact cases and adds accepted failures to the offline dataset.

Attach scores to the smallest useful unit and keep policy violations separate from an average quality score. Use Agent Memory to test stale or cross-user recall, and the vLLM guide to connect model quality with inference measurements.

How do offline experiments and online evaluation complement each other?

Offline evaluation asks whether a proposed change handles a fixed set of cases. It is useful before deployment because inputs, expected results, and rubric versions can be reviewed together. Include cases where the correct behavior is refusal, escalation, or a clear incomplete result, not just successful answers.

Online evaluation asks what happens on actual traffic. Sampling and route filters control volume, while reference-free checks can identify issues that do not require a known answer. Production traffic rarely arrives with a reliable expected output, so an online judge score should be triaged rather than assumed correct.

Human review connects the two. A reviewer can inspect the retrieved policy, identify that the wrong revision was used, and add that failure to the offline dataset. This creates a concrete regression case instead of an aggregate “quality fell” alert with no reproduction path.

A practical workflow could use one tool for local experiments and another for production traces, but that is optional. Before splitting systems, verify that identifiers, rubric versions, and dataset lineage remain understandable. A single backend with a complete review loop may be easier to operate than multiple disconnected evaluation dashboards.

Which Platform Fits Your Deployment and Budget?

Shortlist LangSmith if its managed workflow and hosting terms fit. Shortlist Langfuse if its self-hosted stack fits your operational capacity. Shortlist Phoenix if its tracing and evaluation workflow fits the intended local or deployed environment. Validate access control, storage, retention, and export on the selected deployment rather than assuming them from a product label.

For each candidate, separate trace ingestion/storage charges, evaluator model calls, underlying infrastructure, and operator time. Record volume, retention, sampling, and workload before comparing estimates. Pricing and packaging change; the list prices in the head-to-head sections were read on October 20, 2026, and this article provides no workload quote or claim that one tool is always cheaper.

Connect the decision to the framework selection guide and AgentCore comparison. For related SDK changes, browse the AI Frontier Weekly archive.

Frequently Asked Questions

What is the difference between LLM observability and traditional monitoring?

Infrastructure monitoring establishes system behavior such as latency and errors. LLM observability adds the model inputs, retrieved evidence, tool outcomes, and evaluation signals needed to assess an answer. Use both when operating an agent.

Does self-hosting keep all evaluation data private?

Not automatically. The backend can be self-hosted while an evaluator still sends inputs to an external model. Review exporter configuration, evaluator endpoints, redaction, retention, and access controls together.

Can I use these tools without LangChain?

Yes, each documents integrations beyond LangChain. Verify the supported instrumentation and field mapping for your framework and provider; receiving a span does not guarantee that every LLM-specific field is interpreted correctly.

Does Phoenix register() start the local server?

No. Start or select a Phoenix server, configure its collector endpoint, and then register tracing. Verify that a fixture trace reaches the server before relying on the setup.

Can OpenTelemetry move every dataset and evaluation score between products?

No such guarantee follows from trace transport support. Check vendor mappings and export/import paths for datasets, annotations, prompts, and scores separately. Use the migration checklist above on a small fixture before changing production collection.

LangSmith vs Langfuse: what is the main difference?

LangSmith is a managed, commercial platform with per-seat plus per-trace pricing and the deepest LangChain and LangGraph integration; self-hosting is an Enterprise add-on. Langfuse is MIT-licensed at its core, can be self-hosted on any tier, and bills a flat plan fee plus units with no seat fees. Both trace, evaluate and manage datasets; the choice turns on who operates the backend and how your team size and trace shape map onto the two pricing models.

Langfuse vs LangSmith: which is cheaper?

Neither is always cheaper. LangSmith's list price starts at $0 for one seat with 5k base traces a month and $39 per seat a month on Plus, then per-trace usage with a 14-day base retention. Langfuse's list price starts free at 50k units a month for two users and $29 a month on Core with 100k units included, then $8 per additional 100k units, where a unit is a trace, an observation or a score. Seat-heavy teams with shallow traces tend to favor Langfuse's model and small teams with deep traces may not; price your own volume on both pricing pages before deciding.

Langfuse vs Phoenix: which should you choose?

Choose Phoenix when you want a free, local-first tracing and evaluation server that you run yourself, your instrumentation already uses OpenInference, or Arize AX is your eventual managed platform. Choose Langfuse when a team needs shared prompt management, online evaluation and retention controls on a documented production stack, with a managed cloud available on the same code. Both ingest OpenTelemetry, so test a representative trace in each before committing.

Phoenix vs Langfuse: which is better for self-hosting?

Phoenix is lighter to start: Docker, Compose, Kubernetes, Helm and one-click cloud templates are documented, and self-hosting has no license fees, usage limits or feature gates. Langfuse's self-hosted stack is heavier, with Postgres, ClickHouse, Redis or Valkey and object storage, but it is the same deployment the vendor runs in production, and its enterprise controls such as data retention policies and audit logs are available with a license key. Check the Elastic License 2.0 if you plan to redistribute Phoenix or offer it to others as a service.

Phoenix vs LangSmith: when does each fit?

Phoenix fits local debugging, notebook-driven evaluation and self-hosted deployments where you accept the operating work and want no license cost; its commercial path is Arize AX. LangSmith fits teams already on LangChain or LangGraph that want managed tracing, datasets and online evaluation without running infrastructure, with self-hosting reserved for Enterprise. If you are not using LangChain, compare the instrumentation effort for your framework on both before weighing price.