What Is LangSmith: Complete Observability Platform for LLMs

TL;DR: LangSmith is the official observability, testing, and evaluation platform from LangChain for debugging and monitoring LLM applications in production. It provides distributed tracing for every LLM call, tool invocation, and agent decision, plus dataset management, prompt versioning, and automated evaluations to catch regressions before deployment. LangSmith works with any LLM application, not just LangChain, making it the de facto standard for AI agent observability in 2026.

Key Takeaways

• LangSmith provides end-to-end tracing for LLM applications, capturing every model call, tool execution, prompt template, and token count in a unified timeline with sub-millisecond accuracy.
• The platform includes evaluation primitives for testing AI outputs against datasets with LLM-as-judge, semantic similarity, exact match, and custom evaluators that run automatically on every commit.
• Datasets in LangSmith serve as ground truth for testing — capturing real production traces as test cases enables regression detection and fine-tuning data collection without manual labeling.
• Prompt Hub centralizes prompt versioning with Git-style commit history, A/B testing infrastructure, and the ability to update production prompts without code deployment.
• LangSmith is model-agnostic and framework-agnostic, supporting OpenAI, Anthropic, Google, AWS Bedrock, local models, and raw API calls with zero LangChain dependency.
• Pricing follows a freemium model: free tier for individual developers with 5,000 traces per month, paid tiers starting at $39/month for teams with volume discounts at enterprise scale.

What is LangSmith and why does it exist?

LangSmith is an observability and evaluation platform built by LangChain Inc. specifically for the unique debugging challenges of LLM-powered applications. It launched in closed beta in mid-2023 and reached general availability in early 2024, becoming the most widely adopted observability tool for AI agents by 2026.

The platform exists because traditional application monitoring tools break down when applied to LLM systems. When your agent fails to complete a task, you need to answer: Which step failed? What was the exact prompt sent to the model? What did the model return? Which tool was called with what arguments? What was the conversation context at that moment? How much did this run cost in tokens?

Traditional APM tools like Datadog or New Relic can show you latency and error rates, but they cannot show you the semantic trace — the chain of LLM calls, tool executions, and context transformations that led to the failure. LangSmith captures that semantic layer as first-class data, making it possible to debug non-deterministic AI systems that change behavior between identical requests.

How does LangSmith work?

LangSmith operates through three core mechanisms: instrumentation, storage, and analysis. Understanding each is essential to getting value from the platform.

Instrumentation and Tracing

You instrument your application by adding the LangSmith SDK and setting environment variables. For LangChain applications, tracing is automatic — every chain, agent, and tool call is captured without code changes. For non-LangChain applications, you use the @traceable decorator or RunTree API to mark functions for tracing.

Every traced operation creates a run in LangSmith. A run captures inputs, outputs, start/end timestamps, token counts, model parameters, error traces, and parent-child relationships. Complex agent workflows create nested runs forming a tree structure that shows exactly how execution flowed through your application.

Storage and Retrieval

Traces persist in LangSmith's managed storage for the duration specified by your plan (30 days free tier, 180+ days paid). The platform indexes traces by project, run ID, tags, metadata, and content, enabling searches like "show me all runs where the tool search_wikipedia was called" or "find traces that cost more than 100K tokens."

Projects organize traces by environment or application. A common pattern is separate projects for development, staging, and production, allowing you to debug locally while monitoring production separately. You can filter, search, and compare traces across projects.

Datasets store input-output pairs for evaluation. You create datasets manually or by promoting production traces to test cases. This "production-to-test" workflow is LangSmith's killer feature for regression detection — if a user reports a failure, you save that trace as a dataset example, fix the issue, and ensure the fix works by running evaluations against the saved case.

Analysis and Evaluation

The Evaluations feature runs your application against a dataset, compares outputs to expected results, and scores quality using evaluators. LangSmith provides built-in evaluators for common tasks:

• Exact match: Output must match expected string exactly
• Semantic similarity: Embeddings-based comparison (configurable threshold)
• LLM-as-judge: Use an LLM to score output quality on custom criteria
• Custom evaluators: Python functions that return scores based on your logic

Evaluations produce a scored comparison showing which examples passed, which failed, and why. You can run evaluations manually during development, automatically in CI/CD pipelines, or on a schedule to detect production regressions.

What are the key features of LangSmith?

LangSmith's feature set spans observability, testing, and operational workflows. The features that matter for production systems are detailed below.

Distributed Tracing

Every LLM call, tool invocation, retrieval query, and prompt template render becomes a traced run. Runs are organized hierarchically: an agent execution is the root run, with child runs for each LLM call, which may have their own child runs for tool executions. The trace view shows this as an interactive tree with timing information, token counts, and full input/output payloads.

Key metrics captured per run:

• Latency: Start and end timestamps with microsecond precision
• Token usage: Input and output tokens per LLM call
• Cost: Calculated based on model pricing (configurable)
• Status: Success, error, or cancelled
• Metadata: Tags, session IDs, user IDs, or custom key-value pairs

The trace view includes a timeline visualization showing parallelism — when multiple tools execute concurrently, the timeline makes it obvious. This is critical for debugging performance bottlenecks in multi-agent systems.

Prompt Hub

Prompt Hub is a centralized repository for prompt templates with versioning, access control, and deployment management. Instead of hardcoding prompts in application code, you store them in LangSmith and reference them by name and version.

Prompt Hub supports:

• Versioning: Git-style commit history with semantic versioning
• Branching: Test prompt changes in separate branches before merging
• A/B testing: Deploy multiple prompt versions simultaneously and compare performance
• Access control: Public, private, or organization-scoped prompts
• Change tracking: See exactly what changed between prompt versions

The operational value is updating production prompts without code deployment. When you discover a better prompt through testing, you push it to Prompt Hub, tag it as the new latest, and your production agents pick it up on the next cold start without a redeploy.

Datasets and Examples

Datasets are collections of input-output pairs used for evaluation, fine-tuning data collection, and regression testing. You create datasets by:

1. Manual entry: Define examples in the UI or via SDK
2. Promotion from traces: Click "Add to dataset" on any production trace
3. Bulk upload: Import CSV or JSONL files
4. Programmatic creation: Use the SDK to generate synthetic test cases

The production-to-dataset workflow is transformative. When a user reports "the agent failed to book my flight," you search traces by session ID, find the failed run, click "Add to dataset," label the expected behavior, and now you have a regression test. Fix the issue, run evaluations, and confirm the fix works on the exact failure case without manual test case authoring.

Datasets support:

• Splits: Train/test splits for fine-tuning workflows
• Versioning: Datasets evolve as you add examples
• Metadata: Tag examples by difficulty, category, or source
• Export: Download datasets for fine-tuning or external analysis

Evaluations and CI/CD Integration

Evaluations run your application against a dataset and score outputs. LangSmith provides a built-in evaluator library covering common patterns:

• Correctness: LLM-as-judge scoring based on criteria (requires OpenAI or Anthropic API key)
• Relevance: Whether output addresses the input question
• Helpfulness: Subjective quality scoring
• Hallucination detection: Whether output contains unsupported claims
• Exact match / regex: Deterministic string comparison
• Semantic similarity: Embedding-based comparison with configurable threshold

Custom evaluators are Python functions with signature (run: Run, example: Example) -> dict that return scores and feedback. This enables domain-specific evaluation logic — checking whether generated SQL queries are syntactically valid, whether API calls use the correct authentication, or whether customer support responses match brand voice guidelines.

Evaluations integrate with CI/CD through the LangSmith SDK and GitHub Actions. A typical pipeline:

1. Developer changes agent prompt or code
2. CI runs evaluations against staging dataset
3. If pass rate exceeds threshold (e.g., 95%), allow merge
4. If not, block PR until regressions are fixed

Monitoring and Alerting

The Monitoring dashboard aggregates trace data into operational metrics:

• Request volume: Traces per hour/day
• Error rate: Percentage of runs ending in error
• Latency percentiles: p50, p90, p99 response times
• Token usage: Input/output tokens over time
• Cost: Spend by model, project, or user

You configure alerts on thresholds: "notify if error rate exceeds 5% for 10 minutes" or "alert if p99 latency exceeds 30 seconds." Alerts route to Slack, email, PagerDuty, or webhooks.

For production agents, monitoring answers three critical questions:

1. Is the system up? Error rate and request volume trends
2. Is it fast enough? Latency distribution and outliers
3. Is it expensive? Token usage and cost tracking

How does LangSmith compare to alternatives?

The LLM observability space in 2026 includes several competitors. LangSmith, Langfuse, Phoenix (Arize AI), Braintrust, and Helicone are the most widely deployed. Each has different strengths.

LangSmith advantages:

• Deepest LangChain integration with zero-config tracing
• Most mature Prompt Hub with versioning and A/B testing
• Largest user base and community
• Official support from LangChain Inc.

LangSmith disadvantages:

• No self-hosting option (SaaS-only)
• Pricing scales quickly for high-volume applications
• Requires LangChain API key (privacy-sensitive teams may prefer on-premise)

When to choose LangSmith:

• You are already using LangChain and want zero-friction tracing
• Prompt management and versioning are critical workflow components
• You need enterprise support from the LangChain team
• You prefer managed SaaS over self-hosting

When to choose alternatives:

• Langfuse: Need self-hosting for data sovereignty or cost control
• Phoenix: Already using Arize for ML monitoring and want unified observability
• Braintrust: Prefer fast iteration on evaluations with flexible scoring

What are the common use cases for LangSmith?

LangSmith serves five primary workflows in production LLM applications.

Debugging Failed Agent Runs

An agent fails to complete a user's request. You search LangSmith traces by session ID or user ID, open the failed run, and see:

• The exact prompt sent to the model at each step
• What the model returned (including tool calls)
• Which tools were invoked with which arguments
• Tool outputs and error messages
• Total token usage and cost

The trace shows that the agent called a search tool, received 0 results, and gave up instead of reformulating the query. You fix the agent's prompt to include retry logic, run an evaluation against the saved trace, and confirm the fix works.

This workflow is impossible with traditional logs because the semantic context — what the model was asked to do versus what it actually did — is lost in unstructured text. LangSmith's structured traces make debugging deterministic.

Regression Testing with Evaluations

Before deploying a new prompt or model version, you run evaluations against a curated dataset of known-good examples and known-failure edge cases. If the new version scores below the current production baseline, you block the deployment.

Example workflow:

1. Production agent has 200 saved test cases from past traces
2. Developer tests a new prompt claiming better accuracy
3. CI runs the new prompt against all 200 cases
4. Results show 198/200 pass (99%) versus 195/200 for current prompt (97.5%)
5. Deployment proceeds because improvement is validated

This prevents regressions that would be invisible in manual testing. A "better" prompt may improve one behavior while breaking three edge cases you forgot to test manually.

Cost Optimization

LangSmith's cost tracking shows which parts of your application consume the most tokens. A common finding: an agent's retrieval step returns 50KB of context, but only 2KB is relevant, wasting 96% of input tokens.

Using the trace view, you see:

• Retrieval step: 12,500 input tokens
• Agent reasoning: 500 output tokens
• Total cost: $0.15 per request

You add a re-ranking step that filters context to 3KB. Evaluations confirm quality stays the same. Cost drops to $0.04 per request — a 73% reduction. Over 100K monthly requests, this saves $11K/month.

Prompt Engineering and Iteration

Prompt Hub enables rapid iteration without code changes. You test five prompt variations in separate branches, deploy them to staging with A/B testing enabled, and measure which version achieves the highest user satisfaction score (collected via feedback buttons in your app).

LangSmith aggregates scores by prompt version. Winning prompt gets promoted to latest and deployed to production by updating a single tag — no code deployment, no API changes, no downtime.

Building Fine-Tuning Datasets

Fine-tuning requires high-quality input-output pairs. LangSmith's dataset export captures production traces with human feedback as fine-tuning data:

1. Users mark agent responses as helpful/unhelpful
2. Helpful responses are added to a dataset
3. You export the dataset in JSONL format
4. Fine-tune a model on this data using OpenAI, Anthropic, or AWS Bedrock

This closes the loop: production traces become training data, improving the base model, which improves future traces. Without LangSmith, collecting and labeling fine-tuning data requires separate tooling.

How do you get started with LangSmith?

Getting LangSmith running takes under 5 minutes for LangChain applications, longer for non-LangChain applications.

Step 1: Create Account and Get API Key

1. Go to smith.langchain.com
2. Sign up with email or GitHub
3. Create an API key in Settings → API Keys
4. Store the key securely (treat it like a production secret)

Step 2: Instrument Your Application

For LangChain applications, set environment variables and tracing is automatic:

For non-LangChain applications, install the SDK and add @traceable:

Step 3: View Traces in Dashboard

Run your application. Open LangSmith dashboard and navigate to your project. You will see:

• List of all runs with timestamps and status
• Click any run to see the full trace tree
• Inspect inputs, outputs, token counts, and latency
• Add tags or comments for organization

Step 4: Create a Dataset from Production Traces

Find a trace you want to use as a test case:

1. Click the trace in the dashboard
2. Click "Add to dataset" button
3. Choose or create a dataset
4. Label the expected output (if different from actual output)
5. Repeat for more examples

Step 5: Run Your First Evaluation

This runs your agent against the dataset and scores outputs. Failed examples show you where the agent regressed.

What are best practices for using LangSmith in production?

After deploying dozens of production agents with LangSmith tracing, these patterns consistently deliver value.

Organize by Environment

Create separate projects for development, staging, and production. This isolates traces by environment, preventing development noise from obscuring production issues. Use environment variables to control which project receives traces:

Tag Traces with Metadata

Add custom metadata to traces for filtering and analysis. Common tags:

• User ID: Track traces by user for debugging user-specific issues
• Session ID: Group traces by conversation session
• Feature flags: Track which features were enabled during execution
• Model version: When A/B testing models, tag which version was used
• Cost center: For chargeback to departments

Capture User Feedback

Integrate feedback buttons in your application's UI. When users mark a response as helpful or unhelpful, send that signal to LangSmith:

Feedback appears in the trace view and enables filtering: "show me all runs with negative feedback" surfaces the exact failures to fix.

Build Evaluation Datasets Continuously

Do not wait until you have a problem to create test cases. Promote high-quality production traces to datasets as they occur:

• Positive examples (great outputs) ensure you do not regress quality
• Edge cases (unusual inputs) ensure robustness
• Failure cases (after fixing) ensure regressions do not reoccur

Aim for 50-100 examples covering common and uncommon scenarios. Run evaluations on every deploy.

Set Up Cost Alerts

Configure alerts on token usage and cost:

• Alert if daily cost exceeds budget
• Alert if a single run costs more than 100K tokens (potential loop or retrieval explosion)
• Alert if average cost per request increases by more than 20% week-over-week

These alerts catch cost regressions before they appear on your AWS or Anthropic bill.

Monitor Latency Distribution, Not Averages

Average latency hides outliers. A p99 latency of 60 seconds means 1% of users wait a full minute, even if average latency is 3 seconds. Monitor latency percentiles and investigate high-percentile outliers:

• p50: Typical user experience
• p90: Slower-than-usual but tolerable
• p99: Edge cases, may indicate bugs or infrastructure issues

Prune Traces Strategically

The free tier provides 5,000 traces per month. For high-volume development, you will exceed this. Use sampling:

Always trace 100% in production. Cost optimization happens in production; sampling hides the data you need.

What are the common mistakes with LangSmith?

After reviewing hundreds of LangSmith implementations, these are the failure modes to avoid.

Treating LangSmith as Optional

Teams add LangSmith "when we have time" after launching. This is backwards. Without observability from day one, you cannot debug production failures effectively. By the time you add tracing, the critical failure happened last week and the trace is gone.

Integrate LangSmith during development, not after production issues force you to.

Not Creating Datasets from Failures

When a user reports a failure, developers fix the issue but do not promote the failed trace to a dataset. The fix works, but three months later a refactor reintroduces the same bug. Without the dataset example, there is no regression test, and the bug returns.

Always add failure cases to datasets after fixing them.

Ignoring Feedback Signals

LangSmith captures user feedback, but teams do not act on it. A trace with negative feedback sits in the dashboard unreviewed. The signal is worthless unless you investigate why users marked it unhelpful and fix the underlying issue.

Create a weekly ritual: review all negative feedback traces, identify patterns, and fix the root causes.

Over-Relying on LLM-as-Judge Evaluators

LLM-as-judge evaluators are powerful but imperfect. They hallucinate, have biases, and cost money (they make additional LLM calls). Do not use LLM-as-judge for easily deterministic checks:

• Use exact match for structured outputs with fixed format
• Use regex for outputs requiring specific patterns
• Use custom evaluators for domain-specific validation

Reserve LLM-as-judge for subjective quality evaluation where programmatic checks are insufficient.

Not Setting Sampling in High-Volume Applications

Production agents handling 1 million requests per day generate 1 million traces per day. At $0.01 per 1,000 traces (typical paid tier pricing), that is $10K/month just for tracing. Most applications do not need 100% trace capture.

Sample intelligently:

• Trace 100% of errors (always debug failures)
• Trace 100% of requests with user feedback
• Trace 1-10% of successful requests for statistical monitoring

This cuts costs by 90% without losing critical debugging data.

Hardcoding Prompts After Using Prompt Hub

Teams adopt Prompt Hub, centralize prompts, then hardcode a critical prompt in application code "just this once" because it is easier. Now prompts are split between Prompt Hub and codebase, and nobody remembers which version is canonical.

If you use Prompt Hub, commit fully. All prompts live there, or none do. Mixed approaches create confusion.

FAQ

What is LangSmith used for?

LangSmith is used for debugging, testing, monitoring, and evaluating LLM applications in production. Its primary use cases include tracing agent execution to diagnose failures, running evaluations against test datasets to catch regressions before deployment, managing prompts with versioning and A/B testing, collecting fine-tuning datasets from production traces with user feedback, and monitoring cost, latency, and error rates across production deployments. LangSmith works with any LLM application, not just LangChain, making it useful for teams using OpenAI, Anthropic, Google, AWS Bedrock, or custom models. The platform provides visibility into the semantic layer of AI systems — what the model was asked to do, what it returned, and how that led to success or failure.

Is LangSmith free?

LangSmith offers a free tier for individual developers with 5,000 traces per month, unlimited projects, and 30-day trace retention. This is sufficient for prototyping and small personal projects. Paid plans start at $39/month for the Developer plan (50K traces, 90-day retention) and scale to Team ($199/month, 500K traces) and Enterprise (custom pricing, unlimited traces, 180+ day retention, SSO, dedicated support). Educational and open-source projects may qualify for extended free tier access. The free tier includes all core features — tracing, datasets, evaluations, and Prompt Hub — so you can fully evaluate the platform before paying. For production applications, most teams exceed the free tier's 5,000 traces within days and upgrade to paid plans.

How does LangSmith integrate with LangChain?

LangSmith provides zero-configuration tracing for LangChain applications. Set the LANGCHAIN_TRACING_V2 and LANGCHAIN_API_KEY environment variables, and every LangChain chain, agent, tool call, and retrieval operation is automatically traced without code changes. LangSmith captures the full execution tree, including all intermediate steps, token counts, latency, and errors. Prompts from LangSmith Prompt Hub integrate natively with LangChain via hub.pull(), enabling centralized prompt management. LangChain also provides built-in evaluation helpers that integrate with LangSmith datasets, making it easy to run regression tests. While LangSmith was built by the LangChain team for LangChain users, it is not LangChain-exclusive — the @traceable decorator and RunTree API work with any Python code calling any LLM provider.

What is the difference between LangSmith and LangChain?

LangChain is an open-source framework for building LLM applications with composable abstractions like chains, agents, tools, and memory. LangSmith is a paid SaaS platform for observing, testing, and monitoring those applications in production. LangChain helps you build the application; LangSmith helps you debug and improve it. They are complementary tools from the same company. You can use LangChain without LangSmith (but lose observability), and you can use LangSmith without LangChain (by manually instrumenting your non-LangChain code with the @traceable decorator). Most production teams using LangChain also use LangSmith because the tracing integration is automatic and the debugging value is immediate. LangChain is free; LangSmith has a freemium pricing model.

How does LangSmith compare to Langfuse?

LangSmith and Langfuse are the two most popular LLM observability platforms in 2026. LangSmith has deeper LangChain integration, a more mature Prompt Hub with versioning and A/B testing, and official support from LangChain Inc. Langfuse offers both a managed SaaS option and an open-source self-hosted option, making it attractive for cost-sensitive teams or those with data sovereignty requirements. Langfuse is also model-agnostic and framework-agnostic with excellent non-LangChain support. Pricing is similar for SaaS tiers. Choose LangSmith if you are heavily invested in LangChain and want zero-friction tracing. Choose Langfuse if you need self-hosting, want open-source transparency, or prefer a vendor-neutral observability layer.

Can I self-host LangSmith?

No, LangSmith is a SaaS-only platform with no self-hosting option as of 2026. All traces, datasets, and prompts are stored on LangChain Inc.'s infrastructure. For teams requiring on-premise deployment due to data sovereignty, compliance, or security policies, alternatives include Langfuse (open-source with self-hosting), Phoenix by Arize AI (open-source), or building custom observability using OpenTelemetry and Jaeger. LangChain Inc. offers enterprise contracts with extended data retention, dedicated support, and security certifications (SOC 2, GDPR compliance), but self-hosting is not available. If self-hosting is a hard requirement, LangSmith is not an option.

How do I trace non-LangChain applications with LangSmith?

For non-LangChain applications, install the LangSmith SDK (pip install langsmith) and use the @traceable decorator on any function you want to trace. Mark your main agent function, individual LLM calls, tool executions, or retrieval steps with @traceable. Each decorated function creates a run in LangSmith capturing inputs, outputs, latency, and errors. You can nest traceable functions to create hierarchical traces. For applications where decorators are not feasible (e.g., instrumenting third-party libraries), use the RunTree API for manual tracing. This approach works with any LLM provider — OpenAI, Anthropic, Google, AWS Bedrock, local models, or custom APIs. LangSmith is framework-agnostic despite being built by the LangChain team.

What are LangSmith evaluators?

Evaluators are functions that score LLM outputs against expected results during evaluation runs. LangSmith provides built-in evaluators for common tasks: exact match (output must match expected string exactly), semantic similarity (embeddings-based comparison with configurable threshold), LLM-as-judge (use an LLM to score quality based on criteria), relevance (whether output addresses the input), and hallucination detection (whether output contains unsupported claims). You can also define custom evaluators as Python functions with signature (run: Run, example: Example) -> dict that return scores and reasoning. Evaluators run when you call client.run_on_dataset(), comparing your agent's outputs against dataset examples and producing pass/fail results plus aggregate metrics. Custom evaluators enable domain-specific validation like checking SQL syntax, verifying API call correctness, or ensuring brand voice compliance.

How much does LangSmith cost at scale?

LangSmith pricing scales with trace volume. The free tier provides 5,000 traces/month. Paid plans: Developer ($39/month, 50K traces), Team ($199/month, 500K traces), and Enterprise (custom pricing, unlimited traces). For a production agent handling 1 million requests per month, assuming 10% trace sampling (100K traces), you would be on the Team plan at $199/month. At 100% trace capture, you would exceed the Team plan and need Enterprise pricing, which typically starts around $1,000-2,000/month for 1M traces with volume discounts for higher volumes. Additional costs include evaluation runs (each run consumes traces) and LLM-as-judge evaluators (which make LLM API calls). For high-volume production applications, evaluate whether the observability value justifies the cost, or implement intelligent sampling (trace 100% of errors, 1-10% of successes) to reduce costs by 90% without losing critical debugging data.

Sources

• LangSmith Official Documentation
• LangSmith Python SDK on GitHub
• LangSmith Pricing and Plans
• LangSmith Tracing and Observability Guide
• LangSmith Evaluation and Testing Documentation
• LangSmith Prompt Hub Documentation
• LangChain Hub on GitHub
• LangSmith API Reference