Datadog LLM Observability: Examples, Demo and Pricing

As large language models (LLMs) become more and more integrated into our daily lives, the importance of observability in these systems is growing. LLM Observability ensures that the models are working as expected and that they are safe and reliable in real-world scenarios.

Datadog is a popular monitoring tool for infrastructure and applications. It is known for its comprehensive observability solutions.

In this article, we'll explore Datadog's new LLM observability features and how it can be used to observe and monitor LLMs.

Datadog's LLM Observability Features

Datadog offers a range of features for infrastructure and applications observability, including:

• Automatically collect and aggregate logs, metrics, and traces from your infrastructure and applications
• Monitor critical user journeys
• Detect and alert on performance issues for users in various regions
• Visualize load time and frontend performance for every user session

Recently, Datadog has added support for observability of LLMs.

With LLM Observability you can monitor, troubleshoot and evaluate your LLM powered applications.

You can also investigate the root cause of errors and performance issues and evaluate the quality of your LLM outputs.

Trace LLM Application with Datadog

Here is a quick demonstration of Datadog's LLM Observability using a simple Python SDK which they provide.

If your application is written in another language, you can use their API to send traces to Datadog.

Prerequisites

• Datadog API key
• OpenAI API key
• Python SDK

Datadog API keys

In order to obtain the Datadog API key, you need to create an account on their website and obtain the key from the settings page. Follow along with us to register an account and obtain the API key.

Register an account on Datadog LLM Observability by clicking on the "Get Started" button. Once you fill in the required information and create your account, you will be redirected to the dashboard.

In the dashboard, You can navigate to the section for LLM Observability by clicking on the "LLM Observability" button given in sidebar.

Now for obtaining the Datadog API key, you need to navigate to the "API Keys" section which can be found by clicking on your profile.

Now click on "API Keys" and you be able to manage your existing API keys or create new ones.

By default, you will see some pre-existing API keys. Click on the key entry to open up the key details and then you can copy it and start using it.

Code example

Install the SDK by adding the ddtrace and openai packages:

pip install ddtrace openai

Create a Python script and save it as quickstart.py. This Python script makes a single OpenAI call.

import os
from openai import OpenAI

oai_client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

completion = oai_client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "system", "content": "You are a helpful customer assistant for a furniture store."},
        {"role": "user", "content": "I'd like to buy a chair for my living room."},
    ],
)

Run the Python script with the following shell command. This sends a trace of the OpenAI call to Datadog.

export OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>

DD_LLMOBS_ENABLED=1 DD_LLMOBS_ML_APP=onboarding-quickstart \
DD_API_KEY=<YOUR_DATADOG_API_KEY> DD_SITE=datadoghq.com \
DD_LLMOBS_AGENTLESS_ENABLED=1 ddtrace-run python quickstart.py

After executing the above command, you can examine the LLM call trace in the Traces section of Datadog's LLM Observability dashboard.

The displayed trace consists of a single LLM span, automatically captured by the ddtrace-run command from Datadog's supported integrations.

Traces can illustrate various aspects of LLM operations:

1. Single LLM inferences, encompassing token usage, error details, and response times
2. Predefined LLM workflows, which group related LLM calls and associated operations like tool invocations or data preprocessing
3. Dynamic workflows executed by LLM agents

Each trace is composed of spans that represent individual decisions or steps within a workflow. These traces may also include additional information such as inputs, outputs, latency metrics, potential privacy concerns, and error logs.

You can find all the details about the trace in the "Trace Explorer" by simply clicking on the trace.

For applications with more sophisticated prompting techniques or intricate LLM-based workflows, you can implement tracing by referring to the Setup and SDK documentation provided by Datadog.

To explore further into the workings of LLM Observability with Datadog's Python SDK, you can checkout their comprehensive guide.

Datadog's ability to seamlessly integrate LLM traces with Application Performance Monitoring (APM) and Real User Monitoring (RUM) data helps organizations to quickly identify and resolve issues, facilitating the efficient scaling of AI-driven applications in production environments.

Datadog's pricing

Datadog offers a tiered pricing model that can be challenging to understand. The platform divides its pricing for several different categories based on the services they offer.

Each service has a plan based on personal use, small business, or enterprise. The pricing model is either on per-host or per-device or per-GB. This is flexible but can be confusing.

Since the pricing of Datadog is directly based on the number of hosts, It becomes a problem for infrastructures that are on microservices architecture.

In microservices architecture, services are often distributed across multiple hosts for scalability and fault tolerance. This can lead to higher costs, especially if the infrastructure is large or the retention period is long.

Some users have reported that Datadog costs more to monitor AWS.t3.medium instance than the actual instance.

This pricing favors users who uses larger and higher resource instances which often don't make sense for a proper microservice system.

No matter if you are using tiny host or inactive host. They all costs the same under datadog.

Not just it is hard for smaller companies to afford it, even big companies like Coinbase have been hit with unexpected high Datadog bills at massive scale.

Talking specifically about LLM observability, the pricing should be based on the number of traces and the amount of data stored.

Since they haven't offered a clear breakdown of how the pricing works, we can roughly estimate it between $20k to $100k per year given their pricing model for other services.

Lunary: An LLM-native alternative

While Datadog offers good observability features, there's a better alternative specifically designed for LLM-based applications: Lunary.

As a GenAI-native solution, Lunary provides a more tailored approach to monitoring and observing large language models applications in production.

Lunary stands out as a superior choice for companies building with LLMs due to several key factors:

1. Predictable Pricing: For larger companies, Lunary offers a simple, transparent pricing model with a fixed annual subscription fee. This makes budgeting easier and more predictable.

2. LLM-native design: Unlike Datadog, which has adapted its general-purpose observability tools for LLMs, Lunary is built from the ground up with LLMs in mind. This results in more intuitive and relevant metrics and visualizations for LLM applications.

3. Full suite of GenAI features: Lunary goes beyond basic observability, with:

   • Prompt Management: Manage and version your LLM prompts in a centralized way
   • Real-time Evaluations: Enrich LLM outputs with data about their quality
   • LLM Guardrails: Protect your chatbot and filter content
   • Custom dashboards: Create custom dashboards for managers and product owners

4. Open-source and Self-hostable: Lunary's core is open-source, allowing for greater transparency, community contributions, and customization. It's designed to be self-hosted, giving you full control over your data and infrastructure.

5. Integrations: Lunary provides integrations with popular LLM frameworks and tools such as OpenAI, LangChain, Anthropic, LiteLLM and more.

By choosing Lunary, teams working with LLMs can benefit from a tool that speaks their language, addresses their specific needs, and provides the flexibility and control that modern AI-driven applications demand.

For companies leveraging LLMs, the choice of an observability tool is an important step towards moving their applications to a production-ready state.

Reach out to us to learn more about Lunary and how it can compare with Datadog's LLM observability.