Overview

Prompt Management provides a centralized registry for the prompts used by your LLM applications. Instead of hardcoding prompt templates in application code or configuration files, create, version, and update prompts through Agent Observability, then retrieve them at runtime.

Runtime retrieval is supported in Python through the ddtrace SDK. Prompt retrieval and Prompt Tracking are separate: LLMObs.get_prompt() can retrieve a managed prompt without enabling Agent Observability, but Agent Observability must be enabled to create LLM spans and associate prompt metadata with them.

Prompt Management works alongside Prompt Tracking. When Agent Observability is enabled, managed prompts passed directly to supported, automatically instrumented LLM calls are associated with the resulting spans.

Prerequisites

• Python 3.9 or later.
• Your Datadog site and a Datadog API key. The API key is required for prompt retrieval even if traces are sent through the Datadog Agent.
• A Datadog application key with the llm_observability_read, feature_flag_config_read, and feature_flag_environment_config_read permissions to resolve prompts by environment. If you select an existing application key in Datadog, ensure that it has these permissions.
• To manage prompts through the API or Python SDK, the application key also requires the llm_observability_write and feature_flag_config_write permissions.

Install the SDK

Install or upgrade the latest ddtrace package in the Python environment used by your application:

pip install --upgrade ddtrace

Use a managed prompt in Python

Integrate Prompt Management with a coding agent

Integrate a managed prompt with a coding agent of your choice by pasting in the following prompt:

Follow the instructions at https://docs.datadoghq.com/llm_observability/instrumentation/agentic.md to integrate the Datadog managed prompt <PROMPT_ID> into this application for environment <DEPLOYMENT_ENVIRONMENT> and track its use in Agent Observability.

Prompt variables: <PROMPT_VARIABLES>

When configuring the environment, use the following values:

DD_SITE=
DD_ENV=<DEPLOYMENT_ENVIRONMENT>

Optionally, append selected Datadog credentials so the coding agent can configure and verify the integration in the same session:

Selected Datadog credentials:

DD_API_KEY=<DATADOG_API_KEY>
DD_APP_KEY=<DATADOG_APP_KEY>

Treat these values as secrets and handle them according to the linked guide. Do not repeat or expose them.

Note: Including the API and application keys in the prompt is optional and is not required for the coding agent to integrate Prompt Management. Include them only in a trusted coding-agent session.

After the integration is complete, run your application and trigger the modified LLM flow. Return to the prompt page to view usage; new prompt calls may take a minute to appear.

Configure prompt retrieval

Provide the Datadog site, credentials, and deployment environment through the configuration and secret-management workflow already used by your application. For example, use the application’s environment file, Docker Compose or Kubernetes configuration, deployment platform, or secret manager. At runtime, the following environment variables must be set before importing ddtrace:

export DD_SITE="<DATADOG_SITE>"
export DD_API_KEY="<DATADOG_API_KEY>"
export DD_APP_KEY="<DATADOG_APP_KEY>"
export DD_ENV="<DEPLOYMENT_ENVIRONMENT>"

DD_ENV selects the environment used to resolve the prompt version and must match an environment where the prompt is deployed.

Retrieve, format, and use a prompt

Preserve the prompt already used by your application as the fallback. The fallback keeps the application working if registry, environment-resolution, network, or server failures occur.

The following example retrieves and formats a chat prompt, then passes the formatted messages directly to OpenAI:

from ddtrace.llmobs import LLMObs
from openai import OpenAI

default_messages = [
    {"role": "system", "content": "You are a support agent for {{company}}."},
    {"role": "user", "content": "{{question}}"},
]

variables = {
    "company": "Acme Inc.",
    "question": "How do I reset my password?",
}

prompt = LLMObs.get_prompt(
    "customer-support-greeting",
    fallback=default_messages,
)
messages = prompt.format(**variables)

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
)

prompt.format() returns a string for a text prompt and a list of messages for a chat prompt. Pass the formatted value to the corresponding text or messages parameter of your LLM provider call.

If retrieval fails and no fallback is provided, get_prompt() raises a ValueError. A fallback does not replace authentication: DD_API_KEY is always required, and DD_APP_KEY is also required when DD_ENV is set.

Managed prompts cannot reference other managed prompts in their templates. To compose prompts, combine them in application code or manage the final provider-facing prompt as a single prompt.

Select a version

Without DD_ENV, get_prompt() retrieves the latest prompt version:

prompt = LLMObs.get_prompt("customer-support-greeting")

With DD_ENV, get_prompt() resolves the prompt version for that environment. This requires DD_APP_KEY with the read permissions listed in Prerequisites.

To retrieve an exact numeric version independently of DD_ENV, pass version:

prompt = LLMObs.get_prompt("customer-support-greeting", version=2)

The version argument takes precedence over environment resolution.

Track prompt usage

To associate a managed prompt with an LLM span, enable Agent Observability and run the application with automatic instrumentation through its existing execution workflow.

If the application receives its configuration before the Python process starts, use ddtrace-run. For example, the equivalent shell command is:

DD_SITE="<DATADOG_SITE>" \
DD_API_KEY="<DATADOG_API_KEY>" \
DD_APP_KEY="<DATADOG_APP_KEY>" \
DD_ENV="<DEPLOYMENT_ENVIRONMENT>" \
DD_SERVICE="<SERVICE_NAME>" \
DD_LLMOBS_ENABLED=1 \
ddtrace-run python app.py

If the application loads its configuration in Python, load the configuration first, then import ddtrace.auto before importing the LLM provider or other application modules:

from dotenv import load_dotenv

load_dotenv()

import ddtrace.auto

from ddtrace.llmobs import LLMObs
from openai import OpenAI

Run this setup with the application’s normal Python command, such as python app.py. Do not also use ddtrace-run; it initializes ddtrace before the application can load its configuration.

If the application does not send data through a Datadog Agent, also set DD_LLMOBS_AGENTLESS_ENABLED=1.

For a supported automatically instrumented provider, pass the value returned by prompt.format() directly to the provider call, as shown in Retrieve, format, and use a prompt. This automatically associates the managed prompt with the resulting span.

Copying, rebuilding, or converting the formatted value can discard its prompt-tracking metadata. For example, concatenating a managed system prompt with a user question creates a new string without that metadata. Use LLMObs.annotation_context() to associate the managed prompt with the resulting LLM span:

prompt = LLMObs.get_prompt(
    "customer-support-system-prompt",
    fallback="You are a helpful support agent writing for a {{audience}} audience.",
)
variables = {"audience": audience}
system_prompt = prompt.format(**variables)
combined_prompt = f"{system_prompt}\n\nUser question: {question}"

with LLMObs.annotation_context(
    prompt=prompt.to_annotation_dict(**variables),
):
    response = client.responses.create(
        model="gpt-4o",
        input=combined_prompt,
    )

Pass the same variables to to_annotation_dict() that you pass to format() so that the tracked prompt includes the values used for that call.

annotation_context() associates metadata with an LLM span created inside the context; it does not create the span. For providers that are not automatically instrumented, first manually instrument the LLM call to create an LLM span. An explicit annotation_context() takes precedence over automatic prompt tracking. See Prompt Tracking for more information.

Create and manage prompts

Create prompts and publish new versions in the Prompts UI, through the Python SDK, or through the API.

Create a prompt

Promote a tracked prompt

To promote a prompt already tracked in Agent Observability to a managed prompt, navigate to the Prompts page, open the prompt, and click Register. You can then update the prompt in the UI and retrieve it at runtime.

In the UI from scratch

Navigate to the Prompts page and click + New Prompt.

In the Prompt Editor:

1. Add one or more messages and assign each a role: System, User, or Assistant.
2. Use {{variable_name}} syntax in any message to add dynamic content.
3. Optional: Click Run to test the prompt with sample values.
4. Click Save Prompt to open the save dialog.

Structure the prompt so the user query and context are injected as variables:

In the save dialog:

Click Create Prompt to save the prompt to the registry.

Update, list, and delete prompts

In the UI

Open a prompt in the Prompts page to:

• Create a new version: Click Edit and update the messages in the Prompt Editor.
• Deploy a version to another environment: Select a version and update its Deployment environments.
• Delete a prompt: Select Delete from the prompt’s options menu. This removes the prompt and its version history from the registry.

Use the Python SDK

Use LLMObs.create_prompt() to create a prompt and deploy its first version to one or more environments. The env_ids values are Feature Flags environment IDs, which you can obtain from the List environments API:

from ddtrace.llmobs import LLMObs

chat_template = [
    {"role": "system", "content": "You are a support agent for {{company}}."},
    {"role": "user", "content": "{{question}}"},
]

created_prompt = LLMObs.create_prompt(
    "customer-support-greeting",
    chat_template,
    env_ids=["<FEATURE_FLAG_ENVIRONMENT_ID>"],
)

To publish and deploy another version, use LLMObs.create_prompt_version():

created_version = LLMObs.create_prompt_version(
    "customer-support-greeting",
    updated_chat_template,
    env_ids=["<FEATURE_FLAG_ENVIRONMENT_ID>"],
)

Treat prompt creation, versioning, and deployment as setup operations. Do not perform them during application startup or from a request path. At runtime, retrieve deployed prompts with LLMObs.get_prompt().

These methods require the API and application key permissions listed in Prerequisites.

Use LLMObs.list_prompts() and LLMObs.list_prompt_versions() to inspect managed prompts, LLMObs.update_prompt() and LLMObs.update_prompt_version() to update metadata or deployments, and LLMObs.delete_prompt() to delete a prompt and all of its versions.

Use the API

Use the Prompt Management API to create, retrieve, update, and delete prompts and prompt versions. See the LLM Observability API reference for endpoint schemas, request media types, and examples.

Further reading

Additional helpful documentation, links, and articles:

Prompt TrackingDOCUMENTATION PlaygroundDOCUMENTATION Agent Observability SDKDOCUMENTATION