📊 feat: Surface Message Feedback as Langfuse Scores by graphaelli · Pull Request #13544 · danny-avila/LibreChat

graphaelli · 2026-06-06T02:58:54Z

What

When Langfuse tracing is enabled, message feedback (👍/👎 plus the reason
tag/comment) is now surfaced as a Langfuse score on the assistant message's
generation trace. Previously feedback was stored only on the message in MongoDB
and never reached Langfuse, so there was no way to turn real user signal into
response-quality analysis or evals in the tool that already captures every
generation.

How

The feedback endpoint posts a boolean user-feedback score (1/0 + tag/comment)
to Langfuse /api/public/scores for that message's trace; clearing feedback
deletes the score. Fire-and-forget and env-gated (LANGFUSE_*), so the
feedback UX never blocks on (or requires) Langfuse.
Linking is lookup-free: the run opts into deterministic Langfuse trace ids
(langfuse.deterministicTraceId, passed to the agents Run), so the trace id
is sha256(messageId)[:32]. The feedback route recomputes the same id and
scores by it.

Changes

api/server/services/Langfuse.js — POST/DELETE /api/public/scores (env-gated)
api/server/utils/langfuseTrace.js — traceIdForMessage(messageId)
api/server/routes/messages.js — fire the feedback score after the Mongo write
packages/api/src/agents/run.ts — pass langfuse: { deterministicTraceId: true }
bump @librechat/agents to ^3.2.21

Dependency

Requires @librechat/agents ≥ 3.2.21, which adds the opt-in
LangfuseConfig.deterministicTraceId (danny-avila/agents#220, merged & released).

Verification

Running end-to-end locally (LibreChat + @librechat/agents@3.2.21 + Langfuse on
ClickHouse 25.8): a 👍/👎 in the chat UI lands as a user-feedback score on the
message's named, deterministically-identified trace.

Closes #13537

When Langfuse tracing is enabled, the message feedback endpoint now posts a boolean `user-feedback` score (1/0 + tag/comment) to Langfuse for the assistant message's trace; clearing feedback deletes the score. Fire-and- forget, so the feedback UX never blocks on Langfuse. Linking is lookup-free: the run opts into deterministic Langfuse trace ids (`langfuse.deterministicTraceId`, passed to the agents Run), so the trace id is sha256(messageId)[:32]. The feedback route recomputes the same id and scores by it. - api/server/services/Langfuse.js: POST/DELETE /api/public/scores (env-gated) - api/server/utils/langfuseTrace.js: traceIdForMessage(messageId) - api/server/routes/messages.js: fire feedback score after the Mongo write - packages/api: pass langfuse.deterministicTraceId to the run - bump @librechat/agents to ^3.2.21 (adds LangfuseConfig.deterministicTraceId) Closes danny-avila#13537 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

danny-avila · 2026-06-06T13:09:19Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 663166e9d3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@librechat/agents passes no environment to its Langfuse tracer, so @langfuse/otel falls back to LANGFUSE_TRACING_ENVIRONMENT and otherwise to Langfuse's "default". The score helper instead fell back to NODE_ENV, so a deployment with only NODE_ENV=production filed scores under "production" while the trace stayed on "default" — the score never landed on the trace. Use LANGFUSE_TRACING_ENVIRONMENT only, and omit `environment` when unset so Langfuse defaults both score and trace to "default". Addresses Codex review on danny-avila#13544. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The agent tracer emits traces with just the public/secret keys (defaulting to Langfuse Cloud, or via the legacy LANGFUSE_BASEURL alias), but the score helper disabled itself unless LANGFUSE_BASE_URL was set — so an otherwise-traced deployment silently posted no scores. Resolve the base URL the same way the tracer does (LANGFUSE_BASE_URL -> LANGFUSE_BASEURL -> Cloud) and gate enablement on the credentials only. Addresses Codex review on danny-avila#13544. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The feedback route is shared by all message types, but deterministic Langfuse trace IDs are only enabled for agent runs. Rating a message from a non-agent endpoint (with Langfuse configured) posted a user-feedback score for sha256(messageId) that no trace will ever match, leaving orphan scores. Gate scoring on isAgentsEndpoint(message.endpoint); `updateMessage` now returns `endpoint` so the route can check it. Addresses Codex review on danny-avila#13544. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

danny-avila · 2026-06-06T14:26:35Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6fa13c7ab3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…oint The previous gate used isAgentsEndpoint, which only matches the literal `agents` endpoint. But provider endpoints (anthropic, openai, custom, …) run through the agents runtime as ephemeral agents and DO emit deterministic AgentRun traces, so isAgentsEndpoint('anthropic') === false suppressed scoring for the common case. Only the OpenAI/Azure Assistants endpoints use a separate runtime with no agent trace, so gate on !isAssistantsEndpoint instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…back-scores # Conflicts: # api/package.json # package-lock.json # packages/api/package.json

danny-avila · 2026-06-07T14:46:11Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d2e9d2f676

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

danny-avila · 2026-06-07T15:08:40Z

@codex review

danny-avila · 2026-06-07T15:12:14Z

@codex review

chatgpt-codex-connector · 2026-06-07T15:16:06Z

Codex Review: Didn't find any major issues. 👍

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

* feat: surface message feedback (thumbs up/down) as Langfuse scores When Langfuse tracing is enabled, the message feedback endpoint now posts a boolean `user-feedback` score (1/0 + tag/comment) to Langfuse for the assistant message's trace; clearing feedback deletes the score. Fire-and- forget, so the feedback UX never blocks on Langfuse. Linking is lookup-free: the run opts into deterministic Langfuse trace ids (`langfuse.deterministicTraceId`, passed to the agents Run), so the trace id is sha256(messageId)[:32]. The feedback route recomputes the same id and scores by it. - api/server/services/Langfuse.js: POST/DELETE /api/public/scores (env-gated) - api/server/utils/langfuseTrace.js: traceIdForMessage(messageId) - api/server/routes/messages.js: fire feedback score after the Mongo write - packages/api: pass langfuse.deterministicTraceId to the run - bump @librechat/agents to ^3.2.21 (adds LangfuseConfig.deterministicTraceId) Closes danny-avila#13537 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: match Langfuse trace environment for feedback scores @librechat/agents passes no environment to its Langfuse tracer, so @langfuse/otel falls back to LANGFUSE_TRACING_ENVIRONMENT and otherwise to Langfuse's "default". The score helper instead fell back to NODE_ENV, so a deployment with only NODE_ENV=production filed scores under "production" while the trace stayed on "default" — the score never landed on the trace. Use LANGFUSE_TRACING_ENVIRONMENT only, and omit `environment` when unset so Langfuse defaults both score and trace to "default". Addresses Codex review on danny-avila#13544. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: don't require LANGFUSE_BASE_URL to post feedback scores The agent tracer emits traces with just the public/secret keys (defaulting to Langfuse Cloud, or via the legacy LANGFUSE_BASEURL alias), but the score helper disabled itself unless LANGFUSE_BASE_URL was set — so an otherwise-traced deployment silently posted no scores. Resolve the base URL the same way the tracer does (LANGFUSE_BASE_URL -> LANGFUSE_BASEURL -> Cloud) and gate enablement on the credentials only. Addresses Codex review on danny-avila#13544. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: only post feedback scores for agent-endpoint messages The feedback route is shared by all message types, but deterministic Langfuse trace IDs are only enabled for agent runs. Rating a message from a non-agent endpoint (with Langfuse configured) posted a user-feedback score for sha256(messageId) that no trace will ever match, leaving orphan scores. Gate scoring on isAgentsEndpoint(message.endpoint); `updateMessage` now returns `endpoint` so the route can check it. Addresses Codex review on danny-avila#13544. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: gate feedback scoring by !isAssistantsEndpoint, not isAgentsEndpoint The previous gate used isAgentsEndpoint, which only matches the literal `agents` endpoint. But provider endpoints (anthropic, openai, custom, …) run through the agents runtime as ephemeral agents and DO emit deterministic AgentRun traces, so isAgentsEndpoint('anthropic') === false suppressed scoring for the common case. Only the OpenAI/Azure Assistants endpoints use a separate runtime with no agent trace, so gate on !isAssistantsEndpoint instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: sort message method imports * fix: honor Langfuse tracing gates for feedback scores * refactor: move Langfuse feedback logic to api package * fix: support Langfuse host for feedback scores * test: type Langfuse feedback fetch mock * chore: compact Langfuse feedback comment --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Danny Avila <danny@librechat.ai>

danny-avila changed the base branch from main to dev June 6, 2026 13:09

chatgpt-codex-connector Bot reviewed Jun 6, 2026

View reviewed changes

Comment thread api/server/services/Langfuse.js Outdated

Comment thread api/server/services/Langfuse.js Outdated

Comment thread api/server/routes/messages.js Outdated

Comment thread api/package.json Outdated

graphaelli and others added 3 commits June 6, 2026 10:09

chatgpt-codex-connector Bot reviewed Jun 6, 2026

View reviewed changes

Comment thread api/server/routes/messages.js Outdated

Comment thread api/server/services/Langfuse.js Outdated

Comment thread api/server/services/Langfuse.js Outdated

graphaelli and others added 2 commits June 6, 2026 12:43

Merge remote-tracking branch 'clickhouse/dev' into feat/langfuse-feed…

d2e9d2f

…back-scores # Conflicts: # api/package.json # package-lock.json # packages/api/package.json

style: sort message method imports

8971eca

chatgpt-codex-connector Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread api/server/services/Langfuse.js Outdated

Comment thread api/server/services/Langfuse.js Outdated

Comment thread api/server/services/Langfuse.js Outdated

danny-avila added 3 commits June 7, 2026 10:57

fix: honor Langfuse tracing gates for feedback scores

b147c56

refactor: move Langfuse feedback logic to api package

fef7106

fix: support Langfuse host for feedback scores

76e1f0d

test: type Langfuse feedback fetch mock

b5772b1

chore: compact Langfuse feedback comment

e91e52f

danny-avila changed the title ~~feat: surface message feedback (thumbs up/down) as Langfuse scores~~ 📊 feat: Surface Message Feedback as Langfuse Scores Jun 7, 2026

danny-avila merged commit 90ebecb into danny-avila:dev Jun 7, 2026
19 checks passed

graphaelli deleted the feat/langfuse-feedback-scores branch June 8, 2026 13:06

graphaelli mentioned this pull request Jun 10, 2026

Add Playwright E2E tests for stack functionality (daily, dummy-provider, secret-free) ClickHouse/agentic-data-stack#12

Open

Uh oh!

Conversation

graphaelli commented Jun 6, 2026

What

How

Changes

Dependency

Verification

Uh oh!

danny-avila commented Jun 6, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danny-avila commented Jun 6, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danny-avila commented Jun 7, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danny-avila commented Jun 7, 2026

Uh oh!

danny-avila commented Jun 7, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants