Repository navigation
Stop paying for work already done - #12
Merged
Merged
Conversation
The per-call context block sat at character 32 of a 26,389-character system prompt, so every minute rollover changed the prefix and invalidated the provider's prompt cache for the whole prompt. Moved to the end. cached_tokens across minute rollovers: [0, 8448, 0] -> [4864, 8448, 4864].
Trace CA591f56d4f5055bd64cb598b2de7a58cd: lookup_booking fired 8 times in one 204s call. Each firing cost 1-3 sequential Appwrite round trips at ~250ms apiece — a caller-phone miss falls through to a name query and then a 100-document scan — so tool turns landed at 1.1-1.6s against an 800ms TTFA budget while the model itself answered in 341ms. Measured on the dyno: a single reservation query is 1220ms cold and ~270ms warm, and pooling the HTTP client saves only ~20ms of that. The cost is the number of round trips, not the connection. - Per-call memo in front of lookup_motel_reservation. It caches the task, not the result, so concurrent callers join one in-flight request, and it drops failed entries so a transient error cannot poison the call. - Any tool outside the read-only set invalidates the memo. A tool added later defaults to safe, not fast. - Prefetch the caller's own booking when the call context is built, so it overlaps the first model round instead of landing inside the turn.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The per-call context block sat at character 32 of a 26,389-character system
prompt, so every minute rollover changed the prefix and invalidated the
provider's prompt cache for the whole prompt. Moved to the end.
cached_tokens across minute rollovers: [0, 8448, 0] -> [4864, 8448, 4864].
Second commit —
perf(voice): stop re-asking Appwrite for the same booking mid-call:Trace CA591f56d4f5055bd64cb598b2de7a58cd: lookup_booking fired 8 times in
one 204s call. Each firing cost 1-3 sequential Appwrite round trips at
~250ms apiece — a caller-phone miss falls through to a name query and then
a 100-document scan — so tool turns landed at 1.1-1.6s against an 800ms
TTFA budget while the model itself answered in 341ms.
Measured on the dyno: a single reservation query is 1220ms cold and ~270ms
warm, and pooling the HTTP client saves only ~20ms of that. The cost is the
number of round trips, not the connection.
not the result, so concurrent callers join one in-flight request, and it
drops failed entries so a transient error cannot poison the call.
later defaults to safe, not fast.
overlaps the first model round instead of landing inside the turn.