DEV Community

Cover image for How Do We Extract the “Why” from a PR into a CHANGELOG with Jev? 🤔
nyaomaru
nyaomaru

Posted on AI-assisted

How Do We Extract the “Why” from a PR into a CHANGELOG with Jev? 🤔

#opensource #githubactions #ai #typescript

Balancing strict LLM constraints with raw PR text

Hoi hoi! 👋

I'm @nyaomaru, a frontend engineer currently building an AI-assisted CHANGELOG generator and experimenting with Jev. 😸

Recently, I have been thinking about one problem.

How can we extract this

## Why

The previous implementation could leave stale release data after an update.
Enter fullscreen mode Exit fullscreen mode

from a pull request and put it into a CHANGELOG?

### Fixed

- Fix cache invalidation

  Why: The previous implementation could leave stale release data after an update.
Enter fullscreen mode Exit fullscreen mode

Getting the WHAT is relatively easy.
Getting the WHY is much harder.
And I didn't want AI to simply invent a nice-looking reason.

Let's look at how changelog-bot currently approaches this! 👀

GitHub logo nyaomaru / changelog-bot

Automatic create changelog with AI. 🤖 It provides CLI and github actions. 🚀 https://www.npmjs.com/package/@nyaomaru/changelog-bot

changelog-bot type logo

changelog-bot logo

Releases should feel exciting, not tedious.

@nyaomaru/changelog-bot 🤖 turns your Git history and release notes into a polished changelog entry (and optional PR) in a single run. Drop it into CI, run it locally, or hand it to your release captain—either way, you ship with a crisp changelog and zero copy-paste fatigue.

Why changelog-bot?

  • Automated storytelling: Combines commit history, PR titles, and release notes to produce human-ready changelog sections.
  • LLM superpowers (optional): Connect OpenAI or Anthropic for tone-aware summaries or skip API keys entirely and rely on a robust heuristic fallback.
  • PR-ready output: Can open a pull request with updated changelog, compare links, and release notes already wired up.
  • Safe defaults: Detects duplicate versions, keeps compare links current, and won’t fail a release if AI is down.
  • CI-native: Works as a GitHub Action, reusable workflow, or plain CLI—no fragile scripting required.

Important

This project is currently in its early stages…

View on GitHub

📝 WHAT Is Easy. WHY Is Hard.

Suppose a PR contains.

## Description

Extract shared runtime configuration into ProviderBase.

Update OpenAI, Anthropic, and Gemini adapters to use the shared implementation.

This reduces duplicated setup logic and makes adding new providers easier.
Enter fullscreen mode Exit fullscreen mode

The first two sentences mainly describe WHAT and HOW.

But this one is different

This reduces duplicated setup logic and makes adding new providers easier.
Enter fullscreen mode Exit fullscreen mode

That is much closer to WHY.

We could simply give the whole PR to an LLM and ask

Why was this change made?
Enter fullscreen mode Exit fullscreen mode

But then the model may summarize, paraphrase, or infer a reason that was never explicitly written.

For a CHANGELOG, I wanted something stricter.

Maybe WHY extraction is not a generation problem.

Maybe it is a selection problem.

✂️ First, Find Possible WHY Candidates

Before calling a model, changelog-bot preprocesses the PR description.

It looks for signals such as

const STRONG_CANONICAL_SECTION_NAMES = new Set([
  "why",
  "reason",
  "because",
  "motivation",
  "context",
  "background",
  "problem",
  "rationale",
]);

const RATIONALE_MARKER_RE =
  /\b(why|because|so that|in order to|reason|rationale|motivation|to avoid|to prevent|context|problem)\b|,\s*so\b/i;
Enter fullscreen mode Exit fullscreen mode

It also removes template noise such as checkboxes, placeholders, and irrelevant fields.

Then it builds small candidate snippets

const candidates = buildWhyCandidateSnippets(
  extracted.sections,
  body,
  options.maxCharsPerPr,
  !extracted.hasTargetSection,
);
Enter fullscreen mode Exit fullscreen mode

So instead of sending a large PR body, we might get

[
  "Extract shared runtime configuration into ProviderBase.",
  "This reduces duplicated setup logic and makes adding new providers easier.",
];
Enter fullscreen mode Exit fullscreen mode

Now the problem is much smaller.

Which candidate is actually the WHY?

🧠 Asking Jev Two Questions

This is where I started experimenting with Jev.

I don't ask Jev to write a WHY sentence.

Instead, each candidate is evaluated with two questions.

First

question: "Does the target candidate explicitly state why this changelog change was made?";
Enter fullscreen mode Exit fullscreen mode

Basically

Is this actually a reason?

But that alone is not enough.

Imagine

We changed the cache invalidation logic.

The README was updated because the previous explanation was confusing.
Enter fullscreen mode Exit fullscreen mode

The second sentence definitely contains a reason.

But it explains the README change, not the cache invalidation change.

So we also ask

question: "Does the target candidate state a reason that applies to the identified changelog change?";
Enter fullscreen mode Exit fullscreen mode

Now each candidate has two signals

{
  probability: explicitWhyProbability,
  relevanceProbability: changeRelevanceProbability,
}
Enter fullscreen mode Exit fullscreen mode

So the real question becomes

Is this a reason?

AND

Is it the reason for this change?
Enter fullscreen mode Exit fullscreen mode

🎯 Selecting the Candidate

changelog-bot compares the candidate probabilities

const bestCandidate = candidateProbabilities.reduce((best, candidate) =>
  combinedWhyCandidateProbability(candidate) >
  combinedWhyCandidateProbability(best)
    ? candidate
    : best,
);
Enter fullscreen mode Exit fullscreen mode

Then both minimum thresholds must pass

const accepted =
  selectedCandidate !== undefined &&
  bestCandidate.probability >= JEV_MIN_EXPLICIT_WHY_PROBABILITY &&
  bestCandidate.relevanceProbability >= JEV_MIN_CHANGE_RELEVANCE_PROBABILITY;
Enter fullscreen mode Exit fullscreen mode

If the candidate does not pass

if (!accepted) continue;
Enter fullscreen mode Exit fullscreen mode

No WHY is added.

And I think that is important.

Sometimes the correct answer is

There is no reliable WHY in this PR.
Enter fullscreen mode Exit fullscreen mode

That is better than inventing one.

🔒 Keep the Original PR Text

This is probably my favourite part.

When a candidate is accepted

items.push({
  prNumber: item.prNumber,
  why: selectedCandidate,
  confidence: mappedConfidence,
});
Enter fullscreen mode Exit fullscreen mode

Notice

why: selectedCandidate;
Enter fullscreen mode Exit fullscreen mode

The final WHY is the original PR text.

Not newly generated prose.

So the pipeline becomes

PR description
      ↓
candidate extraction
      ↓
Jev evaluates candidates
      ↓
select candidate
      ↓
original source text
      ↓
CHANGELOG
Enter fullscreen mode Exit fullscreen mode

Jev decides.

The PR author still provides the sentence. 😸

🧱 Where Does WHY Fit in the v1 Pipeline?

While working on WHY extraction, I have also been redesigning the changelog-bot pipeline for v1.

The main idea is deterministic-first.

A complete CHANGELOG should exist even when AI is disabled or fails.

So AI should enrich the result, not own the result.

The target v1 pipeline looks roughly like this

source changes
      ↓
ReleaseChange[]
      ↓
deterministic classification
      ↓
complete ReleaseDraft
      ↓
optional editorial enrichment
      +
optional WHY enrichment
      ↓
ReleaseResult
      ↓
deterministic renderer
      ↓
CHANGELOG
Enter fullscreen mode Exit fullscreen mode

WHY extraction is therefore an independent optional enrichment stage.

Conceptually

ReleaseDraft / ReleaseChange[]
      ↓
find eligible WHY targets
      ↓
collect PR rationale candidates
      ↓
LLM or Jev
      ↓
trust / confidence validation
      ↓
attach structured WHY notes
      ↓
ReleaseResult
Enter fullscreen mode Exit fullscreen mode

One important change here is that the WHY engine does not need to control the final Markdown.

Today, some of the pipeline still works by finding targets from generated Markdown and inserting WHY text afterward.

For v1, I want to remove that boundary.

Instead

{
  prNumber,
  why: selectedCandidate,
  confidence,
}
Enter fullscreen mode Exit fullscreen mode

becomes structured release data first.

Then one deterministic renderer decides how it appears

- Fix cache invalidation
  - Why: The previous implementation could leave stale release data after an update.
Enter fullscreen mode Exit fullscreen mode

So the responsibility becomes clearer

WHY engine
→ find the evidence

Renderer
→ format the evidence
Enter fullscreen mode Exit fullscreen mode

This is also part of a larger rule I am using for the v1 migration

Deterministic code owns the CHANGELOG structure. AI only enriches it.

Phases 1–4 of this migration are already implemented, and the structured rendering and enrichment boundary is part of the next phase.

You can see the full v1 design here 👇

https://github.com/nyaomaru/changelog-bot/blob/main/v1-design.md

🧪 But What Threshold Should We Use?

Of course, once we have probabilities, another question appears.

0.50?
0.60?
0.80?
Enter fullscreen mode Exit fullscreen mode

I didn't want to pick one only because it felt reasonable.

So I created an evaluation harness with an initial 14-case corpus.

For example

{
  candidates: [
    "Separate changelog execution into input-resolution and output-finalization phases.",
    "This simplifies the main orchestration flow and runs independent GitHub lookups concurrently.",
  ],
  expectedSelectedCandidateIndex: 1,
}
Enter fullscreen mode Exit fullscreen mode

We also need negative examples

{
  candidates: [
    "Extract shared provider runtime configuration into ProviderBase.",
    "Refactor OpenAI, Anthropic, and Gemini adapters to inherit it.",
  ],
  expectedSelectedCandidateIndex: null,
}
Enter fullscreen mode Exit fullscreen mode

There is plenty of implementation information.

But no strong explicit WHY.

So null is the correct result.

🤝 Then John Joined the Project

While I was working on this experiment, @johnnylemonny joined the project as a contributor. 🎉

johnnylemonny (John) · GitHub

Full‑stack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. Open to opportunities. - johnnylemonny

favicon github.com

John proposed expanding the small benchmark into roughly 50 cases covering

Explicit Rationale
Implementation-Only
Template & Noise
Multilingual PRs
Enter fullscreen mode Exit fullscreen mode

I liked the idea, but I wanted one rule to stay clear

Ground truth should be independent of the current Jev threshold.

So the benchmark first defines whether a candidate is actually valid.

Then we evaluate different thresholds afterward.

John implemented the expanded corpus in PR #212. 🙌

The benchmark grew from 14 cases to 50, including tricky negative cases.

He also added finer threshold sweeps

[0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9];
Enter fullscreen mode Exit fullscreen mode

Now we can compare precision, recall, and F1 instead of saying

0.8 feels safe! 👍

Let's measure it instead. 😸

🧩 The Current Pipeline

Today, WHY extraction roughly looks like this

Pull Request
     ↓
Normalize PR body
     ↓
Extract WHY candidates
     ↓
Remove noise
     ↓
Local trust check
     ↓
Jev:
  explicit rationale?
  relevant to this change?
     ↓
Apply threshold
     ↓
Candidate or none
     ↓
Original PR text
     ↓
CHANGELOG
Enter fullscreen mode Exit fullscreen mode

There are quite a few steps for one small Why: line. 😂

But that is what made this problem interesting.

🎯 What I Learned

When I started this feature, I thought the main question was

How can AI write a good reason?
Enter fullscreen mode Exit fullscreen mode

Now I think the more interesting question is

How can we find evidence that the reason already exists?
Enter fullscreen mode Exit fullscreen mode

So my current approach is simple

Use deterministic code to find evidence.

Use Jev to evaluate the evidence.

Keep the original evidence as the output.
Enter fullscreen mode Exit fullscreen mode

And thanks to John's contributions, we now have a much better way to test whether that approach actually works.

Open source is fun when someone looks at your experiment and says:

What if we test this properly?

and then builds it with you. 🙌

changelog-bot is still evolving, and Jev support is still experimental.

I'm curious to see where this approach goes next!

https://github.com/nyaomaru/changelog-bot

Thanks for reading!

See you in the next article! 👋😸

Top comments (46)

Collapse
 
johnnylemonny profile image
𝗝𝗼𝗵𝗻 •

Thanks for the generous shoutout and writeup, @nyaomaru! 😸

Collaborating on this has been an absolute blast. When tackling the WHY problem, your core decision to treat it as an evidence-selection problem - keeping the author's original words instead of letting an LLM hallucinate or paraphrase reasons is what makes the output genuinely trustworthy.

Designing the 50-case evaluation harness together in PR #212 was a great experience. Your insistence that the ground truth corpus must stay independent of any specific threshold was spot on: it gave us a true yardstick to measure precision and recall instead of guessing.

Seeing this integrate cleanly into the deterministic-first v1 pipeline (structuring the enrichment before rendering instead of patching Markdown) really brings the whole architecture together.

Excited to keep building and refining changelog-bot with you! 🚀

Collapse
 
nyaomaru profile image
nyaomaru •

Thank you so much, John! 😸
Working on this together has been a lot of fun for me too!!

Your work on the 50-case evaluation harness and threshold sweeps made this experiment much stronger. It gave us a real way to measure the approach instead of just guessing what “feels safe.”

I’m really happy with how the deterministic-first v1 direction is coming together, and I’m excited to keep building changelog-bot with you! 🚀

Collapse
 
antonioprosperi2svg profile image
Antonio • • Edited

Thanks for this article, really useful! 😸

Funny thing: it took me slightly off track, in a good way. Reading about WHY extraction made me realize my own project was missing something basic: a CHANGELOG. I maintain a small vanilla JS 2D game engine (BeeEngine) and went from v2.0 to v2.10.0 with only a README 😅 So thanks to your post, it now has a proper CHANGELOG.md, and future developers will actually know what changed and why.

The idea of treating WHY as a selection problem instead of a generation problem, and keeping the original PR text as evidence, is really smart. changelog-bot looks like a big and well-thought-out project. Good luck with v1, and kudos to John for the benchmark! 🙌

Thread Thread
 
nyaomaru profile image
nyaomaru •

Thank you so much! 😸
This genuinely makes me really happy to hear.
The fact that the article actually pushed you to add a proper CHANGELOG.md to BeeEngine is such a nice outcome 😹

I’m really glad the “WHY as selection, not generation” idea resonated with you too.
Thanks as well for the kind words about changelog-bot and the shoutout to John! 🙌

Good luck with BeeEngine and future releases with CHANGELOG.md! 🚀

Thread Thread
 
johnnylemonny profile image
𝗝𝗼𝗵𝗻 • • Edited

Chiming in here - thanks so much for the kind words, @antonioprosperi2svg !

Skipping changelogs until a project matures is painfully relatable! Really awesome that the post nudged you to set up a proper CHANGELOG.md for BeeEngine.

Teaming up with @nyaomaru on that benchmark was a blast, especially making sure we tested against real-world edge cases rather than guessing. Wishing you all the best with BeeEngine and future releases!

Thread Thread
 
antonioprosperi2svg profile image
Antonio •

Thank you so much for stopping by! Your post was exactly the push I needed. BeeEngine finally has a proper CHANGELOG.md, and I'll keep it updated with every release from now on.

Testing real edge cases instead of guessing is a great lesson too. I'll try to apply the same approach to my own engine. Thanks again to both of you!

Collapse
 
panthpatel profile image
Panth Patel •

"Deterministic code owns the structure, AI only enriches it" is the line we ended up drawing too. Our first version let the agent take more of the pipeline steps; it was slower and worse at the boring parts, so now a script opens the PR when a developer moves the ticket, and the reason for the change sits in the ticket a person wrote. Which raises a question for your selector: when an agent writes the PR body, its "This reduces duplication" line is still original text. Does the 50-case corpus include agent-written PR descriptions?

Collapse
 
nyaomaru profile image
nyaomaru •

Good point 😸 Not explicitly yet.
The current corpus mixes real PR-derived cases with adversarial fixtures, but it doesn’t track whether the source PR description was human or agent written.
I think that’s worth adding, because “quoted from the PR” doesn’t necessarily mean “grounded in human intent” if the PR body itself was generated.
I’ve opened an issue for this, and I’d like to support it in the future. 👍

Collapse
 
panthpatel profile image
Panth Patel •

Glad it turned into an issue. On our side the PR and commit links get posted back into the ticket thread, so the human ask is always one hop from the PR, never inside it. How do you plan to tell the two apart: a flag from the tool that wrote the body, or a guess from the text?

Thread Thread
 
nyaomaru profile image
nyaomaru •

I wouldn’t try to guess from the text.
My current direction is to treat the PR itself as the source of truth. If some intent exists only in an external ticket or tool metadata and isn’t reflected in the PR, I don’t plan to fetch it for WHY extraction. 😸

So if an agent wrote the PR body, the important question is whether the rationale is actually present and explicit there, not whether I can infer who wrote it.

Collapse
 
onizuka profile image
Onizuka •

The selection-not-generation framing is the right call. I tried pure LLM summarization on changelogs last year and roughly 30% of entries ended up being plausible-sounding reasons the author never actually wrote — worse than no explanation at all. One thing that bit me: regex markers like so that and in order to also match a lot of commit messages that describe how, not why. You'll probably need a second filter pass on the matched sentences or the noise will creep back in.

Collapse
 
nyaomaru profile image
nyaomaru •

Exactly! That’s the failure mode I was worried about too 😸

The regex markers are only used to find possible WHY candidates, not to accept them.

After that, Jev asks two separate questions:

  • does this candidate explicitly state a reason?
  • does that reason actually apply to this changelog change?

So something containing so that or in order to can still be rejected if it’s really describing implementation rather than rationale.

I also added implementation-only negative cases to the evaluation corpus for exactly this reason. Really good point though that candidate-extraction layer can get noisy very quickly if it starts doing too much. 🙌

Collapse
 
timmkal01 profile image
Timothy Kelvin •

The "no WHY is better than an invented one" rule is the part I'd steal. I build a website audit tool and landed on the same thing... when a site blocks my checker it comes back as null with "couldn't audit" instead of a score, so a blocked site never passes for a healthy one
curious whether the benchmark has any cases where the why only lives in a linked issue ("fixes #123") and not in the PR body

Collapse
 
nyaomaru profile image
nyaomaru •

Yeah 😸 The benchmark already has a similar case: Resolves #123. Closes #456... is expected to produce no WHY.

Right now, changelog-bot doesn’t follow linked issues for rationale. The actual WHY candidates come from the PR body, so if the reasoning only lives in the issue, it returns nothing rather than guessing.

I could see linked issues becoming an explicit evidence source later, though. 👍

Collapse
 
jaredchuvn profile image
Jared Chu •

A useful counterfactual test would keep the cache-fix title and implementation text fixed, then swap only the rationale: one version explains stale cache data, another explains an unrelated README change, and a third has no rationale. The latter two should produce no WHY even when the supplied sentence is an exact quote.

Would you keep those related variants together in a held-out group, tune thresholds on separate cases, and report wrong-change acceptance separately from missing-reason rejection? That would test relevance beyond whether the output came from the PR.

AI-generated test suggestion. I have not run Jev or this benchmark.

Collapse
 
nyaomaru profile image
nyaomaru •

That’s a really useful test shape 😸

Jev already scores two things separately for each candidate "whether it explicitly states a WHY", and "whether that WHY actually applies to the changelog change". So the “wrong change, but perfectly valid quote” case is exactly the kind of false positive I want to reject.

For this use case, I intentionally care more about precision than recall, missing a valid WHY is acceptable, but attaching a wrong WHY is not.

The benchmark currently sweeps thresholds and reports aggregate precision/recall/F1, but I’m not yet grouping counterfactual variants as held-out families or reporting wrong-change acceptance separately from missing-reason rejection.

I like that direction though, especially keeping related variants together so threshold tuning can’t accidentally overfit the same example family. I'll try add this test! 👍

Collapse
 
jabinpou profile image
Mahdi JabinPour •

Keeping the quoted sentence is useful for traceability. Would you also store the PR description’s revision or a hash alongside it? A PR body can be edited after release, so a link alone may no longer show the evidence that the selector actually saw.

Collapse
 
nyaomaru profile image
nyaomaru •

That’s a really good point 😸
Right now I preserve the exact selected source text, but I don’t store an immutable revision or hash of the PR body alongside it.

A hash or snapshot of the evidence seen at extraction time would make the provenance much stronger, especially if the PR description is edited later. I think that’s worth exploring. 👍

Collapse
 
kartik-nvjk profile image
Kartik N V J K •

Not wanting the model to invent a nice-looking reason is the right instinct, because a plausible WHY is more dangerous than a blank one; once it reads well, nobody re-checks it against the diff. I would ground the WHY against the PR discussion and the linked issues as much as the code change, since the intent usually lives in the review comments more than the patch. Are you pulling from the thread, or just the diff?

Collapse
 
nyaomaru profile image
nyaomaru •

Exactly, and the current Jev path is actually narrower than that 😸

Right now it doesn’t read the diff or the PR discussion thread. It extracts candidate rationale from the PR description, then scores each candidate separately for:

  1. whether it explicitly states a WHY, and
  2. whether that WHY actually applies to the PR title / changelog change.

Linked issues and review comments aren’t evidence sources yet. I agree they’re valuable though, especially because intent often gets clarified during review.
The important constraint for me would be keeping the same rule "use them as additional source evidence, not as material for the model to invent a rationale from."

I’ll definitely consider exploring that in the future!

Collapse
 
aireadyleader profile image
AI Ready Leader •

Love that the core tension you picked is 'don't let the AI invent a why'. That's the same line I ended up drawing when we started summarizing AI-assisted work.

One guardrail that has held up well for us: treat a missing 'why' as a signal, not a gap to fill. If the PR trail doesn't contain the reasoning, the generator should leave the field empty (or flag it) rather than synthesize something plausible - a changelog with a confidently wrong 'why' is worse than one with no 'why', because readers will trust it.

The second guardrail is making the why cheap to write: a two-line 'intent' section in the PR template, authored by the person who did the work, gives the extractor real material instead of forcing it to guess from the diff. Extraction gets dramatically more reliable once the input is honest.

Curious whether you've experimented with a confidence signal - e.g., marking entries where the why was inferred vs. quoted - and how readers respond to that.

Collapse
 
nyaomaru profile image
nyaomaru •

Yes, that’s very close to the conclusion I reached too 😸

AI inference can produce a WHY that sounds completely plausible and is even logically consistent with the diff. The problem is that “plausible” is not the same as “what actually motivated the change.”

Once that inferred explanation gets written into the CHANGELOG.md, the documented intent can quietly drift away from the real specification or decision that was made.
So for the Jev path, my current rule is pretty strict. "If the WHY isn’t grounded in actual PR evidence, don’t use it". Missing WHY is better than a convincing but invented one.

I do keep confidence internally for candidate selection, but the output itself is still the original PR text. I’m not currently mixing quoted and inferred WHYs.

And I really like your idea of making the WHY cheap to write in the PR template. Better input is probably much more valuable than trying to make the extractor increasingly clever. 😸

Collapse
 
fgrandon profile image
Francisco Grandón •

Extracting the architectural 'why' rather than just summarizing the git diff diffs is the holy grail of automated changelogs. Most LLM-based release tools just list changed functions, which creates noise for end users. Focusing on semantic intent and user impact makes the resulting notes genuinely readable. Love seeing this level of curation!

Collapse
 
nyaomaru profile image
nyaomaru •

Thank you! 😸
That’s exactly the goal.
Diffs are good at telling us what changed, but not always WHY it matters.
The hard part is extracting that intent without letting the model invent it, which is why I’m leaning toward evidence selection over free-form generation.
Really glad that came through! 🙌

Collapse
 
mudassirworks profile image
Mudassir Khan •

the framing of WHY as a selection problem rather than a generation problem is the right call. once you let the model paraphrase it starts hallucinating intent and that ends up worse than leaving the field blank. the candidate snippet approach keeps the context window tight and forces the model to pick instead of invent.

what I am curious about: how does the bot handle PRs that skip all the rationale markers entirely? does Jev still get a chance to infer from diff context or does it just return an empty WHY and let the human fill it in?

Collapse
 
nyaomaru profile image
nyaomaru •

Yep, no diff inference in the Jev path 😸
If there’s no recognized WHY section, it can still use the PR body as a fallback candidate source, but only if it passes the local trust checks. If nothing explicit and trustworthy survives, it returns no WHY rather than inventing one.

I’d rather leave that gap for a human than turn implementation context into a plausible story.

View full discussion (46 comments)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.