Diff-first analysis

Most release-monitoring tools summarize what a maintainer wrote in the release notes. DevUpdate.io reads that and the actual git diff between the new tag and the previous one. The two sources almost never agree, and the gaps are where bugs live.

How the analysis runs #

1. We detect a new tag on a tracked GitHub source via the GitHub API.
2. We fetch the diff between the new tag and the previous tag using GitHub's compare API.
3. The diff plus the release notes are sent to a language model that's prompted to identify breaking changes, function signature modifications, removed exports, type modifications, and security-sensitive edits, and to flag every place the changelog and the diff disagree.
4. The output drives the risk score and the release summary you see in the dashboard.

What this catches that changelogs miss #

• Silent signature changes. A function went from (a, b) to (a, b, c) with no mention in the release notes.
• Removed exports. A symbol you might be importing is gone, but the changelog only mentions "internal cleanup."
• Behavioral shifts. Same signature, different return shape or side effect: the kind of thing a git diff makes obvious but a changelog glosses over.
• Documentation drift. The changelog says "no breaking changes" but the diff shows several. We flag the discrepancy explicitly.

What it doesn't catch #

• Non-tagged releases. We rely on git tags. A repo that ships from main without tagging won't surface here.
• Source not on GitHub. Diff analysis requires the GitHub compare API. Sources backed by static release-notes pages (e.g. Google Ads API) don't get diff-derived risk scores, only the documented-change signals from the page itself.
• Closed-source dependencies. We can't see code we don't have access to.

Frequently asked questions #

Do you use AI, and how accurate is it? #

Yes, we use OpenAI's models to analyze diffs and generate summaries, prompted specifically to identify breaking changes, undocumented modifications, and security implications. No AI is perfect, so the prompts are deliberately conservative: when there's ambiguity we flag it for your review rather than ignoring it.

What happens if GitHub rate limits you? #

We degrade gracefully. If we hit GitHub's API rate limit while fetching a diff, we fall back to analyzing just the release notes for that release and mark its diff_fetch_status as rate_limited (you still get a summary, just without diff-based insights for that release), and we retry automatically when the limits reset.

See Undocumented changes for the specific signals we surface, and Breaking changes for how detection and false-positive handling work.