Comparison Report
Last updated: October 27, 2026
The Comparison report (page title AI Impact) splits your merged pull requests by how much of their code an AI agent wrote, then compares delivery, review, and quality metrics between two groups. It measures AI code from Agent Traces.
Why it matters
Adoption numbers tell you who uses AI tools. This report asks what happens to the work: does AI-heavy code ship faster, need more review rounds, or cause more bugs than work with little AI? Use the differences as signals to investigate, not as proof. The report compares groups of PRs in the same period, and larger or riskier work can land in one group for reasons unrelated to AI.
How it works
1. Measure AI code per PR. Span matches each merged PR's lines to the author's recorded AI sessions and computes the PR's AI code ratio. Only qualified PRs get a ratio. A PR qualifies when at least 20% of its lines match the author's sessions, or when the author's traces ran without a gap around the PR, so genuine low-AI work stays in. See How AI Code Ratio is calculated and How are PRs considered qualified in the AI Impact report when using traces?
2. Bucket the PRs. Each qualified PR lands in one bucket:
No AI: 0-5% AI-generated code
Low AI: 5-25%
Medium AI: 25-50%
High AI: 50-100%
Unqualified: PRs Span could not measure. They sit out of the comparison unless you turn on Show unknown PRs.
3. Pick two groups. The sticky selector sets a baseline (default No AI) and a comparison (default High AI). Each group can hold several buckets, for example No AI against Low, Medium, and High AI together. The Cohort by menu at the top switches the grouping between AI code ratio and Prompt clarity score.
4. Compare. Each card shows the comparison group's difference from the baseline as a percentage: (comparison − baseline) ÷ baseline.
Card | Metrics |
|---|---|
AI code ratio | Share of merged code AI wrote, with the Qualified PRs count |
Velocity | Weighted PRs per developer-week, weighted PRs per cycle-time day, and PR cycle time and its stages |
Code Review | PR review cycles and PR diff size |
Quality | Estimated bug density (90 days). When bug density is off for your organization, the card says the metric is not enabled. |
Weighted PRs scale each PR by size and complexity, so a large change counts for more than a one-line fix. A developer-week is one person's seven-day window. For throughput, Span pools a person's qualified PRs in the week and buckets the whole week; the week qualifies when at least 30% of its merged PRs qualify.
Review cycles count the back-and-forth between author and reviewer. One cycle is open to approve with no revisions. Each round of feedback followed by a new commit adds a cycle.
5. Break it down. The table below the cards repeats the comparison by team, person, level, job title, tenure, or location. Switch between Comparison (two groups) and Bucketed (one bar per bucket) views.
Setup
Agent Traces must be on for your organization. Without it, Comparison does not appear under AI insights, and a direct link to the page shows Trace processing is not enabled. Contact your Span account team.
A connected source control integration and permission to read pull requests.
The Quality card uses bug density, which Span turns on for your organization on request. Every source control connection must be GitHub or GitLab, and the Est Bug PR Density setting must stay on in Settings → Metrics. See Estimated Bug Density Metric.
Find the report under AI Effectiveness → AI insights → Comparison.
Things to know
It shows correlation. The report compares groups of PRs in the same period, not your team before and after AI.
It compares work, not people. Span buckets each PR and each developer-week on its own, so one person's work can land in both groups. A difference between groups doesn't measure any one developer's productivity.
Attribution has limits. Span credits a line to AI only when it matches an agent edit from the PR author's own recorded sessions. A line a person changed after the agent counts as human.
Small groups swing. A handful of PRs can move a percentage a lot. Check the counts behind each group.
Some rows show no percentage. In the breakdown table, a row shows N/A when either group has no value for that metric or the baseline value is zero.
New organizations wait a week. The report shows Not enough data yet until it has at least one week of trace activity.
Recent PRs take a few hours. Attribution and qualification run several hours after a PR merges.
Quality looks 90 days back. The Quality card can't be selected while your date range includes PRs still inside their 90-day observation window. Use Jump before window to move the range.
Worked example: No AI developer-weeks average 2.0 weighted PRs and High AI developer-weeks average 2.5. The Velocity card shows +25% ((2.5 − 2.0) ÷ 2.0). If High AI PRs average 3 review cycles and No AI PRs average 2, Code Review shows 50% more review cycles.
Related articles
How AI Code Ratio is calculated
How are PRs considered qualified in the AI Impact report when using traces?
Understanding differences between AI Impact (Comparison), Tool Utilization, and Agent Traces
Need help?
Contact your Span account team or reach out to support@span.app.