Comparison Report

Last updated: August 28, 2026

This article describes the Comparison Report (formerly AI Impact), which estimates AI-generated code with Span's span-detect-1 detection model.

Notice: the currently recommended way to measure AI Code Ratio is Agent Traces, which attributes AI-authored lines from IDE sessions rather than inferring them with a machine learning model.

If your organization uses traces, start here:

The rest of this article applies only if your Comparison report still uses span-detect-1.

What this report shows

The Comparison report (also called the Impact Scorecard - AI Impact) estimates how much of the code that shipped was AI-generated, and how that correlates with velocity and review dynamics. It is powered by span-detect-1, Span's model that classifies merged pull-request diffs as AI or human.

It answers:

  • How much of our shipped code looks AI-generated? Track AI code ratio across teams, people, repositories, job level, and location.

  • Does higher AI dosage correlate with faster throughput? Compare PR throughput between AI-dosage groups.

  • Are review cycles changing? Compare review-cycle counts between those same groups.

span-detect-1 is a directional signal on merged code. It does not see local or uncommitted AI usage, and it does not use IDE session data. For that, use Agent Traces.

Supported languages

This report uses language-specific models. Today it supports:

Python, TypeScript, JavaScript, Ruby, Java, C#, Go, Kotlin, and Swift.

How to access the report

The page title is AI Insights. Find it under AI Effectiveness → AI insights → Comparison.

This tab is only present if the Comparison report is enabled for your organization. If you do not see it, contact your Span representative.

Prerequisites

  • An active GitHub or GitLab connection

  • Merged PRs that contain code in a supported language

Initial setup

No extra configuration is required. The report analyzes merged PRs from connected repositories. Default filters cover the whole organization. Filter selections persist in local storage.

Qualified PRs

Not all PRs are included. There are two qualification levels:

Level 1: AI Code Ratio qualified

The PR must have at least one code chunk in a supported language. Lines are grouped into contiguous chunks. Only added and modified lines are analyzed, and only chunks with at least 700 characters.

Level 2: AI Dosage Analysis qualified

At least one chunk in a supported language, and at least 30% of non-ignored lines in supported languages.

PRs with too little supported code produce unreliable detection results. The 30% threshold keeps the dosage comparison trustworthy.

Turn on Show unknown PRs to see what is being filtered out.

For the full criteria, see How are PRs considered qualified in the AI Impact report when using Span-detect-1?.

If your report is powered by traces instead, qualification is different: see How are PRs considered qualified in the AI Impact report when using traces?.

Where the data comes from

  • Pull request data from your VCS integration (GitHub or GitLab)

  • AI detection by span-detect-1, which classifies code patterns in the merged diff

  • Aggregation at organization, team, person, and repository levels

  • Benchmarking from anonymized industry data

How span-detect-1 works

  • Tool-agnostic. It classifies the shipped code, not which tool produced it (Copilot, Cursor, ChatGPT, Claude, or others).

  • Pattern-based. It analyzes code characteristics, not IDE telemetry.

  • Confidence intervals. Lower and upper bounds reflect detection uncertainty.

  • Automatic. It runs when PRs are merged or reverted. No manual action is required.

This is the main difference from traces. Traces attribute specific lines to the author's AI sessions. span-detect-1 infers AI-looking code from the merged diff, so it cannot see planning, research, or local work that never shipped.

Data freshness

Analysis runs on merged PRs as part of asset processing after VCS syncs, typically every few hours.

Configuration options

Filters

  • Date range. The data-availability banner shows the full analyzed period you can select.

  • Repository. Filter to specific repos.

  • Team / people. Drill down by organizational structure.

  • Pull request has tests. Include or exclude PRs with tests.

  • AI work classification. Filter by work type (New Features, Maintenance, Productivity).

Baseline vs comparison

Define AI-dosage groups to compare:

  • Baseline options: No AI (0-5%), Low AI (5-25%), Medium AI (25-50%), High AI (50-100%), Unknown, or combinations

  • Comparison options: the same buckets

  • Default: No AI (0-5%) vs High AI (50-100%)

Example: compare No AI vs any AI (High, Medium, Low) to see whether AI-assisted development correlates with better outcomes.

View controls

  • Metric selection. AI Code Ratio, PR throughput, PR review cycles, and Estimated detected defects (early access)

  • Breakdown dimensions. Group (teams), Person, Repository, Job title, Tenure, Location

  • Show unknown PRs. Display PRs where AI usage could not be determined (default: off)

Benchmarking filters

Compare against your organization's baseline or industry percentiles (P50 shown by default). Segment by IC level, job title, manager status, or team.

Key metrics

1. AI Code Ratio

Percentage of new or modified lines detected as AI-generated in supported languages.

AI Code Ratio = (Lower Bound + Upper Bound) / (2 × Total Supported Lines)

  • Lower bound: conservative, high-confidence AI detections

  • Upper bound: optimistic estimate including likely AI

  • Total supported lines: lines in languages span-detect-1 can analyze

Ignored file patterns are excluded from the ratio, including common generated-file patterns such as *.generated.js. Change ignore patterns in Settings → Metrics.

2. Velocity (PR throughput)

Merged PR weight per developer-week. A developer-week is a 7-day period identified by its start date and a code contributor ID.

The report shows the difference between baseline and comparison groups. Example: developers in High AI (50-100%) complete 15% more work-weight per week than No AI developers.

Weighted PRs:

  • Distance: a PR's weight, between 0.1 and 2.0, normalized by global PR complexity and line count

  • Time: duration from first commit to merge

A higher value means teams are merging more work-weight per day.

3. Quality (PR review cycles)

The back-and-forth between author and reviewer:

  • 1 cycle: open → approve (no revisions)

  • Multiple cycles: open → comment → commit → approve (each full loop adds 1)

If High-AI PRs have higher review cycles, reviewers may be pushing back more, or be less certain about AI-generated changes. A corresponding rise in the rework stage of the lifecycle often supports that reading.

Report features

Hotspots

  • Bright spots: teams, people, or repos showing the most positive change with AI usage. Example: "Alice Johnson (+15%)" means Alice's velocity improved 15% with AI.

  • Pressure points: segments showing negative changes. Example: "Backend Team (-8%)" means that team's review cycles increased 8%.

Impact metrics

Shows the delta (change %) between baseline and comparison groups, with visual indicators for positive and negative trends. Updates when you change the baseline or comparison selection.

Data tables

Sortable, filterable breakdowns by group, person, or repo, with absolute values and comparison deltas.

FAQ

Why don't some PRs show AI usage?

They must meet qualification criteria: at least 30% of lines in supported languages, and code blocks long enough to classify. Turn on Show unknown PRs to see filtered PRs.

What languages are supported?

Python, TypeScript, JavaScript, Ruby, Java, C#, Go, Kotlin, and Swift. If you do not see a language you need, ask your Span representative whether it is on the roadmap.

How reliable is AI detection?

span-detect-1 is directionally accurate in aggregate (about 95%). Heavily guided or edited AI output may read as human-written. Short chunks (under 700 characters) and unsupported languages are skipped. Use this report for trends, not for any single PR.

Does this work with all AI tools?

Detection is tool-agnostic. It classifies shipped code patterns regardless of which tool was used. It does not prove which tool authored the lines. Traces do.

Can I compare multiple teams?

Yes. Use the breakdown dimensions to drill down by team, and apply filters to compare specific groups.

How do benchmarks work?

Span calculates industry percentiles (P50, P75, P90) from anonymized data across similar organizations. Your metrics are compared to those benchmarks automatically.

Are contractors included?

Only qualifying developers are included.

Can I export this data?

Data tables support standard interactions. Contact your Span CSM for custom export options.

What if we don't use AI tools yet?

The report can still establish a baseline. As the org starts using AI coding assistants, you can track adoption trends. Traces will give a more complete picture once devices are collecting sessions.

Does this track local or uncommitted AI usage?

No. span-detect-1 only analyzes merged PRs. It does not see AI usage in development branches or local work that has not shipped. Agent Traces captures that session-level activity. See Understanding Agent Traces in Span (auspex).

Limitations

  • Legacy detector path. This report infers AI from merged diffs. It is not the recommended measurement of AI impact.

  • Language support. Only the languages listed above are analyzed.

  • Minimum chunk size. Code blocks must meet the 700-character minimum.

  • PR-centric. Only merged PRs, not in-progress or local work.

  • Qualification filtering. The 30% supported-code threshold is required for dosage analysis.

Related