Harness health: weekly fixes for your AI coding setup

Harness health is a weekly, ranked list of fixes for the repositories your engineers work in with AI coding agents. Each fix comes with the evidence behind it and a prompt you can hand to a coding agent. It is in Beta.

Why it matters

An agent works inside its harness: the context files, skills, tools, and tests a repository gives it. When the harness has a gap, a person has to step in to re-explain a convention, run a command the agent couldn’t, or check work the agent had no way to check. Harness health finds those gaps in your AI coding sessions (agent traces) and ranks them by how often people had to step in. Fix the top ones and your agents get more done for each unit of human attention they need.

• For platform and DevEx leads: a weekly fix list for your busiest repositories, with the evidence to justify each change.

• For engineering managers: see where your team keeps stepping in for agents, and confirm when a fix took hold.

• For engineers: copy a ready-made fix prompt into your coding agent and stop re-teaching the same rule.

How it works

A weekly run. Once a week, over the weekend (UTC), Span analyzes the previous 7 days of AI coding sessions. It covers each repository that had at least 30 linked sessions with at least one tool call, up to your 10 busiest repositories.

What it reads. For each repository, the analysis reads:

• the AI coding sessions linked to it

• the pull requests linked to those sessions, including review comments and follow-up fixes

• a read-only copy of the repository’s default branch, to confirm setup insights and check for linters, type checkers, and observability tools

Step-ins. The core measure is a step-in, a moment a person had to step in for the agent. Span sorts each failed step by how it ended:

1. A person stepped in.

2. The agent gave up.

3. The agent recovered after a struggle.

4. One retry fixed it.

Insights are ranked by step-ins. Time is shown for context only, with each wait capped at 15 minutes.

Nine principles. Every insight ties to at least one principle:

• Tooling and config errors: commands and config that fail as written.

• Hand-backs for lack of self-verification: the agent can’t reach logs, the database, or a browser, so a person checks its work.

• Preventable review feedback and follow-up fixes: reviewers keep flagging what a test or lint rule would catch.

• Missing linters and type checkers.

• Missing debugging and observability.

• Out-of-date docs: docs that sent the agent the wrong way.

• Tacit knowledge people re-teach: instructions people keep typing that belong in a context file such as AGENTS.md or CLAUDE.md.

• Skills followed by corrections: skills that lead to a correction or error.

• Environment differences: setups that the people who get the most from agents have and others lack. These are shown as counts, such as “4 of 11 people”, never names.

The listing bar. An insight is listed only if it reached at least 3 people, 2 people across 8 sessions, or 1 person re-teaching the same thing across 10 sessions. Insights below that bar are counted under the list as not shown, so a single bad session can’t top your list.

Resolved, verified by the next run. Each run re-checks earlier setup insights against the current repository. If the problem is gone, the insight moves to the Resolved tab.

What you see

Go to AI Effectiveness → Harness health (marked Beta). The header shows the report’s date range, how many sessions and people it covers, and when it was generated.

Tabs

• Active: the current ranked list. Filter it by Repository and Principle.

• Resolved (N): insights the latest run found gone.

• Archived: everything archived, marked done, or dismissed. You can restore any of them.

Columns on the Active tab

• Step-ins: how often people stepped in, in the insight’s own unit, such as “3 sessions” or “2 times re-teaching it”.

• Insight: the problem and the change to make, tagged New when first found.

• Repository: where the problem lives.

• Reach: people and sessions against the repository’s totals, such as “4 of 11 people”.

Below the list, a line counts the insights not shown; click Show to see them with the reason. A Checkouts line lists linters, type checkers, or observability tools missing from each repository.

The insight drawer

Click an insight to see how often it happened, when it was first seen, quotes from the sessions, and How to resolve, with a Copy fix prompt button. People with the Agent Trace Data permission also see Traces and People tabs with the sessions behind it.

The … menu offers Archive, Mark as done, and Dismiss. Dismiss needs a reason, and later reports won’t raise that insight again.

On the Overview page. AI Effectiveness → Overview has a Recommendations card with the top 5 insights and a View all link.

Fixing with your coding agent

Use Copy fix prompt, or connect your coding agent to Span’s MCP server from the Autotune your harness with Span MCP card on the page. Your agent can also call the Span API directly. Through either, your agent can:

• list insights, for all repositories or one

• get an insight with its evidence, history, and fix prompt

• mark an insight done, archive it, dismiss it with a reason, or restore it

Your agent makes the change and a person on your team merges it. Changing an insight’s status needs the harness.insights:write scope, which you approve once, and every call follows the caller’s Span permissions.

A simple weekly workflow to value

1. Early in the week, open Harness health and look at what is new in the Active tab.

2. Pick the top one to three insights per repository and assign an owner.

3. Use Copy fix prompt to start each fix, and review the change like any other pull request.

4. Dismiss anything that doesn’t apply, with a reason.

5. After the weekend run, check the Resolved tab.

Setup

Harness health needs:

• auspex collecting traces from your engineers’ AI coding tools: Claude Code, Cursor, Codex CLI, GitHub Copilot CLI, and VS Code Copilot Chat. Claude Code and Cursor cloud agents are in Beta. See Understanding Agent Traces in Span (auspex).

• Trace processing on for your organization. Ask your Span representative to turn it on.

• Permissions:

Admins can grant each permission per person, and grant Agent Trace Reporting and Agent Trace Data per team. See Agent Trace permissions.

The first results arrive after the next weekend run once at least one repository qualifies.

Things to know

• Weekly only. There is no on-demand run. Changes you make show up after the next weekend run.

• No report if nothing qualifies. If no repository qualifies, no report is produced that week and the page keeps showing the most recent one.

• GitHub only for repository checks. Checking the repository copy and analyzing pull request reviews work only for GitHub repositories.

• Quotes follow trace access. A quote shows only if you can read that session’s contents. Otherwise it reads Quote withheld — no access to this session.

• Actions apply to everyone. Archive, Mark as done, and Dismiss remove the insight for your whole organization.

• Mark as done isn’t a check. It only moves the insight off your list. The next run checks setup insights against the repository.

• Screened traces’ contents are never read. Trace Screening keeps a screened trace’s contents out of the analysis.

Related articles

• Understanding Agent Traces in Span (auspex)

• Agent Trace permissions

• Trace Screening: Automatically restrict personal or sensitive traces

• auspex: Default redaction, custom patterns, and privacy mode

• Span Agent Trace Analytics: Security Overview (auspex), including which AI providers process trace data

• Agent Readiness Survey

Need help?

Contact your Span account team or reach out to support@span.app.