Code2Skill: Mining a Million Grounded Agent Skills from 19,769 GitHub Repositories

Code2Skill: Mining a Million Grounded Agent Skills from 19,769 GitHub Repositories

Sources: the DeveloperSkillHubs project page by Ant International AI Research, the technical report Grounded Skill Synthesis from Code at Scale for Agentic Intelligence (arXiv:2609.05571), and the companion Code2Skill repository with its Hugging Face dataset.

Every serious agent builder eventually hits the same wall: the model knows a lot, but the procedure for getting a specific job done correctly lives somewhere else, in runbooks, in senior engineers' heads, in the error-handling branches of a mature codebase. The agent-harness community calls that missing layer "skills": reusable procedural knowledge packaged together with the conditions under which it applies, so an agent can retrieve and execute the relevant procedure at inference time. Skills can be versioned, deployed, and updated independently of model weights at relatively low cost, which makes them the most practical plug-and-play interface for feeding continuously evolving domain knowledge into agentic systems.

The hard part is acquisition. Trajectory-based synthesis distills skills from an agent's own successful episodes, so the resulting bank is coupled to the model, tools, and harness that produced it, and its quality is bounded by that agent's competence. Document-based synthesis avoids trajectories, but yields descriptions with no executable evidence behind them: there is nothing concrete to check an operational claim against. Ant International's Code2Skill proposes a third substrate that sits between the two: source code. Maintained repositories record implemented control flow, state changes, constraints, and failure paths. They exist before any target agent interacts with a task, and they remain inspectable after abstraction. Code2Skill mines them at repository scale and, crucially, verifies every abstraction against the implementation it came from.

The headline numbers: from 19,769 actively maintained GitHub repositories, Code2Skill produces CodeSkillBank, a bank of 1,006,822 accepted skill records. Across 72 protocol-matched evaluations spanning nine model settings and eight benchmarks, models augmented with retrieved skills lift the macro-average from 42.90 to 47.90, an 11.7% relative gain, with 57 of 72 pairs improving. Under one shared downstream interface, Code2Skill also beats three trajectory-derived skill banks on all seven shared benchmarks. This post walks through how the pipeline works, what a skill record actually contains, what the million-record bank is made of, and where the gains hold and where they do not.

The pipeline: from source units to verified skill records

Code2Skill runs in four stages. It first scans GitHub repositories available by April 14, 2026 and retains projects with more than 500 stars, yielding a source pool of 19,769 repositories. The median repository in that pool has 3,133 stars and 82 merged pull requests, 78.3% have at least 1,000 stars, and 66.0% were pushed within the previous year: this is mined from broadly used, actively maintained implementations, exactly the kind of codebase where reusable conventions, boundary checks, and repair patterns accumulate.

Stage 1, selecting candidate procedural evidence. Within each repository, the parser builds candidate functions, methods, command-line entry points, and file-level components, excluding unsafe files, test files, binary payloads, and units too short to expose reusable behavior. An LLM tagger then scores each unit against a six-signal rubric: reusable intent beyond project-local glue; ordered operational steps; control-flow, state, or resource transitions; boundary conditions, invariants, and failure handling; interface and context sufficiency for later reconstruction; and non-triviality relative to getters, wrappers, and boilerplate. Units that clear the gate, subject to a per-repository cap, remain candidates: nothing is accepted yet.

Stage 2, typed skill record generation. The extractor maps each selected unit and its structural context to one of three granularities, because procedural knowledge appears at different source scopes. Atomic skills capture a single well-defined operation inside one function or method; composite skills capture ordered workflows that coordinate multiple operations; recurring-pattern skills capture higher-level implementations that span beyond a single localized operation. A single record type would either fragment multi-step procedures or overgeneralize local behavior.

Stage 3, source-body-blind reconstruction and consistency checking. This is the quality gate that separates Code2Skill from "ask an LLM to summarize this function". A reconstructor regenerates code using only the synthesized record: no source body, no repository name, no file path. If the record omits a critical step, invariant, or failure branch, the mismatch surfaces when a source-aware judge compares the reconstruction against the original implementation. Sufficiently consistent reconstructions are accepted directly; the rest go to an adjudicator that distinguishes an unsupported skill record from a failed reconstruction. Accepted records keep their final status, decision rationale, reconstruction outcome, and repository-, file-, symbol-, and source-span-level provenance.

Stage 4, retrieval-oriented tagging and purpose indexing. Feature tagging creates one task-oriented retrieval record per accepted skill using closed vocabularies, preserving a one-to-one correspondence with the evidence archive. Purpose indexing then filters low-value candidates, groups records by a normalized key of task family, intended action, and intended target, and selects an existing record as each group's representative. Both transformations leave the raw evidence archive intact, so audit and refresh always reach back to the original spans.

Anatomy of a skill record: boundaries and receipts

What does an accepted record look like in practice? The project page shows one extracted from AdGuard's public DNS proxy repository, and it is worth reading closely, because it is the clearest statement of what Code2Skill considers a skill.

The record, recoverDoQAfterCachedQUICFailure, captures a DNS-over-QUIC resolver that reuses QUIC connections, where a cached connection can fail after server restart, idle close, stateless reset, NAT breakage, or 0-RTT token rejection. Its reusable insight is not "retry on error". It is a bounded recovery policy: retry only failures that came from a cached connection; close the failed connection before reacquiring; clear the QUIC token store only for the specific 0-RTT rejection error. The invariants field forbids retrying a freshly opened connection's failure, forbids leaving a failed cached connection in the cache, and requires restoring DNS message IDs. The anti-goals field names the tempting but incorrect behaviors explicitly: no unbounded retry loop, no global resolver reset, no silent error swallowing, no token-store reset for unrelated transport errors. And the source-evidence field is the receipt: the upstream resolver at commit f35ca3e, with the exact symbols that expose the cached-failure branch, the close-and-reacquire boundary, the one-retry budget, and the error-specific token reset.

That combination is the point. A code summary tells you what a function does; a Code2Skill record tells an agent when to apply a procedure, which steps are essential, what must stay true, how failures change the path, which generalizations are out of scope, and where in the source every one of those claims can be checked.

What a million-record bank actually contains

Scale means little if the bank is a pile of shallow API facts, so the team annotated all 1,006,822 accepted records along semantic dimensions. The profile that emerges is emphatically procedural. Multi-step procedures account for 57.3% of records and constraint reasoning for another 33.5%, while surface API calls are only 8.6%: 90.8% of the bank preserves either an execution sequence or the conditions governing correct execution.

On the target side, data transformation and state updates are the most common problem archetypes; on the solution side, parameterization, validation, state-machine control, and API usage lead the reuse mechanisms. The two axes are many-to-many but structurally coherent: state-update problems align with state-machine reuse, decision validation with validation patterns, external interaction with API usage, and error recovery with robustness patterns. That coherence is what makes multi-dimensional retrieval possible: an agent can query for "state update plus state-machine control" instead of relying on broad keyword overlap.

The knowledge and risk profile explains why provenance is not optional here. Library or framework behavior appears in 49.4% of records, programming-language semantics in 47.5%, and data formats or schemas in 47.2%; nearly half of all records depend on context-sensitive assumptions where an unsupported generalization silently changes behavior. In parallel, 79.6% of records are correctness-critical, with substantial coverage of user-visible behavior, data-loss risk, and evaluation-sensitive logic. On the agent-facing side, the leading use cases are code generation (69.1%), SWE-agent workflows (58.3%), and tool use (41.8%), and most records teach what to check (79.1%) and how to structure a solution (66.7%) rather than providing code to copy.

Human annotation backs up the automated gates. Annotators scored records sampled from four separately sampled construction outcomes on description accuracy, reconstruction correctness, and retention value. In the accepted bank, 92% of skill descriptions are judged accurate, 80% of records are worth retaining, and 84% of directly accepted records support correct reconstruction. The rejected pool, by contrast, reaches only 32% description accuracy, 28% retention value, and zero correct reconstructions. Purpose-level cards also retain workflow-level content: 31.12% composite-only and 3.53% mixed cards, against 65.35% atomic-only.

Does the bank actually help agents?

The main evaluation covers nine model settings (DS4-Flash, Qwen3.5 27B, Qwen3.6 27B, Gemini 2.5 Pro, and GPT 5.2, each with its reasoning mode) across eight benchmarks: BigCodeBench and SWE-bench Verified for programming and software engineering; TerminalBench, LongCLI-Bench, and AgentBench-OS for terminal and OS control; AIME 2026, HMMT 2025, and GPQA for mathematical and scientific reasoning. The default harness is a draft, review, revise loop, and every comparison is protocol-matched: only skill access changes. For computational efficiency the default retrieval pool is a random 10% sample of the available records.

Model setting	Reasoning	No skill	+ CodeSkill	Delta
DS4-Flash	No	35.62	43.01	+7.39
DS4-Flash	Yes	44.47	51.76	+7.29
Qwen3.5 27B	No	32.89	36.61	+3.72
Qwen3.5 27B	Yes	47.60	50.94	+3.34
Qwen3.6 27B	No	37.74	40.00	+2.26
Qwen3.6 27B	Yes	50.59	57.33	+6.74
Gemini 2.5 Pro	Default	37.32	42.09	+4.77
GPT 5.2	No	43.15	49.16	+6.01
GPT 5.2	Yes	56.73	60.23	+3.50
Macro-average		42.90	47.90	+5.00

Every complete model-setting average improves, with relative gains from 6.0% to 20.7%, and 57 of 72 protocol-matched evaluations improve. The gains are broad but not uniform, and the paper keeps the variation visible rather than hiding it behind the headline.

Benchmark	Improved pairs	Observed pattern
SWE-bench Verified	9 / 9	Every matched pair improved
BigCodeBench	6 / 9	Three pairs were lower
AIME 2026	9 / 9	Every matched pair improved
HMMT 2025	8 / 9	One pair was lower
GPQA	8 / 9	One pair was lower
TerminalBench	8 / 9	One pair was lower
LongCLI	4 / 9	Five pairs were flat
AgentBench-OS	5 / 9	Three lower, one flat
All benchmarks	57 / 72	79.2% of protocol-matched evaluations improved

The pattern is interpretable. Improvements are largest on SWE-bench Verified and TerminalBench, where repository navigation and multi-step interaction give stored workflows, invariants, and error cases room to apply. Gains on AIME, HMMT, and GPQA suggest that structured checking transfers beyond coding. The mixed results on BigCodeBench under reasoning mode mark the boundary: short, self-contained problems that a capable model already solves directly leave procedures little leverage.

The sharper question is whether the gains come from the code-derived bank or merely from having some skills in the loop. RQ2 answers it by rebuilding three trajectory-derived banks, Trace2Skill's Creation+Error pipeline, ExpeL's experience memory, and a SkillRL-adapted SkillBank, on a held-out task split with Qwen3.5-397B-A17B, then evaluating every bank with DS4-Flash reasoning in the same agent loop over five runs.

Method	SWE	BigCode	AIME	HMMT	Terminal	LongCLI	AgentBench	Avg
Trace2Skill	6.0	31.9	63.3	46.7	6.9	18.8	43.1	31.0
ExpeL	7.5	29.4	50.0	20.0	22.2	18.8	47.4	27.9
SkillRL-Bank	36.8	29.4	43.3	26.7	31.9	10.4	51.2	32.8
Code2Skill	44.7	42.3	70.0	53.3	45.2	30.0	61.3	49.5

Code2Skill ranks first on all seven shared benchmarks (means over five runs). Even a per-benchmark oracle that picks the strongest trajectory-derived baseline separately for each benchmark averages only 40.1, still 9.5 points below Code2Skill; on individual benchmarks the margin over the strongest baseline is 6.6 to 13.3 points. Repository-derived skills therefore provide useful procedural knowledge before an agent accumulates its own interaction experience, which is precisely the regime trajectory-based methods cannot serve.

Placement matters: where the skill enters the workflow

The same retrieved record can play different computational roles depending on where the agent sees it. Appended to the first-pass generation prompt, the model must translate guidance directly into a complete answer; supplied during planning, its conditions and workflow constrain task decomposition; revealed only after a draft exists, its invariants, failure cases, and anti-goals become concrete review criteria. The experiments separate these roles cleanly.

• Planning-time guidance improved all eight shared model-benchmark pairs: retrieved records can structure decomposition before execution.
• Generation-time prompting was mixed and model-dependent: DS4-Flash improved on all four shared benchmarks with large gains on AIME and TerminalBench, while Qwen3.5 benefited less consistently.
• Post-generation review showed the broadest and most stable positive pattern, consistent with the 57-of-72 result in the main table.

The same choice persists in training. In a coding-RL pilot on a shared Qwen3-32B SWE-World checkpoint, evaluated at training step 150 under simulated test-pass rewards, the no-skill control resolved 24% of tasks. Exposing CodeSkillBank to the policy with full records reached 32%, compact summaries 31%, a reward-side verifier reference 31%, and a post-generation reviewer 38%. All four interfaces beat the control, with review highest. The authors are careful about what this pilot does and does not show: one checkpoint, no repeated seeds, no aligned learning curves, so it signals that skills can supervise learning, not that they speed convergence.

Interface	CodeSkill placement	Resolve rate at step 150
No CodeSkill	None	24%
Full prompt	Policy, full records	32%
Summary prompt	Policy, summaries	31%
Reward reference	Verifier	31%
Post-generation review	Reviewer	38%

Less context, same utility

A million-record bank is only useful if agents can query it without exhausting their context windows, and the retrieval study delivers one of the most practically useful findings in the paper. Increasing retrieval depth from k=1 to k=10 expanded average rendered context from 2.1K to 17.8K characters with little additional utility: Qwen gained 1.40 points while DS4-Flash stayed at or below its no-skill baseline. Rendering, not depth, is the lever. At k=3, summary rendering cut average skill context by 88.9%, from 6,352 to 707 characters, while Qwen preserved its full-record score and DS4-Flash improved from 28.40 to 31.80. Purpose indexing, which replaces near-duplicates with cluster representatives, reduced full-record context from about 6.4K to 5.0K characters but had mixed downstream effects, presumably because a canonical representative can displace a locally relevant candidate.

The design lesson for anyone building a skill harness: compact, relevant procedures beat volume. Keep the full evidence archive for provenance and audit, but render the procedural core at decision time.

AI-generated code as the next source

If skills come from code, and AI writes a growing share of code, can the pipeline feed on its own era? RQ6 tests this with a source-controlled protocol: 25 tested implementations sampled from the accepted archive, each re-implemented by Codex with GPT-5.1 at extra-high reasoning effort under an interface-matched contract, with both versions required to pass the same public and hidden tests. Origin labels were withheld from the extractor, and each implementation yielded one "how to implement" record and one "how to verify" record, giving two 50-skill banks evaluated on the same 400-task LiveCodeBench subset with identical retrieval slots.

The human-code bank scored 93.00% and the AI-code bank 93.50%. The aggregate gap is negligible, but the banks disagree on 16 tasks: the human bank uniquely solves seven, the AI bank nine. Tested AI-generated implementations therefore carry non-identical but comparably useful procedural signal, which means CodeSkillBank can keep expanding as AI-written software accumulates, rather than freezing at the boundary of human-authored repositories.

Boundaries, caveats, and what to take home

Three honest boundaries deserve emphasis. First, the verification gate is an LLM consistency check, not a proof of program equivalence: the repository README states this plainly, and reconstruction-based checking is positioned as a scalable repository-level filter rather than a substitute for test-based verification. Second, several results are pilots by design: the RL study reports a single checkpoint, and the default retrieval pool is a 10% sample chosen for compute efficiency. Third, the gains are interface- and task-dependent: short self-contained problems and some OS-control settings show flat or negative pairs, and generation-time prompting helps some models more than others.

Even with those boundaries, the contribution is a new scaling axis for agentic systems. Beyond model parameters and inference-time compute, Code2Skill treats procedural knowledge itself as a scalable, versioned, auditable artifact: built offline before any target agent acts, grounded in implementations that can be re-inspected as code evolves, and consumable at whichever decision point can actually use it, from planning and review to reward-side verification in RL. For teams building coding agents today, the transferable lessons are concrete: verify abstractions against their source rather than trusting summaries; keep provenance and anti-goals in the record, not just steps; place skills at decision points that can consume them; and render the procedural core compactly while the evidence archive stays one hop away.

The bank, the pipeline, and the project page are public: the DeveloperSkills-Code2Skill dataset on Hugging Face, the Code2Skill repository (currently a publication-oriented, pre-release implementation), and the DeveloperSkillHubs page with interactive semantic panels over the full million-record bank.

---

Sources: DeveloperSkillHubs: Developer Skills from Real Code, Ant International AI Research project page; Grounded Skill Synthesis from Code at Scale for Agentic Intelligence, arXiv:2609.05571, September 4, 2026; ant-intl/Code2Skill repository README. Figures reproduced from the project page and paper.