EvoOntology: A Self-Evolving Ontology Layer for Data Agents

One-Line Positioning

EvoOntology (github.com/ruc-datalab/EvoOntology, MIT, Python, created Sep 15, 2026, ~950 stars) is an open-source self-evolving ontology layer from the Data Lab at Renmin University of China. It freezes the domain semantics scattered across table names, columns and file paths into a versioned Ontology Layer, serves it to coding agents such as Claude Code and Codex through MCP tools for on-demand querying, and keeps refining it from real execution trajectories. The paper (arXiv:2609.15779) positions it as the first self-evolving ontology layer for data agents.

The authors' own metaphor is the shortest way in: the ontology layer is trainable agent state, not model weights. The model stays fixed; what changes is the layer of semantics the agent relies on to understand data — built, evaluated and gated for release like weights, except the training signal comes from the agent's actual interaction trajectories with the data.

The Problem: The Agent-Data Gap

The paper frames the gap plainly: heterogeneous data (tables, files, databases) lives outside the agent, and the agent can only reach it through generic tools — it sees column names and file paths, not metric definitions, entity relations or business constraints. Three concrete pains follow:

Pain	Status quo	Consequence
Semantic uncertainty	Raw-data semantics are implicit: table names, columns and scattered observations do not explain how a metric is computed or how entities relate	Every request re-guesses business meaning, and semantic errors do not raise exceptions
Repeated data exploration	No reusable, grounded knowledge accumulates; each task rediscovers the environment	The same tables are re-understood every task; turns and tokens go into repeated labour
Costly semantic maintenance	Hand-written semantic layers need expert upkeep and go stale as data and workloads change; injecting the full layer into the prompt carries ever-growing context overhead	The "more complete" a static layer is, the more expensive and more obsolete it gets

The two existing routes both fall short: let the agent explore raw data directly (every task starts from zero), or inject a hand-built semantic layer wholesale into the prompt (expensive, brittle, cannot keep up). EvoOntology takes a third route: workload-grounded, queried on demand, evolved under control.

The Ontology Layer: Three Tiers

The whole system ships as an MCP server with three interlocking tiers answering "what is the semantic knowledge", "what are the representation rules", and "how is it accessed at runtime":

Tier	Contents	Role
Content Layer	A typed semantic graph with four node families: Term, Mapping, Constraint, Evidence. Semantic Relations connect Terms; Structural References link Terms to Mappings and attach Constraints or Evidence to the objects they govern or support	The knowledge itself: every concept traces back to "which table, which column, which aggregation, on what evidence"
Schema Layer	Defines the fields of the four node families, the permitted Semantic Relation types, and the legal Structural Reference patterns	Sets the representation boundary — evolution may only make typed edits inside it, never free-form ones
Tool Layer	Exposes the ontology via browse_semantics, resolve_semantics and a compact session manifest	The runtime access surface: only the manifest is injected at session start; records are fetched on demand

Each published version is five record files on disk (Term / Mapping / Relation / Constraint / Evidence) under .evoontology/versions/ in the project root. The same directory holds trajectories/ (one tool-call-granularity trajectory per task, no chain-of-thought), evolution/run_N/ (diagnosis, attribution, patches and evaluations for each run), and state.json (trigger checkpoint and thresholds).

_{The Content Layer in Explorer: concepts, mappings, constraints, evidence and their relations are all inspectable — "checkable" is this design's answer to semantic maintenance cost.}

How It Works: Query on Demand, Not Inject in Full

Once the plugin is installed, the data agent gets three things at the MCP level: browse_semantics(query, kind, limit) to discover relevant concepts, resolve_semantics(mentions, context) to resolve mentioned concepts to grounded mappings plus their relations, constraints and evidence, and the compact evo-semantic://session-manifest read once at session start. Detailed records and linked objects are always retrieved on demand — this is the README's "Active access" principle in practice: fetch only the semantics the current step needs, never pour the whole ontology into context.

Paired with it is "Grounded construction": the Builder extracts candidate concepts around the real workload and only commits semantic objects once they validate against the underlying data source. Before publishing there is also a deterministic gate, validate_semantics: valid JSON, complete references, loadability. Note this is structural validation only — whether tables and columns exist, mappings execute, evidence reproduces belongs to the Builder's exploration phase. The two layers stay separate.

The Self-Evolution Loop: Diagnose → Attribute → Patch → Gate

This is the real watershed between EvoOntology and a static semantic layer. The lifecycle runs five steps: Build (derive candidate concepts from the workload, validate against raw data, publish ontology_v0) → Use (the agent queries on demand while tool interactions and results are recorded) → Evolve (diagnose recurring behaviours, attribute them to the Content / Tool / Schema tier, generate localized candidate patches) → Evaluate (compare Parent vs Candidate under identical data, agent, decoding settings and interaction budget) → Publish or reject (only a passing candidate becomes ontology_vN+1; otherwise the Parent is kept and the finding feeds the next round).

Evolution is hosted by the EvolutionSession state machine in the deterministic core. The skill decides "what to change and why"; the state machine guarantees "a run cannot end the wrong way":

_{The builder and evolution framework: the left half is "where the ontology comes from" (the Builder constructs around real workloads and validates against raw data); the right half is "how it changes" (diagnose → attribute → patch → paired gated evaluation).}

Transition	Meaning
running ──Reject──▶ running	A rejected candidate is not terminal: append a summary to rounds.jsonl, update attribution and the problem map, design the next candidate within the same run
running ──Accept──▶ accepted	Search ends: deterministic validation → publish the new version (never overwriting a published one) → update active.json → advance the checkpoint exactly once
running ──budget exhausted / external block──▶ incomplete	No publish, no checkpoint advance; the same batch is retried next run. Judgment-based stop reasons (missing data, unreliable evaluation) require at least 2 formal candidate rejections in the same run

Defaults: 8 candidate rounds per run; evolution triggers after 30 new tasks since the checkpoint or 7 days; zero configuration — there is no config.yaml. To change behaviour, just tell Claude or Codex "remind me every 60 tasks" and the agent writes it into state.json. Versioning has discipline too: published versions are ontology_vN, candidates vN-cK, and only acceptance maps vN-cK to ontology_vN+1.

Two Modes and the Gated Evaluation

Build Step 0 writes the project mode into project.json. Both modes share the same workspace, versions and checkpoint mechanism; they differ only in how workloads enter construction, evolution and evaluation:

• fixed_split: for benchmarks with a fixed question set, ground truth and evaluation boundary. The Construction Pool serves Build and diagnosis; the Validation Reserve is used only for the final gate and must never flow back into construction, diagnosis or patch generation — an isolation design that keeps exam questions out of training.
• rolling_trajectory: for production or cold starts with no fixed test set. A seed workload initializes ontology_v0; later tasks accumulate in trajectories/. After a trigger, a batch is frozen and compared with independently sampled tasks or an LLM Judge A/B; no forced fold split.

The gate itself runs two protocols: absolute scoring when ground truth exists, anonymous LLM Judge A/B when it does not. The paired comparison rests on the paper's keyword — backbone-conditional: an evolved ontology is bound to the backbone that produced it (see the evaluation section), so Parent and Candidate must be compared on the same backbone under the same budget.

Evaluation: Three Benchmarks, Six Backbones

The paper experiments on three widely adopted data-agent benchmarks: BIRD (text-to-SQL on real databases, Oracle Knowledge setting, main metric execution accuracy EX), DDR-Bench (open-ended deep research over 10-K financial data, main metric Trajectory-Wise), and InsightBench (iterative business analysis and insight generation). Backbones cover GPT-5.5, GPT-5.6-sol, Claude-Sonnet-5, Claude-Opus-4.8, DeepSeek-V4-Flash and Qwen3.5-Flash; the analysis subset uses the first four. Below are the paper-reported four-backbone averages:

Benchmark / Metric	ReAct (no ontology)	Initial ontology	EvoOntology (evolved)
DDR-Bench · Trajectory-Wise	69.5	81.8	89.5 (+20.0)
BIRD · EX	63.6	68.7	72.4 (+8.8)
InsightBench · Insight	53.2	54.0	54.2 (+1.0)

Read the table honestly: DDR-Bench is the main battlefield, where self-evolution adds another 7.7 points on top of the initial ontology; InsightBench is nearly flat (+0.2), meaning open-ended insight generation captures little ontology dividend. In the same table, ReAct + episodic memory sits at 75.8 — so of EvoOntology's +20.0, roughly six parts come from "having an ontology" and four from "being able to evolve it".

The ablations answer "which step of the loop actually pays" (DDR-Bench, four-backbone average, Trajectory-Wise; full loop 89.5):

Variant	Score	Δ
w/o Gate	78.3	−11.2
w/o Attribution	83.2	−6.3
w/o Diagnose	84.7	−4.8
w/o typed Patch (free-form)	87.8	−1.7

The gate is the most valuable single component of the design (−11.2), outweighing attribution and diagnosis combined-in-importance: "evolving" is not scarce; only publishing candidates that reproducibly beat the parent is. Among the three editable tiers evolved alone, Tool +13.2 > Content +8.7 > Schema +3.6, and only together do they reach +20.0. In the ontology-structure ablation, Mapping matters most (−13.4) and Evidence next (−8.7) — "which column of which table a concept binds to" and "on what grounds" are this ontology's load-bearing walls.

Overhead: More Context per Turn, Fewer Tokens per Task

On-demand querying is not free. The paper measures the ontology layer's overhead on DDR-Bench (four-backbone average): input tokens per turn rise from 3.2K to 4.6K (initial 4.1K) while output tokens stay flat; but turns per task drop from 14.6 to 8.4, and total tokens per task from 52.6K to 42.0K (−20%). The tokens saved by fewer wrong turns pay for the semantics carried — this is the ledger difference between "active access" and "wholesale injection".

Another finding worth remembering: injecting a static semantic layer (SL) loses points on strong backbones (Claude-Sonnet-5 Trajectory-Wise −15.0). Strong models lean on whatever the prompt pours in; pouring in something wrong is worse than pouring nothing. And the Term identifiers evolved per backbone show pairwise Jaccard overlap of at most 0.62, with cross-backbone transfer dropping at least 6.6 points — an ontology is a backbone-bound asset; there is no cheap "one ontology fits all agents". That is exactly why the evaluation must be backbone-conditional.

Install and Use

The product ships as two self-contained plugins through each client's Marketplace — no cloning the repo, no virtualenv, no separate pip install:

• Claude Code: claude plugin marketplace add ruc-datalab/EvoOntology → claude plugin install evoontology@evoontology
• Codex: codex plugin marketplace add ruc-datalab/EvoOntology → codex plugin add evoontology-codex@evoontology

In a new session, three skill entry points: Claude Code uses /evoontology:build-ontology, /evoontology:evolve-ontology, /evoontology:explore-ontology; Codex uses the same-named $build-ontology workflows. The plugin spawns the service as a module via .mcp.json, started automatically by the client, with the default workspace at .evoontology/ in the project root. The repo also contains benchmarks/ (BIRD, DDR-10K and InsightBench environments, each implementing an EvolutionAdapter; python -m benchmarks list enumerates them) and a ~4,600-line deterministic Python core — the core only implements the runtime, minimal deterministic validation and the evolution lifecycle state machine; all intelligent analysis lives in the skills, and the product has no CLI.

Where It Sits in the Harness Landscape

This site's harness domain (agent runtimes) collects "what an agent runs on": plugins and sandboxes, tool surfaces, skill systems, long-term memory and self-improvement. EvoOntology is a rare specimen on this line — it does not change the agent itself; it bolts a layer of evolving semantic state onto the agent. It occupies all four boxes: plugin form (Marketplace install, Session Start reminder), tool surface (MCP browse/resolve navigation tools plus the manifest resource), skill system (build / evolve / explore), and self-improvement (the gated evolution loop). It explicitly leaves sandboxing, resident workers and automatic loops out of the first release — stated here plainly rather than oversold.

Against peers in this section: the deepagents family solves "what fits in context"; EvoOntology solves "where the data-related part of context comes from and how it updates". If a coding agent's skeleton manages task state, EvoOntology manages data-semantic state — and it is one of the few that builds the evaluation gate into the evolution loop: not "evolve and then show you", but "publish only when reproducibly better". For readers building data agents, Text-to-SQL or analysis agents, cross-check the numbers section against the paper; for readers building agent infrastructure, the EvolutionSession state machine (Reject is not terminal, only Accept advances the checkpoint) is an engineering pattern worth borrowing directly.

Boundaries and Caveats

• Structural validation ≠ semantic validation: validate_semantics only guarantees valid JSON, complete references and loadability; whether mappings execute and evidence reproduces is the Builder's exploration-phase job. Do not treat the gate as omnipotent.
• Ontologies are backbone-bound: cross-backbone transfer loses at least 6.6 points. When switching models, re-run Build/Evolve rather than moving the workspace.
• InsightBench barely moves: do not expect ontology dividends on open-ended insight generation; its sweet spot is structured querying and deep data research.
• Out of scope for v1: Web UI, SaaS, multitenancy, message queues, resident workers, parallel candidates and fully automatic loops are not in the first release — unattended evolution needs a resident background worker; this version provides detection and reminders, with execution initiated by a human.
• Paper-reported data: all scores above come from arXiv:2609.15779 (submitted Sep 14, 2026), not from re-runs on this site; per house convention, treat them as "paper-reported" until third-party reproduction.