Check a knowledge graph's own numbers before believing them.
Graphs built by LLM pipelines report node counts, edge counts and "confirmed by N sources" scores. graphaudit asks of each number what it would be if nothing interesting were happening, and lists what does not hold up.
$ graphaudit relationships.csv --nodes entities.csv --doc text_unit_ids --undirected nodes 7945 → 7718 unique · edges 7536 → 7453 · multi-source edges beating chance: 75/804 report: graphaudit_out/report.md
What it finds
Every check runs on a CSV export of the graph, with no model and no network. Each one has turned up something real in a graph built by an LLM.
Duplicate nodes
Certain duplicates (case, accents, punctuation, articles), likely ones (typos, plurals, titles, legal forms) and partial names, with guards for numbers, honorifics and opposites.
SUPERINTENDENCIA DE BANCOS DELAREPÚBLICADOMINICANA
Sources vs. chance
Is "found in N documents" more than two frequent entities produce anyway? Exact hypergeometric test, Benjamini–Hochberg across all edges.
SANTO DOMINGO — REPÚBLICA DOMINICANA: the letterhead
Edges with no source
Counts relationships with no provenance and how connected the graph is without them.
17–30% inferred; 2–16 pieces without them
Self-loops
A node linked to itself usually means two entities were merged into one.
gpt4.1 — is_faster_than — gpt4.1
Copied from the prompt
Entities named after the worked example in GraphRAG's default extraction prompt, which a model can copy into the graph.
AURELIA · FIRUZABAD · TIRUZIA · ALHAMIA PRISON
Broken types and structure
Missing or self-named types, edges to entities that do not exist, unused entities, disconnected pieces, two-way duplicates.
"person" typed "bret_taylor"
Tested on real graphs, several of them blind
Run unchanged on graphs no rule had been tuned on; each run also exposed at least one error in the tool, fixed and recorded in the README.
| System | Graph | What the audit found |
|---|---|---|
| GraphRAG | Banking regulation, 7,945 entities | The regulator is about 40 nodes (OCR); 4 entities copied from the prompt's example; the top "beyond chance" edge is the letterhead (161 chunks, 15 without it) |
| GraphRAG | A Christmas Carol, three indexes | Each ghost split in two; only 78 of 642 distinct entities appear in all three indexes; a GLM-4-9B index contains Firuzabad and Aurelia |
| LightRAG | Wars of the Roses | Edward IV is three nodes; the same story is told twice and never connected |
| kg-gen | Apple Inc., 1,188 entities | The company is split; most of what the graph knows about Jobs sits on his surname alone |
| ai-knowledge-graph | Five samples | 17–30% of edges are inferred, and they are what holds each graph together |
| itext2kg | OpenAI news, 2025 | Entity resolution over-merges: 16 self-loops such as gpt4.1 — is_cheaper_than — gpt4.1 |
Run it
Build with Cargo, export your graph to CSV (converters included for GraphRAG, LightRAG, kg-gen, ai-knowledge-graph and itext2kg) and point graphaudit at it.
# build $ git clone https://github.com/ernestoariasdiaz/phantom-convergence $ cd phantom-convergence/graphaudit && cargo build --release # GraphRAG output (parquet) to CSV, then audit $ python tools/graphrag_to_csv.py path/to/output graph_csv $ ./target/release/graphaudit graph_csv/relationships.csv --nodes graph_csv/entities.csv \ --id title --label title --doc text_unit_ids --undirected # letterheads and repeated text, without re-indexing $ python tools/boilerplate.py path/to/output graph_csv/relationships.csv clean
Reliable no model
- Duplicates, partial names, edge inflation
- Source support against chance
- Unsourced edges, self-loops, types, structure
- Entities copied from the prompt example
Experimental local model
--judge: a local LLM decides whether two named entities are the same; useful for review, wrong on generic nodes--embed: embedding similarity; measured, and it rates related things above true duplicates
graphaudit audits; it does not clean. Every list is for a person to review before changing the graph.
Cite
The method comes from auditing ANIMUS, an autonomous knowledge-graph system, where 52% of nodes were duplicates and the flagship "32 sources" edge was at chance level.
Software, latest archived version: https://doi.org/10.5281/zenodo.23671604