Evaluation & Reliability
A layer-by-layer view of how ImpactGraph performs across deterministic code analysis, evidence retrieval, and grounded answer generation. Metrics are grouped by evaluation layer and must not be combined into a single accuracy score. Detailed diagnostic metrics are included alongside headline results to provide additional context on retrieval and generation behavior.
Analyzer Correctness
These metrics evaluate deterministic static-analysis output against a source-derived benchmark created independently of analyzer output. Results are distinct from retrieval and LLM generation — no probabilistic variation applies here.
All independently asserted changed symbols were detected. TP=33, FP=9, FN=0.
The 9 extra detections (FP) are module-level symbols emitted by current analyzer semantics — not missed assertions.
15 of 16 independently reviewed expected graph relationships were recovered.
The one unresolved edge involves static type/call resolution for OrderRiskService.assess. Graph precision is not independently asserted.
Every file reported as changed was genuinely in the diff. Zero false positives.
Zero structural graph failures across deterministic integrity checks: duplicate edges, dangling sources/targets, invalid edge types, cycles, broken hops, propagation depth mismatches.
Retrieval Quality
Measures whether the retrieval system surfaces relevant evidence before an LLM generates an answer. Aggregate metrics are from the golden retrieval benchmark. The production hybrid retriever diagnostic is a separate 8-case study and must not be mixed with aggregate figures.
How often the top retrieved result is relevant.
Fraction of expected relevant evidence recovered within the first five results.
Fraction of expected relevant evidence recovered within the first ten results.
Mean Reciprocal Rank — measures how early the first relevant result appears.
RAG / LLM Reliability
DEEPEVAL · EVAL-2Evaluates the generated answer after evidence retrieval, including grounding, relevance, correctness, and citation behaviour. RAG/LLM reliability was evaluated with DeepEval using the EVAL-2 benchmark. All metrics are judge-model scored (gemini-3.6-flash). Context diagnostics are provided in the detail section below.
Generated claims remain grounded in the supplied evidence.
Generated response directly addresses the question.
Factual correctness of the generated answer relative to the reference.
Claims requiring evidence are accompanied by citations.
Fraction of citations referencing non-existent or fabricated sources.
Methodology
Evaluation layers are independent. Analyzer, retrieval, and RAG/LLM metrics measure separate system layers. Numbers from different layers must not be combined into a single accuracy score.
Assertion scope (analyzer only). Fully asserted, partially asserted, and not asserted labels apply exclusively to analyzer benchmark categories. Retrieval metrics carry a Golden Benchmark provenance label. RAG/LLM reliability was evaluated with DeepEval using the EVAL-2 benchmark, so those metrics are labelled Judge-Scored. Judge-scored metrics used gemini-3.6-flash.
Retrieval benchmarks differ. Aggregate retrieval metrics cover the full golden benchmark. The Candidate C production diagnostic is a separate 8-case study — its figures must not be averaged with aggregate benchmark figures. Current corpus: 353 documents / 353 embeddings (Gemini gemini-embedding-001).
Context diagnostics. Contextual Relevancy (0.00%) is a context-retrieval diagnostic, not an answer quality measure. It is distinct from Answer Relevancy (88.89%). All diagnostic values are recorded judge-model results from Eval-2.
Scope of results. All values are measurements against specific, controlled benchmark fixtures and do not guarantee identical performance for every repository or diff. Dynamic patterns (e.g. runtime imports) are reported as analyzer limitations, not silently omitted.