You ask an AI a question, it reads a stack of documents, and it hands you one confident answer. What that answer quietly hides is the shape of the evidence behind it: that the sources disagree, that five of them are the same wire story reposted under different mastheads, that it’s contradicting something it told you last week, that the honest response was “nobody actually knows.” Retrieval-augmented generation is very good at producing an answer. It is structurally bad at telling you how much the answer is worth.

I wanted the other thing. Not a chatbot — an evidence layer. Something that answers not just “what’s the answer,” but “what do the sources agree on, what’s contested, what changed, and why should you believe any of it.” So I built a POC, called knowl, to test whether one specific design could deliver that. It’s early and it runs on a corpus I made up. But the core idea held up well enough that I want to write it down.

The one rule: weight and contestation are different axes

Here’s the mistake almost every system makes. It takes a claim, weighs the evidence, and gives you back a single number — a confidence, a score, a relevance. And that one number is doing two completely different jobs at once: how much support does this have and are the sources fighting about it. Those are not the same question, and mashing them together produces a specific, dangerous lie — a high number on a claim where three good sources say X and three equally good sources say not-X. The system tells you it’s solid. It’s actually a coin flip wearing a solid number.

So the design’s load-bearing rule is that those two axes never touch. Weight is computed from supporting evidence only — how much independent backing a claim has, saturating as real corroboration accumulates (roughly three genuinely independent sources gets you to about 0.85, and piling on more barely moves it). Disagreement does not lower the weight. Disagreement flips a separate label — singular, consensus, or contested. And the two come back welded together: you cannot get the weight without the contestation state attached to it. A claim is allowed to be high-weight and contested, and when it is, you’re told both, because that combination is exactly the situation you most need to know you’re in.

Volume can’t buy truth

The reason this matters shows up fastest in a manipulation case, which is the probe I care about most.

Give the system a fact backed by one authoritative registry, and the opposite fact backed by a handful of blog posts that are all, on inspection, the same article reposted. Count naively and it’s five-against-one: the blogs win, and your knowledge base now confidently believes a lie because someone hit copy-paste. So before anything gets weighed, observations run through an independence pass — same source, or near-duplicate text, collapses into a single “voice.” Five reposts of one article count as roughly one. The authoritative single source isn’t outvoted by echoes of itself. Volume can’t buy truth; only independent corroboration can, and the system’s whole job is to tell the difference.

Why I wrote “DO NOT SIMPLIFY” over a truth table

When two observations disagree about the same thing, there’s an overwhelming temptation to resolve it cheaply: latest wins, or highest-confidence wins, pick a survivor and move on. I put the words DO NOT SIMPLIFY at the top of that file, because the instant you pick a winner you’ve thrown away the contestation — which was the entire point of building this instead of using RAG.

So conflicting observations don’t get resolved to a winner; they get routed. A small truth table asks the questions that actually distinguish the cases: did the two claims’ time ranges overlap, or did one come after the other? Is this one source correcting itself, or two different sources disagreeing? Is the second observation reinforcing the first or opposing it? Out of that come distinct outcomes — a genuine supersession, a self-correction, consensus building up, or a real live contestation that gets preserved as a contestation. And it’s deliberately timid where it’s ignorant: if I can’t tell whether one fact predates another, the system treats them as coexisting, never as one silently overwriting the other. A knowledge layer’s job is to preserve the disagreements it can’t legitimately resolve, not to launder them into a clean-looking answer.

The number that looks too good

I built a small adversarial benchmark to see if any of this actually worked — consensus questions, contested ones, unanswerable ones it was supposed to refuse, cases where the documents flatly contradict what the model already “knows,” and the manipulation case. To keep the model’s memorized trivia from winning, the corpus is deliberately wrong about the world — the Eiffel Tower relocated to the wrong city, a company that doesn’t exist — so the only way to score is to actually follow the evidence. Then I scored the contestation-aware layer against plain retrieval and a bare model, graded by a blind LLM judge working from a rubric.

The evidence layer went nine for nine. Naive RAG got five; the bare model got one.

That is a great-looking number, which is precisely why I don’t fully trust it and neither should you — the result that flatters you is the one to audit hardest. It’s nine questions, on a benchmark I designed, built specifically to exercise the thing I was trying to prove. It’s a first signal that the design does what I hoped on the cases it was built for. It is not evidence that it survives a real corpus, and I haven’t run it on one yet. The one place it did slip — the manipulation case, at a lenient de-duplication setting — turned out not to be a bug but a dial: turn the skepticism up and it passes, and the honest version ships that as a posture, “balanced” versus “skeptical,” because how paranoid to be about duplicated sources is genuinely a choice and not a constant.

What I actually believe after building it

I don’t yet know whether the whole system earns its keep. It’s a POC, on invented data, tested against a benchmark I stacked in its favor and was careful to say so. Most of it is a hypothesis.

But I came out fairly sure about the one rule, and it’s the part that generalizes past this project: a number you’re not allowed to contest is worse than no number at all, because it takes real disagreement and hands it back to you as false certainty. Whatever you’re building that scores or ranks or believes things on your behalf — the useful version tells you how much and who’s fighting about it, in the same breath, and never lets you have one without the other.

---

Companion code: ramwise-examples/evidence-weight — the weight-vs-contestation split, independence discounting, and the no-latest-wins conflict router.