Knowing whether it improved

Problem

ClipBase is my own lightweight, semantic knowledge base. It holds the repos, articles, papers, and videos that I save. Its job is to keep agent conversations, and the content ideas that come from them, anchored to what I save. The same save motion that feeds the NoCodeWorkflows pipeline feeds ClipBase.

The corpus is 472 items with usable text, in a Turso database. Agents search it in three ways. Keyword search uses a full-text index of the stored document. Semantic search uses an embedding of each chunk. Hybrid search fuses the two rankings on position, not on score.

It had a bug that I saw. For the query “how do I test and debug an MCP server I am building,” the top two semantic hits were items that a human graded zero. The second hit was a chunk that contained three words: “## MCP Tools”. That heading was embedded as a passage, and it outranked each chunk that answered the query. Semantic search scores an item by its single best chunk. Three on-topic words get a better score than a 1,400-character passage that is only mostly on-topic.

The question was not how to fix that chunk. The question was how much of the corpus had that shape, and whether a fix makes retrieval better. Most people who ship an AI feature can answer neither question. The demo query looks better after the change, so the change ships.

System

I counted the shape across all 8,522 stored chunks before I wrote a fix. Five chunks were a heading and nothing else. A fix for only that one chunk covers 0.06% of the corpus.

The same count found the real defect. 985 more chunks ended on a heading, and the body of that heading went to the next chunk. The same accumulator flushed on the other side of the boundary, and the same split divided a passage from its title. That is 990 chunks, 11.6% of the corpus, across 368 of 472 items. Only 316 chunks started with their heading. A heading went to the wrong side of a boundary 3.1 times more often than to the correct side.

I measured the cheap fix and refused it. That fix absorbs undersized chunks into the next chunk. It is 2.6 times cheaper to reprocess, and it does fix the chunk that I reported. But it fixes 5 of the 990 chunks, and it keeps 985 passages apart from their titles. It also moves 329 badge and link rows into real passages, and that dilutes good vectors with URL blobs.

The fix that shipped binds a heading to the first piece of the block under it, and it stops at the size ceiling. A second split of an oversized bound pair sends the pair through the word packer again. That process changed one 2,395-character markdown table into three fragments on one line, and 83 newlines were lost. A stranded heading costs less than a flat table.

A second defect shipped in the same chunker version, and that was deliberate. Page furniture survived extraction. The last three words of a site footer were a text-to-speech ad, and that ad took semantic rank 2 on a query about voice cloning. Both changes move chunk boundaries. Two boundary changes in one version cost one re-embedding of the corpus. The same changes in two versions cost two re-embeddings.

Measurement

I applied the fix to the live corpus. It rewrote 383 items, moved the corpus from 8,522 chunks to 8,491, and re-embedded 7,504 chunks for 2.3 million tokens. That is about $0.35 at the list rate. The net count moved by 31. The work was 7,504 embeddings, almost a full rebuild from empty. That number is the number on which the decision turns.

Then I ran the eval: 30 labeled queries with hand-graded answers, across all three rankers. Keyword search is the control. It indexes the stored document, not the chunks, so the chunk changes cannot touch it. It reproduced its pinned baseline, Success@1, Success@5, and MRR, to three decimals. If the control moves, the cause is a change other than mine.

Both stated failures closed. The MCP-server query moved from semantic rank 6 to rank 4. The bare “## MCP Tools” chunk does not exist now. It is bound to the 1,410-character table that it labels. The voice-cloning query returns its first correct answer at semantic rank 1.

And the aggregate became slightly worse. Hybrid Success@1 decreased from 0.800 to 0.767. MRR decreased from 0.854 to 0.830. nDCG decreased from 0.663 to 0.654. Recall decreased from 0.682 to 0.678.

One query of Success@1 on a 30-query set is the resolution of the instrument, not a result. But recall and nDCG moved in the same direction, and that shows a mild net reorder, not noise alone. A boundary change on 383 items changes the best-chunk score for each item. Slightly more queries lost a slot than won a slot.

I kept the change. The defect was real and measured. No content was lost that was not meant to go. The heading fix moved boundaries and did not touch a single line. The only deletions were the 900 footer lines that the second fix exists to remove. A reversal means a second 7,500-chunk re-embedding, back to a state with a known precision bug.

The eval says that the fix costs about one rank slot. That is a price, not a refutation.