Knowing whether it improved
The reported bug was 5 chunks. The real one was 990.
Problem
ClipBase is my own lightweight, semantic knowledge base. It holds the repos, articles, papers, and videos that I save. In the map of the three memories, it is the in membrane: what I have read, measured before it is trusted. Its job is to keep agent conversations, and the content ideas that come from them, anchored to what I save. The same save motion also publishes the NoCodeWorkflows directory.
The corpus is 472 items with usable text, in a Turso database. Agents search it in three ways. Keyword search uses a full-text index of the stored document. Semantic search uses an embedding of each chunk. Hybrid search fuses the two rankings on position, not on score.
It had a bug that I saw. For the query “how do I test and debug an MCP server I am building,” the top two semantic hits were items that a human graded zero. The second hit was a chunk that contained three words: “## MCP Tools”. That heading was embedded as a passage, and it outranked each chunk that answered the query. Semantic search scores an item by its single best chunk. Three on-topic words get a better score than a 1,400-character passage that is only mostly on-topic.
The question was not how to fix that chunk. The question was how much of the corpus had that shape, and whether a fix makes retrieval better. Most people who ship an AI feature can answer neither question. The demo query looks better after the change, so the change ships.
System
I did not fix the chunk that I saw. I counted the shape across all 8,522 stored chunks first. The count showed 990: the 5 chunks I reported, plus 985 more whose heading was split from its body at the same kind of boundary. A fix for the reported chunks covers 0.06% of the corpus. The defect covers 11.6%.
I measured the cheap fix and refused it. That fix absorbs undersized chunks into the next chunk, and it is the cheaper reprocess. But it repairs 5 of the 990 chunks and keeps the other 985 passages apart from their titles. The fix that shipped binds a heading to the first piece of the block below it and stops at the size ceiling. A stranded heading costs less than a flat table.
A second defect shipped in the same chunker version, and that was deliberate. Page furniture survived extraction, and the last three words of a site footer were an ad that took semantic rank 2 on a query about voice cloning. Both changes move chunk boundaries. Two boundary changes in one version cost one re-embedding of the corpus. The same changes in two versions cost two re-embeddings.
Measurement
I applied the fix to the live corpus. It rewrote 383 items, moved the corpus from 8,522 chunks to 8,491, and re-embedded 7,504 chunks for 2.3 million tokens. That is about $0.35 at the list rate. The net count moved by 31. The work was 7,504 embeddings, almost a full rebuild from empty. That number is the number on which the decision turns.
Then I ran the eval: 30 labeled queries with hand-graded answers, across all three rankers. Keyword search is the control. It indexes the stored document, not the chunks, so the chunk changes cannot touch it. It reproduced its pinned baseline, Success@1, Success@5, and MRR, to three decimals. If the control moves, the cause is a change other than mine.
Both stated failures closed. The MCP-server query moved from semantic rank 6 to rank 4. The bare “## MCP Tools” chunk does not exist now. It is bound to the 1,410-character table that it labels. The voice-cloning query returns its first correct answer at semantic rank 1.
And the aggregate became slightly worse. Hybrid Success@1 decreased from 0.800 to 0.767. MRR decreased from 0.854 to 0.830. nDCG decreased from 0.663 to 0.654. Recall decreased from 0.682 to 0.678.
One query of Success@1 on a 30-query set is the resolution of the instrument, not a result. But recall and nDCG moved in the same direction, and that shows a mild net reorder, not noise alone. A boundary change on 383 items changes the best-chunk score for each item. Slightly more queries lost a slot than won a slot.
I kept the change. The defect was real and measured. No content was lost that was not meant to go. The heading fix moved boundaries and did not touch a single line. The only deletions were the 900 footer lines that the second fix exists to remove. A reversal means a second 7,500-chunk re-embedding, back to a state with a known precision bug.
The eval says that the fix costs about one rank slot. That is a price, not a refutation.
What runs today
My agents search ClipBase, and the new chunker runs against the live corpus. The measurement has one hole, and the hole is mine. I cannot say which query lost that Success@1 slot. No per-query rank table was recorded before I applied the change. That table prints on each eval run and costs nothing to keep. The rule now is to put it into the decision log before each change that moves chunk boundaries.
The next drop will be attributable. This one is not.
The artifact
The panels show the reported bug, the count that reframed it, and the measurement that priced the fix. The first panel alone is the version that everybody publishes.
semantic Q13: how do I test and debug an MCP server I am building was: 268 18 13 78 331 212 333 334 305 233 now: 434 463 268 18 13 78 331 212 333 334 rank 1: id 434 — graded 0 by hand rank 2: id 463 — graded 0 by hand item 463 — the chunk store row#8438 seq=7 len=12 wc=3 :: "## MCP Tools"
A human graded both top hits zero. The second hit is twelve characters, embedded as a passage. It outranks each chunk that answers the query.
{ "items": 472, "changed": 368, "v2": { "chunks": 8522, "headingOnly": 5, "trailingHeading": 985, "leadingHeading": 316, "newlines": 102653, "overMax": 0, "longest": 2380 }, "v3": { "chunks": 8532, "headingOnly": 0, "trailingHeading": 7, "leadingHeading": 1270, "newlines": 102633, "overMax": 0, "longest": 2398 }, …
Panel one is a bug. This panel is the defect. In the live chunker (v2), five chunks are a heading and nothing else. And 985 more chunks end on a heading whose body went to the next chunk. That is 990 chunks, 11.6% of the corpus, across 368 of 472 items. The revision (v3) decreases the trailing headings to seven.
the billembedded 7488/7504 chunk(s)embedded 7504/7504 chunk(s)applied: embedded 7504 chunk(s), 2333013 token(s) pinned baseline — beforeretrieval eval · 30 queries · k=10 collection · items 1–487 · eval/queries.collection.json (pooled 2026-08-05) FTS semantic hybrid Success@1 0.533 0.700 0.800 Success@5 0.867 0.900 0.900 MRR 0.670 0.784 0.854 Recall 0.562 0.670 0.682 nDCG 0.495 0.643 0.663 after apply FTS semantic hybrid Success@1 0.533 0.667 0.767 Success@5 0.867 0.933 0.900 MRR 0.670 0.762 0.830 Recall 0.562 0.660 0.678 nDCG 0.495 0.625 0.654
The FTS column is the control. It indexes the stored document, so the chunk changes cannot touch it. It reproduces its baseline to three decimals. The count made the change worth 7,504 re-embeddings. The eval says that the change cost about one rank slot.
The panels are typeset verbatim from the recorded 2026-08-08 session transcripts (fold f5923944, apply d6bb3efc). The pre-fix corpus does not exist now, so a new run is not possible. The recorded measurement is the honest exhibit.