How Silver Bullet learns

For anyone who uses Silver Bullet 3.0, and for anyone deciding whether to buy it. It explains how Silver Bullet gets better from the corrections you make, what you check before a correction is used for every way of asking, what each kind of fix does, what was measured, and the limits.

The screens are described step by step in the Administration Guide (Your corrections, verified answers) and the User Guide (“No, fix it”).

The short version

• No AI model is fine-tuned or retrained. Silver Bullet learns by keeping verified answers, rules and words in its own store, where you can see, change and remove each one.
• Nothing is learned until you check it. A correction is in use for the same question at once; it becomes a verified answer, used for every way of asking, only after you check it under Your corrections.
• Every fix is checked before it is saved. A safety replay reads your verified questions and recent questions again with the fix, and shows any answer that would change; its model check asks the AI model again about the recent questions only the model reads, wherever the fix would change what the model is told.
• Your data stays where it is. Learning works on questions and their readings. The AI model never receives your rows or query results.

The five parts

1. Questions are read into explicit choices first

Every way of asking (tapping a card, building a question, a shortcut, or typing) ends in the same explicit choices: the measure, how to split it, the period, the filters, the ranking and the kind of answer. The SQL is written from those choices by Silver Bullet itself, the same way every time.

Silver Bullet reads a typed question with its own parser first, using the data dictionary’s names and other names, business measures, rules, the financial year (April to March) and the values in the data. Only what the parser cannot read goes to the AI model, which returns choices, not SQL; its choices are checked against the data like any others.

This is what makes learning possible: a correction is a change of choices (“the period should have been the same quarter last year”), which can be stored, checked and replayed exactly. The “How I read your question” chips on every answer show the choices, and where each came from.

2. Verified answers, matched by meaning

A verified answer is a checked question with its choices (or, for an answer that did not fit the usual choices, its query, checked by the safety layer before it is kept). When you type a question on the same data source, Silver Bullet looks for a verified answer that means the same, before anything else:

1. The same question, ignoring case, extra spaces and punctuation at either end.
2. By meaning, with an embedding model: nomic-embed-text in Ollama by default or, when the AI model is Google Gemini, Gemini’s own gemini-embedding-001 (the questions already go to Google). Each question becomes a list of numbers that places it by meaning; two questions match when their similarity (cosine) reaches the model’s threshold: 0.85 for nomic-embed-text, 0.87 for gemini-embedding-001 (each measured; see Measured results). “Net sales by region last quarter” and “region-wise net sales for the last quarter” match this way. Vectors of two models are never compared: after a change of model each verified answer is embedded again. Admin → AI model and Admin → Health and logs say how matching works now.
3. By words, when no embedding model is available (or “No AI model” is chosen, or the embedding model is not answering, in which case it is tried again every five minutes): the questions must share enough of their words (a score of at least 0.45), after every name Silver Bullet knows (chat names, other names, measures, rule words, values in the data) is folded into one word, and periods into their dates.

A match is used only if the anchors agree (next section). The answer then carries the badge Verified answer · checked by you on 3 Oct, and, when the wording differed, Answered from: “…”. A period such as “last quarter” is read again for today, so a verified answer does not freeze on the day it was checked.

The anchors check

Two questions can be close in meaning and still ask for different numbers. Embeddings put “net sales in FY 2024-25” and “net sales in FY 2025-26” at about 0.93, well above 0.85. So before a verified answer is reused, both questions are read by the same parser and must name the same specifics:

A specific that only one of the two questions names is not a conflict: a paraphrase may leave it unsaid. Why it matters: without the polarity anchors, “paid on time” and “still unpaid” both matched “paid in full, but later than their due date” at 0.92 to 0.94. A wrong match is worse than no match, because it would carry the Verified badge.

3. Examples given to the model

When a question still has to go to the AI model, a verified reading is given to it as an example only when the two questions are nearly the same:

• a similarity of at least 0.78 by meaning (0.38 by words), close to the 0.85 needed to match outright, so in practice a paraphrase just below the match, or the same question for another period or value;
• about the same thing: both name at least one same column or business measure (by any of its names), and they are not on opposite sides of a yes/no (“paid on time” / “still unpaid”);
• at most two examples.

They are given as examples of how this business words things (checked readings of other questions, not answers to copy), with the instruction to take this question’s periods, numbers, values, groupings, ranking and kind of figure from its own words. Their periods are read again for today.

Earlier, examples went to any question similar at 0.60 (0.20 by words), up to three, as “approved readings of similar questions”. On the Test Kit that made six right answers wrong after a few approvals (below); the rules above, with the model check, made none wrong.

4. Corrections

1. You tap No, fix it, say what went wrong, and pick what you meant, or press None of these. Keep a note.
2. The corrected answer is in use for you at once, marked Not yet reviewed. If you ask the same question again, you get your correction again.
3. The correction is listed under Admin → Your corrections, with what you tapped, what Silver Bullet assumed, and the fixes it suggests.
4. You pick a fix and press Run the safety check (the safety replay).
5. Make it a verified answer saves the fix: from then on it is used for every question it fits, including the same question in other words. Or press Dismiss: answer was fine, and nothing changes.

5. Measured, in the product

Your corrections shows four numbers for the current month, each beside last month’s:

What you check, and what each fix does

Your corrections suggests fixes from what you tapped. You may edit the question, word or period before saving.

You can also press Mark as verified on any answer, and Remove verified to stop using one (it is kept in the history). Marking runs the same safety replay.

Every verified answer in one place: Admin → Your corrections (Review queue on a server) → Verified answers. Each shows its data source, who checked it and when, and how often it has been used. Open asks it again in the chat, answered from the verified reading on today’s data; Remove verified stops using it, with a confirmation; removed ones stay under Show removed. Search and a data-source filter help when the list is long. (Waiting and Done list only “No, fix it” corrections.)

Rules and words can also be written directly in the data dictionary; there they are drafts until you press Publish. The safety replay belongs to Your corrections and to Mark as verified.

Answers that need their own query

Some wrong answers are not a misread word or period: the question needs a calculation, a join or a condition Silver Bullet does not know. I will write the right query myself under No, fix it keeps the question, how Silver Bullet read it, its query and your note under Your corrections, where you (or someone who knows the database, sitting with you) write the query the question should have had. That query teaches Silver Bullet the right lesson:

1. It runs through the same safety layers as every question (the read-only login, the safety check, the time and row limits) beside Silver Bullet’s answer, with the differences marked. Nothing in the database can change.
2. It is in use for you at once, marked Verified by you, for that question; for every way of asking once you publish it after the safety replay.
3. The general lesson first. Silver Bullet compares the two queries and proposes what kind of fix it is: a business measure (a calculated figure, or “count each order once, not each line”), a rule (leave out cancelled orders; periods apply to another date), a saved view (tables joined, then used by the chat like one more table), the right reading, or the query itself (this question only). Each is tried on the question first. The safety replay reads every verified and recent question with the measure, rule or view applied before it is saved.
4. Changeable parts. You tick the parts of the query likely to vary: the period, a value such as a region, a top N. Questions that mean the same with another period or value (“for FY 2024-25”, “only West”), and short follow-ups (“the same for last month”), reuse the query, filled in by Silver Bullet with values it writes itself, never with the question’s text.
5. Kept correct. Every night each expert query and saved view runs again through the safety layers. A failure, or a result whose columns changed, marks it needs attention: it appears under Your corrections, and its answers say Please check until a later check finds it fine. These queries are also regression questions: their question must still be answered by them, the safety replay runs their query again with any fix and compares the figures with those checked, and the nightly check notes when they moved.
6. Examples for the question → SQL path. A published query is given to the model as an example only when a new question is nearly the same (the same closeness and relatedness rules as the readings’ examples), and only to the step that writes a query for questions the choices cannot express. The model’s choices prompt is unchanged.

The safety replay and the model check

Before a fix is saved, Silver Bullet reads again, with and without the fix:

• every verified question on that data source, and
• the questions typed on it recently (the last 90 days, at most 200).

It does this in a sandbox (a copy of the vocabulary and verified answers with the fix applied), so nothing is saved, no AI model is asked and no database is queried. Each question is read as it would be today: verified answers first, then the parser with the rules, else the reading kept from when the question was first answered. Two readings are the same when the SQL written from them is the same.

The result reads, for example, 12 replayed, 11 same, 1 changed. The change is the question being fixed. Show what changes lists each changed question, before and after.

The model check covers what the replay cannot see: a question only the AI model reads. Its answer can still change when the fix changes what the model is told (an example, a rule, a word). With Check with the model too (on by default whenever an AI model is answering), Silver Bullet:

1. builds, for every recent question only the model reads, what the model would be told with and without the fix. Where it is exactly the same, the answer cannot change, and the model is not asked. (Most fixes leave most questions untouched: on the Test Kit, 1,007 of 1,047 such checks needed no model at all.)
2. asks the model the others (at most 40, those about what the fix touches first, then spread across tables), with and without the fix, under a time budget of 5 minutes. Nothing is run against your data.
3. asks again, without the fix, any question whose reading differed, so that a question the model reads two ways on its own is listed as the model varies rather than blamed on the fix.

The page says beforehand how many questions it will ask and about how long it will take. K answers would change stops saving exactly like the replay’s changes, unless you accept them.

Make it a verified answer runs the replay again (and reuses a model check of the same fix from the last 30 minutes). If anything other than the question being fixed would change, saving stops and lists the changes; you save only by pressing Accept these changes and save, after checking that every change is right.

Measured results

Matching by meaning was calibrated on 16 questions (14 from the Test Kit and two short ones), with 39 paraphrases of them and 46 different questions, on PostgreSQL 18 with a data dictionary like a real installation’s:

Paraphrases scored at least 0.807 against their own question (median 0.922); the different questions that passed the anchors scored at most 0.723. The two paraphrases missed (two wordings of “monthly net sales” at 0.82 to 0.85) went to the model, with the verified answer as an example.

Gemini’s gemini-embedding-001 (measured 10 October 2026 with the same questions and the same method: SB_TEST_EMBED=gemini:gemini-embedding-001 go test -tags integration -run Calibrate ./internal/learn/) places questions on another scale: paraphrases scored at least 0.882 (median 0.947), the different question that passed the anchors 0.791. The threshold 0.87 sits between them (every paraphrase matched, nothing else did), and its example floor is 0.82 (nomic’s 0.78, scaled the same way). nomic’s 0.85 would have worked for Gemini on these questions too, but with less room above the different question. OpenAI’s text-embedding-3-small and Mistral’s mistral-embed are not measured: they are not chosen by themselves (the local nomic-embed-text is used), and when you name one it is used at the 0.85 threshold.

Without an AI model (“No AI model”, measured 10 October 2026 with go run ./tools/testkit answer -model none, results in tools/testkit/answer-results-none.json): Silver Bullet answered 10 of 205 Test Kit questions on the five databases and 1 of 29 on the files by itself, every one a request to change data, refused as it should be. The built-in reader read none of the kit’s other questions whole: they are written as an analyst asks them, with exclusions and definitions in the sentence ("…excluding cancelled orders? Use the region of the branch that fulfilled the order."), which the reader leaves to the model; each got the note of SB3-4004 with questions it can read. With a model, the same questions were answered right 199 of 205 times (Gemini) and 189 of 205 times (the local qwen3-coder-next). The reader answers the home page’s cards and questions said like them (“net sales by region last quarter”).

The learning loop on the Test Kit: every question of the Test Kit was asked on the five test databases and on the kit’s files with the local model; wrong answers were corrected through “No, fix it”, the review of corrections, the safety replay and the model check; four right answers per database (and on the files) were also marked as verified (readings that become examples for nearly the same questions); then the corrected and marked questions and their paraphrases were asked again from a second test account, and every question was asked once more. Every answer that changed from right to wrong was put down to its cause (the model was told something else, or read the same thing differently).

Measured on 3 October 2026 (release 3.0.0-rc.1, qwen3-coder-next and nomic-embed-text; tools/testkit/learn-results.json and learn-results-2.json), twice:

• Each correction’s safety replay read 37 to 40 questions again and found only the question being fixed changed; its model check found nothing to ask (a corrected query is never given to the model as an example).
• Of the 15 marks per run, the replay stopped 5 (the mark would have answered a different question) and the model check stopped 8 (for example, marking Q06 would have changed how the model read one or two other questions on each database, and marking Q05, Q06 or Q08 on the sales register file six to eight of its other questions: a small file’s questions are close to each other). Each model check asked the model about at most 8 questions; all the checks of a run took about 10 minutes.
• The files’ three wrong answers (Q03, Q40, Q42) were not corrected in this loop.

The same loop with the earlier example rules (0.60 / 0.20, three examples worded as answers, no model check; tools/testkit/learn-results-old-rules.json): 9 corrected, but six right answers became wrong (Q10 on PostgreSQL, Q26 on MySQL 9.7 and Oracle, Q27 on MySQL 9.7, Q28 on MySQL 8.4, Q25 on the files), each time because what the model was told had changed. This is what the first release candidate’s report saw (three answers there).

These measurements used the local model qwen3-coder-next and the nomic-embed-text embedding model (the matching above also Gemini’s). Cloud models were not measured for this release. Measure on your own data: check a few corrections, then ask the same questions in other words.

Expert queries on the Test Kit (9 October): a DBA’s query for each question answered wrong, saved through the review of corrections and the safety replay, then every question asked again:

¹ Q22 on MySQL 8.4 failed once with “invalid connection”: the test databases were restarted (packaging/demo.sh) during the run. Asked again alone it is right (go run ./tools/testkit answer -only Q22 -half online -targets mysql84), so it is counted right here. ² The files’ question → SQL path takes no expert examples yet (Q03); Q42 is not a query problem: the file wizard set rows aside when loading, so no query on the loaded table can reach the expected figure.

The DBA’s queries are internal/testkit/kit/expert-sql.json (ten questions, each on PostgreSQL, MySQL, SQL Server, Oracle and DuckDB, every one checked against the kit’s expected answer with -check-fixture); the run’s results are in tools/testkit/expert-results/. Run with the local model (qwen3-coder-next), one database at a time: go run ./tools/testkit expert -targets pg18 -files=false -out …, and -targets "" -files for the files.

Limits

Said plainly:

• Nothing becomes a verified answer until you check it. Corrections wait under Your corrections until you decide them. An unchecked correction teaches nothing beyond its own question.
• An unchecked correction applies only to the same question asked again, not to other wordings of it.
• The polarity anchors cover listed word patterns only: paid / unpaid (and “outstanding”, “not settled”…), on time / late (and “overdue”, “delayed”…), and including or excluding a thing (“excluding GST”, “with cancelled orders”). Other opposites, such as “new customers” against “returning customers”, are not caught by polarity; they are kept apart only if another anchor (a value in the data, a measure) differs. The word patterns are English.
• A paraphrase below 0.85 does not match. It goes to the AI model, with the verified answer as an example only when it is at least 0.78 and about the same columns, and may then be read differently.
• Matching is per data source. A verified answer on one source does not answer questions on another.
• The model check covers recent questions only. It asks the model about the recent typed questions (90 days, at most 200 per source) whose instructions a fix changes, at most 40, within 5 minutes; others are listed as not checked. A question nobody has asked recently is not checked, and a model can still read a question differently from one day to the next on its own. Turn the model check off only when no AI model is available (the replay alone then cannot see questions only the model reads). Re-run your own regression questions after a batch of checks.
• The model check says what would change, not whether it is right. It stops a fix that changes other answers; you decide whether the new answers are right before accepting them.
• The replay covers verified questions and recent typed questions (90 days, at most 200 per source), not every question ever asked.
• Without the embedding model, matching falls back to words, which matched 29 of 39 paraphrases instead of 37. If the embedding model stops answering, Silver Bullet uses words for five minutes, then tries the model again. The AI model page says whether the embedding model is installed.
• A verified answer keeps its choices. If the data changes shape (a column renamed, a table left out), a verified answer that relies on it may no longer work; remove it and check a new one.
• No model is trained, so Silver Bullet does not get better at questions you have not corrected or verified, beyond what the dictionary, measures and rules already teach it.