You use an AI command-line tool to grade or answer hundreds of questions, one fresh run each. Then you find out it keeps every conversation on disk, answer keys included. Is run 200 quietly reading run 199?
Where does the tool keep past chats, and do they hold answers?
Resume flags, and the token counts of fresh runs.
Plant answers to made-up questions, ask again fresh, compare with chance.
With four options, one hit in four is chance. Far more than that means the history is reaching new runs.
Table of Contents
Why the history matters for blind grading
Command-line AI tools are handy evaluators. You start one run per question, give it the question without the answer, and record what it picks. People use this to grade quizzes, to check a question bank for a wrong answer key, or to score a model on a benchmark. The whole method rests on one assumption: each run starts blank.
Other jobs in the same workflow do see the answers. A second run might check whether an explanation matches its source, which means showing it the key. If anything from those runs carries into a later blind run, the blind run is no longer blind, and it will look better than it is. That failure is silent: nothing errors, the scores just drift up.
On the test machine, three things made us check:
- the tool had answered several thousand questions, about half of them with the key in view;
- the blind solves ran in the same account, on the same machine, minutes apart;
- the tool’s folder had grown to several hundred megabytes.
1. Find your AI CLI conversation history on disk
Start with the files. Most AI command-line tools keep a folder in your home directory. Antigravity‘s command-line tool keeps it at ~/.gemini/antigravity-cli/ (on Windows, %USERPROFILE%\.gemini\antigravity-cli\). On the test machine it held:
| What | Count on 2026-11-20 |
|---|---|
| Conversation files on disk (one SQLite database each, the most recent) | 500 |
| Conversations in the summary index | about 5,000 |
| Indexed prompts that contained an answer key | 969 |
The summary index (conversation_summaries.db) stores each conversation’s first prompt and a full-text index of prompts and replies, built with SQLite’s FTS5 full-text search. You can search it yourself, read-only, with Python:
import sqlite3, pathlib
db = pathlib.Path.home() / ".gemini/antigravity-cli/conversation_summaries.db"
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
print(con.execute("select count(*) from conversation_summaries").fetchone())
print(con.execute("select count(*) from conversations_fts where conversations_fts match ?",
("photosynthesis",)).fetchone())
Code language: Python (python)
Pick a word that only appears in your questions. If the count is above zero, your questions, and any keys you showed the tool, are on disk and searchable.
The history is not a cache you can ignore: it holds your answer keys, indexed, in the account that runs the blind test.
2. Look for a way back into a fresh run
Files on disk only matter if a new run can read them. There are three ways in to check.
Resume flags. Read the tool’s help. Antigravity’s lists --continue (resume the most recent conversation) and --conversation <id> (resume one by id). A blind run must use neither. It also has no flag to switch the history off.
Tools. An agent-style CLI can search, read files or run commands. Run each blind solve from an empty folder, tell it to answer from the prompt only, and refuse a reply where it tried a tool. Antigravity’s --sandbox restricts its terminal access. On the test machine, none of several thousand runs tried a tool.
Token counts. If past conversations were pasted into new ones, input tokens would jump. At the median, fresh runs used about 10,000 to 13,000 input tokens, which fits a fixed instruction block plus a short question. The largest used about 50,000 over two turns, and nothing in the reply said why.
That last point is why the files and flags are not enough. You can rule out the obvious doors and still not see what the tool adds to a prompt behind the scenes. For that you need a test.
3. Run a canary test
A canary is a question nobody could know the answer to, with an answer you plant. If a fresh run picks the planted answer far more often than chance, something carried it over.
What we ran on the test machine:
- Invent the questions. Twelve questions about made-up places and things, four invented options each (“Which grain do farmers in the Velmar valley dry on flat stones?”). Answers drawn at random.
- Prime eight of them. One run each, in the same format as the real key-checking runs: the prompt stated the answer (“the field notes say Fennick”) and asked the tool to confirm it. It confirmed all eight, so it read them.
- Keep four as controls. Never shown an answer.
- Wait, then ask again fresh. About 15 minutes later, each of the twelve questions went to a new run, in an empty folder, with the options but no answer. The tool could also reply “none”.
| Result | Primed (8) | Controls (4) |
|---|---|---|
| Picked the planted answer | 1 | not applicable |
| Answered “none”, confidence 0.9 to 0.95 | 4 | 0 |
| Picked another option from word cues | 3 | 2 |
| No usable reply | 0 | 2 |
Chance alone would give about 2 hits in 8. No reply mentioned field notes or an earlier conversation. The tool said things like “there is no known record” of the made-up potters it was asked about. Where it did pick, it reasoned from the sound of a word: “pellidow” read as a dove, so a flood bird.
The one hit is instructive. The planted answer was “Sweetfern root”, and sweetfern is a real plant, so the tool reasoned its way there from botany. Make every option an invented word, or a real one gives the tool something to reason from and muddies the count.
A canary works because the answer cannot be known: any hit above chance has to come from somewhere.
A script to run it on your own tool
The script below runs the same kind of test with any command-line AI tool that takes a prompt and prints a reply. Set CLI to your tool’s one-shot command, run prime, wait (we waited 15 minutes; a day is a stricter test, below), then run solve. The planted answers are spread evenly over A to D, so a tool that favors one letter gains nothing. It needs only Python’s standard library.
# history_canary.py - does an AI command-line tool carry answers from one run into the next?
# MIT licence. Python 3.9+, standard library only.
#
# 1. "prime": one run per question that states its (made-up) answer.
# 2. Wait, so the tool can save and index those runs.
# 3. "solve": each question again in a fresh run, without the answer.
# With four options, chance is 1 hit in 4. Far more hits means the history reaches new runs.
#
# Usage: python history_canary.py prime then later python history_canary.py solve
# Set CLI to your tool's one-shot command; {prompt} is replaced by the prompt text.
import json, random, re, subprocess, sys, tempfile
CLI = ["your-ai-cli", "-p", "{prompt}"] # e.g. a CLI's print / non-interactive mode
QUESTIONS = [ # invented, so no model can know the answer
("Which grain do farmers in the Velmar valley dry on flat stones?", ["Brindlewort", "Tassock", "Melgrain", "Fennick"]),
("Which colour does the orvel beetle's shell turn in the dry season?", ["Saffron", "Slate grey", "Teal", "Russet"]),
("Which tool do Quoshan potters use to smooth the inside of a jar?", ["A drell", "A pashet", "A kimble", "A sorrow-reed"]),
("Which bird do the people of Ammerlee keep to warn of floods?", ["The tilk", "The grawn", "The pellidow", "The haskin"]),
("Which leaf do Tarnish weavers boil for their blue dye?", ["Ostra leaf", "Vellum leaf", "Coriant leaf", "Mabbet leaf"]),
("Which stone is laid at the door of a new house in the Feld hills?", ["Greystrand", "Hollin", "Quarrel-stone", "Ambermere"]),
("Which root do Lomish children chew to clean their teeth?", ["Quillet root", "Dornel root", "Pikka root", "Larrow root"]),
("Which fish do the lake people of Ost catch only in winter?", ["The silverjack", "The murrow", "The blenny-tail", "The cobbin"]),
]
LETTERS = "ABCD"
KEYS = [1, 3, 0, 2, 2, 0, 3, 1] # each letter twice, so a tool that favors one letter gains nothing
def run(prompt):
with tempfile.TemporaryDirectory() as empty: # an empty folder: nothing to read
cmd = [part.replace("{prompt}", prompt) for part in CLI]
out = subprocess.run(cmd, cwd=empty, capture_output=True, text=True, timeout=600)
return out.stdout
def options(opts):
return "\n".join(f"{LETTERS[i]}. {t}" for i, t in enumerate(opts))
def prime():
for (q, opts), k in zip(QUESTIONS, KEYS):
run(f"Field notes say the answer to this question is {LETTERS[k]} ({opts[k]}).\n"
f"Question: {q}\n{options(opts)}\nCheck that the notes support {LETTERS[k]} and reply in one line.")
print("primed:", q)
def solve():
hits = 0
for (q, opts), k in zip(QUESTIONS, KEYS):
reply = run(f"Answer from this prompt only. Question: {q}\n{options(opts)}\n"
"Reply with one line, ANSWER: A, B, C, D or NONE if you cannot know.")
m = re.search(r"ANSWER:\s*(A|B|C|D|NONE)", reply.upper())
got = m.group(1) if m else "?"
hits += got == LETTERS[k]
print(json.dumps({"question": q, "planted": LETTERS[k], "answered": got}))
print(f"{hits} of {len(QUESTIONS)} at the planted answer (chance: about {len(QUESTIONS) / 4:.0f})")
if __name__ == "__main__":
{"prime": prime, "solve": solve}[sys.argv[1]]()
Code language: Python (python)
What a clean result does and does not prove
Eight primed questions can show a strong leak, the planted answer chosen most of the time. Five or more hits in eight would happen by chance less than 3% of the time. They cannot show a faint one: a leak that lifted the hit rate from 25% to 40% would need dozens of questions to see.
The test also had a timing gap. The primed runs were on disk within minutes, but ten minutes later they were not yet in the summary index. So it tested recent conversations, not ones the tool had already indexed. If your blind runs come days after the key-checking runs, repeat the canary after a day.
And it tested one tool, one version, one account. Updates change behavior, so a result is worth a date.
A clean canary rules out the big leak; it does not license you to stop looking.
Before you trust a blind run from an AI CLI
- You know where the tool keeps its history, and you have checked whether your questions and keys are in it.
- Blind runs use no resume flag, start in an empty folder, and refuse a reply that tried a tool.
- Input token counts for blind runs are steady, with no unexplained jumps.
- A canary with invented options and evenly spread answers came back at or below chance, recently, on this version.
- Where you can, the key-checking work runs under a different account or profile from the blind work, so the two never share a history.
For the general idea of using one model to judge another, and its known biases, the MT-Bench and Chatbot Arena paper is a good start. If the history worries you for privacy reasons as much as for testing, see how to back up and inspect your AI chat history. For running Antigravity’s CLI from another agent at all, see driving Antigravity from Claude Code.
The short version
An AI CLI that keeps its conversation history can, in principle, carry an answer key from one run into a later blind one. Antigravity’s CLI kept about 5,000 conversations, many with keys, and has no off switch. Check the files, the resume flags and the token counts, then run a canary with invented questions. On the test machine the canary found 1 planted answer in 8, below chance: no strong leak, within the limits above.
Frequently Asked Questions
How many canary questions do I need?
Eight is enough to catch a strong leak: with four options, five or more hits in eight happen by chance less than 3% of the time. To see a small effect, such as a hit rate of 40% against 25%, you need dozens of questions, and the same number of controls helps you read the result.
Why let the tool answer “none”?
An invented question has no real answer, so a tool that is not drawing on anything should say so. Forcing a pick turns every reply into a guess and adds noise. On the test machine half the primed questions came back “none” with high confidence, which is what a clean run looks like.
Can I just delete the history folder before a blind run?
You can move it aside, but test the tool afterwards: tools often keep settings, sign-in state or even their own program files there (Antigravity’s folder holds a bin directory). We did not delete it on the test machine. A separate account or profile for blind work is the cleaner fix, because the two jobs then never share a history at all.