A research note · DAOS
Asking twice
What it takes to get the same answer from an AI model.
We built DAOS to make an AI model give the same answer every time. It works, within narrow limits. This is what we measured, what did not pay off, and when you do not need any of it.
The problem
Same question, different answers
Ask an AI model the same question twice and you may get two different answers, even with the randomness turned off (temperature=0). When Thinking Machines asked one large model the same question 1,000 times at temperature 0, they got 80 different answers. All of them began with the same 102 tokens (pieces of words) and then went their own way [1].
A study of AI models used as graders found the same thing with GPT-4o, Gemini and Claude: identical inputs, different scores, even at temperature 0 [2].
Why it happens, even with the randomness off:
- 1a busy server processes your request together with other people’s, and the size of that batch changes the arithmetic (the main cause found in [1])
- 2a GPU adds numbers in a different order, and tiny rounding differences add up
- 3a smaller, compressed copy of a model is really a slightly different model
- 4a hosted model can be updated without notice
Why it matters: if an answer can change, you cannot prove what the model said, you cannot compare two prompts fairly, and a wrong answer can come and go.
First finding
On one machine, turning the randomness off was enough
Before building anything, we measured. A small open model (Llama-3.2, 1 billion parameters) on one laptop chip (Apple M3 Pro), served by LM Studio:
runs of the same request, with four different seeds: one single answer
runs with another request in flight at the same time: identical to the quiet runs
So on your own machine, with one model, temperature=0 and a plain cache may be all you need. The 80 answers in [1] came from a large shared server, which is where most people are when they use a hosted model. Our own concurrency test only overlapped requests on the wire; it did not reproduce a busy server’s batching.
What we built anyway
Stable is not the same as provable
A stable answer is still just a claim. We wanted three more things: proof of what the model said, a way to replay it, and a way to fix one wrong answer without touching the rest. That is DAOS, a Python library:
1. Ask once. You send the input with a schema, the exact shape the answer must have: fields, allowed values, number precision. An answer that does not fit is an error, never passed on.
2. Tidy. Two answers that mean the same thing should be the same bytes: a raw 0.8500000119 becomes 0.85, text is normalised, fields are put in a fixed order.
3. Sign and store. The answer is signed, so an edit is caught, and stored under a key that includes the exact model file and its settings. Swap the model and the old answers are not used.
4. Replay. With the log turned on, every run is recorded in a signed log. A logged run replayed in a fresh process, with a new signing key, gave back the same bytes with zero model calls.
# read back a logged run: the signature checks out read_run("ticket-01...") → MAC-verified ✓ # someone edits the stored answer by hand (billing → refund), read again: → IntegrityError: event 'ticket-01...'/1 failed HMAC verification
The rule that limits everything
The input has to be exactly the same
DAOS recognises a request only if it is the same byte for byte, after evening out the order of fields, the unicode form and the number format. A different capital letter, an extra space or a typo inside the text is a new request: a new model call, and possibly a different answer.
We could match “similar” requests instead. We chose not to: deciding what counts as similar is itself a judgement that can change, and it would bring the randomness back through the side door.
So DAOS fits records you already hold and process again, such as a stored document, a ticket in your database or an eval set. It does not fit things people type.
# the same request: only the layout of the JSON differs {"question":"Refund?","plan":"annual"} { "plan": "annual", "question": "Refund?" } → same request, stored answer # a new request: one capital letter differs inside the text {"question":"refund?","plan":"annual"} → new request, new model call
The experiment that did not pay off
Three copies of one model, voting
Could several copies of a model vote their way to an answer that is both stable and better? We ran the same model at full precision and compressed to 8-bit and 4-bit, and had them vote on each field of the answer.
Voting on the priority field
Over 40 fields, the full-precision copy was never the only one to disagree; the 8-bit copy was, 2 times; the 4-bit copy, 7 times.
fields where the copies split two to one; the majority still gave one consistent answer
labelled tasks used to check accuracy: sorting customer requests into 150 intents
Consistent, yes. Better, no. On the 600 tasks the vote scored 33.2%; the full-precision copy alone scored 33.7%. The gap is too small to call either way, but there is no gain. The reason: compressed copies of one model make the same mistakes. Two copies that were both wrong gave the same wrong answer 91–99% of the time, and 380 of the 600 tasks were wrong on all three. A majority of voters who fail together just repeats the failure.
Consistent is not correct. Voting only helps when the voters make different mistakes. That is why the vote stayed a research mode and never became part of the library’s main path.
Ideas borrowed from neuroscience
In the specBrains also get one clear thought out of noisy parts
Neurons are noisy and unreliable, yet a brain settles on one thought. The DAOS spec borrows three tricks that make that possible, and deliberately rejects a fourth. None of them is built. One limit has been measured.
Neurons settling on one state → pooled answers
Let the copies agree before picking each word
A group of neurons given a noisy signal falls into one stable pattern, and cleans up the noise on the way. The spec applies the same idea to copies of a model: at every word, combine how likely each copy finds each option, pick one word together, and move on in step.
Not built. It needs every copy running inside one program, word by word, which common model servers do not allow.
Many neurons voting → a limit on voting
Voting only helps if voters make different mistakes
Adding up many neurons cancels noise, but only when each one makes its own random errors. If they go wrong together, adding more does not help. Compressed copies of one model go wrong together, so adding more of them stops helping quickly.
Measured: the copies were wrong together almost every time (correlation up to 0.978). This is why the vote did not improve accuracy.
Expectation → a stated preference
Break a near-tie with a stated preference
Faced with something ambiguous, the brain leans toward what it expects. The spec breaks a near-tie between two words the same way: the schema can say which value to prefer. It deliberately does not use “whatever we answered last time for something similar”, for the same reason as the exact-input rule.
Not built. Without a stated preference, the spec falls back to a fixed rule.
The one we rejected: making up a story
The part of the mind that invents a reason
Studies of split-brain patients show the mind inventing a convincing explanation after the fact, and believing it. That is exactly what DAOS must not do: giving the same answer every time does not make the answer trustworthy.
Rejected on purpose: a system that invents its own certainty is the failure, not the feature.
Wrong answers repeat faithfully
So we added a check and a fix
A stored answer is repeated just as faithfully when it is wrong. Two tools deal with that.
A daily check
Once a day it takes the new answers, plus a rotating share of older ones, and asks a stronger model (Claude) and a web search whether they hold up. It withdraws an answer on its own only where your rules say so; everything else goes into a report for a person to review. The check is paid, and capped by a budget you set.
Correcting one answer
A wrong answer can be withdrawn, or replaced with a fixed one, without touching anything else. Until a fix is in place, that request returns a clear error instead of the wrong answer. Who changed it, and why, is recorded with the change.
# check stored answers every day (03:30 by default) uv run daos-inspect schedule install --config inspector.toml # withdraw or replace one stored answer, saying why uv run python scripts/admin/quarantine.py apply daos-cache.sqlite KEY \ --reason "wrong capital" --replacement-file fixed.json ...
When you need this
When it is worth it, and when it is not
- You process the same records again and again (documents, tickets, a catalogue) and must be able to prove or correct any single answer.
- You use an AI model as a grader, and a score must only move when the answer or the grader changes.
- Someone may ask, months later, exactly what the model decided, and whether it was changed since.
- Chat, or anything people type: the wording is never exactly the same twice.
- One model on your own machine, where temperature=0 and a plain cache already give one answer.
- You want better answers: DAOS makes them stable, not right.
- You use a hosted model: today DAOS only drives local models, through LM Studio.
Open questions
What we have not answered
Copies built differently
Would copies compressed with different methods (MLX, GPTQ, AWQ) make different mistakes, and make voting worth it? Not measured yet.
Noticing that a model changed
Replay reuses the stored answer, so a model that now answers differently still replays fine. Detecting that drift is designed, not built.
The same answer on other software
DAOS promises the same answer for one model file with the same settings. Moving to other inference software is not covered; batch-invariant GPU code [1] might change that.
Acting on near-ties
In research mode DAOS can measure when the model almost picked a different word. Doing something about it, such as sending the answer for review, is not built.
What we take away
Stable is cheap. Provable is the hard part. Correct is a different problem.
On one machine, turning the randomness off already gives one answer. What DAOS adds is proof, replay and per-answer correction, and only for inputs that come back exactly the same. No amount of agreement makes a wrong answer right.
Setup and sources
Measured in July 2026 with Llama-3.2-1B-Instruct in three sizes (F16, Q8_0, Q4_K_M), in LM Studio on an Apple M3 Pro. Accuracy: 600 tasks from the CLINC150 set. Design: DAOS specification v1.6. The source repository is private.
- Horace He, Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, September 2025.
- Fiona Lau, Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge, arXiv 2603.04417.
DAOS · Deterministic Agent Output System
A research note · spec v1.6