What question does this note answer?
Data teams are asked to move years of warehouse logic to a new platform: stored procedures, ETL packages, scripts and schedules. AI coding assistants can draft that code quickly. The question for the people who approve the migration is different: can the result be reproduced and proven?
This note reviews what published research and model vendors say about one property of large language models (LLMs): whether the same request gives the same code. It then sets out what that means for migration evidence.
Method. We reviewed a peer-reviewed empirical study of ChatGPT code generation, the reproducibility documentation of two model providers, and a 2025 engineering analysis of why LLM inference varies. We quote each figure from its source. This note uses no Novuz customer data. Where it describes how Novuz works, it says so.
How often does an LLM give different code for the same prompt?
Often. Ouyang, Zhang, Harman and Wang studied ChatGPT on 829 code-generation problems from three standard benchmarks: CodeContests, APPS and HumanEval. They sent the same prompt several times and compared the code that came back, including how it behaved on each problem’s tests.
They report that the share of problems with zero equal test output across requests was 75.76% for CodeContests, 51.00% for APPS and 47.56% for HumanEval. In plain terms: for three in four of the hardest problems, no two answers to the same request behaved the same way on the tests.
They also tested temperature, the setting that controls how much randomness the model uses. Their conclusion: setting the temperature to 0 does not guarantee determinism in code generation. It lowers the variation compared with the default, but it does not remove it.
Problems where no two requests produced code with equal test output
75.76%
CodeContests problems
51.00%
APPS problems
47.56%
HumanEval problems
Source: Ouyang, Zhang, Harman and Wang, An Empirical Study of the Non-determinism of ChatGPT in Code Generation (829 problems).
These are general coding benchmarks, not warehouse migrations. They show the property of the method, not the error rate of any migration.
What do the model vendors say?
The vendors are clear that their models are not guaranteed to repeat themselves.
OpenAI introduced a seed parameter in November 2023 so that developers could get “mostly deterministic” outputs. Its own guide says that “determinism is not guaranteed”. It adds that a system fingerprint changes when OpenAI updates its numerical configuration, and that “there is a small chance that responses differ even when request parameters and system_fingerprint match”.
Microsoft’s documentation for Azure OpenAI says the same in stronger terms: “Determinism isn’t guaranteed with reproducible output.” Even when the seed and the system fingerprint match, it is “not uncommon to still observe a degree of variability in responses”. Longer responses tend to vary more.
Why does the same request give different code?
Temperature is only part of the story. A widely read 2025 analysis from Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference”, looked for the root cause. It argues that the common explanation, parallel floating-point arithmetic finishing in a random order, is not the main one.
Its finding is that the result of a request depends on how many other requests the server is processing at the same moment. Server load changes the batch size, and the arithmetic inside the model is not invariant to batch size. The same prompt can therefore produce different numbers, and different tokens, depending on who else is using the service.
The same analysis shows that this can be fixed. With batch-invariant kernels, the authors made an open model (Qwen3-8B, served with vLLM) deterministic. The lesson for a migration team is practical: determinism is achievable when you control the whole inference stack. It is not something you can assume from a shared, hosted API.
| Source of variation | What it changes | Can a migration team control it? |
|---|---|---|
| Sampling temperature | How much randomness the model uses when picking each token | Yes. Set it to 0. This reduces variation but does not remove it. |
| Server load and batch size | The arithmetic inside the model for the same request | Only on a self-hosted stack with batch-invariant kernels. Not on a shared API. |
| Model updates | The model or its numerical configuration, sometimes silently | Partly. Pin model versions where offered and log the system fingerprint. |
| Prompt and context | What the model sees: instructions, examples, surrounding code | Yes. Version every prompt and record the context sent. |
| Output length | Longer outputs drift more, according to Microsoft’s guidance | Partly. Convert in smaller units and check each one. |
Why does this matter for a migration?
Because change control asks two things of every change: show what changed, and show you can produce it again. A warehouse migration is thousands of changes at once.
If a conversion can be rerun and gives the same output, any object can be regenerated, compared and signed. If it cannot, each converted object is a one-off. It must be tested on its own, and tested again whenever it is regenerated. The testing becomes most of the cost, and an auditor has nothing stable to check against.
There is a second, quieter risk. LLM-written code usually compiles. The failures that matter are behavioral: a changed join, different NULL handling, a rounding rule, a branch that did not run in testing. These pass a code review and show up later, in a report someone trusts.
- Reproducibility
- Can the same input be converted again, with the same result, for an audit a year later?
- Review cost
- If two runs differ, every line of every run needs review. If they match, review once.
- Evidence
- What proves the converted object gives the same numbers as the original, on the same inputs?
- Accountability
- Who approved each piece of code a model wrote, and is that approval recorded?
What can a team check before trusting AI-converted code?
Five checks work for any conversion method, including a hand rewrite. They need no special tools.
- The two-run test. Convert the same object twice and diff the outputs. An empty diff means a rule produced it. A non-empty diff means every output needs its own review.
- A record for every model-written change: the model and version, the prompt, the parameters, the system fingerprint where offered, and the engineer who approved it.
- Reconciliation on the same inputs. Run the original and converted object on one frozen input and compare schema, row counts, column totals and sampled rows. Mark each object pass, fail or review.
- Inputs that exercise the risky branches: NULLs, ties, duplicates, mixed case, boundary dates and error paths. A test that never runs a branch proves nothing about it.
- A written counting rule. If someone reports that a share of objects was “automated”, ask how a one-line fix is counted and whether dead code is in the total.
Where does AI belong in a migration?
At the edges, with approval. The research does not say AI is useless for code. It says AI output varies, and that the variation must be controlled and proven.
A sensible split uses fixed rules for the bulk of the code, where patterns repeat, and a model for the cases the rules do not cover. Every model proposal is reviewed by an engineer, and every object is reconciled whichever way it was produced.
This is the design Novuz uses, stated here so readers can weigh this note accordingly. About 95% of objects convert by fixed, versioned rules with no model call, so two runs give an empty diff. The rest go to a resolver on a model the customer approves, including a local model. An engineer approves each proposal before it is used, and every object gets a reconciliation report.
What are the limits of this note?
The figures above come from general coding benchmarks and from vendor documentation, not from warehouse migrations. They show that non-determinism is a real property of hosted LLMs. They do not measure how often AI-converted warehouse code is wrong; we know of no public study that does.
Models and serving stacks change quickly. Some providers now offer more control over determinism, and self-hosted stacks can be made deterministic, as the 2025 analysis shows. Check the current documentation of any model you plan to use. The checks in this note stay useful whatever the model does.
Sources
- Ouyang, S., Zhang, J. M., Harman, M. and Wang, M., “An Empirical Study of the Non-determinism of ChatGPT in Code Generation” (also published as “LLM is Like a Box of Chocolates”), arXiv:2308.02828
- OpenAI Cookbook, “How to make your completions outputs consistent with the new seed parameter”, 6 November 2023
- Microsoft Learn, “How to generate reproducible output with Azure OpenAI”
- Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference”, September 2025, as summarized by Simon Willison, 11 September 2025