GEPA improves a prompt by watching it fail on a few examples and asking a language model to rewrite it. It picks those examples at random. I spent a few months trying to pick them better. Nothing I tried beat random.
The more useful thing I found is about the step after the rewrite. GEPA keeps a rewrite only if it beats the old prompt on the same three examples. Judge the same rewrite on two different sets of three, and the two verdicts agree about as often as two independent verdicts would. On IFBench, the rewrites that pass weren't detectably better on average, either: on 150 validation examples they beat their parent by 0.001 (interval −0.003 to +0.005). At three examples, at least on these two benchmarks, the test mostly measures which examples were drawn and how they happened to score.
Here is one rewrite from my IFBench experiments, the first one in the data rather than a hand-picked one, scored against its parent prompt on two batches of three examples. The rewriting model wrote it after seeing the parent fail on the first batch. Each score is the total over the three examples, from 0 to 3; higher is better.
- rewrite
- 1.08
- parent
- 1.33
thrown away
- rewrite
- 1.00
- parent
- 0.75
would have been kept
The left panel is the test GEPA applies, and it threw the rewrite away. On a batch it had never seen, the same rewrite would have passed. Same rewrite, same parent. The only thing that changed is which three examples it was judged on.
This is not a rare case. Across 2,292 rewrites scored this way, the two verdicts agreed 53.9% of the time. If the verdicts had nothing to do with each other, you would expect 54.4%.
That is where this project ended up. It started somewhere else.
How GEPA works
GEPA (code) improves a prompt by trial and error, with a language model doing the editing. It keeps a pool of prompts, starting with yours, and repeats one loop:
- Pick a parent from the pool. GEPA favors prompts that are the best on at least one validation example.
- Draw three training examples at random and run the parent on them.
- Rewrite. Show a second model, the rewriter, each example's input, the parent's output, and the feedback, and ask it for a better prompt.
- Keep or discard. Run the rewrite on the same three examples. Keep it only if its total score is strictly higher than the parent's. I'll call this step the gate.
- Validate. Score each kept rewrite on the full validation set and add it to the pool. The parent stays in the pool too.
When the budget runs out, GEPA returns whichever prompt scores highest on the validation set. The budget is counted in metric calls: one call is the task model answering one example and getting scored.
Two details matter later. The same three examples are used to write the rewrite and to judge it. And the judgment is a single comparison of two totals over three examples.
What I was trying to do
Each round, the rewriter learns from three examples picked at random. My bet was that some failures make better lessons than others. A failure whose feedback names a specific fixable problem should teach more than one that just says "wrong." If you could spot those cheaply, you could feed the optimizer better lessons.
I tested that on two benchmarks.
IFBench: nothing to pick
IFBench asks a model to follow instructions with checkable rules ("answer in three paragraphs," "no commas"). Each example is scored by the share of its rules the output follows, and the feedback is a checklist:
Satisfied 2/3 constraints.
✓ [keywords:word_once] satisfied — Include keyword internet in your response.
✗ [keywords:keyword_specific_position] FAILED — Include keyword freedom in the 10-th sentence, as the 26-th word of that sentence.
✓ [keywords:forbidden_words] satisfied — Do not include keywords ['camp', 'catch', 'friend', 'heavy'] in the response.I built five ways to score how useful an example's feedback was. Then I found I had been scoring only the checklist, while the rewriter actually reads the full example: the input, the parent's output, and the checklist. I rebuilt everything on the full text. None of the five predicted how much a batch's rewrite added to the run's best validation scores any better than knowing which rules had failed.
I also tested whether the rewriter's examples matter at all, with what I'll call the swap test. I took a parent prompt at a logged point in a real run and had the rewriter learn from two batches: the parent's own batch, and a different batch where the parent also failed. Then I scored both rewrites on both batches. If the examples matter, a rewrite should do better on the batch it learned from than the rewrite that learned from the other batch. On IFBench it didn't, detectably: +0.005 on the three-example total, with an interval from −0.012 to +0.024.
The most direct test: I deleted the feedback entirely. Across 150 batches, rewrites were kept 34% of the time with the feedback, 31% without it, and 37% with a generic placeholder ("This example's output did not meet requirements."). No detectable difference. The test was designed to detect differences of roughly 10 to 12 points.
The likely reason is simple in hindsight. The checklist is basically a list of which rules failed, and the rewriter can work that out from the input and output it already sees. So the feedback adds little to select on. This does not prove selection can never help on IFBench. It makes one-step effects bigger than these tests could detect unlikely.
One caveat on the deletion test: it reads the same keep-or-discard verdict that, as I'll show below, mostly follows the batch. But all three versions were judged on the same batch against the same parent, so that luck cancels out of the comparison. What limits it is the rewriter's own randomness: with identical inputs, asking for the rewrite twice flipped the verdict 32% of the time. The average margin over the parent didn't move either.
The obvious objection is that checklist feedback is unusually easy to work out. Fair. So I moved to a benchmark where the feedback tells the rewriter something it can't see on its own.
HoVer: a real signal, then one that didn't hold up
HoVer asks a model to find the three Wikipedia articles needed to check a claim, in three search steps. Each example is scored by the share of those articles it finds, and the feedback names the ones it missed:
Correctly retrieved 1/3 gold documents: ['50th Berlin International Film Festival']. Documents remaining to be retrieved: ['Jason Robards', 'Magnolia (film)'].The rewriter can't work out the missing titles from the example alone, and a good prompt here has to be built from specific failures. (One other difference: the rewriter on HoVer was gpt-4.1-mini, where on IFBench it was gpt-4.1. So the comparison between the two benchmarks changes the feedback and the rewriter at once.)
Here, which examples the rewriter sees did matter, at least for one step. In the swap test, on the batch it learned from, a rewrite beat the rewrite that learned from the other batch by +0.027 on the three-example total (interval +0.015 to +0.042). That is about one more needed article found every 12 batches: small, but clear of zero. Whether rewrites beat their parent on the batch they hadn't learned from was too close to call (+0.017, interval −0.004 to +0.037).
So I went looking for which examples to prefer. I screened 108 candidate rules: how new a batch's failures were compared with ones the run had already learned from, how fixable the feedback looked, how hard the claims were, and so on, each summarized over the batch in a few ways. One passed every test I'd set for it, with p = 0.0013 before accounting for the 108 tries. It prefers batches whose failures all look different from the failures the run has already learned from.
I built a proper experiment around it. Three arms, eight paired runs each, same budget (300 metric calls per run):
- Stock GEPA draws three examples, as usual.
- Control draws six and keeps three at random, so it pays the same extra cost as the treatment.
- Treatment draws six and keeps the three the rule likes best.
Each run's final prompt was picked on 50 separate claims, instead of GEPA's own 10-claim validation set, then scored on 150 held-out claims. The design was frozen before the runs, and the analysis plan and the smallest effect it could detect were written down before I read any results. The selection did what it was supposed to: whenever the rule had something to rank, the treatment took its top three, and its picks overlapped with the control's about as much as two unrelated picks would.
It came back flat:
| comparison | difference | 95% interval | p |
|---|---|---|---|
| treatment vs control | −0.001 | −0.026 to +0.032 | 0.97 |
| treatment vs stock GEPA | −0.013 | −0.039 to +0.021 | 0.48 |
Then I checked the screen itself. I shuffled the data so no real pattern could exist, re-ran the exact same screen 200 times, and counted. It still crowned a winner in 134 of the 200 shuffles, and one at least as strong as mine in 43, about 1 in 5. Accounting for the 108 tries, my winner's p-value was about 0.2, not 0.0013. The finding I built the experiment on was the kind of thing the screen produces from pure noise.

The experiment also couldn't have detected the rule's effect even if it were real. It could reliably detect a difference of about 0.063 in final test score. The rule was expected to improve a single rewrite step by 0.027 on the three-example total, about 0.009 per example, and how a one-step gain carries into a final score is unknown. Before I read the results, my plan noted it was underpowered, but its formal power check passed, and only because it compared numbers on two different scales (the mix-up I describe at the end). On one scale, the smallest effect it could detect was about 7 times the effect it was built to find. For scale, stock GEPA's own gain over its starting prompt on this test was about 0.024 (interval −0.002 to +0.050), so the smallest effect this experiment could see was more than twice what GEPA itself delivered at this budget. In 5 of the 24 runs, the prompt picked at the end was the unoptimized starting prompt.
The gate
Which brings it back to the opening example.
Every rewrite GEPA proposes has to pass the same test: beat its parent on the three examples it was written from. The swap test scored every rewrite on two batches, so I could measure how much that test depends on the rewrite versus the examples:
- Same rewrite, different examples: the verdicts agree about as often as chance. True on both benchmarks (IFBench 53.9% vs 54.4% expected, HoVer 54.7% vs 54.3%).
- Different rewrites, same examples: the rewriter wrote three rewrites from each batch. Judged on the same batch, they agree 72% of the time on IFBench (55% expected) and 69% on HoVer (53%). Rewrites written from different batches agree about as often when judged on the same batch: 70% on both benchmarks, against about 55% expected.

Rewrites mostly don't beat their parents even on the three examples they were written from. On IFBench, judged on that batch, 34% scored higher than the parent, 32% scored lower, and 34% tied. On HoVer it was 37%, 27% and 35%.
Much of what the batch decides is how well the parent happened to do on it. On IFBench, when the parent totals under 0.5 out of 3, 76% of rewrites pass. When it totals 2.5 or more, 8% do. About a third of all verdicts are exact ties, which count as rejections. That's mostly the coarse scoring: the same parent prompt, re-scored on the same three examples in a later re-run, tied its first score 46% of the time.

The validation scores point the same way. In the eight IFBench runs with three-example batches, every kept rewrite had beaten its parent on its three examples. On the 150-example validation set, kept rewrites beat their parents by 0.001 on average (95% interval −0.003 to +0.005). For scale, the starting prompt, scored on the same 150 examples in each of the 10 IFBench runs, came out anywhere from 0.505 to 0.560. And the differences are mostly noise: their standard deviation is 0.021, about the same as for the gap between two scorings of one prompt (0.019 to 0.021, depending on how it's estimated).
What this can't show is whether the gate at least throws out rewrites that are clearly worse. Rejected rewrites were never scored on the validation set.
One bias also runs against the rewrites: each parent's validation score is the same single scoring GEPA used to pick it as a parent, and prompts that scored lucky get picked more often. The first kept rewrite in each run escapes that bias, since its parent, the starting prompt, was the only candidate. Those 8 beat the starting prompt by 0.011 on average (6 of 8 higher, interval −0.006 to +0.028). That isn't clear of zero either, and later rewrites have less room to improve, but it is the comparison the bias can't touch. Both checks also only measure average gains. GEPA's Pareto pool can value a rewrite that wins on different examples at the same average, and I didn't measure that.
I also can't tell from this whether three examples are too few or the rewrites are mostly too similar to their parents to tell apart. At a fixed batch size those are the same problem: the real difference is small next to the noise from which three examples were drawn and how they happened to score. Batch size, together with the acceptance rule, is the most direct way to change either. These are also settings where GEPA itself gained little: on HoVer its gain was about 0.024 with an interval that includes zero, and the HotpotQA study in Details was similar. Where rewrites make bigger improvements, three examples may be enough.
None of this proves that choosing better examples can't help. Nothing that measurably changes a rewrite was ever followed through a full run with enough power to see a small effect. But it is the best explanation I have for why selection was so hard to see: a better input goes through a test this noisy and mostly gets washed out.
What's next
Batch size does two jobs in GEPA. It sets how many examples the rewriter learns from, and it sets how many examples the gate uses. Right now it is three for both.
Two GEPA issues propose changing it during a run: #132 treats it like a learning rate that should follow a schedule, and #395 proposes setting it automatically to save time. Neither accounts for the second job. Recent GEPA versions let you pass in a different acceptance rule, but both built-in ones (StrictImprovementAcceptance and ImprovementOrEqualAcceptance) still judge on the same three examples, and the second accepts ties, which matters more than it sounds: about a third of verdicts are exact ties. Newer versions can also write several rewrites per round and keep only the ones with the biggest margin on the minibatch (BestImprovement and TopKImprovements). In my IFBench data, picking the best of three rewrites that way raised the margin on their batch to +0.23 out of 3, but gained +0.006 on a different batch (interval −0.006 to +0.020).
The second job also decides where the budget goes. At three examples, the gate costs about six metric calls per round. Each kept rewrite then costs 150 more to validate. In the three-example IFBench runs, validating kept rewrites took 83% of all metric calls, and those rewrites weren't detectably better than their parents on average.

So a bigger gate costs more per round but could pay for itself by keeping fewer rewrites that don't help.
There's a catch with the current rule, though. At three examples, a third of verdicts are ties, which the strict rule rejects, and ties get rarer as the batch grows. A rewrite that's no better than its parent passes a strict 'beat the parent' test about half the time once ties are rare, at any batch size. One that's truly 0.01 better per example, about the size of the real differences here, passes only about 70% of the time even at 150 examples. Telling those two apart reliably would take a margin and a batch of a few thousand examples. So a bigger batch alone could keep more of the rewrites that don't help, not fewer.
An earlier study of mine doesn't settle it either. There, judging rewrites on a fixed 20-example batch, with examples otherwise drawn at random, didn't beat stock GEPA (held-out F1 0.632 vs 0.639, three runs each, on a different task model). Neither moved far from the starting prompt's 0.632 (see Details). That batch was also the same 20 examples every round, which rewrites can come to fit, so it isn't the bigger fresh batch I have in mind.
So the number that should steer batch size is the final held-out score at equal budget, not how consistent the gate looks. That's what I want to build next: a sampler that can vary the batch size and the acceptance margin, possibly separately for the rewriter and the gate, and a measurement to steer it. At three examples, the same rewrite's verdict on two batches agrees at chance. The question is which batch size and margin give the best final score for the same budget, and whether the validation calls saved cover the cost.
Details
For anyone checking the work. Every number in this post traces to a committed file. Most are in the number register with the file and line they come from, and every figure is drawn from a CSV next to it. The three-arm results are in results.md.
- Units. One-step numbers (gate verdicts, swap results, the planned effect) are totals over a three-example batch, from 0 to 3. Validation and test scores are per-example averages, from 0 to 1.
- Statistics. Swap results use within-pair permutation tests, with intervals that resample the 8 runs. The three-arm results use exact sign-flip tests over 8 paired runs, with paired bootstrap intervals (results.md also has BCa intervals). Agreement intervals resample batch pairs. Intervals that resample 8 runs are optimistic, so for comparison: per-pair intervals, which ignore that grouping, are wider and point the same way (IFBench swap −0.015 to +0.026; HoVer swap +0.005 to +0.050, and −0.015 to +0.049 on the batch a rewrite hadn't learned from). For kept rewrites I use a t-interval over the 8 run means. The batch-size numbers in What's next use a normal approximation that ignores ties.
- Agreement beyond chance. Cohen's κ is −0.01 on IFBench (interval −0.06 to +0.04) and +0.007 on HoVer (−0.06 to +0.07). Chance means two independent verdicts at the observed pass rates.
- Every null is bounded, not zero. The IFBench swap test was designed to detect 0.033 on the three-example total, the feedback-deletion test about 10 to 12 points of keep rate, and the three-arm experiment 0.063 on the final test score.
- One-step and full-run results measure different things. The swap test measures three-example batches; the three-arm endpoint is a 150-claim held-out test.
- The opening rewrite comes from the swap test, which replays the rewriting step at logged points of real runs. It is not a live run's proposal.
- Rejected rewrites were never scored on the validation set, so the validation comparison covers kept rewrites only.
- Setups. IFBench: 150 training and 150 validation examples, 2,500 metric calls per run, gepa 0.1.1. HoVer three-arm runs: 100 training claims, a 10-claim validation set, 300 metric calls per run, gepa 0.0.27, with each final prompt picked on a separate 50-claim split and scored on a 150-claim test split.
- Models. IFBench: gpt-4.1-mini runs the task and gpt-4.1 writes the rewrites. HoVer: gpt-4.1-mini for both.
- Parent baselines differ. On HoVer, the parent's score on its original batch is the logged one from the run; its score on the other batch is fresh. On IFBench both are fresh.
- Two IFBench runs used one-example batches. They're in the repo. The kept-rewrite numbers above use only the eight three-example runs.
- One run carries the treatment arm. Treatment seed 4 beat control seed 4 by 0.100. Leaving out one seed at a time, treatment vs control ranges from −0.016 to +0.007, and treatment vs stock GEPA from −0.027 to −0.006. The control trails stock GEPA in every leave-one-out too (−0.017 to −0.008), so drawing six examples seems to cost something at this budget. Every number above includes all eight runs.
- GEPA's own gain on HoVer compares stock GEPA's eight final prompts with the starting prompt's test score in the 5 runs that returned it. Two of those 5 were ties broken toward the starting prompt. The interval is a Welch t-interval over those 8 and 5 scores.
- An earlier study on HotpotQA (frontier-gepa) came out null too, but there was little for better selection to show. 21 of its runs kept at least one rewrite and logged the validation scores this needs: 3 of stock GEPA and 18 of my variants that judged rewrites on a fixed 20-example batch. A later 12-run study in the same repo didn't log validation scores and is left out. In every one of the 21, the returned prompt beat the starting prompt on the 75 validation examples used to choose it (+0.062 on average, a best-of score inflated by the choosing). On 300 held-out test examples the average gain was +0.001, only 11 of the 21 beat the starting prompt, and stock GEPA alone gained +0.007 (source). That study used a different task model, Qwen2.5-7B.
- Exact ties passed on floating-point rounding five times on HoVer, and on one batch in the IFBench feedback-deletion test, which I count as a rejection. GEPA's built-in rule compares float sums, so this can happen in any run with fractional scores.
How this happened
This was an independent project on a laptop, alongside a summer internship.
The lesson I kept relearning was to check the actual bytes, not a summary that looks right. More than once, a number looked correct while measuring less than it claimed. I scored the feedback checklist when the rewriter reads the whole example. I computed a noise figure that mixed two different quantities. I reported design-time numbers as if they came from the live runs. I compared an effect on one scale with a detectable effect on another. The costliest was the screen: I didn't measure how often it produces winners from noise until after I'd run the experiment it picked. I only caught it because someone asked a question I couldn't answer, and the only way to answer it was to run the screen on noise.
Both repositories are public, including the failed analyses and the corrections: gepa-project and frontier-gepa.
ndankar@berkeley.edu