From 86% to 71%: learning to trust a brain decoder's accuracy

I set out to train a small neural network to read someone’s brain — given half a second of MEG signal, guess whether they just heard a sound on the left/right or saw something on the left/right. The model worked. But the part worth writing down isn’t the model; it’s how the number I reported swung from 65.5% → 86.2% → 71% ± 5% as I fixed one evaluation mistake after another. The honest answer was never 86%.

This ran on Google Colab (see how I use Colab for the setup), using Meta’s neuroai stack — neuralset for data, neuraltrain + PyTorch Lightning for training, and an EEGNet from braindecode.

🧠 The task

The data is the MNE sample dataset: one person in an MEG scanner (≈300 magnetic-field sensors around the head), shown a stream of stimuli about every 0.75 s. Four stimulus types, interleaved:

Each training example is a slice of brain activity around one stimulus — (306 channels × 60 timepoints), i.e. 306 sensors sampled at 120 Hz over a 0.5 s window. The label is which of the four stimuli caused it. The model is EEGNet (Lawhern et al., 2018) — a deliberately tiny conv net (6,100 parameters) that learns which frequencies and which sensors matter.

In one line: read half a second of someone’s brain and guess what they just heard or saw.

(One subject, one session — so everything here is within-person decoding on a single recording, not a universal brain reader.)

1️⃣ A first result, and a sanity check

The first training run landed at 65.5% test accuracy. Is that real? A 4-class problem has a 25% chance baseline, so I evaluated the untrained (randomly initialized) model:

24.1% sitting right on the 25% chance line is exactly what you want to see — it confirms the input pipeline isn’t leaking the label, so the trained model’s jump to 65.5% is real learning, not a trivial artifact. So far so good.

2️⃣ Trap 1 — the split wasn’t stratified

The pipeline did a random 60/20/20 train/val/test split with no class balancing. On a small dataset that produces lopsided splits:

Class 1 got 56 of its 73 trials into training but only 9 into test, while class 0 was barely trained. The dataset is essentially balanced (≈72/class) — the imbalance is purely a small-sample split artifact. The one-line fix:

default_config["data"]["study"][1]["stratify_by"] = "code"

…which rebalances every split to ~14–15 per class in test.

3️⃣ Trap 2 — comparing two single runs lied to me

With stratification on, I retrained and compared:

So… unstratified is better?! That conclusion is wrong, for three compounding reasons:

• Different test sets. The two runs hold out different trials, so the numbers aren’t comparable. With only 58 test samples, 1 trial = 1.7% — the 5-point gap is 3 trials.
• Tiny test set. A 58-sample accuracy has a 95% interval of roughly ±9 points. 81% and 86% are statistically indistinguishable.
• Checkpoint resume. This was the real killer: the same unstratified config had scored 65.5% earlier and 86.2% now. Identical code, deterministic seed — but every run loaded and overwrote the same last.ckpt, so later runs silently resumed training and accumulated epochs. The 86.2% was a model that had trained far longer than I thought, scored on a lucky test set.

Lesson: never compare single runs, and train each model from scratch (load_checkpoint=False, save_checkpoints=False).

4️⃣ Doing it right — controlled, multi-seed

To isolate the real effect of stratification, I trained from scratch over many split seeds and compared the distributions instead of two lucky draws. Over 20 seeds:

Two findings:

1. The means are tied (0.663 vs 0.651) — stratification does not raise accuracy.
2. It tightens the spread and clips the worst-case tail (unstratified swings 0.517 → 0.810).

There was even a meta-lesson here: at 5 seeds the spread gap (std) looked like a dramatic 4×; at 20 seeds it shrank to ~1.35× (0.066 vs 0.049). The dramatic version was a small-sample mirage — you need enough seeds to estimate variability itself.

So stratification is free insurance against bad-luck splits, not a performance lever.

5️⃣ The real ceiling — test-set size

Here’s the deeper point hiding under all of this: the test set is only ~58 trials. Scoring a model on 58 examples is like polling 58 people — even with a perfect sample, the result can swing by roughly ±12% (95%) just from which trials happened to land in the test set. That’s the same kind of margin of error a 58-person political poll carries, and stratification can’t touch it: balancing the classes doesn’t make the test set any bigger. The only real fix is to stop betting the whole answer on a single 58-trial draw.

6️⃣ The capstone — repeated stratified k-fold

The honest evaluation: repeated stratified 10-fold cross-validation, every trial tested exactly once per pass, predictions pooled over all 288 trials, repeated 5× to check reproducibility, each model trained from scratch with an inner val split for early stopping.

Repeated stratified 10-fold × 5:  accuracy = 0.709,  honest 95% CI = ±0.053 (from 288 trials)
(the ±0.016 spread across the 5 repeats is reproducibility, NOT generalization certainty)
fold-level:                       0.709 ± 0.074 (std over 50 folds)

70.9%, with an honest 95% CI of about ±5% — and the numbers above are the whole lesson. A single ~29-trial fold bounces with std 0.074 (the small-sample lottery, alive and well). Testing on all 288 trials at once tightens the 95% interval to ±0.05 — about 2× tighter than the 58-trial split’s ±0.12, because you’re finally scoring on every trial. The ±0.016 spread across the 5 repeats looks tighter still, but it’s a trap: the repeats reuse the same 288 trials, so it measures how reproducible the procedure is, not how well you actually know the accuracy. The data floor is ±5%:

For context, the full journey of the number:

Notice the k-fold mean (0.709) is even a bit higher than the multi-seed sweep (0.66) — because 10-fold trains each model on ~90% of the data instead of 60%, so it’s less pessimistic. More training data, same center, and an honest ±5% error bar set by the 288 test trials — tighter than the 58-trial split’s ±12%, but no tighter than the data allows.

7️⃣ Where the model actually wins

The flat 71% hides a bimodal story. Pooling out-of-fold predictions across all 5 repeats (288 trials × 5 = 1,440 predictions) into a confusion matrix (rows = true, columns = predicted):

Dashed lines mark chance level: 25% for the full 4-class guess, 50% for telling left from right within one modality.

The model is doing three very different things:

• Sound vs. sight: ~94% — cross-modality confusions are tiny. It nails what kind of stimulus.
• Visual left vs. right: ~85% — cleanly separated.
• Auditory left vs. right: ~57% (≈61% once you condition on getting the modality right) — reliably above the 50% coin-flip across hundreds of trials, but far weaker than vision. It only weakly recovers which ear a tone came from.

And that lines up with the neuroscience. The visual field is strongly contralateral and retinotopic — a left-field stimulus lights up right occipital cortex with a clear spatial signature. Auditory “which ear” is processed bilaterally with much weaker lateralization in MEG, so it’s genuinely harder to decode. The signal clearly still carries some ear information — the model stays reliably above chance — but far less than the visual field offers, and on ~70 trials per class I can’t cleanly separate “weak signal” from “tiny model starved of data.” Either way, the decoder’s error pattern recovers a known asymmetry of the brain, and that one weakness is what drags the ~71% down.

🎯 Takeaways

The model was the easy part. The evaluation was where all the learning happened:

1. Always stratify classification splits — free insurance against bad-luck imbalance. It won’t raise your mean; it clips the worst case.
2. Train from scratch when comparing. Shared checkpoints silently resume and inflate results — the single most dangerous bug here (65.5% → 86.2% from nothing but resume).
3. Never trust a single split. Compare distributions over seeds, not two numbers.
4. Test-set size caps your precision. A 58-trial test carries a ~±12% margin (95%) no matter what; pooling k-fold over all 288 trials tightens it to ~±5%, but no cleverer split can — and averaging repeats only buys reproducibility, not certainty.
5. Report mean ± CI, and always look at the per-class breakdown — a flat accuracy hid the fact that 3 of 4 classes were strong and one was near chance.

The difference between a number and an understanding: I started with “65 vs 81 vs 86 — which is real?” and ended with “71% ± 5%, and here is precisely where the model wins and loses.” 🌱