Black Friday Discount! Until December 1st, 50% off premium membership
Tutorials

LLM Coding Leaderboard: My Methodology and Scoring Formulas

November 11, 2026
3 min read

After looking at my LLM benchmark table, many people ask how exactly I test models and what the evaluation criteria are. To avoid repeating it in every video, I published this article explaining my methodology.

Each model receives 8 projects to be tested, with a total of 33 prompts.

The projects cover Laravel/PHP, React + TypeScript, Flutter/Dart, Go and Python, and every one of them starts from existing code, not a blank folder. That's what most of our real work looks like.

What kind of tasks these are

  • The model is the implementer. One coding agent (Claude Code, Codex CLI, OpenCode, Cursor…) gets one prompt and a repository, and does the work itself.
  • It's not orchestration. There are no subagent swarms, no multi-agent pipelines, and no human feedback in between. I'm not testing how well a model manages other agents.
  • The tasks are not long-running. The median task takes 4–11 minutes, depending on the project. These are focused, ticket-sized jobs, not overnight "build me a SaaS" sessions.
  • No retries or coaching. Every prompt starts in a fresh copy of the project, and the model never sees the hidden tests or the scoring rubric.

The 8 projects

The newest one is Pricing Engine, in Python. It has two prompts, each in a fresh session: first the model reworks a prototype engine for pricing rules nobody has written yet, then it adds six new rule families on top of its own code. So it measures whether the first version was actually built to be changed.


Starting point → prompt → expected result


How big is one task?

Median values per prompt, across 2,242 runs of all tested models since August 2026:

In other words, these are tasks a senior developer would plan for an hour or two, and a good agent finishes in under 10 minutes.


How it becomes leaderboard points

The same prompt is launched several times, each in a fresh copy of the project, so that one lucky (or unlucky) run doesn't decide the score:

Consistency penalty

An average alone hides how far apart the runs were. A model that scores 95 and 55 gets the same average as one that scores 75 twice, but only the second one is predictable.

So the leaderboard deducts a consistency penalty when a model's runs on the same project are too far apart. A small spread is free, and every point above that allowance costs a little:

The penalty is subtracted from that project's points (it can never take them below zero), and it's shown in its own Penalty column on the leaderboard, so you can see which models are reliable and which ones just got lucky.


If you want more information about methodology or you have ideas or suggestions, email me povilas@laraveldaily.com.

Share this video

Povilas Korop

Get Weekly AI Coding News

Sent every Wednesday. No spam, ever. Unsubscribe anytime.