LLM Coding Leaderboard: My Methodology and Scoring Formulas

After looking at my LLM benchmark table, many people ask how exactly I am testing models, and what are the evaluation criteria. To not repeat it in every video, I decided to publish this article, explaining my metholodogy.

This benchmark measures how reliably a coding model can take an unfamiliar, partially implemented project and finish a production-shaped task from one prompt.

TL;DR

• 4 projects: CSV Import, Offline Sync, Bank Feed, and Shipping Quotes.
• 5 fresh attempts per model on every project.
• The model receives the same prompt and starter template each time; evaluator tests remain separate until the model is finished.
• Evaluation is performed by automated, deterministic tests using fixed inputs and assertions, not by a subjective reviewer watching the model work.
• Each attempt is graded on how many of the mapped evaluator checks it fails, not on a strict all-or-nothing pass. A clean attempt is worth 1 point; an attempt with a small number of failing checks earns partial credit; an attempt with several failing checks earns 0.
• Maximum score: 20 points — 4 projects × 5 attempts × 1 point.

In other words, this benchmark measures consistent, repeatable coding performance rather than a model's best result after retries or coaching, while still distinguishing a near-miss attempt from a badly broken one.

Example: one CSV Import attempt

After the model finishes editing the starter project, the evaluator runs the automated suite. Suppose the model imports valid rows but rejects the entire mixed file instead of salvaging the valid row:

$ php artisan test .eval-tests --filter="(EvalTest|ArchTest)"

FAILED ... reports skipped rows and row-level errors for a mixed import
Failed asserting that 0 is greater than or equal to 2.

Tests: 29, Passed: 28, Failed: 1

That single failure means the attempt did not pass cleanly, but it is not treated the same as an attempt that fails many checks. CSV Import is scored on a failure-weighted scale: an attempt with 1 failing check earns partial credit rather than being zeroed out outright.

CSV Import, attempt with 1 failing check: 0.5/1 leaderboard point

An attempt with 2 failing checks would earn 0.2 points, and an attempt with 3 or more failing checks earns 0. Only a fully clean attempt (0 failing checks) earns the full 1 point. The detailed 28/29 result is retained for diagnosis; the failing-check count is what drives the leaderboard score.

Projects at a glance

The sections below explain the starter template, prompt, evaluator examples, and scoring coverage for each project.

What the model starts with

Every project has a committed initial template. The template is a small, runnable project prepared specifically for the task. It contains the existing application structure, dependencies, database schema or public interfaces, a deliberately incomplete or first-pass implementation, and a small public smoke test where appropriate.

For every attempt, the harness:

1. Creates a fresh workspace from the project template.
2. Installs or prepares the pinned dependencies and initializes the application.
3. Gives the coding agent the project prompt and the workspace.
4. Lets the agent inspect, edit, and verify the project within its configured session limits.
5. Discards the workspace after the attempt unless it is explicitly kept for debugging.

The five attempts are independent. A model does not inherit code, notes, or feedback from an earlier attempt. This is intentional: the benchmark measures repeatable first-pass performance rather than the best result after a sequence of coaching rounds.

The initial templates are:

• CSV Import: a Laravel contacts application with an existing /api/contacts/import route, Contact model, table, and first-pass CSV importer.
• Offline Sync: a Laravel notes application with Note, SyncChange, and SyncMutation persistence infrastructure and a deliberately naive synchronization service.
• Bank Feed: a Flutter app where TransactionFeed has its fixed constructor and data-source interfaces, but the widget itself is only a stub.
• Shipping Quotes: a Go package with the public carrier/request/result contract, helpers, and a sequential first-pass QuoteService.

The model is expected to work inside this template. It is not asked to build an entire application from scratch, and it may not change the public contract where the prompt says that contract is fixed.

Prompts and evaluator tests are separate

The task prompt is stored separately from the evaluator tests. The model sees the prompt and the candidate workspace; it does not receive the hidden evaluator test files or the scoring map.

After the model finishes, the harness checks that it produced candidate changes, copies the evaluator tests into the disposable workspace, runs them, and records every passing and failing test. This prevents a model from optimizing against test implementation details and keeps the test suite out of the normal coding context.

The tests are behavioral. They exercise the public route, widget, or package API and inspect observable results such as stored records, JSON responses, rendered widgets, returned quotes, errors, ordering, and resource behavior. Tests do not award style points for a particular implementation; different internal designs can pass as long as they satisfy the contract.

The verification command depends on the project type:

Laravel:  php artisan test .eval-tests --filter="(EvalTest|ArchTest)"
Flutter:  flutter test .eval-tests --reporter=json --no-pub
Go:       go test -json -race -count=1 ./evaluator_tests

The real runner also writes a machine-readable test report, records the error message for failed cases, and treats a timeout or missing report as an unsuccessful attempt.

The repository also keeps a score map for the four current projects. It separates basic floor checks from more discriminating signal checks, with an optional resource check in the importer. That split is useful for analysing why an attempt failed, and the count of failed checks it produces is also what feeds the graduated leaderboard score described below.

Public scoring

Each attempt is graded from the number of mapped evaluator checks it fails (its error count), then converted into a point value on a graduated scale. Two scales are currently in use, both anchored at 1 point for a clean attempt and 0 once failures pile up, but differing in how forgiving the middle ground is:

Failure-weighted scale (used for CSV Import, Bank Feed, and Shipping Quotes):

Graduated-failure scale (used for Offline Sync, which has the largest check count and the most ways to fail partially):

An attempt that produces no usable change or that fails verification entirely is scored as having failed every relevant check, which lands at 0 points on either scale.

A project's score is the sum of its five attempt scores, so it can land anywhere from 0 to 5, including fractional values such as 3.7/5. The leaderboard total is the sum across the four projects, from 0 to 20. A result such as 3.7/5 for a project means the model's five independent attempts were, on average, close to but not always fully clean — not that two of five attempts simply failed outright.

Internally, we also retain the detailed passed/total test result, the raw failing-check count per attempt, and the floor/signal breakdown. Those details are useful context for videos and debugging, and they are exactly what drives the point value above — the leaderboard is not a separate, coarser pass/fail judgment layered on top.

1. CSV Import

Prompt

The model is asked to harden an existing contact CSV importer without changing its public API.

The fixed contract is POST /api/contacts/import, with the upload under the file multipart field and CSV columns name,email,phone. Email is the natural key. The importer must be idempotent, salvage valid rows from mixed input, return a structured JSON reconciliation summary, handle malformed or hostile uploads cleanly, and work for production-sized files.

In practice, the prompt tests whether the model thinks beyond explode(',', ...) and one successful demo file: quoted commas and newlines, UTF-8 BOMs, CRLF files, invalid bytes, invalid rows, duplicate emails, large files, and injection-shaped content all matter.

Evaluator example

One evaluator sends a mixed file containing one valid row, one invalid email, and one missing email. It expects a successful response with at least one imported row, at least two skipped rows, and row-level errors:

name,email,phone
Alice,alice@example.com,111
Bad,not-an-email,222
NoEmail,,333

Representative failure output when the implementation uses all-or-nothing validation instead of salvaging rows:

FAILED ... reports skipped rows and row-level errors for a mixed import
Failed asserting that 0 is greater than or equal to 2.

The exact formatting depends on the Pest/PHPUnit version; the important part is that the response did not report the required skipped rows.

Scoring coverage

There are 29 mapped checks:

• 1 floor check for importing a clean file;
• 27 signal checks covering structured responses, duplicate prevention, CSV parsing and encoding, row validation, partial acceptance, batching, large inputs, bad uploads, security, and database invariants;
• 1 optional memory-bound check for a roughly 20,000-row file.

2. Offline Sync

Prompt

The model must harden POST /api/sync for clients that edit notes on multiple devices while offline. The API accepts a device_id, cursor, and ordered mutations for create, update, or delete, and must continue returning results, changes, and next_cursor.

The server is authoritative. Mutation IDs provide idempotency within a device, versions detect stale edits, conflicts must not overwrite newer state, deletes must produce tombstones, mixed batches must salvage valid siblings, and pull-only requests must return an accurate ordered change feed. Malformed JSON or malformed mutations must produce clean reconciliation responses rather than server errors.

Evaluator example

An evaluator submits the same create mutation twice, then performs a pull from another device. The note must be created once, the retry must describe the original result as applied or duplicate, and the observer must see only one canonical change.

Representative failure output from an implementation that applies retries twice:

FAILED ... retrying a create mutation does not apply it twice
Expected collection to have count 1. Got 2.

Other tests replay conflicts, reuse mutation IDs with different contents, apply stale updates and deletes, verify tombstones, check cursor boundaries, and simulate a failed change-log write to ensure canonical state is not left half-committed.

Scoring coverage

There are 41 mapped checks:

• 3 floor checks for creating, updating, and deleting a note correctly;
• 38 signal checks covering retries, batch idempotency, device scoping, conflicts, tombstones, cursor delivery, patch semantics, malformed and mixed input, ordering, transactions, and production-sized batches.

This project is intentionally stateful: a solution can look correct for one request and still fail when requests are retried, reordered, replayed from another device, or combined in one batch.

3. Bank Feed

Prompt

The model must implement the Flutter TransactionFeed widget without changing its constructor, its data-source interface, or its required widget keys. The source returns pages of untrusted raw maps. Records may be duplicated, out of order, malformed, in the wrong currency, or delivered slowly.

The feed must show valid records only, count skipped records, parse integer amounts, normalize timestamps using the supplied display offset, sort newest first, group by calendar day, calculate the signed total, show loading/empty/error states, retry failed pages, paginate without duplicate requests, and remain responsive with years of history and very long descriptions. Rebuilding the widget with changed inputs must also update the display correctly.

Evaluator example

One widget test provides five unusable records and one valid record. It expects only txn-ok to be rendered and the feed-skipped text to say 5 skipped.

Representative failure output from a widget that renders malformed records instead of salvaging only valid ones:

Expected: [txn-ok]
Actual:   [txn-no-identity, txn-42, txn-bad-amount, ...]

Another useful example is pagination: a later page can fail, but already loaded rows and the current total must remain visible while an inline feed-page-error and retry control are shown.

Scoring coverage

There are 48 mapped checks:

• 27 floor checks covering the basic rendered feed, parsing, ordering, grouping, totals, initial errors, retries, pagination, and layout safety;
• 21 signal checks covering malformed records, deduplication, cross-page ordering, skipped counts, duplicate requests, lazy list construction, partial failures, and in-place widget updates.

The checks use a phone-sized viewport and include large pages, long text, emoji, right-to-left text, and large amounts so a logically correct implementation that cannot render reliably does not receive full credit.

4. Shipping Quotes

Prompt

The model must harden the Go QuoteService without changing the exported contract in contract.go. It aggregates quotes from multiple third-party carriers, all of which are treated as untrusted: a carrier may be slow, panic, fail, return malformed data, ignore cancellation, or return duplicate services.

The request must be normalized and validated. Carrier calls must run concurrently with a configurable bound, honor per-carrier and overall timeouts, preserve valid sibling quotes, reconcile failures deterministically, and return partial success when possible. A successful non-empty result may be cached, equivalent requests must share cache entries, concurrent identical misses must coalesce, and callers sharing a flight must cancel independently.

Evaluator example

The concurrency test configures 12 carriers with MaxConcurrency: 3. It observes the number of active calls and requires the peak to be no more than three while still proving that calls actually overlap.

Representative failure output from a sequential or unbounded implementation:

bad concurrency peak 12, err=<nil>

Timeout tests similarly require a fast carrier's quote to survive a slow or stuck sibling, and cache tests verify that returned slices cannot mutate the cached result or another caller's result.

Scoring coverage

There are 19 mapped Go test groups:

• 1 floor group for basic carrier fan-out and a usable quote;
• 18 signal groups covering validation, normalization, quote reconciliation, deterministic ordering, panic and nil-carrier isolation, ownership, concurrency, timeouts, cancellation, cache lifecycle, cache keys, in-flight coalescing, and production-sized carrier sets.

What a score means

A high score means the model can repeatedly produce a complete solution that survives both the obvious happy path and adversarial behavioral checks. A lower score does not necessarily mean the model cannot write code; it often identifies a specific reliability gap, such as treating a whole batch as atomic, forgetting replay semantics, rendering eagerly, or allowing one broken upstream dependency to affect all valid results.

The five-attempt format makes that distinction visible. One successful attempt may be a lucky completion; repeated successful attempts indicate a more dependable coding agent.