7-minute video for Premium members. I tested Sol vs Opus as code REVIEWERS, running the same prompt 8 times: 4 on Opus, 4 on Sol, also on different thinking levels of Medium/High. Is there a clear winner?
I have built a demo project in 4 phases with Composer 2.5 (intentionally weaker model) and then executed the same prompt 8 times:
Review the code changes in commits 0c9fdbf, 5ed05b2, 2beca84 and 7699a8c ("Phase 1" through "Phase 4") as a senior engineer reviewing a pull request.
It's a Laravel/PHP application — apply whatever standards you'd hold such a codebase to.
Do not modify any files: this is a review only, so report your findings instead of fixing anything.
Point out the issues that actually matter, each tied to a specific file and line, with a short explanation of why it matters and how you'd fix it.
Order them by how much you'd want them addressed before merging.
What do I mean by 8 times?
- 2 times on GPT-5.6-Sol Medium effort level
- 2 times on GPT-5.6-Sol High
- 2 times on Opus 5 Medium
- 2 times on Opus 5 High
Questions I was trying to answer:
- Obvious: is Sol or Opus better as a reviewer?
- Does High effort vs Medium effort change the results?
- I run the same review twice on the same model+effort, may result be vastly different?
