Independent Research
Reproducible, transparent AI model comparisons. Evidence over opinions. No sponsored rankings. Every experiment documents exact prompts, parameters, and evaluation methods.
Build the most trustworthy public collection of AI benchmark experiments.
Every benchmark publishes exact prompts, API parameters, execution environment, and reproducibility hashes.
Scoring rubrics, human review protocols, and raw outputs are documented. No black-box evaluations.
No sponsored rankings. No pay-to-win placements. No invented results. Full disclosure of funding.
Rigorously designed benchmarks with full documentation.
Four frontier models attempt to fix a subtle async/await bug in a FastAPI endpoint. Evaluated on correctness, test pass rate, and minimal diff size.
Five models solve a classic reasoning puzzle with added constraints. Scored on logical validity, step completeness, and constraint satisfaction.
Recently published experiments.
Four frontier models attempt to fix a subtle async/await bug in a FastAPI endpoint. Evaluated on correctness, test pass rate, and minimal diff size.
Vision-capable models extract structured data from a bar chart image. Scored on numerical accuracy and schema compliance.
Models write a 1,200-word technical blog post explaining Kubernetes networking to intermediate developers. Evaluated on accuracy, structure, and clarity.
Five models solve a classic reasoning puzzle with added constraints. Scored on logical validity, step completeness, and constraint satisfaction.
Models synthesize findings from 8 provided research papers on carbon pricing. Scored on citation accuracy, synthesis quality, and hallucination rate.
Market and benchmark-planning reports kept separate from experimental results.
7/16/2026
A report-driven benchmark proposal comparing Claude Sonnet 5, GPT-5.6 Terra/Sol, DeepSeek V4, and Gemini 3.5 Flash on ScarfBench-style Java migration tasks.
Explore benchmarks by domain.
Code generation, debugging, and software engineering tasks
ReasoningLogic, math, and multi-step problem solving
WritingClarity, tone, structure, and editorial quality
BusinessAnalysis, strategy, and professional communication
SEOSearch optimization, metadata, and content structure
ResearchLiterature synthesis, citations, and fact-finding
AgentsTool use, planning, and autonomous workflows
VisionImage understanding, OCR, and visual reasoning
MathSymbolic math, proofs, and quantitative accuracy
TranslationMultilingual fidelity and nuance preservation
AutomationScripting, APIs, and workflow execution
EnterpriseCompliance, security, and production readiness