RN coding tests — the benchmark suite

the three axes

What's being measured

Every scenario is scored on three independent axes, then blended into one composite. The axes are the point: generic benchmarks check whether code looks like TypeScript. These check whether it is React Native.

composite = 0.4·correctness + 0.3·adherence + 0.3·guardrails\n# no --live run? correctness is excluded and the rest renormalized:\ncomposite = (0.3·adherence + 0.3·guardrails) / 0.6

benchmark 1 · every night

The model leaderboard

The headline benchmark: a real model drives generation across all 43 scenarios and is scored on all three axes — with correctness now measured live. What it's for: a public, reproducible RN-specific model leaderboard — the same harness, any provider. The nightly workflow runs a [local · openai · anthropic] matrix; tonight only the local row has results.

Generated 2026-08-15 — model qwen2.5-coder-1.5b (local), scored with --live --install: a real npm install, then jest (tests, weight 0.5), tsc --noEmit (typecheck, 0.25) and eslint . (lint, 0.25) in a throwaway project. Tests pass on 4 of43 scenarios; where typecheck or lint fails, it is a real defect in the model's output. The guardrail floor holds at 88–100% on every scored scenario.

benchmark 1b · the model presets, same harness

Three local models — which one should you run?

The same live-scored harness, three GGUF presets you can actually run on your laptop — no API key, no source leaves your machine. What it's for: init auto-selects a tier from your RAM (fast 8 GB / balanced 16 GB / quality 32 GB), and this table is the honest cost/quality curve behind that choice — all three rows live-scored, measured the way the leaderboard measures everything else. The gradient is the story: fast → balanced is a dip this pass (79% vs 67% — the nightly 1.5B re-score landed above the 3B run; model variance, expect it to flip), and balanced → quality is the jump — the 7B scores 97% composite, 114% of the 86% human reference across the full 33-scenario pack, perfect on 29 of 32 scored scenarios — including every one of the 20 new real-world app scenarios at 100%. If your machine has 32 GB, this is why you run the big model.

Every tier is scored with --live --install, exactly like the leaderboard above — composite deltas are per-scenario, from the three committed runs (bench/results/local.json + local-3b.json + local-7b.json). Run your own row — or your own machine's row — with vectalon bench --model local --preset <fast|balanced|quality> --live --install -o bench/results/local-<tier>.json, then merge everything with vectalon leaderboard.

benchmark 2 · sliced by area

Where it wins — and where it doesn't

The same nightly run, aggregated by suite. What it's for: a leaderboard that hides variance is a lie — this shows exactly which area of React Native the harness handles today, so the roadmap and the model choice chase the weak spots.

The gradient is the point: navigation and core-ui sit at 90–100% while perf lags at 45% — the model's weakest muscle is media-heavy rendering and async orchestration (image feeds, pagination, offline queues), which is exactly where the next model or a fine-tune should spend its budget.

benchmark 3 · honest about the ceiling

Relative to a human

Every scenario ships with a human-authored reference solution, scored by the same rubric. What it's for: it defines what "passing" means. The generated pass reaches 80% of the 89% human-reference composite — up from 30% the moment correctness started being scored for real.

The human reference is not automatically 100% — it's scored by the same rubric, so a reference with a hardcoded hex literal scores below 1.0 on adherence. Generated code can therefore beat the human: rn-05 (multi-field form) and rn-06 (offline queue) score 100% composite at 117% and 148% relative — the generated code out-scored the reference on its own rubric. That's honest scoring, not an error.

benchmark 4 · every PR

The regression gate — the harness protecting itself

A different kind of benchmark: no model, every pull request. Nine scenarios — the six scaffold-able add-scenarios plus three dependency-removal scenarios (rn-11, rn-34, rn-35), now deterministic via the removal seam — run through the deterministic generator, and the scores are compared against the committed baseline. What it's for: any PR that improves a guardrail rule or rubric check must move the benchmark up; any PR that silently breaks the scaffold, a rule, or score detection fails CI. The harness can't regress without the leaderboard noticing.

$ npx vectalon bench --baseline bench/baseline.json\n# exit 1 on any axis regression

benchmark 5 · every fix

The fix-bench — 100 real failures, auto-fixed

A different kind of reliability number: vectalon fix-bench takes the "fix my React Native issue" wedge and measures it against 100 real failures across the ten families the roadmap names — Gradle dependency conflicts, Kotlin / AGP / Gradle incompatibilities, CocoaPods, Xcode, Metro resolution, Hermes, RN upgrade breakages, native module linking, and TypeScript regressions. Each scenario materializes a healthy RN 0.74 project, injects one real failure, and runs the vc fix pipeline end-to-end — diagnose → plan → sandbox-apply — hermetically (no build ever runs, CI-safe). What it's for: both product-milestone targets are now cleared — 100% correct diagnosis (target ≥ 80%) and 69% of fixes applied without human modification (target ≥ 50%) — with zero false positives on the healthy control. The same pack is a regression gate in CI: hermetic tests in __tests__/fixBench/seams.test.ts re-run all 100 scenarios and fail the build if either target slips.

$ npx vectalon fix-bench
┌─ vc fix-bench — 100 real RN failures, measured ──────────────┐
  Diagnosis accuracy            100.0%   target 80.0% ✓
  Fix accuracy (auto, no human) 70.0%    target 50.0% ✓
  Build success (post-fix)     27.0%
  False positive rate          0.0%
  Human intervention           34.0%
  Time: median 15ms/scenario · total 1.6s
  Estimated time saved: 50.0 hours vs a 30-min-per-failure human baseline
✔ Both product-milestone targets met — 80%+ correct diagnosis and 50%+ fixes applied without human modification.

The six axes

Per suite — where the auto-fix lands

The gradient is the point: version-alignment families (Kotlin, AGP, Gradle, SDK) and the pod/autolinking seams auto-fix at 100% — the pipeline edits the exact version pin, Podfile, or settings.gradle line. The remaining manual cases are the genuinely judgment-heavy ones: code signing, provisioning, linker configuration, and SDK toolchain installs, where the deterministic edit would be guesswork — the pipeline says so and gives the exact command instead. Honest numbers, not a vanity 100%.

benchmark 6 · the engineering benchmark

The Vectalon RN Engineering Benchmark — competitors can't copy this

A benchmark no generic coding eval can replicate: vectalon rnbench scores the committed 43-scenario pack across eight engineering dimensions a team actually cares about — architecture, native integration, dependency management, testing, performance, security, upgrades, debugging — with every scenario mapped to exactly one dimension. What it's for: the material is the moat. The 43 scenarios, the 43 human references, and the RN-specific rubric (correctness = typecheck + lint + tests actually run; adherence = the craft checklist; guardrails = the bans) are all committed and exported — anyone, including a competitor, can run the same task set and be scored by the same rubric. The pack is 35 build tasks plus 4 upgrade-breakage repairs (rn-36..39) and 4 debugging repairs (rn-40..43), so the upgrades and debugging dimensions now score from real pack tasks — the deterministic fix seam applies the declared repair to the broken fixtures (scored by a `fix-applied` adherence check), the Human row reads the reference composites (100/100 both), and the 7B tier is scored live. Rows that haven't run yet render as pending; a committed competitor result renders in the leaderboard. Never a cherry-picked number.

$ npx vectalon rnbench
┌─ vectalon rnbench — Vectalon RN Engineering Benchmark ─────┐
  Architecture (6) · Native integration (3) · Dependency mgmt (3) ·
  Testing (9) · Performance (10) · Security (4) · Upgrades (4) ·
  Debugging (4)
  Vectalon            100%  100%   99%  100%  100%  100%  100%  100%
  Generic LLM (7B)     96%  100%    —  100%   98%   90%   80%   59%
  Generic LLM (3B)     63%    —    —    —    —   72%    —    —
  Generic LLM (1.5B)   89%   88%    —   74%   74%   73%    —    —
  Human                92%   85%   99%   82%   86%   85%  100%  100%
  Claude Code / Cursor / Cline / Windsurf / Aider — pending — run the protocol
✔ Computed from committed artifacts — publish the methodology, export the bundle, run competitors.

The eight dimensions

The anti-cherry-picking rules