Relevant Context · creator of SpyneJS
The Acme Dashboard experiment
The same product — Vercel's Next.js Learn dashboard — evolved feature-by-feature on Next.js 16.3 and on SpyneJS, every feature built by a fresh AI agent session from identical, framework-neutral task statements. Pre-registered predictions, transcript-counted metrics, misses published.
The takeaway
Given the identical agent setup Next.js ships for itself,
a framework the models have never seen reached parity
— and in places exceeded the experience — with its
knowledge base as the primary supplied context.
Cost, stated honestly: ~1.9× the agent tokens on
Next's home terrain, with zero shipped defects on either
side.
Try both — they're live
Demo login for both: user@nextmail.com / 123456 —
5,000 invoices, live payment activity, Cmd-K search, bulk editing.
What was measured
- Six metric-scored rounds (twelve agent builds) plus a polish round: sorting, freshness, global quick-search, 5,000-row scale, real-time payments, spreadsheet-grade bulk editing.
- Each arm ran its own framework's best practice for agents: Next.js with its full first-party agent apparatus (AGENTS.md, bundled docs, Skills); SpyneJS with its served agent knowledge base — the SpyneJS Knowledge Base.
- Every number counted from agent transcripts — never from
agent self-reports. Per-round
pre-*/post-*git tags on both repos.
What we found
- A framework the models have never seen reached parity — and in places exceeded the experience — with the SpyneJS Knowledge Base as the primary supplied context.
- Zero shipped defects on either side; framework-semantics corrections 2 vs 2, all caught in-session.
- Zero third-party dependencies added by either side, across every round.
- Honest cost: the SpyneJS side spent ~1.9× the agent tokens on Next's home terrain.
- Freshness: staleness-zero on Next.js took handling four distinct caching/restore mechanisms (one undocumented) across five incidents; on SpyneJS it held by construction.
- Latency (identical probes, both live deployments): each architecture wins where its model predicts — sorting settles 24 ms vs 132 ms and deep page jumps 58 ms vs 130 ms in SpyneJS's favor; optimistic toggle flips (11 vs 68) and cold-load-to-content favor Next.js.
- An ablation arm re-ran a round with the Knowledge Base withheld — published with everything else, including a confounded pilot run. Publishing the misses is the method.
The evidence
| spynejs.com | the framework |
| npm: spyne | published package, full readable source |
| benchmark-protocol | the kit: verbatim tasks, scoring receipts, fixtures, counting rules — run it against your framework |
| acme-dashboard-nextjs | Next.js arm, per-round tags |
| acme-dashboard-spynejs | SpyneJS arm, per-round tags + ablation branches |