Original sourceventurebeat.com
Summary
A study by Shandong University found that increasingly complex harnesses around coding models (system prompts, tools, and context rules) do not deliver measurable accuracy gains and may triple the cost per task. The study tested three production harnesses (Claude Code, OpenCode, mini-SWE-agent) wit…
Key points
- Teams procuring or standardizing coding agents can base decisions on total cost per task rather than advertised benchmark scores.
- If more complex agent wrappers only increase bills without increasing accuracy, enterprises' logic for procuring AI development tools needs to be rewritten.
- Teams should build internal evaluations using task success rate and cost per task as metrics, avoiding expanding deployment based solely on vendor benchmarks.
Editorial note
This page is Code & Chain's editorial summary of public sources. It may be prepared with AI assistance and published through an automated workflow. Refer to the original sources; this content is not investment, legal, or tax advice.