SWE-bench scores for top coding agents now clear 70%. Inside Databricks' production codebase—millions of lines spanning distributed systems, Scala runtime code, and internal APIs built over a decade—those numbers fall off a cliff. The benchmark captures something real about model reasoning, but it does not capture what breaks in enterprise code at scale. The open question is whether the gap is closeable with more context, better retrieval, or whether it points to a fundamental mismatch between how agents reason and how large codebases actually work.
The current benchmark narrative is clean. Top models cross 70% on SWE-bench Verified. Leaderboard positions shift by single percentage points between releases. The press reads this as agents can code.
This is the wrong frame for anyone deploying these systems in production.
SWE-bench draws from 12 Python repositories—requests, Flask, sympy, scikit-learn, and similar. Well-maintained, heavily documented, mostly single-language, with small-to-medium issue scopes. The benchmark is useful for comparing model reasoning under controlled conditions. It is not useful for predicting whether an agent can close a bug in a Scala codebase with 4 million lines of history, custom test infrastructure, and internal APIs that are documented nowhere except in the code itself.
Databricks sits at exactly this intersection. Their engineering team maintains Apache Spark, Delta Lake, MLflow, and the Databricks Runtime—each a large, multi-decade codebase with deep interdependencies. Understanding what happens when you run coding agents against systems at this scale is not an academic exercise. It is the actual deployment condition for every company running serious software infrastructure.
The gap between SWE-bench performance and what happens in practice is where the interesting engineering work lives.
The SWE-bench paper (Jimenez et al., 2023) built a benchmark from real GitHub issues with verified fixes. This is meaningfully better than synthetic coding tasks. The evaluation criterion—does the agent's patch pass the test suite?—is objective. These are genuine strengths.
But the test set has structural properties that favor specific agent behaviors:
None of these properties hold for Databricks' Spark codebase. A typical runtime bug touches the query optimizer, the execution engine, and possibly the catalog. The test suite requires a full cluster to run. The relevant context may be in a Scala trait hierarchy three levels deep, in a Python integration test written in 2017, or in a comment in a JIRA ticket that never made it into the codebase.
Scores are flat across recent model releases. Pass rate on actual enterprise issues tells a different story.
Databricks documented their experience with AI coding assistance across their engineering organization, covering both suggestion-based tools and autonomous agents. Several patterns emerge consistently:
Context depth is the primary bottleneck. Agents working on Spark's query optimizer routinely need to trace through 10+ files before they have enough context to form a correct hypothesis about the bug. At 200k context windows, this is theoretically tractable. In practice, the model's ability to maintain coherent reasoning across 150k tokens of Scala code is substantially lower than SWE-bench performance would predict.
Internal API conventions break generalization. Databricks' internal codebases use patterns—custom configuration systems, internal RPC frameworks, proprietary serialization formats—that do not appear in training data. Agents that score in the 60–70% range on public benchmarks drop substantially on tasks that require understanding these internal abstractions. The model has not seen them. It halluminates plausible-looking API calls that do not exist.
Test infrastructure is a barrier to agent loops. Autonomous agents depend on tight test-run loops to verify patches. On Spark, a full test suite run takes hours, requires cluster provisioning, and may fail for environmental reasons unrelated to the patch. Agents that rely on this architecture—which is what drives SWE-bench performance—stall in this environment. Either the agent loops indefinitely, burning tokens without converging, or it exits early with an unverified patch.
Token consumption is non-linear with task difficulty. Simple bug fixes—null checks, off-by-one errors, typos—are handled efficiently at 20k–50k tokens per task. Anything involving distributed systems semantics, cross-module refactors, or performance optimization burns 200k–500k tokens depending on model and scaffolding. At current pricing on frontier models, complex tasks can cost $1–$5 each. That changes the deployment economics entirely.
Not all models fail the same way on large codebases. Here is where they stand:
| Model | SWE-bench Verified | Context Window | Scala/JVM Handling | Internal API Hallucination | Cost per 1M Tokens |
|---|---|---|---|---|---|
| Claude 3.7 Sonnet | 70.3% | 200k | Strong | Moderate | ~$3 |
| o3 (OpenAI) | 71.7% | 128k | Moderate | Lower | ~$30–60 |
| Gemini 2.5 Pro | 63.8% | 1M | Moderate | Moderate | ~$7 |
| GPT-4o | ~49% | 128k | Moderate | Higher | ~$5 |
| DeepSeek R1 | ~49% | 64k | Weaker | Higher | ~$0.50 |
| Llama 3.3 70B | ~35% | 128k | Weak | Highest | ~$0.20 (self-hosted) |
The 1M token context window of Gemini 2.5 Pro is theoretically an advantage for large codebases. In practice, attention quality at extreme context lengths degrades. The model's effective recall of content at position 800k is substantially worse than at position 10k. The context window is a ceiling, not a working memory.
o3's lower hallucination rate on novel APIs is meaningful for enterprise deployment. The cost premium—roughly 10–20× more expensive per task than Claude Sonnet—is significant at the volume an engineering organization actually generates.
For self-hosted deployment on internal infrastructure, Llama 3.3 70B is the only model with a reasonable cost profile at scale. The 35% SWE-bench score is a floor with proper fine-tuning. Out of the box, it is not competitive for autonomous agent tasks.
The benchmark landscape as currently reported gives you the wrong priors for deployment decisions. Here is what to actually measure.
Don't deploy on tasks with inadequate test coverage. Agents optimize for passing tests. If the test suite does not cover the behavior you care about, you will get patches that pass CI and break production. This is not a model problem. It is a test infrastructure problem that agents expose aggressively.
Don't use autonomous agents on security-sensitive code without mandatory human review. Agents generate plausible-looking patches for authentication logic, encryption, and access control. The patches may pass tests. They may introduce subtle vulnerabilities that static analysis will not catch. The failure mode is silent and delayed.
Don't assume SWE-bench performance transfers to your language stack. Most benchmark data is Python-heavy. If your codebase is primarily Scala, Go, Rust, or C++, treat all published scores as upper bounds on your actual performance. The gap varies by language and is consistently larger than most teams expect.
Don't use agents on tasks requiring business logic that isn't in the code. If fixing a bug requires understanding a rule that lives in a product spec, a Confluence doc, or a Slack conversation from six months ago, the agent does not have that context. It will reason from code alone. That may be wrong.
Context scaling will help, but not linearly. Research on 10M-token contexts is active. The theoretical benefit for large codebase navigation is real. The practical benefit depends on attention quality at those lengths, which degrades non-linearly. Expect continued progress here, but not the step change the context window numbers imply.
Domain-specific fine-tuning is being systematically underexplored. Every company running agents on a large internal codebase should be collecting fine-tuning data now. Agent traces—inputs, intermediate steps, patches, test results, human corrections—are the training data for a company-specific model that will substantially outperform a general-purpose model on internal tasks. Almost no one is doing this systematically. The actual competitive moat is being left on the table.
Enterprise benchmarks are coming. SWE-bench is 12 public Python repos. The next generation of evaluations—currently in development at several research labs—will include proprietary codebases, cross-language tasks, and criteria beyond test passage. Databricks' internal evaluation work is part of this broader push toward more realistic assessment. When those benchmarks publish, expect the leaderboard order to shift.
Agent orchestration will matter more than raw model capability. The difference between a 70% model and a 65% model on a real enterprise task is small compared to the difference between a well-designed agent loop and a poorly designed one. Retrieval architecture, context management, tool use strategy, and error recovery are where performance variance actually lives. Open-source orchestration frameworks—like the multi-model coordination approach that projects such as Omnigent are exploring—make it practical to experiment with these patterns without building evaluation harnesses from scratch.
The evaluation gap will widen before it narrows. Benchmark scores will continue to improve. Real-world performance on enterprise codebases will improve more slowly, because the constraints are structural—internal APIs, test infrastructure, organizational context—not just model capability. The gap between reported benchmark performance and deployment performance is a research problem. It needs to be named as such.
Does a higher SWE-bench score reliably predict better performance on a large internal codebase?
No. SWE-bench measures performance on localized bugs in public Python repositories. Enterprise codebases introduce scale, language diversity, internal API complexity, and test infrastructure constraints that the benchmark does not capture. Treat benchmark rankings as a prior, not a prediction. Run your own evaluation on 20–30 internal tasks before committing to a model.
What is the actual token budget for running an agent on a complex Spark bug?
It varies by task complexity and agent architecture. Straightforward localized bugs: 20k–50k tokens. Cross-module issues requiring context gathering: 100k–300k tokens. Complex performance bugs or refactors: 300k–500k+ tokens. At current pricing, the latter category costs $1–$5 per task on frontier models, which materially changes the economics of autonomous deployment at scale.
Is Gemini 2.5 Pro's 1M context window actually useful for multi-million line codebases?
Partially. You can fit more of the codebase in context, which helps with retrieval. But the model's attention quality at extreme context lengths degrades—content at position 800k receives meaningfully less effective attention than content at position 10k. For most tasks, the practical benefit saturates well below 1M tokens. A well-designed retrieval system that pulls the right 50k tokens will outperform naive full-codebase injection in most cases.
Why do agents specifically fail on internal API conventions?
The model has not seen your internal APIs in training data. When it encounters a call pattern it does not recognize, it synthesizes a plausible-sounding API call based on naming conventions and surrounding context. The hallucinated call looks correct syntactically and may pass linting. It does not exist. The fix is either explicit documentation injection in the agent prompt or fine-tuning on internal codebase data. Neither is trivial at scale.
Should we wait for better models before deploying agents on large codebases?
Not necessarily. The benchmark gap reflects structural problems—test infrastructure, internal API coverage, context management—that better models will only partially address. The more productive path is to identify task categories where agents already perform well on your codebase—localized bugs, boilerplate generation, test writing—and deploy there now, while collecting traces for future fine-tuning and incrementally expanding scope.
What is the single most important thing to measure when evaluating a coding agent for enterprise deployment?
Hallucination rate on internal APIs, measured separately from overall task completion rate. This is the failure mode most likely to slip through code review, cause subtle production bugs, and erode engineer trust in agent output. A model that completes 60% of tasks correctly with zero internal API hallucinations is more valuable in practice than one that completes 70% of tasks but invents function signatures 30% of the time. Completion rate is the metric that gets reported. Hallucination rate on novel APIs is the metric that determines whether you keep the system running six months later.
How should ML researchers think about designing better enterprise benchmarks?
The key structural requirements are: task scope unconstrained by file count, non-trivial test infrastructure that agents must reason about rather than just execute, internal API surfaces not in public training data, and at least one cross-language component. The evaluation criterion needs to go beyond test passage—patch quality, regression risk, and code style adherence are all part of what a senior engineer actually evaluates. Building this is hard, which is why it hasn't happened at scale yet. It is the right problem.
![Open-source single-GPU reproductions of Cartridges and STILL for neural KV-cache compaction [P] image](/_next/image?url=https%3A%2F%2Fbestaifor.s3.eu-west-1.amazonaws.com%2Fopen_source_single_gpu_reproductions_of_cartridges_featured_954b9a265f.jpg&w=3840&q=75)
Neural KV-cache compaction — using learned compression rather than heuristic eviction — is one of the more credible paths to running long-context LLMs without bleeding GPU memory. Cartridges and STILL are two recent...
Frontier model benchmark scores climbed between 8 and 45 percentage points across MMLU, SWE-bench, and ARC-AGI over the past 12 months. Workflow impact is real but concentrated — reasoning tasks and agentic code...