BestAIFor
A Brief History of Artificial Intelligence: From the Beginning to Today
An accessible overview of the history of artificial intelligence, from early theoretical ideas to modern deep learning.
December 19, 2025•3 min read
![Elastic Attention Cores for Scalable Vision Transformers [R]](/_next/image?url=https%3A%2F%2Fbestaifor.s3.eu-west-1.amazonaws.com%2Felastic_attention_cores_for_scalable_vision_transf_featured_7a30bc1faf.jpg&w=3840&q=75)
This paper introduces VECA (Visual Elastic Core Attention), a vision transformer that eliminates direct patch-to-patch attention entirely, replacing it with a fixed set of learned core tokens that mediate all communication — dropping complexity from O(N²) to O(N). The accuracy cost at standard resolution is real: roughly 1.6 percentage points behind DeiT-B on ImageNet-1K. The trade-off only flips once resolution climbs past the point where quadratic attention becomes a deployment blocker, which is sooner than most practitioners expect.
The Vision Transformer scaling problem has been understood for years. The original ViT paper from Dosovitskiy et al. gave the field a universal backbone. DeiT made it trainable without industrial-scale data. Swin Transformer introduced hierarchical shifted-window attention to cut the quadratic cost. EVA, DINOv2, DINOv3 progressively closed the accuracy gap with CNNs. Through all of it, one cost stayed structurally fixed: all-to-all self-attention scales quadratically with sequence length.
At 224×224 with 16×16 patches, you get 196 tokens. Manageable. At 512×512, 1,024 tokens — getting expensive. At 1024×1024, the sequence hits 4,096 tokens. Attention cost scales as N², so you're computing roughly 16.7 million pairwise interactions per head per layer. Multiply by 12 heads, 12 layers, and you're running a number that most mid-range GPUs can't hold in VRAM, let alone compute at useful throughput.
VECA arrives at a moment when practitioners are being asked to run ViT-based models on satellite imagery, whole-slide pathology scans, and dense video frames — exactly the regimes where quadratic scaling is not a theoretical inconvenience but an operational ceiling.
Swin's answer to quadratic scaling was locality: restrict each token to attend only within a 7×7 or 8×8 patch window, then shift windows between layers to allow cross-region communication. It works. But window-based local attention introduces an architectural inductive bias toward spatial locality and adds implementation complexity that creates real pain when deploying on non-standard accelerators.
Token pruning and merging approaches (ToMe, EViT) take a different angle: progressively discard or merge redundant tokens so the sequence shortens through the network. Effective at reducing compute, but you discard information. Whether the dropped tokens matter depends on the task.
VECA takes a third path. Rather than restricting which tokens attend to each other, or reducing how many tokens exist, it restructures how tokens communicate.
The paper introduces a fixed set of C learned "core" tokens — typically 64 to 256 — that persist across all transformer layers. These cores are randomly initialized and trained end-to-end. In each layer, instead of running full N×N self-attention over all patch tokens, the model runs two smaller attention operations:
Because C is fixed regardless of image resolution, total attention cost becomes O(N·C). Since C doesn't grow with N, this is effectively linear — the key mathematical claim in the paper.
Critically, all N patch tokens are retained throughout every layer. Cores don't replace patches or compress them. They mediate. The patch representation stays full-resolution; only the communication pathway is restructured. This separates VECA from earlier cross-attention designs like Perceiver, where the patch tokens are discarded after the initial cross-attention step and you're left with a fixed C-length representation. In VECA, you have all your patches and efficient communication.
VECA-B hits approximately 81.93% top-1 on ImageNet-1K. DeiT-B, the standard non-hierarchical ViT baseline at comparable parameter count, reaches 83.56%. That's a 1.63-point gap — worth stating plainly, because some efficiency paper summaries bury the accuracy cost in qualifications.
The honest framing: at 224×224, you're paying an accuracy penalty to save compute you don't actually need. Standard ViT-B at that resolution runs fine on any modern training cluster. The trade-off inverts as resolution climbs. The paper benchmarks VECA against DINOv3 and reports competitive performance across classification and dense tasks — suggesting the architecture can absorb self-supervised pretraining in the same way standard ViTs do, which is important for anyone planning to use pretrained backbone weights.
For teams evaluating like-for-like on standard ImageNet at 224px, VECA-B is not the right pick if top-1 accuracy is the primary constraint. That calculus changes completely at 512px, 768px, and above.
Dense tasks — segmentation, detection — surface something architecturally important. Performance consistently improves as core token budget increases, while classification accuracy stays comparatively stable across core counts.
This asymmetry makes mechanical sense. Classification collapses the image to a single global prediction. A small set of well-trained cores that summarize global image structure handles this reasonably well. Dense prediction requires spatially fine-grained feature maps — each output pixel or object region needs local context that may not fully propagate through a tight communication bottleneck. More cores → more spatial routing capacity → better dense results.
The practical implication: if you're targeting detection or segmentation workloads and considering VECA, budget toward the high end of the core range. If you're targeting classification, you may be able to run leaner during inference.
Most coverage of this paper focuses on the linear complexity claim. The elastic inference behavior is the part with the most direct deployment value.
The paper's nested training strategy trains the model such that the first k cores carry the most essential information, with each increment of cores adding refinement. Think of it as Matryoshka representation learning applied to the core dimension rather than the embedding dimension. During training, the model periodically runs a forward pass using only the first 64 of 256 cores, with gradients flowing through the active subset. This forces the early cores to be sufficient on their own.
At inference time, you select how many cores to activate. A single trained checkpoint runs with 256 cores on a data center GPU, 128 on an edge server, 64 on a mobile chip. One model, three hardware tiers, no retraining. This is genuinely different from post-hoc pruning (which requires a fine-tuning pass) or knowledge distillation (which requires training a separate smaller model).
For teams managing deployments across environments with different compute budgets — cloud training, edge inference, mobile — that reduction in model management overhead is worth quantifying directly. The same framing applies to quantization-side flexibility: understanding tradeoffs between representation quality and compute at inference time is a theme in recent efficiency work, including FP4 quantization-aware training approaches like DeepSeek V4's, though VECA's elasticity operates at the architectural level rather than the bit-width level.
Don't apply VECA to 224×224 classification tasks where accuracy is your primary constraint. The 1.6-point gap against DeiT-B is real, the compute savings at this resolution are minimal, and you're paying accuracy for nothing.
Don't use small core counts for dense prediction. The paper's own ablations show segmentation and detection degrade more steeply with core reduction than classification does. If you cut core count for inference speed on a detection task, benchmark first — the accuracy cliff may appear earlier than you expect.
Don't treat VECA as a drop-in for hierarchical architectures. Swin Transformer's hierarchical design produces multi-scale feature pyramids that downstream task heads (FPN, UPerNet) explicitly require. VECA, like vanilla ViT, outputs a flat sequence of patch tokens. Task heads built for Swin need modification.
Don't expect strong out-of-box results on tasks requiring very precise spatial localization without benchmarking. Dense depth estimation, instance segmentation with fine boundaries — these need spatial correspondence that full attention handles naturally. VECA preserves all patch tokens (no discarding), which helps, but indirect communication through cores may introduce artifacts. This is a question empirical testing should answer, not the ImageNet and semantic segmentation numbers alone.
Linear attention designs are converging. VECA, RWKV-based vision models, Mamba-style SSMs, ELSA, and Perceiver-IO variants are all attacking the quadratic ceiling from different angles. The next benchmark cycle will need to compare across all of them systematically — cherry-picked comparisons against only standard ViT baselines are becoming inadequate as the alternative space fills out.
Elastic compute will become a deployment requirement. As inference runs on increasingly heterogeneous hardware — cloud GPUs, edge accelerators, mobile SoCs — single-checkpoint models that adapt compute at test time have a structural advantage. VECA's nested training approach is one path. Early-exit transformers and token-budget-conditioned models are others. Expect convergence on some form of inference-time elasticity as a standard feature within the next architecture generation.
High-resolution vision is the next benchmark frontier. ImageNet-1K at 224×224 is a solved problem. Competitive research attention is moving toward satellite imagery, pathology whole-slide analysis, and multi-frame video — all domains where quadratic attention is prohibitive. Papers demonstrating clean linear scaling with competitive accuracy at 1K+ resolution will drive adoption decisions in the next product cycle.
Self-supervised pretraining with linear attention is underexplored. VECA trains against DINOv3 as a teacher — it doesn't run DINO self-supervised from scratch. Whether a core-periphery architecture can generate the rich patch-level representations needed for masked autoencoder or contrastive pretraining without a quadratic-attention teacher is an open question. If it can, compute savings compound across pretraining and fine-tuning.
Core count as a structured hyperparameter opens new research directions. Dynamic core allocation per layer, task-conditioned core selection, learned routing over core subsets — none of this is in the paper, but it follows naturally from the architecture. The analogy to mixture-of-experts (applied to the communication bottleneck rather than the FFN) is close enough that integration with MoE training recipes seems plausible.
Is VECA competitive with Swin Transformer on dense prediction tasks like COCO detection? The paper's abstract and summary results compare VECA primarily against DINOv3 as a teacher, not a direct Swin head-to-head table. For the full benchmark breakdowns including detection and segmentation numbers, the complete tables are in arXiv:2605.12491. Swin remains a strong hierarchical baseline for multi-scale dense prediction at standard resolution. VECA's structural advantage over Swin shows up at high resolution, where Swin's window computation still scales and VECA's core count stays fixed.
Does adopting VECA require rewriting the attention module from scratch? Yes, with caveats. The core-periphery attention replaces the standard self-attention block with two sequential cross-attention operations, plus management of the core token state across layers. That's a real implementation change. The patch token input/output interface is preserved, so task heads designed for standard ViT outputs should be compatible — but you can't swap in a VECA attention layer by changing two lines in a standard transformer codebase.
How does the nested training actually enforce the core hierarchy? The model trains such that the first k cores are sufficient for a functional forward pass with acceptable performance. During training, the model periodically runs with a prefix of the full core set (e.g., the first 64 of 256), with gradients flowing through the active subset. This enforces that early cores carry the most essential signal — similar in spirit to Matryoshka Representation Learning, applied to the core dimension rather than the embedding dimension.
What does "competitive with DINOv3" mean in concrete numbers? The paper uses DINOv3 as a distillation teacher and reports that VECA reaches comparable downstream task performance. "Competitive" here means the linear-attention student approximates the teacher's quadratic-attention representations well enough to close most of the gap on the measured tasks. The exact per-task deltas are in the paper's tables — the claim is architectural, not that VECA beats DINOv3 outright.
Can this approach be applied to language models? Core-periphery attention is architecturally general, but language has different demands. Vision scenes have a roughly stable set of "things worth summarizing" (objects, textures, spatial layout). Natural language salience is highly context-dependent. A fixed C-core bottleneck would need to route very different information across diverse text distributions. Linear attention for language (RWKV, RetNet, Linear Transformer) is active and somewhat mature, but those designs are built differently — not by adding cores to standard attention. Porting VECA to language needs empirical validation before any claims.
Is the 87.1% reduction in attention interactions a practical throughput gain? Theoretical complexity reduction and wall-clock speedup don't map directly. GPU kernels are optimized for large dense matrix operations; smaller N×C matrices may underperform relative to optimized N×N flashattention kernels, depending on batch size and whether a custom CUDA implementation is provided. The paper's reported throughput numbers (if published) are the right metric to check — the theoretical reduction is necessary but not sufficient evidence of real-world latency improvement.
Are there released model weights or training code? No released checkpoints or official code repository appear available as of the paper's May 2026 arXiv submission. Watch the arXiv page for supplementary materials or linked GitHub repos. This is a common gap in preprint releases that resolves within weeks for papers that attract community attention.
An accessible overview of the history of artificial intelligence, from early theoretical ideas to modern deep learning.
Machine learning vs deep learning explained with clear differences, real world use cases, and guidance for beginners and professionals
Z.ai’s GLM-5 scores 77.8% on SWE-bench Verified and 62.0 on BrowseComp, nearly doubling Claude Opus 4.5’s 37.0. First open-weights model above 50 on the Artificial Analysis Intelligence Index.
ARC-AGI-3 launched March 26, 2026. Every frontier model scored below 1%: Gemini 3.1 Pro Preview led at 0.37%, GPT-5.4 at 0.26%. Here’s what the interactive agentic benchmark reveals about current AI reasoning limits.
Z.AI's GLM-5.1 scored 58.4 on SWE-Bench Pro, edging GPT-5.4 and Claude Opus 4.6 by less than 1.1 points. The benchmark lead is real — the hardware requirement to run it locally is not consumer-grade.