inference
84 posts tagged inference.
- The CMP 170HX A100 unlock is real, useful, and not a shopping list
- Apple’s rumored local AI app is really a packaging test
- B3 speeds up diffusion language models by overlapping the slow part
- FP4 KV Cache Done Right: What BitDecoding-FP4 Changes for Long-Context Serving
- Tencent’s Qwen DSpine hint is about inference cost, not model size
- The RTX 5060 Ti 16GB local AI signal
- Used A100s may be the first real crack in the GPU squeeze
- StepBack targets the first bad reasoning step instead of rerunning the whole chain
- LoopVQ and the Hidden Memory Cost of Looped Transformers
- Zyphra's tree-structured attention aims at million-token-plus context
- Neural Thickets May Be an Inference Trick, Not a Weight-Space Discovery
- SpecSlot splits image tokens by what needs global attention
- TextSeal2 keeps watermarks alive under speculative decoding
- Hugging Face points open AI back at the iPhone
- Hexels push 3D reconstruction toward fewer primitives
- dSpark and the Order Problem in Diffusion Language Models
- AdaBPE makes tokenization care about LaTeX
- Cerebras Says It's Running Trillion-Parameter Models: What That Actually Means for Builders
- What Samsung's On-TV TensorFlow Lite Tracker Tells You About Edge AI
- Pruning reasoning candidates by optimizing pass@k directly
- Adaptive Depth in Recommenders: What LoopRec Changes
- Adaptive MTP makes speculative decoding less wasteful
- LatComm makes long context reusable across models
- MiniMax M3.1 Flash puts Space Bunny back under test
- TorchNeuron points at the real Trainium bottleneck: developer friction
- 552B parameters on one RTX 4090 is an inference trick, not a free lunch
- Local LLMs do not always need a four-GPU box
- What DeepSeek V4.1 Flash on Dwarfstar tells us about SSD-streaming local models
- What OpenAI's 'ultrafast' service tier for Agents actually signals
- ORL Makes Diffusion LLMs Look Practical for Low-Latency Inference
- PathFM Treats Forecasts as Paths, Not Quantile Spreads
- Reading OpenAI's SDK Changelog: Caching, Voice Cloning, and the Mystery of 'Decisions'
- DoubtLess uses a small model to make a big one stop overthinking
- Latent Thinking, Text Acting: What HybridAgent Changes About Agent Cost
- Temperature Should Read the Room
- On-device agents are a systems bet, not just a model release
- 100B-parameter phone models are a battery and memory bet
- Tokenizer compression is not the same as multilingual quality
- Asana’s browser-agent test points to the real AI cost story
- GPU scheduling is where AI infrastructure gets real
- Running Anomaly Detection on the Satellite Instead of the Ground
- GALA makes Gaussian avatars lighter by distilling animation into shared blendshapes
- SILSA compresses 3D generation by slicing the shape, not voxelizing everything
- Looped Mixture of Experts: Two Efficiency Tricks That Finally Stack
- LeapQuant targets the hidden inference cost in linear attention
- Not All Quantization Errors Are Equal: STEPQuant and the Memory Cost of Linear Attention
- Foil's Recipe for Looped MoE: Flatten the Experts, Untie the Attention
- OpenAI's DevDay agents move the product boundary from chat to operating system
- Self-Improving Agents Without RL: What Retrospection, Creativity, and Token Forecasting Actually Show
- Telescopic Language Models: one training run that serves every compute budget
- Teaching a Reasoning Model to Know When It's Sure Cuts Its Token Bill
- Free local AI models are not charity
- Prompt lookup drafting gets faster in llama.cpp, but the win is workload-specific
- Qwen architecture rumors do not make a 3090 fast by default
- Ling 3.0 Tiny makes old CPUs interesting again
- Swift Qwen’s speed claim is a local AI reminder, measure the whole loop
- MISVO steers frozen language models at inference time
- A Small Model That Steers a Bigger One's Reasoning
- CliffCompaction makes long coding runs cheaper by refusing to summarize
- Softmax attention needs an off switch
- CXMT’s reported memory ramp matters more for inference than model hype
- LACE compresses speech tokens one codec layer at a time
- The Skill Router You Already Have: Gavel Reads Routing From a Frozen LLM
- Local LLMs are getting useful because constraints are back
- ConvMem Turns Long-Context Reasoning Into a Tree Instead of a Chain
- Declarative attention lets models skip the context they do not need
- Reasoning Without Tokens: What Soft Latent Thinking Actually Changes
- Synthetic data needs a weight limit
- llama.cpp’s CPU backlog is a map of local AI’s next gains
- Qwen 3.8 27B and the local OCR cost test
- Your Local LLM Isn't Dumb, Your Defaults Are
- A Strong Model Can Scaffold a Weak One Without Any Retraining
- BDH-CQ Makes ARC Reasoning Cheaper by Thinking in Latent Space
- A 16-node DGX Spark cluster at home: what running trillion-parameter models locally actually takes
- DeepSeek-V4-Flash on a 3090 shifts the bottleneck to DDR5
- DeepSeek-V4-Flash on a Mac is an I/O story, not a parameter-count story
- Kimi K3 on 8 GB RAM is a systems lesson, not a serving plan
- llama.cpp support is becoming the real local AI distribution layer
- The Qwen3.8 rumor is really a VRAM planning signal
- Mesh LLM makes distributed inference a networking problem
- vLLM 0.25 Deletes PagedAttention, and the Transformers Backend Catches Up
- Long-context KV caches are getting selective
- Local LLMs are becoming a workflow choice, not a hobby project
- OrbitQuant makes diffusion quantization less tied to calibration sets