AI Agents & Evals
AI agents get hyped as autonomous coworkers and dismissed as glorified chatbots. Both miss the point. An agent is a loop: a model that calls tools, hits errors, retries, and sometimes finishes the job. What separates a demo from something you'd run in production is rarely the model. It's the harness, the tool design, and whether you can actually measure when it works. That's why agents and evaluation belong together here. I track the architectures that hold up, the eval methods that catch failure before your users do, and the honest gap between a slick demo and a system that ships.
410 posts
Recent articles (30)
- Amplitude's Wave and the agent loop that starts before anyone asks for a feature OpenAI says Amplitude's Wave agent finds what to build, drafts the change, and measures the result in days, a full-loop pattern that most teams still run as four disconnected jobs with humans stitching the gaps.
- OpenAI’s $2M Child Safety Research Bet Is Really About Measurement OpenAI’s child safety funding is small next to model training budgets, but the useful signal is the focus on shared measurement tools for youth risk and benefit, not another vague trust-and-safety pledge.
- Contrastive Discovery: Why Science Agents Should Weigh Rival Explanations, Not Just Reject One A new arXiv paper argues that LLM discovery agents waste most of their evidence by testing one hypothesis at a time. Swapping the single claim for a set of competing explanations resolves far more questions on the same budget, and human experts mostly agree with the answers.
- ReTrace turns repeatable agents into code paths ReTrace points to a practical middle ground for business automation: use an LLM agent to discover the workflow, then compile its traces into cheaper, faster code that handles repeat runs without planning every step.
- Tether’s BotPsy points to smaller phone agents, but the proof is still thin Decrypt reports that Tether’s 800 million parameter BotPsy agent can control phone tasks locally and beat larger models. The interesting part is not the crypto angle. It is the claim that small, task-shaped agents may be enough for everyday device control.
- What Your Agent Evaluation Harness Can Actually See A three-phase study of enterprise analytics agents shows that the information you feed an evaluation harness decides whether it catches real failures, and that functional testing and security red-teaming find completely different bugs.
- Agentic-OPD says your best RL checkpoint is not the whole model Agentic-OPD argues that discarded RL checkpoints contain capabilities the final checkpoint forgot, and shows a practical distillation method for recovering them without rerunning training. The useful idea is simple: treat an RL run as a policy library, not a winner-take-all race.
- Dense rewards made clinical agents ask better questions A paper on clinical multi-turn agents shows why final-answer rewards can hide bad procedures, especially premature stopping and missed rare cases, and why builders need process-aware reward design plus evaluations that inspect the path, not just the final outcome before deployment.
- HiVe-Mem and the Case for Two-Dimensional Agent Memory A new multimodal memory framework called HiVe-Mem claims a 90 percent token cut by rethinking how agents retrieve past interactions, splitting the problem into broader coverage and query-adaptive evidence rather than flat semantic search.
- Near AI’s breach shows where agent privacy really breaks Decrypt reported that an SGLang flaw exposed Near AI user details and data from 18 agents, a useful reminder that privacy claims fail at the weakest operational layer, not the most impressive cryptographic one.
- A Generalist Model Beat Specialist Robot Policies Without a Single Demo The AMP1 paper claims a generalist AI commanding a robot arm directly beats trained vision-language-action policies zero-shot, cutting failure rates up to 20x and learning from memory across episodes. Here is what that actually means for builders.
- Radiology agents are only as good as their tool catalog An arXiv paper on agentic radiology reporting finds that orchestration matters, but missing FDA-cleared tools by modality are the bigger blocker to comprehensive AI-assisted reports.
- A World Map of LLM Answers Is an Eval, Not a Belief System The llm-beliefs project asks models whether censorship exists in each country, then maps the answers. It is a useful reminder that simple political questions can expose model behavior, but not cleanly separate knowledge, refusal policy, prompt sensitivity, and dataset residue.
- Agent memory is architecture, not a chat history The useful way to think about agent memory is not as one database or longer context window, but as a set of separate systems with different jobs, failure modes, and product choices.
- Cloudflare’s agent hints look like SEO’s next control layer Cloudflare’s CEO is framing AI agent traffic as a new SEO problem, but the real shift is not keyword tactics. It is negotiation over crawl access, freshness, intent, and whether bots can explain why they are visiting.
- Local AI on 16GB VRAM is a scope problem, not just a model problem A r/LocalLLaMA thread about running useful local agents on 16GB VRAM gets at the real bottleneck: local AI can work, but only when the workflow is narrow, testable, and built around tools instead of vague autonomy.
- Local GUI agents need disposable desktops, not prettier demos A LocalLLaMA report on Open WebUI and Open Terminal points to the useful part of computer-use agents: local models driving observable, sandboxed desktops that humans can interrupt, inspect, and reset.
- OpenAI's LifeVal and the gap between knowing and doing OpenAI says it built LifeVal to measure how well its models handle everyday personal tasks like meal planning and grocery ordering, not just coding and math. Here is what the eval signals, what is still unverified, and how a builder should read it.
- Same outputs are not enough for AI mechanism audits The arXiv paper on integrated information is a useful warning for AI audits: finite experiments can certify some mechanism claims, but behavior alone and partial access can make the important differences invisible.
- The Label Wording Problem Nobody Tests in Agent Routers A study of a 307M Russian cross-encoder shows that how you word your routing labels moves accuracy by up to 0.24 points, and that the fix works for plain words but fails cold on abbreviations and codes.
- AI exploit generation is getting cheap enough to change patch work Decrypt reported that an automated OpenAI system found 948 Linux security flaws, produced working exploits, and drafted fixes at about $1,279 per flaw. The useful takeaway is not panic. It is that security teams need to treat AI as a patch prioritization and exploitability engine.
- Animal ID Models Need Evidence, Not Just Retrieval Scores A paper on animal re-identification shows why retrieval accuracy is not enough: builders need evidence profiles that separate real foreground signal from background shortcuts, especially when datasets become shared benchmarks and foundation-model training fuel. The CFR protocol is a useful pattern for evaluating vision retrieval before leaderboard scores become fragile product claims.
- Forecasting an LLM’s answer before token one Context activation traces may let evaluators estimate how a model is likely to behave before it starts generating. The useful idea is not mind-reading, it is cheaper behavioral forecasting for reliability work.
- Richer Context Does Not Automatically Fix LLM Social Agents A new arXiv framework tests whether richer context makes LLM social agents less brittle, and the answer is useful but awkward: stability depends on the kind of context, the social change being tested, the model underneath, and not just prompt length.
- What Anthropic's SDK Error Codes Tell You About Running Claude in Production Two quiet releases of the Anthropic Python SDK add structured error codes, thread-safe parsing, and path-injection guards. Here is what those changes mean for anyone running Claude agents at scale, and why error handling is where agent reliability actually lives.
- An Independent Replication of Direct Nash Optimization, and What Its Ablations Actually Prove A new empirical study re-implements Direct Nash Optimization on Llama 3.1 8B and Mistral 7B, showing an 8B model can beat Llama 3.1 70B on AlpacaEval while quietly admitting the circularity that inflates the headline numbers.
- Cognitive symbionts shift the agent question back to the human The arXiv paper on cognitive symbionts makes a useful distinction: the next agent frontier may not be more autonomy, but better augmentation of one person’s memory, attention, context, and decisions over time.
- Fine-tuning can break alignment without harmful data The arXiv paper “Knowingly Wrong” argues that training a model to say things it still internally rejects can create broad misalignment, even when the training examples are harmless.
- Fixed Personas Break on Context in Synthetic User Tests A paper titled Personas Are Not People argues that simulated users built from fixed profiles miss context-driven behavior, especially in communities where identity and norms shift by situation. The useful move is not bigger personas, but person-in-context tests with real validation before trusting outputs.
- MedPhysBench Tests Whether Medical AI Can Catch the Mistakes It Can Solve MedPhysBench tests language models on medical physics calculations and error detection, exposing a practical gap: some models can solve a task while missing the same mistake in someone else’s work. That matters for clinical QA, agent design, and any high-stakes workflow.
Earlier articles (379)
- The OCC puts AI agent autonomy on the bank examiner checklist
- Websites need an agent policy before agents become customers
- DiabAgent tests medical agents where mistakes compound
- Google Search Is Becoming an AI Agent Manager
- Strategist treats agent reasoning as code that can evolve
- WebHarbor Puts 153 Fake Real Websites in a Box for Agents to Train On
- Claude-written Linux exploit puts patch lag in the spotlight
- KLM Wants AI to Show Its Evidence. Its Own Negative Results Show Why That's Hard.
- LangChain’s Python 3.10 cutoff is a maintenance warning for agent stacks
- LifeVal puts AI evals closer to real paperwork
- Claude, game reverse engineering, and the product boundary problem
- Factorio Peaceful Mode Is a Useful Agent Test, Not a Finish Line
- Malware using LLMs for victim triage is the practical AI security problem
- Memdoor's Claim That Five Small Models Beat One Big One on Hard Coding
- Reward hacking is the failure mode behind too many shiny AI demos
- LangChain’s tiny core releases are really about state discipline
- The Economics of Proactive AI: When a Single 1B Decoder Beats a Two-Model Pipeline
- Verifying Computer-Use Agents by Letting the Verifier Go Look
- ChatGPT browsing access is now part of site visibility
- OpenAI and ServiceNow move agents closer to enterprise work queues
- What OpenAI's Misalignment Detection Post Actually Describes
- Hidden AI Opinions Can Steer Expert Analysts
- Latent world models are passing tasks without learning the world
- What LLM Group Memory Keeps, and Why It Still Gets the Answer Wrong
- The .agent domain fight is about AI distribution, not vanity
- Inbox agents need labels before models
- SemantEq makes formal-math evals less easy to game
- When an LLM's 'True' and 'False' Don't Sum to One
- When a Benchmark Claim Has No Benchmark Behind It
- Claude Duet tests the proactive assistant, not a new chatbot
- OpenAI’s Dots demo turns bug triage into an agent workflow
- Agents fail when true tool outputs omit the one fact that matters
- CausalCoT treats chain-of-thought as a dependency graph
- Medical exam scores are not enough for clinical LLMs
- METTA splits agent memory into moments and episodes
- RAG Scores Move When Ground Truth Gets Better
- When Deferring to a Human Beats Letting the LLM Guess
- ASTEROID teaches agents to explore before they act
- Quant-Arena Tests Whether LLMs Can Actually Build Trading Strategies
- Why the Best Solver Model Is Rarely the Best Environment Author
- The Teacher That Scores Worse Than It Plays: Fixing Agentic Distillation
- What Arabic Dialect Math Problems Reveal About Model Reasoning
- When AI Agents Start Finding Bugs Faster Than Maintainers Can Patch Them
- AfriBED makes speech bias measurable in Bemba and Nyanja
- LatComm makes long context reusable across models
- WeAct Turns LoRA Adapters Into Something an Agent Can Pick Mid-Task
- WildAWM trains computer-use agents in synthetic worlds
- An AI Agent That Reproduces Cancer Studies and Checks Where They Wobble
- Reasoning distillation works better when the trace fits the student
- What OpenAI's 'ultrafast' service tier for Agents actually signals
- StepFun’s agent phone is a bet against the app grid
- FDE-Gym Puts Agents in the Messy Last Mile of Customer Delivery
- MLExpert and the Case for Training Agents on Whole Experiments, Not Single Turns
- When Models Behave Differently in Hindi Than in a Safety Test
- CoPE trains agents to forecast their own future actions
- The Case for Making Research Agents Forget Their History
- When Renaming a Tool Breaks Your Agent: What CIPR Fixes
- ChunkAct moves robot agents from constant planning to reusable routines
- Latent Thinking, Text Acting: What HybridAgent Changes About Agent Cost
- Self-Evolving Agent Harnesses Need Change Control More Than Hype
- Tev's Dense Progress Rewards and the Sparse-Reward Problem in Long-Horizon Agents
- Gemini 4 Argon needs harder evals than its launch pitch
- On-device agents are a systems bet, not just a model release
- When the Vulnerability Scanner Becomes an Attack Planner
- Agent red teams need maps, not just scary prompts
- Hide and Seek tests black-box vision models by removing the thing you care about
- Tabular regression still punishes lazy early stopping
- Tool retrieval should know when the tool is broken
- Marketing AI has budget before it has proof
- Oracle’s Codex story is about killing the report backlog
- A chatbot reliability formula is only useful if it predicts your failure mode
- Accuracy is not enough when agents meet conflicting evidence
- Legal AI citations are not proof of legal reasoning
- OnTrack watches agent trajectories before the bill is paid
- ViSkill and the case for keeping agent memory visual
- Alignment generalization is a behavior problem, not a values checklist
- Asana’s browser-agent test points to the real AI cost story
- Manus, Meta, and the new deal risk around AI agents
- Probes That Catch an Agent Lying Before It Says a Word
- What Sophos actually automated to cut threat response from 38 minutes to 89 seconds
- OpenAI’s withdrawn math results are a reminder that AI math needs audit trails
- SciExam Tests AI Agents on Climate Modeling Without Giving Them the Answer
- Scoring LLMs When There Is No Right Answer: A Lesson From Survey Economics
- When a hallucination enters the context, can the model talk itself out of it?
- AI-assisted breaches are becoming an operations problem
- Co-Evolving the Attacker: How AdvSim2Real Hardens Web Agents Against Prompt Injection
- CLIFT turns web-agent judging into reusable verification
- MemPilot and the Case for Deciding Memory at Query Time, Not Ingest Time
- Robot agents need indexed video memory, not bigger prompts
- RL Environments on Hugging Face Make Agent Work Less Hand-Wavy
- Agent memory is not a substitute for readable docs
- LeCun’s “zero concerns” is the wrong bar for AI safety
- The risky lesson in Meta Muse’s reported household-authority prompt
- When an Agent Reports Success and the Database Says Otherwise
- A simulated canvas turns vision models into operators
- Stratego Is a Better Agent Test Than Another Chess Win
- KaliBench Measures Whether LLMs Can Actually Type the Right Security Command
- OpenAI’s Python SDK is getting more agent-shaped
- ScholarCatalyst: When Agentic Search Can't Beat a Plain Embedding
- Source learning makes RAG less forgetful
- DynaHarness puts a governor between robot planning and robot motion
- Prompt optimization should chase ranking, not accuracy, in imbalanced clinical AI
- The multilingual loophole in LLM unlearning
- AdviSD trains a small model to steer bigger frozen models
- Correct math answers can hide broken reasoning traces
- Robinhood’s trading agents make permissioning the product
- Robots Need Correction Loops, Not Just Better Planners
- The Harness Is the Model: What 'Meta-Skills for Agent Harness Design' Actually Shows
- KV-streams make long-agent training cheaper by not throwing memory away
- One uncertainty layer for graph neural networks
- OpenAI's DevDay agents move the product boundary from chat to operating system
- Australia’s AI agent hearing is really about incident disclosure
- Flow matching gives neural dissimilarity metrics one common frame
- The useful part of Roetzer’s architect-orchestrator-apprentice AI work model
- What LLMs Miss About Haitian Creole, and Why Low-Resource Culture Breaks Evals
- When Documentation Doesn't Help Coding Agents: A Negative Result Worth Reading
- Agent control failures are moving from demo risk to operating risk
- Drawgent puts a coding agent on an Excalidraw canvas: what a visual work surface actually changes
- AI agents cut the estimated cost of quantum-safe Bitcoin transactions
- NSA’s reported AI testing spend makes evals look like infrastructure
- Ollaya points at a missing layer for decision models
- When Agents Hack: What the OpenAI-Hugging Face Story Actually Tells Builders
- AI Agents Make Audience Data Debt More Expensive
- Bitcoin Lightning joins x402, and agent payments get a little more real
- Conspiracy Detection Needs Context, Not Just Better Keywords
- ExplorationBench and the Gap Between Recall and Real Discovery
- When a model’s rejection reason changes its next choice
- Australia’s OpenAI agent breach is a disclosure problem, not a sci-fi one
- Multi-agent shutdown sabotage is a real eval target now
- SWE-Flux: The Benchmark That Asks If Coding Models Can Predict What Code Actually Does
- The World Model That Edits the Agent's Mind, Not the Environment
- Translation Fine-Tuning Breaks the Controls General Evals Miss
- When a Model Gets Math Right but Reads the Same Problem Differently
- AI agents need payment rails before they need crypto hype
- Fact-checking agents need receipts, not vibes
- Known puts domain identity into the agent stack
- Local tool-use evals are measuring your server too
- Long-context models can miss the answer sitting behind nearby noise
- Speaker-Centered Memory: Why Group Chats Break Your AI Agent
- Critical-State RL trains the one agent call that actually matters
- DolphinBench Tests Agent Memory by Task, Not Trivia
- Gemini’s reported company breaches show the AI disclosure gap
- Iterative unalignment tests the failures too rare for normal evals
- Jev shows why science workflows need semantic evals, not just final-answer grading
- Long-horizon agents can learn to cheat the checker
- Where Harness Self-Improvement Actually Stands in Late 2026
- DiaVLo Turns Vision-Language Model Failures Into Named Behaviours
- Full-Duplex Voice With Tool Calls: What NemotronLabs VoiceChat Actually Ships
- Multi-hop RAG needs confidence before the answer
- OpenAI wants shared AI standards, but the hard part is enforcement
- OpenAI’s math advisory group is about claim control
- Decontamination reports are a promise, not a proof: the case for evaluation-side benchmarks
- A cipher win is not an eval without the working
- Hex, GPT-6 Astra, and the shift from answers to visual reports
- LangChain 1.4.2 fixes a small but real agent reliability problem
- What Two Boring openai-python Patches Reveal About Agent Reliability
- LangChain’s typesafe alpha points at safer model routing
- The Harness Matters as Much as the Model in Coding Agents
- Toxicity Scores Can Miss Sanitized Bias in GPT Outputs
- When a Coding Agent Drives a Robot, Task Success Isn't Safety
- When the Teacher Knows to Quit: RetireOPD and Distillation for Agents
- Agent memory helps after the action loop stops breaking
- Probabilistic Linear Explanations make interpretability more honest
- When Rewording the Answer Key Reshuffles the Leaderboard
- ENCP gives navigation agents a usable uncertainty check
- ScienceBuddy and the case for training the harness before the model
- Teaching Models to Say 'I Don't Know' With a Prompt, Not a Retrain
- K-Bench tests mental health chatbots where generic safety evals do not
- LLM personas fail when opinions have to change
- The Skill Router You Already Have: Gavel Reads Routing From a Frozen LLM
- A vulnerability is not fixed because an AI bot saw it
- Siri as a model router, not a single assistant
- What Fyxer's AI inbox assistant gets right about trust
- When Agents Lie to Pass the Eval
- AI math systems are still optimizing for the wrong proof
- Devin testing its own work with GPT-6 Astra: what's real, what's reported
- GPT-6 Astra and the problem with moving-target models
- Perplexity Handing GPT-6 Astra Production Access: What OpenAI's Claim Actually Means
- The RubyGems agent report is a supply-chain warning
- An 86% Bitcoin quantum benchmark cut is not an 86% attack
- DataShifts turns distribution shift into an error-budget problem
- GEO needs exposure data before it can claim revenue
- Recursive Self-Improvement Has a Roadmap Now, and Most of It Isn't Built
- The Hallucination Detector That Doesn't Transfer to Your Domain
- CEO clone chatbots expose the limits of personality wrappers
- Instruction vs. Example: How Vision-Language Models Actually Moderate Content
- OpenAI training-data accusations need a provenance test
- The Benchmark Says GPT-5. Your Users Get a Serving Route.
- Browser agents are becoming practical comment analysts
- Can AI Agents Do Their Own Interpretability Research? A New Benchmark Says Not Yet
- OpenAI’s reported Millennium math claim needs proof, not applause
- Procedural Graphs give agents a memory of what to do, not just what happened
- SPINE Shows Sycophancy Gets Worse When Users Keep Pushing
- OpenAI's Python SDK Gets More Honest About Agent Failures
- How a 9B model learned to chain Korean government APIs by actually calling them
- LexFlip Shows Legal Meaning Metrics Are Still Token Counters
- When You Upgrade the Model, Does the Agent's Memory Come With It?
- Agent Harnesses Are Becoming a Token Efficiency Fight
- AI can help with PCB design, but it cannot own signoff
- GPT-6 Astra's Big Claim: Agentic Skill Meets Alignment
- What LangChain Core 1.6.2 Fixes for Agent Builders
- Compiling a Prompt Into a Small Neural Function You Can Version
- ESPO's fix for prompt bloat: diagnose errors, then stop appending rules
- Hosted LLM judges are not fixed instruments
- The OpenAI agent hack report is a disclosure problem first
- AICOME works best when the missing variable is narrow
- Bare assertions are a weak spot for medical reasoning models
- Repo-distilled skills are the missing middle layer for research agents
- Web agents need useful predictions, not prettier page snapshots
- When a Language Model's Odds Don't Add Up
- AI agent interviews are a workflow fix, not a content strategy
- Grading Coding Agents on How They Solve, Not Just Whether They Pass
- Proactive writing agents need timing more than autocomplete
- SAGE Uses a Big Model as a Coach, Not a Crutch
- Verbal reinforcement learning is a feedback routing problem
- Gemini’s agentic video framing shifts the hard part from seeing to checking
- The Evaluate-First Turn in AI Research Agents
- Reward choice changes how LLM forecasters are wrong
- Small dialogue agents need repair loops, not just bigger training
- Synthetic data needs a weight limit
- A Minecraft clone is a weak coding-agent test
- What Happens When You Let AI Agents Build Their Own Society
- Agent payments are a product problem before they are a currency problem
- Terminal-Bench 4.0 and the eval gap for smaller coding agents
- Treat the smart TV like an untrusted computer
- WikiSkill and the Case for Giving Agents a Memory That Compounds
- Anthropic’s Python SDK is tightening the agent plumbing
- Correct SQL answers can still hide broken agent work
- SwarmWorld makes a case for agents that coordinate through artifacts
- VBVR-Pro makes visual reasoning a training loop, not a demo
- Claude Code as a domain renewal research assistant
- Recuris and the Case for Memory That Rewrites Itself
- The Retrieval-Integration Gap: When Your AI Analyst Reads the Risk and Ignores It Anyway
- Prime Agent treats the harness as part of the model
- Gemini and Apex point prediction markets toward brokerage plumbing
- Verification becomes the operator loop for AI-built silicon
- A 36-node DGX Spark homelab points at agent infrastructure, not just bigger inference
- Munder Difflin and the real work behind AI clone offices
- Treat the Texas AI hacking story as a disclosure systems test
- What LangChain's perplexity 1.4.1 patch says about agent plumbing
- Agent memory works better when skills are small and written in text
- Binance Agent OS puts trading agents behind user-controlled gates
- Self-improvement audits need a measured null
- Task Model Induction turns messy screen traces into reusable agent skills
- What AI4AI-Bench Says About Recursive Self-Improvement Right Now
- When a Zero-Shot LLM Ties a Random Forest on Travel Behavior
- Measure AI precision before you trust AI capability
- SPADE Makes the Training Environment a Thing the Model Learns to Build
- The Self-Improving Agent Demo That Falls Apart When You Shuffle the Tasks
- AutoSR turns symbolic regression into a research-state search
- CaliBench tests whether video models roll fair dice
- PIHF turns prompts into a versioned policy layer
- Stop Making One Prompt Read the Sources and Cast the Vote
- AI coding feels like managing a very literal junior engineer
- What Actually Makes a Claude Code Session Productive
- AutoDesign turns paper-to-poster into a harness the agent rewrites itself
- OmniScientist argues that AI scientists need eyes, not just workflows
- A Strong Model Can Scaffold a Weak One Without Any Retraining
- When Your User Simulator Is Secretly One Person: Simulator Collapse in Multi-Agent RL
- AMIE’s video consult result is about perception, not replacement
- Auto-research agents need fuzzer-style feedback
- The TTS Score That Hides Your Voice Model's Real Problems
- PsychoAgent makes memory retrieval less purely semantic
- SABRE turns VLM evals into a repeatable stress-test pipeline
- SkillProx and the Case for Agents That Prune Their Own Playbooks
- The 2011 Link That Died on Schedule, and What It Says About Agent Memory
- Kitesurf and the browser built for agents, not humans
- TutorMoments and the Hard Part of AI Tutoring: Knowing When Not to Answer
- Agnostic PAC learning gets its optimal bound
- Benchmarks Need QA Before They Judge Agents
- Heart-failure feature engineering gets an agent pipeline, not a chatbot
- RAG For Table-Heavy Reports Needs Search You Can Audit
- Selective Trust: Why RAG Systems Should Learn When to Ignore Their Own Context
- The Harness Is the Product: What HarnessOpt-Bench Actually Measures
- OpenAI’s cyber eval issue is really a process story
- Video deep research agents need to look before they search
- What 'Test-Time Scaling' Actually Means When You Read a Benchmark
- DNS identity for AI agents needs more than a name
- LiveMem reframes long-context memory as state continuity
- Where multimodal embeddings and collaborative coding agents actually stand
- DungeonBench puts tactical reasoning where agents usually break
- ExtractBench tests document extraction where demos usually fail
- On-policy imitation helps when the student is smaller than the expert
- Post-training is now the behavior layer
- The useful question behind a multiplayer agent harness
- Claude’s test escape is a security design problem, not a sci-fi story
- Computer-use agents need stricter judges, not prettier demos
- ORCA-bench shows oncall agents are not ready for pager duty
- When You Count the Tokens, Self-Reflection Loses to Just Sampling More
- AI research agents can code, but they still can’t judge the work
- APEX-Accounting: The 2.6% Number That Should Scare AI Bookkeeping Startups
- The Office-Task Benchmark That Prices Agents Against Human Labor
- πR² makes robot policies react inside the action chunk
- Autonomous research needs a budget scoreboard
- ChatGPT Work turns sales AI into a revenue feedback loop
- On-Policy Distillation Is Becoming the Default Move for Agent Training
- CausalForge makes AI research agents prove their work
- Coinbase’s AI agent bet is payments plumbing, not an AI pivot
- Anthropic's SDK Just Named Claude Opus 5 and Added Mid-Stream Tool Swaps
- Treat rogue AI hacker stories as incident reports, not movie trailers
- GS-Agent Builds 4D Worlds by Driving a Physics Engine, Not Replacing It
- LangChain’s gateway env var is production plumbing
- Training Agents Inside the Harness They Actually Ship With
- World’s $52.5M bet on proof of human for agents
- PoTRE makes the case for heterogeneous test-time reasoning
- Safety bounds are becoming probabilities, not vibes
- LangChain’s xAI update is really about model contracts
- When a Model Eval Turns Into an Actual Breach
- No Best Harness: Automated Discovery Systems Fail to Generalize
- O-VAD treats factory video anomalies as object histories
- CRAFT Turns Rubrics Into a Fine-Tuning Map
- Goal prompting is not a solver for NP-hard search
- Kimi K3’s SpreadsheetBench win is a signal, not a coronation
- ChatGPT’s computer control turns the browser into the new agent runtime
- Kimi K3 and the narrow win that matters for Next.js builders
- Shopify’s agent lesson is delegation, not autonomy
- Paper Revisions as Training Data: What SciDiagramEdit Gets Right About Figure Editing
- The One-Shot Trap in Agent Optimization
- Bonsai 27B makes local agents smaller, not magically smarter
- Memory Failures Hide Behind Correct Answers: What MemOps Exposes
- PalmClaw and the Case for Tool-First Mobile Agents
- Self-repair in small code models may be measuring retry form, not error content
- Useful work per dollar is the agent metric that matters
- Voice AI needs a human-quality test, not another pretty demo
- Judge Bias Lives in the Activations, Not Just the Prompt
- What a Model Knows About What It Knows
- IoT exploit agents are getting useful in controlled labs
- Vision models are getting the scene right, but not always the gaze
- Intelligence Is Still Not the Product
- What Stampli's 'one person doing four people's work' claim actually shows
- DiaLLM separates dialect understanding from dialect writing
- LangChain’s small July fixes point to bigger agent runtime problems
- GaP treats robot policies as editable graphs
- Verification as a Scaling Axis: What the LLM-as-a-Verifier Paper Actually Changes
- LangChain’s Mistral update is about provenance, not flash
- LangChain’s OpenRouter header fix is a production clue
- MiniCPM5-1B points at the phone-sized agent layer
- The AI job diamond still needs a bottom rung
- The Model Got Smarter and My Tool Got Dumber
- Speaker recognition is a better agent test than another chat demo
- Unlearning That Hides Instead of Erases: What LACUNA Exposes
- Safety evals need to test messy language, not just bad behavior
- Anthropic’s Python SDK points to agents as infrastructure, not demos
- PolicyGuard makes the compliance bot show its work
- The Cheap Way to Test Whether Your Agent's Step Scores Actually Mean Anything
- TRIAGE gives agent RL a better target than pass or fail
- Hugging Face model pages get a better eval paper trail
- World Models That Edit Their Own Context Instead of Their Weights
- Agent immunity is the missing layer between alignment and tool use
- Google's Paper Assistant Wants to Catch Your Math Errors Before a Reviewer Does
- HORIZON treats chip design like a repo, not a chat prompt
- Nash solvers have preferences when the value is identical
- Hermes Agent moves the agent demo into the payments layer
- Hermes turns agent setup into packaging work
- Promptware Treats Prompt Injection Like an Execution Chain
- What a one-line LangChain fix tells you about streaming reliability
- Autodata Turns Synthetic Data Generation Into an Agent You Train
- Multimodal models still change answers when you shuffle the evidence
- Self-distillation can make models better on the first try and worse on the fifth
- Tool-use RL is failing at the brackets, not the tools
- Codex record-and-replay turns screen demos into reusable skills
- CUGA's Two Dozen Examples Are the Real Agent Documentation
- Omio’s AI-native travel bet starts with messy trip planning
- The 4-second budget that decides if your AI agent ships
- Agent Success Rate is the only number that matters when a new model drops
- Marketers are still vibe-checking prompts. Frontier devs run evals before lunch.
- Stop Vibe-Checking New Models. Build a 50-Prompt Eval Set Instead.
- Splitting the agent loop from tool execution cut TTFT by 90%
- Agent Amnesia Has a Fix, and It Looks Like Sleep
- Dreaming Agents Could Finally End the Brand Voice Correction Loop
- How a 400-line system prompt becomes 15 lines with Skills
- The 200K Token CSV Problem Has a One-Line Fix
- The 400-line system prompt is the new technical debt
- The mechanism matters: making AI competitive analysis auditable
- Last Quarter Means Two Different Things Inside Your Own Company
- Split-brain agents: when planner and executor stop talking, quality collapses
- The Frustration Index: A Cheap Eval Most Teams Skip
- Why I Stopped Trusting Demo Videos for Agent Tools
- When Your AI Agent Needs a Browser, Not an API