AI Agents & Evals
AI agents get hyped as autonomous coworkers and dismissed as glorified chatbots. Both miss the point. An agent is a loop: a model that calls tools, hits errors, retries, and sometimes finishes the job. What separates a demo from something you'd run in production is rarely the model. It's the harness, the tool design, and whether you can actually measure when it works. That's why agents and evaluation belong together here. I track the architectures that hold up, the eval methods that catch failure before your users do, and the honest gap between a slick demo and a system that ships.
191 posts
Recent articles (30)
- Browser agents are becoming practical comment analysts Marketing AI Institute’s LinkedIn comment workflow shows where browser-based AI agents already help: fast qualitative sorting, theme detection, and follow-up planning. The catch is that sentiment is not truth, and operators still need sampling, source checks, and human judgment.
- Can AI Agents Do Their Own Interpretability Research? A New Benchmark Says Not Yet SAEScientist-Bench tests whether frontier agents can autonomously find and steer interpretable model features, and the results show real discovery ability paired with a stubborn gap in causal control and reading their own experiments.
- OpenAI’s reported Millennium math claim needs proof, not applause CoinDesk reports that OpenAI used 10,000 agents and an unreleased internal model on a Millennium Prize Problem. The interesting part is not the headline win, it is whether AI-assisted proof work can be audited, attributed, and trusted.
- Procedural Graphs give agents a memory of what to do, not just what happened A new arXiv paper proposes storing agent know-how as a graph of procedures instead of a growing chat history, and lets an LLM edit that graph from its own wins and losses. Here is what that changes for anyone shipping long-horizon agents.
- SPINE Shows Sycophancy Gets Worse When Users Keep Pushing The SPINE benchmark is a useful reminder that single-turn AI evals miss a common product failure: models can know the right answer, then concede anyway after enough user pressure.
- OpenAI's Python SDK Gets More Honest About Agent Failures OpenAI’s Python SDK v3.9.0 is a small release, but it points at a bigger shift: production AI apps now need cache visibility, retry sanity, and cleaner event handling more than another demo.
- How a 9B model learned to chain Korean government APIs by actually calling them A new benchmark and data-synthesis recipe show that grounding tool-calling training in live API execution can push a small open model close to a much larger one, with real implications for on-premise government agents under data-sovereignty rules.
- LexFlip Shows Legal Meaning Metrics Are Still Token Counters A legal text can keep almost all its words and flip its force entirely. LexFlip is a useful diagnostic because it separates surface similarity from legal meaning, exposing how weak many common preservation metrics are for clause simplification.
- When You Upgrade the Model, Does the Agent's Memory Come With It? A controlled study finds that agent memory can silently break during a model upgrade, and the failure depends on how you stored it: verbatim history, chunks, compressed notes, or a fixed schema. Here is what transfers and what quietly rots.
- Agent Harnesses Are Becoming a Token Efficiency Fight A Reddit benchmark comparing Claude Managed Agents, LangChain DeepAgents, and TrueForge points to a practical question for builders: not which agent feels smartest, but which runtime gets the same work done with fewer tokens, fewer tool calls, and less operational drag.
- AI can help with PCB design, but it cannot own signoff A practical read on the Hacker News question, can AI design circuit boards yet, with the useful distinction between drafting help, engineering judgment, and manufacturing-ready PCB signoff.
- GPT-6 Astra's Big Claim: Agentic Skill Meets Alignment OpenAI says GPT-6 Astra leads on computer use, coding, cybersecurity, and science while being its most aligned model yet. Here is what an operator should test before believing the pitch, and where the announcement stays thin.
- What LangChain Core 1.6.2 Fixes for Agent Builders A close read of the langchain-core 1.6.2 release: async tool support for OpenAI, two content-mutation bugs squashed in the standard content layer, and what those small fixes actually mean for people running agents in production.
- Compiling a Prompt Into a Small Neural Function You Can Version A new arXiv paper proposes turning a natural-language spec into a reusable local neural function trained by teacher models, hitting 83.6% semantic accuracy on a hard benchmark where a faster compiler scored zero exact matches.
- ESPO's fix for prompt bloat: diagnose errors, then stop appending rules A new prompt optimizer called ESPO beats GEPA by 3.76 points on seven benchmarks while writing prompts 47% shorter, by clustering errors first and selecting candidates with bootstrap stability instead of endlessly appending caveats.
- Hosted LLM judges are not fixed instruments A preregistered arXiv audit found that black-box LLM observers on shared endpoints failed basic repeatability checks, which should change how teams design eval gates, leaderboards, data filters, and model-as-judge workflows.
- The OpenAI agent hack report is a disclosure problem first Decrypt reported that OpenAI agents were involved in hacking a German website to share rule-breaking tactics. The bigger operator lesson is not agent panic. It is that auditability, containment, and disclosure timing now belong in the product spec.
- AICOME works best when the missing variable is narrow AICOME reframes AI measurement as contextual estimation, not prediction, and its strongest lesson is practical: synthetic survey variables can help when datasets are rich, targets are narrow, and validation checks split group effects from individual deviations without pretending the model knows what was never observed.
- Bare assertions are a weak spot for medical reasoning models A clinician-reviewed benchmark finds medical reasoning models are easier to steer with unsupported assertions than fabricated evidence, and response-only monitoring misses much of the corruption that open reasoning traces can reveal.
- Repo-distilled skills are the missing middle layer for research agents The arXiv paper Repo-To-Skill argues that agents need compact operational knowledge, not just bigger models or longer context, and reports sizable benchmark gains from turning ML repositories into verified reusable skills.
- Web agents need useful predictions, not prettier page snapshots The arXiv paper Discriminative World Models for Web Agents points at a practical fix for web agents: train world models to make action outcomes distinguishable for rankers, not merely to recreate the next browser state.
- When a Language Model's Odds Don't Add Up A new arXiv paper measures the incoherence of language model probability forecasts using Dutch books, and finds the numbers models give you can be arbitraged for guaranteed profit. Here is what that means for anyone using them to reason about risk.
- AI agent interviews are a workflow fix, not a content strategy Marketing AI Institute’s agent-interview workflow points to a practical content use case: not replacing experts or writers, but capturing expertise asynchronously, cleaning the handoff, and forcing teams to define what counts as approved knowledge before they publish. The catch is governance, not prompts.
- Grading Coding Agents on How They Solve, Not Just Whether They Pass A new arXiv framework called PTA-IRT uses agents' execution trajectories, not just pass/fail results, to estimate benchmark scores from small task subsets. Here's what trajectory-aware evaluation changes for anyone trying to test SWE agents without burning a fortune.
- Proactive writing agents need timing more than autocomplete A small arXiv writing study points to a useful design shift: AI helpers should not just finish sentences, they should act like configurable thought partners that know when to interrupt, when to stay quiet, and how to frame advice without taking over the draft.
- SAGE Uses a Big Model as a Coach, Not a Crutch A new arXiv paper called SAGE trains a cheap RL agent by querying a vision-language model only when the learner is uncertain, then distilling that advice into a policy that needs no model calls at deployment. Here is what operators can take from it.
- Verbal reinforcement learning is a feedback routing problem A useful arXiv taxonomy frames natural language feedback as something that can define tasks, steer reasoning at runtime, or change model weights. The practical lesson is not that every agent needs training, but that builders need to route human feedback to the right layer.
- Gemini’s agentic video framing shifts the hard part from seeing to checking Google DeepMind’s “Introducing agentic video understanding with Gemini” points to a useful product pattern: video AI should inspect, rewind, compare, and verify, not just summarize a clip in one pass.
- Reward choice changes how LLM forecasters are wrong The arXiv paper How Proper Scoring Rules Shape LLM Forecasting shows that scoring rules can produce similar top-line forecasting performance while changing calibration, probability use, and error structure. That matters if you are training or evaluating models for real decisions.
- Small dialogue agents need repair loops, not just bigger training The arXiv paper “Acquire, Repair, Preserve” shows that small dialogue-game agents can improve sharply when training targets diagnosable failures, but the gains stay narrow unless the repair signal transfers beyond one task family.
Earlier articles (160)
- Synthetic data needs a weight limit
- A Minecraft clone is a weak coding-agent test
- What Happens When You Let AI Agents Build Their Own Society
- Agent payments are a product problem before they are a currency problem
- Terminal-Bench 4.0 and the eval gap for smaller coding agents
- Treat the smart TV like an untrusted computer
- WikiSkill and the Case for Giving Agents a Memory That Compounds
- Anthropic’s Python SDK is tightening the agent plumbing
- Correct SQL answers can still hide broken agent work
- SwarmWorld makes a case for agents that coordinate through artifacts
- VBVR-Pro makes visual reasoning a training loop, not a demo
- Claude Code as a domain renewal research assistant
- Recuris and the Case for Memory That Rewrites Itself
- The Retrieval-Integration Gap: When Your AI Analyst Reads the Risk and Ignores It Anyway
- Prime Agent treats the harness as part of the model
- Gemini and Apex point prediction markets toward brokerage plumbing
- Verification becomes the operator loop for AI-built silicon
- A 36-node DGX Spark homelab points at agent infrastructure, not just bigger inference
- Munder Difflin and the real work behind AI clone offices
- Treat the Texas AI hacking story as a disclosure systems test
- What LangChain's perplexity 1.4.1 patch says about agent plumbing
- Agent memory works better when skills are small and written in text
- Binance Agent OS puts trading agents behind user-controlled gates
- Self-improvement audits need a measured null
- Task Model Induction turns messy screen traces into reusable agent skills
- What AI4AI-Bench Says About Recursive Self-Improvement Right Now
- When a Zero-Shot LLM Ties a Random Forest on Travel Behavior
- Measure AI precision before you trust AI capability
- SPADE Makes the Training Environment a Thing the Model Learns to Build
- The Self-Improving Agent Demo That Falls Apart When You Shuffle the Tasks
- AutoSR turns symbolic regression into a research-state search
- CaliBench tests whether video models roll fair dice
- PIHF turns prompts into a versioned policy layer
- Stop Making One Prompt Read the Sources and Cast the Vote
- AI coding feels like managing a very literal junior engineer
- What Actually Makes a Claude Code Session Productive
- AutoDesign turns paper-to-poster into a harness the agent rewrites itself
- OmniScientist argues that AI scientists need eyes, not just workflows
- A Strong Model Can Scaffold a Weak One Without Any Retraining
- When Your User Simulator Is Secretly One Person: Simulator Collapse in Multi-Agent RL
- AMIE’s video consult result is about perception, not replacement
- Auto-research agents need fuzzer-style feedback
- The TTS Score That Hides Your Voice Model's Real Problems
- PsychoAgent makes memory retrieval less purely semantic
- SABRE turns VLM evals into a repeatable stress-test pipeline
- SkillProx and the Case for Agents That Prune Their Own Playbooks
- The 2011 Link That Died on Schedule, and What It Says About Agent Memory
- Kitesurf and the browser built for agents, not humans
- TutorMoments and the Hard Part of AI Tutoring: Knowing When Not to Answer
- Agnostic PAC learning gets its optimal bound
- Benchmarks Need QA Before They Judge Agents
- Heart-failure feature engineering gets an agent pipeline, not a chatbot
- RAG For Table-Heavy Reports Needs Search You Can Audit
- Selective Trust: Why RAG Systems Should Learn When to Ignore Their Own Context
- The Harness Is the Product: What HarnessOpt-Bench Actually Measures
- OpenAI’s cyber eval issue is really a process story
- Video deep research agents need to look before they search
- What 'Test-Time Scaling' Actually Means When You Read a Benchmark
- DNS identity for AI agents needs more than a name
- LiveMem reframes long-context memory as state continuity
- Where multimodal embeddings and collaborative coding agents actually stand
- DungeonBench puts tactical reasoning where agents usually break
- ExtractBench tests document extraction where demos usually fail
- On-policy imitation helps when the student is smaller than the expert
- Post-training is now the behavior layer
- The useful question behind a multiplayer agent harness
- Claude’s test escape is a security design problem, not a sci-fi story
- Computer-use agents need stricter judges, not prettier demos
- ORCA-bench shows oncall agents are not ready for pager duty
- When You Count the Tokens, Self-Reflection Loses to Just Sampling More
- AI research agents can code, but they still can’t judge the work
- APEX-Accounting: The 2.6% Number That Should Scare AI Bookkeeping Startups
- The Office-Task Benchmark That Prices Agents Against Human Labor
- πR² makes robot policies react inside the action chunk
- Autonomous research needs a budget scoreboard
- ChatGPT Work turns sales AI into a revenue feedback loop
- On-Policy Distillation Is Becoming the Default Move for Agent Training
- CausalForge makes AI research agents prove their work
- Coinbase’s AI agent bet is payments plumbing, not an AI pivot
- Anthropic's SDK Just Named Claude Opus 5 and Added Mid-Stream Tool Swaps
- Treat rogue AI hacker stories as incident reports, not movie trailers
- GS-Agent Builds 4D Worlds by Driving a Physics Engine, Not Replacing It
- LangChain’s gateway env var is production plumbing
- Training Agents Inside the Harness They Actually Ship With
- World’s $52.5M bet on proof of human for agents
- PoTRE makes the case for heterogeneous test-time reasoning
- Safety bounds are becoming probabilities, not vibes
- LangChain’s xAI update is really about model contracts
- When a Model Eval Turns Into an Actual Breach
- No Best Harness: Automated Discovery Systems Fail to Generalize
- O-VAD treats factory video anomalies as object histories
- CRAFT Turns Rubrics Into a Fine-Tuning Map
- Goal prompting is not a solver for NP-hard search
- Kimi K3’s SpreadsheetBench win is a signal, not a coronation
- ChatGPT’s computer control turns the browser into the new agent runtime
- Kimi K3 and the narrow win that matters for Next.js builders
- Shopify’s agent lesson is delegation, not autonomy
- Paper Revisions as Training Data: What SciDiagramEdit Gets Right About Figure Editing
- The One-Shot Trap in Agent Optimization
- Bonsai 27B makes local agents smaller, not magically smarter
- Memory Failures Hide Behind Correct Answers: What MemOps Exposes
- PalmClaw and the Case for Tool-First Mobile Agents
- Self-repair in small code models may be measuring retry form, not error content
- Useful work per dollar is the agent metric that matters
- Voice AI needs a human-quality test, not another pretty demo
- Judge Bias Lives in the Activations, Not Just the Prompt
- What a Model Knows About What It Knows
- IoT exploit agents are getting useful in controlled labs
- Vision models are getting the scene right, but not always the gaze
- Intelligence Is Still Not the Product
- What Stampli's 'one person doing four people's work' claim actually shows
- DiaLLM separates dialect understanding from dialect writing
- LangChain’s small July fixes point to bigger agent runtime problems
- GaP treats robot policies as editable graphs
- Verification as a Scaling Axis: What the LLM-as-a-Verifier Paper Actually Changes
- LangChain’s Mistral update is about provenance, not flash
- LangChain’s OpenRouter header fix is a production clue
- MiniCPM5-1B points at the phone-sized agent layer
- The AI job diamond still needs a bottom rung
- The Model Got Smarter and My Tool Got Dumber
- Speaker recognition is a better agent test than another chat demo
- Unlearning That Hides Instead of Erases: What LACUNA Exposes
- Safety evals need to test messy language, not just bad behavior
- Anthropic’s Python SDK points to agents as infrastructure, not demos
- PolicyGuard makes the compliance bot show its work
- The Cheap Way to Test Whether Your Agent's Step Scores Actually Mean Anything
- TRIAGE gives agent RL a better target than pass or fail
- Hugging Face model pages get a better eval paper trail
- World Models That Edit Their Own Context Instead of Their Weights
- Agent immunity is the missing layer between alignment and tool use
- Google's Paper Assistant Wants to Catch Your Math Errors Before a Reviewer Does
- HORIZON treats chip design like a repo, not a chat prompt
- Nash solvers have preferences when the value is identical
- Hermes Agent moves the agent demo into the payments layer
- Hermes turns agent setup into packaging work
- Promptware Treats Prompt Injection Like an Execution Chain
- What a one-line LangChain fix tells you about streaming reliability
- Autodata Turns Synthetic Data Generation Into an Agent You Train
- Multimodal models still change answers when you shuffle the evidence
- Self-distillation can make models better on the first try and worse on the fifth
- Tool-use RL is failing at the brackets, not the tools
- Codex record-and-replay turns screen demos into reusable skills
- CUGA's Two Dozen Examples Are the Real Agent Documentation
- Omio’s AI-native travel bet starts with messy trip planning
- The 4-second budget that decides if your AI agent ships
- Agent Success Rate is the only number that matters when a new model drops
- Marketers are still vibe-checking prompts. Frontier devs run evals before lunch.
- Stop Vibe-Checking New Models. Build a 50-Prompt Eval Set Instead.
- Splitting the agent loop from tool execution cut TTFT by 90%
- Agent Amnesia Has a Fix, and It Looks Like Sleep
- Dreaming Agents Could Finally End the Brand Voice Correction Loop
- How a 400-line system prompt becomes 15 lines with Skills
- The 200K Token CSV Problem Has a One-Line Fix
- The 400-line system prompt is the new technical debt
- The mechanism matters: making AI competitive analysis auditable
- Last Quarter Means Two Different Things Inside Your Own Company
- Split-brain agents: when planner and executor stop talking, quality collapses
- The Frustration Index: A Cheap Eval Most Teams Skip
- Why I Stopped Trusting Demo Videos for Agent Tools
- When Your AI Agent Needs a Browser, Not an API