“Ask an LLM” Is Not an Answer to Every Question

A Hacker News post titled “Stop Telling Me to Ask an LLM” captures a frustration I keep seeing in technical spaces.

Someone asks a question. Maybe it is about a weird bug, a product choice, a legal gray area, a career move, or a library that changed under them. The replies come fast: “Ask ChatGPT.” “This is what Claude is for.” “Why are you asking people?”

That answer is sometimes useful. Often, it is lazy.

LLMs are now good enough that ignoring them is a mistake. They can explain unfamiliar APIs, draft test cases, compare options, summarize docs, and generate first-pass plans. I use them daily. But “ask an LLM” has become a conversation stopper. It treats every question as an information retrieval problem, when many questions are really requests for experience, taste, warning, or accountability.

The missing part is not always information

The default assumption behind “ask an LLM” is that the asker lacks facts. Sometimes true. But in practice, people ask communities because they want something narrower and harder to fake.

They want to know what broke in production for someone else. They want to hear which abstraction looked clean for six months and then became a tax. They want a senior engineer to say, “Yes, that error message is misleading.” They want social calibration: is this a dumb idea, a common trap, or a reasonable bet?

An LLM can simulate that kind of answer. It can also make stale advice sound current, flatten minority edge cases into bland consensus, and present guesses with the same tone as known facts. The problem is not that models are useless. The problem is that fluency changes the surface area of trust.

When a person answers in a forum, you can often inspect the trail: reputation, past posts, employer context, code snippets, scars. With an LLM, you inspect the answer. That can be enough for low-stakes work. It is not always enough for choices with operational cost.

“Ask an LLM” can be good advice, if it has shape

The better version is not “go away and ask a bot.” It is: “Here is the prompt I would try, here is what to verify, and here is where the model is likely to lie.”

That is a very different contribution.

For example, if someone asks how to debug a React hydration issue, “ask Claude” is noise. A useful reply might say: paste the exact warning, the component tree, framework version, and any server/client conditionals into the model. Ask it for three likely causes, not one. Then check for nondeterministic rendering, date formatting, random IDs, and browser-only APIs. That gives the asker a workflow and a checklist.

The same applies outside coding. “Ask an LLM to review this contract” is unsafe advice. “Use an LLM to produce a plain-English issue list, then have counsel review termination, indemnity, data rights, and liability caps” is practical. The model is a drafter and spotter. Not the authority.

This is the line I wish more AI advice respected. Models are strong at generating candidates. They are weaker at owning consequences.

Communities still matter, just differently

Forums, teams, and expert networks do not become obsolete because models can answer common questions. Their job shifts.

The high-value human answer now includes provenance, constraints, and lived edge cases. “We tried that at 10 million rows and the migration locked writes.” “That vendor discount disappeared at renewal.” “The model passed evals but failed when customers used screenshots.” These are not just answers. They are field reports.

AI also raises the bar for askers. If a question can be answered by a quick model query, do that first. Bring back the result, the uncertainty, and the specific point where you got stuck. That makes the human conversation better. It stops wasting community time on boilerplate and focuses attention on judgment.

The worst pattern is using LLMs as a way to dismiss people. The best pattern is using them to prepare better questions.

Practitioner’s Take: If you run a team or community, do not ban “ask an LLM.” Make it specific. Encourage people to show the model output, what they checked, and what still feels uncertain. For builders, add this to your workflow: model first for breadth, human second for judgment, docs or tests third for verification. The catch most readers miss is that the model is not replacing the expert. It is changing what an expert answer should contain.