Methodology

How a verdict gets made

Short version: we judge each task against the same checklist, write the prompt we'd use ourselves, and say plainly where AI falls short.

The checklist

Every task is judged on four questions:

  1. Can a current AI app do the core of the job? ChatGPT, Claude and Gemini, including free tiers.
  2. How often does it get it wrong? Made-up facts, bad math and confident mistakes count against it.
  3. What does a mistake cost? A clunky packing list is fine. A wrong tax form or a missed legal deadline isn't.
  4. Can a regular person check the result? If you can't easily spot the error, the verdict gets stricter.

What the verdicts mean

Bot AI does the job well and mistakes are easy to catch.

Half-bot AI does most of the work, but a person has to finish, check or decide something.

Nah AI is wrong too often, or the stakes are too high. We still show how AI can help around the edges.

How we test

Every verdict is judged against the checklist above. Then we test the prompt: we fill in the [BRACKETS] with a made-up sample (never real personal data), run it, and grade the answer against the prompt's own rules. Did it follow the MUST and NEVER rules? Did it make up details we never gave it? Did it stay inside its word limits?

Every verdict is being re-tested twice, with two different made-up samples, on Bot or Nah's own AI server (Qwen 3.5). Earlier runs used Llama 3.3 70B through Cloudflare Workers AI, graded by GPT-OSS 120B. An AI grader checks each answer, and every run on the page names the models it used. Word limits are counted by code, and every quote it flags as made up is checked against the full answer. Re-tested pages show both samples in full, every check with a pass or fail, and the complete answer the AI gave. Pages still waiting for their re-test show our earlier single run with an excerpt; our first 50 of those used Claude (Anthropic). A check that fails is shown as failed. We have not logged runs in ChatGPT or Gemini, and we don't claim we have. A page without a test log says “Reviewed” instead.

The paid-app directory

Every paid-app page shows the plan, the price, the date it was checked and a link to its source. On those pages, the verdict answers a different question: can an AI coding tool build you a working replacement?

Readers keep us honest

Every verdict has a “Did this work for you?” vote. We only show the tally once enough people have voted for it to mean something. If readers report a different result, we re-review the verdict.

Money never changes a verdict

Sponsors can buy labeled placements. They can't buy, change or remove a verdict, and a sponsor's own product gets the same checklist as everything else.

Not professional advice

Verdicts about money, law and health are general information, not advice. For anything with real stakes, check with a qualified professional.