You run a link checker. It flags a link as broken. You click it yourself, in an ordinary browser tab, and the page opens perfectly fine. You run the checker again, just to be sure. Still "broken." You're not losing your mind, and the link probably isn't actually dead — your checker most likely got blocked by a firewall, and reported that rejection as if the page didn't exist.
Two completely different reasons a request can fail
A link check is really just an automated HTTP request, and there are two fundamentally different reasons that request can come back looking like a failure:
- The page is actually gone. The domain doesn't resolve, or the server responds with a genuine 404 (Not Found) or 410 (Gone). This is a real, dead link — nothing more to investigate.
- Something in front of the page rejected the request itself — before the page even had a chance to answer. This is a block, not a failure of the page.
The second case is far more common than most people realize, because a huge share of the modern web sits behind bot-protection services: Cloudflare, Akamai, Incapsula/Imperva, PerimeterX, DataDome, and Sucuri collectively protect an enormous number of sites. Every one of them is specifically designed to detect and challenge traffic that doesn't look like a real browser — and an automated link checker, almost by definition, doesn't look like a real browser.
What actually gives an automated checker away
A real visitor's browser sends a rich, consistent fingerprint with every request: a normal user-agent string, cookies from prior visits, a referring page, JavaScript execution, and a request pace that looks like a human clicking around. A basic link checker often sends none of that — a bare request with a generic or missing user-agent, no cookies, no JavaScript, and a burst of requests to many URLs in quick succession. To a firewall built to catch scrapers and bots, that pattern is exactly what it's watching for, and it will often respond with a challenge page, a rate-limit response, or an outright rejection — never reaching whatever the actual page would have shown a real visitor.
How to tell a real 404 from a block
The status code alone is often the first clue, though not always a reliable one on its own:
| Signal | Usually means |
|---|---|
| HTTP 404 or 410, no firewall fingerprint in the response | Broken — a genuinely dead page. |
| HTTP 403, 429, or 503, especially with a known firewall's headers or challenge page | Blocked, likely fine — the page is probably fine for a real visitor. |
| HTTP 404, but the response body matches a known firewall's block-page fingerprint | Blocked, likely fine — some firewalls use 404 specifically to hide pages from scanners. |
| Timeout, SSL error, or a 5xx server error | Needs a re-check — often temporary, not proof the page is gone. |
Notice that last row in the "broken" case: even a 404 isn't automatically a safe signal. Some bot-protection systems deliberately return a 404 to make a page look like it doesn't exist to a scanner, rather than admitting a block happened — which means telling the two apart reliably requires actually recognizing the firewall's response fingerprint (specific headers, or phrases in the page body like "checking your browser before accessing" or "attention required"), not just reading the status code in isolation.
Why this matters more than it sounds like it should
A checker that reports every blocked request as "broken" doesn't just create annoying false alarms — it actively wastes time. Someone sees a growing "broken links" report, spends time investigating or replacing links that were never actually dead, and after enough false positives, starts ignoring the report altogether. That's the worst outcome for a tool whose entire purpose is catching the links that really are dead.
It also skews toward exactly the wrong sites. The sites careful enough to run serious bot protection tend to be larger, more established, more trustworthy sources — precisely the kind of sites worth linking to. A checker with this blind spot will systematically flag your best outbound links as broken more often than your sketchiest ones, simply because well-protected sites are more likely to challenge an automated request in the first place.
How to actually catch this instead of guessing
This is the exact gap Link Verdict was built to close. Every checked link gets classified against known firewall/bot-protection fingerprints before it's ever called broken — a rejected request that matches Cloudflare, Akamai, Incapsula, PerimeterX, DataDome, or Sucuri's signature gets reported as "blocked, likely fine," not lumped in with genuinely dead pages. A real 404, a domain that no longer resolves, or a page confirmed gone still gets called exactly what it is.