A product page that ranked for two years is suddenly gone from results. Your first instinct is to open every tool you own and run all of them at once. Resist it. Firing off a full crawl, a live inspection, and a log analysis simultaneously buries the signal under noise, and you end up reading four reports that each answer a slightly different question. The faster path is to name the symptom precisely, then pick the one tool built to explain that symptom.
Start from the symptom, not the tool
Most technical problems announce themselves as one of a handful of complaints: a page dropped out of the index, rankings slipped without any visible breakage, a whole section stopped getting crawled, or a change you shipped didn’t take. Each of those has a natural first stop. Choosing the tool before you’ve described the symptom is how people spend an afternoon in a crawler when a thirty-second live check would have answered them.
So before touching anything, write down the observable fact. Not “SEO is broken” but “these three URLs return in the browser and no longer appear in results.” That sentence usually points straight at the right instrument.
Pages vanished from search: where do you look first?
When specific URLs disappear from the index, start with a live URL inspection rather than a crawl. A crawler tells you what your site currently serves; it does not tell you what the search engine last saw or decided. The inspection view shows the last crawl date, the indexing verdict, and whether a canonical or noindex directive is steering the page elsewhere.
Reach for a site-wide crawler only after the live check names the cause. If one page is deindexed, inspect it. If a hundred pages vanished together, then a crawl earns its keep by revealing the shared pattern, a template change, a stray noindex tag, a canonical pointing at the wrong URL.
When rankings dropped but nothing looks broken
This is the trickiest symptom because the obvious tools stay green. Pages are indexed, the crawler reports no errors, and yet position slipped. Here the useful move is comparison over time rather than a fresh scan. Pull performance data and look at which queries and pages lost ground, then check whether the loss clusters around a date, a device, or a specific template. Technical crawlers rarely explain ranking softness, so don’t keep re-running one hoping the answer appears. The tool that helps is the one showing change, not the one showing current state.
Choosing between a crawler and a live URL check
These two get confused constantly, so it helps to draw the line clearly. A crawler is for breadth: it walks many URLs and surfaces patterns you’d never spot one page at a time, like a swath of redirect chains or a directory returning soft 404s. A live URL check is for depth on a single address: it reflects exactly what the search engine fetches and how it interprets that fetch, right now.
Use the crawler when your question contains the word “which” or “how many.” Use the live check when your question names one specific URL and asks “why this one.” Reaching for the wrong one wastes the most time, because a crawl of a large site takes real minutes while a single-URL question sits unanswered.
The case for testing crawl access and robots rules before anything else
There is one check worth running first no matter which symptom you face, because it invalidates so many other readings when it’s wrong. If a rule is quietly blocking a path, a crawler will report the page as inaccessible, an inspection will show it was never fetched, and you’ll chase phantom causes for an hour. Before you interpret any other report, it’s worth a moment to test Googlebot crawl access and robots.txt rules using a free utility, which teams around the Denver tech scene and elsewhere lean on to confirm a directory really is fetchable. Rule this out and the rest of your diagnosis rests on solid ground.
A quick routing table for your next emergency
Keep this short mapping handy. One URL gone from the index: live inspection. Many pages gone at once: crawler for the pattern, after confirming access rules. Rankings down with no visible breakage: performance comparison over time. A recent change not showing up: access check first, then re-inspect the single URL. A whole section uncrawled: robots and internal-linking check before anything heavier.
Your next practical step is simple. The moment a symptom appears, write one plain sentence describing what you actually observe, then open only the single tool that sentence points to. Everything else can wait until that first tool has spoken.
