How Google finds your site

Before a page can rank for anything, something has to find it. Most small-business sites lose here, not at the ranking stage.

5 min read

Almost every small business I talk to about search assumes the problem is ranking — that Google has looked at their site, weighed it against the competition, and put it on page four. Usually that isn't what happened. Usually Google hasn't meaningfully looked at it at all.

Crawlers follow links, and nothing else

Google finds pages by following links from pages it already knows about. That is the entire mechanism. There is no directory of every website, no registration desk. A crawler lands somewhere, reads the links, follows them, and repeats.

The consequence catches people out constantly: a page that nothing links to is effectively invisible. You can write the best page on the internet about drain repair in Torrance, publish it, and have it sit there indefinitely, because nothing points at it and nothing ever told Google it exists.

This is why site structure matters more than most people expect. Every page worth ranking needs a real link from somewhere else on your site — navigation, a related-services list, a link in the body of another page. Not a link that only appears in a dropdown built with JavaScript. An actual anchor tag a crawler can read.

Three things that quietly block it

When a site genuinely can't be found, it's nearly always one of these, and all three are invisible to the owner because the site looks perfectly fine in a browser.

Content that only exists after JavaScript runs

Google can execute JavaScript, but it does it on a delay and not always thoroughly. If your text, links, and headings only appear after a script fetches them, you are relying on the slowest, least reliable part of the process. Anything you want ranked should be in the HTML that arrives from the server.

A robots.txt or noindex tag left over from development

This one is common and brutal. A site gets built on a staging URL, blocked from search so the half-finished version doesn't get indexed — and the block ships to production with everything else. The site looks live. It is deliberately hidden. I have found this on sites that had been up for a year.

Two versions of the same site

If example.com and www.example.com both load, Google may treat them as two separate sites showing identical content, and split whatever credibility you've earned between them. One should redirect to the other. This is a five-minute fix that people leave broken for years.

What to do about it

Set up Google Search Console. It's free, it takes about ten minutes, and it is the only place Google tells you directly what it thinks of your site: which pages it has indexed, which it has deliberately skipped and why, and what queries you're already appearing for. Everything else in SEO is inference. This is the primary source.

Then submit a sitemap. A sitemap doesn't make anything rank — it's a list of URLs saying "these exist, here's when they changed." What it does is remove any excuse for a page not being discovered, which matters most on new sites with few inbound links.

The quickest check you can run right now: search Google for site:yourdomain.com. That lists what Google has actually indexed. If pages are missing, or the number is far smaller than the number of pages you have, you have a discovery problem — not a ranking problem, and the fixes are completely different.

Why this comes first

Crawling is the least glamorous part of search and the only part that is strictly required. You can improve your writing, your speed, and your reviews indefinitely, and none of it moves a page that was never found. Getting this right isn't optimisation — it's the precondition for optimisation mattering at all.

Want this handled for you?

I build and maintain sites for small businesses around the South Bay. Call or text for a same-day estimate — a real answer from me, usually within the hour.

Keep reading