How search engines work.

What a search engine actually is

A search engine is a piece of software that helps people find information on the open web. The three big ones used by Australians are Google (around 94 percent of searches), Bing (about 4 percent) and a long tail of DuckDuckGo, Yahoo and Ecosia. Bing also powers DuckDuckGo and ChatGPT's web search, so it is worth keeping in mind even though Google is the priority.

Under the bonnet every modern search engine does the same three things:

1. Discover URLs and download their content. This is called crawling.
2. Process and store the content in a searchable database. This is called indexing.
3. Match a user's query against the stored content and show the best results. This is called ranking and serving.

Get those three stages working for your site and you are in the game. Break any one of them and you are invisible, regardless of how good your content is.

If you want the parent pillar with the full overview of SEO, read what is SEO. If you want each stage in much more depth, read crawling, indexing and ranking explained.

The three stages: crawl, index, rank

Here is the pipeline in a single line of text.

Web → Crawler (Googlebot) → Renderer → Indexer → Index → Query → Ranker → SERP

Each arrow can fail. When we audit a Perth site that is "not ranking", roughly 70 percent of the time the problem is in the first three boxes, not the ranking algorithm. The page either is not being found, is not being rendered properly, or is not making it into the index at all.

Stage 1: Crawl

Google starts with a list of URLs it already knows about. It hits each one, downloads the HTML, looks for links, and adds any new URLs it discovers to its queue. The cycle never stops. Bigger, more authoritative sites get crawled more often. A brand-new site might get crawled once a week. A news site every few minutes.

Stage 2: Index

Once a page is downloaded, Google renders it (executes the JavaScript, builds the DOM, takes a snapshot), then runs natural-language processing on the text, the headings and the structured data. If the page passes Google's quality threshold, a processed version is stored in the index. If it does not (thin content, duplicate, blocked by noindex), it gets dropped quietly.

Stage 3: Rank and serve

When you type a query, Google's ranker runs the query against the index in milliseconds. It scores every candidate page using hundreds of signals, applies machine learning layers like RankBrain to understand fuzzy intent, blends in fresh content if the query is newsy, and personalises the result by your location, language and search history. Then it builds the SERP you see.

Googlebot and the crawl in detail

Googlebot is the user-agent name for Google's web crawler. It is not one bot, it is a swarm of them, hitting your server from many IP addresses at once. The two main variants are:

• Googlebot Smartphone. The default for indexing since 2021. It crawls and renders your site as a mobile browser would. If your mobile experience is broken, your rankings are too.
• Googlebot Desktop. Still active, mostly used for comparison and double-checking.

There are also specialist crawlers like Googlebot Image, Googlebot Video, AdsBot and Google-Extended (the one you opt out of if you do not want your content used to train Google's AI).

How Googlebot decides what to crawl

Three inputs feed Google's crawl queue:

1. Sitemaps. The XML file at /sitemap.xml tells Google which URLs you want crawled.
2. Links. Internal links and external backlinks pointing at a URL are the strongest discovery signal.
3. Manual submission. Submitting a URL inside Google Search Console pushes it into the queue. Use this sparingly.

If you are running a large site, Google also enforces a crawl budget: a soft limit on how many URLs it will crawl per visit. Tighten your site, kill duplicates and the budget stretches further.

The index: where your pages live

Google's index is the part most business owners never think about. It is the database of every page Google has decided is worth storing. As of recent disclosures it holds well over a trillion documents.

A page being in the index is a hard prerequisite for ranking. No index entry, no ranking, no traffic. Plenty of Perth sites we audit have whole sections that have been quietly dropped, often because of duplicate content or accidental noindex tags.

To check if a specific page is indexed:

1. Open Google Search Console.
2. Paste the URL into the URL Inspection tool at the top.
3. Read the status. "URL is on Google" means indexed. Anything else, you have work to do.

The site: operator in regular Google (e.g. site:yourdomain.com.au) gives a rough count of indexed pages, but it is approximate. Trust Search Console for the real number.

Ranking signals that matter in 2026

Google has confirmed "hundreds" of ranking signals, and the 2024 antitrust documents and the May 2024 Content Warehouse leak gave us partial visibility into many of them. Here is the practical ranking list a Perth business should care about, in rough order of weight:

1. Relevance. Does your page match the query, by topic and by intent? This is the entry ticket. Everything else is a tiebreaker.
2. Content quality. Original analysis, real expertise, useful structure. The Helpful Content system penalises thin or scaled content hard.
3. Backlinks. Other credible sites linking to you. Still one of the strongest off-site signals.
4. User signals. Click-through rate, dwell time, and "did the user come back to Google and try again". Google denies using these directly, the leak suggests otherwise.
5. Page experience. Core Web Vitals, HTTPS, mobile usability. A tiebreaker between equal pages.
6. Freshness. For "deserves freshness" queries (news, recent updates), newer wins.
7. Location and language. Where the searcher is, what language they speak. Huge for Perth-targeted searches.
8. E-E-A-T. Experience, expertise, authority, trust. Heavier in finance, health and legal.

These are not switches you can flip. They are outcomes of doing the underlying SEO work well across the three layers we describe in what an SEO actually does.

Trace a single query end to end

To make this concrete, follow a real query: "emergency plumber Joondalup". This is exactly what happens in the ~400 milliseconds between hitting Enter and seeing results.

1. Query parsing. Google identifies "plumber" as a service, "emergency" as an urgency modifier, "Joondalup" as a Perth suburb, and the implied intent as commercial, local, urgent.
2. Index retrieval. Google pulls candidate pages from its index that touch any combination of these terms. We are talking thousands of candidates initially.
3. Initial ranking. Each candidate gets a base relevance score from the original PageRank-style link graph plus content signals.
4. ML re-ranking. RankBrain and BERT re-score for query intent, neural matching looks for semantically related pages even without exact keyword matches.
5. Local stack. Because the query has clear local intent, Google fires up the Local Pack: it pulls candidates from Google Business Profile, scores them by proximity to Joondalup, review count and category match.
6. Personalisation. Your search history, your device, your IP location, your language settings all nudge the final order.
7. SERP assembly. Google decides which features to show: AI Overview, local pack map, ads, organic blue links, "people also ask". Different searchers get different layouts for the same query.
8. Delivery. The final SERP is rendered, cached and sent to your browser. Total elapsed time: about a third of a second.

The reason a plumber in Joondalup might rank above a competitor with a flashier website often comes down to two unsexy things: their Google Business Profile has more recent reviews, and their on-page content matches the exact phrase the searcher typed. Boring beats fancy.

Common mistakes that break the pipeline

Tools to see what Google sees

You do not need a paid stack to inspect the pipeline. Four free tools cover it:

1. Google Search Console. The URL Inspection tool shows you exactly what Googlebot saw the last time it visited a page: the rendered HTML, the screenshot, indexing status, any errors.
2. The site: operator. Type site:yourdomain.com.au into Google. Browse the results to spot pages you forgot you had.
3. Screaming Frog SEO Spider. Free up to 500 URLs. Runs a crawl of your site the way Googlebot would and surfaces broken links, duplicate titles, blocked URLs.
4. Google's Rich Results Test. Pastes any URL and tests whether your schema markup is valid and what Google can parse from it.

If you want all four bundled into one report, our free SEO audit tool runs them against your URL and produces a single dashboard.

Perth and WA context

Perth has a quirk: it is a relatively small media market with a high proportion of trades, mining services and professional services. That changes how the ranking pipeline behaves.

• Local pack dominates commercial queries. A search like "electrician Fremantle" or "physio Subiaco" almost always shows a 3-pack map result above the regular organic results. If your Google Business Profile is weak, you lose the click before your website even gets a look in. Read our local SEO pillar for the GBP playbook.
• Mining services have a small but high-value SERP. Queries like "shutdown maintenance Karratha" or "mine site labour hire Pilbara" have low volume but enormous lifetime value. The pipeline still applies but the ranking signals tilt heavily towards demonstrated experience and case studies, which is good news for genuine operators. See mining SEO.
• Tradie SEO is mostly a Google Business Profile and reviews game. The regular organic listings still matter for "best plumber Perth" style queries, but most tradies' phones ring from the local pack. Trades SEO covers it.
• Regional towns reward niche pages. Bunbury, Mandurah, Geraldton: these markets are small enough that a single well-built service page targeting "service town" can rank within weeks. Our Bunbury, Mandurah and Fremantle service pages were built on exactly this principle.