How search actually works
Crawl, index, rank. Three jobs in that order, and a surprising number of SEO problems are one of the first two failing while you optimise for the third.
Most SEO advice starts at the end. Write good titles, build links, add schema. All of that is downstream of three jobs a search engine does, in order, and a surprising number of SEO problems are one of the first two jobs failing while you spend months optimising for the third.
Crawl, index, rank
- Crawling. A program follows links around the web and fetches pages. If nothing links to a page and it is not in a sitemap, it may never be found at all.
- Indexing. The fetched page is understood and filed. Not everything crawled gets indexed. Thin pages, near-duplicates and pages that look like a hundred others often get dropped quietly.
- Ranking. When somebody searches, the indexed pages that match are ordered. This is the part everyone talks about and the last part that matters.
The useful thing about this order is diagnostic. "I am not ranking" is three different problems wearing one sentence. If the page was never crawled, no amount of better writing helps. If it was crawled but not indexed, the problem is usually that the page is not worth indexing. Only if it is indexed and still invisible are you actually in a ranking competition.
How to check which one you have
This takes about five minutes and settles the question, so do it before reading anything else about keywords.
- Open Google Search Console and paste the exact page address into the inspection bar at the top. It will say whether the URL is on Google.
- If it says the page is not indexed, the reason is given underneath. Read it literally: it usually names the cause.
- If you have no Search Console, search for the address with the site: prefix. If nothing comes back, it is not indexed. This is cruder and good enough to tell you which problem you have.
- Check the Pages report for the whole site, which groups every page by why it is or is not indexed. The groups are more informative than the total.
The reasons you will actually see, and what each one means:
- Crawled, currently not indexed. The page was fetched and judged not worth filing. This is a quality or duplication problem, and it is the most common one.
- Discovered, currently not indexed. It knows the page exists and has not bothered fetching it. Usually a sign the site has many low-value pages competing for attention.
- Excluded by noindex tag. Self-inflicted and instant to fix, and it happens most often when a staging configuration ships to production.
- Blocked by robots.txt. Also self-inflicted, and worth understanding properly because it is widely misunderstood.
- Alternative page with proper canonical tag. Working as intended: something else is the canonical version. Only a problem if you disagree with which one was chosen.
What ranking is actually weighing
There is no single score you can see, and nobody outside the company knows the full list. What is well established is the shape of it: does this page answer the thing that was asked, is it better than the alternatives, and is there any reason to trust it.
- Relevance. Does the page actually cover the thing somebody searched for, in the sense they meant it?
- Quality. Is it a real attempt at the subject, or a page assembled to catch traffic?
- Authority. Do other sites that already have standing point at this one, or at the site it sits on?
- Experience. Does the page load, work on a phone, and let the reader get to the content without a fight?
Notice what is missing from that list: keyword density, word count, how many times you repeated the phrase, and almost every number people quote as a target. Those were proxies, and the proxies stopped working years ago while the advice kept circulating.
On crawl budget
You will read about crawl budget, the idea that a crawler will only fetch so many pages from your site. It is real and it almost certainly does not apply to you. It becomes a consideration on sites with hundreds of thousands of pages, usually generated ones.
On a site with fifty pages, if something is not being crawled, the cause is that nothing links to it or that the site has given the crawler reasons to lose interest. Both are fixed by having fewer, better pages that link to each other, not by budget management.
Why new sites take time
A new site has no history, no links and no record of anyone finding it useful. That is not a penalty, it is an absence of evidence. The practical effect is the same: you will not win competitive searches in your first months, however good the page is.
What does happen early, and is worth watching for because it confirms the machinery is working: you start ranking for your own name, then for very long and specific phrases nobody is competing over, then for slightly shorter ones. That progression is the signal. Its absence after a couple of months, when pages are indexed, points at the pages rather than at time.
This is the single most common reason people give up on search. They publish fifteen pages, see nothing after six weeks, and conclude SEO does not work. Six weeks is roughly when a new site starts appearing for the searches nobody is fighting over, which is exactly where you should be aiming anyway.
Work to do
- Inspect your three most important pages in Search Console and write down, for each, whether it is indexed and what reason is given.
- Open the Pages report and note which non-indexed group is largest. That group is your actual problem.
- For anything saying crawled but not indexed, read the page and write one honest sentence about what it offers that the pages already ranking do not.
- Submit a sitemap if you do not have one. It does not make you rank, but it removes discovery as an excuse.