Nowadays, there are numerous search engines available on the internet that we use daily to make thousands of queries and the large amount of information that they provide us with is truly overwhelming.
Search engines such as Google, Yahoo, Bing, Baidu, etc. are examples of this, each with its own capabilities and characteristics, but currently there is a king when it comes to search engines and that is Google, the most popular and used search engine. currently all over the world.
But, now that we have established its importance, have you really wondered how a search engine works to answer user questions? How is it possible for Google to find what you are looking for with such speed and accuracy?
To understand it, we will focus on explaining how the Google search engine works in detail.
How do search engines work?
It all starts when we enter a word or phrase in the search engine, which, after giving it a search, Google shows us millions of web pages at the moment that they contain or may contain information to answer our query.
The main objective of Google is to offer the user the information they need, that is, relevant information, and to do this, it selects which results to show you first and orders them according to the priority it deems appropriate for your search.
The operation of the Google search engine can be divided into three phases: Crawling, indexing and returning search results.
1. Google Crawling
The first step carried out by the search engine is the tracking of the millions of web pages that are on the Internet since new pages are continually being created or those that are already created are updated.
To do this, the first thing Google must do is find out what pages are within a website, but how does it do it?
This crawl starts from a list of web addresses that Google has obtained from previous crawls or also from the Sitemaps files that have been created by the owners of the websites.
A sitemap is an XML file that contains a list with all the URLs that we want Google to index and with this list you communicate to Google the updates you have made to your website, the pages you want it to index and how often your website is updated. .
To crawl or crawl these websites , Google uses the help of the Googlebot or more colloquially called the “Google spiders” or “Google Crawler” that enter, read the source code of your page and analyze the content to see what has changed with compared to previous versions or follow the links contained in these pages to discover new ones.
All this information is taken to the server so that it is processed, classified and weighed by the SEO optimizations that this web page has. We will talk about it later in the indexing phase.
On the other hand, just as there are pages that you want Google to crawl, there are many others that you want not to be visited , for example, those pages that are irrelevant to the business such as legal pages or blocking parts of the web that generate duplicate content by example the website categories, and for this you must tell Google not to track them and this is done with a robots .t xt file. These robots.txt files are also used to manage traffic from crawlers to the website.
All this tracking process has a finite time , it is what is known as Crawl Budget , that is, you have a number of milliseconds assigned for the bots to walk through your website to track it.
Google assigns more or less crawl time based on authority, accessibility, speed and quality, and this Crawl Budget can be optimized.
tracking issues
There may be cases where Google spiders cannot crawl the web page well. The problems can be the following:
- Do not include URLs in the Sitemap
- Being blocked by robots.txt
- Having too many levels of depth in the web page.
- Link with non-HTML links
- Abusing Java Script in code
- Do not link (Let there be orphaned URLs)
- Excessive loading times
2. Indexing in Google
After receiving all the information that the crawlers obtain from the web pages, the next step is to process and sort all that data.
If we go to the RAE, “indexing is orderly registering data in information, to prepare its index” which, extrapolated to the digital environment, the clearest index we have is Google’s SERP (Search Engine Results Page).
Google indexing is nothing more than that classification of content based on the information it contains to add it to the large Google database.
The objective of this is that, after a user search, Google only has to go to the part of its index where the information that the user is looking for is classified to show it and position it in order of relevance to the user.
There are several ways to check if a page has been indexed:
- Google Search Console: You must enter the tracking section of the tool → URL Inspection → Enter the address you want to check. When Google Search Console has finished the analysis, If the URL was not indexed, the message “Request indexing” will appear.
- Do a manual check on Google by adding “site:” in the search engine followed by the URL of the website and if it appears as a search result, it is already indexed.
- In the search engine itself, it tells you how many URLs of your web page are indexed.
Analyze your website with an SEO tool such as Screaming Frog and see which URLs are indexed.
What to do to get Google to index us?
- Configure the Google search console on your website.
- Create a sitemap.
- Create the robots txt files.
- Keyword study.
- Add quality content frequently.
- Create a good internal link and kill.
- Check that there are no broken URLs.
3. Return of search results
After the tracking and indexing phase, the moment comes when the user makes a query in the search engine, and that is, how does the search engine select the most relevant content for you?
For a specific user search intention, Google’s algorithm identifies in its index what you want to find and what would be the most relevant answer to your question.
Depending on what you are looking for and thanks to the SEO techniques implemented in the web pages, it determines the order in which you are going to see the list of pages , taking into account different factors such as the relevance of the web page and the authority.
Classification of URLs according to their indexation and crawlability
Crawling and indexing are often confused, because they are two totally different processes, because as we indicated before, crawling is that content search and indexing is its classification in the Google index.
But if a URL can’t be crawled, it can’t be indexed either?
This gives rise to confusion, since, although a URL may not be crawled by Google’s boot, it can be indexed at the same time. There are many cases that can occur in the URLs of our website, including:
Crawlable and indexable URLs:
These are those URLs in which Google can access and view your content and is also indexed by this search engine. But this does not mean that being crawlable is always indexed, since if Google does not consider it relevant to the search, it may not index it.
Crawlable and non-indexable :
These URLs are those where Google can access it and crawl its content but has been told that we don’t want it to be indexed in search results.
Non-crawlable and indexable:
These are those URLs that we do not want Google to access and cannot be read by meta-robots, but they can be indexed through other content (external links, sitemap, etc). Normally these URLs are defined with the robot.txt so that they are not tracked.
Non-crawlable and non-indexable:
URLs defined as <noindex> and also with access blocked to Google crawlers so that they cannot be crawled or indexed.
Why is it important for Google to index your website?
As we have mentioned before, not being indexed means not appearing in search results and therefore you do not exist for Google and consequently you do not exist for users. Therefore, if you have a web page and it is not indexed, you do not directly exist.
As the website is not indexed, you do not obtain any benefit, but when you achieve correct indexing, this becomes an increase in traffic to your website and improve your positioning.
For this reason, it is also very important that apart from the content being well indexed, it is of quality and has a good SEO positioning, since there is a directly proportional relationship between the improvement of organic positioning and the increase in web traffic.
Google will assign your website a position in the SERP that will vary depending on our SEO work done on the page, both SEO on Page and SEO off Page.
Now that you understand how the Google search engine works, you know the importance of crawling and indexing your web page to appear in Google. If you have a website and it is not indexed, your website does not exist directly.
At SEOraiseup we are experts in SEO services and we can ensure that your website is displayed optimally in search results. Do not hesitate in ask us for a budget.