The most convenient way to find necessary information on the Internet is to use search engines. The desired query is typed into the search bar, and the user receives the result – the list of pages that correspond to the search query as precisely as possible. To get into the search engine results, the pages of the site must first of all be indexed by the search robot. The search engine processes and adds to its base all information that can be found on the site; it indexes articles, links, illustrations.
Main types of site indexing
- Forced indexing – information about the site is added to the special form of a search engine.
- Natural indexing – the search robot finds the site himself by following the links from the sites that refer to your website. It implies that external link amount of the site must not be zero.
The term of site indexing may vary from several days to several weeks. The process of indexing is cyclic and repeats periodically every 2-4 weeks depending on a particular search robot. The search spider can index either the whole website or separate internal pages. Sometimes the site may not be indexed because of technical troubles or when the site pages cannot be accessed. In this case the search engine may take the following measures:
- it can exclude non-indexed pages from its database at once;
- it can carry out one more indexing of the site within a couple of days;
- it can exclude non-indexed pages if it cannot find them during the second indexing.
For the management of site indexing we use the sitemap, a special file (robots.txt) and special meta tags.
Partial closing of the page content from Yandex search robots is performed with the help of <noindex> tag. There also exists a special robots meta tag that can allow or forbid indexing and following the links of each particular page of the site.
Robots.txt is a special file in which you can forbid or allow indexing of the site by particular search engines or forbid indexing of the whole site or its separate sections. For this purpose special directives are used; they are: Disallow, Allow, User-agent and Crawl-delay.
Sitemap is the list of the site pages liable to indexing; this list is always relevant. A special sitemap file may be created in XML format or in common text format. To indicate the location of the sitemap file you can use the robots.txt file. For this purpose you should add the following line to the robots.txt file:
Sitemap: http://site_name/sitemap.xml
The use of the sitemap allows to inform the search engine about:
- the pages of the site that have to be indexed;
- the frequency of the information update on the pages of the site;
- the importance of indexing of every site page.
Acceleration of site indexing
There are a lot of ways to accelerate the indexing of the site by the search robots. But first let us consider the norms that should be observed if you want to solve the problems of the site indexing.
- Do not create too large pages (the size of the page should be no more than 100-200 kilobytes).
- The links to the inner pages of the site should explain the page content using simple headings.
- Use only friendly URLs – they make site indexing easier. Session parameters and identifiers are excluded from URLs.
You can order search engine optimization of your site in web studio WebStudio2U right now!
TAGS
site indexing,
sitemap,
robotstxt,
search engine optimization |
SEO 
