To avoid unwanted information while in the lookup indexes, site owners can instruct spiders not to crawl specified documents or directories throughout the conventional robots.txt file in the basis directory from the area. Also, a website page could be explicitly excluded from a internet search engin