Robots.txt is really a file that exists on the foundation directory of every Internet site and can be used to instruct search engines like google and yahoo on which directories/documents of the website they will crawl and consist of of their index. rank sth In keeping with/by sth Metropolitan areas https://technik.forum