Being a online search engine crawler could preserve a cached duplicate of the file, it might on occasion crawl pages a webmaster isn't going to desire crawled. Internet pages typically prevented from becoming crawled include things like lo…