Crawler – (aka Robot or Spider)

Crawler – (aka Robot or Spider) A script running for a search engine that “travels the web,” through hyperlinks, storing URLs and indexing content of pages it encounters. These results are then ranked and appear in Search engine result pages (SERPs)

Leave a Reply

You can use these XHTML tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>