Crawler – (aka Robot or Spider)
Crawler – (aka Robot or Spider) A script running for a search engine that “travels the web,†through hyperlinks, storing URLs and indexing content of pages it encounters. These results are then ranked and appear in Search engine result pages (SERPs)




Leave a Reply