web crawler
Example Sentences
Examples are provided to illustrate real-world usage of words in context. Any opinions expressed do not reflect the views of Dictionary.com.
Publishers could also try to protect their content from Google by forbidding its web crawler from sharing any content snippets from their sites.
From New York Times ● Jun. 1, 2024
The company originally operated a web crawler that searched for GIFs on the internet.
From Slate ● Nov. 30, 2021
He rewrote the code for a slow web crawler that Mr. Page had created to understand the relationship between links on different websites.
From New York Times ● Aug. 20, 2021
Bumgarner said the domain on Sony Corporation of America's network where the application was located was carefully hidden from view, so a web crawler or casual surfer would not have located it.
From Reuters ● May 13, 2011
