Once upon a time, web crawling was the providence of manly men. Men who wrote hairy systems code that optimized DNS queries, tweaked operating system TCP kernel options, avoided robot traps, and honored complex, dynamic revisit schedules.
Nowadays, thanks to 80legs, you can fork over 99 bucks a month, setup 3 crawl jobs, repeating if you like, and get the contents of a million URLs. All from the comfort of your own home.
This is progress.