Spidering a site with Wget

Depending on your activity….this can be a fun example.  Wget can actually be used to crawl an entire website.  A number of dubious uses for this come to mind, but from a practical view it can be used for transferring pre-rendered pages, archival and even syncing.

wget -U "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" --execute robots=off --recursive --no-parent --continue --no-clobber http://www.vassox.com

About the author

Lima is the visual nautical indicator for "stop instantly."

Leave a Reply