Need recommendation for a site scraper / web page bulk downloader

I'm looking for an application (preferably Windows) or a Unix script that can scrape a portion or entire website to local files. "Why I need this" is a long story, but I need to be able to slurp down HTML files, graphics, CSS, javascript, etc., and have the URL paths rewritten to work from a local hard drive.

Any software recommendations for software that people have used? As for cost, 'free' would be nice, but I'm willing to pay some $ if the product is good and professional.

thanks!
LC

 

hehe... replying to my own thread.

I found a great one that does exactly what I needed: MetaProducts "Offline Explorer" ... http://www.metaproducts.com/

LC

 

$70...ouch.

I don't have much use for this sort of program, but, when I have had the situation come up, I just use wget. It takes a little bit (though not much) to learn, but I'm comfortable using command line tools. And it's free.

I'm not sure about a windows version, though. If there's not, I bet it would work with cygwin.

 

yeah, wget should do it most of times

 

wget -r http://www.somesite.com

and there is windows version http://gnuwin32.sourceforge.net/packages/wget.htm and http://www.interlog.com/~tcharron/wgetwin.html

 

Maybe you can consider to use FEAR::API, another site scraping framework based on Perl. You can use it to create your own specific scraping scripts.

Best,
Yung-chung Lin