Spider Simulation Tools

Hello,

Can anyone recommend a tool that can be used to determine if a page can be crawled or seen by a search engine spider?

We recently changed our links page from a static page to a dynamic page such as.../links.php page and after a week or so, I still can't see any evidence that the page has been listed in Google's cached.

Our site or main page is crawled almost every day and I can see the cached listing updated every few days. During this same time, I have seen any evidence of the new links page. I have looked at the omitted pages when I search in Google.

Thanks

 

Try Xenu Link Sleuth. I use if for 4 years and it does the job for me.

 

THis will do the trick:

http://www.delorie.com/web/ses.cgi

 

yeah link sleuth is a very good tool. i use it to find out how many pages my sites have so i can compare it to what google has indeexed so far.

-Dan