Is it necessary to list all pages in a google sitemap?

Anyone have any opinions about how important it is to list all pages in your google sitemap? Not too painful now, since I don't have many pages, but in the future I can see it becoming more difficult.

I suppose I could look into the sitemap generator, but that probably wouldn't be too smart about assigning the right priority and changefreq to each page.

Anyhow I was just wondering if I listed some of the main pages in my site, if the sitemap crawler would be smart enough to crawl pages linked from the main pages, or if it's really important to list them all.

Thanks,
Gary

 

Someone can correct me on this if I'm wrong, but...

If you want google to know about changes to your pages before its bot comes around to crawl your site, sitemaps is the way to go. It really depends on how fast you want your site to be indexed and for google to see any new pages or changes to existing pages.

I use it just for the main purpose of making sure google has a good representation of what pages my sites do have. I've seen sites where google crawls every page in the sitemap after I've added a sitemap, and I've seen googlebot simply just grab the sitemap and not crawl the site.

In conclusion I think its safer to go with sitemaps then without, because I don't think it hurts.

sorry if this confused more than helped.

 

No that does help, thanks.

While I'm asking questions for which probably no one outside of google has answers:

What about "priority"? Anyone have any inklings on whether that makes any difference? When I created my sitemap I just assumed I should probably assign a higher priority to things like my homepage, and maybe less to pages deep in the site. Anyone have any better ideas?

 

Priority will tell Google which of your pages should get higher priority in the crawling queue. Usually your most important page, your home page, would get a priority of 1.0, and then lower that for your internal pages. Go by the importance of the page to yourself in relation to the other pages on your site.

 

That's about what I figured, thanks. Another question if I still have your attention:

What's your experience been with sitemap generation tools? Is the python script google offers any good? (Don't currently have python installed, so I haven't tried it yet). Are there any others out there worth looking at?

As I said I only have 20 pages or so at this moment, so doing it by hand is not a problem, but I'm afraid of what might happen as things get bigger...

Thanks again,
Gary

 

The thread below has a couple of cool online tools for generating sitemaps. I use the python script from Google for one of my sites, and its not bad. But that site only has 11 pages so far.

http://forums.digitalpoint.com/showthread.php?t=25121

 

Nifty, thanks for the info.

-Gary

 

I generally add my newer pages at the top of the sitemap and leave out pages that have already been indexed. Works out best for me.

 

Yes the crawler is smart enough to find the other pages. If you do the top 3 levels of a site, Goog will find the fourth level on it's own.

As for the Site Map priority or change freq, set them to a default setting and don't worry about it. What is important is the last date modified. This is what Goog uses to decide if to index the page or not

Jim Catanich

 

As for site map generator, we have been using the http://gsitecrawler.com/ crawler. It takes some time, but the results will show you where you have broken navigation, link issues, etc.

And the results have been used by Google, Yahoo, and MSN. There is even formats that have been approved.

It is also a free tool so check it out.

Jim Catanich
http://www.catanich.com/

 

Well, I have written this article called
XML Google sitemaps and priority values

I personally believe, if done right, it is possible to get a slight advantages if using them. Close to minute though, but depends on circumstances.