Sitemap used by Google to REMOVE pages (?)

That's really interesting. I'm not the only one

I submitted a sitemap about 2 days ago thinking all my pages are gonna get indexed. But hey, i was wrong! Not only the pages which were not indexed havent been indexed but also the pages which were in G index have been removed. Not all the pages though, i guess im lucky to still have 10 pages there lol. Anyways if this continues, im gonna get rid of the sitemap.

 

After more experimentation lets look at the following:

First off remember I'm not praising or punishing google or anyone else, these are facts and notes from past experience.

1. A sitemap is used by google to streamline the crawl process by identifying pages to crawl. Not all pages that are crawled need to be listed on the site map, it could be considered basic directions around your site.

2. Goog will, and we've probably all seen this happen, compare indexed links against your sitemap. So, for example, if you had a forum up, say a year ago, and took it down. You may find http errors listed because these older pages are now 404.

3. The sitemap tool does give goog a way to communicate with you that they have links for your site indexed that are no longer available. By showing both "in sitemap" errors as well as non sitemap errors it does give you a chance to fix the problem before those links are removed. Take the forum example above. 301 those links to the homepage as a fix.

4. The sitemap is a tool, but every time I see errors listed against a sitemap I've found the errors in the site logs as well. On most occasions, not only googlebot but MSNbot and others have run across the same error.

5. When checking the errors be sure that you pay attention to the date your sitemap was submitted, the last sitemap crawl and the error date. Quite often you'll find that the error was listed prior to the last submission and will probably go away, assuming the problem has been fixed. An example of this would be to click on the link provided in the error log. If the page comes up fine then your ok. Investigate the ones that are really broke. I don't loose a lot of sleep of an error that really isn't. There's a thousand things that could have gone wrong when googlebot tried to hit an individual page. As I said before sitelogs will almost always point at something, even if it was just a server burp.

6. SEF work will give you a sitemap headache. End of story, if you're playing with the URLs for your site with an SEF tool any small change could effect the entire site and thus effect your sitemap and indexed links. Don't get me wrong I am completely in favor of most of these tools and think they can do wonders in the long run, but just don't be surprised if it takes your sitemap errors a while to calm down and be sure to 301 any links that changed. Especially the ones that show up in your sitemap error list.

7. Don't loose sleep over it, if you feel that sitemaps are not for you then.... remove it. I've seen a hundered stories in this forum alone, that would justify removing the sitemap. All I can say is that I would be willing to bet the reasons fall into one of these categories. If not, heck, get rid of the thing and move on.

8. Last item, I promise. Sitemap + sitelogs will give you the best results. Goog Sitemaps is new and is bound to cause a little heartache but as I said earlier it helps the site owner, the developer and goog all by ensuring that the index is accurate and updated on a timely basis.

 

Did I read this right. You changed the 1,000,000 page extension from ASP to HTML?

If you changed your file name for webpage.asp to webpage.html, this is the cause of loosing everything. You have altered the PR ranking completely. You "deleted" the pages that had PR assigned to them and added new ones that did not have PR assigned to them.

This is one of the major problems with upgrading sites from ~ ASP to DotNet. The extensions change and you will lose everything. This also happened with the MIVA shopping cart when they changed extensions from MV to MVC. Total PR reset of the site.]

You will have about 3 months from the time you did the file name change to undo it.

If you need help, send me a private message. Many of our clients have done this.

Jim Catanich
www.catanich.com

 

Yep, a change in the file extension is a killer unless you remember to 301 the older files. This can be done via your .htaccess file so it's not a deal breaker but I can't tell you the number of times I've forgotten it myself.

You'll probably loose some PR even with a good 301 but you'll should pick it back up in the long run.