Google adds pages into index if they are not in sitemap

I released a new site bit over month ago. It's a sub-domain site to a directory I'm publishing. Naturally I added sitemap of the new site to Google Sitmaps and it took only few days when I saw that my sitemap was noticed by Google -> sitemap was downloaded and status was OK. Crawl stats told that the pages were crawled successfully and some of the pages had even a medium PR.

But for some reason I couldn't see my pages in Googles index - it took more than month before I saw any results more than frontpage of the site. And what is the most strange thing - when I saw new pages in the Googles index, these pages were not included in the sitemap! These pages were followed by dynamical links from my main site and there is big chance that these pages does not include any content!

I have a script that tells me everytime when Gbot crawls my pages and I know that the pages from my sitemap are crawled but not added into index. The biggest difference with the indexed pages and the pages in sitemap is that the indexed pages are crawled by Mediabot too. So that much for the lie that Mediabot does not affect the Google index.

 

Hi,

Two things:

1. Google says

2. Matt Cutts explains why Google uses Mediapartners information in the search engine here.

I think that these two pages explain what happens with your new subdomain.

Jean-Luc

 

I have also bought a new domain and set up a site bit over a month ago. Almost immediately I submitted the sitemap and very short after that it was noticed by Google and showed status OK.

But I also had to wait about a month before my pages were showing in the index. So I believe that's more or less a common practice with Google for all sites...

And the fact that the pages listed in your sitemap weren't indexed by Google, here's what I have to say...
I know that your pages need to be linked to from some other indexed page in order to be indexed, simply putting them in the sitemap won't place them in Google's index. You said it yourself about the pages that were indexed, that "These pages were followed by dynamical links from my main site".

If you have links from other indexed pages, then I suggest you first review the Google's Webmaster Guidelines (http://google.com/support/webmasters/bin/answer.py?answer=35769?hl=en) to see if by any chance your pages are not following their guidelines.

In the Sitemaps Login area, after you select your site, you can see your indexing summary - does it report any problems? Also, on the left, you have different types of crawl error reports. Have you checked those?
For example, my Indexing summary still tells me that some of my pages are partially indexed and until recently I had my home page under "Unreachable URLs" reporting as a "DNS error" (if I'm not mistaken, as it's no longer there so I'm going by my memory) and this stayed there for quite a long even though my whole website was in fact fully, not partially, indexed, along with the home page.

What I'm saying is Google Sitemaps is still in beta so if you're sure your site follows Google's guidelines for webmasters and that you have incoming links from other sites, I suggest you try with Google Sitemaps Group at https://www.google.com/webmasters/sitemaps/googlegroup?hl=en. They might be able to tell you what's happening with your site.

Hope this helped.

 

You might want to generate HTML sitemap as well... And make sure it is linked from your highest PR pages. Might solve some of the problem.