My website is google/yahoo battlefield!

Google and yahoo are battling it out on my website, the googlebot and yahoo slurp are competing on who is faster at indexing my web store, it is eating up my bandwidth very fast. I hope in the end it is worth it. However I'm scared that Google might penalize me for dup content since my store is an affiliate program and therefore all the products, descriptions, etc there are the same as some other sites might have. Do you guys think I should write in robots.txt for google not to index my store?

 

wouldn't writing a robots.txt file contradict your original purpose of wanting to have your website in the google index...

 

I think it is possible to specify individual pages, I'm not sure though

 

if you're interested in preventing google from indexing individual pages, you can use the meta tag in the header of the page.

however, asking google not to index your site because you're worried you are too 'indexed' is just madness...

 

I know where you are coming from on this. On one of our sites we have a book section. The pages in this section are dynamically created with a common layout and thus very similar. Recently Google nailed almost all of those pages.

"If" Google penalizes the rest of the site because of this "Then" excluding that section via Robots.txt makes sense.

 

Resale Broker---

Please excuse my ignorance in advance ...

Is it possible to modify the content of the dynamically created pages?

 

I use robots.txt usually to exclude styles, scripts and templates from being indexed, while I "allow" Google and others to index my dynamic pages. Although most dynamic pages I have show totally different content text- and graphic-wise. If you have an amazon.com affiliated store e.g. I would suggest to exclude it for google bot because you might get penalized for "stealing" amazon.com content. It's stupid, but who said the google bot is smart ;-)

 

It's very possible to modify the "layout" of dynamically created pages. The content, in this case, is the data pulled from a database that populates the page.

 

Yes, that is exactly it, I modified the layout as much as possible and how is suits my site, so hopefully I won't get penalized.

 

modifying the layout won't help you much if you're worried about dup content, you need to eliminate some of the text you get from the affiliate feeds, and add more related text from some other source.