Stupid SpamBots Getting to me!

Hello, I own blitzgamer.com and currently have been getting spam bots coming to my site through google (I think) and writing reviews on my site that are just plain spam about various things.

What I don't understand is that you have to be a member to write a review, the reviews in the admin panel shows the review to not have a member associated with it. SO they are not a member..

I am pretty confused on the whole thing.

If anyone can help much thanks and rep will be awarded!

The site is being ran with a custom php script that the person I bought the site from created.

Regards,
Zack B
BlitzGamer.com

 

I got this robots.txt from another site having the same problem, just drop it into your root directory.

User-agent: BotRightHere 
User-agent: larbin 
User-agent: b2w/0.1 
User-agent: Copernic 
User-agent: psbot 
User-agent: Python-urllib 
User-agent: NetMechanic 
User-agent: URL_Spider_Pro 
User-agent: CherryPicker 
User-agent: EmailCollector 
User-agent: EmailSiphon 
User-agent: WebBandit 
User-agent: EmailWolf 
User-agent: ExtractorPro 
User-agent: CopyRightCheck 
User-agent: Crescent 
User-agent: SiteSnagger 
User-agent: ProWebWalker 
User-agent: CheeseBot 
User-agent: LNSpiderguy 
User-agent: Alexibot 
User-agent: Teleport 
User-agent: TeleportPro 
User-agent: MIIxpc 
User-agent: Telesoft 
User-agent: Website Quester 
User-agent: WebZip 
User-agent: moget/2.1 
User-agent: WebZip/4.0 
User-agent: WebStripper 
User-agent: WebSauger 
User-agent: WebCopier 
User-agent: NetAnts 
User-agent: Mister PiX 
User-agent: WebAuto 
User-agent: TheNomad 
User-agent: WWW-Collector-E 
User-agent: RMA 
User-agent: libWeb/clsHTTP 
User-agent: asterias 
User-agent: httplib 
User-agent: turingos 
User-agent: spanner 
User-agent: InfoNaviRobot 
User-agent: Harvest/1.5 
User-agent: Bullseye/1.0 
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) 
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 
User-agent: CherryPickerSE/1.0 
User-agent: CherryPickerElite/1.0 
User-agent: WebBandit/3.50 
User-agent: NICErsPRO 
User-agent: DittoSpyder 
User-agent: Foobot 
User-agent: SpankBot 
User-agent: BotALot 
User-agent: lwp-trivial/1.34 
User-agent: lwp-trivial 
User-agent: BunnySlippers 
User-agent: URLy Warning 
User-agent: Wget/1.6 
User-agent: Wget/1.5.3 
User-agent: Wget 
User-agent: LinkWalker 
User-agent: cosmos 
User-agent: moget 
User-agent: hloader 
User-agent: humanlinks 
User-agent: LinkextractorPro 
User-agent: Offline Explorer 
User-agent: Mata Hari 
User-agent: LexiBot 
User-agent: Web Image Collector 
User-agent: The Intraformant 
User-agent: True_Robot/1.0 
User-agent: True_Robot 
User-agent: BlowFish/1.0 
User-agent: JennyBot 
User-agent: MIIxpc/4.2 
User-agent: BuiltBotTough 
User-agent: ProPowerBot/2.14 
User-agent: BackDoorBot/1.0 
User-agent: toCrawl/UrlDispatcher 
User-agent: suzuran 
User-agent: TightTwatBot 
User-agent: VCI WebViewer VCI WebViewer Win32 
User-agent: VCI 
User-agent: Szukacz/1.4 
User-agent: Openfind data gatherer 
User-agent: Openfind 
User-agent: Xenu's Link Sleuth 1.1c 
User-agent: Xenu's 
User-agent: Zeus 
User-agent: RepoMonkey Bait & Tackle/v1.01 
User-agent: RepoMonkey 
User-agent: Openbot 
User-agent: URL Control 
User-agent: Zeus Link Scout 
User-agent: Zeus 32297 Webster Pro V2.9 Win32 
User-agent: Webster Pro 
User-agent: EroCrawler 
User-agent: LinkScan/8.1a Unix 
User-agent: Keyword Density/0.9 
User-agent: Kenjin Spider 
User-agent: Iron33/1.0.2 
User-agent: Bookmark search tool 
User-agent: GetRight/4.2 
User-agent: FairAd Client 
User-agent: Gaisbot 
User-agent: Aqua_Products 
User-agent: Radiation Retriever 1.1 
User-agent: Flaming AttackBot 
User-agent: Curl 
User-agent: Web Reaper
User-agent: Firefox
User-agent: Opera
User-agent: Netscape
User-agent: WebVulnCrawl
User-agent: WebVulnScan
Disallow: /

Code (markup):

The last two in the list are well know for crawling your site in the quest of vulnerabilities.

 

So let me see if I understand what you are suggesting.

Basically that robots.txt will replace mine which allows all robots..

The list that you provied only allows those robots and no others? While my robots.txt file allows all bots which is leading to the spam..

Is there any disadvantage of limiting to just those bots? Like what if a new search engine comes out or a smaller search engine that isn't listed in the robots.txt file?

Anyone else have any ideas opinions on the problem and the robots.txt file?

Regards,
Zack B

 

That list avoid the above robots crawl your site.

If you want to allow other mix this with your current robot.txt

 

Ah.... so the long list is too block those bots... and all others will be allowed?

Regards,
Zack

 

I put the robots.txt on my server.. but still getting spam... =(

 

I don't know much about it.. but it seems adding captcha in the registration page and review pages, should help you.

I was getting a lot of spam from the feedback page, i just added captcha and now it is good for me. Let's see if that works for you as well.

For ASP:
http://sourceforge.net/projects/asp-captcha

For PHP:
http://www.webhosting.net/forum/showthread.php?p=154

Have a great day!

 

If it's a custom made script then it has a hole somewhere.

The bots are smart enough to search out and post..

I would suggest both adding a captcha and tightening your code.

O

 

switch to captcha-based contact form.

 

Check you forms...75% of all coders leave loose forms,not by fault but they just aren't aware of the security that is required


I have bought so many scripts in the past, and the forms on the site were so wide that I could stick an elephant through it

Have a PROFESSIONAL coder look at your forms, and tighten them all up

I used to have this issue as well, and then I hired some really good people that new php real well and they locked my shit up tight

 

Yea I did the same thing I owned TheeForum and I had bots there all the time.

 

i had a simple solution to my forums i killed them First

1. Update your php version on your forums - Especially PHPBB - mine got Owned Several Times
make sure they are up to date

2. administrator approval on all messages or blog entrys

3. Or just kill the forums from your website - takes the fun out thou....

 

Hey I have a question to that list you posted.

Will that block all users having these browsers or do I missunderstand something?

 

robots.txt will not help against spammers at all because they ignore the directives. spambots must be fought using technology not polite suggestions.

 

Do spambots really care what rules you set in robots.txt?

tom

 

lol, yes, the robots.txt def didn't stop the spamming..

does anyone know of a good php coder who has knowledge of putting in image verification for posting comments...?

Thanks

 

im using the Textual Confirmation Mod and it has been able to block 95% of spam registrations in the site.

 

Is that only for phpbb forums?

I am looking for just a form submit image verification system. Not for a forum.

 

You need to go back to the person who wrote it and make them fix it. This has nothing to do with spam bots. There is a simple problem here : Members should be the only ones posting reviews and yet, non-members can post reviews.

That is a bad script. There will ALWAYS be spammers, but you don't always have to have a bad script. Blocking spam bots by useragent name means nothing at all. I could crawl your site now with a useragent name from google. You could tell by IP that I was not google, but it would go right by that ridiculously long robots.txt. robots.txt is NOT for security.. AT ALL.

 

A couple thoughts:

1) A robots.txt will NEVER block spambots because they never obey robots.txt.
2) Implement CAPTCHA on your forms and make sure people have to be logged in to post. Log their IP addresses, and if one of them spams, ban them.
3) When it is released, install Crawlwall.

If you want to know more about banning bots, read these blogs:
1) http://willmacc.wordpress.com/
2) http://spamhuntress.com/category/bots/
3) http://incredibill.blogspot.com/