Bad robots, steal somewhere else

Bagi Zoltán · 12-28-2007, 08:13 AM

I usually support th UKWW with my programming solutions, but this time i will post it here. Last week a guy asked me if i could develop some protection against bad robots. Previously i enhanced a google proxy hack defending script and another one which is suiable to make your robots.txt file only for the robots, which are authorized to check it so he was very enthusiastic.

Some days ago i started a little survey on topic and found a lot of htaccess rule, where certain hosts are rejected via .htaccess, but they were not automatic, so the challange was given. Basically i used this resource to create this automatic solution.

1. Open your robots.txt and insert this line into it

Code:

Disallow: /core

if you don't have any create one with this content

Code:

User-agent: *
Disallow: /core

The name of the limited folder is not important, but would be great if the humanoid atteckers would find it very attractive to check it since this will be the live-bait. Bad robots and hackers are not interested in the crwaling limitation can be found in the robots.txt file

2. Create the folder on your hosting space which is specified in the robots.txt file, in my example this is root/core and upload an index.php file with the following content.

PHP Code:

   <?php
$ip = $_SERVER["REMOTE_ADDR"];
$logfile = 'bannolnilog.txt';
//collect the IP adresses or something else into the logfile
$fp = fopen($logfile, 'a');
fputs($fp, "$ip
");
fputs($fp, " "); 
fclose($fp);
echo "your IP was logged for security reasons and your visit is now over";
?>

3. As you may see i defined a $logfile where the IP adresses will be collected hence we need to upload to the same (core) folder a blank txt file called bannolnilog.txt (644 attributum).

4. We need to upload an other php file which will check if the visitor is bannished whenever a page is requested, i named this file validator.php and its content is the following.

PHP Code:

   <?php
$ip = $_SERVER["REMOTE_ADDR"];
$logfile = 'bannolnilog.txt';
$target = file(dirname(__FILE__). "/core/bannolnilog.txt");
foreach($target as $item){
$item = trim($item);
if(stristr($ip, $item)){
header("HTTP/1.0 403 Forbidden");
exit;
}
}
?>

5. You need to insert this line into the very front of your script header

PHP Code:

   <?php require "/you/need/to/insert/the/path/here/validator.php";?>

This will make the validater.php run before the page would be diplayed.

I warrant nothing, but works very well

You may truncate the logfile deleting the collected IPs.

midlandi · 12-28-2007, 09:47 AM

Great post Bagi, rep given. This is the most usefull thing I have seen posted here for ages.

Bagi Zoltán · 12-28-2007, 10:19 AM

I'm glad that you like it Midi, I have some posted already, but have a plan about creating an integrated security script system, but i have no idea when it will be ready, i guess in the next 6 months

midlandi · 12-28-2007, 10:27 AM

Bagi, would you mind if I added this to the Tutorials section at TriPHP
http://www.triphp.com/articles.php?cat_id=3

If so what link would you like to go with it?

Bagi Zoltán · 12-28-2007, 10:29 AM

Midi, feel free to add this post to the tutorial section. I think this time i need no link

midlandi · 12-28-2007, 10:33 AM

Great Bagi, thanks. That will be a great addition there

©®azy-¢øÐ3R · 01-02-2008, 09:23 AM

thx nice tut.

urstop · 01-02-2008, 01:33 PM

This is cool, even though it adds a little over head to each of the file, I think it is good , than get over content stealed :-)

We just need to make sure that the IP file does not get too big that users fell that our site pages are loading slowly.

Bagi Zoltán · 01-02-2008, 01:40 PM

Your suggestion is 100% justified urstop: When finalizing the future release of the script i will consider the auto truncate option after x day to easy the script load.
The new version is already completed with an url checker module which may help to log and ban the IP adresses of those visitros, who tries to attack the sql database via injection techniques.

12-28-2007, 08:13 AM	#1
Bagi Zoltán iWEBTOOL Moderator Contributor Join Date: Aug 2006 Location: Hungary Posts: 1,855	Bad robots, steal somewhere else I usually support th UKWW with my programming solutions, but this time i will post it here. Last week a guy asked me if i could develop some protection against bad robots. Previously i enhanced a google proxy hack defending script and another one which is suiable to make your robots.txt file only for the robots, which are authorized to check it so he was very enthusiastic. Some days ago i started a little survey on topic and found a lot of htaccess rule, where certain hosts are rejected via .htaccess, but they were not automatic, so the challange was given. Basically i used this resource to create this automatic solution. 1. Open your robots.txt and insert this line into it Code: Disallow: /core if you don't have any create one with this content Code: User-agent: * Disallow: /core The name of the limited folder is not important, but would be great if the humanoid atteckers would find it very attractive to check it since this will be the live-bait. Bad robots and hackers are not interested in the crwaling limitation can be found in the robots.txt file 2. Create the folder on your hosting space which is specified in the robots.txt file, in my example this is root/core and upload an index.php file with the following content. PHP Code: `<?php $ip = $_SERVER["REMOTE_ADDR"]; $logfile = 'bannolnilog.txt'; //collect the IP adresses or something else into the logfile $fp = fopen($logfile, 'a'); fputs($fp, "$ip "); fputs($fp, " "); fclose($fp); echo "your IP was logged for security reasons and your visit is now over"; ?>` 3. As you may see i defined a $logfile where the IP adresses will be collected hence we need to upload to the same (core) folder a blank txt file called bannolnilog.txt (644 attributum). 4. We need to upload an other php file which will check if the visitor is bannished whenever a page is requested, i named this file validator.php and its content is the following. PHP Code: `<?php $ip = $_SERVER["REMOTE_ADDR"]; $logfile = 'bannolnilog.txt'; $target = file(dirname(__FILE__). "/core/bannolnilog.txt"); foreach($target as $item){ $item = trim($item); if(stristr($ip, $item)){ header("HTTP/1.0 403 Forbidden"); exit; } } ?>` 5. You need to insert this line into the very front of your script header PHP Code: `<?php require "/you/need/to/insert/the/path/here/validator.php";?>` This will make the validater.php run before the page would be diplayed. I warrant nothing, but works very well You may truncate the logfile deleting the collected IPs. __________________ █ Time may come when you will need an excellent eCommerce shopping cart software █ free seo related web directory mérlegek Free shopping cart

12-28-2007, 09:47 AM	#2
midlandi iWEBTOOL Moderator Contributor Join Date: Oct 2006 Posts: 2,172	Re: Bad robots, steal somewhere else Great post Bagi, rep given. This is the most usefull thing I have seen posted here for ages. __________________ Webmaster Resources & Forums SubmitterBot - Semi-automatic directory submission tool

12-28-2007, 10:19 AM	#3
Bagi Zoltán iWEBTOOL Moderator Contributor Join Date: Aug 2006 Location: Hungary Posts: 1,855	Re: Bad robots, steal somewhere else I'm glad that you like it Midi, I have some posted already, but have a plan about creating an integrated security script system, but i have no idea when it will be ready, i guess in the next 6 months __________________ █ Time may come when you will need an excellent eCommerce shopping cart software █ free seo related web directory mérlegek Free shopping cart

12-28-2007, 10:27 AM	#4
midlandi iWEBTOOL Moderator Contributor Join Date: Oct 2006 Posts: 2,172	Re: Bad robots, steal somewhere else Bagi, would you mind if I added this to the Tutorials section at TriPHP http://www.triphp.com/articles.php?cat_id=3 If so what link would you like to go with it? __________________ Webmaster Resources & Forums SubmitterBot - Semi-automatic directory submission tool

12-28-2007, 10:29 AM	#5
Bagi Zoltán iWEBTOOL Moderator Contributor Join Date: Aug 2006 Location: Hungary Posts: 1,855	Re: Bad robots, steal somewhere else Midi, feel free to add this post to the tutorial section. I think this time i need no link __________________ █ Time may come when you will need an excellent eCommerce shopping cart software █ free seo related web directory mérlegek Free shopping cart

12-28-2007, 10:33 AM	#6
midlandi iWEBTOOL Moderator Contributor Join Date: Oct 2006 Posts: 2,172	Re: Bad robots, steal somewhere else Great Bagi, thanks. That will be a great addition there __________________ Webmaster Resources & Forums SubmitterBot - Semi-automatic directory submission tool

01-02-2008, 09:23 AM	#7
©®azy-¢øÐ3R Smurf Join Date: Jan 2008 Posts: 21	Re: Bad robots, steal somewhere else thx nice tut. __________________ Samari.org <----- webmasters forum comming soon... crazy-coding@hotmail.co.uk im a php coder msg me for jobs.

01-02-2008, 01:33 PM	#8
urstop Junior Member Join Date: Dec 2007 Location: India Posts: 52	Re: Bad robots, steal somewhere else This is cool, even though it adds a little over head to each of the file, I think it is good , than get over content stealed :-) We just need to make sure that the IP file does not get too big that users fell that our site pages are loading slowly. __________________ UrStop.com India's Property and Rental listings Search Tool UrStop.com India's Property and Rental listings Search Tool

01-02-2008, 01:40 PM	#9
Bagi Zoltán iWEBTOOL Moderator Contributor Join Date: Aug 2006 Location: Hungary Posts: 1,855	Re: Bad robots, steal somewhere else Your suggestion is 100% justified urstop: When finalizing the future release of the script i will consider the auto truncate option after x day to easy the script load. The new version is already completed with an url checker module which may help to log and ban the IP adresses of those visitros, who tries to attack the sql database via injection techniques. __________________ █ Time may come when you will need an excellent eCommerce shopping cart software █ free seo related web directory mérlegek Free shopping cart

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Need Help So Bad... :(	Evan	Help and Support	2	12-27-2007 02:59 AM
Even BIG Sites Steal Your Work: How Iwebtool steals from Directory Kit	xefus	Directories	2	10-27-2007 03:30 AM
Adding a robots.txt page, some questions	jfranktoo	Search Engine Optimization (SEO)	3	10-09-2007 01:14 PM
Not bad, I think	cross_on	Introduce yourself	6	05-27-2007 01:08 PM
Is too much really bad?	tag909	Discuss iWEBTOOL	10	09-30-2006 12:04 AM