Pesky Bots :)

compuXP · 12-30-2005, 05:43 PM

Okay, so I'm viewing the users online list, and I see that MSN Bot, Googlebot, and Yahoo bot are all on the forums indexing them.

Ironically, each one is viewing a "No permission" message - and Google was even trying to create an event in the calendar :rolleyes:

But actually, these are pretty smart bots, because they know not to index those "No permission" pages...

Clever

-Matt

Trojanfan · 12-30-2005, 07:31 PM

Yes, they are very good. Crazy to think how they designed them

Zack · 12-30-2005, 08:36 PM

That's cool. Wait, did they sign-up by themselves?
HTMLGuy

Trojanfan · 12-30-2005, 10:05 PM

That would be nice, I would ask for more

compuXP · 12-30-2005, 10:37 PM

No, they didn't sign up. Hence the permissions error.

dre · 12-31-2005, 12:51 AM

Whats are these bots coded in?

compuXP · 12-31-2005, 01:43 AM

C++

(text text text so that I can submit a short message)

DogFaceBoy · 12-31-2005, 03:46 AM

The bots are already apart of Vbulletin, as in they automatically get logged in by Vbulletin. How? By using a Robots.txt. They don't actually know not to index the no permission page, the Robots.txt disallows them to access them.

Read up on a Robots.txt file to learn more about how to use them on your site. Some search engines actually don't even crawl your page without one. But I don't think I would call them pesky bots, because truly you want them to be visiting your page as much as possible. Google visits my site 10,000 times everyday, which is a great thing. Everytime I add more content, in a few minutes it is already in the search engines gaining me traffic.

Hope this helps you understand how that works...

Regards,
Doggie

Zack · 12-31-2005, 02:04 PM

Got it thanks! I was just wondering how he saw it.
HTMLGuy

Edit: This is also my 100th Post

microdude431 · 01-01-2006, 01:21 AM

MSN bot is always indexing every single page. Google bot is still pretty cool, would be funny if it actually did create an even in the calender.

12-30-2005, 05:43 PM	#1
compuXP Retired Member Join Date: Dec 2005 Posts: 139	Pesky Bots :) Okay, so I'm viewing the users online list, and I see that MSN Bot, Googlebot, and Yahoo bot are all on the forums indexing them. Ironically, each one is viewing a "No permission" message - and Google was even trying to create an event in the calendar :rolleyes: But actually, these are pretty smart bots, because they know not to index those "No permission" pages... Clever -Matt __________________ Retired Moderator/Member/Friend/Helper - it was great working with all of you!

12-30-2005, 07:31 PM	#2
Trojanfan Member Join Date: Dec 2005 Posts: 150	Re: Pesky Bots :) Yes, they are very good. Crazy to think how they designed them __________________ Wood Sheds

12-30-2005, 08:36 PM	#3
Zack Junior Guru Join Date: Dec 2005 Location: North Carolina Posts: 759	Re: Pesky Bots :) That's cool. Wait, did they sign-up by themselves? HTMLGuy __________________ yaxal releases '07

12-30-2005, 10:05 PM	#4
Trojanfan Member Join Date: Dec 2005 Posts: 150	Re: Pesky Bots :) That would be nice, I would ask for more __________________ Wood Sheds

12-30-2005, 10:37 PM	#5
compuXP Retired Member Join Date: Dec 2005 Posts: 139	Re: Pesky Bots :) No, they didn't sign up. Hence the permissions error. __________________ Retired Moderator/Member/Friend/Helper - it was great working with all of you!

12-31-2005, 12:51 AM	#6
dre wo0h0o Join Date: Dec 2005 Location: Birmingham, UK Posts: 457	Re: Pesky Bots :) Whats are these bots coded in? __________________ Scrizzle.com - Dre's Blog Enjoy Yourself! SEOContest2008 Myspace Proxy

12-31-2005, 01:43 AM	#7
compuXP Retired Member Join Date: Dec 2005 Posts: 139	Re: Pesky Bots :) C++ (text text text so that I can submit a short message) __________________ Retired Moderator/Member/Friend/Helper - it was great working with all of you!

12-31-2005, 03:46 AM	#8
DogFaceBoy iWEBTOOL Staff Join Date: Dec 2005 Location: Canada Posts: 207	Re: Pesky Bots :) The bots are already apart of Vbulletin, as in they automatically get logged in by Vbulletin. How? By using a Robots.txt. They don't actually know not to index the no permission page, the Robots.txt disallows them to access them. Read up on a Robots.txt file to learn more about how to use them on your site. Some search engines actually don't even crawl your page without one. But I don't think I would call them pesky bots, because truly you want them to be visiting your page as much as possible. Google visits my site 10,000 times everyday, which is a great thing. Everytime I add more content, in a few minutes it is already in the search engines gaining me traffic. Hope this helps you understand how that works... Regards, Doggie

12-31-2005, 02:04 PM	#9
Zack Junior Guru Join Date: Dec 2005 Location: North Carolina Posts: 759	Re: Pesky Bots :) Got it thanks! I was just wondering how he saw it. HTMLGuy Edit: This is also my 100th Post __________________ yaxal releases '07

01-01-2006, 01:21 AM	#10
microdude431 Senior Member Join Date: Dec 2005 Posts: 349	Re: Pesky Bots :) MSN bot is always indexing every single page. Google bot is still pretty cool, would be funny if it actually did create an even in the calender. __________________ Free Webmaster Directory

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode