Robot.txt File

myareanetwork · 05-28-2007, 01:51 PM

Is the robots.txt file extremely important. I'm not sure if I really need it since I don't mind if my website is crawled by the search engines every day. But is there something that I'm not understanding and should really consider using a robots.txt file?

Bagi Zoltán · 05-28-2007, 02:04 PM

No, the robots.txt file is not important unless you want to exclude some files or folders from indexing. If you have no problem with the indexed files, i don't suggest you to use that.

myareanetwork · 05-28-2007, 02:12 PM

Ok thank you. Maybe in the future but my website is still young so I want the search engines to keep crawling it as much as possible anyways. Appreciate it!

Chart Music · 05-28-2007, 03:00 PM

Not important really, but having a site map listed there might be of use once you have created a site map file.

Flipflop · 05-28-2007, 03:15 PM

Can also handy for restricting Google image if you don't wish your images to be list in Google images searches, can help control bandwidth, if you have restrictions.

User-agent: Googlebot-Image
Disallow: /

DjFlush · 05-29-2007, 11:30 AM

Important in future I must say. Sitemaps.org has updated their policy recently.

All search engines like Google, Yahoo and MSN will check your robots.txt to see the link to your XML Sitemaps [if you have any]. So its a good chance that you dont manually need to submit your sitemaps to Google Sitemaps and it will ensure that your site gets indexed by search engines

chromatography · 05-31-2007, 04:35 PM

Quote:

No, the robots.txt file is not important unless you want to exclude some files or folders from indexing

The inclusion of certain files/directories in robots.txt may increase the security risk for your site. I've seen some robots.txt files that had way too much information regarding the structure of the site, the admin folders and etc. Anyone can view robots.txt from their browser, and hackers may use them to get access to your server.

sri_gan · 06-01-2007, 02:35 PM

myareanetwork,

robots.txt is one way of controling the web robots and its a very good practice to have one and its part of the web standards.

Even though like chromatography says, hackers may have to chance to get into the restricted area, if your coding is tight its not possible for them to get in.

Chart Music · 06-01-2007, 02:59 PM

It's a good idea to have one. But it's not that important unless there is something you wish to tell passing bots.
Would recommend getting little familiar with the subject by reading the FAQ here
http://www.robotstxt.org/wc/faq.html

adstorm · 06-01-2007, 05:57 PM

When your site grows, or you have loads of dynamic pages, or you have a slow host and low bandwidth then you need one but for many web sites it's not necessary.

Albert Tai · 06-01-2007, 11:15 PM

Quote:

Originally Posted by myareanetwork

Is the robots.txt file extremely important. I'm not sure if I really need it since I don't mind if my website is crawled by the search engines every day. But is there something that I'm not understanding and should really consider using a robots.txt file?

If you have some important files or password directory or personal info you dont want search engines listing it right?
So use robot.txt

05-28-2007, 01:51 PM	#1
myareanetwork Junior Member Join Date: May 2007 Posts: 19	Robot.txt File Is the robots.txt file extremely important. I'm not sure if I really need it since I don't mind if my website is crawled by the search engines every day. But is there something that I'm not understanding and should really consider using a robots.txt file? __________________ Tampa Florida City Guide Baltimore Maryland City Guide

05-28-2007, 02:04 PM	#2
Bagi Zoltán iWEBTOOL Moderator Contributor Join Date: Aug 2006 Location: Hungary Posts: 1,844	Re: Robot.txt File No, the robots.txt file is not important unless you want to exclude some files or folders from indexing. If you have no problem with the indexed files, i don't suggest you to use that. __________________ █ Time may come when you will need an excellent eCommerce shopping cart software █ free seo related web directory online marketing Free shopping cart

05-28-2007, 02:12 PM	#3
myareanetwork Junior Member Join Date: May 2007 Posts: 19	Re: Robot.txt File Ok thank you. Maybe in the future but my website is still young so I want the search engines to keep crawling it as much as possible anyways. Appreciate it! __________________ Tampa Florida City Guide Baltimore Maryland City Guide

05-28-2007, 03:00 PM	#4
Chart Music iWEBTOOL SEO Advisor Contributor Join Date: May 2006 Posts: 1,260	Re: Robot.txt File Not important really, but having a site map listed there might be of use once you have created a site map file. __________________ Eco Tours \| Gift Baskets \| Random Keyword See a good post? <--Give thanks by clicking left of post.

05-28-2007, 03:15 PM	#5
Flipflop Member Join Date: Mar 2007 Location: Somerset, UK Posts: 219	Re: Robot.txt File Can also handy for restricting Google image if you don't wish your images to be list in Google images searches, can help control bandwidth, if you have restrictions. User-agent: Googlebot-Image Disallow: / __________________ █ Pregnancy and Parenting Advice █ Windows Hosting and Windows Reseller Hosting by Stormhosts █ 10% Discount on all hosting plans for iWebTool Members

05-29-2007, 11:30 AM	#6
DjFlush Newcomer Join Date: May 2007 Posts: 26	Re: Robot.txt File Important in future I must say. Sitemaps.org has updated their policy recently. All search engines like Google, Yahoo and MSN will check your robots.txt to see the link to your XML Sitemaps [if you have any]. So its a good chance that you dont manually need to submit your sitemaps to Google Sitemaps and it will ensure that your site gets indexed by search engines

06-01-2007, 02:35 PM	#8
sri_gan Member Join Date: Apr 2007 Posts: 116	Re: Robot.txt File myareanetwork, robots.txt is one way of controling the web robots and its a very good practice to have one and its part of the web standards. Even though like chromatography says, hackers may have to chance to get into the restricted area, if your coding is tight its not possible for them to get in. __________________ compare hotel prices from online travel industry leaders @ vtrip Make your suggestion about my website

06-01-2007, 02:59 PM	#9
Chart Music iWEBTOOL SEO Advisor Contributor Join Date: May 2006 Posts: 1,260	Re: Robot.txt File It's a good idea to have one. But it's not that important unless there is something you wish to tell passing bots. Would recommend getting little familiar with the subject by reading the FAQ here http://www.robotstxt.org/wc/faq.html __________________ Eco Tours \| Gift Baskets \| Random Keyword See a good post? <--Give thanks by clicking left of post.

06-01-2007, 05:57 PM	#10
adstorm Newcomer Join Date: Apr 2007 Location: Lichfield, near Birmingham Posts: 28	Re: Robot.txt File When your site grows, or you have loads of dynamic pages, or you have a slow host and low bandwidth then you need one but for many web sites it's not necessary. __________________ Rob from AdStorm Advertising Agency \| Marketing Agency

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
New MSN Robot	Katey	MSN	15	06-18-2007 01:32 AM
Expload Upload - Free file hosting + File SHARING Website	hemanthjava	Websites for Sale	0	02-25-2007 08:09 AM
Free File Hosting + File Sharing Website	hemanthjava	Websites for Sale	0	02-25-2007 08:08 AM
what Difference between RSS-sitemap-url.txt	mhmdkhamis	Search Engine Optimization (SEO)	0	12-15-2006 08:19 AM
site blocked with robot.txt for 6 months!	walkthetalk	Google	8	07-24-2006 08:31 AM