| Return to iWEBTOOL |
| |
|
| |||||||
 | Welcome to iWEBTOOL Talk, where you talk about
webmaster-related stuff.
1 Register
2 Browse the board
3 Discuss whatever may interest you! | |||||||||||||
 |
| | Thread Tools | Search this Thread | Display Modes |
10-05-2007, 08:18 PM
| #1 |
| Member Join Date: Aug 2006 Location: chichester england
Posts: 189
 |  Adding a robots.txt page, some questions I added a robots.txt blank page to our site as advised when some pages did not get indexed by Google. But I don't think I did it correctly because it went into a sub directory and now I see it should have gone into the root, is that right? But also, I'm a bit worried about the disallow function. Am I in any danger of doing something I shouldn't if I move it to the root? The page is blank with just robots.txt as the extension. John ps The pages were indexed after linking to them, but I'm thinking of the future and want to do things the right way. |
|
|
| |||||||
|
10-05-2007, 09:41 PM
| #2 |
| iWEBTOOL SEO Advisor Contributor Join Date: May 2006
Posts: 1,267
| Re: Adding a robots.txt page, some questions Hi Frank, yes robots.txt is only understood in the root. You might disallow something you don't intend to. Just check that it's correct. You can use the Google robots check in the webmasters tool or maybe even easier to use this tool http://tool.motoricerca.info/robots-checker.phtml
__________________ Eco Tours | Gift Baskets | Abundance and Prosperity See a good post? <--Give thanks by clicking  left of post.
|
|
|
|
10-09-2007, 11:30 AM
| #3 |
| Smurf Join Date: Sep 2007
Posts: 8
| Re: Adding a robots.txt page, some questions Does robots works for google? I think that noindex and nofolow pages work.
__________________ Webhostingbuzz.com No web host is perfect. But we're damn close... |
|
|
|
10-09-2007, 01:14 PM
| #4 | |
| iWEBTOOL SEO Advisor Contributor Join Date: May 2006
Posts: 1,267
| Re: Adding a robots.txt page, some questions Quote:
Yes robots.txt works for google. The Meta 'Robots' tag is a seperate issue and perhaps you can prevent some bots from crawling that way. However John's question is about getting Google to crawl the site and not stopping the site from being crawled.
__________________ Eco Tours | Gift Baskets | Abundance and Prosperity See a good post? <--Give thanks by clicking left of post.
| |
|
|
| (Threads which have no activity for more than 30 days are automatically closed.) |
| Quick Reply | ||
|
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | Search this Thread |
| Display Modes | |
|
|
Similar Threads | ||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| problem with IE not doing the same as FF - Not adding up $ | CircleOfLinks | Programming | 0 | 08-19-2007 01:11 AM |
| open and retrive a text file(.txt) using JSP | noche_arya | JavaScript | 1 | 07-25-2007 01:40 AM |
| Robot.txt File | myareanetwork | Programming | 10 | 06-01-2007 11:15 PM |
| what Difference between RSS-sitemap-url.txt | mhmdkhamis | Search Engine Optimization (SEO) | 0 | 12-15-2006 08:19 AM |
| site blocked with robot.txt for 6 months! | walkthetalk | 8 | 07-24-2006 08:31 AM | |
Linear Mode
