Increasing Crawlers and Spiders

metalcherry · 12-30-2007, 10:40 PM

I have been told crawlers and Spiders help your site, can someone give some advice how to try and increase the activity of crawlers and spiders.

I have recently installed a robots.txt file which i found on the internet but any other advice which would be great.

Phil2511 · 01-01-2008, 03:34 PM

I have found that the more people that post on my forum and upload photos and then the subsequent comments those photos attain has increased the activity of crawlers on my site.

Basically the crawlers see that there is more content added and so need to send more crawlers to be able to crawl the ever increasing content.

Last weekend i looked at my google webmaster tools and the average crawl rate was 112 pages per day. Today the average is 244.

If you dont add new content or add only very little then the cawlers dont need to crawl as often because they already know whats on your site.

mars guitars · 01-01-2008, 10:45 PM

You can set the Google crawl rate to 'Faster' using Google Webmaster Tools.

As Phil says, the more frequently your content changes the more often the spiders will visit - if they keep seeing 'nothing changed' they will see no need to keep coming back to check.

Phil2511 · 01-01-2008, 11:00 PM

In my google webmaster tools it says that i have no need to set to faster crawl rate at this time and the option is unavailable to select.

Only normal and slow is there to choose from

jumpenjuhosaphat · 01-02-2008, 12:55 AM

It's based off of how often your content changes(I know Mars and Phil, just reiterating

)......The less your content changes, the less need the spiders have to crawl your site. If your site changes daily, then you will see the spiders nearly daily, and in some cases, several times per day.

usearchme · 01-02-2008, 07:56 PM

Hi

Its really a non issue phil, if your content changes a lot, the frequency of the spiders visits changes, but it wont get you any higher on the search engines, its handy if you want any changes on ur site to be seen quicker.

Woc

Webnauts · 01-03-2008, 06:49 AM

Make sure that your web server supports the If-Modified-Since HTTP header, because this feature allows your web server to tell Google and other search engines whether your content has changed since they last crawled your site. Also supporting this feature will save you bandwidth and overhead.

Spency · 01-03-2008, 11:58 AM

I have recently introduced a daily news service, which means we add at least one new page every day, and change three others every day.

Since this has been done, the crawl rate has risen dramatically.

But, as others have mentioned, it bears no effect on your placings in all honesty.

frutta · 01-03-2008, 02:53 PM

Use XML-Sitemaps.com to generate free sitemap.html, ror.xml, urlist.txt and sitemap.xml

Webnauts · 01-04-2008, 01:21 AM

Quote:

Originally Posted by frutta

Use XML-Sitemaps.com to generate free sitemap.html, ror.xml, urlist.txt and sitemap.xml

And what will happen then?

hasan_889 · 01-04-2008, 02:14 AM

Without changing content, speeding up crawlers is useless ... hope you will understand

mit · 01-04-2008, 07:20 AM

Yes, it needs continuous updates (content refreshment + something new which your visitors and SEs spiders can like to surf).

hem1234 · 02-13-2008, 11:31 AM

Frequently update your website, change the content, add new pages in short time. Google will automatically crawl your website fastly. Also change crawl speed from google webmaster option. Digg.com, Forums, etc will also helpful. So place the link of your website there.

12-30-2007, 10:40 PM	#1
metalcherry Junior Member Join Date: Mar 2007 Posts: 55	Increasing Crawlers and Spiders I have been told crawlers and Spiders help your site, can someone give some advice how to try and increase the activity of crawlers and spiders. I have recently installed a robots.txt file which i found on the internet but any other advice which would be great. __________________ Music Music Music alterchatter.com - the Alternative Community alternativeforums.com

01-01-2008, 03:34 PM	#2
Phil2511 Junior Member Join Date: Nov 2007 Posts: 70	Re: Increasing Crawlers and Spiders I have found that the more people that post on my forum and upload photos and then the subsequent comments those photos attain has increased the activity of crawlers on my site. Basically the crawlers see that there is more content added and so need to send more crawlers to be able to crawl the ever increasing content. Last weekend i looked at my google webmaster tools and the average crawl rate was 112 pages per day. Today the average is 244. If you dont add new content or add only very little then the cawlers dont need to crawl as often because they already know whats on your site. __________________ Phil My Neapolitan Mastiff Information Site

01-01-2008, 10:45 PM	#3
mars guitars Junior Guru Contributor Join Date: Aug 2007 Posts: 745	Re: Increasing Crawlers and Spiders You can set the Google crawl rate to 'Faster' using Google Webmaster Tools. As Phil says, the more frequently your content changes the more often the spiders will visit - if they keep seeing 'nothing changed' they will see no need to keep coming back to check. __________________ Weird guitars! - in my guitar collector forum Find out about Guitar string gauges, Guitar string manufacturers and best guitar strings.

01-01-2008, 11:00 PM	#4
Phil2511 Junior Member Join Date: Nov 2007 Posts: 70	Re: Increasing Crawlers and Spiders In my google webmaster tools it says that i have no need to set to faster crawl rate at this time and the option is unavailable to select. Only normal and slow is there to choose from __________________ Phil My Neapolitan Mastiff Information Site

01-02-2008, 12:55 AM	#5
jumpenjuhosaphat Forum Admin Contributor Join Date: Jun 2006 Location: Denver Posts: 4,457	Re: Increasing Crawlers and Spiders It's based off of how often your content changes(I know Mars and Phil, just reiterating )......The less your content changes, the less need the spiders have to crawl your site. If your site changes daily, then you will see the spiders nearly daily, and in some cases, several times per day. __________________ Storage Sheds Lost Forum

01-02-2008, 07:56 PM	#6
usearchme Member Join Date: Nov 2006 Posts: 247	Re: Increasing Crawlers and Spiders Hi Its really a non issue phil, if your content changes a lot, the frequency of the spiders visits changes, but it wont get you any higher on the search engines, its handy if you want any changes on ur site to be seen quicker. Woc __________________ Free Chat Rooms Free Online Chat Rooms

01-03-2008, 06:49 AM	#7
Webnauts Smurf Join Date: Feb 2006 Location: Germany Posts: 18	Re: Increasing Crawlers and Spiders Make sure that your web server supports the If-Modified-Since HTTP header, because this feature allows your web server to tell Google and other search engines whether your content has changed since they last crawled your site. Also supporting this feature will save you bandwidth and overhead. __________________ SEO Workers - Organic Search Engine Optimization

01-03-2008, 11:58 AM	#8
Spency Member Join Date: Nov 2007 Posts: 199	Re: Increasing Crawlers and Spiders I have recently introduced a daily news service, which means we add at least one new page every day, and change three others every day. Since this has been done, the crawl rate has risen dramatically. But, as others have mentioned, it bears no effect on your placings in all honesty. __________________ Bueno Property Investments - http://www.buenoinvestments.com / Directory 4 Property - http://www.directory4property.co.uk

01-03-2008, 02:53 PM	#9
frutta Member Join Date: Aug 2007 Posts: 90	Re: Increasing Crawlers and Spiders Use XML-Sitemaps.com to generate free sitemap.html, ror.xml, urlist.txt and sitemap.xml __________________ No.1 Gelato in Singapore. Free Tasting. Frutta La Viva :: Best Gelato

01-04-2008, 02:14 AM	#11
hasan_889 iWebTool Moderator Join Date: Aug 2006 Location: London Posts: 953	Re: Increasing Crawlers and Spiders Without changing content, speeding up crawlers is useless ... hope you will understand __________________ Join, Ask & Share Your Knowledge - Online Money Making & Internet Entrepreneurs Forum Search Engine Optimization Blog \| Latest Coolest High-Tech Gadgets \| SEO Consultant in London

01-04-2008, 07:20 AM	#12
mit Senior Member Join Date: Feb 2006 Location: India Posts: 368	Re: Increasing Crawlers and Spiders Yes, it needs continuous updates (content refreshment + something new which your visitors and SEs spiders can like to surf). __________________ SEO Services Offshore software development WEBMASTER FORUM * CONTESTS * REVENUE SHARING

02-13-2008, 11:31 AM	#13
hem1234 Junior Member Join Date: Jan 2008 Posts: 87	Re: Increasing Crawlers and Spiders Frequently update your website, change the content, add new pages in short time. Google will automatically crawl your website fastly. Also change crawl speed from google webmaster option. Digg.com, Forums, etc will also helpful. So place the link of your website there. Last edited 02-20-2008 at 05:25 PM.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Spiders	detoam	Google	7	05-17-2007 09:27 PM
increasing optimisation PR etc	cindylittle	Introduce yourself	2	02-17-2007 07:14 PM
Increasing PR?	lauraddirectnet	Google	10	01-24-2007 12:59 AM
How can I tell when spiders last visited my site?	Cat	Search Engine Optimization (SEO)	9	11-18-2006 05:32 PM
How to make the spiders visit	travj1	MSN	9	11-06-2006 09:15 PM