metalcherry

I have been told crawlers and Spiders help your site, can someone give some advice how to try and increase the activity of crawlers and spiders.

I have recently installed a robots.txt file which i found on the internet but any other advice which would be great.

__________________

Phil2511

I have found that the more people that post on my forum and upload photos and then the subsequent comments those photos attain has increased the activity of crawlers on my site.

Basically the crawlers see that there is more content added and so need to send more crawlers to be able to crawl the ever increasing content.

Last weekend i looked at my google webmaster tools and the average crawl rate was 112 pages per day. Today the average is 244.

If you dont add new content or add only very little then the cawlers dont need to crawl as often because they already know whats on your site.

__________________
Phil
My Neapolitan Mastiff Information Site

mars guitars

You can set the Google crawl rate to 'Faster' using Google Webmaster Tools.

As Phil says, the more frequently your content changes the more often the spiders will visit - if they keep seeing 'nothing changed' they will see no need to keep coming back to check.

__________________
Guitar strings and accessories, Rotosound bass strings, Guitar Capos and Guitar Gifts from Mars Guitars UK
. . . .under development.. guitar string guide

Phil2511

In my google webmaster tools it says that i have no need to set to faster crawl rate at this time and the option is unavailable to select.

Only normal and slow is there to choose from

__________________
Phil
My Neapolitan Mastiff Information Site

jumpenjuhosaphat

It's based off of how often your content changes(I know Mars and Phil, just reiterating )......The less your content changes, the less need the spiders have to crawl your site. If your site changes daily, then you will see the spiders nearly daily, and in some cases, several times per day.

__________________
Storage Sheds
Lost Forum

usearchme

Hi

Its really a non issue phil, if your content changes a lot, the frequency of the spiders visits changes, but it wont get you any higher on the search engines, its handy if you want any changes on ur site to be seen quicker.

Woc

__________________
Free Chat Rooms Free Online Chat Rooms

Webnauts

Make sure that your web server supports the If-Modified-Since HTTP header, because this feature allows your web server to tell Google and other search engines whether your content has changed since they last crawled your site. Also supporting this feature will save you bandwidth and overhead.

__________________
SEO Workers - Organic Search Engine Optimization

Spency

I have recently introduced a daily news service, which means we add at least one new page every day, and change three others every day.

Since this has been done, the crawl rate has risen dramatically.

But, as others have mentioned, it bears no effect on your placings in all honesty.

__________________
Bueno Property Investments - http://www.buenoinvestments.com / Directory 4 Property - http://www.directory4property.co.uk

frutta

Use XML-Sitemaps.com to generate free sitemap.html, ror.xml, urlist.txt and sitemap.xml

__________________
No.1 Gelato in Singapore. Free Tasting.
Frutta La Viva :: Best Gelato

Webnauts

And what will happen then?

__________________
SEO Workers - Organic Search Engine Optimization

hasan_889

Without changing content, speeding up crawlers is useless ... hope you will understand

__________________

mit

Yes, it needs continuous updates (content refreshment + something new which your visitors and SEs spiders can like to surf).

__________________
Search engine optimization company
Webmaster Forums - Fun Place to Join
Raksha Bandhan

hem1234

Frequently update your website, change the content, add new pages in short time. Google will automatically crawl your website fastly. Also change crawl speed from google webmaster option. Digg.com, Forums, etc will also helpful. So place the link of your website there.