SEO Article:
Search Bots, Crawlers, and Spiders
If you are a webmaster and you review your logs, often you will see a bunch of really strange hits. They aren't humans, you can't tell their operating system or their browser! Who are these pesky little creatures who rummage around the internet all the time?
Not quite sure what I am talking about? Here is a few examples of various bots searching my website:
207.68.146.40 (msnbot.msn.com)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
This is the MSN Search bot.
207.68.146.40 (lj2070.inktomisearch.com)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
This is Yahoos Search Bot.
66.249.65.147 (crawl-66-249-65-147.googlebot.com)
Mediapartners-Google/2.1
This is Googles bot, that searches your webpages for AdSense.
What is a Bot, Crawler, Spider?
These terms are all the same, they all refer to an automated program that goes from website to website caching and processing the pages for search engines. As you know, "WWW" means World Wide Web, thus "Spider" seemed like an appropriate term. Crawler is another term that just describes what it does, crawling from site to site and page to page endlessly. Bot, is actually short for "robot" and again is just an automated program to index websites.
What is the purpose of a Spider?
A spider looks at all the pages of your website, and uses that information to rank you in search engines (how high you will list in a search result), and cache a copy of your page on their server for quick reference, and if your site ever goes down. Spiders jump from link to link on the Internet and run endlessly, even if you never submit your website to a search engine, odds are your site will still be spidered.
Can I stop bots and spiders from searching my website?
Yes and no. Legitimate spiders are run by reputable organizations that follow certain rules. For instance, most companies have a policy that their robot will search for a file called "robots.txt" in the root of your website. This text file is filled with information telling the bots what and what not is allowed to be viewed. Unfortunately, there are also bad bots out there, they search the internet harvesting e-mail addresses for spam and other bad things, these bots often don't comply with the "robots.txt" standard.
How many bots are there?
It's impossible to guess how many bots are out there searching websites. On any given day I will get roughly 10 different ones check my website. Some of them only search one or two pages, others go over my entire website. Not all of them give you a good description of what they do, or who owns them. If you cut and paste their name and IP address in to Google, quite often you can find more information about what they do.
How can I get my site spidered?
As I mentioned before, if your website is up long enough, it "will" get spidered eventually. However, if you want to ensure that it gets done within a few months, go to the various search engine websites and look for the "Add URL" or "Suggest a Link" pages. DMOZ is one of the big directories which you should submit your site. When you sign up for these search engines, your website is automatically queued up to be spidered. It may take several weeks or months to actually start showing up on the search engine, even after you see the robot spidering your website.
What about pay search engines?
There are a bunch of different search engines that make you pay to have your website listed. I personally don't support these search engines, I find that most people use the big free search engines anyway. However, if you do wish to get included in some search engines faster, many have payment options which will get your site listed within a couple of days.
Ken Dennis
Related Search Engine News and Articles
Corsavoo.com, France - 7 hours ago
Let's say I want to target the keyword, Search Engine Optimization. So I put my title as "Tips on Search Engine Optimization". So, my title is 60% optimized
Promotion World (press release), CA - 16 hours ago
So why does the average search engine optimization company pay such little regard to professional design? Because search engine optimization specialists are
Search Engine Roundtable - 11 hours ago
Earlier this week, we reported that Google Suggest was going to become more mainstream -- that is, that it would be the search default.
Best Syndication
Best Syndication, CA - Aug 26, 2008
The medium used for promoting a website is search engine optimization (SEO). If you are seeking services for getting maximum visibility for your website,
PR Web (press release), WA - Aug 26, 2008
Brick Marketing, an internet marketing company, has recently revised and re-launched their search engine optimization services to adjust with growing trends
Search Engine Roundtable - Aug 26, 2008
Vanessa Fox blogged about her experience at SES San Jose last week. She attended the highly talked about White Hat, Black Hat session and explained her
Related Search Engine Optimization Articles:
How To Choose Keywords Before they Skyrocket in Popularity
Long before the days of researching phrases with the helpful online resources of today, the art of keyword/phrase selection was often left just to guesswork. However, guesswork by today's highly competitive standards is just not good enough. So how is it possible to select a powerful traffic-pulling phrase in this fashion without researching it in real time? Better yet, is it possible to choose such a phrase and get positioned before it becomes popular?
Professional SEO: Hand Off to Bob or Outsource the Job
We are often asked if professional SEO (search engine optimization) can be done effectively utilizing in-house talent. Despite our obvious self-interests on the subject, our answer is always a qualified "yes"- you can achieve professional SEO results using existing talent. However, for every company we have known that has met with great in-house SEO success, we know of many more that have seen their in-house efforts fail. We have also discovered the companies that have succeeded share some common traits.
Sales And Crawlers, Update! Update! Update!
The importance to the algorithmic web crawlers that speed throughout the web is crucial to the successful marketing campaign of your site. Your site would simply be a pretty compilation of HTML, XML, Java and the sort if search engine crawlers did not come around and look over your site. These amazing little creatures are absolutely incredible software inventions those only purpose in life is to "crawl" the web.
Search Engine Optimization - A Beginners Guide
Getting your site listed in the top search engines, such as Google, Yahoo, or MSN is no small job. There is lots of work that needs to be done to guarantee the highest placement possible, and even more work is needed to keep your ranking for any period of time. Here are some simple tips and strategies to keep your site listed, and listed well, without spending any extra cash on pay per clicks.
Finding Targeted Keyword Phrases Your Competitors Miss
Finding Targeted Keyword Phrases Your Competitors Miss
How to Improve Your Search Engine Positioning and Increase Traffic Today
Every website has times when traffic is higher than others. However, in the downtimes you need to figure out why your traffic is lower and what you can you about it. The following suggestions have been proven to increase website traffic and will be effective in getting you more customers. However, before implementing these tips into your website promotion plan make sure you have a clear understanding of how to perform them effectively because if you are not aware of how to do something right, it could possibly backfire and work against you.