Spiders the Good and Bad
and Sometimes Ugly
How To Get The Most Out Of Them:
I'm not talking about spiders that you see crawling around on webs; well that's not exactly right, maybe all spiders crawl on webs. Let me try this again, these spiders won't be the 8 leg variety out crawling around where you can see them; without some special tools. The Internet (web) spiders are just like the real arachnid (had to look it up) in the fact, they can be both good and bad. The ugly, is when things change and your rankings go down the drain. Lets hope you don't get caught up in any of the schemes to fake out search engines and the spiders shouldn't hurt you.
If you are into SEO (Search Engine Optimization) you will want to get visited by the various search engine and directory spiders. They can help you get moved up in page ranking, and lead to more visitors to your site (good). Then there are the second type and they are a lot more sneaky; they go out and try to harvest email addresses and other information. Probably for either selling to list developers, or just plain old spamming. I hate spam.
Spiders are hidden to most of the people on the Internet, it is after you begin studying visitors to web pages that you will actually detect, or see them. Some are invisible even then, you just see a number without any identification, so you can't be sure who visited. Sometimes you can use "Who Is" and find out whose site it is by typing the number in. Others are not as sneaky, they leave there calling card, so you will know they visited. The search engines stay in business by finding the most and latest updated data or content for their visitors.
When people come on the net, they are generally looking for something, or sending email. The major search engines make their money selling advertisements, and the more visitors they have; the more they can get for their ads. They also add cost to others; if you have to encrypt, or scramble your email address, or be added to some undesirable lists. I think all of us have been caught up in this type of bad spider. There are several encrypting programs available to keep email addresses from being harvested. Some are better than others.
I will be the first to admit, the Internet as we know it today without search engines; probably wouldn't even be in existence except for spiders. On the other hand, they are being well paid for offering an indexing service. Today, I actually wonder, if we are getting an unbiased opinion when we search? I know that suppliers pay for top listings on the search engines. I also am told that the people that does the searching rarely go more than 3 pages deep in their searches. It is this aspect, that allow the major search engines to be paid big dollars for getting sites keywords placed upon the first few pages.
Search engines use something called keywords for creating the list you see when searching. You wouldn't believe what some suppliers pay for each hit they receive for some of these keywords. I guess, I just wish I could afford to pay for these hits, I know they have to have a high conversion rate or very expensive prices to recover expenses. Keywords is another subject and will require another page to explain how they are used.
I sometimes wonder, if I miss some of the best deals by not digging deeper in the search engine pages. I tend to jump around and check out some of the pages below the first 3. If I'm looking for a good deal, I check out more pages and go deeper in the listings. Sometimes I find better deals by doing this little extra searching at other times I don't. Well, now you know a little about spiders and how they work; some have worse bites than others. I hope you and I only get visited by the good spiders and they see the value of our websites.
Maybe I've gone too far, I was just looking at my statistics and I have more spiders visiting than prospects. Now, if they would only move me up in the page rankings. I must be doing something that Yahoo doesn't like, or they have their spider disguised. I see just about every other search engine listed but only a few prospects; must mean I don't have good keywords. I really don't worry too much about SEO, I figure if I keep adding content I'll stay listed. I'm pretty sure my linking strategy is way behind and I may have some bad links.
Spiders have been around for a long time and probably from the popularity of the Internet they will remain in some form or another for a long time to come. Security has become an issue of some and must be changed to keep orderly conduct among webmasters and Internet Users. Spiders have been developed for many reasons, some good, some bad. Spiders are used to help us find information easier and now can be given orders to "not follow" those spiders are not as harmful as others that is searching for ways to overcome your privacy. It looks as if we might as well become adjusted to the idea that we have to live with both the good and bad spiders; so we might as well get used to it.
