Tags |
|
|
Links |
|
|
Other Added - The Robots Text File Or How To Get Your Site Properly Spidered, Crawled, Indexed By Bots
Many advertisers had experienced much frustration with Google not so long ago. Some tried to adjust to changes and lower they cost per click in their campaigns, some gave up all together and, at least temporarily, moved their efforts to Yahoo and MSN.com. Looking at these giants in search engine industry a parallel can be drown to a set of other two giants: Pepsi-Cola and Coke.The competition between Pepsi-Cola and Coke included one common element. While both may have had different approach to managing and marketing, both tried to improve their product so that each would become a favorite drink of many people. But in the midst of all of their efforts one thing is clear, in order to succeed Pepsi-Cola and Coke needed each other. They have learned from each other successes and failures.Google, Yahoo and MSN.com the three leading giant in search industry need each other. They need to compete on two fronts, to offer advertisers great product and provide good Internet experience to every one else. The most common explanation from Google about recent changes was that they strive to provide Internet user with better experience. After those changes many advertisers had adjusted to changes and one can only hope that the end of this competition will result in great products in pay per click advertising industry as well as in wonderful Internet experience for the every one else.
On a different note, nowadays, all search engines look for the robots.txt file as soon as their robots arrive on your site. There are unconfirmed rumors that some robots might even 'get annoyed' and leave, if they don't find it. Not sure how true that is, but hey, why not be on the safe side?
Again, even if you don't intend to block anything or just don't want to bother with this stuff at all, having a blank robots.txt is still a good idea, as it can actually act as an invitation into your site.
Don't I want my site indexed? Why stop robots?
Some robots are well designed, professionally operated, cause no harm and provide valuable service to mankind (don't we all like to "google"). Some robots are written by amateurs (remem
Try this; in a Google search page, enter the word, 'Google.' How many results do you get?Over 2.5 Billion.Zap back to the mid-90's; two students named Larry Page & Sergey Brin are working on a new research project at Stanford University, California. Using the leading search engines of their day; Inktomi and AltaVista, they're entering the word 'Inktomi' into the Inktomi search engine and 'AltaVista' into the AltaVista search engine.How many results do you think they got; a Thousand? A Million?Zero.Larry & Sergey couldn't believe it. Because Inktomi and AltaVista search engines were the leaders, but couldn't even find their very own websites in an online search. This added to the boys' conviction. Now they were sure they'd soon be blasting the competition with their own new creation; Google.Google didn't get to be king of search and PPC (Pay-Per-Click) by thinking like everyone else. They worked 'outside the box' to develop an alternate plan of attack. The results; obscene profits and domination of the search engine world.In short, Google used imagination to get where they are today.To beat competitors in your business category you must use imagination too. Here are 5 top techniques that'll boost your performance. While your ad is climbing the Results ladder, you'll be getting far more attention from prospective customers.Introducing these tips may take a little time and effort (perhaps that's why most advertisers aren't using them). But there are resources available to help you. What's more, these methods won't cost you a penny.1. Tailor the Web Address Displayed in Your AdsLet's say Mr Smith is searching online for apples. Meanwhile, you own 'Fruit' dot-com. If you happen to have the word, 'apples' in your ad's web address, w
There are many folks out there who vehemently insist on the uselessness of the robots.txt file, proclaiming it obsolete, a thing of the past, plain dead. I disagree. The robots.txt file is probably not in the top ten methods to promote your get-rich-fast affiliate website in 24 hours or less, but still plays a major role in the long run.
First of all, the robots.txt file is still a very important factor in promoting and maintaining a site, and I will show you why. Second, the robots.txt file is one of the simple means by which you can protect your privacy and/or intellectual property. I will show you how.
Let's try to figure out some of the lingo.
What is this robots.txt file?
The robots.txt file is just a very plain text file (or an ASCII file, as some like to say), with a very simple set of instructions that we give to a web robot, so the robot knows which pages we need scanned (or crawled, or spidered, or indexed - all terms refer to the same thing in this context) and which pages we would like to keep out of search engines.
What is a www robot?
A robot is a computer program that automatically reads web pages and goes through every link that it finds. The purpose of robots is to gather information. Some of the most famous robots mentioned in this article work for the search engines, indexing all the information available on the web.
The first robot was developed by MIT and launched in 1993. It was named the World Wide Web Wander and its initial purpose was of a purely scientific nature, its mission was to measure the growth of the web. The index generated from the experiment's results proved to be an awesome tool and effectively became the first search engine. Most of the stuff we consider today to be indispensable online tools was born as a side effect of some scientific experiment.
What is a search engine?
Generically, a search engine is a program that searches through a database. In the popular sense, as referred to the web, a search engine is considered to be a system that has a user search form, which can search through a repository of web pages gathered by a robot.
What are spiders and crawlers?
Spiders and crawlers are robots, only the names sound cooler in the press and within metro-geek circles.
What are the most popular robots? Is there a list?
Some of the most well known robots are Google's Googlebot, MSN's MSNBot, Ask Jeeves's Teoma, Yahoo!'s Slurp (funny). One of the most popular places to search for active robot info is the list maintained at http://www.robots.org.
Why do I need this robots.txt file anyway?
A great reason to use a robots.txt file is actually the fact that many search engines, including Google, post suggestions for the public to make use of this tool. Why is it such a big deal that Google teaches people about the robots.txt? Well, because nowadays, search engines are not a playground for scientists and geeks anymore, but large corporate enterprises. Google is one of the most secretive search engines out there. Very little is known to the public about how it operates, how it indexes, how it searches, how it creates its rankings, etc. In fact, if you do a careful search in specialized forums, or wherever else these issues are discussed, nobody really agrees on whether Google puts more emphasis on this or that element to create its rankings. And when people don't agree on things as precise as a ranking algorithm, it means two things: that Google constantly changes its methods, and that it does not make it very clear or very public. There's only one thing that I believe to be crystal clear. If they recommend that you use a robots.txt ("Make use of the robots.txt file on your web server" - Google Technical Guidelines), then do it. It might not help your ranking, but it will definitely not hurt you.
There are other reasons to use the robots.txt file. If you use your error logs to tweak and keep your site free of errors, you will notice that most errors refer to someone or something not finding the robots.txt file. All you have to do is create a basic blank page (use Notepad in Windows, or the most simple text editor in Linux or on a Mac), name it robots.txt and upload it to the root of your server (that's where your home page is).
On a different note, nowadays, all search engines look for the robots.txt file as soon as their robots arrive on your site. There are unconfirmed rumors that some robots might even 'get annoyed' and leave, if they don't find it. Not sure how true that is, but hey, why not be on the safe side?
Again, even if you don't intend to block anything or just don't want to bother with this stuff at all, having a blank robots.txt is still a good idea, as it can actually act as an invitation into your site.
Don't I want my site indexed? Why stop robots?
Some robots are well designed, professionally operated, cause no harm and provide valuable service to mankind (don't we all like to "google"). Some robots are written by amateurs (rememb
For many new high school graduates (and those approaching graduation) the question looming in your mind is whether you should go on to college, or jump right out into the workforce.It's a confusing time. And you are bound to get all kinds of different answers from different people. This only adds to the confusion.So what do you do?Well, everyone is different, with varying goals and desires, so it's impossible to give an answer that fits all. But let me go over the benefits of one possible course of action:Work a while, and then go to college.Again, this is not the answer for everyone, but let's explore this option a bit anyway.Many young adults just don't know what they want to do when they finish high school. They have been forced to go through years of unending primary and secondary education, and that's about it. They understand that once school is finished, there's this thing called work that is going to consume them for the rest of their lives; they understand that Mom and Dad aren't going to take care of them forever...but what do most of them REALLY know about working for a living?How can they make an informed decision about what sort of work to throw their energies into before they've really had a chance to feel what it's like to work?If you work for a couple of years, and actually experience the realities of trying to make a living and support yourself, you might:1) stumble onto something you can become passionate about2) discover what you definitely DON'T want to spend the rest of your life doing and be able to go on to college with the resolve and determination to do what it takes to avoid that sort of lifeAlso, working for a few years before college might:A) help you save money for collegeB) make you
Let's try to figure out some of the lingo.
What is this robots.txt file?
The robots.txt file is just a very plain text file (or an ASCII file, as some like to say), with a very simple set of instructions that we give to a web robot, so the robot knows which pages we need scanned (or crawled, or spidered, or indexed - all terms refer to the same thing in this context) and which pages we would like to keep out of search engines.
What is a www robot?
A robot is a computer program that automatically reads web pages and goes through every link that it finds. The purpose of robots is to gather information. Some of the most famous robots mentioned in this article work for the search engines, indexing all the information available on the web.
The first robot was developed by MIT and launched in 1993. It was named the World Wide Web Wander and its initial purpose was of a purely scientific nature, its mission was to measure the growth of the web. The index generated from the experiment's results proved to be an awesome tool and effectively became the first search engine. Most of the stuff we consider today to be indispensable online tools was born as a side effect of some scientific experiment.
What is a search engine?
Generically, a search engine is a program that searches through a database. In the popular sense, as referred to the web, a search engine is considered to be a system that has a user search form, which can search through a repository of web pages gathered by a robot.
What are spiders and crawlers?
Spiders and crawlers are robots, only the names sound cooler in the press and within metro-geek circles.
What are the most popular robots? Is there a list?
Some of the most well known robots are Google's Googlebot, MSN's MSNBot, Ask Jeeves's Teoma, Yahoo!'s Slurp (funny). One of the most popular places to search for active robot info is the list maintained at http://www.robots.org.
Why do I need this robots.txt file anyway?
A great reason to use a robots.txt file is actually the fact that many search engines, including Google, post suggestions for the public to make use of this tool. Why is it such a big deal that Google teaches people about the robots.txt? Well, because nowadays, search engines are not a playground for scientists and geeks anymore, but large corporate enterprises. Google is one of the most secretive search engines out there. Very little is known to the public about how it operates, how it indexes, how it searches, how it creates its rankings, etc. In fact, if you do a careful search in specialized forums, or wherever else these issues are discussed, nobody really agrees on whether Google puts more emphasis on this or that element to create its rankings. And when people don't agree on things as precise as a ranking algorithm, it means two things: that Google constantly changes its methods, and that it does not make it very clear or very public. There's only one thing that I believe to be crystal clear. If they recommend that you use a robots.txt ("Make use of the robots.txt file on your web server" - Google Technical Guidelines), then do it. It might not help your ranking, but it will definitely not hurt you.
There are other reasons to use the robots.txt file. If you use your error logs to tweak and keep your site free of errors, you will notice that most errors refer to someone or something not finding the robots.txt file. All you have to do is create a basic blank page (use Notepad in Windows, or the most simple text editor in Linux or on a Mac), name it robots.txt and upload it to the root of your server (that's where your home page is).
On a different note, nowadays, all search engines look for the robots.txt file as soon as their robots arrive on your site. There are unconfirmed rumors that some robots might even 'get annoyed' and leave, if they don't find it. Not sure how true that is, but hey, why not be on the safe side?
Again, even if you don't intend to block anything or just don't want to bother with this stuff at all, having a blank robots.txt is still a good idea, as it can actually act as an invitation into your site.
Don't I want my site indexed? Why stop robots?
Some robots are well designed, professionally operated, cause no harm and provide valuable service to mankind (don't we all like to "google"). Some robots are written by amateurs (remem
I’d just finished a speech with group of sales managers at a Fortune 500 company. After everyone cleared out of the room, my client came up to me with a huge smile on his face.“Great job Scott! I’ve been getting awesome feedback. You truly resonated with my staff!”“Cool, that’s what I like to hear,” I replied.Steve sat down in the chair in front of me. He leaned back, put his feet up and said, “You know Scott, I gotta be honest: I hire a lot of outside trainers just like yourself. And as valuable as your message of approachability is, the real reason I chose you … is because I LIKED you.”Wow. So there it was. Just like that. Because he liked me.LESSON LEARNED: people buy people first.Before your company. Before your products. Before your services.They buy YOU first.Before your ideas. Before your suggestions. Before your work.They buy YOU first.THEREFORE: you owe it to yourself to put your values before vocation. Beliefs before business. Person before profession. Individuality before industry.Here’s how. I call it The ABC’s of Leading with Your Person:A is for attitude. (Clich? but true!) Sun Tzu said, “What you believe about yourself, the world will believe about you.” So, before you sell a product, idea or service, first sell yourself on yourself. Because if you don’t like you, nobody else will.How much time do you spend each day selling yourself to yourself?B is for breathing. ...your person through every possible touch point, that is. The way you answer the phone, type emails, engage in person, or appear on paper – all of these are different channels through which you have an opportunity to communicate your person FIRST. It's like Seth
What is a search engine?
Generically, a search engine is a program that searches through a database. In the popular sense, as referred to the web, a search engine is considered to be a system that has a user search form, which can search through a repository of web pages gathered by a robot.
What are spiders and crawlers?
Spiders and crawlers are robots, only the names sound cooler in the press and within metro-geek circles.
What are the most popular robots? Is there a list?
Some of the most well known robots are Google's Googlebot, MSN's MSNBot, Ask Jeeves's Teoma, Yahoo!'s Slurp (funny). One of the most popular places to search for active robot info is the list maintained at http://www.robots.org.
Why do I need this robots.txt file anyway?
A great reason to use a robots.txt file is actually the fact that many search engines, including Google, post suggestions for the public to make use of this tool. Why is it such a big deal that Google teaches people about the robots.txt? Well, because nowadays, search engines are not a playground for scientists and geeks anymore, but large corporate enterprises. Google is one of the most secretive search engines out there. Very little is known to the public about how it operates, how it indexes, how it searches, how it creates its rankings, etc. In fact, if you do a careful search in specialized forums, or wherever else these issues are discussed, nobody really agrees on whether Google puts more emphasis on this or that element to create its rankings. And when people don't agree on things as precise as a ranking algorithm, it means two things: that Google constantly changes its methods, and that it does not make it very clear or very public. There's only one thing that I believe to be crystal clear. If they recommend that you use a robots.txt ("Make use of the robots.txt file on your web server" - Google Technical Guidelines), then do it. It might not help your ranking, but it will definitely not hurt you.
There are other reasons to use the robots.txt file. If you use your error logs to tweak and keep your site free of errors, you will notice that most errors refer to someone or something not finding the robots.txt file. All you have to do is create a basic blank page (use Notepad in Windows, or the most simple text editor in Linux or on a Mac), name it robots.txt and upload it to the root of your server (that's where your home page is).
On a different note, nowadays, all search engines look for the robots.txt file as soon as their robots arrive on your site. There are unconfirmed rumors that some robots might even 'get annoyed' and leave, if they don't find it. Not sure how true that is, but hey, why not be on the safe side?
Again, even if you don't intend to block anything or just don't want to bother with this stuff at all, having a blank robots.txt is still a good idea, as it can actually act as an invitation into your site.
Don't I want my site indexed? Why stop robots?
Some robots are well designed, professionally operated, cause no harm and provide valuable service to mankind (don't we all like to "google"). Some robots are written by amateurs (remem
"PRAISE LOUDLY, BLAME SOFTLY" - Catherine the GreatIn a Leadership position, the pressure to perform is ceaseless. Hitting your numbers. Achieving your goals. Meeting your metrics. That's what's expected quarter after quarter. But, why is the race getting more intense? Why does each day seem like a perpetual revved-up Indianapolis 500? And, why is it that the pressure you are getting from above, that you send below and place within, seems to be getting "louder, longer, meaner"? That's what Aubrey Daniels, in his performance management bible Bringing Out the Best in People, contends. We're all hearing and sending the same messages:Work harder. Work faster. Work smarter. Be more creative. Take more initiative.And, if we don't get the desired results, we tell the same people again, usually the same thing. This time a little louder, or longer, or perhaps a little meaner.Is this sounding vaguely familiar? If so, it's probably the right time for a quickie refresher on the ONE unbelievably simple thing you can do as a leader to keep your peoples' momentum and engagement soaring and ease the strain of relentless pressure.PRAISE – A Management Tool Honest, genuine praise. The stuff that NONE of us gets nearly enough of. Especially those sitting in the board room.The Power of Positive Praise: Fast Facts * The number-one reason people leave their jobs: They don't feel appreciated.* Sixty-five percent (65%) of Americans received no recognition in the workplace last year.* Bad bosses could increase the risk of stroke by 33%.* A study found that negative employees can scare off every customer they speak with -- for good.* Nine out of 10 people say they are more productive when they're around positive pe
There are other reasons to use the robots.txt file. If you use your error logs to tweak and keep your site free of errors, you will notice that most errors refer to someone or something not finding the robots.txt file. All you have to do is create a basic blank page (use Notepad in Windows, or the most simple text editor in Linux or on a Mac), name it robots.txt and upload it to the root of your server (that's where your home page is).
On a different note, nowadays, all search engines look for the robots.txt file as soon as their robots arrive on your site. There are unconfirmed rumors that some robots might even 'get annoyed' and leave, if they don't find it. Not sure how true that is, but hey, why not be on the safe side?
Again, even if you don't intend to block anything or just don't want to bother with this stuff at all, having a blank robots.txt is still a good idea, as it can actually act as an invitation into your site.
Don't I want my site indexed? Why stop robots?
Some robots are well designed, professionally operated, cause no harm and provide valuable service to mankind (don't we all like to "google"). Some robots are written by amateurs (remem
Running or owning a company is not an easy task. The pressure to perform is intense. Payrolls and profits have to be met and satisfied. People have to be hired and treated well. And an overall air of dynamic energy has to be created. For some folks the “running a business” learning curve can be quite emotional.About a year ago, a friend of mine, along with a group of other investors, purchased a sizable company and she became the CEO. The organization is national in scope and reputation, has existing clients, an experienced and trustworthy workforce and, when she bought it, had a significant cash flow.The company’s focus is in the financial consulting area. Its clients are secured through contracts running anywhere from a few months to a few years. They are paid well in exchange for making and/or saving their clients substantial sums of money.With approximately seventy-five contracts in place at the time the company was secured by my friend, she found herself overseeing a staff of about one hundred and monitoring some hefty deals. But it wasn’t long before cracks began to appear in the organization’s foundation. Unbeknownst to my friend (we’ll call her Kelly for the sake of this article), the company who previously owned her organization had not supported it with any sizable sales or marketing effort.Two months after assuming control and with a few contracts at their termination point, she began to realize that there had not been (and was not currently) a sustained sales effort to secure new contracts. The existing staff was viewed as the first line of salespeople. However, these folks were consultants and not a professional sales staff. Everyone was told to look for opportunities within the existing client base or search out other prospects. One problem…none of them knew ho
On a different note, nowadays, all search engines look for the robots.txt file as soon as their robots arrive on your site. There are unconfirmed rumors that some robots might even 'get annoyed' and leave, if they don't find it. Not sure how true that is, but hey, why not be on the safe side?
Again, even if you don't intend to block anything or just don't want to bother with this stuff at all, having a blank robots.txt is still a good idea, as it can actually act as an invitation into your site.
Don't I want my site indexed? Why stop robots?
Some robots are well designed, professionally operated, cause no harm and provide valuable service to mankind (don't we all like to "google"). Some robots are written by amateurs (remember, a robot is just a program). Poorly written robots can cause network overload, security problems, etc. The bottom line here is that robots are devised and operated by humans and are prone to the human error factor. Consequently, robots are not inherently bad, nor inherently brilliant, and need careful attention. This is another case where the robots.txt file comes in handy - robot control.
Now, I'm sure your main goal in life, as a webmaster or site owner is to get on the first page of Google. Then, why in the world would you want to block robots?
Here are some scenarios:
1. Unfinished site
You are still building your site, or portions of it, and don't want unfinished pages to appear in search engines. It is said that some search engines even penalize sites with pages that have been "under construction" for a long time.
2. Security
Always block your cgi-bin directory from robots. In most cases, cgi-bin contains applications, configuration files for those application (that might actually have sensitive information), etc. Even if you don't currently use any CGI scripts or programs, block it anyway, better safe than sorry.
3. Privacy
You might have some directories on your website where you keep stuff that you don't want the entire Galaxy to see, such as pictures of a friend who forgot to put clothes on, etc.
4. Doorway pages
Besides illicit attempts to increase rankings by blasting doorways all over the internet, doorway pages actually do have a very morally sound usage. They are similar pages, but each one is optimized for a specific search engine. In this case, you must make sure that individual robots do not have access to all of them. This is extremely important, in order to avoid being penalized for spamming a search engine with a series of extremely similar pages.
5. Bad bot, bad bot, what’cha gonna do...
You might want to exclude robots whose known purpose is to collect email addresses, or other robots whose activity does not agree with your beliefs on the world.
6. Your site gets overwhelmed
In rare situations, a robot goes through your site too fast, eating your bandwidth or slowing down your server. This is called "rapid-fire" and you'll notice it if you are reading your access log file. A medium performance server should not slow down. You may however have problems if you have a low performance site, such as one running of your personal PC or Mac, if you run poor server software, or if you have heavy scripts or huge documents. Is these cases, you'll see dropped connections, heavy slowdowns, in extremes, even a complete system crash. If this ever happens to you, read your logs, try to get the robot's IP or name, read the list of active robots and try to identify and block it.
What's in a robots.txt file anyway?
There are only two lines for each entry in a robots.txt file, the User-Agent, which has the name of the robot you want to give orders or the '*' wildcard symbol meaning 'all', and the Disallow line, which tells a robot all the places it should not touch. The two line entry can be repeated for every file or directory you don't want indexed, or for each robot you want to exclude. If you leave the Disallow line empty, this means you are not disallowing anything, in other words, you are allowing the particular robot to index your entire site. Some examples and a few scenarios should make it clear:
A. Exclude a file from Google's main robot (Googlebot):
User-Agent: Googlebot
Disallow: /private/privatefile.htm
B. Exclude a section of the site from all robots:
User-Agent: *
Disallow: /underconstruction/
Note that the directory is enclosed between two forward slashes. Although you are probably used to see URLs, links and folder references that do not end with a slash, note that a web server always needs a slash at the end. Even when you see links on websites that do not end with a slash, when that link is clicked, the web server has to do and extra step before serving the page, which is adding the slash through what we call a redirect. Always use the ending slash.
C. Allow everything (blank robots.txt):
User-Agent: *
Disallow:
Note that when a "blank robots.txt" is mentioned, it is not a completely blank file, but it contains the two lines above.
D. Do not allow any robot on your site:
User-Agent: *
Disallow: /
Note that the sin
HTTP = HTML link (for blogs, profiles,phorums):
<a href="http://www.otheradded.com/article/78142/otheradded-The-Robots-Text-File-Or-How-To-Get-Your-Site-Properly-Spidered-Crawled-Indexed-By-Bots.html">The Robots Text File Or How To Get Your Site Properly Spidered, Crawled, Indexed By Bots</a>
BB link (for phorums):
[url=http://www.otheradded.com/article/78142/otheradded-The-Robots-Text-File-Or-How-To-Get-Your-Site-Properly-Spidered-Crawled-Indexed-By-Bots.html]The Robots Text File Or How To Get Your Site Properly Spidered, Crawled, Indexed By Bots[/url]
Related Articles:
Productive Internet Marketing - 5 Steps for Internet Marketing
Bookmark it:
|