刚看到的
And then there were four
Google is one of about four search engines that matter. There are many more than four engines, but only about four have the technology to crawl much of the web on a regular basis. As of July 2003, Yahoo owned Overture, Alltheweb, AltaVista, and Inktomi, and finally dumped Google in February 2004. Everything needed to turn Yahoo into a major search engine was now under Yahoo's roof.
It is still possible that Yahoo will shoot themselves in the foot with all of this firepower -- their desire to monetize everything appears to be high on their agenda. But so far, after only a year, Yahoo has shown that their main index search results are on a par with Google's. This is true despite the fact that Yahoo has has infiltrated some pay-per-click links into the main index. One reason for Yahoo's success is that Google's main index, though free from paid results, has declined considerably since early 2003. Amazingly, there is on average only a 20 percent overlap between Yahoo's first 100 results and Google's first 100 results for the same search -- and still, Yahoo is just as good as Google. These days there is so little room at the top of the search results heap, that any combination of algorithms will produce acceptable results. The main difference now is in the depth of the crawl.
Microsoft recently developed their own engine because they found themselves squeezed between the advertising engine of Overture and the search engine Inktomi -- both of which became Yahoo property. In 2003 Microsoft began experimenting with their own crawler. Their new engine was launched in early 2005. If Microsoft puts their greed on a back burner for a few years, by doing deep crawls and presenting a clean interface, they could do to Google what they did to Netscape. There is no "secret sauce" at Google -- we now believe it was all hype from the very beginning. (To the extent that there ever was a secret sauce, the recipe is now known by countless ecommerce spammers, which makes it a liability rather than an asset.) Thousands of engineers in hundreds of companies know how to design search engines. The only real questions are whether you can commit the resources for a deep, consistent crawl of the web, and how aggressively you want to use your search engine to make money.
That gives us Google, Yahoo, and Microsoft. The last one worth watching is Teoma/AskJeeves. Their search technology is good, and they seem serious about expanding their crawl. It remains to be seen how deeply and consistently they will be able to crawl websites with thousands of pages.
Google is easily top dog. They provide about 75 percent of the external referrals for most websites. There is no point in putting up a website apart from Google. It's do or die with Google. If we're all very lucky, one of the other three will soon offer some serious competition. If we're not lucky, we will be uploading our websites to Google's servers by then, much like the bloggers do at blogger.com (which was bought by Google in 2003). It would mean the end of the web as we know it.
It is worthwhile to understand the pressures that the average, independent webmaster is under. And given that Google is so dominant, it's important to understand the pressures that are being brought to bear on Google, Inc. It does not take too much imagination to recognize that there's a struggle going on for the soul of the web, and the focal point of this struggle is Google itself.
At one level, it's a struggle for advertising revenue. The pundits look at only this level, and are unanimous that the only advertising model on the web with any sort of future is one where little ads appear after being triggered by keyword searches, or by the non-ad content of a web page. For example, a search for Google Watch may show some ads on the right side of the screen for wrist watches. While the technique doesn't work for this example, often it serves its purpose. There is only so much pixeled real estate that the average user can be expected to survey for a given search. Today up to half of each screen is dedicated to paid ads on Google, as compared to the ad-free original Google. Everyone wants a piece of this new wave in web advertising, and Google is making a lot of money.
Unfortunately, early evidence suggests that Yahoo is less interested in pure search algorithms, than in acquiring market share in a pay-for-placement and/or pay-for-inclusion revenue stream. The same may be true for Microsoft. Even Google, dazzled by the sudden income from advertising, must be wondering why they go to all that trouble and expense to crawl the noncommercial sector. Those public-sector sites, such as the org, edu and gov domains, do not provide direct income, even though the web would be unattractive without them. All the excitement over a revived online ad market, pushed by pundits hoping for another dot-com gold rush, is beginning to look like the days when AltaVista decided that portals were the Next Big Thing. That notion caused AltaVista to lose interest in improving their crawling and searching -- which is how Google succeeded in the first place.
There has been almost no interest in establishing search engines that specialize in public-sector websites. Where is the Library of Congress? Where are the millions of dollars doled out by the Ford Foundation? How about the United Nations? Why can't some enlightened European entity pick up the slack? Everyone is asleep, while the Internet is getting spammed to death.
At another level, it's a struggle over who will have the predominant influence over the massive amounts of user data that Google collects. In the past, discussions about privacy issues and the web have been about consumer protection. That continues to be of interest, but since 9/11 there is a new threat to privacy -- the federal government. Google has not shown any inclination to declare for the rights of its users across the globe, as opposed to the rights of the spies in Washington who would love to have access to Google's user data.
Much of the struggle at this new level is unarticulated. For one thing, the spies in Washington don't talk about it. Congress has given them new powers, without debating the issues. Google, Inc. itself never comments about things that matter. The struggle recognized by Google Watch has to do with the clash of real forces, but right now all we can say is that potentially this struggle could manifest itself in Google's boardroom.
The privacy struggle, which includes both the old issue of consumer protection and this new issue of government surveillance, means that the question of how Google treats the data it collects from users becomes critical. Given that Google is so central to the web, whatever attitude it takes toward privacy has massive implications for the rest of the web in general, and for other search engines in particular.
