|
Showdown News Vol. 2 No. 1
|
|
= = = = = = = = = = = = = = = = = = = = = = = = = =
Having survived both the holiday rush and the year 2000 frenzy it is time once again to focus on changes, news, updates, and trends in the search engine industry. The new year is bringing in many new initiatives and efforts on the part of the search engine companies and technologies. This issue takes a look at the search engine size wars and Fast's quick jump to the top. FAST PASSES 300 MILLION Today, Fast is announcing its new search engine database of 300 million records. The new larger database is available at
alltheweb.com, and their press releases notes both the new size and the speed with which it delivers results. Before their launch they contracted with me to do a size comparison of Fast with Northern Light and AltaVista, the three
largest. Using a new set of 25 search terms, my study verified their claims. All the Web found more hits than either of the
other two on 20 of the 25 searches. Overall it found significantly more. In addition, since I was using 25 different queries, I was able to put up more details about my methodology and the specific
results. Anyone who would like to dig into the numbers and technique is welcome to take a look. Can they maintain the growth? An interesting side note is that the 300 million record database was built by visiting 700
million pages. Then, the company says, they removed spam, duplicates, and some pornography. While I don't have all the
details of the removal, it certainly built a much cleaner database than their first version that was rife with duplicates. Also, in the past, Fast would put up a new, larger database and then it would shrink as they fine-tuned their duplicate and
spam removal. Meanwhile the database would also get more out of date until the next major relaunch of the database. Fast
explains that it had not run a refresh spider in the past, whose job it would be to check links in the database and to add new
records as well as remove dead ones. Now, they will be using a refresh spider, so we should see a more accurate database
between now and the next major size increase. By April, Fast plans on offering a 400 million record database and "All the Web" by this summer. While no one can deliver an index of the entire Web, Fast is making some significant strides forward in indexing previously inaccessible Web pages. LYCOS IS BACK After several years of giving up on hope of finding much beyond portal and directory content at Lycos, this grandfather of
search engines can be considered a serious research search engine again. As one of the earliest spider-generated databases of
Web pages, Lycos has been languishing behind the newcomers as its database has stagnated or at least not grown
significantly in size. During that time it has been building its traffic with lots of portal and general interest content, from news, to personalization,
to communities, and more. Yet in its portal-building stage, its basic search functionality changed a few times and the
underlying database remained small. They swapped their own directory for the Open Directory in April of last year and many
search engine observers began pegging them as more of a directory and portal site rather than as a search engine. All that has changed with one sudden shift. Lycos Pro, their name for the advanced search, has abandoned its own database
and the detailed Boolean and proximity operators that went with it. Last week (or perhaps even earlier) Lycos Pro switched
to a Fast database. I suppose it is not too surprising given Lycos' investment in Fast Search and Transfer last month and its
use of the Fast FTP, rich media, and MP3 search databases, but it would have been nice if they had announced it. Look close at the bottom of the results when doing a Lycos advanced search of the Web. "Portions powered by Fast" it
says, and it finds exactly the same results as Fast at All the Web. The search features have changed completely and work like
Fast. If you were a searcher fond of using OFAR and the other old Lycos Pro operators, they are gone. Nested Boolean
searching is only available via the menus. In Lycos' own words "In an effort to give you the best search experience, we have redeveloped advanced search." Just don't
count on their help pages for a detailed explanation of the change. They only explain the details of the old syntax. Perhaps in
a few days we will have updated documentation. In the meantime, this raises some interesting questions for the future. At this point, the regular Lycos search does not use
Fast. Instead, it draws from the Open Directory, Direct Hit, and the old Lycos search engine database. It seems like a not
unlikely scenario for the near future would see Fast replacing the old Lycos search engine database even on the regular
Lycos search. And then there is that other Lycos-owned search engine: HotBot. Once the largest search engine and the researcher's
delight, HotBot has also been languishing of late. No longer is it the only Inktomi search engine, and it tends to find less
than other Inktomi services. With its clustered results and an apparently diminishing database size, I have found myself using
it less and less often. Since Lycos has invested in Fast, might HotBot dump Inktomi and replace it with a Fast database? Fast has set itself up to compete with Inktomi and has said that it plans on selling its databases and search interfaces to more than one customer. While Lycos Pro is the second user of the Fast Web database and All the Web was the first, they will probably not be the last. SEARCH ENGINE NEWS: Go2Net has relaunched MetaCrawler with a new interface and additional categories: music and auctions. In addition, both
MetaCrawler and Dogpile now include Google in the group of Web search engine databases that they query. Wherever HotBot is heading with its underlying database, it already has a new beta version running. It looks pretty much
like the regular version, but it does use a different source for its related terms. The suggested searches under Refine Your
Search come from E-Cyc. The regular version uses terms from LexiQuest. Intelliseek is now offering a beta version of BullsEye 2. The regular version looks like it will be available for free, but there
will still be a commercial BullsEye Pro that includes tracking abilities. Microsoft Network (MSN) search has switched back. Initially, it used an Inktomi database. Then in January 1999, AltaVista
announced that it would replace Inktomi as the MSN search database. Then in September of 1999 MSN finally made the
switch to AltaVista. Now, MSN has switched back to an Inktomi database. The advanced search has even more capabilities, some of which are not available on other Inktomi search engines. For
example, users can choose to limit to one record per domain or turn site clustering off. A new option is available for sorting
"equally relevant results" by date, depth, and title. Unfortunately, since the MSN results do not display dates or relevance
score, it is difficult to tell how well the sorting is working. So why the change back to Inktomi? It seems that MSN now views AltaVista as more of a competitor given AltaVista's
relaunch with more portal content. Dialog's entry into the Web search engine market has debuted as WebTop.com. It reputedly has a 50 million URL database
but plans to grow to 200 million in the next several months. The interface promotes its options for dragging and dropping or
copying and pasting text into the search box. WebTop supports the + and - operators but none of the traditional Boolean,
adjacency, or field searching commands available on Dialog. Instead it promises natural language processing. It does have a
few language limits and the ability to restrict by domain in its advanced search. But it does not count its results. WholeWeb.net announces plans to have a one billion page search engine by June of 2000. They have a demo up, but it is not
available for public view. One of the features in the demo not available on other search engines is that the results display
both the date of the page and the date it was indexed. SEARCH ENGINE SHOWDOWN NEWS: Lycos and Fast reviews have been updated with changes to search syntax for Lycos Pro and the change in databases
discussed above. AltaVista review updated with a link to the new URL for press releases. Infoseek review revised to include
information on their advanced search. Google review updated to reflect addition of RealNames database. The Search Chart
and the Search Engines by Features pages have also been updated to reflect all these changes. Two new books added to the Search Engine Readings page: Nora Paul's Great Scouts!: CyberGuides for Subject Searching
on the Web and T.R. Halvorson's Law of the Super Searchers. Pages added describing the Fast special supplementary study including a summary with commentary and analysis,
methodology, specific results, query list, and Why Size Matters. Many pages updated to make the new URL work. Search Engine Showdown is now available at your choice of URLs. Both
of the following now work, linking to the same content. Please let me know if you run across any broken links or with other
comments. And the URL works just fine in all lowercase as well.
= = = = = = = = = = = = = = = = = = = = = = = = = =
= = = = = = = = = = = = = = = = = = = = = = = = = = |
| A Notess.com Web Site ©1999-2023 by Greg R. Notess, all rights reserved |
Search Engine Showdown Greg's Writings Greg's Presentations |