[an error occurred while processing this directive]

Search Engine Statistics: Freshness Showdown

Data from May 17, 2003.
by Greg R.

All search engines are pictures of the past, but which search engine has taken its picture most recently? This comparison tries to begin to answer that question and tracks how the search engines change over time.

Key points from results:

• Most have some results indexed in the last few days
• But the bulk of most of the databases is about 1 month old
• And some pages may not have been re-indexed for much longer

This freshness showdown evaluated the search engines based on 6 searches (with 10-46 total useful matches per search engine). The useful matches were hits for specific pages that Are updated daily And report that date And the reported date is also visible from the search engine results. The chart to the right shows how old the most recently refreshed pages were and the age of the most ancient page retrieved. All URLs analyzed showed the current date at the time of the test, but the search engines had all indexed older versions.

Search Engine Newest Page Found Rough Average Oldest Page Found MSN (Ink.) 1 day 4 weeks 51 days HotBot (Ink.) 1 day 4 weeks 51 days   Google 2 days 1 month 165 days   AlltheWeb 1 days 1 month 599 days* AltaVista 0 days 3 months 108 days Gigablast 45 days 7 months 381 days Teoma 41 days 2.5 months 81 days WiseNut 133 days 6 months 183 days

* On one search of the six, AlltheWeb continues to find several extremely old records (Sept. 25, 2001, Feb. 25, 2002, June 29, 2002, Dec. 24, 2002, and Jan. 2, 2003). If these were not included, AlltheWeb's oldest would only be 103 days old.

Details

The graph below shows the distribution of pages by date. The methodology of this study attempts to identify the date the page was crawled, and those are the dates shown below. The graph should give some idea of the crawling pattern of each of the search engines.

Also, due to the non-random nature of the searches used and the small number of results, the points on the graph only give a broad, general idea of the age groupings of the databases.

Since the freshest search engines are so crowded at the top, here is another graph with a close-up of the five most current databases looking at the recent hits from about a month and a half.

Past Freshness Showdowns

• Oct. 20, 2002
  Newest: Inktomi (MSN & HotBot), Google, AltaVista, AlltheWeb
  Oldest: Northern Light, WiseNut
• April 4, 2002
  Newest: Google, MSN, HotBot, AltaVista
  Oldest: WiseNut
• March 7, 2002
  Newest: Google, MSN, HotBot
  Oldest: WiseNut
• August 13, 2001
  Newest: Google
  Oldest: WiseNut, Northern Light

More Stats Showdowns

See the Search Engine Statistics page for links to other search engine comparisons.

See also my article for more about freshness of search engines:

"Freshness Issue and Complexities with Web Search Engines." ONLINE 25(6): 66-68, Nov.-Dec. 2001.