Search Engines: Many Movements in Mere Months
The Internet moves quickly, and no more so than when looking at search engines. In the last few months (or even weeks) there's been a lot of movement, with some major changes in the field. Consequently, rather than just look at one search engine in detail, or a particular subject area, I thought it might be useful to play catch-up in this issue of Ariadne; hopefully you'll be aware of all these news stories, but in case you've actually got a life and don't sit glued to your computer screen, some of them may be new to you.
Google [1], as always, has been in the news of late, for a variety of reasons. It has recently bought Pyra Labs, which owns Blogger, a web log company. There's been a lot of speculation about why exactly they've done this, and it caught a lot of people by surprise. However, with the benefit of hindsight, it's actually fairly obvious. Well, it's obvious if you know anything about blogging! If you don't, here's a brief outline. A blog or web log is basically an online diary, and most bloggers tend to do one of two things (or indeed a combination of them): they write about things that interest them, often from a personal perspective, just like a diary, or they'll observe what's going on in the world and will link to sites that interest them. My blog for example [2] tends to be the latter - I'll briefly mention news stories that are of interest to searchers and point them to a resource that goes into more detail. Consequently, if it's possible for a company, (such as Google), to get access quickly to a company that hosts blogs and index the data, they can do a number of things. Firstly, since blogs are very much 'here and now', if Google can index them quickly, (which it can), it's very easy to see new and emerging trends, news stories and so on. Secondly, since people link to other sites, this gives Google another way of working out which sites are particularly authoritative. For more details on the purchase you can visit Blogger [3] and read their FAQ on the matter [4].
On the search engine acquisition front the big news is of course that Overture has bought both AltaVista and FAST as well. There's a very good and interesting article, produced by the folks over at ResearchBuzz [5], on why Overture is doing this, and what the possible outcomes might be.
While some search engines grow and prosper, others fall away. NorthernLight [6] has finally closed its doors; there's a slight crack though, since it's still possible to search its special collection, but you can't see the full text, so it's pretty much worthless. It's a great shame - I can think back (not long ago either) to the Online Conference, and if you didn't know better, you'd think that this was the only search engine on the planet. I think the crucial mistake that they made was to close their doors and move to a subscription-based model. Once they did that people stopped talking about them, webmasters stopped trying to get a high ranking with them, (or indeed getting into their database in the first place), since they couldn't see what they were doing, which would have meant that the indexed database would have been less useful and so on.
Talking about making money (or at least trying to) Yahoo! [7] is apparently changing its layout in an attempt to get more advertising on the page. The idea is to put more adverts down the side of the screen à la Google. Personally I think the new look is dreadful; the main reason that people want to go to Yahoo is to search, and now the search option (the hierarchical directory) is hidden down in the bottom left-hand corner of the screen - on my monitor it isn't even on the first screen, I have to scroll down to see it. I can fully understand that Yahoo wants to make money, but, (to my mind at least), the whole point of Yahoo is its use as a search engine - that's why I and millions of others go there. No amount of advertising is going to help me find what I want, and I think they're in danger of losing the plot a little. There's an interesting news story from ZDNet that goes into their thinking a little bit more deeply than I'm able to here [8].
Some search engines have recently been increasing their sophistication. Alltheweb has now added full Boolean search capabilities [9] while AltaVista has increased search functionality when it comes to using their news feature [10]. They've also released an expanded version of their multimedia database which now contains over 240 million files. If you're looking for that sort of data, I certainly think it's worthwhile spending a little bit of time checking it out. There's a good story on it at URLWire[11]. Teoma is also getting in on the act, and has increased their advanced search tools, among other new features [12]. I still keep trying Teoma, and I keep hearing good things about it, but I still remain fairly unimpressed with it - I updated an article comparing search engines (available on my site ) recently [13], and it still lags behind the others. A new service called Googlert [14] has been produced by Gideon Greenspan, a PhD student at Israel's Technion. This is a way of storing searches and getting Google to run it for you on a regular basis. Obviously as time goes on you get fewer results, but it's also a useful way of seeing how much new material gets added to their database, and when their major updates take place.
I wrote about image search engines a very long time ago now [15], and was considering updating the article, but I've been beaten to the draw on it. There's a really nice article on image search engines produced by TASI [16] which is definitely worth reading.
As for new search engines, if you're interested in exploring the whole blogging phenomenon in more detail you might want to take a look at Daypop [17] which indexes 14,600 news sites and weblogs for current events and breaking news.
I think I'll stop at that point and set all the events in stone - the longer I delay doing that, the more likely they are to keep changing, and this column will never get finished! However, if you do want to keep up to date with events in the search engine field, feel free to pop in and have a look at my web log (well, since everyone else is writing them these days I thought I should as well!) to see who is buying who today! [18]
Stop Press!
Just when I thought it was safe to send my column off to Ariadne word reached me that AlltheWeb [19] has changed its interface again. I thought I should take a quick look, and I'm very impressed with what I've seen. They've increased the size of their index to try and stay within touching distance of Google, and they've made one or two cosmetic changes which are fine if you're interested in that sort of thing. However, in the area of searching they've added in the AlltheWeb URL investigator. It quickly allows you to see who owns a site, who links to it, who refers to it, sub-domains, size, what it used to look like and so on. This is going to be an extraordinarily useful tool for evaluation of sites and for lateral searching. Add to the fact that AlltheWeb has recently improved their search tools as well [20] makes this search engine an increasingly attractive proposition for the experienced searcher.
References
- Google front page
http://www.google.com - Phil Bradley's Blog front page
http://www.philb.com/blog/blogger.html - Blogger front page
http://www.blogger.com/ - Blogger and Google FAQ
http://www.blogger.com/about/blogger_google_faq.pyra - Search Engine News and My Exploding Head http://www.researchbuzz.com/news/2003/feb20feb2603.shtml#searchengine
- Divine Northern Light front page
http://www.northernlight.com - Yahoo front page
http://www.yahoo.com - Yahoo! tests new search layout
http://news.zdnet.co.uk/story/0,,t269-s2129882,00.html?rtag=zdnetukhompage - Search Engine Showdown: Full Boolean at AlltheWeb
http://www.searchengineshowdown.com/newsarchive/000629.shtml - Additional Search Options on Alta Vista News
http://www.researchbuzz.com/news/2003/jan30feb503.shtml#additionalsearch - URL wire Alta Vista Launches Largest Multimedia Search Database
http://www.urlwire.com/news/021203.html - Ask Jeeves Inc news item
http://www.irconnect.com/askjinc/pages/news.mhtml?d=35741 - Phil Bradley site: Search Engines: How to get the best out of the Internet
http://www.philb.com/compare.htm - Google Alert
http://www.googlert.com/ - Finding images on the Internet, Ariadne issue 25, September 2000
http://www.ariadne.ac.uk/issue25/search-engines/ - TASi: A Review of Search Engines February 2003
http://www.tasi.ac.uk/resources/searchengines.html - DAYPOP
http://www.daypop.com - Phil Bradley's Blog
http://www.philb.com/blog/blogger.html - alltheweb front page
http://www.alltheweb.com - alltheweb FAQs: Query Language
http://www.alltheweb.com/help/faqs/query_language.html
Author details
Phil Bradley
5 Walton Gardens
Feltham
Middlesex
Email: philb@philb.com
URL: http://www.philb.com
Phil Bradley is an Independent Internet Consultant.