A Glimpse at EEVLs' Evaluation

EEVL [1] and the other projects in the Access to Network Resources (ANR) area of the Electronic Libraries (eLib) Programme [2] aim to provide gateways to selected quality networked resources within focus subject areas. They attempt this within the relatively immature and turbulent environment of the Internet where there is no clear picture as to the ways users search for, integrate and utilise different networked resources. While a considerable body of information exists relating to engineers use of traditional information resources [3] [4] relatively little is known about their use of the Internet. Why do engineers use the Internet? How do they navigate? What types of information are they looking for? How are the plethora of new electronic options being integrated with other more established information resources? Few answers to questions such as these are apparent yet these issues are clearly important if the provision of new services such as EEVL are to be successful.

Pilot testing [5] of the EEVL service although principally concerned with interface evaluation, offered tantalising glimpses into some of these more fundamental issues. The service was introduced to six university test sites by conducting workshops for engineering academics, researchers, and postgraduate students. Initial profile questionnaires from 93 workshop attendees indicated that 95% had used the WWW with over 85% having utilised web search engines such as Lycos or Alta Vista. While opinion was divided on the usefulness of these large search services even those who did find them useful commented upon the lack of descriptions, unfocused nature of results and the time investment required to locate suitable materials. The engineers in the study indicated that they were attempting to use the web to locate a diverse spectrum of resources ranging from software to bibliographic databases. However, the most frequently reported usage related to locating full text papers, researchers conducting similar research, and product information from commercial companies.

Analysis of the EEVL database transaction logs during the pilot study revealed a 63% versus 37% spit between searching and browsing. Search technique tended to be rather unsophisticated with over 50% of search statements utilising only a single keyword. Boolean logic was specified very rarely and truncation was utilised in only 25% of cases. Indeed, the way the majority of searches were conducted raised the issue of whether best match searching would be more appropriate than the currently supported Boolean system. Although browsing the EEVL subject classification was less popular than searching it did offer a certain serendipity which was liked by a number of users. This ‘opportunism’ was particularly appreciated by less experienced users who used it to gain an overview of resources in a subject area. One commented ‘while browsing the subject categories I come across sources I’d never have searched for’.

By recording which links users followed during the first twenty five weeks of the live EEVL service a picture of the most commonly accessed sites was built up. Over 14,000 connections to sites described within the database were made during this period.

Evidence from the pilot study suggested that the majority of users (77%) utilised the resource descriptions before assessing whether to visit a site held in the database. The EEVL resource descriptions were rated positively by over 90% of users many of which felt that the descriptions saved them time and helped the location of more relevant sites. The listing of most accessed sites should therefore reflect resources that a number of users have found genuinely useful. A typically mixed group of resources are represented including electronic journals, commercial servers, software archives, conference announcing services, engineering associations, and bibliographic databases.

During the pilot stages and the first few months of the live service EEVL users have provided a substantial amount of predominantly positive feedback. One user summed up the major advantages of the service as follows; ‘searching only in a relevant subject area, selected and annotated third party descriptions, no duplicates, faster than US searches’. Criticism of the service, when it occurred, was mostly directed at the small size of the database during the period of the pilot study. However, while appreciative of the EEVL service, many end users seemed unable to point out problems or suggest improvements because they perceived themselves as relative novices. Comments such as ‘this is such a new area that I’m not sure what to suggest’ were fairly common.

This initial evaluation of the EEVL service clearly only scratches at the surface of the issues related to use and user behavior which are relevant to ANR services. The findings represent no more than one small piece in the complex jigsaw of the information seeking behavior puzzle. Some of the types of resources highlighted by engineers as important (e.g. company information, product information, and technical software) emphasize the applied character of engineering compared to other disciplines. Consideration should be given to the nature of such interdisciplinary differences in the design of ANR gateways. Future comparison between ANR services on other aspects of patterns of use and user behavior, such as searching and browsing preferences, may provide further clues to help tailor individual services.

References

1. Edinburgh Engineering Virtual Library (EEVL) Web Site
   http://www.eevl.ac.uk/
2. Electronic Libraries Programme Web Site
   http://www.ukoln.ac.uk/elib/
3. Pinelli, T.E., 1991. The information-seeking habits and practices of engineers. Science & Technology Libraries, 11 (3), 5-25
   
4. King D.W., Casto J. and Jones H., 1994. Communication by engineers: a literature review of engineers’ information needs, seeking processes, and use. Washington D.C.: Council on Library Resources.
   
5. King I. and Moffat M.O., 1996. Evaluation of the EEVL Pilot Service
   http://www.eevl.ac.uk/evaluation/report.html

Author Details