Are They Being Served?

Preparing to write this article, I sought inspiration in the form of a recurring question. How can we improve our knowledge of users and their use of networked information services? What should we be doing to develop our understanding in the context of emerging digital library developments? And how might we make progress?

My starting point came from a recent reading of the executive summary of the TULIP project final report. The goal of the project was jointly to test systems for the networked delivery of journals to the user’s desktop, and the use of those journals. Some 43 Elsevier and Pergamon materials science and engineering journals were provided in electronic form, and delivered to workstations. The objective of the user behaviour research was to obtain specific feedback about TULIP from end users to guide future developments for delivering journal information to the desktop. In addition, insights were sought on the user requirements for electronic services to be attractive and valuable, both from the content provider’s side, and from the infrastructure provider’s side.

The findings are very clear and unsurprising. The end user’s definition of convenience includes an intuitive and familiar interface, access to all information from one source, high processing and publishing speed, good image text quality, sufficient journal and time coverage, and linking of information. Most users considered the coverage (in terms of titles and in time) of the journals in the project to be insufficient. This required them to search additional information elsewhere, which was considered time-consuming and redundant, contributing to increased inconvenience.

One of the main conclusions was that users will only move to electronic publications when they find the content they need in sufficient quantity. Having journals in electronic form and bringing them to the desktop are necessary, but not sufficient, conditions for the scholarly user. You must deliver a certain critical mass of needed information to warrant learning a new system or accessing information in a different way.

We encountered a similar reaction here at the London School of Economics when meeting with departmental representatives who had volunteered to assist in a similar user behaviour study for a European Union funded research project, Decomate, which aims to deliver some 70 core economics journals from Elsevier to the desktop. The general concept underlying the project was well received, but considerable concern was expressed at the lack of electronic backruns and the lack of integration with catalogues and subject bibliographies, such as Econlit and The International Bibliography of Social Sciences, which are commonly searched electronic entry point tools to this journal material. Other complaints related to the speed of downloading and printing, and the timeliness (or otherwise) of information. In particular there was disappointment that the anticipated advantage of more timely availability of information through electronic versions is at present a far cry from the actuality of a phototypesetting-driven, image-based process.

These two examples were salutary reminders of the need to look much more carefully at the relationship between different electronic services, and how best to integrate them with the broader approaches to information seeking, for them actually to be of sustained use to researchers and students. This means thinking much more widely than the individual publisher level. Valuable as the UK national pilot site licence initiative is, the very idea that in the long term, electronic journals deriving solely from a single publisher is the best approach from the user perspective, is to misunderstand or to be ignorant of user needs. And the expectation that they will hop from one publisher server to another is simply unrealistic. A more productive starting point would be to ask our academic colleagues to list their most regularly consulted journals and to see how best they might be electronically clustered, interlinked, and integrated with a range of associated traditional and electronic information services.

Perhaps the eLib Superjournal project approach to user evaluation will take us further? At least here we start in a promising way with a consortium of publishers and journals clustering around discipline areas, and including at least 30% of the most frequently cited journals in each discipline grouping. You may argue about the value of citations and impact factors, but as a first proxy in a research project for usefulness to students and academics, arguably it beats the single publisher approach hands down.

Of course our knowledge of user requirements of electronic journals is bound to be rudimentary at this stage of development, but we have an opportunity to do better than we have in the past with printed journals. It has always been a source of professional embarrassment to me that serials, the most costly non-staff area of library expenditure, has received so much emotional and so little analytical attention in our libraries. Even methodologies for deselection are generally suspect. Peter Stone, formerly of the University of Sussex Library, rigorously defends its track record as a leader in use-driven and user-based decision making in collection management. But the work done at Sussex is fairly exceptional, and by and large academic libraries know little of the ways in which users seek information and integrate it into their research and teaching. The situation is made more complex by the plethora of new electronic options.

There are also new tensions. In policy terms an open Web philosophy encourages and enables access, an 'all you can eat' approach, equivalent to our traditional open access browsing collections. Detailed authentication controls, on the other hand, which would enable the collection of 'micro use' data, can seriously inhibit beneficial and widespread use of networked information services. Users' preference for anonymity, coupled with widespread and unrestrained access to networked services, pose real potential problems to the researcher.

Within JISC a very clear and successful policy for dataset provision has been established through a model service, free at the point of use, which has enabled every higher education institution in the UK to obtain unlimited access to a range of data files for a flat subscription. The success rate is easily measurable and there are inexorable increases in use as each new academic year rolls on. And so we obtain justifiable requests for increasing hardware, storage, and extensions to user licence arrangements.

But have you ever really wondered what academics and students are really doing with this information? What is its real impact? Pioneering work in this field has been done by Harry East. His 1995 report for the British Library on the development of access to database services in British universities contains, for example, a wealth of data on BIDS/ISI use. There are tantalizing interview survey quotes, frequency of use data, broad reasons for use, document follow-up activities, information on linkages to home library’s holdings, hints about the inadequacy of subject coverage, comments on the US bias, glimpses of the uses made of abstracts, and other fascinating data. All this is excellent and I believe that we should build on this work in a systematic manner, for example to examine carefully differences at discipline and sub-discipline level, and to see how use of such databases fit with the changing information seeking behaviour of students and academics.

It does of course raise questions about the development of appropriate methodologies for this kind of evaluation and the need to consider a long-term programme of 'market research', given that shifts and comparisons over time are always more valuable than any single exercise. Peter Stone argues cogently that there are good lessons to be learnt from more traditional book and collection use assessment methods, and he has persuaded me that TV and radio audience research methods could usefully be applied to JISC-funded information services as well as local database and catalogue services. These approaches might be complemented by data collected automatically from service providers (at the aggregate level) and subject and institutional survey approaches to enable a balance to be struck between the global quantitative data and the richness of personal qualitative assessments.

The specific question of non-use is a most interesting one in the context of JISC-funded information services. The overall use and traffic growth figures indicate huge success, yet we can only speculate about what percentage of the potential user population is actually reached. NISS, for example, has some 200,000 accesses per day. Assuming an average of four accesses per session, and one user session every other day, this equates to around 100,000 individual users, or 8% of the potential target population. Similar calculations can be done for dataset and network software services. These figures are suggested, not in any negative way, but to propose that it is timely to construct a more systematic framework for use and user behaviour assessments so that JISC and others can see, at a level of complexity, the impact and relative benefits of its services, including the equality of take-up across UK universities.

From this can flow a more informed assessment of inhibiting factors (technological, cultural and organisational) and of appropriate pricing structures and their potential impact on use and non-use. This would provide a baseline for exploring at the institutional level a range of complementary issues - linkages between information strategies, operational policies, and the quality of information skills training and its integration into the curriculum - all affecting the take-up of campus and national information services.

Through these musings I would like to make a plea for bringing the user of networked information services to centre stage. As Chairman of JISC’s new committee on electronic information I will give priority to the development of this more analytical and holistic approach to the evaluation of use and user behaviour. We are already doing this type of full evaluation at the JISC programme level, and we should now develop it further, with Harry East and others, for our services. There is a danger otherwise that we might be opening up an increasing range of services with precious little understanding of the frustrations they will cause users, their true economic costs, and costs to users in lost time, or thwarted ways of working. This is a shared responsibility - the effectiveness of national services is inextricably linked with the effectiveness of any local campus information policy and its management and service delivery. I hope you will wish as information professionals to engage with this new range of challenges in partnership with JISC, and as another element in the development of a better trained cadre of information specialists and information managers in higher education.