View from the Hill

The main reason why the eLib projects are so aware of the need to evaluate what they do is that the Follett Implementation Group in Information Technology (FIGIT), in establishing the programme, commissioned the Tavistock Institute [1]to assist projects in the preparation of self-evaluation strategies. Founded in London in 1947, the Tavistock is an independent not- for-profit organisation which combines social science research with professional practice. At around the time that it was setting up its Evaluation Development and Review Unit (EDRU) about six years ago, John Kelleher - a social science researcher who had done some European-sponsored research and development work into technology transfer - moved from County Cork to the Tavistock. "I think I was hired because I was considered to know something about technology." But technology is not what interests him. "What we find time and time again is that technology is the pretext for organisational change. That's what I find really interesting."

The EDRU was closely involved with eLib in the early months of the programme, producing guidelines for projects to follow in preparing their evaluation strategies, running workshops around the country and offering free clinics at the Institute's base in Tabernacle Street. "I would hope that by now there is no one involved in an eLib project who has been untouched by evaluation, or who will look upon evaluation in quite the same way again." Having done the work of empowerment, Kelleher and his colleagues in the EDRU have now taken what he calls a "supporting, back seat role". Programme-level evaluation is still to take place, but the Tavistock does not expect to be involved in that. "We're probably too close to the programme now to be the best choice" he remarks.

His knowledge of academic library culture is impressive, as is the eloquence with which he speaks about the distinguishing characteristics of the eLib programme. "I think eLib represents a break from older models of higher education funded programmes" he comments. "These often became quite autarchic once the grant was made. FIGIT was much more directive about what it wanted. There is a real emphasis upon the programme rather than just a basket of projects. ELib has an architecture."

Some of the features of that architecture have come in for criticism. What does he feel, for instance, about the consortium approach? "It has strengths and weaknesses. Certainly the overheads are enormous, in the costs of meetings and in the investment in time required by project managers. But ultimately the products and services which result have a greater generalisability, and are less idiosyncratic." The legacy of eLib, Kelleher believes, will be not only organisational change, but sectoral cultural change.

Perhaps there has been an assumption, not very successfully borne out in practice, that a programme like eLib will take advantage of communications technologies to overcome the project management difficulties of consortia? Video-conferencing seems an obvious example. Kelleher is wary of overstating the role of technology. "I have a colleague who believes that video-conferencing is unsatisfactory because the participants cannot smell each other, which - at a subliminal level - has a real influence on the way we interact emotionally and cognitively. He's thinking of patenting a 'scratch and sniff' device for video-conferencing suites." More seriously, the communication difficulties experienced by consortia should not be blamed upon the limits of e-mail or video-conferencing. "It's to do with cultural and organisational differences, with the fact that people are coming from organisations with different perceptions and ways of doing things."

For Kelleher, it is crucial that evaluation does not happen in a way that is detached from the main activity of projects, and he sees it linking vitally to the production of business cases and exit strategies. Many projects are producing products and services which will require investment if they are to be lifted out of the R&D phase into stable implementations. At an early stage, projects need to think about whether they are producing something genuinely likely to have a market. "Behavioural change in the community is what it's all about - and one of the most effective indicators of behavioural change is the decision to invest in something new."

But what about those projects which will fail to persuade anyone to put their hands in their pockets and pay for what they have produced? Is it not that very fear that makes project personnel so wary about evaluation? Kelleher admits that this is the difficulty. "We all hate failure. But eLib is an innovative programme. If no project failed, the only conclusion you could reach is that the programme should never have been funded in the first place. It is unfortunate" he continues "that in our culture we look upon evaluation as a punitive thing. If things are going awry, then it's time to stop and think, 'OK - let's do it differently. ' That, after all, is what FIGIT wants to read in the project annual reports. Not that everything is perfect."

He has been impressed by the "instinctive tendency to collaboration" which he has found in working with librarians, in sharp contrast to academics, whose approach is much more competitive. Could it be a survival strategy? Does he see the products of eLib hastening the end of the profession? "Not at all. What they do will change, but their role will not. And the same goes for publishers. It's an academic fantasy to say that we can cut out the publisher. I find that sort of rhetoric deeply boring." Ultimately, in Kelleher's view, the Electronic Libraries Programme is not about electronics or libraries. It's about people and behaviours. And this is reassuring.

References

[1] Tavinstock Institute Web site,
http://www.tavinstitute.org

Author Details

John Kelleher works for the Tavinstock insititute, significantly on evaluation within the eLib programme.
Email: j.kelleher@tavinstitute.org