The European Library: Integrated Access to the National Libraries of Europe
The European Library (TEL) Project [1] completed at the end of January 2004. The key aim of TEL was to investigate the feasibility of establishing a new Pan-European service which would ultimately give access to the combined resources of the national libraries of Europe [2]. The project was partly funded by the European Commission as an accompanying measure under the cultural heritage applications area of Key Action 3 of the Information Societies Technology (IST) research programme. The project ran for 36 months with the following partners: the Conference of European National Librarians (CENL), the national libraries of Finland, Germany, Italy, Netherlands, Portugal, Slovenia, Switzerland and the United Kingdom, along with ICCU, the national central cataloguing institute from Italy.
The main strands of work in the accompanying measure were:
- Investigation of the potential agreements such a service might negotiate with publishers concerning the provision of digital resources across Europe
- Establishing a consensus among the national libraries of Europe on the service requirements and the creation of a model for the development, management and funding of a service acceptable to all the libraries
- Development of metadata profiles to which national libraries supplying data to the service agreed to conform and a metadata environment which would enable the evolution of these profiles as required
- The testing of a technical environment which would combine access via some kind of portal to existing national library distributed Z39.50 targets and an XML-based central index of data collected from various national libraries' databases.
At the end of the accompanying measure the partners had established with sufficient confidence that it was feasible to develop an operational service on the basis of the work done. Development of the service, to be known as "The European Library", started in January 2004 and the first version of it is expected to be launched by late 2004 - early 2005.
The Nature of The European Library Service
The European Library service will be a portal which offers integrated access to the combined resources of the national libraries of Europe. It will offer free searching for both digital and non-digital resources and will deliver digital objects - some free, some priced. At launch there will be no central authentication mechanisms, any authentication will be carried out via services offered by the national libraries themselves. There will be limited multi-lingual features at launch - the partners are committed to multi-lingual interfaces, with links to translation services, but there will be no integrated multi-lingual search facility. A languages working group will be set up to investigate the development of multi-lingual searching. The service will grow incrementally - both in terms of collections available and additional contributing partners.
What is unique about The European Library:
- It will bring together on the desktop disparate collections and will allow for cross-collection searching
- It will present integrated results and will deliver digital objects
- It will enable types of collection-level searching which would otherwise be impossible
- It will be a major contribution to research both in making resources widely available and by making possible new connections through exploitation of a huge virtual library collection - The European Library will itself encourage new research
The European Library will be integrated at a service level with Gabriel [3], the World Wide Web service for the 43 European national libraries represented in the Conference of European National Librarians (CENL). The mission of Gabriel is to provide information about Europe's national libraries, their collections and their services in order to facilitate access to them, and to foster the development of new services based on a shared infrastructure. Given the closeness of purpose of Gabriel and TEL, it has become clear that there is synergy between Gabriel and the proposed European Library operational service.
The European Library service is aimed at informed citizens world-wide who want a powerful and simple way of finding European cultural resources. Moreover, it is expected to attract researchers as there will be a vast virtual collection of material from all disciplines. It will offer anyone with an interest a simple route to access European cultural resources.
Technical solution
The technical starting point at the beginning of the project was the heterogeneous nature of access to the data of the partner libraries: some offered access to bibliographic data via the Z39.50 protocol, some did not. Furthermore, not all collections were included in the Online Public Access Catalogues (OPACs) of the national libraries.
The first task for TEL was to find a solution for pooling the metadata of all collections and to offer solutions for integrated search.
Two testbeds were established: one for Z39.50 targets and one for the metadata not offered via Z39.50 for which a different protocol was required. For this data it was decided to use the Extensible Markup Language (XML) as the record structure and HTTP (Hypertext Transfer Protocol). The Z39.50 testbed was built up at Die Deutsche Bibliothek (DDB), while the HTTP /XML testbed was established by the Koninklijke Bibliotheek (KB) in The Hague.
Meanwhile the Open Archives Initiative (OAI) published the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). This protocol was adopted by TEL for the harvesting of metadata for the central index of those resources not available via Z39.50.
While the Z39.50 testbed already offered all functionality necessary for search and retrieve, the HTTP /XML testbed lacked the required search functionality. This was achieved by indexing the XML metadata and adding an interface for searching.
SRU Protocol
During the time the decision for a search protocol was on the TEL agenda, the ZING initiative (Z39.50 International Next Generation) developed a new search protocol called SRW (Search and Retrieve for the Web). SRW is a Web-based protocol which aims to integrate access across networked resources and to promote interoperability between distributed databases by providing a common platform. A sibling of this protocol is the SRU protocol (Search and Retrieve for the Web using Unified Resource Locators) [4] which places the search parameters within the URL. At present this is considered to be simpler to implement within Web browsers than say, the Simple Object Access Protocol (SOAP). For this reason the SRU protocol was adopted for the HTTP/XML testbed.
Another task of the technical workpackage of TEL was to explore how these different approaches could be integrated into some kind of portal. Using a portal would allow the potential user access to all information and services offered by the participating databases and servers. The workpackage produced an exhaustive list of requirements for the portal software which was sent out to nine software vendors in a "Call for Information". It was expected that this Call would help to recommend a specific portal software for the eventual TEL service. None of the responses from vendors fully matched the requirements and it became clear that significant costs and implementation effort would be involved. The project partners began to explore alternative solutions for the integration of the distributed and centralised resources.
Meanwhile the client component of the HTTP/XML testbed had been improved significantly, evolving from a simple testbed to an elaborated SRU search interface which proved to be fast and stable. The success of this development prompted the idea of integrating Z39.50 searching into the SRU client through the use of a protocol converter. SRU searches would be converted into Z39.50 searches and the associated responses converted back from Z39.50 to SRU.
In order to test this concept a prototype gateway was developed to convert between SRU and Z39.50 searches. Using an open source Python implementation of ZOOM, (The Z39.50 Object Model, another development arising from ZING), this proved a simple task and the outline concept was quickly proved. The initial prototype is now being developed into a fully working SRU/Z39.50 gateway on which the TEL service will be based. The software will be open source [5]. The gateway can be both centralised and distributed amongst the TEL participants, the central gateway being used for Z39.50 targets that are not under the control of The European Library.
Fig. 1: SRU portal for The European Library
Functionality of the SRU Portal
The aim of The European Library is to offer integrated access to the collections and catalogues of the European national libraries. This is achieved by encoding collection descriptions as metadata records within the central database. TEL participants can incorporate new collections and catalogues into The European Library by adding new records. The collection descriptions contain data which identify the mechanism for searching a collection or catalogue. Thus the user, by searching for and making specific collection descriptions can invoke new searches selectively or simultaneously across the chosen collections and catalogues. The services available through the portal are thus data-driven.
The SRU protocol generates the search result in a common format using XML encoding. To transform a collection-level description or an object record into a user display, the XML is converted by XSL style sheets to XHTML, which incorporates JavaScript functions to provide the user interface. These functions include for example the selection of collections, simultaneous searching, help and invoking a service for a particular object record. The services available include direct links to documents, access to linking services using OpenURL, translations, searches generated from the content of specific fields and other services such as document requesting.
As the user interface is driven by the use of style sheets, an alternative presentation of the interface can be provided by the use of a different style sheet. This feature will be used to accommodate the large number of native language interfaces that will need to be supported by The European Library. But the feature can also be used by TEL participants to tailor the system to their own requirements, including the addition of their own collection descriptions.
Thanks to this solution the implementation of a central portal became superfluous. Instead the TEL portal runs in the user's browser and the partners keep the control over the functionality and the collection and catalogues through the use of collection description records. As a result the solution has some interesting characteristics in terms of scalability. For example, the portal is highly distributed thus avoiding bottlenecks typically associated with a central portal.
The solution to integrate both Z39.50 and SRU searches into the portal system was reviewed and strongly recommended by an external expert.
Although TEL had never intended to develop a prototype for the TEL service, it turned out that the project finished with a prototype which serves as a basis for the implementation of the functional TEL service.
Metadata Development
To obtain satisfactory search results from a distributed retrieval in heterogeneous data, as far as possible, homogeneous metadata are necessary. To obtain this, a TEL application profile for metadata was developed during the TEL Project. Metadata to search, find, identify, describe and give access to materials and related services are required. The TEL partners as well as publishers all produce and use metadata but the formats, the terminology and the classifications into groups are quite different.
It was the task of the metadata workpackage of TEL to develop a TEL metadata model which would facilitate effective access to the collections of the libraries via The European Library. At the beginning of the project XML was chosen as the record syntax because most relevant formats can be converted to XML in a reversible way without information loss. Moreover, with regard to the semantic aspect, it was decided to use the Dublin Core Library Application Profile (DC-Lib) as the starting point. Dublin Core Simple was expected to be too limited for the purposes of TEL.
To determine which metadata might be needed, current and potential services and functionalities related to publication types had to be investigated. The defined functionalities for TEL and the DC-Lib were mapped in a matrix, missing elements were added and so the TEL application profile was developed. A TEL namespace was introduced for the TEL specific elements, refinements and encoding schemes.
The user scenario which the project group set up to determine which metadata might be needed in TEL, was as follows (with the basic functions that will be addressed in parentheses):
When a user accesses the TEL portal:
- He will see a list of collections and catalogues and may select those he would like to search (collection level services)
- Then he enters the query (search service)
- He may get help in entering the correct terms from a name authority file (authority service)
- or from a facility that translates his query in different languages (multilingual service)
- The result will be a list of short records that enables him to select the relevant ones (identification and description)
- Then he retrieves the full record (retrieval)
- and will further inspect the metadata to determine whether this is the object he wants to access (identification and description)
- Next he will move up or down in the object hierarchy or jump to related metadata records (navigation)
- or he may decide to retrieve the object (link service)
- Before that he wants to know whether he has permission or the rights to access the object or how he can obtain permission (authorisation)
- and whether he has the right equipment (hard- and software)
- Finally metadata might be exchanged with other systems or downloaded
The relations between functionality and metadata can be expressed in a matrix (see Figure 2). This matrix is subject to continuous development. It will be adopted to new technological and other developments in TEL (e. g. adding new collections with special requirements). Therefore the matrix shown here is just a snapshot and a starting point for further developments. Not all of the functionalities in the matrix will be available in the service at launch.
Figure 2: Matrix of functionalities and metadata of TEL
[HTML format] • [PDF format] • [Word format]
There is a distinction between metadata that are used by the TEL portal and metadata that are only used by the actual back-end services. The latter are not part of the TEL application profile. The TEL portal may neglect metadata that are present in metadata records but that are not part of the application profile or may indicate the presence of such metadata depending on the type of presentation that has been selected.
Collection-Level Descriptions
In the eventual TEL service collections will play an important role: TEL can be described as a "collection of collections". As the number of collections can become quite large the TEL user should be able to search for a collection in the same way as searching for objects. For this reason an application profile for collections has been introduced additionally to the one for objects. With this application profile metadata which describe collections (collection-level descriptions) can be included in a search which makes it possible not only to search in a collection but also to search collections themselves. If a collection is part of a search result, it can be transformed into a search target so that the user can start a new search in this special collection.
The Research Support Libraries Programme (RSLP) initiated a number of relevant studies with respect to collection-level descriptions and an extensive study on this subject was carried out by Michael Heaney [6]. The metadata proposed in the RSLP model [7] is used as the basis for the TEL application profile, with the namespace for the collection description element set as it is expected to be registered by the Dublin Core Metadata Initiative (DCMI).
During the development of the TEL application profile it became clear that the model should remain flexible because the TEL functionality would expand. A controlled evolution of the application profile was needed to introduce new collections and services. The TEL metadata registry was set up to allow for such controlled evolution. The registry contains all actual metadata terms and characteristics as well as information about proposed, rejected or obsolete elements. Furthermore it may contain extra characteristics that are specifically meant for use in TEL. The registry is open for inspection by anyone but submission of proposals for new elements has to be done via the national libraries only.
To maintain the TEL application profile, the collection-level descriptions and the TEL registry, and to guarantee a controlled evolution and adjustment to future developments, the establishment of a working group was considered to be sensible. This working group will keep up with external developments on metadata standards, metadata usage and protocols and will review these for The European Library.
The European Library and the Publishers
When the metadata of the collections of European national libraries are offered by an integrated search and access facility, enriched by links to local services, the question of how to handle networked electronic publications arises. From the technical point of view, networked electronic publications could be offered very easily to the end-user via the Internet.
However, it is not only copyright law that presents an obstacle in this respect. Publishers submit their publications to deposit libraries (including all national libraries) for archiving purposes without charge. With regard to conventional materials usage is normally limited to library reading rooms since deposit material is not usually available for lending. The same agreements between publishers and deposit libraries were applied in respect of networked electronic material. Most national libraries are not able to provide any e-business tools; therefore negotiations with publishers about charging for networked deposit materials supplied over the Internet are currently impossible. But The European Library's major potential still remains the provision of access to the entire networked electronic material of the partner libraries across the continent. Therefore only freely available electronic networked publications and digitised material from the national libraries will be available at the start of The European Library together with the metadata of conventional material.
Publisher Relations
The workpackage which dealt with publisher relations during the project comprised two objectives: to study the state of play in respect of the law of digital deposit in Europe on the one hand, and the handling of electronic publications in European national libraries on the other. The aim of this workpackage was to reach a common approach on negotiating legal and voluntary deposit agreements with publishers and to establish co-operative approaches to business, licensing and copyright matters.
The current situation in the European national libraries and the view of the publishers were determined by two questionnaires at the beginning of the project. The results can be viewed at the TEL Project home page [1]. To summarise, one can say that deposit laws which include networked electronic materials are few and far between in most of Europe; however in many countries the deposit laws are under revision with a view to adapting them to reflect the new developments in publishing. Most of the national libraries have made agreements with publishers on voluntary deposit of electronic networked publications and are acquiring experience in handling these materials. On the publishers' side, the European working group CENL / FEP (Federation of European Publishers) produced the CENL/FEP International Declaration on the Deposit of Electronic Publications in 2001; this is now under review again in order to take account of new developments and the experience acquired since the declaration was adopted by deposit libraries [8].
The publishers's view on The European Library is mixed. On the one hand, they see the possibility of new distribution channels by including their networked electronic publications in the national bibliographies and consequently in the catalogues of the national libraries. On the other hand, they fear that their commercial interests could be jeopardised. Provided it is possible to reach individual agreements between publishers and national libraries, especially concerning copyright matters and payment for usage beyond the library reading room, the publishers are willing to become partners of The European Library.
A lot of work has to be done before the access to networked publications of commercial publishers via The European Library can become a reality. The deposit laws have to be amended and negotiations on deposit procedures and the conditions of use have to be completed. Thereafter agreement will have to be reached on the pricing of expanded and remote access for off-site users. The introduction of e-business tools in national libraries will be an essential issue in such negotiations.
CENL Resolution
The results of the TEL Project were presented in detail at the TEL Final Conference at the National Library of Lithuania in Vilnius on 24 September 2003 and at the Annual Meeting of the CENL the following day. The TEL partners made an application to the CENL to merge the project results and the Gabriel service into a single common Web service entitled "The European Library" under the business model described below. The CENL adopted this unanimously and therefore The European Library will be established after the end of the project phase from February 2004 under the aegis of the CENL.
Outlook
In effect The European Library will become 'the public face of the CENL', steered by a board and hosted by the Koninklijke Bibliotheek (KB) in The Hague, Netherlands. The day-to-day work will be done by a small team of 3-4 persons responsible for the management, marketing, implementation and maintenance of the Web site and the portal, as well as for the technical support of the participants.
The European Library team will be established in February 2004 and start to transfer the project results into an operational service, including the parts of Gabriel of interest to the public. The project Web site as well as Gabriel will remain online until the launch of The European Library which is planned for the end of 2004.
Initially there will be two levels of participation in The European Library: basic and full participation. All CENL libraries will be basic participants at the outset, offering information about their libraries and their services via The European Library service. Libraries offering their metadata for the complete integrated search facilities of The European Library will become full participants. The original partners in the European Commission-funded project will all be full participants at the launch of the service and other national libraries will be gradually assimilated once the service is launched.
It is the aim of The European Library that all national libraries in the CENL will eventually become full partners but this will remain a medium-term objective. Other medium-term objectives are the expansion of the functionality of The European Library e. g. by integrating e-business tools and multilingual search functions.
A Handbook will be available online which describes all organisational, financial and technical terms and conditions for participating in The European Library. As one part of this Handbook a metadata handbook was compiled. It comprises the TEL metadata application profile, the TEL metadata registry and the database of the collection-level descriptions. Furthermore it specifies the requirements for the metadata to be accepted for TEL and it contains tools, input forms and links to relevant information.
References
- Homepage of the TEL Project http://www.europeanlibrary.org
- Schwens, U., Solberg. S., Woldering, B. and Dale, P. "The European Library (TEL) - The Gate to Europe's Knowledge", Cultivate Interactive, issue 5, 1 October 2001 http://www.cultivate-int.org/issue5/tel/
- Gabriel Web site: http://www.bl.uk/gabriel/
- Search and retrieve for the web using Unified Resource Locators (URL): http://www.loc.gov./z3950/agency/zing/srw/sru.html
- SRU-/ Z39.50-Gateway: http://herbie.bl.uk:9080/
- Comprehensive study on collection level descriptions by Michael Heany (Research Support Libraries Programme, RSLP): http://www.ukoln.ac.uk/metadata/rslp/model/amcc-v31.pdf
- RSLP model for collection level descriptions: http://www.ukoln.ac.uk/metadata/rslp/schema/
- CENL/FEP International Declaration on the Deposit of Electronic Publications: http://www.ddb.de/news/epubstat.htm