The Dawning of DARE: A Shared Experience
The SURF Programme Digital Academic Repositories (DARE) is a joint initiative of Dutch universities to make their academic output digitally accessible. The KB (National Library of the Netherlands), the KNAW (Royal Netherlands Academy of Arts and Sciences) and the NWO (Netherlands Organisation for Scientific Research) also cooperate in this unique programme. DARE is being coordinated by the SURF Foundation [1]. The programme will run from January 2003 until December 2006.
DARE has several goals:
- Implementing the basic infrastructure by setting up and linking the repositories;
- Stimulating the development of services based on the research information made available through the infrastructure; and
- Initiating and promoting the submission to and use of scientific content from the repositories.
The DARE Programme has been given financial support by the government, through the NAP (National Action Plan electronic highway) fund of €2 million for the period 2003-2006. With this grant the Dutch government is giving a strong boost to innovation in the provision of academic information in the Netherlands.
General Approach
The DARE programme DARE has brought together all (thirteen) Dutch Universities and three major academic institutions to create a network of digital repositories of Dutch academic output. The first year of DARE focused on implementing the basic infrastructure by setting up and linking the repositories. This resulted in the creation of a Dutch network of OAI (Open Archives Initiative) data providers. A demonstrator portal called DAREnet [2] has been set up to access this national network's academic output. With the DARE management at SURF as the beating OAI heart, all universities agreed on a few basics:
- the use of relevant standards.
- the use of existing systems as tools for creating the OAI institutional repositories
- one solution does not fit all, and
- learn hands-on.
Other common agreements were:
- the repository's infrastructural requirements
- the amount of items to be harvestable from the repository,
- all items within a repository are ultimately freely accessible, and
- the number of university faculties participating in and contributing to the repository.
The above represented the breadth of the ambition of the institutions' libraries as a whole and of the tasks that awaited them.
The DARE management created a platform on which knowledge was developed and shared with all DARE key people in the universities. This approach led to rapid results instead of (the illusion) of a complete theoretical framework in which all pros and cons were weighed. The breaking down of the OAI ambition into two levels, i.e. a data level and a service one, kept issues relatively manageable. SURF stimulates and supports projects on a data level and funds projects on a service level aiming at the delivery of content from the academic community.
Data and Service Model
Figure 1 shows the data and service model that repositories use internationally. It comprises a) a basic facility (the data level) and b) services (the service level). These services are developed from the basic facility and provide added value for specific end-users.
- The data level is where the infrastructure is set up and maintained. A scientific institution establishes a repository that stores academic output from that institution and keeps it available for use (or reuse) according to uniform international standards. This includes working papers/preprints, dissertations, research reports, datasets, conference reports, teaching material, graduate essays, multimedia material, etc. including the corresponding metadata.
- The service level: basic material from the data level can be used to develop services providing added value for scientists, students, universities, funding agencies and other interested parties. The possibilities are numerous, including developing current or new services (e.g. management information or the updating of résumés; services the institutions themselves wish to offer, either individually or in association with others (e.g. subject portals); or services provided by third parties such as publishers (e.g. e-journals). The supply and development of services can take place at a local level (e.g. individual homepages), at a national level (e.g. national academic output service, e.g. organised by document type or discipline) or even at an international one (e.g. virtual communities). The most appropriate level of co-operation can be decided upon depending on the situation in question. However, it is crucial that institutions retain control of their own information.
Figure 1: Data-service model
The advantage offered by the data-services model is that it also provides a guide to making sound decisions about the level and extent of services that an institution wishes to offer and the costs involved therein. The data level offers every institution a basic facility for the reliable, structured digital storage of its own intellectual property. This is an important factor in the digitisation of formal scientific communication. The DARE Programme ensures that the data level is set up as simply as possible. As a result, the basic facility can be offered free of unnecessary extras, at the lowest possible operational cost. This can then also better guarantee its financial sustainability in the long term.
All additions to functionality (and thus work and manpower) aimed at activities above and beyond the basic facility belong to the service level. This makes it possible to ascertain what the corresponding costs are for each service, for whom such services are intended, whether they are worth the effort, and how best to finance them. The DARE Programme stimulates the development of services in such a way that good use is made of available (open) technology for improving their effectiveness and efficiency.
Individual Responsibility, Joint Action
An important aspect of the DARE programme is the combination of individual responsibilities with joint actions. Every university is responsible for its own repository, has its own motivation for its implementation and decides itself which services it wants to offer. Within the DARE community, participants work together for greatest effect, in order to complete the job, share knowledge and experience and to achieve interoperability.
There is no single prescribed standard repository solution, as long as the repository complies with the agreed formats. The rule is to do locally what can be done locally and restrict centralised activity to the bare essentials like creating a frame of reference, setting preconditions where and when necessary and working together on shared issues. The DARE Programme stimulates the development of different approaches in concrete projects. This way the participants learn through practical experience, a pragmatic way of pioneering at both data and service levels.
Preservation
It was considered an a priori requirement that public and other data of sufficient importance stored in the repositories would be automatically preserved for long-term future use. To that end the Koninklijke Bibliotheek (The Royal Library) has been involved from the very start of the project. The Royal Library has already set up a system for the long time preservation of data, which will also be used for the data in the DARE repositories. This is therefore one less task for the local repository managers, but one to be dealt with centrally by the Royal Library.
DAREnet: www.DAREnet.nl
DAREnet is a demonstration Web site providing basic information services. It has been developed to check and demonstrate the interoperability and the potential such a network of institutional repositories can offer scholarly communication. It has not been set up as a permanent national harvesting service, but merely as a demonstrator.
Implementation
DAREnet uses i-Tor [3] for harvesting the metadata of all repositories and for providing services to end-users (service level in Figure 1). Some of the DARE participants used i-Tor as a data provider for their institutional repository (the data level in Figure 1, see next section).
For end-users, the standard services include the browsing of the repositories and searching the metadata (on specific or all fields). Full-text searching of the digital objects (referenced in the metadata) is also a standard feature. For administrators with the appropriate permissions it is possible to edit the Web pages and to manage the harvesting of the various repositories.
The i-Tor tool was selected to implement DAREnet because its goal fits the goals of DARE: providing open access while retaining data at the source. It uses open standards and is an open source product, providing a Web content management system. It features the collaboration and coupling of existing data sources, including OAI repositories. It was this feature in particular which was used especially intensively in DAREnet.
Software Development
Whilst DAREnet was being implemented, i-Tor features for handling Open Archives were also under development. Although this meant occasional bugs, a distinct advantage was that missing features identified by DARE participants during their implementation became part of the I-Tor development process, resulting in a more tailor-made product for DARE. The fact that the software developers were a division of one of the DARE partners also represented another advantage.
Using i-Tor
The ability to manage the site remotely, not needing technicians, turned out to be a very useful feature of i-Tor. As a consequence, changes could be made swiftly. The DAREnet site was implemented in both Dutch and English. Multi-lingual interfaces are a matter of concern in most systems. As regards Web interface, i-Tor handles matters reasonably well by providing tools to maintain versions of pages for various languages. With regard to harvesting, however, separate harvesting has to be done for both the Dutch and English version and this unfortunately leads to unnecessary additional work.
Infrastructure
Architecture
At the start of the DARE Programme, a specification document 'Specifications for a Networked Repository for Dutch Universities' [4] served as our compass for the DARE architecture. This document describes DARE as a network of local repositories, with local policies, local repository software, linked together via OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting).
Evaluation of Technology / Software
Starting positions for the DARE partners were very different. Some institutions already had a repository in place, some had a repository but it was not OAI-compliant or which in some other way failed to satisfy requirements. Others started from scratch.
Starting from Scratch
Those who had to start from the beginning found the choice of software problematic. It was hard to find up-to-date comparisons of software. One of the universities had a lot of problems with the available software. The lack of good installation manuals and with software which had not been tested on various platforms made it impossible to implement their own repository on time. However by using the repository of another DARE partner, on a temporary basis, they were able to join the effort.
Adapt or Change
Some institutions used their existing systems and made them OAI-compliant. Another university had to make a workaround (with the help of one of the other DARE partners being their data provider) because of problems with their existing system (DigiTool). For this library, difficulties with crosswalks, XML files and Dublin Core made it necessary to switch to DSpace. Transferring data from old databases, which had been set up for other purposes, to the new repository proved to be a considerable challenge for some.
Systems Already in Place
Three universities were involved in the ARNO-project (Academic Research in the Netherlands Online). This project, also funded by the SURF Foundation, resulted in an OAI-compliant institutional document server that could function as a repository. These universities therefore had their repositories up and running, despite the problems caused by a new release. Another institution already had a lot of experience with their DSpace repository.
Five different systems are in use at present:
- ARNO is being used for five institutional repositories.
- Three of these institutions have been involved in the ARNO Project [5].
- Six universities are using DSpace. Four of them started off with other software but have changed since or are in the process of implementing DSpace.
- i-Tor is being used by three organisations, one being the institution that has developed this software. i-Tor has the advantage of being developed by a DARE partner where tailor-made changes can be made. However, because i-Tor is a new product it has insufficient documentation and is still under development.
- Two institutions use other proprietary software.
Metadata
Each element in the repository is described by a set of metadata. The original specification document for DARE states that the OAI-PMH standard should be used within the DARE Programme. This affected the standard for metadata. The OAI protocol requires that repositories offer the 15 metadata elements employed in simple Dublin Core (DC).
DARE Metadata Guidelines
The application of standards always leaves room for local choices and implementations and the OAI world is hardly an exception to this rule. The OAI standard (OAI PMH 2.0) and metadata formats (oai_dc, qdc) led to a short but intensive study of the DARE use of Dublin Core metadata, which, in turn led to a 'version 1.0' that was acceptable for the DARE community to begin with. This 'DARE use of Dublin Core metadata, version 1.0' [6] benefited a great deal from the best practice observed at UKOLN, Bath, with E-Prints. In this document, which was endorsed by all DARE partners in October 2003, DARE chose to use simple DC as the mandatory metadata set (because of OAI-PMH), and DARE-qualified DC (dare-qdc) as the optional metadata set within DARE. This was because the librarians soon concluded that simple Dublin Core was too simple and limited for DARE and its institutions' needs. At time of writing, all DARE partners use simple DC for data exchange.
The project managers have spent quite some time discussing where to draw the line between the data and service levels. The basic question here was whether to first define services and then determine the use of metadata or vice versa. Some people argued that experience of Z39.50 showed that 'garbage in' produces 'garbage out'. It is difficult to imagine all possible services beforehand and therefore to think of all of the metadata needed in advance. Hence, you either get caught up in endless discussions on services or those on metadata. DARE chose the pragmatic solution to work with simple DC and leave more elaborate metadata exchange until such time as it becomes necessary.
We also discovered that discussing metadata within an OAI context is in fact a discussion of data mapping and data exchange. Each DARE partner has developed some form of mapping from internal systems and richer metadata to the simple DC. Librarians tend to see DC as a set of cataloguing rules, which it is not, it is for data exchange. In a networked, co-operative setting like DARE, one needs some guidelines that answer the question what content should go into which DC element. So some cataloguing rules can be applied to make sure the same language is spoken. Dublin Core, as is, is not specific enough for that purpose.
Problems Encountered and Resolved
Once the DARE metadata guidelines were defined, we decided to freeze them for a period of six months. During the following months (October 2003 - March 2004) we kept track of issues, questions, problems and experiences with mapping, harvesting and using the metadata.
Some issues that arose were:
- Non-UTF-8 characters in local input: The use of UTF-8 is mandatory in OAI-PMH, but some repositories found that users were uploading special characters into their repositories. The DC Checker utility helped here;
- The use of dc:identifier (1): Should it contain the url of the object or the url of a local resolving mechanism like a jump-off page? And where should the identifier lead the service provider to in case of one record with multiple related digital objects (bit streams))?
- The use of dc:identifier (2): should we use OpenURL, the CNRI handle or DOI? We agreed not to use DOI but a persistent URL to be chosen locally;
- How do you harvest the object file itself? OAI-PMH has no provision for this;
- Should we include metadata of printed and digital objects, or purely digital ones? Digital-only was decided upon as the ultimate goal (as an intermediate step metadata referring to non-digital objects were accepted temporarily, as long as the goal was to obtain access to the digital object in the end);
- Should we use sets, and if so what should be the set contents?
- Problems with the correct use of the date stamp: i.e. with the correct use of the date stamp on the protocol level: date of upload to repository or date of mutation.
- Challenges with the dare_qdc namespace and XML schema:
- Use of resumption tokens: we agreed to use one after every 200 records.
During the period when we were building the DARE demonstrator we were obliged to create additional guidelines for metadata in respect of the following elements:
- name (full name and/or initials)
- source (to link the digital object to the printed version we used journal title, ISSN, volume, pages in dc:source when applicable)
- the jump-off page in dc:identifier
- the direct link in dc:source, etc
Within the context of this article it would be inappropriate to explain all the details behind these issues. Some were solved and some were not. The biggest remaining issue is the identifier/jump-off page issue. DARE set up an international OAI expert meeting on this subject in May 2004.
The Implementation Process
Looking back on the implementation period, the DARE community is proud of having achieved its first major milestones: A summary of the evaluation by the programme participants of all institutions involved is given below.
Success factors
DARE as a national initiative, not only made all universities aware of the necessity of an institutional repository but also made it a reality for all universities to set up and operate an institutional repository. With the commitment from all parties and the central co-ordination by SURF, an operational network of repositories now exists, demonstrated by DAREnet. Consequent upon these achievements, a number of new services can now be developed based on this Dutch repository infrastructure.
DARE has given a Dutch boost to the dissemination and use of OAI technology. A simple standard as defined within the Open Archives Initiative has led to rapid consensus on the way to go about retrieving and exchanging data. In addition, the fear that, on presentation of the programme at the end of the project, there would be little to show unless all repositories were live, served as a considerable stimulus to activity. The interoperability of information and the profiling of the different organisations have also proved to be critical success factors.
Obstacles Overcome, Obstacles to Come
Significant results have been achieved by a group of diverse people and institutions. Although the different cultures and approaches were apparent and sometimes represented something of a challenge, they never stood in the way of the common good of the project. Not all institutions started at the same time. Those who followed in a later phase had less time to set up servers, install the software and get content ready for the repository.
Copyright issues need a lot of organisation and will be a determining factor for a lot of academics whether to join the 'DARE movement'. However, this issue, as well as that of filling the repositories with content, will be the major focus for the remainder of 2004 and beyond. It is therefore important to develop a strategy to engage scientists' and managers' commitment to contributing content to the repositories. You could say that the technical implementation was the 'easiest' part; the institutional and organisational embedding will undoubtedly embody the challenge for the coming period.
The DARE Community
Without a doubt, everyone has experienced the added value of working together. This added value was felt in both practical and technical terms. Indeed it turned out that partners frequently provided each other moral support as much as inspiration. Not all institutions started implementing a repository at the same time. Those who followed in a later phase were able to profit from the experience of others. Some confessed it would not have been possible to implement their repository at such short notice without the help of DARE partners. Those with an existing repository on the other hand were inspired and stimulated to enhance the content and management of their operational repositories. The central co-ordination and sharing of knowledge certainly represented a critical contribution to the results achieved.
The community proved to be a lively one, although decision-making on technical details was sometimes perceived to be too slow. To communicate and exchange information and knowledge online, in a private and secure area, an extranet was set up (making use of CommunityZero [7]). However, this extranet needed a bit of getting used to and the software could perhaps have been more user-friendly. Nevertheless, it has certainly helped to generate a feeling of community and to the exchange of information and experience.
Programme Management
Due to sound project management, local initiatives and the central approach have been combined to create a very good working environment. Some participants admit that having been urged to have the repository up and running has provided them with a basis for the further development that they needed.
Presentations by the DARE programme manager to different stakeholders have also had a positive effect. The unbounded enthusiasm from the programme management at SURF represented very tangible support as did good and timely information and communication.
Organisation of the Different Institutions
The different institutions developed varying ways of dealing with the organisation of their institutional repositories.
Here are some examples:
One university set up a new unit that supports scientists with electronic publishing. This unit is also responsible for the IR (Institutional Repository). Another university set up a steering committee and instated project managers for the implementation process, very similar to the national DARE organisation of steering committee and project managers. These people were also responsible for communication within the university to get the DARE notion across. Commitment from a number of important users provided the impetus for the creation of a local repository at another university.
Although some universities were not quite ready for the implementation of a repository, the DARE initiative gave them a start: For some universities the lack of human resources was the main reason why they have only implemented the repository on a very small scale meaning, that they have only concentrated on the technical aspects of the IR and on making content more easily available.
Fragmentation within one of the institutions made it impossible for all institutional sub-groups to participate in the DARE Programme. One group was committed to making a mini-repository available but participants all had different ideas on how to go about this, especially concerning the issue of how to make the digital objects available. Much persuasion was needed to convince the participants. However, having seen DAREnet, they do now realise the potential of DARE and the benefits of sharing scientific information.
As for the submission of content, different organisational approaches have become apparent:
Most institutions use decentralised ways of collecting content. The national Metis research information system [8] is being used in several places for (decentralised) submission. However, such submission is often only done once a year, for the production of annual reports, and this does delay the availability of the digital object. One university library has started a project to integrate its library catalogue, documentation database and the repository in order to make the submission process more efficient and to improve on the quality of the metadata.
Content
All DARE participants started off with readily available content from one or more faculties or departments. Most universities have dissertations and research reports available. Today, the content mainly consists of text documents, but some photographs and videos are also included. Although the repositories can contain digital objects of any kind, this is yet to be the case in practice.
Lessons Learned
Organisational Diversity
An important success of the DARE Programme is combining individual responsibilities with joint actions. Each university is responsible for its own repository, has its own motivation for its introduction and decides itself which services it wants to offer. It was important for the DARE participants to know that they could hold onto their own ways of working, but also to learn of other ways of working. By using each others' knowledge and expertise, the community was able to achieve interoperability.
The DARE Programme encourages partners to try out different approaches in concrete projects. In this way the participants learned in a practical, pragmatic and pioneering manner at both data and services level.
Harvesting
As soon as a repository was harvestable, testing could begin. It is important to make sufficient time for the testing phase. This involves several cycles of adjustment and retesting. One reason for this is that with such a large number of diverse parties involved, several cycles are needed before the repositories deliver exactly what the harvester needs. In the DARE case, several problems arose due to the incorrect use, (mostly misinterpretation), of the Dublin Core standard. In other cases problems were caused by lack of specification (like which character encoding to use, e.g. OAI-PMH requires UTF-8 but one repository contained non-UTF-8 characters in its metadata). Another recommendation would be to set strict specifications with which each party has to comply, and provide examples. The repositories may revert to creating temporary databases, as was done by some of the DAREnet participants, just to be able to deliver the right format for harvesting purposes.
Every time repositories changed their output format, the time-consuming process of harvesting had to be performed again. New software releases of i-Tor (bug fixes or new features regarding harvesting) also occasionally made re-harvesting of all repositories necessary. The main bottleneck that made harvesting so time-consuming was the network connection between the DAREnet harvester and the repositories. This was a matter of trial and error and finally we succeeded in linking all repositories to DAREnet.
Linking to Digital Objects
Another issue with harvesting turned out to be local resolving mechanisms, like the 'jump-off pages' often used in Open Archives. The metadata contains a link to the digital object, but this is not always a link to the object itself but a (jump-off) page on the site of the institution. That page in turn contains a link to the digital object. The jump-off page, however, is a more stable url to link to. When, as is the case with DAREnet, the digital object is needed for the purpose of full text searching, you have a problem with linking to a jump-off page. This problem still exists and cannot be solved easily.
Technical issues
The original assumption that the required technology already existed proved to be true. Several technical problems encountered during this project point to the importance of good hardware, backups and software as well as measures regarding support and maintenance.
Metadata
OAI-PMH is not a set of cataloguing rules. It is a lingua franca for data exchange between partners who have agreed to use simple DC. However, OAI-PMH allows for data exchange based on various other formats. This can be any metadata set. OAI-PMH does not protocolise the exchange of the digital object file itself. An additional protocol is needed here as in most cases service providers also want to have the object and not just the metadata. One solution might be to use MPEG21-DIDL (as described by Herbert van de Sompel in his article in D-Lib Magazine of February 2004, 'Using MPEG-21 DIP and NISO OpenURL for the Dynamic Dissemination of Complex Digital Objects in the Los Alamos National Laboratory Digital Library' [9]. However, so far his solution has only been implemented at Los Alamos.)
OAI-PMH is simple enough to get you started. From a librarian's perspective it might be too simple. From a user's perspective (scientific author, student, professor), it might be elaborate enough. At present we have not yet decided whether we need additional qualifiers in DC. This question is still under discussion and will have to be settled based on experience with actual service development.
The unique identification of metadata, the digital object and its creator has proved to be necessary and useful. DARE is therefore working on a so-called 'digital author identification'.
Future Plans
The libraries' involvement in the DARE experience has shown a movement within traditional library issues of (online) access (i.e. metadata and infrastructure) and (digital) preservation towards actively publishing and the possibility of the multiple use of metadata and digital objects.
Furthermore, libraries and scientists have found themselves in new, more interactive roles between one another. DARE fundamentally improves open access to the university's 'hidden' or 'deep Web' in which scientific publications were at best downloadable but less easy to find. This shift in opportunity, potential and competence is only just being explored and is currently being translated into services that meet the needs of scientists as well as those of the general public.
DARE's focus has been on this process in this its second year, i.e. 2004. This also implies an emphasis on the more efficient reuse of metadata and digital objects for the whole process of its creation, access and long-term digital preservation - and using today's solutions to link the separate systems that manage these sub-processes. As far as the metadata is concerned, further development in DARE's use of metadata is needed.
Communication and marketing are also important issues for the next year(s) in order to inform academics of the potential of repositories for academic communication and publishing. The involvement of the scholarly community in the further submission of content and in service development will therefore be a major area of effort.
Directions for (Re)use
Certain scientific communities already have long-standing experience of sharing their publications in OAI environments, such as Los Alamos. One could state that in these early adopter environments the need for quick access to different versions of a paper was an impetus for the use of the OAI protocol. It served a specific need: the rapid dissemination of information and knowledge and feedback which is often only relevant for a short period of time - a process defined as 'science in the making' by Bruno Latour [10]. Latour makes a distinction between 'science in action' - the process of 'science becoming fact' and 'science as fact'.
One could stress that the new challenge is to create environments, based on the OAI protocol, to gain access to science as it has been established to be a fact. In other words: the pre-print phase and demand for quick access serves the 'science in the making' aspect, which has already manifested itself in the early OAI-adopter environments, serving mainly scientists. The task now is to give the public open access to the static science corpus we now know to be 'scientific establishment', in environments which (re)use data in OAI data providers.
Acknowledgements
Many thanks to all DARE project managers involved in the DARE Programme who have given their input to this article. Also a note of special thanks to Marlon Domingus of Leiden University and Rob Maijers of www.maijers.com.
References
- SURF Foundation http://www.DAREnet.nl
- Digital Academic Repositories (DARE) http://www.darenet.nl/
- i-Tor: Tools and Technology for Open Repositories http://www.i-tor.org/en/
- DARE Report 'Specifications for a Networked Repository for Dutch Universities': http://www.surf.nl/publicaties/index2.php?hb=1&oid=122
- ARNO has been developed by the University of Amsterdam, Tilburg University and the University of Twente, as a result of a project funded by the SURF Foundation. It aims to develop and implement university document servers to make the scientific output of participating institutions available. Read more on http://www.uba.uva.nl/arno. Like i-Tor and DSpace, ARNO is an open source programme.
- DARE report 'DARE use of Dublin Core metadata': http://www.surf.nl/en/download/DARE%20use%20of%20DC%20v.%201.0.pdf
- CommunityZero™ http://www.communityzero.com. This is a research information database system which enables universities, organisational units within universities, research institutes, research groups or individual researchers to register information about their research online and to make this information available worldwide in a multitude of ways.
- http://metis.hosting.kun.nl/metis/default.cfm?i=aboutmetis
- 'Using MPEG-21 DIP and NISO OpenURL for the Dynamic Dissemination of Complex Digital Objects in the Los Alamos National Laboratory Digital Library', J. Bekaert, L. Balakireva, P. Hochstenbach and H. van de Sompel, D-Lib Magazine, February 2004: http://www.dlib.org/dlib/february04/bekaert/02bekaert.html