Implementing Kuali OLE at SOAS Library

1.0: Background

In April 2015, SOAS Library implemented an open-source, next-generation library management system. It is the third library in the world to implement Kuali OLE and the first in Europe. During the 9-month implementation cycle, the project team faced challenges in all areas: functional challenges related to the business's unique library processes and technical challenges related to IT, infrastructure, and system development. The OLE project at SOAS Library required extensive collaboration between library teams and IT teams: more so than a project led by a vendor or a private-sector company. This article will demonstrate how innovation in the information sector requires a blend of library and IT skills. It should also help others to plan large-scale system implementations particularly of open-source systems.

1.1: Kuali OLE

OLE (Open Library Environment) is a next-generation, library management system from the Kuali Foundation, a non-profit corporation that develops open-source software for Higher Education. OLE is the foundation's first piece of library software and was implemented by two early adopters in the USA in August 2014: The University of Chicago Library (Kuali, 2014a) and Lehigh University Library (Kuali, 2014b). Along with Koha and Evergreen, OLE is one of the few open-source library management systems on the market (1). Free and open-source software (FOSS) is becoming more widely used in libraries as an alternative to expensive proprietary software provided by large corporate vendors.

FOSS generally offers greater ownership over the product, more opportunities for bespoke customisation, and crucially allows libraries to keep their money in their mission (2) by spending it on developing services rather than paying for expensive and exploitative software licenses from proprietary vendors. At the second Kuali Days UK event, John Robinson (2014), Director of SOAS Library, explained the moral imperative for libraries to protect their metadata from corporate interests and how FOSS allows freer access to metadata than a big supplier keeping it in a data centre as a 'cloud-hosted' solution. Preater (2012) points out that FOSS is already widely used in libraries since it is embedded in proprietary software: Apache Solr and Lucene are both widely used in next-gen search and discovery engines; other systems make use of PostgreSQL, Apache Tomcat, and the Apache http server. Preater goes on to argue that “while proprietary software suppliers enjoy the benefits of OSS themselves they’re not so keen on passing those freedoms onto us, libraries that buy their software and support services.”

Kuali OLE is developed with a collaborative and open approach emphasising a blend of library and IT. There is a Project Board comprised of library directors and thought leaders and a Functional Council comprised of academic librarians; on the other side is a Technical Council comprised of library systems experts and HTC Global Services, the company developing the codebase. Each institution contributing to OLE assigns staff to the Functional Council and the Technical Council to make their voice heard on both library-specific and technical issues. The councils work together, rely on each others expertise, and in so doing have designed software that is made by librarians for libraries.

1.2: SOAS

The formation of the Bloomsbury Library Management System consortium and its decision-in-principle to adopt OLE are covered in detail elsewhere (3). To summarise: in 2011 staff from a number of academic libraries in the Bloomsbury area of London came together to discuss their shared need for new, next-gen library management systems. After thorough research and horizon-scanning looking at proprietary systems and open-source systems, the group made a decision-in-principle to adopt OLE as a shared service working together for hosting and support. Over time, partners fell away and SOAS Library pressed ahead.

SOAS' resilience on the OLE project is partly due to the organisation's long-standing ethical roots. SOAS (the School of Oriental and African Studies, a college of the University of London) has a tradition of innovative and radical approaches to HE informed by a strong moral core: the school has a particularly politically active student population; trade unions are very active at SOAS ensuring that workers have a say in decision-making; SOAS recently announced its intention to become the first university in London to fully divest from fossil fuels (4). SOAS Library has a higher risk-appetite than other Bloomsbury partners and was therefore more willing to pursue an innovative open-source solution.

1.2.1: FOSS at SOAS

OLE is the latest in a series of FOSS products implemented at SOAS Library. It implemented VuFind (5) as its library catalogue and discovery interface in summer 2014. The library is also in the process of adopting SobekCM (6) as digital repository software for digitisation projects.

Although many FOSS systems are free as in beer (Stallman, 2011) and result in cost-savings, this was not the case for OLE. Implementing OLE rather than a vendor system may result in long-term savings but this was not the main impetus for SOAS: “... a choice between an open-source or vendor-based, proprietary solution is not about saving money. As consortium partners we made a decision early-on that our project is designed to obtain better outcomes for the same levels of expenditure, through a shared-service approach and an open mind about the new possibilities which modern technologies offer. We are also clear that a significant capital investment will be required to realise our vision.” (Robinson, 2012).

SOAS has fewer library systems staff than the previous two OLE implementers and so contracted HTC Global Services – a Kuali Commercial Affiliate and member of the OLE Project Board – to provide development, support, and integration services (7). The internal project team started with a Project Manager who pushed the project through the politics of the University of London and took it from a concept to a reality. A Business Analyst and Analyst Programmer were also recruited alongside additional staff resource from other IT teams, library staff, and a Project Board of library and IT managers.

“Too often, IT departments and information services are separated and even in competition for money, staff or responsibility over systems.” (Schneider and Barron, 2015, p. 55). The challenges that the project team faced - functional challenges and technical challenges - highlight the extent to which this project required IT and information services to work together blending strong technical skills with strong library skills. The OLE project at SOAS was a project driven by librarian-IT hybrids (Barron, 2013).

2.0: Functional challenges

Functional challenges are those related to the ‘business’ side of the implementation: related to the workings of the library and the university. With the exception of the Analyst Programmer, the project team did not have a background in librarianship. Team members had previously worked on systems implementations in the private sector and faced steep learning curves in understanding library working practices and library-specific jargon, metadata, and protocols.

2.1: Millennium

OLE at SOAS is a replacement for the legacy library management system, Millennium. Millennium is a Java-based proprietary library management system from Innovative Interfaces Inc. (or III). The system had been in place at SOAS since 1999 (Robinson, 2015) and many staff members had never used any other library management system.

As often happens under long-running systems, the system had come to define the library’s workflows and processes. With FOSS the system can be changed to meet local requirements but with proprietary software the library needs to change its processes to meet the demands of inflexible systems. At SOAS, circulation staff did things a certain way because the system demanded it and membership was defined a certain way because of the constraints of the system. Even the metadata structure for all records - primarily bibliographic but also item records, serial records, and patron records - had come to be defined not by the structure of MARC or the professional standards of AACR2 or RDA but by Millennium’s database structure. Over 16 years, processes and structures had become so embedded into the workflows of the library that staff no longer saw them as constraints of the system but as the only way to work.

2.1.1: New ways of working

Library systems change is an opportunity for rethinking outdated and inefficient workflows and processes. The open-source nature of OLE meant that the staff had more opportunity to refine the system - as well as their processes - to fit an ideal. In the early stages of implementation, thorough business analysis and process mapping defined and mapped the workflows at SOAS Library.

One example of how OLE led to new ways of working was cataloguing practice. Cataloguing at SOAS used Millennium’s built-in MARC editor: downloading records from a collaborative database using Millennium’s Z39.50 connector and then editing the record directly in the LMS. OLE was not designed for that. Although it has a built-in MARC editor, the software is better equipped for the practice of downloading MARC files outside the LMS, editing them in separate software like OCLC Connexion or MarcEdit, and then batch importing them into the LMS. This is a cataloguing practice more prevalent in the USA than in the UK.

OLE forced the cataloguing staff at SOAS to think of MARC records outside the confines of the LMS: as discrete files and records rather than as part of a single database held solely within Millennium. The project provided the opportunity for them to develop their professional skills, learn new software like MarcEdit, and rethink their approach to bibliographic cataloguing. They also learned the processes of batch importing and creating profile tables for data import - skills previously confined solely to the Systems Librarian. By thinking outside the confines of the pre-existing system, the staff rethought their work, approached processes in a new way, and developed themselves.

2.2: User engagement

Open-source system implementation requires more engagement from end-users than implementation of a proprietary system. Users have more control over how the software works and so engaged users can steer development to meet their unique requirements. Library staff need to be involved so that, prior to go-live, they can make sure that the system meets their operational needs.

OLE’s community-led development is guided by the engagement of library staff from all libraries in the partnership. Project groups hold regular meetings using WebEx, communicate using Skype, and attend face-to-face workshops in the USA. The development roadmap is defined in shared Google Drive documents, documentation is collaboratively written in a Confluence wiki (8), and bugs, issues, and features for development are captured in a Jira issue tracker (9). SMEs (subject matter experts) from each area of library services (cataloguing, circulation, acquisitions, etc.) also represent the library at SME meetings on WebEx. Keeping track of these meetings and ensuring the library has a consistent voice is a lot of work and requires contributions from staff across the library.

The library staff at SOAS Library had never previously been involved in a large-scale system implementation. Staff were not used to working on community-led systems projects, were not used to software testing, were unfamiliar with PRINCE2 project management methodology, and, in some cases, were unwilling or unable to stop using the legacy LMS.

2.2.1: Software development in the library

Particularly for early adopters, FOSS needs more testing and development than proprietary software. OLE is developed using an Agile approach (10) which arguably required more user testing than the traditional waterfall model for software development. After each patch, we performed regression testing and user-acceptance testing. After each enhancement, we checked the new functionality and checked that it didn’t break existing functionality. After every database load, we checked for data integrity issues and ensured that the data matched our expectations. Since we wanted wide staff input from the library, this meant encouraging a rigorous and professional software development from users with library backgrounds: staff with no prior experience of software testing or development.

Software testing is a labour-intensive process and we struggled to get buy-in from staff used to proprietary software working ‘right out of the box’. When preoccupied with the day-to-day work of operational library services, staff found it difficult to make time for rigorous testing. To overcome this, the project team ran a beta test of OLE’s ‘Describe’ module (the cataloguing module) from December 2014 to February 2015. This allowed users to get used to the system and to naturally discover bugs and issues. The beta was partially successful in that cataloguers got to the know the system before go-live and performed on-the-job testing but, due to workloads, several users did not work on the beta until the last couple of days when they rushed to finish their assigned work. A sudden spike in concurrent users led to the performance issues outlined in Section 3.2. It also meant we were less able to respond to bugs and issues because they were reported late in the development process.

2.2.2: Working with staff

For some staff, the control that FOSS provides was not worth the inconvenience of the extra work. This kind of resistance requires firm direction from management and leaders guiding the project. Other staff had a conservative attitude to change and struggled to adjust their existing ways of working. Again this required strong leadership from the top in order to bring all levels of library staff along on the same journey.

The levels of user engagement required for large-scale, open-source implementation should not be underestimated. User expectations need to be managed and users must be engaged in the extra work that FOSS requires. Staff resistance - valid or otherwise - can make a difficult project impossible unless carefully acknowledged and addressed. “It is perfectly clear that large-scale modernist schemes of imperative coordination can, for certain purposes, be the most efficient, equitable, and satisfactory solution… Where such schemes run into trouble... is when they encounter a recalcitrant nature, the complexity of which they only poorly comprehend, or when they encounter a recalcitrant human nature, the complexity of which they also poorly comprehend.” (Scott, 2012, pp. 36-37)

3.0: Technical challenges

Technical challenges are those related to the OLE software itself, to the infrastructure of SOAS’ library systems, and to making the system operational. With a small project team, the project relied on technical expertise in the SOAS IT teams and on HTC as external contractors.

3.0.1: System structure

OLE is a Java application that runs on Apache Tomcat using web application deployment (11). The program runs as a web application with two WAR files: the OLEFS (Financial Services) application which handles all circulation and acquisitions features and the DocStore which handles the retrieval of data from the database. The DocStore uses an Apache Solr search engine and SolrMarc indexer to power its internal search and browse functions for bibliographic, holdings, and item data. All data is held in a MySQL or Oracle database. When a user performs a function in the web application, OLEFS will run a query on the database to check and update it: for example, if a user borrows an item, the application will run queries to update the item table and the loans table to update item status, current borrower, due date, checked-out date, etc. These queries are all logged in Tomcat’s catalina.out file. In contrast to proprietary systems, transactions and their effect on the database are simple and transparent.

The core of SOAS’ infrastructure are three virtual servers for the OLE application and database: dev server, UAT (user acceptance testing) server, and live server. SOAS uses CentOS 6.5 as the Linux operating system (a free-as-in-beer Red Hat Enterprise fork offering the robustness of Red Hat without the cost) and MySQL for the underlying database.

Library systems are a small network within the larger enterprise-level network. SOAS Library’s systems comprised six virtual servers for VuFind, the public-facing catalogue and discovery layer (two versions - one linked to OLE and one to Millennium - each with dev, UAT, and live); a virtual server for Sentry Isis (12); a virtual server for Confluence knowledge management and wiki software; and five desktop PCs running 3M self-service software (13). This new library systems network also required links to other university systems: online fines payment; Identity Vault for student records; Shibboleth for authentication; etc.

3.1: SIP2

The SIP2 server is a representative challenge faced in systems integration. SIP2 is the protocol for communication between LMS and 3M self-service machines (14). Neither Chicago nor Lehigh had required SIP2 functionality in OLE and so SOAS had to lead on the development of this feature. HTC wrote an open-source SIP2 server from scratch to run on the OLE server.

This feature highlights the difficulties of working internationally on a software implementation. HTC’s developers were in Chennai, India with no access to SOAS’ 3M machines in London, UK and so the project team had to test on the actual machines for every code change that HTC made. This required strong and precise communication using every tool available: email, Skype, shared online drives.

As of writing, the SIP2 server on OLE performs all necessary functions including loans, returns, and renews. However there are noticeable performance issues compared to the SIP2 server on Millennium: response times are poor and bad error-handling causes machines to hang whenever OLE generates an error. This puts more pressure on circulation staff to do manual circulation and to help users use the machines. The SIP2 code needs to be cleaned up, the circulation rules in OLE need to be refined, and, ideally, the SIP2 code should run on a dedicated separate server.

3.2: Performance issues

Every early adopter of OLE underestimated the hardware required to run the software. After Chicago went live, they were restarting the application regularly during the working day because it kept crashing. The SOAS Project Manager was aware of this and recommended higher-spec hardware than we predicted we would need. This still underestimated the hardware. While conducting training with 20+ concurrent users or at the end of the beta (Section 2.2.1) when staff scrambled to finish work, the software would run out of allocated Java memory and crash requiring a manual restart. With lots of users, response times for basic functions like loans and returns were also slow.

OLE provides analytics on server performance and application performance through JavaMelody code. Using JavaMelody and htop, we monitored performance in real-time to diagnose performance issues. We iteratively increased the hardware spec on the servers and tested performance by running CPU-intensive tasks (batch exports, bulk renewals, wildcard searches, etc.). By go-live, we had an optimum server specification that didn’t crash with concurrent users: 8-cores, 64GB RAM, and two hard-drive partitions of 32GB for boot and 128GB for the application. Crucially we also increased the RAM allocated to Java for running Tomcat: from 3GB to 16GB. With 3GB, Java’s ongoing memory would quickly fill with ongoing threads and slow to a crawl. 16GB proved to be ample memory for production use.

After go-live and as of writing, the live OLE application has not crashed. The application is automatically restarted at midnight to clear hanging threads and free up memory. In the future, SOAS plans to improves to performance further by separating the application and the database onto separate servers.

3.3: Metadata

Metadata is the core of a library management system. Since one of the main drivers of the OLE project was maintaining control of our data rather than giving it to a vendor, we had to ensure metadata was carefully handled to meet SOAS’ unique requirements.

3.3.1: Internationalisation

As the first library outside the USA to implement OLE, SOAS was the first to look at internationalisation. The software had to get rid of hard-coded US-based values and formats for dates, currency, phrases like ‘Forgive fines’, etc. and replace them with customisable parameters. This was necessary for operations at SOAS and essential for OLE to expand its market outside ot the USA. SOAS pushed for internationalisation at the Functional and Technical Councils ensuring that Kuali kept it as a priority.

3.3.2: Unicode

SOAS’ unique collections required accurate handling of non-Roman characters in metadata. SOAS has a focus on Eastern and Middle-Eastern languages which meant that OLE had to handle languages in non-Roman scripts from Arabic and Farsi to Thai and Chinese. The VuFind catalogue was already optimised for display and search in non-Roman scripts but the underlying OLE database also had to handle the full range of Unicode characters.

The University of Chicago has extensive Oriental collections and had already discovered that the MySQL database needed to be set to use the utf8mb4 character set rather than the standard utf8 set. utf8mb4 uses four bytes per character and supports more complex supplemental character than utf8. As well as this, we had to take care exporting data from Millennium and importing it to OLE to ensure that the process didn’t strip unique characters. Innovative upgraded our export tables to export in UTF8 and we performed thorough testing in OLE after every data import to ensure vernacular languages were preserved.

3.3.3: Clean-up

As mentioned in Section 2.1, Millennium had not only defined the workflows at SOAS Library but determined the metadata structure as well. Cataloguing practices changed over the years but old or legacy records were not updated for parity which meant that, taken as a whole, the bibliographic database was riddled with inconsistencies: fields other than 880’s used for vernacular scripts; 900 fields used all manner of junk data; barcodes entered into item records inconsistently; no consistent practice for notes fields in patron or item records.

The cataloguers and the project team did metadata clean-up in Millennium before import into OLE. Iterative testing on data loading into OLE refined the process to ensure that data came across correctly. Classmarks present an illustrative example of the customisation required (15). SOAS’ practice was to use the 082 field in the Marc record for classmarks where a bib record had one item. Subsequent items added to that bib had classmarks encoded in the item record (so the first item had an empty classmark field and picked its classmark up from the 082). Millennium handled this natively, assigning the 082 classmark to whichever item had been first. Every item record with an empty classmark field in Millennium had an empty classmark in OLE. HTC had to adjust their import logic to copy the 082 field from the linked bib to any item with an empty classmark field.

The metadata clean-up at SOAS was more extensive than expected. Even after go-live, errors in the database had to be fixed using MySQL update queries. Metadata clean-up was a case where technical staff communicated extensively with library staff to understand unique cataloguing practices. Library staff had to take ownership of their metadata and better understand the new system’s data structure.

4.0: Conclusion

Large-scale open-source implementation is challenging. FOSS requires a trade-off of convenience for control. Compared to proprietary software, FOSS offers more control and greater ownership: “...if you can’t open it, you don’t own it.” (Doctorow, 2010) FOSS can be opened, code can be tinkered with, it can be customised and changed. The user is free (as-in-freedom). The downside is that FOSS requires more time, more skills, and more commitment. Open-source software can be buggy, complicated, and less slick than a well-rounded commercial product. For many institutions, it is simply easier and more convenient to spend money on a proprietary system.

SOAS Library valued control and freedom over convenience. Thanks to the long-term vision of management and thanks to the work of a small project team, SOAS Library went live with Kuali OLE on schedule. It now has full control over its own data and the ability to customise both systems and workflows to optimise its service.

Open is the shape of things to come. Open-source library systems implementations require a unique combination of library skills and IT skills whether through collaboration between library teams and IT teams or in the work of specialised librarian-IT hybrids equally comfortable with library and IT (Schneider and Barron, 2015, p. 49). An international systems project requiring such a diverse range of skills is challenging. But facing challenges is the route to innovation. As the third library in the world to implement an open-source next-generation library management system, SOAS Library points the way towards the future of library systems.

5.0: Acknowledgements

Acknowledgements to the extremely dedicated past and present members of the OLE project team: Amirun Ali (Senior Analyst Programmer), Anne Clarke (Business Analyst), Lina Lakhia (Project Manager), and Claudia Mendias (Business Analyst). Thanks to Liz Maggs (Corporate Systems Manager) and John Robinson (Director of Library and Information Services) for their management and leadership. Special thanks to Sharon Penfold (Project Manager) without whose hard work and dedication SOAS could never have implemented OLE and without whom I would not have been involved with the project. This article is an expression of the opinions of Simon Barron and not those of SOAS, the University of London, the Kuali Foundation, or the OLE project team at SOAS.

6.0: Notes

(1) Kuali OLE is licensed under the Educational Community License Version 2.0 (ECL-2.0). The source code is open and freely available at https://www.kuali.org/download/ole

(2) The words of the Kuali Foundation's slogan: http://www.kuali.org/choice

(3) See the Bloomsbury Library Management System blog: http://www.blms.ac.uk

(4) See SOAS press release here: http://www.soas.ac.uk/news/newsitem101976.html

(5) VuFind is developed by Villanova University’s Falvey Memorial Library and licensed under the GNU General Public License (GPLv3). SOAS Library’s VuFind library catalogue is available here: https://library.soas.ac.uk/

(6) SobekCM is developed by University of Florida’s George A. Smathers Library and licensed under the GNU General Public License (GPLv3).

(7) Particular acknowledgement to Srini Induri and Giri Sankar of HTC Global Services for their extraordinarily dedicated work.

(8) As well as the Kuali wiki, Confluence from Atlassian was used to document everything on the BLMS project including contracts, procurement information, implementation documentation, and systems guides.

(9) Jira from Atlassian is an issue and project tracking web application used by OLE and other Kuali projects.

(10) Agile is a software development approach that emphasises coding ‘sprints’: bursts of coding and continuous development interspersed with iterative testing and documentation. See more here: https://en.wikipedia.org/wiki/Agile_software_development

(11) For more information on Apache Tomcat web application deployment, see https://tomcat.apache.org/tomcat-6.0-doc/deployer-howto.html

(12) Sentry Isis access control software from Telepen controls the library turnstiles and patron blocks.

(13) Self-service hardware and software from 3M is the main way for library users to loan, return, and renew items.

(14) More information than you require on the 3M Standard Interchange Protocol version 2.00 is available in 3M’s lengthy documentation: http://multimedia.3m.com/mws/media/355361O/sip2-protocol.pdf

(15) And internationalisation: ‘classmarks’ are referred to as ‘call numbers’ throughout OLE.

7.0: Bibliography

Barron, S., 2013. ‘Rise of the cyborgs: the growth of librarian-IT hybrids’ delivered at Umbrella 2013 in Manchester, UK, 2013-07-03 <http://www.cilip.org.uk/sites/default/files/Beyond%20Information%20Matters%20-%20Simon%20Barron.pdf>

Doctorow, C., 2010. ‘Why I won't buy an iPad (and think you shouldn't, either).’ Boing Boing, 2010-04-02 <http://boingboing.net/2010/04/02/why-i-wont-buy-an-ipad-and-think-yo.html>

Kuali, 2014a. ‘University of Chicago Launches Kuali OLE and New Catalog’ on Kuali Foundation website, 2014-08-21 <https://www.kuali.org/news/2014/08/21/university-chicago-launches-kuali-ole-and-new-catalog>

Kuali, 2014b. ‘Lehigh University Launches New Open-Source Library Environment’ on Kuali Foundation website, 2014-08-19 <http://www.kuali.org/news/2014/08/19/lehigh-university-launches-new-open-source-library-environment>

Preater, A., 2012. ‘Free and Open Source Software and distributed innovation’ on Ginformation Systems blog, 2012-12-01 <http://www.preater.com/2012/12/01/free-and-open-source-software-and-distributed-innovation/>

Robinson, J., 2012. ‘We scanned the horizon and found something interesting’ on Bloomsbury Library Management System blog, 2012-12-10 <http://www.blms.ac.uk/scanning-the-horizon/>

Robinson, J., 2014. ‘UK View On the Implementation of Kuali Products’ delivered at Kuali Days UK in London, UK, 2014-06-19 <http://lanyrd.com/2014/kduk14/sdbdxg/>

Robinson, J., 2015. ‘Library System Change’ at SOAS Library website, 2015-05-01 <https://www.soas.ac.uk/library/lib-sys-change/>

Schneider, L., and Barron, S., 2015. ‘The hybrid librarian-IT expert’ in Schopflin, K., 2015. A handbook for corporate information professionals, London: Facet Publishing, pp. 45-56.

Stallman, R., 2011. ‘What is free software?’ on GNU Operating System website, <https://www.gnu.org/philosophy/free-sw.html>

Scott, J. C., 2012. Two cheers for anarchism: six easy pieces on autonomy, dignity, and meaningful work and play. Woodstock: Princeton University press.