Web Magazine for Information Professionals

Looking for the Link Between Library Usage and Student Attainment

Graham Stone, Bryony Ramsden and Dave Pattern introduce the JISC-funded Library Impact Data Project (LIDP).

In 2010, the University of Huddersfield shared results from its analysis of anonymised library usage data [1]. Data was analysed for over 700 courses over four years - 2005⁄6 — 2008⁄9; this included the number of e-resources accessed, the number of book loans and the number of accesses to the University Library. This investigation suggested a strong correlation between library usage and degree results, and also significant underuse of expensive library resources at both School and course level. At the time, it was highlighted that the correlation between library usage and grade had not yet been significance-tested and that it was not known whether the Huddersfield findings were an anomaly or the norm [2]. As a result, a number of universities approached Huddersfield in order to benchmark against the data.

In the light of the recent Comprehensive Public Spending Review and the Lord Browne”s Review of Higher Education Funding and Student Finance [3], it was thought that, if the Huddersfield experience was found to be of statistical significance across a broad range of universities, there was potential for the results to be used as a factor to enhance student attainment. In parallel, there is a continuing focus on the student experience and a desire that all students should achieve their full potential whilst studying at university. Results could also be used by libraries to target their resources more effectively where budgets are shrinking.

In September 2010, the JISC released a call through the Activity Data programme [4] and in February 2011 the University of Huddersfield along with 7 partners: University of Bradford, De Montfort University, University of Exeter, University of Lincoln, Liverpool John Moores University, University of Salford and Teesside University were awarded JISC funding to prove the hypothesis that:

“There is a statistically significant correlation across a number of universities between library activity data and student attainment”

JISC Activity Data Programme

Part of the wider Information Environment Programme 2009-11 [5], the Activity Data strand aims to experiment with the data stored by universities with the stated aim of improving the user experience or administration of university services. The JISC has funded 8 projects that will run for 6 months, all reporting at the end of July 2011.

For the purposes of the programme, activity data has been defined as follows:

The projects were asked to propose a hypothesis and to record and propose solutions to common challenges surrounding data activity by identifying tools and techniques that prove beneficial to university services.

In addition, the JISC has funded a Synthesis Project [6], which will work alongside the other projects to assist in the way data are exploited and presented. There is potential for the JISC to fund further development of some of the areas investigated in order to assist other Higher Education (HE) institutions to benefit from the findings.

Aims and Objectives

The overall goal of the Library Impact Data Project (LIDP) is to encourage greater use of library resources thus leading to an increase in students” knowledge and understanding of their subject areas, and ultimately to ensure that student attainment is improved in areas of non/low use.

However, the project is keen to acknowledge that the relationship between the two variables is not a causal relationship and there will be other factors, which influence student attainment.

The project hopes that by proving the hypothesis there will also be tangible benefits to the wider HE community. A successful conclusion to the project will help to create a greater understanding of the link between library activity data and student attainment. To this effect, the project will publish its methodology to allow other HE institutions to benchmark their data.

The proof of the hypothesis will also be fed back into the original work from studies on non/low use in order to target improvement in these areas [7], including techniques such as course profiling, targeted promotion of information resources, raising tutor awareness of the link between use of resources and attainment, and improved targeting of resources allocation.

Project Plan

As part of the planning process, the project has been separated into four work packages.

Project Reports and Outputs (Months 1-6)

In guidance issued from Andy McGregor [8], the programme manager for the Activity Data strand, all projects are required to create a number of blog posts throughout the project:

The Library Impact Data Project has chosen to do this in a series of tagged blog posts from the project blog [9]; this will then be combined to create the final report.

An early blog post looked at the hypothesis critically, emphasising that there are other factors that impact on student attainment, and that library usage alone is not an indicator of how well students will score in their assignments, and this will be reflected in our conclusions. There are also a number of other elements we will need to consider when we progress through analysis of our results. Due to the nature of some courses, students may only need a small amount of material to achieve high grades and thereforeborrowing and resource access may not necessarily have any viable link with student attainment. Recreational use of facilities, and the nature of some subject area materials being freely available outside library subscriptions (for example primary sources necessary for history assignments), may lead to increased or reduced borrowing respectively. Some courses have information skills heavily embedded into teaching, which could result in increased access to electronic resources, but may not reflect the skills of students accessing them. Distance learners and placement students may also skew results, and we may wish to consider their use in terms of electronic access rather than book borrowing. In terms of entry data, socialising and non-library facilities will need to be factored into the analysis.

The project will be presenting its findings at a number of conferences in Europe during the summer of 2011, including the 40th LIBER Conference in Barcelona, the SCONUL Conference in Cardiff and the 9th Northumbria International Conference on Performance Measurement in Libraries and Information Services in York; further information on other seminars and conferences will be provided on the project blog.

In addition, the partners will seek to raise awareness of the project within their respective intuitions, e.g. at Huddersfield the team will submit a poster to the annual Poster Prom for academic staff and researchers.

The project is working closely with the Synthesis Project and has already contributed to a series of recipes, which will enable other universities to collect and analyse their own data in order to benchmark themselves against the findings of the project.

Data Collection (Months 1-4)

The approach the project will take is to extract anonymised activity data from partners” systems and analyse the findings.

Potential partners were asked to provide activity data which would span at least one entire academic year (e.g. 2009⁄10), or ideally for multiple years if historic data were available as this would add robustness to the data. Multiple years of data would represent for LIDP the ideal option since it would be possible to analyse library usage by students in each year of their course. Data from the original study at Huddersfield suggests that library usage changes as a student progresses. This is in addition to evidence, which pointed to differences in library usage behaviour between departments; it would be interesting to know if both were replicated across other institutions. This builds on research from Kramer, which suggests certain disciplines have less need for books [10], extending this for the first time into the need for electronic resources.

For each student who graduated in a given year, the following data were required:

It should be noted that the number of times a student accessed e-resources through authentication services such as Athens, Shibboleth, EZProzy or MetaLib is a crude but common measure of actual e-resource usage.

At the bidding stage it was anticipated that there may be problems in getting enough data to make the project viable, especially given the short six-month timeframe with which the project has had to work. To this end, potential partners were asked to confirm that they could provide at least two of the three measures of usage in addition to the student grades.

Huddersfield has provided definitions on the data required and the form the data can be accepted in. At the time of writing, data are now being successfully received from the partners and will be quality-checked, collated and significance-tested at Huddersfield. However, at this early stage of the project, some partners have already run into some problems with data collection, but it is felt that there is still enough information to prove the hypothesis one way or the other. Therefore the project will be seen to have succeeded if the following measurable targets are achieved:

One of the partners has additionally submitted PC login data too; this is also being looked at by Huddersfield to see if there is a correlation between attainment and PC logins, or anticipating a lack of correlation between attainment and library entry, library entry and PC logins.

The project also hopes to make all anonymised data available for reuse under an Open Data Commons Licence [11], building on the recommendations of the MOSAIC Project [12]. The MOSAIC Project, cited as a primary influence on the Activity Data call, was a ground-breaking project that investigated the potential surrounding the exploitation of library data and in doing so, gathered intelligence about the possible use of open data for individual universities and national services.

Unanimous approval will have to be sought from partners about whether individual data will be released; otherwise the data will be further anonymised into one dataset, thereby protecting the identity of each institution. This would probably have to be the case if the hypothesis were proved for all partners as a group, but not for an individual partner. It is thought that this could be construed as potentially damaging to that university, despite the fact that the link between usage and student attainment is not a causal relationship. Ideally, if all partners show a statistically significant correlation then the data sample from each will be large enough for the data to be released by each university, thereby giving a snapshot of different types of university as well as the type of use.

One of the major difficulties for the project so far has been to ensure that legal regulations and restrictions are being adhered to. The data we intend to use for our hypothesis is sensitive on a number of levels, and while we were already making efforts to assure anonymisation, we have liaised with JISC Legal and both the University of Huddersfield”s Legal Officer and Records Manager. We need to use identifiers to ensure we match the correct usage data to the degree result, but once the data are combined the identifier will be removed, and we are excluding data from small courses to ensure there is no traceable data back to student level. Student notification of data use is publicised via a revised version of a statement designed by the Using OpenURL Activity Data Project at EDINA [13] for its collaborators; it is intended to include the statement in the fair processing notice for future research use.

Analysis of Data (Months 1-6)

All partners have agreed to hold a number of focus groups in order to collect qualitative data from students on library usage. The initial project meeting agreed a set of questions that each partner would use in addition to guidelines for holding focus groups in the absence of any individual university guidelines on ethics. It is hoped that the data collected from the focus groups will complement the quantitative data and provide a more holistic picture of how students engage with library resources; this may form the basis for future project work.

As part of the project, Huddersfield recruited a research assistant to analyse the data using appropriate statistical methods, to collate the issues arising in the focus groups and identify the themes that evolve.

In the original project bid it was hoped to analyse National Student Survey data at course level, with a view to finding a correlation between satisfaction levels, library activity data and student attainment. However, as each University has a different Faculty/School set-up, it may not be possible to release this information as part of the final report.

Evaluation and Exit Strategy (Month 6)

The Library Impact Data Project has a finite scope and goal, that of proving or disproving the hypothesis.

With this in mind, some of the potential outcomes from analysing the data that does not specifically relate to the hypothesis may have to be left for future projects to investigate. One of the outcomes of the project, to release the data on an Open Data Commons Licence will allow others to exploit the data further.

However, the hypothesis if proved or partly proved may have potentially far-reaching implications and could be taken forward in a number of ways, which will be further explored in the final report:

Conclusion

The Library Impact Data Project is now at the halfway point; all indications show the project will have enough data to be able to prove that:

“There is a statistically significant correlation across a number of universities between library activity data and student attainment”

for at least some of the elements and that data collected from the partners are similar to that already collected at Huddersfield.

Further information on the project will be available in the final report at the end of July, with the project blog and Twitter (using the #lidp hashtag) being used to post regular developments and lessons learned. Details of all members of the project team can be found on the Library Impact Data Project blog.

References

  1. White, S. and Stone, G. Maximising use of library resources at the University of Huddersfield. In: UKSG 33rd Annual Conference and Exhibition, 12-14 April 2010, Edinburgh International Conference Centre http://eprints.hud.ac.uk/7248/
  2. White, S. and Stone, G. Maximising use of library resources at the University of Huddersfield. Serials, 2010, 23 (2). pp. 83-90 http://eprints.hud.ac.uk/7811/
  3. Lord Browne”s Review of Higher Education Funding and Student Finance http://hereview.independent.gov.uk/hereview/
  4. Activity data: JISC http://www.jisc.ac.uk/whatwedo/programmes/inf11/activitydata.aspx
  5. JISC Information Environment Programme 2009-11 http://www.jisc.ac.uk/whatwedo/programmes/inf11.aspx
  6. JISC Activity Data Synthesis Project Blog http://blog.activitydata.org/
  7. Goodall, D. and Pattern, D. Academic library non/low use and undergraduate student achievement: a preliminary report of research in progress. Library Management, 2011, 32 (3) http://eprints.hud.ac.uk/7940/
  8. Evernote shared notebook http://www.evernote.com/pub/andrewmcgregor/jiscad#
  9. Library Impact Data Project Blog http://library.hud.ac.uk/blogs/projects/lidp/
  10. Kramer, L. and Kramer, M. The College Library and the Drop-Out, College and Research Libraries, 1968, 29 (4), pp.310-312
  11. Open Data Commons http://www.opendatacommons.org/
  12. Kay, D., Harrop, H.,Chad, K., van Harmelen, M., Miller, P., and Pattern, D. Making Our Scholarly Activity Information Count (MOSAIC). Project Report. (Unpublished)
    http://ie-repository.jisc.ac.uk/466/
  13. Using OpenURL Activity Data http://edina.ac.uk/projects/Using_OpenURL_Activity_data_summary.html
  14. Leeds Met STAR-Trak Project http://leedsmetstartrak.wordpress.com/

Author Details

Graham Stone

Electronic Resources Manager
University of Huddersfield

Email: g.stone@hud.ac.uk

Having worked in e-resources for the last 17 years, Graham is Library Electronic Resources Manager at Huddersfield and is responsible for the management of the Electronic Resources Team and University Repository. He has participated in a number of JISC-funded projects and is currently project manager for the Library Impact Data Project and Huddersfield Open Access Publishing Project. A member of the UKSG Committee since 2001, Graham is UKSG Secretary and a member of the Serials, Journal of Electronic Resource Librarianship and University of Huddersfield University Press editorial boards. He is editor-in-chief of E-Resources Management Handbook, an OA e-book, and co-author of the University”s 25 Research Things Web 2.0 course.

Bryony Ramsden

Research Associate
University of Huddersfield

Email: b.j.ramsden@hud.ac.uk

Bryony is Assistant Librarian in the Human and Health Sciences subject team at the University of Huddersfield. Bryony has worked at the University Library for almost 10 years in various guises, including as a research assistant on an internally funded library project investigating learning space use, and is currently working on a PhD based on the results of the data.

Dave Pattern

Library Systems Manager
University of Huddersfield

Email: d.c.pattern@hud.ac.uk

Dave Pattern is the Library Systems Manager at the University of Huddersfield, with responsibility for the continuing development of the Web services and facilities provided by the library. A Web developer with over 15 years of experience, he previously worked for a major UK library book supplier before joining Huddersfield as the lead developer on the JISC-funded INHALE and INFORMS projects. In 2004 he was appointed to his current role and has been responsible for incorporating a variety of 2.0 enhancements into the OPAC and developing in-house services, as well as setting up weblogs and wikis for the library. He is a committed ‘2.0’ advocate and was thrilled to be named as one of the ‘2009 Movers and Shakers’ by the Library Journal and is the current Information World Review “Information Professional of the Year”.

Return to top

Article Title: Looking for the Link between Library Usage and Student Attainment
Author: Graham Stone, Bryony Ramsden, Dave Pattern
Publication Date: 4 July-2011 Publication: Ariadne Issue 67
Originating URL: http://www.ariadne.ac.uk/issue67/stone-et-al/