The JISC User Behaviour Monitoring and Evaluation Framework

The JISC User Behaviour Monitoring and Evaluation Framework (hereafter the Framework) is an ambitious project to design and use a research framework that monitors and maps the development of user behaviour with electronic information resources in higher education. Initially approved for three years, the Framework is currently in its third and final annual cycle. The methodology and findings relating to the first and second cycles are located in the First Cycle and Second Cycle Annual Reports, available through the JISC web site. Aspects of the work conducted under the Framework have also been disseminated variously through conference papers, and professional and academic journal articles.

The concept of the Framework was sponsored by JISC’s Committee on Awareness, Liaison and Training in collaboration with the JISC Committee on Electronic Information. The Framework has three strands which are managed through the two projects: JUSTEIS and JUBILEE, as described below. In the second cycle pilot studies were also conducted under each of the strands in further education and these will be expanded to full studies with both ‘survey’ and action research components during the Third Cycle.

Framework Projects

JUSTEIS (JISC Usage Surveys: Trends in Electronic Information Service)

Strands A and C of the Framework were contracted to the Department of Information and Library Studies at the University of Wales, Aberystwyth.

Under the JUSTEIS project, Strand A is an annual survey which seeks to measure and evaluate the overall awareness, uptake, usage and usefulness of information technologies and information services in higher education in the United Kingdom. The survey was conducted by telephone interview, e-mail and paper-based questionnaire. Several hundred interviews have been conducted with students and staff drawn from institutions and subjects across the entire higher education spectrum. The interviews use a critical incident technique to investigate the types of EIS that are accessed, approaches to searching, and success in searching, and the reasons for use of the EIS. A taxonomy of EIS is used to define the types of EIS covered.

Strand C is a general survey of EIS provision, with the aim of developing profiles of current and planned service provision. The basis for this survey is a Web survey of resources access provided by individual HEI’s. This has been supplemented by telephone interviews with senior LIS managers to explore purchasing determinants and intentions.

JUBILEE (JISC User Behaviour in Information Seeking: Longitudinal Evaluation of EIS)

Strand D of the Framework has been contracted to The Information Management Research Institute, School of Information Studies at the University of Northumbria at Newcastle. The JUBILEE Project team have completed the Second Cycle of a linked programme of ongoing, longitudinal monitoring of the information behaviour, needs and opportunities for specific academic and students communities and for academics and students in general. This ongoing programme of qualitative monitoring centres around a selection of actual (HEI’s, Departments) and virtual (discipline/subject communities/networks) ‘sites’, with an annual reporting cycle. Six institutions and three disciplines are studied per cycle. The focus of monitoring changes over time. For the first cycle the disciplines were: Health Sciences, Business Studies and English. Second Cycle disciplines were History, Computing and Sociology. Research methods have included questionnaires, interviews, focus groups, electronic communication (Chat room and observatory) and feedback on case study reports.

The Development of the Framework

Work on the Framework has two components:

• The establishment of a methodology for monitoring, interpreting and responding to user behaviour
• The findings from the methodology.

As indicated in Table 1 the key focus in Cycle 1 of the Framework was to design and test the methodology. By Cycle 2 the data sets start to be sufficiently well populated to increase confidence both in the methodology and in findings. Cycle 3 will continue with population of data sets, but also focuses more strongly on integration as a basis for a holistic set of deliverables.

Table 1 : Framework Cycles

Profiling User Behaviour

The findings from the Second Cycle replicate those from the First Cycle to a significant extent. These are:

• Undergraduates use EIS mostly for academic purposes connected to assessment, and also for leisure and lifestyle reasons.
• Search engine use predominates over all other EIS, and search engines are an important arena for practising search skills.
• Research postgraduates’ pattern of use of EIS differs from that of taught postgraduates.
• Some postgraduates make more use of JISC-negotiated services and specialist EIS than undergraduates.
• Electronic journals are used by postgraduates and by academic staff (teaching and research) but use is relatively infrequent.
• Services such as Genie (for sending a text message to a mobile phone from a PC) may become increasingly popular. Many students have more than one e-mail address.
• Patterns of use of EIS vary among disciplines, except for search engine use that is predominant for all clusters. From JUSTEIS, for example the PAS (Pure and applied sciences) cluster (and to a lesser extent the CM (Clinical medicine) cluster) make noticeably more use of JISC-negotiated services than other disciplinary clusters. The PASS (Pure and applied social sciences) cluster makes comparatively more use of organisational Web sites than other clusters.
• Academic staff influence student (UG and PG) use of EIS more than LIS staff. Friends and colleagues are also influential. Own experience weighs most heavily, though, suggesting that patterns of EIS use become habitual.