QA Focus

Introduction to the QA Focus Post

The JISC QA (Quality Assurance) Focus post [1], which came into being in January 2002, was detailed in full in the last issue of Vine [5]; but for those unfamiliar with the post a brief introduction follows.

The new QA Focus post is promoting a Quality Assurance framework to ensure a more rigorous approach to the establishment of consistent, high quality standards for all the JISC DNER ⁵⁄₉₉ projects and their associated ‘products’. The decision by JISC to create the post was the result of the culmination of a number of significant factors. Over the past five years projects and programmes involved in the creation of learning materials have expanded rapidly across the FE and HE sectors. In particular, through the DNER and associated initiatives, a range of different digital assets and products have been produced. These include distinct types of digital assets (such as image archives, video or audio clips, and discrete learning materials), and software/applications (such as computer-aided learning packages, support systems, databases and portals). Throughout the creation of these materials there have been no established Quality Assurance procedures against which the quality of these materials can be assessed. It is anticipated that the QA Focus post will reverse this position for the JISC.

The post itself is being provided by an equal partnership of ILRT 2 and UKOLN 3 and is jointly held by Ed Bremner (ILRT) and Marieke Napier (UKOLN). A collaborative approach to working has been established but the two partners are also responsible for individual areas. Marieke covers quality assurance for Web sites; aspects include accessibility, provision of Web sites, access by non-standard browsers and other user agents, compliance with accessibility guidelines and standards documentation, metadata, re-use/repackaging of Web sites and preservation of Web sites. She is also be responsible for deployment of project deliverables in a service environment and Quality Assurance for software development. Ed Bremner covers quality assurance for digitisation; aspects include digital images, technical metadata, digital sounds, moving images and multi media resources. He is also responsible for the creation and delivery of learning and teaching packages, modules and objects. The QA Focus post is given strategic support from Brian Kelly and Karla Youngs, Project Managers in UKOLN and ILRT respectively.

Background to the QA Focus Questionnaire

One of the first QA Focus outings was to the 3rd Joint Programmes meeting for all DNER projects held at the Manchester Conference Centre in late January. At this meeting QA Focus handed out a questionnaire that aimed to both gain some preliminary understanding of what Quality Assurance procedures were currently being carried out within the DNER ⁵⁄₉₉ projects and to obtain a better insight into project’s expectations and hopes for the QA Focus post.

The initial questions given on the questionnaire were used to acquire information about the individual project, such as the name of it, the project’s goals, the materials intended to be produced and how they would be produced. General questions were also asked in order to get projects personnel to think about the procedures carried out in their projects. In total we received 22 replies, which accounts for just under half of the current projects.

The Questionnaire

Products

The products which are being created by the projects are fairly varied. They include images (both 2D and 3D graphics), sound and video material, e-texts, html pages, case studies, interfaces, teaching and learning materials, online tutorials, databases and datasets, metadata and some reusable tools. Almost all of these materials are being created in-house. However some projects are also using the help of partners in the FE and HE sectors. A few projects have used external companies for specific pieces of work such as creating CDs but only one project stated that they have used an external organisation for all of their resource creation.

Goals

A fair number of projects have had to modify their goals in some way. Most have had to scale down the number of images or resources created. Many have also have found that their project has progressed slower than they initially expected, this was mainly due to a late start date or staffing problems. As a result of this a number of projects will be running for a longer period than initially expected. Projects seemed keen for the QA Focus to recognise that there is a need for more understanding of the difficulties involved in the type of projects covered by the DNER ⁵⁄₉₉ programme and that there may possibly have been the necessity for lowering expectations at the start of some projects.

Quality Assurance Procedures

Almost all projects that replied to the questionnaire felt that they had some quality assurance procedures in place already. Many saw these procedures as being carried out through some form of internal monitoring and self assessment. A number of projects have set up internal focus groups dedicated to quality assuring resources created. Others have regular steering group meetings and management evaluation sessions that consider how their project is running. These progress reviews are found by projects to be very useful.

Projects have also been using peer review systems to assess the quality of their resources and have been asking for feedback from other DNER projects as well as from students studying the relevant subject area.

A number of projects mentioned user rating systems. These were explained to be a process of carrying out usability testing on resources to study how the users use them and to also locate any problem areas, such as bad Web site navigation. It is interesting that these particular projects see quality assurance as a system of user evaluation as opposed to internal assessment.

The area of work in which quality assurance procedures have been the most clearly identified is in image creation. For image work projects often have testing mechanisms or a form of quality control in force. Images are often internally vetted, although a number of projects explained that they had also used external consultants to come in and evaluate their projects. This type of evaluation was expensive and had usually been written into the original project plan.

It also seems that metadata is being vigorously validated by a high number of projects. Some projects explained that metadata standards were followed and records were then checked by at least 2 people.

Advisory Services Used

An assortment of advisory services were mentioned by projects including TASI, VADS, HEDS, LTSN, PADS, UKOLN, TechDis, and the CD Focus; although worryingly a number of projects claimed that they had not consulted any yet. There seems to be consensus between projects that further information is needed on the specific roles of advisory service. Many projects found that they were unsure of the exact nature of work carried out by some of the services and the overlap between them. Some projects were also unsure of how advisory services could specifically help them with their work.

Standards Documentation

Of the 22 projects who replied to the survey only one was not aware of the DNER technical standards. Although this information in itself is worrying it is possible that the answer is due to a misunderstanding of the question. All other projects stated that they have found the advice and guidelines offered useful, but were quick to point out that the standards had not been available at the start of the programme. QA Focus will be doing further investigation into this area.

QA Focus Role

As was anticipated there is some apprehension about the QA Focus role. Projects are unsure of what to expect from the position and how the QA Focus work will affect them. This conclusion is probably reasonable as the role is a dynamic one.

The main areas in which project personnel felt that the QA Focus could help them were:

1. As a sounding board, a ‘critical friend’. Projects felt that because of the unique position that the QA Focus was in, of being neither project nor service, they could be a point of formal reference that projects could turn to. They would be able to see the whole picture and provide support and understanding.
2. In demonstrating systematic approaches to processes. This could be achieved through online check lists, information papers, templates and the Quality Assurance toolkit.
3. By using knowledge about the programme and other projects to come up with benchmarking criteria, rating systems and methods (and examples) of best practice.
4. As a communication service that put projects in touch with others working on similar areas or suffering related problems. Also in informing projects about new developments.

The projects felt that although there would be significant benefits to their final resources in having an active QA Focus there were also potential problems. A number of the pros and cons of the role were dealt with in the Vine article but the main one quoted by projects was the extra time and money needed to document quality assurance procedures. This was felt to be particularly pertinent if the project had already finished. However it was noted that the experiences learnt would be beneficial for future project funding and on future programmes. Projects also felt that they would have problems articulating the quality assurance procedures they already had in place and that learning ‘quality assurance speak’ would take time.

The replies in the completed questionnaire indicated that there was also confusion on how the QA Focus role actually differs from the other main JISC services.

From Workplans

In the evaluation process carried out the quality assurance information given in the questionnaires was used alongside quality assurance information given in project’s original workplans. Many projects had given some thought to quality assurance and its role within the project from the start; though as was displayed in the questionnaires Quality Assurance processes were usually defined as summative evaluation undertaken by external organisations.

Conclusion

So what did we learn from the questionnaires? We learnt that quality assurance is already seen as an area of potential importance for projects but that at the moment how this quality assurance should be implemented is unclear to people. People seem unsure of what quality is and what quality assurance is. They almost view the two as some sort of add on that comes at the end of a project rather than a core thread running through a project.

The role of the QA Focus will be to encourage the integration of quality assurance procedures into general working practices. This will mean attempts to move away from post project evaluations carried out by external companies and instead concentrate on ongoing assurance of quality. The QA Focus does recognise that external evaluation has its role within a programme and individual projects, but feels that continual assessment and ‘QAing’ is preferable. Quality assurance, as the QA Focus sees it, examines the processes that shape a project and when good quality assurance is implemented there should be improvement in the overall way a project runs as well as the final deliverables.

QA Focus has established 4 key areas of quality assurance for implementation within a project:

• Strategic Quality Assurance - carried out before development takes place - involves establishing best methodology for your project.
• Workflow Quality Assurance - carried out as formative Quality Assurance before and during development - involves establishing and documenting a workflow and processes.
• Sign-off Quality Assurance - carried out as summative Quality Assurance once one stage of development has been carried out - involves establishing an auditing system where everything is reviewed.
• On-going Quality Assurance - carried out as summative Quality Assurance once one stage of development has been carried out - involves establishing a system to report check, fix any faults found etc.

Consideration of these key areas as the basis of quality assurance would be highly beneficial to ongoing and future projects. They were used in a recent workshop on QA for Web sites given at the institutional Web Managers Workshop at Strathclyde [4]. Further discussion of their implementation will be discussed in future QA Focus articles.

Contact Information

Ed I Bremner
ILRT
Ed.Bremner@bristol.ac.uk
Phone: 0117 - 9287109

Marieke Napier
UKOLN
M.Napier@ukoln.ac.uk
Phone: 01225 - 386354

References

1. JISC QA Focus
   URL: <http://www.jisc.ac.uk/qa_focus/>
2. ILRT, Institute for Learning and Research Technology
   URL: <http://www.ilrt.bris.ac.uk/>
3. UKOLN, a national focus of expertise in digital information management
   URL: <http://www.ukoln.ac.uk/>
   QA Focus Area of the UKOLN Web site
   URL: <http://www.ukoln.ac.uk/qa-focus/>
4. QA for Web sites
   URL: <http://www.ukoln.ac.uk/web-focus/events/workshops/webmaster-2002/materials/napier/>
5. JISC QA Focus Published in Vine, Volume 126.
   URL: <http://susanna.catchword.com/vl=1050753/cl=35/nw=1/rpsv/catchword/litc/03055728/v126n1/contp1-1.htm>

Appendix - QA Focus Questionnaire

What is the name of your project?

What is your project start and end date?

Give a concise outline of your goals and aims for the project?

What digital prodcts/materials have or will be produced by the project? (i.e. what are the deliverables?)

Have your digital products been created in-house or through an external agency? If so who?

Since the project bid have you had to change or modify any of these goals in any way? If so how.

What Quality Assurance processes does your project currently use, if any?

Are you aware of the DNER quality standards (DNER Standards document) and have these been useful in establishing standards for your project?

Which advisory services, if any, have you consulted so far?

What Quality Assurance issues do you feel are especially important to your project?

What do you expect from the QA Focus role?

Is there any particular way in which you hope the QA Focus will be able to help you?

Author Details