Collecting Evidence in a Web 2.0 Context
Although JISC [1] has developed a number of services, (e.g. JORUM [2], JISCmail [3]) specifically for use by the UK Higher Education (HE) sector, people within the sector are increasingly using services developed outside the sector, either in addition to - or in some cases instead of – JISC-provided services. And as well as using such services, people are also engaging in ‘mashups’ where combinations of services and content are used to provide new services or to provide added value to data already held.
UKOLN is currently working on a JISC-funded study looking at the use of Web 2.0 tools and services in the UK HE sector [4], with the aim of finding out who is using these tools and services and the purposes to which they are being put. We are also investigating why specific tools and services are used – is it ease of use, or the needs of multi-institutional collaboration, or is it the tool or service everyone else is using?
In order to do this, we need to collect information from a potentially large number of people working in a variety of roles. However, the study is small-scale, so we have limited staff effort at our disposal; the study has a short timescale, so there is limited time to collect the evidence; and finally, the study has a wide focus, so it is not easy to identify the people we need to contact.
So where to start? The traditional route would be to carry out a literature search plus some form of survey focusing on a specific group of people. Then, having collected enough data, carry out some statistical analyses on the survey data, record the findings and come up with some recommendations. While a literature search was feasible, it was quickly recognised that, in the short space of time available, it was not going to be possible to deploy any of the usual survey techniques. Our answer was to use a Web 2.0 tool – a blog – [5] to collect the evidence, relying on wide-ranging advertising of the survey and trusting that those interested enough to take part would also spread the word to colleagues.
What Are Web 2.0 Tools and Services?
Wikipedia defines Web 2.0 as [6]:
‘a second generation of Web development and Web design. … It has led to the development and evolution of web-based communities, hosted services, and Web applications. Examples include social-networking sites, video-sharing sites, wikis, blogs, mashups and folksonomies.’
Web 2.0 hosted services and Web applications enable people to share information, work collaboratively, engage in discussion, keep in contact with colleagues, and reach new audiences. An incomplete listing would include (in no particular order): GoogleDocs, Flickr, YouTube, Delicious, Facebook, MySpace, Box.net, Snapfish, Photobucket, Bebo, Slideshare, Slideboom, GoogleMaps, GoogleEarth, WikiMapia, Furl, Sumpy, Citeulike, Connotea, Stumbleupon, Digg, reddit, Newsvine, LibraryThing, Wordpress, Blogspot, Blogger and Typepad, LinkedIn, NetVibes, Pageflakes, MyYahoo, iGoogle or MicrosoftLive, Second Life – and more are being developed.
For some of the tasks, people might be using a JISC service, such as a JISCmail discussion list, the JORUM repository for teaching and learning materials [7], the Depot [8] open-access repository for e-prints, and ShareGeo [9] for sharing and re-using geo-spatial data for academic research and teaching. However, while JISC, and also institutional services may be available, that does not mean they are always the service of choice. The aim of this study is to find out why mainstream, non-academic services are often the service of choice.
The Literature Search
Even though we were not conducting a full traditional research project, it was still necessary to carry out a literature search for the usual reason that we needed to find out what other relevant (and recent) work had been carried out (or was in progress) on the use of Web 2.0 in UK HE. At the same time as providing background and context for our own work, scanning of other reports and work in progress has brought important issues to our attention and provided pointers for future directions (even though sometimes conflicting). It has also put us in contact with other researchers. An additional benefit was that several suitable candidates for case studies were identified from reviewing reports containing information about interesting HE initiatives relevant to our focus.
An initial Google search quickly brought several key reports to our attention. We are fortunate that several UKOLN colleagues have a strong interest in Web 2.0 developments, so our initial findings were supplemented by suggestions from them, although it was reassuring that there was some overlap with those we had already identified via our searching. Since then, we have continued to add further items, some on more specific areas of research. These have either been identified from further searching using, for example, Google Scholar, or are the result of serendipity, during the course of our work. A request to complete an online questionnaire has been received from one project carrying out a parallel study, although we had already identified this. We have made our list of key findings available on the blog, which also gives people the opportunity to make further suggestions via the comments facility. However no additional resources have been suggested so far.
Why Not Traditional Survey Methods?
Surveys can be useful but need careful preparation. It is important to identify first the type of information that is required and the target population from which the information will be collected. The next step is to create the questionnaire, making sure that all the data you want to collect are covered and that the form of answer required (tick boxes, rating scales, free text, etc.) is appropriate for the purposes of the investigation. The questionnaire then has to be mailed out to the target population, whether as hardcopy print (less likely these days), as emailed attachments, as text within email messages or print or email requests to visit a Web page. Then you wait to see how many respond, maybe do some reminders, and finally call it a day, hoping you end up with sufficient data to produce meaningful analyses.
The traditional questionnaire, whichever route to the target population was chosen, held disadvantages for this study. We were interested in a range of people – lecturers, researchers and students at various levels in any academic discipline; administrators, public relations and marketing departments; librarians, information professionals and IT services personnel – in any academic institution in the UK. There was no easy way to select the people to approach, given that there were so many. And the challenge was not to find out how many were using Web 2.0 but rather to approach only those who were doing so.
An Internet-based survey still requires the lengthy preparation of a questionnaire, together with the time needed to transform it into either a Web-accessible document or a Web form. While the term ‘Internet-based survey’ could be applied to a PDF or Word file that is printed off or downloaded for completion and then printing, it is more accurately applied to a Web form that is completed online and where the data are recorded in a database. It does not remove the challenge of contacting (and reminding) those you want to respond, although the data transcription element is removed. There are even Internet services (e.g. SurveyMonkey) that allow you to create a survey questionnaire and collect the responses. However, whether it is a form printed from a Web site or an online form with database, this method, as with other questionnaire techniques, is more suited to collecting quantitative data than qualitative data.
There is an interesting feature of Internet surveys, especially where they are linked from an existing Web site, and all promotion of the survey is via that Web site. In this situation, the respondents are self-selecting from the pool of anyone who happens to visit that site while the survey is running. An observation made by someone we interviewed for another study was that some people are ‘survey junkies’ and will happily complete any survey they encounter; whereas others regard surveys as a waste of time and only real incentives will get them to take part.
Time was also against us. Creating a questionnaire, deciding whom to send it to, sending it, reminding non-responders and analysing the data would all take time we did not have. Whatever we were going to do needed to be a realistic proposition in terms of the amount of data collected and be time-effective.
Collecting Qualitative Data
We wanted qualitative data not just figures, so we needed to go beyond the simple ‘which service are you using for images?’ question. Yes, we were interested in whether people were using Flickr, Photobucket or Snapfish (or something else) but, more importantly, we wanted to know why they had chosen a specific service. Perhaps someone else had recommended it to them or they had heard of it somewhere. Or maybe they knew that sort of service was available and did some searching and played around with different ones to find the one that suited their needs the best. They might have started with one service, found problems and moved to another service. Alternatively, they might have chosen a service because they knew it to be frequently used by the people they were trying to reach.
We could, therefore, have tried a telephone survey. This still requires the preparatory stages of identifying the information required and the target population, as well as constructing the questionnaire. In this type of survey the target audience would be defined early on as a list of named individuals. With a telephone survey, the questioning can be more informal and text-based, although based on a standard set of questions, and specific points can be investigated in greater detail as opportunity arises. However, despite the fact that this type of survey often focuses on a smaller target population, it is in fact time-intensive as it requires initial contact to set up interview appointments, the time required for the actual interview, and further time spent on transcribing the interview. Through past experience of this technique we realised we could hope only to interview a small number of people within the study timescale and with the person-hours available.
Our Solution
Having considered the usual methods, we were clear about the information we needed to collect. We wanted to know about people who were using Web 2.0 tools and services; why they were using them, what they saw as their advantages, and how using these tools and services supported different aspects of their work. We were also clear about the range of people we wanted to contact – quite simply, anyone working (in any role) or studying in the UK academic sector.
That’s ‘a big ask’ as they say. While discussing this, a colleague suggested that we try using a blog to collect evidence, noting that UKOLN had recently tried this method to collaborate in the compilation of some best practice guidelines in the Good APIs Project [10]. After some consideration, we decided that it had the potential to deliver what we needed within the constraints detailed above and that we would give it a try.
The first task was to investigate what was possible with the blog software (Wordpress) that UKOLN is using. A three-column layout would enable us to post information about the study in the central block, and to set up the information collection using pages linked from the left-hand column, while the right-hand column would hold links to the most recent comments.
Creating the pages to collect the information was an early task. We decided to break down the types of tasks for which people might be using Web 2.0 tools and services, and added the catch-all ‘any other task’ in case we had missed anything. An early version of these pages was shown to UKOLN colleagues; feedback generally was in favour of our breakdown, but the text was felt to be overlong. So with red pen in hand, the text was pruned to make it clearer and more concise.
At this point we were almost ready to make the blog public. However, we felt that publishing the blog with no comments at all initially might be off-putting to some potential respondents, despite providing some example comment text. We therefore asked UKOLN colleagues to add the first few comments. Once this seeding of comments was in place, we went public.
Spreading the Word
We recognised that one factor in the success or failure of this technique would be getting enough responses from the full range of people we were interested in reaching. Simply making the blog public would not lead people to it and we therefore needed an effective advertising strategy.
Our first approach was to use the UKOLN Web site, since this is well known and frequently visited by the library and information community. Accordingly a news item was posted, appearing on both the Web site and in our RSS news feed. A feature piece was also written for the home page and the link to the survey blog was given prominence on the home page.
Recognising nonetheless that this was a passive approach, we also started more active advertising. UKOLN colleagues were helpful in posting to relevant email discussion lists to which they subscribed, posting in other blogs and tweeting on Twitter; this proved useful and we saw comments being posted almost immediately. However all of the responses were coming from the library and information community, and though pleasing to see, we needed to reach other subject communities as well as institutional administration and management.
So we decided try using other JISCmail lists. Since there are many lists, our first step was to identify which lists were relevant and we decided to concentrate on (a) super-lists, (b) general lists within a specific subject area and © new lecturer lists. It was hoped that this would enable us to get our request seen by a sizeable proportion of our target population. However, putting this into practice turned out to be more difficult than anticipated.
Firstly, posting the same message directly to several lists at a time triggered the spam filtering software. Secondly, posting the request to list owners had varying results; a few posted the message on almost immediately, while others ignored it. Thirdly, investigation of the archives of some of the lists initially chosen revealed that they were moribund, with no posts recorded for a year or more. And having put a lot of effort into this, there appeared to be little benefit in terms of comments being made by communities other than the library and information community.
Why were we not getting input from the other communities? This is difficult to answer definitively, though there are some likely reasons. We made the blog public in early June, at the end of the academic year when departments would be focusing on examinations and graduations. Once those were finished with, and over the next couple of months, staff would be taking annual leave and perhaps working in a different way and maybe checking email less often. We were aware of all this before deciding on the blog as an evidence collection method, but we had expected more input to result from the emails than we actually received.
So did this negative engagement indicate that many people on the lists were not big users of Web 2.0 tools and services? That would be one interpretation, but another would be that it may indicate that those using these tools still appear to be predominantly in the early adopter category. Our original hope was that even a small number of respondents to the email request would be enough, if they would pass on the request to friends and colleagues who were known to be similarly interested, but it became obvious that this was not happening, at least not to the extent we had hoped.
Looking for Case Studies
Although the blog seemed a useful approach, we had always intended to use other methods. One was to identify a selection of people known to be using a range of Web 2.0 tools and services and then compile case studies which would provide a more rounded picture of how Web 2.0 is being employed within working lives. Given the time-consuming nature of this method, candidates needed to be selected carefully.
Our first case study focused on a new lecturer in library and information studies, who was known by us to be using Web 2.0 tools and services. Initial contact was followed by email discussion as a draft document was collaborated on, using a question-and-answer interview style. Further case studies also involved telephone discussions.
The case study approach proved useful in obtaining qualitative data as we were able to explore not only which tools and services were being used, but also the reasons why they were chosen, their limitations and advantages.
The main problem with case studies was finding whom to approach. Ideally we would have several exemplars of each category: lecturers and researchers in different disciplines plus support services such as libraries, IT services, marketing and careers. In practice it quickly became obvious that we would simply have to settle for a less comprehensive mix. However, we were able to publish the first case studies in early July. A further problem was that people who initially seemed to be likely people (e.g. we were aware they were using a Web 2.0 tool) turned out to be unsuitable after all (either that was the only tool they used or it was being used in a very limited way).
Although a few in the library and information community were known to us or UKOLN colleagues, those in other communities were not – so again there was a need to identify suitable cases. We used two approaches to identify further potential case studies. Firstly, we kept a watch on comments posted on the blog, and approached a few people who appeared to fit our criteria. Secondly, we were able to pick out a few likely cases from our final strand of evidence collection – desk-top research. Thirdly, several candidates were identified from information gathered in the literature review.
Desk-top Research
Our final method was to look at Web sites in the UK HE sector and collect relevant information. In one sense this is an easy method, since all institutions now have Web sites, but in practice it can be frustrating and time-consuming. Institutions vary in how their Web site is structured and how much is easily visible to the visitor, what further parts can be located following a site search and what remains password-protected.
The Web site survey followed a consistent strategy. Since other evidence was primarily coming from the library community, the library and learning centre was reviewed first, looking in particular for blogs, podcasts, and Web 2.0 features within catalogues. The next step was to look at the alumni section of the site, where it was expected that social networks might feature. Then it was on to sections for prospective students, including accommodation and student life details. Once obvious links were exhausted, the site was searched for ‘blogs’ and ‘podcasts’.
As noted earlier, there were limitations to this approach. Evidence collected was restricted to what was available on the site at the time it was reviewed. The site searches on ‘blogs’ revealed that it is likely that more blogs are in use than are openly visible, because it appears that they are often set up within an institution’s virtual learning environment (VLE). And it is of course possible that institutions may be planning more use of Web 2.0, but have yet to implement it.
An additional benefit of this research strand was that, as noted above, it was useful in identifying a few further potential candidates for the case studies.
Conclusion
This has been an interesting experiment and a learning experience. The blog structure has proved to be a useful method of data collection, albeit with certain disadvantages. Firstly, the amount of data that can be collected is dependent on either effective advertising and take-up or a small target population that is committed to providing that input. Secondly, once the page topics were decided, it was not easily possible to add further topics once the blog was up and running.
A major issue has been getting the message out to people who are using Web 2.0 and ensuring they contribute. Possible reasons are that we did not find the right route to attract their attention, we were unfortunate in the time of year we were carrying out the study, or we simply received few responses because Web 2.0 users are still limited in number. The final answer is likely to be a combination of these reasons.
Was all this effort wasted? The answer is no, for several reasons, Firstly, we did get some evidence and it was the type of information that we were trying to obtain. Then, even negative evidence can be useful; for example, by mid-July we still had no comments on Web 2.0 tools and services being used for research purposes. Knowing this prompted us to try some targeted approaches for input from that group. We also gained more knowledge as to the value and usefulness of using a blog for data collection. Finally, creating the blog has enabled us to create a Web presence for the study (a funding requirement); it is easy to maintain and update and we have been able to share our data with anyone interested throughout the lifetime of the study. On a final note, we’d like to emphasise that the study is still current at the time of writing and we are continuing to collect evidence. We welcome further comments on our blog until the end of August 2009.
References
- The JISC Web site http://www.jisc.ac.uk/
- JORUM http://www.jorum.ac.uk/
- JISCmail http://www.jiscmail.ac.uk/
- JISC SIS Landscape Study http://blogs.ukoln.ac.uk/jisc-sis-landscape/
- JISC SIS Landscape Study blog http://blogs.ukoln.ac.uk/jisc-sis-landscape/
- Web 2.0: Wikipedia entry, retrieved 30 July 2009 http://en.wikipedia.org/wiki/Web_2.0
- Jorum Home http://www.jorum.ac.uk/
- the Depot http://depot.edina.ac.uk/
- ShareGeo Home Page http://edina.ac.uk/projects/sharegeo/
- Good APIs Project http://blogs.ukoln.ac.uk/good-apis-jisc/