EEVL News: What EEVL Users Really Want
What do the users of EEVL [1], the Internet guide to engineering, mathematics and computing, really want from the service? Do they want EEVL to develop more portal services? Do they want more expansion of EEVL's catalogue of Internet resources? Do they want other things? The only way to find out what users of an information service really want is to ask them. This is what EEVL did earlier this year through a Web-based questionnaire. The questionnaire is now closed, but an archive version is available [2]. In order to encourage people to complete the questionnaire, four £25 Amazon vouchers were offered, with winners chosen at random. The questionnaire was available for three months, and at the end of each of the first two months one winner was chosen and announced on various JISCmail lists, with a final two winners being chosen at the end of the exercise.
Questionnaire
A considerable number of entries were received - 364. Of these, 7 were completely blank, and 8 were obviously duplicate entries, leaving a total of 349 valid entries. Good questionnaires are not so long that entrants become frustrated. There were 18 questions in EEVL's, requiring an estimated 10 minutes to complete.
The EEVL Questionnaire 2004
It may seem obvious to state this, but meaningful answers are only generated from meaningful questions, and therefore considerable thought was given to the formulation of the questionnaire. In addition, there were several opportunities for entrants to note their own comments or to expand on any opinions they might have of the EEVL service. The data collected from the questionnaire is proving to be very useful for planning developments to the EEVL service, and many thanks go to everyone who entered.
Users
A reasonable cross-section of different types of users completed the questionnaire, with the highest number (140) being undergraduate students, followed by information professionals (67), postgraduate students (57), researchers (33), lecturers (22), practising professionals (12), and 18 'others'. The relatively small number of lecturers was disappointing, as EEVL is very keen to encourage use by academic staff. Most entrants were from the UK (304), with a high proportion being based in Higher Education (232) and a smaller number in Further Education (62). The latter figure was again a little disappointing, given that the questionnaire had been deliberately promoted to several FE-related mailing lists, and considering that the EEVL service is potentially very useful to those in FE. Most entrants 'main' subject area was engineering (191), but a healthy number (96) came from computing. As computing is the most recent subject to be added to EEVL, it was encouraging that feedback was received from this sector.
How do people find out about EEVL? Most entrants stated 'personal recommendation', which reinforces evidence which has shown how important the 'invisible college' is to those involved in technological subjects. An encouraging number of 71 stated that they had heard of EEVL via promotional efforts, which suggests that those efforts have considerable effect on usage. A disappointingly high proportion of entrants (144) had not used EEVL before, which implies that some other questions might have been completed without a great deal of knowledge about the service, though it does show that questionnaires and prizes help to promote services.
Services
The main body of the questionnaire concerned EEVL's Internet Resource Catalogue of over 10,000 quality resources, plus EEVL's other services such as Recent Advances in Manufacturing (RAM) [3], the Engineering E-journal Search Engine (EESE) [4], the Virtual Training Suite (VTS) [5] and the OneStep news and job announcement aggregators [6][7]. Some questions also invited opinions on possible new services which could be developed.
A high proportion thought that the EEVL Catalogue was easy to use. Slightly less thought that searching the Catalogue produced useful results, and slightly fewer, though still a considerable number (172), reckoned it to be a valuable alternative to search engines such as Google or AltaVista. 223 thought the IRC descriptions were useful, and only 8 thought they were not useful. A fairly high proportion of entrants had not used EEVL's additional services, which is surprising as these services are very popular. Perhaps it suggests that more effort should be made to promote the additional services or make them more visible from the home page, and in fact this was suggested by more than one respondent. Of those who had used the additional services, many found EESE either OK or good. This in itself is a little surprising, in that at the time of the questionnaire, EESE was in need of maintenance. Obviously, the concept of a search engine which searches freely available full-text engineering e-journals is popular, and effort is currently going into not only upgrading EESE but also into producing equivalents for mathematics and computing. The VTS, as expected, received a high level of user satisfaction (64 OK, 51 Good, 10 Poor), with OneStep Jobs sharing similar results (61 OK, 27 Good, 9 Poor).
Entrants were asked to rate six ideas for possible new EEVL services, and the results were not completely as expected. There was considerable support (80 ranking it Very good idea, 113 Good and 107 OK) for a service which allows searching a range of free databases from one place. There was slightly less support for the same type of service searching subscription databases, with 26 thinking this was actually a poor idea. Considering that they might be thought of as niche services, there was an unanticipated level of support for a journal Table of Contents (TOC) announcement service, a new book announcement service, and an events and conference announcing service, with only slightly less enthusiasm (165 OK, 75 Good, 40 Very Good) for a service which gave details of funding calls.
A very encouraging 86.8% of respondents rated the service EEVL currently provides as Very Good, Good or OK, with the highest appreciation shown by information professionals (77.6% rating either Very Good or Good). Undergraduates were slightly less appreciative (44.3% rated the service OK and 41.5% Very Good or Good), but this figure may have been influenced by the fact that proportionately more undergraduates had indicated that they had not used the service before. 9.1% of lecturers (two respondents), thought the service was poor. Highest satisfaction came from those in engineering, followed by computing and then mathematics.
Perhaps the most useful information resulting from the questionnaire came from answers given to specific questions and comments on various aspects of the service. A considerable amount of time and effort goes into writing Internet Resource Catalogue records, in particular the resource descriptions, and it was therefore important to discover whether, and to what extent, these descriptions were useful or not. One question dealt with this, and respondents were asked to "Please explain briefly why you find EEVL Internet Catalogue descriptions useful/not useful." The vast majority found the descriptions useful. In fact, there were very few negative comments, and even these tended to show that the descriptions were to some extent appreciated. For example: "I very rarely read more than one or two lines before deciding whether or not to look at a site itself" and "Quite useful, would appreciate a little more detail" and "Some are useful, some not". Most comments were along the lines of "Descriptions are necessary because the names of the resources are not self-explanatory" and "Evaluation distinguishes EEVL from search engines" and "They help me decide if the level and content of the materials is appropriate". Recently, the style of EEVL's resource descriptions has been changed, and the content of a resource is now described in the first few lines, so even users who only read the beginning should be able to evaluate a resource's potential.
The answers given to the question "What type of resources would you like to see more of in the EEVL Internet Catalogue" were far less clear cut, and demonstrate that the Catalogue is used in many different ways by different users. In fact, more of most types of resources were identified at least once, ranging from preprint servers, tutorials, ebooks, news, book reviews, current awareness resources, databases, quality bookshops, and free e-journals, and contradicting this, more resources which are not journals. In a similar way, many subject areas were identified once. However, there were numerous requests for more resources in computing, in particular software engineering, programming, artificial intelligence and reviews of software. As stated previously, the computing section of EEVL is younger than the other two subject areas, and considerable effort is currently going into its development, so hopefully these requests for more records will in time be satisfied. There were also a few requests for more resources in learning and teaching, although fewer, perhaps, than might have been anticipated in the light of the emphasis currently placed on learning and teaching e-resources in academia. More resources of a suitable level for further education cropped up three times. This category within the Catalogue will also be increased in the future when the outcomes of the RDNFE Project [8] are incorporated into EEVL.
Although several people stated that the existing balance was about right, and that there was not really a need for more of any type of resources, this sentiment was contradicted by a number of responses received to the next question: "What type of resources would you like to see less of in the EEVL Internet Catalogue?" Over thirty responses of 'None' were given. Most people were of the opinion that there was a need to increase the size of the Catalogue rather than decrease it. This results in a considerable conundrum for the EEVL service. As the size of the Catalogue grows, so too does the amount of maintenance required to keep it current. Two of the subjects covered by EEVL (engineering and computing) are particularly volatile in terms of Internet resources. Many sites in these constantly developing subject areas are not static and the records for these resources require regular editing and review. In addition, what are cutting edge subjects this year often fade into obscurity next year. How to increase the size of the Catalogue without increasing the budget for the service presents a considerable problem. One way to achieve a similar, if not identical, net effect, would be to concentrate on particular type(s) of resources in the Catalogue, but supplement retrieval of other type(s) of resources through further development of EEVL's full-text harvested Web sites. In other words, it will almost certainly become necessary to reduce the rate of increase of the Catalogue for entries which are fully described and catalogued by subject experts, and instead use machine-harvested data from other Web sites to supplement search results. This possibility is currently being investigated by the EEVL team.
Although responses to the type of resources users would like to see more of in EEVL included both company resources and departmental sites, several responses to types users would like to see less of included the very same resource types. Company information was identified in many responses as being an intrusion. One again, this presents something of a conundrum, as some company sites have relatively high click-through rates from the EEVL Catalogue, showing that, for some, they serve a function. Other evaluation exercises, plus incidental evidence, have shown that some users find the inclusion of company sites in EEVL very useful. Both engineering and computing are applied subjects, and it is therefore not surprising that some users have close connections with the commercial world or are looking for information related to industry. The solution is therefore complex. EEVL should almost certainly identify existing Catalogue records for company sites which have not generated many click-throughs, and weed these from the Catalogue. Company sites remaining in the Catalogue should include only the top companies in each sector plus those whose Web sites contain a particularly substantial amount of useful information. Perhaps company sites should have a lower ranking in search and browse results than other resource types. In addition, it should be made much easier for those users not wanting to retrieve company sites to filter them out of results. At present, the only way this can be achieved is by selecting all other resource types from the advanced search page [9], and this involves much effort. Perhaps a toggle option on the main search page which easily allows users to edit company sites out - or in - would achieve this. All of these options are currently being investigated.
Another resource type which was mentioned several times in the 'less records of this type' category was subscription-based services, especially subscription e-journals. Once again, some of these have high click-through rates. Also, subscription e-journals are included in most other subject gateways. An Internet guide which does not include the top (subscription-based) journals in its subject areas would certainly be less useful to some users (especially those who have subscriptions, or who have the means to pay-per-view), yet at the same time it is also clearly annoying for other users to be directed to resources which require a paid subscription when their institution does not subscribe. The solution is probably for such resources to be very clearly identified, at the beginning of the description and also through the use of suitable graphics, as requiring a subscription. Better still would be if EEVL could identify whether a user is likely to have a subscription or not, and this is one aspect of the work currently being undertaken as part of the Subject Portal Project [10]. It should be noted that those subscription-based e-journals which are currently included in EEVL offer at least tables of contents and abstracts, and even where a user is not able to immediately click through to the full text, they can identify articles of interest and apply for the full text via inter-library loans services. Though this will be useful for some, it may not satisfy others.
The next two questions asked, in turn, "What do you consider to be the most useful aspect(s) of the current EEVL service, and why?" and "What do you consider to be the least useful aspect(s) of the current EEVL service, and why?" It was encouraging that far more useful aspects of the service were identified (128) than less useful ones (59), and several took the opportunity of saying that "nothing is not useful". Confusingly, some features appeared in both lists! The full-text harvested search engines were highlighted by several respondents as particularly useful, though two said it was difficult to get good results from these. This is another area which it is hoped can be improved in the future. The OneStep Industry News and Jobs services were identified by 14 respondents as being useful, and by 14 as not being useful. However, of those who identified them as not useful, the reasons included "because I am first year", "not personally relevant" and "because I am not an engineer, but I think it is definitely a good thing to have on the site". A couple of entries questioned the need for news, stating that news was available from other sources, including the BBC. This was a bit surprising, as the news sources included in OneStep Industry News have been selected because they concentrate on sector-specific industry news, and many of the feeds are not generally available elsewhere. Clearly, as with commercial information, industry news is relevant to some users, but not to others. The Virtual Training Suite, RAM and EESE were all identified several times as being useful. More generally, the service as a whole, its role in providing subject-specific access to quality information, the ease of use of the site, the fact that it is intuitive, its currency and its focus were all highlighted by several respondents.
Less useful aspects included the speed of the site. This has already been addressed since the time of the questionnaire through the purchase of a new server resulting in far quicker response times. Another service which was selected as being less useful was the Current Awareness page [11], though it is recognised that this might be more useful for LIS (Library and Information Science) professionals and researchers than students. In fact, the Current Awareness page has quite a healthy hit rate. Other criticisms included "patchy coverage", "coverage of databases that are too high-level for my teaching needs" and the fact that EEVL had not been advertised sufficiently. A couple of respondents had difficulty understanding the difference between Catalogue results and the harvested search engine results, and also with locating some of the additional services from the home page. Both of these points will be addressed in the redesign of the site.
A number of very varied responses were received to the question "Are there any other services/features which you would like to see from EEVL? Please describe." Information on learning materials, including those which can be included in Virtual Learning Environments (VLEs), was mentioned more than once, as was information on funding opportunities. Confirming the interest in books, which was identified previously as a potentially popular service, a facility to purchase books was identified. It is already possible to buy books from one publisher (Pearson) at discounted prices via the EEVL On-line bookstore [12], and so, perhaps, this service should be extended to include other publishers. Better browsing for university departments was mentioned by one respondent, (though, perversely, too many records for departmental sites was mentioned as a less useful aspect of the service in a response to a previous question). Interestingly, the ability to cross-search Google was identified. It may be possible to include this in a future service development. Databases of research, useful references by subject area, more full text services, as well as expert opinions in subject areas, were further suggestions. It is hoped that future developments to the EEVL service will help satisfy all of these.
Some of the answers to the next question "Please provide any examples of how you have used EEVL services in your study, research, teaching or work? We are interested to hear examples of both positive and negative experiences" were slightly disappointing, in that in many instances the answers were fairly imprecise or similar to: "to find technical information in engineering". Although in responses to other questions it was apparent that there was a high level of satisfaction with the EEVL service, some responses to this question were slightly negative, along the lines of "I show it to students. I don't use it much myself". More typical responses, however, were "I recommend it to all my undergraduate and postgraduate students as part of their information skills/literacy sessions." and "I recommend EEVL to Engineering students, as an easy way to find reputable Internet-based information. I use EEVL to discover useful engineering Web sites, and for information gathering on specific subjects in engineering." Another positive response was "We suggest EEVL to students who are beginning project work in engineering and computing with examples showing that it can pick up hits they won't find in Google - in a recent example, a search for bridge construction picked up 12 EEVL sites, 10 of which did not come up in the first few pages of a Google search. I don't know how much they actually go on to use EEVL. I suspect a lot of them are so used to doing a Google search that they don't bother, but we do try!" It is obvious that EEVL is much appreciated by the LIS community, and also that this community does much to promote EEVL to students. In other answers to this question, once again, the potential of the Computing section of EEVL was identified by several respondents.
The final question asked how EEVL could be made more useful, and additional comments or suggestions were invited. Some very interesting responses were received. More promotion of the service was requested several times, along with several requests for more printed flyers. There were suggestions for cross-searching EEVL with OMNI, for having an alert scheme, for providing the very latest information in selected subjects, for providing improved access to full-text materials, and for including access to other information sources allowing the retrieval of papers via cross-searching of subscription-based databases. All of these things should become available via the forthcoming EEVL Subject Portal Demonstrator. Some suggested less subscription-based information, and others requested more. One person said the service should be faster, and another stated that it was already very fast. Various particular subjects were identified as areas for improving coverage, and once again, computing was pinpointed. More resources for researchers was mentioned several times, as were links to recommended text books. There was an indication that the site design could be improved, and once again, this will be addressed in the not too distant future.
Conclusion
Though in a couple of instances the questionnaire provided conflicting feedback, on the whole it has proved to be a very worthwhile exercise. It has provided EEVL with a great deal of useful data and opinion, and once again many thanks go to those who completed the questions. On the whole, respondents were very positive about the EEVL service and its usefulness. They found the Internet Resource Catalogue, and resource descriptions, useful. It is particularly satisfying to note than many of the suggestions for improvements are already being processed, and that there appears to be considerable support, by the users of EEVL, for several new features which will become available as part of the Subject Portal Demonstrator. In other areas, there is still considerable work to be done to improve subject coverage, particularly in computing, and to provide the necessary filtering options or some other facility to allow for more successful searching and browsing of the service.
References
- EEVL the Internet Guide to Engineering, Mathematics and Computing http://www.eevl.ac.uk/
- EEVL Questionnaire 2004 http://www.eevl.ac.uk/2004-questionaire/index_old.htm
- RAM Database http://www.eevl.ac.uk/ram/index.php
- The Engineering E-journal Search Engine (EESE) http://www.eevl.ac.uk/eese/
- Web Tutorials: The RDN Virtual Training Suite (VTS) http://www.eevl.ac.uk/vts/
- EEVL : OneStep Industry News http://www.eevl.ac.uk/onestepnews/
- EEVL : OneStep Jobs http://www.eevl.ac.uk/onestepjobs/
- Enhancing the RDN for Further Education (RDNFE) http://www.rdn.ac.uk/projects/rdnfe/
- EEVL Catalogue Search http://www.eevl.ac.uk/catalogue.htm
- Subject Portals Project Phase II http://www.portal.ac.uk/spp/
- EEVL Current Awareness Services for Engineering, Mathematics and Computing http://www.eevl.ac.uk/current_awareness_services.html
- EEVL On-Line Bookstore http://www.pearsoned.co.uk/bookshop/subject.asp?item=6252&affid=EVL