Putting the Library Into the Institution: Using JSR 168 and WSRP to Enable Search Within Portal Frameworks
Under the aegis of the UK Joint Information Systems Committee's (JISC) Portals Programme [1] development projects have taken place to investigate the use of portals as the presentation path for a variety of search tools. A major output from these projects has been the development of a portal interface, a Web site that users could come to in order to make use of the functionality that the portal provided, particularly searching. Each project, as a key part of its investigations, created its own such interface and Web site. These were tested with end-users to establish that the portal was presenting its services in the most useful manner. The work the projects have carried out on portal interfaces has been valuable in understanding better the types of services end-users wish to interact with and where they perceive value to sit within the portals.
Towards the end of the portal projects it became apparent, though, that the presentation of services through a dedicated Web site was just one way in which they could be delivered. Many institutions are now making use of virtual learning environments/course management systems (VLE/CMS) and a number are starting to implement institutional portals to facilitate the aggregation and presentation of applications, services and information to their staff and students [2]. All universities also work heavily within the general Web environment, providing a vast collection of information to those both inside and outside the institution. These institutional environments offered the capability of bringing information and services to the end-user in the context of their work and/or study. This delivery of information and services, including search, to the end-user contrasts with the more traditional approach of building dedicated Web sites and then expecting or requiring the end-user to find and access them.
Whilst the provision of dedicated Web sites is still the predominant route for the delivery of Internet-based search services, it was considered important to examine more closely the integration of Internet-based search tools within institutional environments to establish how this might take place. The CREE Project was funded by JISC to investigate this.
The CREE (Contextual Resource Evaluation Environment) Project
The CREE Project [3] has been structured around two main goals.
- The investigation of user requirements for the presentation and delivery of search tools through a variety of institutional environments and contexts. The University of Hull undertook this work in conjunction with colleagues at the University of Oxford and Newark and Sherwood College. The cross-section of institutions allowed variances to be identified between institutions and different end-user groups, but also validation of results in common across the three taking part.
- The investigation of the JSR 168 [4] and WSRP [5] portlet standards to allow the presentation of search tools within conformant portal frameworks (e.g., uPortal [6]). This work was undertaken primarily by project technical partners at the University of Oxford, the Archaeology Data Service at the University of York, the EDINA National Data Centre at the University of Edinburgh, and instructional media + magic, inc., a software consultancy in Washington D.C., each investigating the use of these standards with a different type of search tool.
This article, following a previous introduction to the project in Ariadne in 2004 [7], focuses primarily on the technical work undertaken by the project. User requirements investigations from the project are reported in a companion article in D-Lib [8].
Technical Investigations Supporting User Requirements Gathering
The two goals of the project came together in the development of a series of fully functional interactive demonstrators. A clear aim of the user requirements gathering was the use of multiple methodologies to validate the results; the theoretical approaches of a survey and series of focus groups were complemented by the more practical approach of showing users the possibilities of presenting search tools through different institutional environments within demonstrators. The investigation of the JSR 168 and WSRP standards supported the development of the portal demonstrator and enabled a view of the types of services that might be delivered within this environment to be offered.
Portal Frameworks
Web portals have been in existence for many years now, offering "...a starting point ... to other resources on the Internet or an intranet." [9]. Many of these were and still are advanced web pages, presenting a range of services and starting points all as part of one page. The use of such Web portals within organisations (often equivalent to intranets) followed much the same path. In order to provide more advanced services, however, a common set of services has become necessary to underpin the portal and provide the framework upon which services, information and applications could be presented, aggregated and customised for the needs of the organisation and individuals within it. As the uPortal FAQ indicates, "...the framework takes care of the common functionality that every portal needs, so that you can implement the parts that are important and specific to your campus." [10]. As indicated in Wikipedia [9], many such frameworks, commercial and open source now exist.
Portals have also been discussed frequently in the context of libraries as a means for delivering a wide range of services [11] [12]. Systems available under this 'library portal' banner do not yet appear to offer the same level of infrastructure as portal frameworks. They are geared specifically to the needs of the library, but would struggle to play a role in delivering non-library services alongside. Portal frameworks such as uPortal, Oracle portal, IBM WebSphere etc. aim for a more generic approach to enable diverse services, including those from the library, to be delivered so long as they can be technically served through the framework.
As portal frameworks have developed each has developed its own way of programming and building services, or portlets, for presentation within the framework - its own portlet API. Each service, or portlet, has to be able to communicate with the framework to function properly and each framework has offered its own mechanism [13]. This has implications for development, and prevents portlets from being exchanged for use across portal frameworks (which can prevent duplication of effort). This lack of interoperability has led to the two standards processes that have produced the JSR 168 and WSRP portlet API standards [14]. Both are now being widely adopted by commercial and open source portal framework developers.
The portal framework used within the CREE Project has been uPortal. This open source framework has been in use at the University of Hull to deliver its institutional portal since September 2003 [15], and aggregates a range of services, applications and information from across the University. The development of the portal demonstrator also used uPortal, version 2.4.1, and the technical development partners in the project similarly made use of this framework for consistency and ease of availability.
Portlet Development
The intention of the technical strand of CREE was to see whether or not existing library search tools could be adapted for use within a portal framework using the JSR 168 and WSRP standards. The search tools investigated were as follows:
- The GetRef [16] and GetCopy [17] services from EDINA [18]
- The JAFER Toolkit [19], used in this instance for access to library catalogues
- The HEIRPORT [20] portal from the Archaeology Data Service [21]
All but the Google portlet relied on Z39.50 as its underpinning search protocol. As previously reported, initial development was with the JSR 168 standard. The WSRP4J toolkit was then used to enable the search tools to be exposed as Web services via WSRP. Full details of the work undertaken by technical partners can be found on the CREE Project Web site [22].
JSR 168 Development
The JSR 168 standard is a Java Community Process standard and, hence, Java knowledge, skills and software are required in order to make use of it. This is the case even where the original tool being adapted is written in another language, as it was in the case of GetRef and GetCopy (written in Perl). Mapping the search tool functionality to the JSR 168 portlet API was complicated by this fact, but was achievable. Ideally, JSR 168 works best with tools that are originally Java-based, though this limitation is by no means absolute.
Three different possibilities offered themselves for the development of JSR 168 portlets:
- Port as much of the existing code/structure as possible to create a new application offering the same functionality as the original, only within a portal environment
- Use the JSR 168 as a wrapper around the existing system, adding enhancements where deemed suitable
- Rewrite the application completely from the ground up as a native JSR 168 portlet application
For the most part, design 2 was selected. Design 1 was not considered feasible due to dependencies within the existing applications, and the project did not permit the time required to follow design 3. The JSR 168 portlets thus developed acted as a means to send a search to the host application (search tools in the case of CREE) from the portal framework and then receive back and present the results from this search. The application could be located locally (which is the intended role for JSR 168) or remotely, communicating via HTTP or Z39.50, for example.
Notwithstanding the aim of presenting portlets through the uPortal portal framework, portlet development itself was found to be simpler using just the Pluto portlet container [23]. Portlet containers such as Pluto are used by portal frameworks to store available portlets: Pluto itself is used within uPortal. However, it can also be used in its own right for development purposes. The use of Pluto also allowed any dependencies on uPortal itself to be removed from the development path. Although the use of the JSR 168 standard aims to allow JSR 168 portlets to be used within different conformant portal frameworks, different frameworks have tended to offer optional 'extras' beyond what the standard offers, making interoperability practical but potentially non-trivial. Use of Pluto avoids this.
It was found to be valuable to make use of the Model-View-Controller (MVC) design pattern [24] to assist with the building of the portlets (See Figure 1). This way of structuring the Java programming required offers a means to separate out the business logic of the portlet from its presentation and also the control logic that connects them. This flexibility allows different types of presentation, e.g. viewing search results, editing preferences, viewing help, to make use of the same business logic in a controlled fashion.
Figure 1: MVC design pattern (See Reference [24])
The issue of presentation was one that affected all portlets being developed. The development of the Google portlet by im+m was based around an XSLT (eXtensible Stylesheet Language Transformation) portlet framework, a methodology that fits the MVC design pattern by creating a clear separation between business and presentation logic. The development of the JAFER portlet made use of this portlet framework, whilst both the GetRef and HEIRPORT portlets also made use of XSLT to transform XML-formatted search results for presentation within the portal framework. XSLT mark-up was used to enhance the look and feel of the results, as well as the ability to alter the presentation according to the role of the user (e.g., staff or student). Elsewhere, basic HTML tags were also used for fixed text. JSR 168 does define a set of cascading stylesheets that can be used to render text etc, but XSLT techniques were found to be more flexible.
The mechanism JSR 168 uses to carry out its interaction with the search tool involves a two-step process (processAction and doView). This fitted in neatly with the QUERY-PRESENT logic of the Z39.50 protocol. However, with each processAction step the portal refreshes its entire display: this prevents progress updates of a search being presented within the portal, as each one would require an entire screen refresh.
All search tools under investigation were successfully adapted for use within a portal framework using JSR 168. Deployment of these portlets at the University of Hull for the portal demonstrator took a little time to master, but was relatively straightforward: a key addition to the deployment process is the definition of a portlet.xml configuration file alongside the standard web.xml deployment descriptor for a Java Web application.
The portal demonstrator was able to offer access to the portlets in a number of scenarios (on the front page, collected together on a 'library' page (See Figure 2), or singly) and display these for users to comment on: feedback on the views expressed on the presentation of search tools within a portal environment can be found in the companion paper in D-Lib [8].
Figure 2: Portal demonstrator screenshot
WSRP Development
As previously reported, the development path of choice taken by the project's technical partners led from the development of JSR 168 portlets to Web Services for Remote Portlets (WSRP) portlets through the use of the Apache WSRP4J toolkit [25]. This toolkit is not yet a fully mature technology, although it is certainly able to achieve the transition to WSRP required. Two main issues affected use of WSRP4J by technical partners: the documentation available was not as clear as it could be; and the status of the toolkit at Apache as an incubator project. This means that the code needs to be pulled straight from the development Concurrent Versioning System (CVS) repository in order to make use of it locally, (incubator projects do not have specific versions available), which can lead to some instability. A presentation given by Matthew Dovey from Oxford University at a NeSC workshop on Portlets and GridSphere in March 2005 discusses some of the issues surrounding the use of WSRP4J at this time [26].
WSRP is itself a more complex specification than JSR 168. To overcome these difficulties and document a procedure for using WSRP4J to produce a WSRP portlet from a JSR 168 start point, a step-by-step guide was developed for local use at Oxford [27]. These were subsequently adopted successfully by other project technical partners and have been used by developers outside the project as well (e.g. Connect Embedding Project [28], which also gives additional details of the steps required). A summary of the necessary steps is as follows:
- Install WSRP4J on a Web application server. Scripts are supplied with the WSRP4J software to do this.
- Deploy the portlet as a separate Web application on the same Web server.
- Configure the WSRP4J Web application (via the portletentityregistry.xml file) to publish the portlet via WSRP.
- Configure a WSRP consumer channel for the chosen consuming application. This requires the portal administrator to assign values to the four WSRP endpoints (markup interface URL, service description interface URL, registration interface URL, and portlet management interface URL).
As with all Web services, a producer and consumer exist at opposite ends of the interaction occurring. In this instance, WSRP4J acts as the producer, while the portal application displaying the search tool at the other end is the consumer. Note that the communication between the Pluto portlet container and WSRP4J also follows this producer/consumer relationship. Figure 3 shows the overall architecture for the JAFER portlet, including the links between the portlet and the application itself.
Figure 3: JAFER portlet architecture (Matthew Dovey)
All the JSR 168 portlets produced were successfully deployed as WSRP portlets using this step-by-step guide, highlighting the ability of the chosen development path to deliver portlets that can be presented using either standard according to need. As mentioned above, JSR 168 portlets were selected for the portal demonstrator because of the level of local control these offered. WSRP portlets are delivering Web services, and there was a degree of uncertainty about the stability of services being available when required.
By way of comparison in using WSRP, the Go-Geo! team at EDINA, working alongside the CREE development there, developed a procedure to produce a WSRP portlet directly from a Perl application, without going via JSR 168 [29].
Additional Portlet Functionality
In addition to the search functionality provided by the JAFER, GetRef, HEIRPORT and Google portlets, a number of additional functions were developed to support the search process. These functions highlighted some examples of what it is possible to deliver through a portlet.
The GetRef service is closely associated with the GetCopy service, which provides the user with a link from the search results to potential locations for the full text of an article. Initially, the project had intended to develop separate portlets for these services. However, the JSR 168 specification does not currently make any provision for inter-portlet communication, a requirement if GetRef were to have to pass information to GetCopy for it to seek a location for an article. More specifically, it is not currently possible to create a new portlet (the potential GetCopy portlet) from an existing portlet (GetRef) on demand. As such, GetCopy functionality was delivered through an additional page within the GetRef portlet. The user clicks on an 'Attempt to locate article' button displayed next to the GetRef search results, and this then invokes a call to GetCopy.
The development of the HEIRPORT portlet did manage to identify a way in which some form of inter-portlet communication might take place, though this lay outside the JSR 168 specification. This allows for an additional, though existing, portlet to be called when certain conditions are met and information displayed within this rather than the original portlet. In the case of HEIRPORT this was used to display detailed search results and a map showing the location of the archaeological object in question alongside the original search results. An example of this mechanism can be seen in Figure 4.
Figure 4: HEIRPORT portlet screenshot with dual portlets
The HEIRPORT development also provided a notepad facility that allows the user to save results and export them for later use.
One feature of the JAFER portlet is the ability to search more than one resource at the same time. The display of results in such circumstances is problematical: should the results be merged or displayed separately? A possible scenario was drawn up whereby the results from each resource are displayed in additional portlets, separate from the search portlet itself, and this appeared to meet user requirements in feedback. The issue of inter-portlet communication, and specifically the generation of new portlets, unfortunately meant that this could not be put into practice within the project's lifetime. Inter-portlet communication is expected to be a part of the WSRP version 2.0 specification due in 2006, but it remains to be seen exactly how this will enable connection between portlets.
The development of the Google portlet focused not so much on portlet functionality but on making the most of available portlet capability. All portlets have three modes: View, Edit and Help. The View mode consists of the normal content displayed by the portlet (the search box, results etc.). im+m also made extensive use of the Edit mode, allowing the number of results displayed to be altered and allowing for internationalisation through the use of XLIFF [30]. A Google Web API key is required for the portlet and this can again be entered in the Edit mode of the portlet. Finally, a contextual help XML schema was developed to enable structured help to be delivered for any individual portlet.
Conclusions
The development of portlets for use within the CREE Project has been a highly successful one, allowing user testing to take place at a detailed level. The JSR 168 specification offered a relatively straightforward means for creating a portlet from scratch, notwithstanding the preference for a starting point in Java. This allows the basic search functionality of the selected search tools to be reproduced in a portal environment, allowing the user to make use of them within the portal and not have to jump to other Web pages. A certain degree of additional functionality was also included where feasible. The full extent to which search functionality can be reproduced within a portlet, and how this might be displayed using XSLT, will however require further work as CREE has, in the end, only scratched the surface of what is possible. Experience suggests that viewing a range of other JSR 168 portlets greatly helps with their development and there are an increasing number of these now available through sites like POST [31].
The use of WSRP was greatly eased by the use of the WSRP4J step-by-step guide developed at Oxford. The ability of WSRP effectively to serve applications remotely through the use of Web services appears to favour the use of this standard over JSR 168 for many search service providers. A number of issues remain to be resolved, though, not least the security of the Web service messages concerned. From the viewpoint of a service provider, the additional support of a WSRP portlet, alongside supporting access via the search tool's home Web site, also needs to be considered. WSRP4J itself, although used within a number of commercial enterprise portal frameworks to some degree, is yet to mature fully and requires additional effort.
Notwithstanding this, the CREE Project has demonstrated the ability to take existing applications (search tools) and present these within alternative institutional environments such as an institutional portal. Where other environments choose to make use of the JSR 168 and/or WSRP standards, for example virtual learning environments (VLEs), then the same portlets can be used within these. The standards open up flexible options for the presentation of search tools and allow search to be presented wherever it is most useful.
Further Information
All reports from the CREE Project are available via the CREE Web site [3].
Acknowledgements
The authors would like to thank Wayne Thompson in the e-Services Integration Group at the University of Hull for his work in creating the portal demonstrator.
References
- JISC Portals Programme, http://www.jisc.ac.uk/programme_portals.html
- JISC Study of MLE activity, http://www.jisc.ac.uk/project_mle_activity.html
- CREE Project Web site, http://www.hull.ac.uk/esig/cree/
- Introduction to JSR 168 - the portlet specification, http://developers.sun.com/prodtech/portalserver/reference/techart/jsr168/
- OASIS Web Services for Remote Portlets Technical Committee, http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wsrp
- uPortal, http://www.uportal.org/
- Awre, C., Dovey, MJ., Hunter, J., Kilbride, W. and Dolphin, I., "Developing Portal Services and Evaluating How Users Want to Use Them: The CREE Project" Ariadne 41, October 2004 http://www.ariadne.ac.uk/issue41/awre-cree/
- The CREE Project: Investigating User Requirements for Searching within Institutional Environments, Chris Awre, Gabriel Hanganu, Caroline Ingram, Tony Brett and Ian Dolphin, D-Lib Magazine, October 2005, Volume 11 Number 10 http://www.dlib.org/dlib/october05/awre/10awre.html
- Web portal entry in Wikipedia http://en.wikipedia.org/wiki/Web_portal
- uPortal FAQ, http://www.uportal.org/faq.html#4_2
- LibPortal: Library Portal Survey and Review, http://www.jisc.ac.uk/index.cfm?name=project_libportal
- Davies, R., "Library portals - today and tomorrow" Online Information 2004 Proceedings, 2004, 237-241 http://www.online-information.co.uk/2004proceedings/thurspm/davies_r.pdf
- Schaeck, T., Hepper, S., "Portal standards for Web services" Network World, 2002, 19(35): 33
- Moore, C., "Portal standards set for takeoff" InfoWorld, 2003, 25(30): 20
- University of Hull portal tour, http://www.hull.ac.uk/esig/portaltour/
- GetRef, http://www.edina.ac.uk/getref/
- GetCopy, http://www.edina.ac.uk/getcopy/
- EDINA National Data Centre, http://www.edina.ac.uk/
- JAFER toolkit, http://www.jafer.org/
- HEIRPORT, http://ads.ahds.ac.uk/heirport/
- Archaeology Data Service, http://ads.ahds.ac.uk/
- CREE Project Web site presentations page, http://www.hull.ac.uk/esig/cree/presentations
- Apache Pluto portlet container project, http://portals.apache.org/pluto/
- Model-View-Controller description, http://java.sun.com/blueprints/patterns/MVC-detailed.html
- Apache WSRP4J project, http://ws.apache.org/wsrp4j/
- Presentation available at http://www.hull.ac.uk/esig/cree/presentations/
- JSR 168/WSRP environment (developed by Matthew Dovey), http://www.oucs.ox.ac.uk/portal/developers/environment.xml
- Connect Embedding Project documentation, http://demo.ssl.co.uk:8080/index.html
- Description of portlet work done for the Go-Geo! Project, http://www.gogeo.ac.uk/geoPortal10/PortletInfo.html
- OASIS XML Localisation Interchange File Format (XLIFF) Technical Committee, http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=xliff
- Portlet Open Source Trading (POST) Web site, http://portlet-opensrc.sourceforge.net/