Towards an Application Profile for Images
Following on from the project to develop an application profile for scholarly works (SWAP)[1], the Joint Information Systems Committee (JISC) has recently funded through its Repositories and Preservation Programme, a series of projects to establish Application Profiles in the areas of images, time-based media, geospatial data and learning objects [2].
The work on the Images Application Profile (IAP) has been carried out for the six-month period from September 2007 to March 2008, and while the substantive project work is now complete and a draft Images Application Profile is in circulation, the ongoing job of promoting the profile to, and consulting with, the image, repository and metadata communities continues. To this end, JISC has funded a one-year post, based with the Technical Advisory Service for Images (TASI), to promote the IAP and work towards community acceptance.
The first task of the IAP project was to bring together a working group with representatives from a range of backgrounds comprising image experts, repository developers and information specialists [3]. The group met for a day in late October 2007. It has commented subsequently on various iterations of the project as it has advanced through an email discussion list and some further occasional one-to-one meetings and exchanges. As the project nears completion, it is our intention to open the discussion on the IAP to a wider Consultation Group. The core deliverables of the IAP project were: a set of functional requirements based on a set of defined user needs; a conceptual model; an Images Application Profile; and a set of easy-to-follow user guidelines. At time of writing most of these deliverables, some in draft form, can be downloaded from the project's wiki page [4].
This article provides an overview of the various issues concerned with the development of the IAP to date, concentrating on the particular challenges presented by images and image metadata, the potential use of the Functional Requirements for Bibliographic Records (FRBR) concept model for images, and finishes with an outline of future plans as regards community involvement and ultimately acceptance of the profile. It also points to some areas of partial resolution that may require further research to ensure a watertight profile for images, that meets the needs of all image users and image providers across all academic domains.
The Challenge of Images
Institutional Repositories have to date concentrated most of their efforts in storing and managing text-based research outputs. Finding text-based digital data through repository and other database systems is relatively straightforward compared to finding image-based data, in that the text's content can be processed automatically by machines, to help the end-user arrive at the meaning and context of the file where human-generated descriptions are partial or lacking. Images are not self-describing in this way and hold nothing intrinsically within them that can be extracted and used by machines to aid description other than low- level pixel data. Work is ongoing in various content-based image retrieval (CBIR) software developments and related endeavours, which use this pixel data to infer similar objects, similar textures and similar colour within images, and may offer partial solutions to providing some automated description of images and their content in the long term. However, human- generated descriptions of images are likely to remain the primary method of describing the often complex semantics that can be associated with any given image.
This 'semantic gap' - the distance between what descriptions can be extracted by machines, and what humans can provide and need - is a common feature of discussions about digital images and their retrievability within software systems [5]. Of course, the problem is further compounded for the image end-user because human-generated text-based descriptions of images - limited by the knowledge, culture, experience and point of view of the annotator - can only ever partially capture the information, meaning and context of an image in a given setting. As the education community moves closer to articulating an ideal of providing seamless searching across educational and cultural image collections [6], and as institutional repositories begin to move forward from storing and sharing mainly text-based objects to embrace images and other complex digital objects, this information gap is becoming increasingly evident.
Also digital images often require at least three levels of description; technical information relating to the image; the content depicted in the image; and possibly more abstract meaning and interpretation. The technical description of the digital image itself is relatively straightforward for the IAP. This information is generally unambiguous and can in some cases be automated using device-generated EXIF (Exchangeable image file format) or IPTC (International Press Telecommunications Council) tags, or, increasingly with newer file formats, XML-based embedded descriptors. However this sort of metadata must become more complex if an institutional repository is to take seriously the long-term preservation of the digital images it holds, perhaps based upon the open archival information system (OAIS) model [7]. More detailed technical metadata, including format migration, versioning and change history information, will need to be provided if this is the case.
Descriptions of image content and abstract concepts are possibly more challenging still. Descriptions of image content can be quite straightforward; for instance a digital image depicting the Eiffel Tower. However, the Eiffel Tower could also reflect 'nation', or 'belonging' which are legitimate concepts to be searched on, and difficult to define in a standard way.
In addition to the complexity inherent in describing them, images also present a challenge in that they often have relationships with, or appear within, other objects. They could be derived from slides, books, manuscripts, or photographs, or they could appear in lesson plans, course booklets, presentations and lectures. All of which may need to be articulated in some way within the record, and all of which could be stored in or referenced by the repository. These relationships with source and containing objects will become even more relevant to institutional repositories if they decide in future to store institutional slide collections or the outcomes of other large-scale digitisation initiatives based on local archives and other historical documents.
Similarly, some images may have relationships to other images, for example an image made up from composite layers of other images placed on top of each other, each of which could require separate description, or have various ownership and rights issues associated with them.
A further key challenge in developing an application profile for images was to take into account the wide variety of academic domains from which images form a core component, from medical and scientific disciplines to the arts and humanities. It was necessary to identify common descriptors and terms that conveyed appropriate meaning and had applicability across a wide spectrum that incorporated all the subject areas; this involved avoiding a 'lowest common denominator' approach, and balancing the need to convey as much richness in data exchange as possible without being overly complicated.
Scope and Definition
The extent to which these issues presented challenges to the development of the IAP depended much on where we drew boundaries in the project's scope, and also on how we defined the term 'image' when modelling image data. The remit of the IAP project was to create a Dublin Core-based application profile for images that are stored in academic institutional repositories, with the aim of facilitating an exchange of image metadata between those repositories and other aggregators of data. Thereafter, our intention was to develop a DC-based mechanism, but with image-centric add-ons that facilitated an exchange of image metadata between repositories in as meaningful a way as possible; but without it being unnecessarily complicated to use in terms of implementation by repository staff on the one hand, or burdensome to academic depositors or end-users on the other.
The scope of the project also had to consider the range of image collections that repositories hold or plan to hold in future. Such a range could include among others:
- The outcomes of individual research and artistic endeavour
- Institutional photographic and slide collections
- Institutional archive and museum image collections
- Institutional teaching and research collections
In the first instance repositories probably aim to begin storing images related to the outcomes of individual research and artistic endeavour, not least on the back of some current image and multimedia-centric repository start-up projects currently in progress [8]. However in the longer term there is likely to be an institutional requirement to manage the outcomes of all forms of image digitisation including larger archive, museum, teaching and slide collections. We therefore took the opinion that the IAP should endeavour to keep the sharing of metadata relating to all these types of image data collections in project scope.
In terms of defining the term image for the project, we decided on a format-based approach that would include any flat two-dimensional digital image format. This omits paintings, photographic prints, slides and anything which is three dimensional or analogue. As such these are objects depicted within or sources for the digital image but not the image itself. This definition suggests that what end-users will primarily be searching for through repositories is a 'digital image of something', the something being either: an object, or an event, an abstract concept, or a place. For us this format-based approach to definition had the advantage of clearly marking the lines between digital images and their content and sources, and allowed us to make clear decisions on what were describing in our various profile entities
We acknowledge that there may be some issues as regards certain file formats such as PDF, which, although technically an image format, is often treated as text. This could also be true of any image format which has been used to capture a text with the intention of it being 'read' by an end-user as a document. However, we feel these issues can be addressed at the local repository level and were therefore out of scope. Also, what we were intending to model in the IAP was primarily two-dimensional Raster Images. Vector and other three-dimensional image types may also be able to use the IAP, but this would require further research into potentially other specific metadata needs, and thus out of the scope of this project.
The Model
Following on from a recommendation by the Working Group, our first task was to examine the SWAP use of the FRBR model to describe texts, and assess its applicability to image data. Essentially FRBR is a means of modeling the structure and relationships that exist in bibliographic records [9]. It does this by providing a precise vocabulary to describe bibliographic entities, centred on what are known as Group 1 Entities, namely Work, Expression, Manifestation and Item. Work is an abstract notion of a distinct intellectual or artistic creation, which is realised through an Expression – another abstract entity, which in turn is embodied in a physical Manifestation, which is exemplified as an Item. To take the example of a text, the book Playback by Ronald Hayman would be modelled as follows [10]:
w¹ Playback by Ronald Hayman
e¹ the authors manuscript text edited for publication
m¹ the book published in 1973 by Davis-Poynter
i¹ copy autographed by the Author
Subsequent editions of the same manuscript text would become new Manifestations of the same Expression, and other particular copies on library shelves would become other items.
Although an output from the bibliographic world, FRBR is intended to be capable of modelling all library holdings, including images [11]. Two initial questions emerged when applying FRBR to images: do the Group 1 entities Work, Expression, Manifestation and Item (in SWAP renamed Copy) suit images, particularly as regards the use of abstract notions for Work and Expression; and, can FRBR be used successfully to describe images, given the various challenges outlined above, regarding what is being described, and relationships with other objects? We concluded that FRBR could be used quite successfully to model some image types, particularly those that are the product of an artistic or intellectual process. For example, using the FRBR Group 1 Entities and their relationships as described in SWAP:
Figure 1: FRBR Group 1 Entities as defined in SWAP
a piece of sculpture could be described thus:
w¹ The Angel of the North
e¹ Sculpture
m¹ The Physical Sculpture
e² Digital Still Image
m¹ TIFF Format
m² JPEG 2000 Format
e³ Analogue Image
m¹ Photographic Print
m² Slide
However, after much investigation and consultation, it was decided ultimately that FRBR did not address our requirements for the IAP. In essence what is being done by FRBR is not the modelling of the simple image and its relationships, but rather an attempt to model the artistic / intellectual process and all resultant manifestations of it. We decided this was inappropriate for the IAP for a number of reasons. While possible, an application profile of this complexity would require detailed explanation that could be a barrier to take-up. Moreover, it strays from the core remit of the images IAP to facilitate a simple exchange of image data between repositories. While the FRBR approach attempts to build relationships between objects, e.g. slides, photographs, objects and digital surrogates, this facility already exists in, for example, the Visual Resources Association Core [12] (VRA) schema. Our intention was not to reinvent or in any way replicate existing standards that are robust and heavyweight enough to deal with most image types. Rather our intention was to build a lightweight layer that could sit above these standards, and work with them, facilitating a simple image search across institutional repositories.
Figure 2: Images Application Profile Conceptual Model
In our model, we have renamed the FRBR Work entity as 'Image' for reasons of clarity, mainly to avoid confusion between notions of Work as described traditionally in image cataloguing in the cultural sector (i.e. the physical thing) [13] and abstract Work as described in FRBR. As noted above, image as defined in the IAP is a digital image, in line with our notion of end-users searching repositories for digital images of something. Therefore our conceptual model - while still using the language of FRBR and using the areas of SWAP that have applicability across the text and image domains - places the digital image at its centre.
One area where we had particular concerns with FRBR for images was in the notion of an abstract Expression layer in the model. This entity, while very useful in describing the various intellectual and artistic realisations of textual, musical and performance works, was not, it seemed to us, as useful in describing images, as defined in our project scope. In FRBR the Expression points to 'intellectual' differences, which are not always apparent to the same extent in, for instance, a format change in an image. To this end we have omitted this entity from the IAP model.
The IAP Model also makes use of the FRBR Group 3 Entities: Concept (an abstract notion or idea), Object (a material thing), Event (an action or occurrence) or Place (location), which will facilitate simple keyword searching of images. We also added to the FRBR recommendations for subject a Person Entity that we felt was necessary to express either the creator of the object depicted in the image, or a person who appears in the image. This Person information is distinct from FRBR Group 2 Entities which we have followed SWAP in consolidating as a single Agent. In the IAP Agent is typically a funder, associated institution, or the creator of an image Manifestation. Similarly the IAP follows SWAP in its use of Copy instead of FRBR Item. Here a Copy is typically a network location of a particular image Manifestation.
In natural language what this model says is that each Image can have one or more Manifestations, and each Manifestation can be made available as one or more Copies. Each Image can have one or more Subjects, Objects, Events, Places or People associated with it. Each Image can have one or more funders and affiliated institutions and each funder and affiliated institution can be associated with one or more Image. And each Manifestation can have one or more Creator, and each Creator can be associated with one or more Manifestations.
To use the Angel of the North example above in the IAP Model:
image¹ An image depicting The Angel of the North
object¹ Sculpture
object¹ The Angel of the North
place¹ Gateshead
person¹ Gormly, Anthony
manifestation¹ image in TIFF Format
Attributes
The attributes we use to describe each of our entities are summarised below. In addition to Dublin Core and FRBR, the other metadata schema standards we looked at that have particular application for images were VRA, Catalogue Descriptions for Works of Art (CDWA) and Metadata for Images in XML (MIX / NISO Z39.87). In the end we used a combination of MIX for technical information relating to the various manifestations, DC and the FRBR subject entities as described above. We also looked at EXIF and IPTC standards and their recent developments, and recommend that, where possible, repositories enable automated population of the technical fields using them.
For more details on the attributes, there is a work in progress on the project's wiki page with definitions and examples [14]. As well as the standard fields derived from FRBR, DC and MIX, we introduced an 'isImageOf' property of Image which is intended, via an identifier such as a URI, to point to a fuller local record where available, that describes the image and its context in more detail. Also, the isPartOf property of Image is intended to point to the larger image collection in which the single image belongs. The use of these properties is intended to facilitate the inclusion of more detailed information about the images, their context, objects and relationships while keeping the Profile relatively uncomplicated. Two further new properties that came out of our use case analysis were 'relatedCourse' property of Manifestation, designed to include information about related courses where the image manifestation appears and reproductionCost as a property of Copy, which will give the end- user a cost price for various types of publication. We have yet to make a final decision as to whether these attributes remain in project scope. One idea is to relate the course information property to the work going on in the XCRI project [15], but this will require some further research. The reproductionCost attribute seems appropriate if institutions intend to generate revenue from images published in books and elsewhere.
The attributes of each Entity Type are as follows:
Image
Identifier
entityType
hasAsSubject
isImageOf
isFundedBy
isPartOf
Title
Description
Subject
Grant Number
isManifestAs
Manifestation
Identifier
entityType
isCreatedBy
Format
fileSize
imageHeight
imageWidth
dateCreation
colorSpace
Resolution
relatedCourse
isAvailableAs
Copy
Identifier
entityType
accessRights
rightsHolder
Rights
dateAvailable
reproductionCost
Agent
entityType
Name
familyName
givenName
Homepage
Subject Object
entityType
Object
Subject Place
entityType
Place
Subject Concept
entityType
Concept
Subject Event
entityType
Event
Subject Person
entityType
Person
Community Acceptance
As noted above, JISC is funding a one-year post to help take the profile out to the community and engage with relevant stakeholders. Our remit in this development phase of the project is to put together a plan for this work, and to highlight areas that can be usefully researched further with the ultimate aim of achieving a fully implemented IAP. The IAP development has been hampered slightly in that it has yet to be tested with institutional repository software or in a working repository setting. Testing and further discussion with repository software developers were impossible to schedule in the short six-month period of the project, but we hope in the coming year to engage more fully in this area. Similarly a key aim of the IAP Project has been to put a mechanism in place that would work with the Repositories Search Project [16] which intends to include image searching within its remit. Some practical testing of image searching in this context will be imperative in the coming months. The IAP will also need to be introduced to a wider community of image providers, across all academic and cultural domains, if it is to be truly embracing of all image types and uses. The project team had some practical experience in providing disparate collections of images to education through our work at VADS [17], but we would recommend further detailed consultation. We intend to put some of the formal mechanisms of this discussion in place in the next month as we invite key people and organisations to be a part of the Consultation Group. Now that the IAP is in place we also hope to begin to engage more fully with the DCMI [18] and other organisations with interests in image metadata and application profiles. Finally it is our hope to continue to engage closely with the other application profiles being developed as part of this Repositories strand, and exploit common areas which we can use to maximise the combined chances of all the application profiles being accepted by all of the relevant communities.
Conclusion
The development of an IAP threw up issues and complexities unique to this visual format which had not been assessed fully by the repositories community thus far. We hope that this article has outlined these comprehensively and pointed to a possible way forward in the development of a profile for images that will facilitate a rigorous enough means of searching across repositories of image data. There remain some areas where further research may prove useful, and as the initial six-month development phase of the project ends, and the longer job of promoting the profile begins, we hope these areas will be explored in more detail. We look forward to more debate in the coming months as engagement with the wider community begins in earnest and the profile moves closer to acceptance by repository developers, managers and the users of images.
References
- EPrints Application Profile Wiki http://www.ukoln.ac.uk/repositories/digirep/index/Eprints_Application_Profile
- The JISC's Repositories & Preservation Programme Web page http://www.jisc.ac.uk/whatwedo/programmes/programme_rep_pres/
- Images Application Profile Working Group members on the IAP Wiki http://www.ukoln.ac.uk/repositories/digirep/index/Images_Application_Profile#Working_Group
- Images Application Profile Wiki http://www.ukoln.ac.uk/repositories/digirep/index/Images_Application_Profile
- For a good discussion of the Semantic Gap in image retrieval see Jorgensen, Corinne, "Image Access, the Semantic Gap, and Social Tagging as a Paradigm Shift", Classification Research Workshop, 2007, available from the Digital Library of Information Science & Technology Web site http://dlist.sir.arizona.edu/2064/
- See the Defining Image Access Project Wiki at http://imageweb.zoo.ox.ac.uk/wiki/index.php/Defining_Image_Access and the JISC Images Working Group's "Digital Images in Education: Realising the Vision" on the JISC Collections Web site http://www.jisc-collections.ac.uk/catalogue/images_book for recent thoughts on sharing image resources.
- See article on the Open Archival Information System on the Repositories Research Team Wiki http://www.ukoln.ac.uk/repositories/digirep/index/OAIS
- Section on multimedia & images, on the JISC's Areas of Common Interest for Start Up and Enhancement Projects Web page http://www.jisc.ac.uk/whatwedo/programmes/programme_rep_pres/repositories_sue/suethemes.aspx#multimedia
- Tillet, Barbara, "What is FRBR? A Conceptual Model for the Bibliographic Universe", Library of Congress Cataloguing Distribution Service, February 2004 http://www.loc.gov/cds/downloads/FRBR.PDF
- "Functional Requirements for Bibliographic Records, Final Report", International Federation of Library Associations and Institutions UBCIM Publications – New Series Vol 19, 1998, p.23 http://www.ifla.org/VII/s13/frbr/frbr.pdf
- Ibid. pp7-8
- Visual Resources Association Web site http://www.vraweb.org
- See "Cataloguing Cultural Objects: A Guide to Describing Cultural Works and their Images" on the VRA Web site http://vraweb.org/ccoweb/cco/index.html
- Images Application Profile Wiki http://www.ukoln.ac.uk/repositories/digirep/index/Images_Application_Profile
- The XCRI: eXchange of Course-Related Information Web site http://xcri.org/Welcome.html
- Intute's Repository Search Web site http://www.intute.ac.uk/irs/
- VADS Web site http://www.vads.ac.uk
- Dublin Core Metadata Initiative (DCMI) http://dublincore.org/