Open Peer Review & Argumentation: Loosening the Paper Chains on Journals

The emergence of the internet and the World Wide Web (WWW) have potentially profound implications for scholarly practice, particularly in the submission, review, and publication of articles in journals. However to date, much of the impact of these new technologies on journals has been on digitising the products of journal publication; the scholarly processes involved in reviewing articles remain unchanged and unsupported. We are using computer-supported collaborative argumentation (CSCA) tools to rethink and redesign the process of scholarly debate at the heart of journal reviewing. This paper describes the design principles behind our approach, how they are currently being realised in the context of a specific new journal, and discusses some of the issues that this approach raises.

Keywords: digital journals, argumentation, electronic publishing, hypermedia, WWW

The shift from paper to digital media

The emergence of the internet and electronic publishing have far reaching implications for the way in which knowledge is disseminated and sanctioned within scholarly communities, which since the Gutenberg revolution in printing has been shaped by the affordances of paper. Currently, we are witnessing the beginnings of a shift from paper to digital media in the submission, review and publication of scholarly work. In many journals, it is now standard procedure for reviews and submissions to be transmitted electronically. Sometimes, automated systems are in place for matching reviewers with submissions based on keyword analysis. In a few cases, the final version is even published electronically as a set of HTML files available on the WWW. Thus to date, this shift from paper to digital media has mainly affected the products of publication. Various documents are becoming digital and some activities are automated, but the actual process of scholarly work has not changed to a great extent.

What remains completely unsupported is the intellectual 'meat' of scholarly publication, the review process itself. This remains a private affair involving a small number of reviewers, and in most paper-based journals, the gap between initial submission and article publication can be dragged out over a year. The potential of internet technologies for opening up and revitalising the scholarly debate process has yet to be realised.

Argumentation: the heart of scholarly debate

What is the nature of intellectual work in the peer review process? Reviewing a submission involves drawing on certain criteria (e.g. theoretical content; empirical content; presentation quality; appropriateness for the publication) to evaluate the quality of reasoning and evidence provided, to probe for weaknesses, acknowledge strengths, and question background assumptions. In effect, reviewing is an argumentative process where reviewers are engaging in an imaginary debate with distant authors who are not present to respond to their analysis. This paper-based review model has shortcomings in that questions go unanswered; confusions go unclarified; criticisms go undefended. The dynamic cut-and-thrust of debate normally found in face-to-face contexts such as workshops or conferences is not supported by the paper-based review processes, nor is it yet being realised in the new electronic media.

We are currently rethinking the review process to use new technologies in order to recapture the best features of a dynamic scholarly debate. This rethinking is guided by existing research into hypertext-based, computer-supported collaborative argumentation. Argumentation research is concerned with developing notations and tools to facilitate public debate and negotiation.

The remainder of this paper will begin by reviewing previous work in argumentation and distilling important design guidelines for creating argumentation tools. Next, we will describe how these guidelines are being followed in the context of a specific new, tool-supported journal publication. Finally, we discuss some of the issues which may be encountered in the new review model and its supporting technology.

Computer-Supported Collaborative Argumentation (CSCA)

CSCA research has focused on designing notations to support debate processes [1, 3] , creating computational tools to support using the notations [4, 6, 9] and understanding the organisational contexts and work practices necessary for such notations and tools to succeed [5, 11]. Key lessons have been learned from previous research that point to design guidelines in each of these three areas:

• Lesson 1: Avoid elaborate and rigid notations. Much research has focused on finding the "right" notation to support debate. Often, the resulting notations require people to express their thoughts using elaborate sets of provided distinctions such as positions, issues, comments, pros, and cons. This, however, runs the risk of burdening people with excessive representational overhead by forcing them to categorise and commit their thoughts to rigid notations before they are ready. Empirical findings indicate that people are often unwilling and sometimes even unable to do this [1, 2, 10].
   
• Lesson 2: Computational tools must integrate argumentation with the artifacts being discussed. Early Csca approaches separated the argumentation from other artifacts (i.e., the papers and drawings being discussed). This separation hindered users from (i) quickly accessing relevant argumentation when it was most needed during problem-solving, and (ii) easily adding new argumentation. Later work has shown that tools must support users to bridge the separation and move seamlessly in both directions between the artifact and argumentation [4, 8].
   
• Lesson 3: Work practices must be redesigned so that argumentation is integral to the task being performed. Empirical studies indicate that people often do not contribute to argumentation because it is perceived as extra work over and above what they are already required to do [5]. Successful argumentation approaches have redesigned work practices to make contributing to argumentation integral to the overall task being performed [11].

A CSCA environment for scholarly debate

Our goal is to develop a CSCA environment for scholarly review of journal submissions which takes into account these three lessons. This environment is being developed to support the reviewing process of a new scholarly journal, the Journal for Interactive Media in Education (JIME).

The JIME Web environment has been carefully designed in accordance with three design principles deriving from the three lessons introduced above.

Principle 1: Begin with a simple argumentative notation

The notation used to support reviewing in JIME has one primary class of contribution called comments. Optionally, debaters may choose to categorise comments as agreements (flagged as ) or disagreements (). The power of the Web means that reviewers may elect to include pointers to evidence in their comments; evidence can be either plain text (e.g., "See Conklin & Begeman, 1988") or a Web hyperlink direct to an online source. Thus, a body of secondary evidence may grow around an article under review, providing a valuable resource for both readers and authors.

Principle 2: Integrate the publication with the argumentation

Prior to publishing for open debate, JIME submissions are marked-up to include special argumentation tags (). Each major document section and the interactive demonstrations are prefaced with these navigation tags. Clicking on one takes the reader to the argumentation about that document section, in the Review Window. Conversely, from the Review Window, debaters can easily switch back to any part of the publication. Figure 1 shows the user interface for reading an article in the JIME Web environment. Reviewers can also download an Acrobat version since we believe that there is still a strong preference for being able to read a paper version. Thus, a reader/reviewer has two main 'work spaces' in JIME - the Article Window for reading, and the Reviews Window for making review comments, which are automatically opened and brought to the front as needed.

Figure 1: Reading an article in JIME. In the lower left frame, the Comment icon takes the reader to the top of associated Review Window (see Figure 3 below). The printer icon allows the reader to download an Adobe Acrobat version for printing. In the main frame, the Comment icon takes the reader to the review comments about this figure and associated demonstration. The Demo icon takes the reader to a Macromedia Shockwave demo of the system (see Figure 2).

Figures 2-4 show screens for trying out a demonstration (Fig. 2), displaying the Review Debate about an article (Fig. 3), and reading Reviews about a particular section of the article (Fig. 4). The Review argumentation is added and manipulated using an environment which we have created by tailoring the NCSA HyperNews system [7].

Figure 2: Authors describing a new system must submit demonstrations of it using an appropriate technology. The above example shows an interactive demo using the Macromedia 'Shockwave' plug-in. This allows readers and reviewers to interact with the system described in the article, as though they were students using the original application.

Figure 3: Overview of a review discussion in JIME. The example shows the outline view with headings for general categories of comment (e.g. Clarity of Goals; Credibility of Results), followed by headings which match the headings in the article. Section-sepcific comments are placed under these.

Figure 4: Details of a Review discussion in JIME. The example shows review comments about a specific section in the article (3.3).

Principle 3: Make argumentation integral to the review process

The proposed review model for JIME submissions is shown in Figure 5. This model changes the role of participants in the process to directly involve generating and manipulating argumentation. Reviewers return their comments to the editor in the JIME argumentative format. The editor pulls together all the reviews to seed the argumentative debate. The publisher marks-up the publication and the initial argumentation to include the special cross-tool navigation tags. The article under review and the reviewers' initial comments are then published on the Web, and the review process moves into a phase of open peer review, in which authors, reviewers and readers can engage in debate. The editor then decides whether the article should be accepted, and formulates change requirements for the authors. We also wish to allow for the possibility that interesing discussion threads may arise during the review process which could be distilled into commentaries for publication with the final article.

Figure 5: Lifecycle of a JIME article under review.

Once the article is published, debate about it will continue. Wth conventional journals, it is only at this point that debate would begin, perhaps a year or more after initial submission. Even then, the debate is not tightly linked to the article, but located in other articles, often published in other journals, at substantially later times. In JIME, discussion can continue directly linked to the published article, or via external links to related articles at other sites.

To summarise our design philosophy:

• JIME is not simply making the conventional media of paper-based articles available on the Net, but addressing the interactive dimensions which are the essence of the new media;

   

• JIME is not simply adding an e-mail listserver to allow peripheral discussion about published articles, but developing a journal review environment in which submitted articles and open peer review are tightly integrated with each other, and central to the journal's operation.

Discussion

This project is still in its early design stages and the details of the current interface will undoubtedly change as it undergoes further implementation and user testing. However, we feel it is important to discuss work such as this in its early stages in order to encourage more participation and dialogue between scholars concerning future publishing practices. We believe the review model presented here offers several benefits for opening up and revitalising scholarly debate. However, there remain many open issues to be resolved.

Some issues relate to the user interface for supporting collaborative, Web-based argumentation. For instance, the Web introduces new concurrency challenges regarding timely updating of reviewers' screens with new contributions to the discussion. Additionally, Web-based systems offer impoverished degrees of interactivity compared to what we now expect from direct manipulation interfaces, limiting our ability to take advantage of new developments in graphical argumentation tools.

Another issue raised by our proposed model is the perceived risk which authors may feel when exposing publications to large audiences at the review stage. Submissions will be critiqued by many more reviewers than is currently possible in the conventional review process, and this will take place in a public rather than private forum. The willingness of authors to do this depends greatly upon the professionalism and netiquette exhibited by reviewers. The philosophy behind this model is that perceived risks of this sort will be outweighed by the benefit to authors of quicker, more extensive, and more relevant feedback, resulting ultimately in higher quality publications.

To conclude this brief article, we should not be surprised that electronic journals meet with resistance from some quarters, since they force to the surface huge issues which entail rethinking how scholarly knowledge is disseminated and sanctioned. This is the time to reflect radically and creatively on the 'papyrocentric' scholarly practices which have until recently been taken for granted. JIME is pushing the boundaries of electronic journals one step further with its Web-based open peer review. This seems to us the logical direction in which to take journals. We await your reactions with interest.

References

1. Buckingham Shum, S., "Analyzing the Usability of a Design Rationale Notation," in Design Rationale: Concepts, Techniques, and Use, T. P. Moran and J. M. Carroll, Ed., LEA, Hillsdale, NJ, 1996, pp. 185-215. (Abstract :: Preprint - postscript :: Request reprint)
2. Buckingham Shum, S. and N. Hammond, "Argumentation-Based Design Rationale: What Use at What Cost?," International Journal of Human-Computer Studies, Vol. 40, pp. 603-652, 1994. (Abstract :: Preprint - postscript :: Request reprint)
3. Conklin, J. and M. L. Begeman, "gIBIS: A Hypertext Tool for Exploratory Policy Discussion," ACM Transactions on Office Information Systems, Vol. 6, pp. 303-331, 1988.
4. Fischer, G., A. C. Lemke, R. McCall and A. I. Morch, "Making Argumentation Serve Design," Human-Computer Interaction, Vol. 6, pp. 393-419, 1991.
5. Grudin, J., "Evaluating Opportunities for Design Capture," in Design Rationale: Concepts, Techniques, and Use, T. P. Moran and J. M. Carroll, Ed., LEA, Hillsdale, NJ, 1996, pp. 453-470.
6. Hair, D. C., "LEGALESE: A Legal Argumentation Tool," ACM SIGCHI Bulletin, Vol. 23, pp. 71-74, 1991.
7. LaLiberte, D., "HyperNews,". National Center for Supercomputing Applications, University of Illinois at Urbana-Champaign.
8. McKerlie, D. and A. MacLean, "Reasoning with Design Rationale: Practical Experience with Design Space Analysis," Design Studies, Vol. 15, pp. 214-226, 1994.
9. Schuler, W. and J. Smith, "Author's Argumentation Assistant (AAA): A Hypertext-Based Authoring Tool for Argumentative Texts," in Hypertext: Concepts, Systems and Applications, A. Rizk, N. Streitz and J. André, Ed., Cambridge University Press, Cambridge, 1990.
10. Shipman, F. and R. McCall, "Supporting Knowledge-Base Evolution with Incremental Formalization," Proceedings of ACM CHI'94, Boston, Mass., 1994, pp. 285-291.
11. Terveen, L. G., P. G. Selfridge and M. D. Long, "From "Folklore" to "Living Design Memory"," in Proceedings of ACM/IFIP INTERCHI'93, 1993, pp. 15-22.

The Journal of Interactive Media in Education is at: http://www-jime.open.ac.uk/jime/

Open peer review of the first submission to JIME begins on 2nd Sept.'96. We invite all Ariadne readers to participate and leave comments, either on the article itself if this is an area in which you are qualified, or on the journal's design more broadly.

An earlier version of this article was presented at HCI'96, Annual Conference of the British Computer Society on Human-Computer Interaction, London, 20-23 August, 1996.