The Performance-based Funding Model: Creating New Research Databases in Sweden and Norway

The introduction of Performance-based Research Funding Systems (PRFS) models has helped to set the focus on scientific publishing since this is one of the major indicators for measuring research output. As a secondary result, it has also forced the countries that have introduced a model that is not solely based on citations to create a new form of research database; the national portal for scientific literature.

Even in countries that have not adapted a PRFS model, these forms of portals are common. In Sweden, the national database has been the result of local universities’ and Higher Education institutions’ efforts towards a mutual search interface to the different local repositories that have emerged.  The repositories are sometimes created to support a local funding model, but the most common reason is simply to disseminate scientific publishing.

The contribution of publications that cannot be found in the international citation indexes are substantial even for peer-reviewed publications especially for literature in the Humanities and the Social Sciences but also in Natural Sciences by adding a lot of conference papers that would otherwise not be recognised.  This can be seen as a side-effect of the discussions on the introduction of PRFS models. Even if not all publications meet the criteria to be recognised as scientific, a significant number of them are still added to the base of publications that can be analysed from a PRFS perspective.

The incentive to disseminate search results is the other driving force for the national catalogues and, in this case, the portals are playing an important role since many of the publications mainly written in domestic languages are otherwise seldom visible to other researchers and the public.

Background

The change in the Higher Education landscape has been significant over the last decades. One of the major changes has been the introduction of performance-based research funding systems (PRFS) starting with the Research Assessment Exercise (RAE) in the United Kingdom in 1986 [1].

The incentive offered by a PRFS model is that it overturns the allocation of funding often based on historical grounds which are no longer valid. It can also be considered as a part of an orientation towards more effective management of the public sector, based on principles derived from the private sector. The central idea in this transformation is the drive to increase productivity without adding resources to the system, while replacing traditional command-and-control systems with market-style incentives [2].

Publications are playing a more and more important role in the process of evaluating research and also as an indicator in the performance-based research funding systems (PRFS). Of 14 countries which were identified as having PRFSs, at least 10 have publication output as an indicator of research quality [3].

The reason why publications are used as a PRFS indicator is that the latter measures the output of research both quantitatively and qualitatively. Moreover, those publication data are easily found in bibliographic databases.  However, the introduction of an indicator for publications has not been without controversy in the countries that have adopted PRFS. The resulting discussions have generally focused on two main issues:

1. Should publications be used as a first-order or a second-order indicator?
2. Which data source would best reflect the research output in a particular country?

OECD launched a report on various PRFS models in 2010 including a systematic exposition of the different indicators being used [1]. Publications are often found among the first-order indicators which are aimed directly at measuring research performance. This can be done by simply counting publications in relation to the number of research staff of a certain institution but the common practice is to add a qualitative element, either by ranking of publication channels as occurs in Denmark and Norway, or by citation analysis as operates in Belgium, that is, in the Flemish community, and Sweden.

Second-order indicators are based on indexes instead of direct measurements. These indexes are often created as a reaction to the methodological difficulties when dealing with citation analysis. The best known indexes are Journal Impact Factor (JIF) which measures a journal’s impact through the number of citations its articles receive, and H-index which is often used on an individual level when comparing the citation history of different authors. However, both indicators are simplifications of more complex structures and have been criticised when being used for research assessment purposes [4].

The second question has created even more controversy in those countries which have adopted a PRFS model. When deciding on which database would best support the publication indicator, the issue as to which type of publications should be counted immediately comes to the fore.

The main issue is whether the alternative forms of publishing that are often found in the Humanities and the Social Sciences (including books, anthologies, etc) should be included in the PRFS model, or if it should rely exclusively on peer-reviewed articles. If the first approach is adopted, as in Denmark and Norway, the existing public or commercial databases are often found to be insufficient since a lot of the literature from the Humanities and Social Sciences is missing from these databases. This means that national databases supporting the PRFS model have to be implemented [5].

On the other hand, if you decide not to include other publishing forms, and rely on citation databases such as Web of Science, adjustments have to be made in the form of weighting publications from different research areas differently (as in Sweden) or using second-order indicators like JIF to balance the different sectors (Flemish Community, Belgium) [6].

Traditionally the main purpose of bibliographic databases has been to disseminate research results. The databases can be multi-disciplinary and cover research publications from all over the world (eg Web of Science, Scopus, WorldCat, etc). Furthermore, there are also subject-specific databases that cover larger areas (eg Inspec, PubMed). Another form of bibliographic database is the library catalogue, which is often national in scope and based on holdings from different scientific libraries.

The PRFS models including SSH (Social Sciences and Humanities) literature have however introduced a new form of bibliographic database in the countries where the model has been adopted (Denmark, Finland, the Flemish community in Belgium, and Norway). It is a national bibliographic database of scientific publishing which has as its main purpose the support of the PRFS model but which also serves as a conventional bibliographic database, providing access to contemporary research. Moreover, Sweden has adopted such a national database, but with the difference that it is primarily a bibliographic research tool and not part, at this juncture, of the PRFS model.

Systems Employed in Norway and Sweden

In this article I shall examine two examples of national databases either used for or proposed for PRFS purposes, the Frida/CRIStin in Norway and SwePub in Sweden. These are databases or portals that were created from different perspectives. The Norwegian system was designed directly to support the publication indicator in the PRFS model, while the Swedish system started out with a conventional bibliographic rationale. Since the Norwegian system has been object of many studies [7][8], I shall focus in this study on the Swedish system, SwePub. I shall see if the different approaches to creating a national portal for scientific literature in Sweden have any implications for the way in which literature is included in the database, as compared to its Norwegian counterpart. 

Furthermore, I shall also try to analyse the content of the Swedish national portal to see if it has the potential to provide structured and validated data for a PRFS model, especially in respect of concerns relating to SSH literature.

Finally we will also study the role each database plays in disseminating scientific literature in the two countries and how the localisation of their portals helps to achieve this effectively.

Norway: Frida/CRISTin

Norway implemented a performance-based funding system for research in 2002. Publications form one of the two output indicators, the other is the number of PhD graduates. Both indicators are worth 30% each of the total model. There are also two input indicators, external funding from Norwegian Research Council and external funding from the EU, each worth 20% [1].

The publication indicator is based on a model which includes books (ISBN-titles), contributions in anthologies (papers in ISBN-titles) and articles from peer-reviewed journals (papers in ISSN-titles). Some conference papers are also included in the model. The criteria for a publication to be regarded as scientific is that it provides a new insight and appears in such a form that makes its results examinable. Furthermore, it should be written in a language and distributed in a way that it makes it accessible to the research community. Finally it should also be peer-reviewed [9].

For each publication type, there is an evaluation of the different publication channels, eg publishers for books and anthologies and journals for articles. The publication channels are divided into two levels, level 1 which gives a lower score and the more prestigious level 2 which gives a higher score. Not more than 20% of the total number of publications can appear on level 2. These channels are continuously evaluated by field-specific research committees.

To support the model, a national system for research documentation has been implemented by the Norwegian Association of Higher Education Institutions (UHR). The system includes a central function (felles tjeneste vid Norsk samfunnvitenskaplige datatjeneste, NSD ) which delivers bibliographic data about scientific publishing and supplies an authority index to support the Higher Education institutions’ own systems (Frida/Forskdok). The bibliographic data comes from Web of Science, the national library system Bibsys and Norart, which is a portal for Norwegian and other Nordic articles. The data are collected in a common database, ITAR, from which researchers can import records into their own local system and verify their own publications [10]. It is also possible to register records manually into the local system. The records are controlled by special administrators, ‘superbrukare’ on different levels in an institution such as department, faculty, etc. A statistical report is then sent from  Frida/Forskdok to Database for statistikk om høgre utdanning (DBH) which then calculates the publication points which becomes the data upon which the budget is based.

Figure 1: Norwegian CRIStin Web site

In 2011, a new system called CRIStin was implemented to replace Frida/Forskdok. In addition to the previous system it also includes publications that are not defined as scientific in the criteria mentioned above, reports, for example, popular science, lectures, etc. Furthermore, patents and products deriving from Norwegian research are also registered there. The part of the system which contains documented results from research activity is called Norsk vitenskapsindeks (NVI).

The CRIStin system also contains researcher profiles, information about research projects and research units. The system involves about 160 institutions as providers of information. All information in the portal is open to public search.

Sweden: SwePub

If the Norwegian system was created from a top-down perspective, its Swedish counterpart has evolved from exactly the opposite direction.

Sweden did not adapt a PRFS model until 2009 and decided to focus on two indicators only, external funding and publications, both indicators sharing equal weight in the funding [11]. With regard to the publication indicator, Sweden decided to use Web of Science as the data source in spite of the fact that uneven coverage between research areas as well as a bias towards literature written in English is well known [12][13].

The bibliometric indicator itself is rather complex, trying to estimate the productivity of Swedish researchers in different research areas combined with field-normalised citations [14]. In order to compensate for the uneven distribution between research areas in Web of Science, the Ministry of Education introduced different weighting for different disciplinary domains, offering the highest weight to humanities and social sciences. The model has met with criticism from both researchers [15] and the Swedish Research Council and the Ministry has decided that the potential of using peer review should be investigated as a possible means, at least partly, of replacing the present PRFS model [11].

In spite of the relatively late interest shown in using publications as a measurement of research output on a national level, there has been an ongoing trend to include publications in different activities at local universities in Sweden. Uppsala University was first to have a research assessment exercise as early as 1999 (Bastu) followed by further exercises in 2007 (KoF 07) and 2011 (KoF 11). These assessments combined conventional peer review with bibliometric indicators and in order to guarantee that all publications were part of the basic data, a local publication database was created. Later this was merged with the digital publishing system, DiVA, which now is in use in some 30 universities and other Higher Education institutions. Other universities have undertaken similar exercises which together with the aim of collecting publication data for other evaluation purposes, including local PRFS models, have helped to support the call to create local systems for publishing data.

Another trend that has been visible during the last decade is the desire to disseminate research results as much as possible. New technology and new publishing trends, including open access publishing has made it possible to simplify the publishing process and has helped to create the “digital revolution” in scientific publishing [16][17].

This has also increased the usage of publication databases both in the sense that they often are imbedded in a local repository together with a digital publishing tool like the example of DiVA at Uppsala University. In certain fields, especially in the humanities and the social sciences, these publications can often only be found in the local publication database.

Figure 2: SwePub Web site

SwePub is an aggregation of these local publication databases starting in 2007. It includes records from most of the Higher Education institutions in Sweden, although some big universities are still missing, such as the Karolinska Institute and the Swedish University of Agriculture Sciences. Moreover, coverage from the different universities has also been uneven, but in 2010 75% of the universities stated that the coverage was sufficient [15]. 75% of the universities also reported that they operated some kind of quality control including control of affiliations. Due to the different approaches to quality control adopted by Sweden’s HEIs, together with their uneven provision of extra information like subject indexing, SwePub can only be partially analysed.

Results

First we will examine the coverage in SwePub both across publication types and all fields in comparison with existing data indexes and in the humanities and social sciences in more detail.

Publication type			Content
	Total	Peer-reviewed	Other academic	Popular science, debate etc.
Journal article	157,608 (48%)	133,382 (85%)	13,582 (9%)	10,669 (7%)
Conference paper	63,350 (19%)	42,354 (67%)	19,673 (31%)	1,349 (2%)
Book chapter	33,347 (10%)	1,298 (4%)	28,416 (85%)	3,651 (11%)
Doctoral thesis*	20,156 (6%)
Report	15,003 (5%)	6 (0%)	13,940 (93%)	1,079 (7%)
Other publications	10,606 (3%)	2,602 (25%)	2,947 (28%)	5,058 (48%)
Review	6,844 (2%)	66 (1%)	4,855 (71%)	1,924 (28%)
Book	6,164 (2%)	100 (2%)	5,127 (83%)	941 (15%)
Licentiate thesis*	5,033 (2%)
Research review	3,868 (1%)	2,266 (59%)	1,285 (33%)	217 (6%)
Editorial collection	3,817 (1%)	8 (0%)	3,457 (91%)	353 (9%)
Editorial proceeding	665 (0%)	106 (16%)	478 (72%)	81 (12%)
Artistic work	457 (0%)	7 (2%)	338 (74%)	112 (25%)
Patent*	361 (0%)
Sum	326,937	182,195 (56%)	94,098 (29%)	25,434 (8%)

Table 1: Total number of publications and distribution over publication
types for Swedish publications 2004-2011 (whole counts)

               * not applicable

The distribution of publication types in table 1 shows that journal articles represent the most frequent category amounting to nearly 50% of the total records in SwePub. They also mainly come from peer-reviewed journals (85%) and are by far the most common way to disseminate scientific work.

More striking is the fact that conference papers are the next largest group with about 20%. Also here, peer-reviewed papers are the most numerous category but we can also find a rather large category under ‘other academic’ which suggests that this is material which can prove hard to find in other databases.

Also books and book chapters are a large group (12%) and since by definition (with a few exceptions) they are classed as ‘other academic’ or as belonging to the category ‘non-academic’, one may suppose that they are also rarely found on common databases. Table 2 confirms this picture when one examines the distribution of languages. More than half of the books, and nearly half of the book chapters are written in Swedish. This means that they are rarely found elsewhere, except in library catalogues and on a few specialist subject databases.

Publication type		Language
	Total	International	Domestic
Journal article	157,664	138,965 (88%)	18,699 (12%)
Conference paper	63,412	58,965 (93%)	4,447 (7%)
Book chapter	33,390	18,038 (54%)	15,352 (46%)
Doctoral thesis	20,156	17,765 (88%)	2,391 (12%)
Report	15,037	7,394 (49%)	7,643 (51%)
Other publications	10,616	7,883 (74%)	2,733 (26%)
Review	6,843	2,536 (37%)	4,307 (63%)
Book	6,170	2,401 (39%)	3,769 (61%)
Sum	327,492	265,137 (81%)	62,355 (19%)

Table 2: Total number of publications and distribution across publication
type and language for Swedish publications 2004-2011 (whole counts)

When analysing the differences between subject areas, both in terms of language distribution and coverage in the ISI databases, we are confronted by the problem that only parts of the records in SwePub are indexed with subject categories. Therefore, only records from Uppsala University have been analysed, since subject indexing, at least from 2007 onwards, has been almost entirely compulsory.

In Table 3 we can clearly see differences in language distribution and ISI coverage with a high percentage of publishing internationally in Natural Sciences, Health Sciences and Engineering, while the Social Sciences, including Humanities, have a large proportion written in Swedish. This also corresponds with the ISI coverage, with high representation in Natural Sciences and Health Sciences (75-80%) and low representation in Social Sciences and Humanities (20% and 5% respectively). Engineering is the only subject with a high degree of international publishing that still has rather low ISI coverage (ca. 50%). This might have something to do with the fact that conference papers represent a quite frequent publication type in Engineering which is still somewhat under-represented in Web of Science.

Main field	Language		Coverage
	International	Domestic	ISI	Non-ISI
Natural Sciences	6,717 (99%)	64 (1%)	5,069 (75%)	1,712 (25%)
Social Sciences	2,238 (67%)	1,118 (67%)	662 (20%)	2,694 (80%)
Health Sciences	6,679 (97%)	242 (3%)	5,627 (81%)	1,294 (19%)
Engineering	1,459 (99%)	12 (1%)	802 (55%)	669 (45%)
Agriculture	61 (91%)	6 (9%)	41 (61%)	26 (39%)
Sum	18,533 (86 %)	3,004 (14%)	12,350 (57%)	9,187 (43%)

Table 3: Language distribution and coverage in the ISI citation
index for publications from Uppsala University 2007-2011

These figures correspond well with the results from other studies. In Norway, the coverage in existing citation indexes is low for SSH literature; only 11-20 % can be found in ISI. The figures for Natural Science and Health Science come out at about 80 % while Engineering Science here also assumes a middle ranking with 60 % [8]. Furthermore, the Flemish database for SSH literature shows the same proportion for Humanities (15 %), while the figure for the Social Sciences is higher (53 %). This may have something to do with the fact that the Social Sciences represent an increasing share of the publications written in English [18].

Finally, the total coverage of the publications from Uppsala University 2007-2011 in the ISI databases is 57 %. This figure is higher than in the Norwegian study where 49 % of the publications were found in Web of Science [8]. Although this figure may not be representative of the whole country, it is nonetheless interesting to note that it also signifies an increase in ISI coverage for Uppsala University, as compared with the period 2002-2006 [19]. For the SSH sector there is an increase in the Social Sciences from 15% to 20%, but for the Humanities the figures still are very low, about 5%. The higher coverage grade for the Social Sciences can be ascribed to the fact that, as with the Flemish community, there is an increasing trend towards publishing in English in this area.

Development of SwePub

As mentioned, SwePub has been the result of the efforts by local universities in Sweden rather than a result of governmental initiative. However, the discussion of using the portal as a data source in the Swedish PRFS model has been ongoing since the database was introduced and was also mentioned in the latest Governmental Bill in 2012 [20].

The database has to go through several improvements before it can serve as a tool in the PRFS model. First of all, all data have to be validated in terms of author addresses, duplicates have to be identified and, if need be, merged, and so forth. This also calls for a far stricter degree of quality control at the local university level and a system like the Norwegian ‘superbrukare’ ought to be considered.

Furthermore, subject indexing has to be completed at least for the included journals and perhaps also for individual records, depending on which PRFS model is preferred. However, the most important task is to encourage the universities still currently missing to participate in the project, for example the Karolinska Institute and the Swedish University of Agriculture Studies.

In the Governmental Bill 2012 there were additional resources set aside for the development of SwePub with the aim of supporting bibliometric analyses and a general quality assurance of the database [20].

The location of the portal is also of crucial importance in how well the visualisation of the literature is carried out. SwePub is managed by the Royal Library. From the beginning it has been integrated with the national library portal, Libris, as an additional search tool. This means that SwePub has the potential to reach the users of the national library catalogue like researchers, students and the public. However, the statistics show a declining usage from around 22,500 searches per month on average in 2010 to 18,000 in 2012. There are ongoing discussions about the full integration of the search function with Libris. A possible scenario in the future will be that SwePub will divide into a bibliometric tool for analysing and maybe also act as a PRFS database, while the search function will be integrated with the national library catalogue which also will enhance its visibility.

In Norway, the Frida/Forskdok database mainly had the function of being data provider to the Database for statistikk om høgre utdanning (DBH) although the database also included publication categories other than those who were identified with the PRFS model. Frida/Forskdok has never been a part of the national library catalogue in Norway; instead it has been working as a parallel system within the universities and other research institutions.

The introduction of the CRIStin database which has a somewhat different profile and content may however increase the usage and enhance the dissemination of scientific literature.  Since no statistics on its usage are currently available, this has still to be proven.

Discussion and Conclusion

Although the national catalogue in Sweden has developed from different perspectives and needs, it has similar content and distribution between publication types, subject areas and languages as its Norwegian counterpart.  Even if peer-reviewed articles are the most common publication type (ca. 50 %) there are a lot of publications from other categories, such as books, book chapters and conference papers, which cannot be found in other databases.

Even if we are unable to estimate the coverage in Web of Science for the whole database, we can compare the figures with the repository for Uppsala University, DiVA and it confirms the picture from other countries like Norway and the Flemish region, ranging from a high coverage of ISI papers in Health Sciences and Natural Sciences, to lower coverage in Engineering, and to very low coverage in Social Sciences and Humanities, although we can see a changing pattern of more international publications in the Social Sciences.

The differentiation in publishing between subjects is also clear when we examine the language distribution. Almost everything in the Natural Sciences and Engineering are directed to an international audience as well as in Health Sciences. In the SSH sector nearly 50 % of all publications are published in the domestic language, both in Norway and Sweden. Between 2007 and 2011, only some 400 publications written in Swedish were indexed in Web of Science and, of these, the majority are book reviews (75%).

The conclusion is that SwePub, as well as the other national portals for scientific literature, plays an important role in disseminating literature that is published in publication channels other than international journals. The national portal supports visibility of important contributions in primarily the Humanities and the Social Sciences, but also in Engineering and in some parts of the Natural Sciences.

If the coming PRFS model in Sweden includes non-ISI publications, SwePub will prove an essential tool for providing publication data. However, the database is not ready at this point to use for evaluation purposes, while data cleaning in the form of correction of addresses, completion of subject categories, etc, remains a task to be completed.

Therefore it can be argued that the national catalogues and portals have an important role to play in the new scientific landscape in both broadening the base for analysing scientific publishing as well as disseminating research.

References

1. OECD. (2010). Performance-based funding for public research in tertiary education institutions: Workshop proceedings: OECD.
2. Kettl, D. (2005). The global public management revolution: Brookings Institution Press.
3. Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251-261. doi: 10.1016/j.respol.2011.09.007
4. van Leeuwen, T. (2006). The application of bibliometric analyses in the evaluation of social science research. Who benefits from it, and why it is still feasible. Scientometrics, 66(1), 133-154. doi: 10.1007/s11192-006-0010-7.
5. UHR. (2004). Vekt på forskning. Nytt system for dokumentasjon av vitenskapelig publisering (pp. 83).
6. Debackere, K., & Glanzel, W. (2004). Using a bibliometric approach to support research policy making: The case of the Flemish BOF-key. Scientometrics, 59(2), 253-276. doi: 10.1023/b:scie.0000018532.70146.02
7. Schneider, J. W. (2009). an outline of the bibliometric indicator used for performance-based funding of research institutions in norway. European Political Science, 8(3), 364-378.
8. Sivertsen, G., & Larsen, B. (2012). Comprehensive bibliographic coverage of the social sciences and humanities in a citation index: an empirical analysis of the potential. Scientometrics, 91(2), 567-575. doi: 10.1007/s11192-011-0615-3
9. NIFU. (2008). Norsk vitenskapsindeks. In G. Sivertsen (Ed.), Rapport (pp. 66).
10. Local system refers to a locked area in (Virtual Private database) a common Oracle database
11. Proposition 2008/09:50 Ett lyft för forskning och innovation.  Stockholm: Utbildningsdepartementet.
12. Moed, H F. (2005). Citation analysis in research evaluation. Stockholm: Karolinska Univ Press Ab.
13. Norris, M., & Oppenheim, C. (2007). Comparing alternatives to the Web of Science for coverage of the social sciences' literature. Journal of Informetrics, 1(2), 161-169. doi: 10.1016/j.joi.2006.12.001
14. Sandstrom, U., & Sandstrom, E. (2009). The field factor: towards a metric for academic institutions. Research Evaluation, 18(3), 243-250. doi: 10.3152/095820209x466892
15. Carlsson, Håkan. (2009). Allocation of research funds using bibliometric indicators - asset and challenge to Swedish higher education sector. Info Trend, 64(4), 6.
16. Bjork, B. C., Welling, P., Laakso, M., Majlender, P., Hedlund, T., & Guonason, G. (2010). Open Access to the Scientific Journal Literature: Situation 2009. Plos One, 5(6). doi: 10.1371/journal.pone.0011273
17. Kempf, Jean. (2011). Social Sciences and Humanities Publishing and the Digital ‘Revolution’. European Review, 19(03), 469-481. doi: doi:10.1017/S1062798711000032
18. Engels, T. C. E., Ossenblok, T. L. B., & Spruyt, E. H. J. (2012). Changing publication patterns in the Social Sciences and Humanities, 2000-2009. Scientometrics, 93(2), 373-390. doi: 10.1007/s11192-012-0680-2
19. Nordgren, Joseph, Andersson, Per, Eriksson, Leif, & Sundquist, Björn (Eds.). (2011). Quality and renewal 2011. Uppsala: Uppsala University.
20. Proposition 2012/13:30   Forskning och innovation.  Stockholm: Utblidningsdepartementet.

Author Details

Leif M Eriksson
Planning Division
Uppsala University
Sweden

Email: Leif.M.Eriksson@uadm.uu.se
Web site: http://www.uu.se/en/

Leif Eriksson is a librarian currently working as an analyst at the Planning Division of Uppsala University, mainly with research performance issues.