More from TU:Librarian

Posted in May 2017

Evaluation. Another open science perspective.

Finally I had some time to read two reports related to open science, one from the European Commission Expert Group on Altmetrics, and a lengthier one from the RISE High Level Group. It is not my intention to summarize these reports in full, but merely to reflect on some parts I found interesting. Mind you that my focus these weeks is on finding tools or actions that can help in the 3rd line of our national plan open science, i.e. “Recognition of and rewards for researchers”, so I have been reading with some “bias”.

Part 1

The report “Next-generation metrics: Responsible metrics and evaluation for open science” ends with 12 recommendations, including the set-up of a European Forum for next-generation metrics. There were also five headlines provided at the end:

  1. An open science system should be grounded in a mix of expert judgement, quantitative and qualitative measures.
  2. Transparency and accuracy are crucial.
  3. Make better use of existing metrics for open science.
  4. Next generation metrics should be underpinned by an open, transparent and linked data infrastructure.
  5. Measure what matters.

And a denser summary could be: Metrics: use them responsibly, and use numbers together with a qualitative assessment.

From the altmetrics report: “Figure 2 – The basket of metrics for the evaluation of science (Haustein, 2015).”

This report perhaps does not add that many new insights, but offers a good overview of what has been done, the pros and cons of altmetrics, and gives a prospect for what needs to be done. Remarkable however that only nineteen responses were received on the Call for Evidence last summer.

Part 2

The RISE report can be downloaded, and also be purchased as a print book – it covers 228 pages (with a long list of references, and some useful case studies at the end), and reading it on my laptop in the train back from Hannover was not optimal. However, I picked up a few things.

The chapters and paragraphs were written by (groups of) individual members – the full report has not been endorsed by the whole RISE group. There is however “The Mallorca declaration on open science”.  Also here five “headlines” capitalize on achieving open science:

  1. Remove the barriers that extreme competition for limited resources create for Open Science.
  2. Implement Open Access publishing where publication is part of the continuum of research.
  3. Establish competence and confidence in the practice of Open Data.
  4. Ensure research integrity.
  5. A cohesive European approach.

Ad 1.

I invited Megan Carey at a CESAER Workshop on Open Access we organized in Bruxelles early in February (you can find the presentation she gave here). Megan is member of the RISE Group, so I was curious to read her contribution.  She indeed grasped my attention by stating: “True progress on Open Science in Europe will require rethinking the way research is funded and researchers are rewarded, in order to address the underlying forces that currently act to discourage Open Science. Longterm policy changes must directly address and remove the current barriers to Open Science practice. Such actions could fundamentally change research culture – simultaneously improving conditions for researchers, promoting excellence, and encouraging openness.” A few suggestions she made are the need for a relative shift of funds away from largescale collaborative projects towards PI-driven funding schemes; and more generally, that the granting schemes should undergo an overall simplification. Ultimately, there is a need to support a move towards funding centred on “people, not projects,” the approach of several world leading funding organizations. The latter claim is also reflected on a simplified drawing created by the “authors”.

From the RISE report: “Figure OI.3: New dynamics of innovation.”

Ad 3.

In the paragraph on open data, written by Ian Mulvany, I liked the rules (Goodman et al., 2014) Mulvany referred to. These rules are already from 2014, but never brought to me like this. As Mulvany puts it: “Were all of these rules to be adhered to by all researchers, we would have as good an Open Data ecosystem as we could wish for.”

The challenge “Competence in working with data” is addressed by the three rules:

  • Rule 1. Love Your Data, and Help Others Love It, Too;
  • Rule 3. Conduct Science with a Particular Level of Reuse in Mind; and
  • Rule 4. Publish Workflow as Context.

The challenge “Appropriate infrastructure for open data” is addressed by the following four rules:

  • Rule 2. Share your data online with a Permanent identifier;
  • Rule 5. Link Your Data to Your Publications as Often as Possible;
  • Rule 6. Publish Your Code (Even the small bits); and Rule 8. Foster and use data repositories.

Challenge three: “Creating a supporting culture for openness” addresses the three remaining rules:

  • Rule 7. State How You Want to Get Credit;
  • Rule 9. Reward Colleagues Who Share Their Data Properly; and
  • Rule 10. Be a Booster for Data Science.

And here my “bias” is being rewarded after all, because this last challenge and the related rules are about “how we can ensure that the correct incentives are in place to support the sharing of open data”.

Part 3

I noticed that in the RISE report recommendations are “all-over” being given to the “green road” of open access, i.e., “The RISE Open Science Group strongly endorses the use of the Green Open Access/ self-archiving model as the most immediate solution for Open Access publication.” In the part written in the Mallorca declaration this is a bit more specified and for me better understandable, and put in context: “The success of Open Science will depend on Open Access publishing having sufficient resources to implement a fair and transparent evaluation process and to ensure the quality, reproducibility and integrity of published research. Posting on recognized pre-print servers, data publishing platforms and self-archiving on shared platforms (‘Green Open Access’) provide useful complementary solutions for immediate pre-publication sharing of Open Science research.”

And then to end with a sidestep –recently I contributed to the Open Education Consortium Year of Open blog with a perspective on open science. Reading with this “bias” as I did (well not fully) with these two reports, is again enriching my perspective. Am I reading this as Library director, as DataCite member, as participant in the Dutch “big deal” negotiations, or as writer of the national plan open science?

Regardless of my own perspective, a very nice overview of the first National Open Science Day on 29 May 2017 was created by Mark van Huystee: just get inspired by all these perspectives!

Drawing made by Mark van Huystee, 29 May 2017. Part of the opensketching weblog “What does open science means to you?”

 

© 2011 TU Delft