CASRAI Impact 2018 Work Cycle
Interim Progress Report (March 2018)
Anthony Atkin (Chair CASRAI Impact Working Group)
This report summarises the progress of the CASRAI Impact Working Group. This Group forms part of the CASRAI Standards Open Review (2017 Standards Open Review - NOW CLOSED). This completes Action #1 of the March 2018 WG Phone Conference.
1. 2017 Impacts Data Collection Review
The 2017 Working Group (WG) was tasked to scope out the elements for an impact data standard to work towards a harmonisation of divergent information requirements for documenting and reporting impact.
The group was chaired by Anna Clements (St Andrews, Chair of CASRAI-UK Data Management Planning Working Group) The group received feedback from funders, impact specialists, data specialists and others.
The 2017 WG documented how UK universities are currently collecting impact information in their various local software solutions, including commercial and in-house systems and spreadsheets. The WG also documented the various impact reporting information requirements of the UK funders, including REF2014 (the next REF likely to be reasonably similar) and the Researchfish tool.
The 2017 WG agreed an Early Draft Glossary of Impact Elements for inclusion in the impact data standard (June 2017 - Impacts Data Collection Review - NOW CLOSED). These WG reviewed the elements and proposed fields for inclusion in the standard and elements to be parked. These elements are listed in the 2017 draft report.
A draft report was created and shared opened for public review (June 2017 - Impacts Data Collection Review - NOW CLOSED). Feedback was received from data specialists, impact specialists, commercial systems providers and funders (among others). The feedback revealed some of the complexities in creating and implementing a data standard for impact.
The public feedback successfully concluded this phase of the Impacts Data Collection Review.
2. 2018 Working Group
A new WG was assembled in 2018 to continue the work. This group was tasked with reviewing the feedback on the 2017 draft report and progressing towards a data standard.
Julie Bayley (Coventry University) assumed the role of Chair and recruited individuals to the group. To reflect the complex discussion in 2017, individuals with expertise in research impact were invited to join the group. After successfully establishing the group, Julie Bayley recommended the chair be passed to Anthony Atkin (University of Reading) as other commitments limited her available time. Thanks for Julie’s contributions to the group were formally recorded by Anna Clements and David Baker (CASRAI).
A brief assessment of the 2017 Impacts Data Collection Review was conducted by the Chair to propose a suitable way forward for the group. This preliminary assessment and the proposed approach are described below.
3. Preliminary Assessment
The elements identified in the 2017 work were reviewed. It was observed that the elements covered a wide range of potential types of information. These include elements that:
- Directly describe the impact achieved (what changed, who experienced this change)
- Describe how the impact (change) was achieved (activities conducted)
- Characterise the impact within a general impact taxonomy (e.g. societal impact vs. economic or environmental impact)
- Align an impact with different conceptual models of impact (e.g. reach and significance, levels of impact, impact stage)
- Elements that relate to integration with other systems or internal processes (REF, ReserachFish, confidentiality restrictions)
- Allow the storing of repository-style information (e.g. sources of evidence)
In some cases one element covered a discrete piece of information:
“Who is Affected? - Details of the audiences, beneficiaries and end-users.”
In other cases one element covered a variety of different types of information. The following example covers what the impact was, how it was achieved, timelines and impact yet to be achieved:
“Narrative - A detailed narrative of the impact. A textual/human readable form of the details of the activities which have been undertaken to create the impact and the combined result. This should be able to be read independently of other parts of the impact record. Include references to relevant periods and future developments.”
The diversity of types of information is not necessarily a problem for creating a data standard, however deriving an efficient standard that works for multiple scenarios may be challenging which is reflected in the public consultation.
The discussions as part of the public consultations were reviewed. This revealed a number of different perspectives were provided on a number of issues. Selected quotes are provided below (for full context see 2017 draft report: June 2017 - Impacts Data Collection Review - NOW CLOSED):
“I have some reservations about early/mid stage assignment - from experience this promotes potential/aspiration-based records. For those that do not develop further this leave a history of proposed impact that never occurred, something I would consider to be more negative in the long-term than no record at all. I am of the opinion that an impact record should record tangible change, in which case stage is perhaps not required at all”
“I have misgivings about impact stages - in particular a stage labelled ‘planning’ or ‘developing’ would I think will lead to innumerable submissions of ‘potential’ rather than actual impact and make it much harder to focus subsequent effort. But once again, it’s about what the data is collected for…”
“Are separate Description and Narrative elements necessary? This will depend on the purpose of collecting the research impact information and who the audience is – different individuals or groups, HEIs, funders, Government, or Trusts require different levels of information in various formats. So, with this in mind, there needs to be some flexibility built in.”
“Title: not more than 100 characters incl. spaces; include at least one term or phrase to uniquely and memorably identify the record in the mind of a reader. Description: more ‘what’ than ‘how’, i.e. describe the principal output/activity leading to the impact(s) AND the impact(s), but don’t dwell on how the one led to the other - that can be done if a full Case Study is commissioned.”
“To get the most out of this it work would be good to bear in mind the perspective of academics who will, after all, be doing most of the recording and have a big stake in how the information is used.”
“Links to research are also important if this is being based on REF so some fields for DOI or other identifiers - and for grants. These wouldn’t need to be required fields.”
“There are many diverse ways in which impact can arise or be achieved, it can take a significant amount of time for impact to be realised and it can comprise of several feedback loops. Feedback loops are complex in themselves and, even more complex, for the NIHR (and other medical research funders) when considered as part of a complex integrated health system. We suggest that the focus here should be on actual impact that can be verified, evidenced and/or triangulated – obviously, how it’s evidenced will depend on the nature of the impact claimed. Where actual impact is being claimed, evidence of external validation should be included, where appropriate. The introduction of ‘potential’ will add noise into the data systems.”
These quotes are provided out of context, it is clear that many factors beyond the technical requirements of a future data standard are being raised as well as more technical considerations.
By placing the characterisation of elements and the public consultation discussion together, it can be seen that there are too many issues at play to successfully reduce the perspectives down to refine a data standard. Similarly, the proposed elements do not in themselves satisfy the broad needs of the communities represented in the consultation; the technical aspects of the data standard and the scenarios in which the data standard do not converge easily.
4. Proposed approach
On the basis of this brief assessment an approach was developed and proposed to the 2018 WG by the Chair.
It was proposed that the technical aspects of the data standard and the scenarios in which the data standard will be used be separated and developed in parallel streams by the 2018 WG.
It was proposed that once sufficient progress is made on each stream that steps are undertaken to tailor the technical aspects to specific scenarios.
This proposal was accepted by the 2018 WG.
Therefore the objectives for the 2018 WG are:
- To define the minimum amount of information required to accurately describe an impact
- To define the scenarios that the data standard will be used in
- To consider how to tailor the minimum description to the scenarios
The Chair will define work packages to achieve each objective and that will be completed via the CASRAI platform and associated phone conferences.
It is proposed by the Chair that these objectives be completed before July 2018.
The long-term objective is to create a data standard that can be readily adopted by different individuals and organisations to improve their impact recording and reporting.
Anthony Atkin (University of Reading)