6.1.5 Analysing the impact of joint calls

A final step in assessing the joint call is the analysis of generated impact. This stretches beyond monitoring of funded projects and is also different from analysisng the mechanics of call implementation.

The basis for all impact assessments should be the defined goals for the joint call or the programme. The type of research funded by the ERA-NET will have bearings on what the expected impact of the funded research should be. ERA-NETs funding academic research should have different impacts than ERA-NETs funding industrial R&D for the benefit of SMEs.

For some programmes of industrial research it may be most relevant to ask the enterprises involved about the revenues generated through their innovative product/service resulting from funded projects, whereas in other programmes – in both basic and applied research – it may be relevant to search for scientific gains but also for societal impact in a broader sense.

Some ERA-NETs are specifically designed for providing a basis for policy making in a specific field – in such cases the impact must be measured as to which degree knowledge developed in the supported project is taken up and implemented in policy making, or ultimately, to which degree this knowledge has lead to solving the problem addressed by the specific policy.

There is agreement among all major agencies that have undergone this task that the impact of project outcomes even at the national level is difficult to measure and even less, in quantitative figures. Whatever quantitative measures are applied, i.e. (i) publications/impact factors in basic science, (ii) patents/market revenue/socio-economic factors in applied science, and (iii) the only recently emerged issue of dissemination of results /creation of public awareness (in all sectors), these lack sound, commonly agreed and undisputed indicators. This puts transnational impact assessment of outcomes of ERA-NET calls/programmes on a rather poor basis.

Impact assessment of the joint calls should focus on added value when compared to national funding. The ERA-NET should develop indicators reflecting the goals of the joint call, decide how the data on these indicators should be gathered and by whom, and set target figures where appropriate.

The toolbox provides examples of objectives and indicators that can be used as a point of reference.

An example of a fully developed system, with objectives and quantitative targets is found here. This system could be used as a starting point for other ERA-NETs, especially ERA-NETs funding applied, industrial research.
An according example for ERA-NETs with a focus on basic research is found here, since basic science is (traditionally) well grounded in international cooperation, there is less focus on the aspect of internationalisation.
For ERA-NETs directly linked to policy making it is important to include indicators on relevance and use of the knowledge developed in projects funded by the ERA-NET. An example of such a system is available here.

In any case, impact assessment of projects goes beyond the usual monitoring process during the execution of an individual project and could thus extend beyond the life time of an ERA-NET, too. It involves both a time and a cost component, and additionally it involves a direct approach channel to the project participants, whereby the number of questionnaires and reports from the project participants should be held at a minimum.

1. Time component: Usually, the projects do not turn out direct results that can be immediately measured in terms of impact. Therefore, the project impact assessment will only start operatively when first projects resulting from an ERA-NET call are finished or even later. However, a timely set-up of ex-ante-indicators (i.e. indicators that can be clearly quantified and that are (indirect) project goals and thus form part of the funding decision and may be stipulated in the funding contract) is strongly recommended already for the call planning phase, because these can only be assessed properly if they have been involved and formed part of the evaluation process. Contrary, ex-post-indicators (i.e. indicators that are set up after the evaluation phase and without direct project context) will allow for a later set-up, but lack qualitative stringency and quantitative robustness.

2. Cost component: As for common project reporting in joint calls, impact assessment (if jointly done by the call secretariat on behalf of all the agencies in the ERA-NET) will have to be paid for beyond the ERA-NET Grant Agreement. Financing of this long-term task has to be considered by the ERA-NET and may be stated in a Memorandum of Understanding.

3. Channels for collecting data: In planning which channels to use for collecting data, effort should be made to minimise workload for all parties involved. Possibilities of integrating questions for project impact assessments and the analysis for optimising call procedures (see 6.1.2) into joint reporting forms should be explored. If it is necessary to develop specific questionnaires, these should be easy to answer (possibly with Y/N answers and/or using pull down answers). There are several tools available for online surveys. Online tools can make it easier to reply to the survey, and not least make the processing of the collected data much easier than analysing data from questionnaires in text or spreadsheet format. As such tools are often used routinely in funding agencies for collecting feedback, these may be made available through one of the agencies of the ERA-NET. Suitable tools may also be found on the web by searching “online survey tool” (http://www.surveymonkey.com/).