Assessing cross-thematic impacts in networked projects
Commissioners often seem to take it for granted that their interventions will produce integrative effects automatically and somehow by design. The same impression follows from my recent review of available evaluation studies of networked projects in the EU’s ERA-NET.
ERA-NET is a funding instrument of the European Union’s Horizon 2020 research and innovation programme. It is designed to support public-public partnerships for mature networks with strong long-term financial commitment from participating states for preparation of joint activities, establishment of networking structures, programming, coordination, and implementation.
A review of relevant literature quite surprisingly reveals an extensive grey area in the methodology of assessment of cross-thematic impacts for ERA-NET projects. It is underrepresented and misconstrued already in the evaluation guidelines and consequently, it is largely missing or is futile in subsequent impact evaluation studies.
The guideline document actually suggests assessing networks linearly by responding to a set of 45 prescribed assessment questions. In this way, the guideline decomposes a network to its quite provisionally defined details and explains it piece by piece. Describing a network from the aspect of its widely different parts entirely misses the point. It invites a static understanding of the network as a complicated structure. Just the opposite: the network implies chaotic logic of dynamic self-organisation emerging from initial disorder (see distinction between complicated, complex and chaotic in Stacey).
The most intuitively selected approach to the assessment of networked projects would probably be network analysis. The Institute for Prospective Technological Studies (IPTS) of the EU’s Joint Research Centre explored methodological approaches for assessing networked projects. It explained that the methodology of network analysis is mainly concerned with demonstrating links between numerous entities and the position of entities among each other. These entities are named ‘nodes’ or ‘vertices’, and they are linked by ‘edges’ or ‘ties.’ Network analysis describes the coherence of a network, its density, clustering, compactness, etc., where the denser, more connected network is comprised as a better network.
However, IPTS itself presented international networks in the Health research field that are shaped quite differently from the previous description. Networks often do not take the shape of an anarchic structure between many networked elements but more as a network of networks (Figure), as a structure of networks that are linked loosely to other networks.
This is a very important difference. Networks of networks are not appropriately assessed from the aspect of a network (analysis) but of a matrix. Not between parts and the whole but between intersected networks as domains, sectors, or themes at the middle level of evaluation. With other words: the network of networks is not a micro-macro setup. A matrix is an organised network of relations at the meso level. The meso level operates as a matrix, a meso-matrix.
In turn, networked projects need to evaluate their impacts by matrical assessment of connectedness between narrower networks as independent domains, themes, or branches of a broader network. Cross-thematic assessment arises as an imperative for studying integration between rather independent networks.
Yet, a cross-thematic view is not the only relevant approach to studying networks. In ERA-NET it can progress in three distinctive ways: by enhancing cross-thematic, cross-border, or cross-cutting connectedness. These three operate simultaneously but produce connectedness between networks with distinctively different logics.
Picture: Network of networks
The cross-border (or cross-country) networking is probably the most regularly addressed in the impact assessments of ERA-NET projects. Cross-border actions increase connectedness as such, even when not pursuing cross-thematic or cross-cutting goals. In turn, it seems that cross-border (or cross-country) cooperation can be properly assessed with evidence of its bare existence, with outputs (not necessarily with results or long-term impacts!). Cross-border connectedness is for the same reason often applied as an eligibility (yes/no) criterion that formally precedes assessment of a project proposal’s quality in substance.
The third approach to connectedness in networks (of networks) is linked to pursuing cross-cutting or overarching issues. Such as harmonisation between network partners, preparation of common guidelines, frameworks, standards, vocabulary, classifications schemes, measuring scales, modelling principles, or assessment methodologies. Cross-cutting as an essentially horizontal approach is characteristic for implementing core concerns with vertical means.
As result-oriented, cross-cutting issues are then best evaluated with indicators of their effectiveness (not impact or output).
After all, the matrical approach is imperative only for assessing the cross-thematic impacts of networked projects, but not for assessing cross-country or cross-cutting effects that also take part in strengthening project like ERA-NET. This is far from claiming that a matrical evaluation approach is useless in cross-country or cross-cutting impact assessment, only that it is not imperative. Yet, without meso-matrical effects, a network of networks cannot produce the unity in diversity that is needed for coping with contemporary transformative challenges that are themselves cross-thematic.
