Sounds of Silence in Evaluation
Evaluation contributes an essential part to our understanding of the world, along with philosophy and science. It enables us to grasp ambiguous and radically uncertain issues, such as opposite but well-justified claims stated from independent standpoints. Evaluation sees things that are invisible for science. Moreover, in its peculiar way, evaluation grasps the uncertain world through what remains invisible to an enlightened rational mind.
Despite its ability to understand the world beyond science, an evaluation must remain humble. The evaluation itself fails to see many visible things. As a policy impact evaluator, I recall various examples when evaluation remained blind and deaf about arguments that contradicted conventional wisdom.
This post discusses an indicative example about excessive use of the notion ‘not relevant’ in policy impact evaluations.
When A does not impact B, B is ‘not relevant’ for A. ‘N.r.’ describes something that is absent or present only in an unimportant way; a small quantity or dispersed. Evaluators and commissioners generally welcome n.r. as a very useful tool, as it eliminates many hypothetically possible but irrelevant questions, thus enabling them to focus on the most important concerns of the evaluation.
I took part in an ex-post evaluation study recently that assessed the impacts of more than fifteen narrow projects that were implemented under a broader project designed as a network consisting of more than 40 partners from more than 30 countries, financed by the EU’s Horizon 2020 Programme. Before the completion of their projects, evaluators asked managers to fill out a questionnaire asking them to assess the contributions of their narrow project to the main thematic goals of the broader project.
The results were quite suggestive. Respondents assessed positive contributions mostly in their thematic field. 30% to more than 90% of potential cross-thematic impacts were assessed as not relevant. A considerable discrepancy emerged between what was relevant for the broader project and what was relevant for the narrower projects. The evaluation needed to look at this gap and attempt to explain it.
Even in cross-thematic projects, it is not the case that everything is connected to everything else. Not all potential connections arise naturally or are not needed in a given context. The connections between themes may be absent due to some obstacles. For instance, certain cross-thematic overlaps did not materialise because of potential project partners’ administrative or financial weaknesses.
At this juncture n.r. scores locate empty space in a network of connections. They also track where unused cohesive potentials are hidden.
In the evaluation of cross-thematic impacts, n.r. scores nonetheless usually signal an alarm. They point to the absence of cooperation in a project designated for networking (I have written on the evaluation of three network effects: cross-thematic, cross-country, and cross-cutting). Different narrow projects of course work independently on different thematic questions, but even so they may address the same end-users or stakeholders, work with similar technical procedures and methodological tools or face similar constraints. Circumstantial or contextual similarities grant narrow projects plenty of opportunities to connect indirectly to resolve overlapping issues. Such as sharing some of their resources or co-implementing certain tasks or exchanging experiences.
The absence of cross-thematic impacts may result from the intervention logic of narrow projects. N.r. signifies an absence of something primarily important or directly related (to me). Analogously, in cross-sectional evaluation, n.r. signifies the absence of my impact on somebody else with entirely different thematic concerns (thus, ‘indirect impacts’). Direct impacts (or their absence) are evident to all in the same way. Indirect impacts, however, are usually hard to detect with conventional appraisals. In turn, the presence (and relevance) of indirect impacts remain hidden! They become visible to managers only when they make extra effort to identify them. They need to look beyond the horizon of their narrow scopes in search of potential synergies with other projects on other themes.
One of the project managers explained that cross-thematic impacts are of secondary importance. They are not relevant because they do not fall within their project’s primary scope. But is this not exactly the reason why the broader project wants to assess the indirect impacts of narrower projects?
Another respondent similarly asserted that cross-sectional as horizontal impacts are relevant only for assessing projects with vertical goals, such as profit-motivated projects. The respondent is supposing that the impacts of horizontal projects are somehow neutral about one another. This conjecture is not founded on solid ground, as it is well known that different public interests can work against one another. For instance, the construction of a new student campus can have a greater or smaller ecological footprint (an example originates from my evaluation study on EU cohesion policy in Croatia).
In both examples, numerous n.r. scores reveal a divergence between lower and higher project levels about which impacts are relevant. This finding called for a consistency check to determine whether the narrow projects appropriately employed the intervention logic of the broader project.
Extensive use of n.r. scores is particularly alarming when they distribute unevenly across the assessment matrix and concentrate only on certain cross-thematic concerns. Asymmetry usually points to systemic inequality. For instance, in conservative traditional communities, managers sometimes rate the impact of their project on gender equality as being neutral and thus not specifically relevant. This sort of neutrality reproduces existing social exclusion patterns and subverts cross-thematic concerns.
In such examples, n.r. scores signal the location of exclusion due to, for example, cultural or institutional prejudices.
The observed patterns of n.r. scores in various studies hide important messages and even alter messages. Jacques Derrida claimed that essence is no longer at the heart of meaning (primary goals, direct impacts) but void — that which is missing. Void is an indestructible trace not only of prejudice, insensibility or lack of knowledge but also of radical indeterminacy in every single thing due to the radical uncertainty in our world. When radical uncertainty rules, the void becomes the most ‘realistic’ feature of the world, following Jacques Lacan. In the darkness of radical indeterminacy, the truth can only come from the void. Alain Badiou wrote that the site from which truths might be grasped must remain empty, which is to say open for incomplete, different but legitimate understandings — a customary situation in cross-thematic undertakings.
Selected examples from impact assessment studies confirm one thing. What is initially invisible becomes visible when passing through the evaluative procedure. In a cross-thematic situation, the void is neither mute nor invisible. The bias involved in my claim is invisible only to me, but it is visible from the perspective of many others with a different bias. In the evaluative frame, different instances of void locate and deconstruct one another. In the same manner, the careful appraisal of n.r. scores is an effective tool for the deconstruction of bias in the cross-thematic impact assessment.
To grasp what is missing, an evaluation must assess policy impacts as blindsighted. This means neither being blinded by what is primarily relevant nor blind to what appears irrelevant at first sight. Hence, the evaluation must not only analyse available data and listen to the voices of those included in the study, but must place equal importance on the ‘sounds of silence’ of everything that is in various ways ignored in the data, absent, or voiceless — a large majority of us, not always but far too often.
The conclusion is that governance of deeply uncertain issues needs to rely on the evaluative wisdom of silence. Contemporary societies have entered into a completely new situation in which only the void can comprehend void. No longer an enlightened mind, but an evaluative one.
As in the case of this post. It illustrates that void and integration logic are connected in evaluation. Even more, it suggests that the void is at the core of the integrative logic in evaluative thinking.
