Hard to Swallow: ‘Trans’ in Transformation
A book review of “Transformational Evaluation for the Global Crises of Our Times,” Van den Berg R., C. Magro, M-H. Adrien (Eds.). 2021. Exeter, IDEAS — International Development Evaluation Association, 448 pp.
Introduction
This is not an ordinary conference book. It is a book with a mission. Its framing and format call for immediate attention. The collection consists of 18 articles, contributed by prominent authors in the field of evaluation. Among them are Michael Quinn Patton, Linda Morra Imas, and Robert Picciotto. Three eminent authors edited the book: two of them are former Presidents of an organisation that published this book, the third is its former Secretary-General. The book gains particular significance by presenting the Prague Declaration on Evaluation for Transformational Change, endorsed at the IDEAS Third International Conference on Evaluating Environment and Development in 2019.
The book brings together relevant discussions about how the evaluation of policies, programmes, or projects can improve the evaluation of appropriately transformational change, triggered by a series of global crises from the beginning of this century. The editors have organised authors’ contributions into six parts presenting practical experiences with transformational evaluations, specifically relevant themes and cases, such as the professionalisation of evaluation and methods. The core message of the book is about the crucial importance of systemic and participatory approaches in transformative evaluation.
In the conference book format, the fragmented presentation of a complex theme is unavoidable. This makes an editorial contribution not only indispensable but also decisive for shaping the book’s overall message and reach. The editors connect fragmented contributions to tell how the book answers a conference theme. Even more important is their role in publishing a project with a mission.
‘Real transformational problems are very hard if not impossible to define; it is unclear which mechanisms promote them, who are main stakeholders, far less what are their planned goals.’
This review rests on the analysis of answers to three questions about the editorial contribution to the book’s mission:
- Scope: What is transformation? The same as reform, revolution, transition, or something different?
- Theory of transformational change:
o How are systems and transformation connected? At first view, they appear to be like apples and apple pie: the former about an order, the latter between order and chaos.
o How system thinking integrates with participatory approaches in transformational evaluation taking into account their contradictory, realist, or relativist philosophies (Stacey, 2011)?
Transformation as a concept
The editors immediately remind overly enthusiastic readers that there is much hype around word transformation. The term has become widely used. It has ‘taken on a trendy cachet’ as a fashionable buzzword (Picciotto; similarly to the term complexity). The problem, in their view, is that we still lack a broad understanding of what constitutes transformational change.
The book contains a diverse conceptual presentation of transformation. The editors see transformation as a major revolutionary modification in a relatively short period between two known states, such as in crisis management (return to normality) or transition to sustainability with clearly defined sustainable development goals, or climate change with a clearly specified threshold limit of ‘acceptable’ global warming.
Understanding transformation as a short-term phenomenon, from one or two to five, or maybe even ten years, is indeed sensible in many considerations. Such as for the evaluation of high-frequency technological changes, or political projects and programmes. Their impacts may be impressive in narrow areas of concern (institutional, locational, thematic, sectoral, the targeting of a specific social group, etc.) but the achieved outcomes usually do not interact with each other and especially not with other crucial areas of social concern. The overall structural, let alone transformational change in society remains in this way limited and is not transformational.
Short-term radical changes are usually accompanied by predefined transformation goals, with an identified roadmap (such as UN Agenda 2030), key stakeholders are known and instruments of change are ready for application. This sort of change cannot really comprehend transformation as radically uncertain. Uncertainty means that some of the essentials about ourselves and the changing world are not knowable irrespective of our personal or institutional efforts. Real transformational problems are very hard if not impossible to define; it is unclear which mechanisms promote them and who the main stakeholders are, far less what their planned goals might be.
‘The evaluator is not a social activist but one who knows how to bridge between contradictory contributions without giving a priori advantage to one or the other.’
Transformation then decisively differs from short-term revolutionary changes. First, transformation does not overthrow and destroy the old order like revolutions do, but intermediates transition between old and new. Similarly, reforms are incremental and result from unprincipled compromises. Second, transformation is a radical transition between known and unknown situations. Finally, if revolution is a dialectical negation of a certain ignorant position, transformation is a negation of the negation — absorption of the negative elements of the anti-thesis (revolution). Transformation emerges from a synthesis of contradictions instead of through revolutionary fractures.
The short-term concept points to the transitional rather than transformational orientation of the editors. Their transformation somehow lost its ‘trans.’ ‘Trans’ points to something so radically new that it cannot even be properly expressed without some sort of revelation of novel meaning or logic. The evaluation of complex trans-phenomena requires radically innovative approaches. The editors nevertheless ask evaluators to accomplish transformational evaluations with a good old systemic approach.
This brings me to a second question about how the theory of systems stole the story in a book about transformational evaluation.
Systems and transformation
The book aims from the start to strengthen and spread the paradigm of systems thinking in evaluation, ‘micro to macro, and macro to micro’ (Quinn Patton). Transformational change such as from non-sustainable to sustainable development is for them primarily a systemic challenge.
The editors remind readers that most if not all of the transformations of the past happened to humanity without any master plan. The present transformations regarding the climate change agenda and sustainable economic development are different due to established global collaboration and agreements on the goals to be achieved in the next decade. We can act collectively now as one big almighty system to overcome even the most complex problems.
‘The editors’ move away from positivist science to systemic science is not radical enough to overcome the realist frame of discussion.’
Evaluators only need to redefine the role and competencies to enhance the transformational potential (Rodríguez-Bilella et al), revise criteria for the evaluation of transformation (Quinn Patton), develop a new educational programme (Linda Morra Imas) and advance evaluators’ professional skills (Ocampo). When existing standards, models, and especially data sufficiently improve by incorporating knowledge about uncertain issues, we will for the first time in history be able to manage a social transformation toward an aspired future in an ordered and controlled way. If we are devoted and persistent enough, the good will win over the bad and order over disorder.
This probably sounds naïve but it is not, at least in terms of an underlying realistic philosophy of knowledge, one of the Western modern era’s constituents since the 18th century. Realists generally do not see any inherent limitation to the human ability of comprehending reality in its entirety (Stacey, 2011). For them, it is only a matter of time before we progressively uncover more and more of its secrets.
‘It seems to me that evaluation emerges as Western mind’s gateway to holistic thinking after centuries persisting under the ruthless rules of either reductionism or relativism.’
Systems thinking is also that of objective observation. Magro and Van Den Berge wrote: ‘Systems thinking requires … ways of thinking … that lead us to understand the functioning of things’, a network of independent causes and intertwined effects about to the overall system goal. They continue: ‘What we need to do for transformational change is to provide … data and insight for the system that is supposed to be changed.’
Is this not pretty much the same as good old positivism and its hard evidence-based mantra? When applied to organisations, as in Stacey (2011), systems are done in a realist way. How does this concern us? It shows that the editors’ ‘transformational’ move away from positivist science to systemic science is not radical enough to overcome the realist frame of discussion. Their transformative change is not really trans.
The esteemed editors observe transformation as if located on the edge of order, where systems are already completely swamped with uncertainty and complexity. Complex systems are everywhere in contemporary society and their management has proved so successful that we often forget the complexity behind everyday normality. We see only systems, only drained land. It appears that systems are always able to somehow domesticate uncertainty. Such as with risk assessment plans, by the precautious installation of new safety buffers, or by improving existing technology, rules, and by updating standard professional procedures. Systems create the macroscopic perspective of transformation, from the top to down, such as from leaders to citizens.
If the idea of systems extends too far beyond the realist world of certainty and objectivity, it can soon pervert from a solution to a problem. When everything touched by the system becomes a system then systems are like Medusa in Greek mythology, turning to stone everyone who has gazed into her eyes.
Despite its clear preference for systems, the book decisively avoids the threat by including collaboration and participation as key approaches besides systems to transformational evaluation. Which brings me to the third assessment question about how the systemic and participatory approaches overlap in transformational evaluation.
Systemic and participative approaches
The participatory approach (or co-operative, communicative, partnership) to transformative evaluation belongs to relativist philosophy. Relativism has been the defining characteristic of postmodernism since the second half of the previous century. Those who hold this position maintain that there is no predetermined reality outside of human interpretation. There is no reality, only the stories we tell each other and, according to those who take this position to the extreme, one story is as good as another is (Stacey, 2011). ‘Anything goes.’
‘When everything touched by the system becomes system then systems are like Medusa in Greek mythology, turning to stone everyone who gazed into her eyes.’
The philosophy of relativism gives ground for understanding complex trans-phenomena in a chaotic way, observed from micro to macro, such as from people to systems. It belongs to a world of freedom such as civil society (as the opposite of the State or System) so it gives rise to anti-systemic dispositions (Wallerstein, 1996). This sort of complexity develops on the edge of chaos. This is where spontaneous self-organisation emerges from chaos such as through dialogue or political negotiation. When chaotic complexity is measured, it creates a world of big data, the heavens for researchers equipped with basic mathematics and capable computers. Democratic deliberation and computational approaches again promise that evaluators can see through uncertainty, so they probably do not need to bother much about the ‘trans’ in transformation.
Synthesis in the middle
Systemic and participatory approaches are both indispensable in transformational evaluations. However, they belong to contradictory philosophies with incompatible intervention logics and incommensurable outcomes. How to integrate different theories to shape non-confusing guidelines for transformational evaluation? This is a rather urgent question that arises in the book precisely through an editorial approach, but it is not addressed with due concern. This leaves transformational evaluation on thin ice about the crucial question. A prerequisite for acheiving the ‘trans’ in transformation is, in my view, to find a logically consistent solution (with the complex nature of a challenge) to evaluative the integration of systemic and participatory approaches.
Professor Ralph Stacey can help. He suitably illustrated complexity as a concept between order and chaos (1996). A complex trans-phenomenon is a hybrid between the oppositions that determine it, sharing characteristics of both, so it can bridge between (Latin com-plexus) order and chaos or in the concrete case between systemic and participative approaches to evaluation. This suggests that transformation needs to be seen as a mesoscopic phenomenon — partly realist and partly relativist. Sibanda and Ofir emphasise the same point more succinctly in their contribution by writing ‘there is a middle way between determinism and uncertainty … [so] that intrinsic validation rather than empirical verification is required for evidence’.
The mesoscopic nature of trans-phenomena infers that transformational evaluation needs to apply both systemic and participatory (antisystemic) approaches but only as partial tools with inconclusive contributions that obtain full sense when evaluatively overlapped. On the one hand, evaluation is a rational and systemic effort with a standard systemic (professional) procedure. However, evidence is never irrefutable, always exclusive and conditional. In turn, assessment results must be validated with the evaluative interpretation that needs to take place in a certain sort of participatory process.
It follows that realism is not really value-free. And vice versa, participation (and relativism) must be about realist questions otherwise its results become unrealistic (Radej, 2021), which would be entirely unhelpful for the direction of action motivated by the maximisation of a public good. This simply means that facts and values rarely travel alone, which can be properly appreciated in evaluation.
Transformational Evaluation
The transformational situation requires a paradigm shift with a fundamentally new way of addressing the phenomena (Rodríguez-Bilella et al) including radical changes in evaluators’ mindsets (Picciotto). Micro and macro, systems and participation are several decades old concepts, even in evaluation.
If evaluators wish to achieve the ‘trans’ in transformation, they need to evaluate changes from the perspective of the middle, which enables the synthesis of systemic and participatory approaches to evaluation. The task can be accomplished for instance with mesoscopic oscillation between the two:
- Systemic approach: The evaluated matter is first defined as complex (trans-phenomenon, like sustainable development) at the meso level, which is at the intersection between foundational contradictions from which a given complex matter has emerged.
- Participatory approach: Participants then convene to identify three horizontal domains of a given complex matter on the meso level, such as the economic, social, and environmental domains of sustainable development.
- Systemic approach: The evaluator continues by rigorous analysis of the impacts of the evaluated phenomenon on three domains and their cross-sectional links at the meso level.
- Participatory approach: The evaluation concludes with participants discussing synthesized findings and evaluatively interpreting them in the pursuit of shared understanding.
In transformational conditions, evaluation gains much broader relevance compared to the mainstream understanding of evaluation as merely an instrument of effective policy-making. Picciotto in fact emphasised its broader relevance by writing about the need to enhance evaluation as an autonomous field of knowledge (!). This is an essential remark for transformational evaluation. In my view, evaluation is equal to philosophy (wisdom) and science (truth) as ‘a third culture’ (Wallerstein, 1996) of comprehending the world. ‘A third culture’ that intermediates between physis and metaphysis, or in our case, between systems and participation, for a more complete understanding of complex issues, as further discussed here.
In brief: standard science distances itself from taking into account diverse values. In a complex situation, it is therefore not uncommon that somebody’s bare truth is sharply opposed to what is highly valued by many others. Genuine insight into complex trans-phenomena can only emerge from the evaluative standpoint, which means by comparing different truths about a certain matter with incommensurably different valuations accompanying it, at the middle level between micro and macro (for instance with the square input-output matrix and Venn diagram, as in Radej, Golobič, 2021). It seems that evaluation emerges as the Western mind’s gateway to holistic thinking after decades and centuries persisting under the ruthless rules of either reductionism or relativism.
‘When many essential things are unknown, the evaluator’s distancing from blinded polarisations becomes a virtue.’
The last remark is somewhat of the track in this review since it refers more to the authors than to editors. In their contributions, the authors forwarded strong positive aspirations for the enhancement of public goods and collective welfare. I wish to emphasise that the evaluator’s preference for any specific form of good or welfare would be inappropriate in light of the evaluation challenge. A transformational evaluator is not a social activist but one who knows how to bridge between contradictory contributions without giving a priori advantage to one or the other. Such as between physical and mental health during coronavirus-related lockdowns, or between economy and environment in the triangle of sustainable development. Distancing from the opposing poles that complexify the world does not point at the moral weakness of transformational evaluation to take sides in a moral dilemma. Just the opposite is true. When many essential things are unknown, the evaluator’s distancing from blinded (exclusivist) polarisations becomes a virtue.
Conclusion
The esteemed editors do not seem to fare comfortably with the idea of a middle ground. Ignorance about the middle is quite strange for a systemic approach. Systems theory triggered breakthroughs in science precisely because of its ability to explain the world with relations between things rather than by explaining things themselves in a uniform (polar, either micro or macro) perspective as the positivist philosophy of knowledge had previously insisted.
The findings obtained from the three assessed questions are convergent. A paradigm shift is not achieved in this book. This is not a transformational book about transformational issues. It is therefore hard to expect that its mission can be fulfilled as envisioned.
‘If evaluators wish to achieve the ‘trans’ in transformation, they need to evaluate changes from the middle perspective.’
Nevertheless, this book is a valuable document that accurately reflects the challenges of transformational evaluation in the mainstream discourse of an irreversibly post-normal condition. If the book succeeds in triggering discussions that advance transformational evaluation, as invited in the Foreword, its book’s purpose can be nevertheless achieved indirectly and more surprisingly.
List of references:
Radej B., M. Golobič. 2021. Complex Society: In The Middle of a Middle Word. Wilmington, Vernon Press.
Radej B. Strateške izbire v temeljni negotovosti (eng. Strategic choice in radical uncertainty). Ljubljana, Slovensko društvo evalvatorjev, Delovni zvezki št. XIV/1(Junij 2021).
Stacey R. D. 1996. Complexity and Creativity in Organizations. San Francisco, CA Berrett-Koehler Publishers.
Stacey R. D. 2011. Strategic management and organisational dynamics. The challenge of complexity to ways of thinking about organisations. 6. ed. Harlow, Pearson.
Wallerstein I. 1996. The Rise and Future Demise of World-Systems Analysis. Paper delivered at 91st American Sociological Association, New York, Aug. 16.
Wallerstein I. 1999. Globalization or The Age of Transition? A Long-Term View of the Trajectory of the World-System. http://fbc.binghamton.edu/iwtrajws.htm
Žižek S. 2008. Violence: Six Sideways Reflections. New York, Picador.
Link to the reviewed book: https://ideas-global.org/wp-content/uploads/2021/07/2021-IDEAS-book-Transformational-Evaluation.pdf
