Magenta Complexity
The UK Treasury adopted new central government guidance on evaluation in 2020, so-called The Magenta Book. The new guidance emphasises particular concerns for evaluation in complex conditions highlighting complex systems thinking implications for policymaking that cuts across many areas of governance. CECAN (The Centre for the Evaluation of Complexity Across the Nexus, 2020) welcomes the adoption of a new guide as evidence of a major shift in an approach to policy challenges, analysis, design, and evaluation.
The Magenta Book is accompanied by a special Annex comprising a supplementary guide on handling complexity in policy evaluation, prepared by CECAN. The guide’s goal is the adoption of a more integrated approach to evaluation. It explains what complexity thinking is, what the features of complex systems are, and how new methodologies and tools can equip policymakers to work with unavoidable complexity. In particular, the guide identifies which complexity-appropriate evaluation strategies can be used in various complex conditions.
Recognition of the fundamental challenges of complexity by a central government is an important milestone in policy impact evaluation. Evaluators in other countries must acknowledge this achievement. As practitioners but also as learners or critical observers.
In short review, I focus on what I perceive as the main problem of the guide — its highly problematic theory of change. The authors understand complexity in a reductionist way, even though complexity is an integrative concept, com-plexus originally means braiding together. Two illustrative examples are addressed. First, they decompose the leading features of complexity, and identify partial solutions for each of them. Second, the authors apply either complicated or chaotic explanations of complexity, as if complexity were not an independent concept, containing part of both but also sublating them.
In efforts to define complexity, the authors rely on two sources. The first is the Complexity Evaluation Framework (CEF) commissioned by The Department for Environment, Food and Rural Affairs (Defra). CEF presents complexity-appropriate evaluation as nested components organised in concentric circles. However, this is a standard fractal representation of chaotic phenomena, not of complex ones (see Gleick, 1987).
The second source is Stacey (1996), who defined complexity in the renowned Certainty Agreement Matrix (CAM) as accommodated between ordered and chaotic rules. Order is located in the area with high certainty and high agreement, while chaos emerges where both agreement and certainty are low. Complexity is a hybrid concept located between order (systems) and chaos (disorder), partly ordered and partly disordered, containing both of them but also overcoming them at a higher level of synthesis.
CECAN adapted Stacey’s explanation by merging chaos and complexity, without providing any theoretical justification. Their ‘adaptation’ completely changes the nature of complexity by collapsing it from the mesoscopic understanding (between order and chaos) to a micro/macroscopic understanding — complexity vs order.
It turns out this was not a slip. The authors assume complexity emerges in two situations: either when an evaluated object is “Close to agreement — Far from certainty” in CAM or “Close to certainty — Far from agreement”. These two types of complexity refer to marginal cases with one foot always firmly stationed in certainty. In this way the authors attempt to avoid the real problem here — when neither agreement nor certainty can be assured. Stacey understands complexity as an area where both are achieved only to a constrained extent. This is where one must accept radical uncertainty with both feet, as equally important to order, instead of trying to outsmart it with good old scientific tricks.
Despite previous graphical illustrations of the concept, the authors decided not to define their core object of concern, explaining: “There is no single agreed definition of complexity, and to choose one would not do justice to the diversity of work in this area.” Therefore, they grant themselves the freedom to explain complexity as a reduced list of ‘widely agreed’ eleven properties. An evaluator must only recognise which type of complexity prevails in the evaluated object and then press the appropriate button that suits the challenge best. This request reveals that complexity is treated here as a sort of a complicated issue (in Stacey’s setting). Furthermore, the authors did not hide that some of the recommended approaches to complexity work better when the level of complexity in an evaluated object is lower.
Overall, the supplementary guide is undoubtedly valuable as an excellent literature review with a presentation of various relevant evaluation case studies dealing in diverse ways with specific aspects of complexity. However, what could be reasonably expected from any guide is a synthesis of diverse evidence and experiences from independent cases to find out what is similar in different approaches.
A supplementary guide is valuable for learning about complexity in policy impact evaluation, but I would very much hesitate to recommend it as a guide for practitioners. The challenge of complexity requires a shift of mind, not more of the same (Stacey). We need new approaches that are able to deal with complexity in a complex way. Only if complex challenges are understood in an original, mesoscopic way, can policy evaluation become simple and surprisingly intuitive despite the complexity of policy challenges.
Used sources:
- CECAN. 2020. Handling Complexity in Policy Evaluation — Supplementary Guide. Magenta Book. London, HM Treasury.
- Gleick J. 1987. Chaos: Making a new Science. Viking Press.
- Stacey R. D. 1996. Complexity and Creativity in Organizations. San Francisco, CA Berrett-Koehler Publishers.
