Defining the analytical complexity of decision problems under uncertainty based on their pivotal properties

Two contrastive examples

To motivate the development of our framework, we will review two simplified examples of uncertainty from the large list of scenarios underlying this study. The first example is the problem originally considered by both Knight and Keynes of entrepreneurial investment (Keynes, 1921a; Knight, 1921). An investor is seeking to allocate funds among firms and the goal is to maximize the long-term value of the total investment portfolio. Suppose the business models or technologies are fundamentally new and so there is little quantitative basis for forecasting profitability any determinants of profitability, e.g. whether the market will see value in the products, whether the firms will be well-managed, or whether the economic and financial conditions will be favorable over the next 10 years. General benchmarks and trends are of course available for such investments, but they suffer from high dispersion and provide little specific guidance for choosing investments. For a second example problem, consider the resource allocation problem of a manager of a small health department. Novel pandemics are rare and hard to predict (Marani et al., 2021) creating a challenging planning problem. For more concreteness, suppose both decision-makers have wide discretion over a budget of 10 million dollars that must be allocated over 2 years and both must choose between several firms or several projects in the health department, respectively.

Contrasting the disparate problems in these scenarios suggests two properties of decision problems that we will call pivotal. While investment decisions are nearly entirely based on a single objective (future capital value), pandemic decisions must weigh multiple objectives (e.g. mortality, morbidity, economic impacts) and multiple stakeholders (e.g. different demographic or political groups). Secondly, risk in financial decisions can often be transferred to another party–for example, by pooling resources with another investor. In pandemic preparedness the major risks to health cannot be transferred in a similar sense.

Yet the two scenarios are similar in some surprising ways. Uncertainty in both scenarios, one might argue, could potentially be reduced through the acquisition of knowledge, i.e. the problems are Learnable: e.g. research in the technology or hiring expert trainers to educate the staff could fruitfully inform the decisions. This property should not be taken for granted because in some situations we lack a clear guide as to where our knowledge gaps are, or we might lack the ability to carry out a study since the system is too complex, inaccessible with current methods, or not subject to experimentation, as occurs in complex systems in fields such as public policy. Indeed, both problems are also what might be called well-understood: there is a body of scientific and practical knowledge around both investment and infectious diseases.

It is also relatively easy, one might argue, to find a feasible solution for the allocation of resources, i.e. both problems are easily Satisficeable. By contrast some problems are much harder to satisfice (e.g. building a rocket that will reach orbit or guarantee at least 10% annual return). Additionally, in what could be called indexability, the outcomes in both situations could be expressed adequately using random variables (e.g. the market value of the investment, the number of infections and a few others). Indexability should be distinguished from quantification (e.g. the ability to model or predict): the price of a company’s stock indexes the value of the investment but does not eliminate the problem of economic forecasting or black swan events. Indexability enables estimation of volatility and an interactive trial-and-error approach to decisions until the outcome is improved (even if the system is difficult to study or model).

Lastly, the decision-makers in both problems are not mere passive observers to the phenomenon but are like engineers and have a large ability to configure or reconfigure the solution. They can hire staff, design procedures, prepare contingency plans and so on. This Configurability property is used extensively in engineering, management and other settings to make physical systems or organizations robust to uncertainty.

Generalizing from these observations, we will attempt now to identify a comprehensive list of pivotal properties. With these properties we will define analytical complexity of a problem, i.e., how hard it would be for a decision-maker or analyst to find a satisficing answer to the problem?

Setting

Generalizing from the above examples, we are interested in characterizing a broad range of problems affected by a high degree of uncertainty and identifying properties that could aid in their classification according to a scale of analytical complexity. The problems of interest are all problems studied in decision theory and risk analysis: individual choices and large-scale societal problems; decisions with stakes both low and high. The goal of the problem might be optimization of an outcome subject to uncertainty or reduction of risk where the risk is difficult to quantify.

To be a little more precise, we will utilize the states-actions-outcomes framework (Borgonovo et al., 2018). The decision-maker (or makers) face an uncertain current state of the world, $s\in S$ and have to choose between actions $a\in F$ from a set that might be infinite. An action is a function that maps states of the world to outcomes (also called objectives): $G_a\left(s\right)\in O$. Our decision-makers are often unaware of the possible actions and outcomes (Steele & Orri Stefansson, 2021), unlike in classical settings. The action set might be very small (e.g. choosing between two options) or very large (as happens in problems of system design). In most cases we will think of the action as occurring at a single point in space and time but in certain situations we face multi-stage problems as well as an infinite sequence of decisions. In most problems, uncertainty about the state of the world $s$ could be decomposed into two factors: $Y$-the state of the system managed by the decision-maker (a vector-valued random variable) and $X$-context external to the system (another vector-valued random vector). The desired knowledge might be about these variable’s present or future state, or certain performance outcomes that are functions of these variables. Many of the problems involve limiting the undesirable effects of $X$ on $Y$. Lastly, when speaking of risk we are referring to the qualitative notion of possibility of unfortunate occurrence (Aven et al., 2018).

Solving a problem here usually means finding a satisficing solution in the tradition of Herbert Simon (Simon, 1955; Artinger, Gigerenzer & Jacobs, 2022). Only in exceptional cases would the solution be optimal in any sense of the word. The pivotal property of a problem is defined as that property of the problem (i.e., context, system or outcome) that could assist a decision-maker to solve the problem.

Data source

The basis for this work is two lines of research. The first, coming from behavioral economics, is a list of decision heuristics being used in diverse fields (Kahneman, Slovic & Tversky, 1982; Gigerenzer & Goldstein, 1996; Artinger, Gigerenzer & Jacobs, 2022). The second, a catalog built by this author and expanding the work of Todinov, (2015) is a list of strategies used by practitioners in fields such as engineering, management and others for decision making and risk mitigation (Gutfraind, 2023). Combining these lines of research resulted in a list of over 90 diverse strategies ranging from multi-layered defense popular in engineering to event tree analysis used in decision analysis¹ . The strategies were sourced from articles, monographs and interviews. The author also contributed examples from personal experience, primarily in engineering, software development, consulting and counter-terrorism. Important examples came from monographs across diverse fields: military science (Freedman, 2015), project management (Flyvbjerg & Gardner, 2023), safety and reliability engineering (Stamatis, 2014), software devops and security (Howard, Curzi & Gantenbein, 2022) and biology (Ellner & Guckenheimer, 2011). Additionally, the author hired anonymous assistants through the site fancyhands.com. They were tasked to list examples of problems they solved in their daily lives (all details anonymized). The solution strategies were then included in the list, if they presented a general-purpose technique for uncertainty.

Identification of properties

Based on this knowledge base the next step was to find pivotal properties. The process was entirely qualitative and several prompting techniques were used. The first approach was to review each of the strategies in the knowledge base and to ask which properties must the setting have in order for the technique to be applicable to a new domain. It was generally quite clear what some of the properties must be. The second approach was to contrast two problems and ask, what are essential differences in the problem settings. The third technique was to ask, for a given problem, what parameters could make the problem easier or harder. The fourth step was to review the emerging list of properties and consolidate closely related properties into a group with a common definition. Because each problem has many unique properties, a property was only included in the list below if, (1) it was logically necessary for applying a solution heuristic and, (2) it was shared by two or more problems or strategies coming from different application domains. Finally, the properties were put into clusters.

Pivotal properties

Analysis of the knowledge base produced 19 pivotal properties that are listed below in alphabetical order. The properties could be organized into four loose clusters based on what they relate to: the problem objectives, the action space, the role of time, and the nature of uncertainty. To clarify the meaning of each property, we list solution heuristics (i.e. strategies) that depend on it.

Properties related to the problem objectives

Properties related to the action space

Temporal properties

Properties of the uncertainty

Well-understood

Definition: A body of knowledge exists to describe the phenomenon and problem such as scientific understanding, data or human experience.

Solution methods: System models (mathematical, computational, predictive), maximization of expected utility, decision templates, expert elicitation and judgment (Meyer & Booker, 2001).

For each property Fig. 1 lists the total number of dependent solution strategies. Because the assignment is a matter of judgment, the count should be viewed as suggestive of the importance of the property.