PyCID: A Python Library for Causal Influence Diagrams

Why did a decision maker select a certain decision? What behaviour does a certain objective incentivise? How can we improve this behaviour and ensure that a decision-maker chooses decisions with safer or fairer consequences? This paper introduces the Python package PyCID, built upon pgmpy, that implements (causal) influence diagrams, a widely used graphical modelling framework for decision-making problems. By providing a range of methods to solve and analyse (causal) influence diagrams, PyCID helps answer questions about behaviour and incentives in both single-agent and multi-agent settings.


Introduction
Influence-diagrams (IDs) are used to represent and analyse decision making situations under uncertainty [HM05], [MIMH + 76]. Like Bayesian Networks, IDs have at their core a directed acyclic graph (DAG), but IDs also specify decision and utility nodes. Relationships between variables are given by conditional probability distributions. When these are specified, we call it an influence model (IM). In an IM, a decision-maker selects a distribution over its available actions at a decision (a decision rule) based on what it knows (the values of its parents in the ID) to maximise its expected utility. To demonstrate, consider the following example: Grade Prediction: To decide who to admit, a university uses a model to predict the grades of applicants based on information in their application forms.  shows the DAG for this example, which displays clearly the structure of the decision situation. The decision being made by an agent, the model, is the grade prediction (decision node). The agent selects a decision rule for this decision, based on information about the applicant (chance node), in order to optimise their prediction accuracy (utility node). The edges denote associational relationships in the case of a statistical IM, but denote causal links in causal influence models (CIMs). This difference in semantics [ECL + 21] allows one to use CIMs to query the effect of causal interventions and provides a setting to ask counterfactual questions [Pea09]. (C)IMs have also been extended to multi-agent settings by [KM03], [HFE + 21], and [HFE + ].
Statistical and causal IDs have shown promise for a wide variety of applications. In business and medical decision making, statistical IDs provide a simple yet powerful model for optimising decisions by making assumptions explicit and revealing what information is relevant [Góm04], [KM08]. Moreover, for the design of safe and fair AI systems, causal IDs have been used to help predict the behaviour of agents arising due to their incentives in an environment [ECL + 21], [CLEL20], [EHKK21], [Hol20], [EKKL19], [LE21], and [CVH20]. Nevertheless, although Python libraries exist for Bayesian networks, perhaps most prominently pgmpy [AP15], these libraries lack specific support for IDs. We found two Python wrappers of C++ influence diagram libraries: pyAgrum [DBDSMW20] and PySMILE [Bay]. These were limited by usability (hard to install), maintainability (using multiple languages) and versatility (they did not cover multi-agent or causal IDs). A Python library that focuses on implementing statistical and causal IDs is therefore needed to ensure their potential application can be explored, probed, and fully realised.
Consequently, this paper introduces PyCID 1 , a Python library built upon pgmpy [AP15] and NetworkX [HSS08], which implements IDs and IMs (including their causal and multi-agent variants) and provides researchers and practitioners with convenient methods for analysing decision-making situations. PyCID can solve single-agent (C)IMs, find Nash equilibria in multiagent (C)IMs, and compute the effect of causal interventions in CIMs (e.g., fixing the prediction model in Figure 1 to always predict a high grade regardless of the applicant's information). PyCID can also find which variables in an ID admit incentives. For example, positive value of information [How66] and value of control [Sha86] tell us when an agent can benefit from observing or controlling a variable. Meanwhile, other incentives concepts, recently proposed in [ECL + 21], reveal which variables it can be instrumentally useful to control and when a decision-maker benefits from responding to a variable. Reasoning patterns are a related concept in multi-agent IDs: they analyze why a decisionmaker would care about a decision [PG07], and these can also be computed in PyCID.
The first two sections of this paper provide the necessary background on (C)IDs and describe the architecture of the Py-CID library. We then move to showcasing some of PyCID's features through applications for discovering agent incentives and analysing games. In the * Instantiating Causal Influence Diagrams* section, we demonstrate how to instantiate a (C)ID for the Grade Prediction example in PyCID. In the Analysing Incentives section, we demonstrate how to find the nodes which admit value of information, response, value of control, or instrumental control incentives for more complex (C)IDs. We then turn to multi-agent (C)IDs (MA(C)IDs) and show how to use PyCID to compute Nash equilibria. Next, we explain how PyCID can construct random (MA)CIDs. Finally, we discuss the future of PyCID.

Notation
Throughout this paper, we will use capital letters, X, for random variables and let dom(X) denote their domain. An assignment x ∈ dom(X) to X is an instantiation of X denoted by X = x. X = {X 1 , . . . , X n } is a set of variables with instantiation x = {x 1 , . . . , x n }. We also let Pa V denote the parents of a node V in a (MA)CID and pa V be the instantiation of Pa V . Moreover, we define Desc V and Fa V := Pa V ∪ {V } to be the descendants and family of V . We use subscripts to index the elements of a set and, in a multi-agent setting, superscripts to indicate a player i ∈ N; e.g., the set of decisions belonging to player i is

Causal Influence Diagrams
A Bayesian network is a model consisting of a directed acyclic graph (DAG) and a joint distribution that is Markov compatible with that graph [Pea09]. The nodes in the DAG denote random variables and the directed edges represent the associational relationships between them. To parameterise the DAG and encode the joint distribution, each random variable, V , in the DAG is assigned a conditional probability distribution (CPD), P(V |Pa V ), dependent on its set of graphical parents, Pa V . Taken together, these CPDs define the Bayesian network's joint distribution. A causal Bayesian network is a Bayesian network where the directed edges in the DAG now represent every causal relationship between the Bayesian network's variables. This enables the model the ability to answer questions about the effect of causal interventions from outside of the system.
Causal Influence Diagrams (CIDs) are DAGs where the nodes are partitioned into chance, decision, and utility nodes and the edges adopt the same causal semantics as causal Bayesian networks [ECL + 21]. Causal Influence models (CIMs) are parameterised CIDs where, at the outset, the CPDs for chance and utility nodes are defined, but only the domains for the decision variables are fixed.
Definition 1 [ECL + 21] A Causal influence Diagram (CID) is a directed acyclic graph (V, E) where the set of vertices (V) connected by directed edges (E ⊆ V × V) are partitioned into chance (X), decision (D), and utility (U) nodes. Utility nodes lack children.
Definition 2 [ECL + 21] A Causal influence Model (CIM) is a tuple (V, E, θ ) where (V, E) is a CID and θ ∈ Θ is a particular parametrisation over the nodes in the graph specifying for each 1. This paper describes PyCID version 0.2.6. node V ∈ V a finite domain dom(V ), for each utility node U ∈ U a real-valued domain dom(U) ⊆ R, and for every chance and utility node a conditional probability distribution (CPD) P(V | Pa V ).
Multi-agent Causal Influence Diagrams (MACIDs) partition decision and utility nodes further into sets associated with each agent. In a (MA)CID, a decision rule, π D (D|Pa D ), is a probability distribution over the actions available at decision node D conditional on the value of its parents in the graph, Pa D . A policy, π i , assigns decision rules to all of agent i's decision nodes, and, in a MACIM, a policy profile, π, assigns policies to every agent. In a (MA)CID, each agent i's expected utility, U i M (π), under a policy (profile) π is the sum of the expected values of their utility nodes.

Package Architecture
In this section, we outline the structure ( Figure 2) and describe the key classes of the PyCID library 2 . PyCID is under continual development and so one can install the latest developmental package using a git checkout from the PyCID repository on GitHub: https://github.com/causalincentives/pycid.

2.
PyCID is under continued development, so more features will be added over time. Any updated documentation may be found in the repository's README file.

Classes Inherited from pgmpy
PyCID's key classes inherit from pgmpy's BayesianModel, TabularCPD, and BeliefPropagation classes [AP15]. The BayesianModel class represents a Bayesian network and CPDs are assigned to each random variable in the model using instances of the TabularCPD class. These CPDs define the Bayesian Network's joint distribution and the BeliefPropagation class is then used to perform probabilistic inference on a BayesianModel object; for instance, one can query the probability that node V takes value v given some instantiation of other variables in the DAG (known as a context).
The pycid.core module PyCID's base class is CausalBayesianNetwork. This class inherits from pgmpy's BayesianModel and represents a causal Bayesian network. In particular, it extends BayesianModel by adding the ability to query the effect of causal interventions. It also adds methods for determining the expected value of a variable for a given context (again under an optional causal intervention) and for plotting the DAG of the Causal Bayesian Network using NetworkX [HSS08]. CPDs for a CausalBayesianNetwork object can be defined using pgmpy's TabularCPD class, but we also allow relationships to be specified more directly with stochastic functions (under the hood, these are implemented via a StochasticFunctionCPD class). This can be used to specify relationships between variables with a stochastic function, rather than just with a probability matrix (see the Instantiating Causal Influence Diagrams section). CausalBayesianNetwork also has an inner class, Model, which keeps track of CPDs and domains for all CausalBayesianNetwork objects' variables in the form of a dictionary.
The MACIDBase class, which inherits from CausalBayesianNetwork, provides the underlying methods necessary for single-agent and multi-agent causal influence diagrams. The class includes methods for determining the expected utility of an agent, for finding optimal decision rules and policies, and for finding various new graphical criteria defined in influence diagrams (e.g. r-relevance).
CID and MACID are classes, inheriting from MACIDBase, that represent single-agent and multi-agent (C)IDs and are the models of most concern in PyCID. They include methods for finding the optimal policy for an agent in a (C)IM and for finding Nash equilibria [N + 50] and subgame perfect Nash equilibria [Sel65] in a MA(C)IM. It is important to highlight here that statistical (i.e., non-causal) single-agent and multi-agent influence diagrams can also be defined as CID and MACID objects using PyCID. In their case, all class methods are permitted except those that involve causal interventions.
The pycid.core module also contains functions that exploit relationships between the (MA)(C)ID's variables such as finding all (active) (directed) paths between variables and classes that find the relevance graphs [KM03] associated with MACIDBase objects.

PyCID's other modules
The pycid.analyse module includes functions for determining incentives in (C)IDs [ECL + 21], reasoning patterns in MA(C)IDs [PG07], and a function for computing the total effect of intervening on a variable with different values. pycid.examples contains pre-implemented (C)IDs and MA(C)IDs, whilst pycid.random contains functions for generating random (C)IDs and MA(C)IDs. pycid.notebooks contains jupyter notebooks with demonstrations of how to use the codebase; these can also be run directly as Colab notebooks. Finally, pycid.tests houses unit tests for all functions and public class methods.

Instantiating Causal Influence Diagrams
Having covered PyCID's basic library structure, the remaining sections will demonstrate some use cases. We begin, in this section, by instantiating the structure of the simple (C)ID given in the introduction (Figure 1). For many purposes, including finding incentives, the graph is enough for analysis.
A (C)ID for the Grade Prediction example is created as an instance of our CID class. Its initializer takes a list of edges as its first argument and then two more lists specifying the (C)ID's decision and utility nodes. All other nodes introduced in the edge pairs, which are not decision or utility nodes, are chance nodes. For conciseness, we abbreviate and use P to denote the prediction model's decision node, A for the applicant's information, and Ac to denote the accuracy of the predictions: import pycid cid = pycid.CID( [("A", "P"), ("A", "Ac"), ("P", "Ac")], decisions=["P"], utilities=["Ac"], ) cid.draw() The CID class method, draw, plots this (C)ID ( Figure 3) with a node colour and shape convention that matches what is given in Figure 1's legend. To then parameterise this (C)ID as a (C)IM by adding a domain for P and CPDs for A and Ac, we pass keyword arguments to the add_cpds method: CPDs in PyCID can be instantiated directly as TabularCPD objects, but more often PyCID's StochasticFunctionCPD subclass is used. This provides multiple ways to easily specify how a chance or utility variable's CPD depends on its parents or follows some distribution; it then converts that expression into a TabularCPD object under the hood. On line 2 above, we assign variable A a discrete uniform distribution over its domain, dom(A) = {0, 1}; on line 3, we specify dom(P) = {0, 1}; and on the final line, we specify how the value of Ac depends on the values of its parents, A and P. Within the lambda function, other variables are referred to by their lower case form to denote that variable's instantiation. Using a CID class method, solve, we can now solve this (C)IM by finding the agent's optimal decision rule for P. This returns the following output, saying that the optimal decision rule for P is to choose action 0 (low grade prediction) when the value of A is 0 (the quality of the application is poor), and action 1 (high grade prediction) when the value of A is 1 (the quality of the application is high): If the agent behaves according to this optimal decision rule, we find that their expected utility is 1 using the code below; ex-pected_utility accepts optional dictionaries for specifying contexts and causal interventions: There are several other ways to specify CPDs for variables. For example, on line 1 below, the CPD for A is updated to now follow a Bernoulli(0.8) distribution and line 2 specifies that now Ac just copies the value of P with probability 0  [EHKK21]), and comparing the promise of different AGI safety frameworks [EKKL19]. We believe that PyCID can further mature these enquiries. PyCID currently finds the following incentives in singledecision CIDs using their graphical criteria: Intuitively, a variable has positive value of information (VoI) if a decision-maker would benefit (get more utility) from observing its value before making a decision: VoI Definition: For a CIM 4 M , and a node X ∈ V \ Desc D , let M X →D and M X→D be M modified by respectively removing and adding the edge X → D. The value of information for X is then max VoI has been applied to a wide array of problems in economics and computer science [BP16]. Although PyCID's function 3. Nodes can be specified further as admitting indirect or direct Value of Control.
4. This definition is also valid in (non-causal) statistical influence models.
quantitatative_voi returns the quantitative VoI of a variable in a CIM, for the remainder of this section we shall focus on its graphical criterion, which depends upon which nodes are requisite observations in the CID. Requisite Observation Graphical Criterion: Let U D ∈ U ∩ Desc D be the utility nodes downstream of D.
VoI Graphical Criterion: A single decision CID, G , admits VoI for X ∈ V \ Desc D if and only if X is a requisite observation in G X→D , the graph obtained by adding X → D to G .
To demonstrate how to find nodes that admit VoI using PyCID, we extend the Grade Prediction example given in the introduction: Extended Grade Prediction: [ECL + 21] The university wants to admit the brightest students using their grade prediction model, but doesn't want to treat students differently based on their gender (Ge) or race (R). The model uses the gender of the student and the high school (HS) they attended to make its grade prediction. We make the following assumptions: • Performance at university is evaluated by a student's grades (Gr) and this depends on the quality of education (E) the student received before university (which depends on the high school they attended).
• A student's high school is assumed to be impacted by their race, but not by their gender.
We want to know whether the predictor is incentivised to behave in a discriminatory manner with respect to the students' gender or race. A CID for this example is defined below: cid = pycid.CID( [ ("R", "HS"), ("HS", "E"), ("HS", "P"), ("E", "Gr"), ("Gr", "Ac"), ("Ge", "P"), ("P", "Ac"), ], decisions=["P"], utilities=["Ac"], ) PyCID finds that HS, E, and Gr can all have positive VoI for the predictor model (line 1). We can also display this visually ( Figure 4)  Our implementation of this example in PyCID has revealed that there exists a parameterisation of this setup (i.e., a CIM with the given CID) where the model would benefit from knowing the value of one or more of 'High School', 'Education', or the student's true 'Grade' before making a grade prediction.

Response Incentives (RI)
Response incentives (RI) are a related type of incentive and we explain how implementing them in PyCID can help improve the fairness of AI systems. A variable admits an (RI) if a decisionmaker benefits from making its decision causally responsive to the variable [ECL + 21] 6 . RI Graphical Criterion: A single decision CID, G , admits a response incentive on X ∈ X if and only if there is a directed path X D in the requisite graph 7 G req where G req is the result of removing from G all information links from non-requisite observations.
To demonstrate how to find the nodes which admit RIs, we will again consider the Extended Grade Prediction example. As we did with VoI, we can list all of the nodes that admit RIs in the CID (line 1) or we can display the result visually (line 2) with the result shown in Figure 5  Implementing CIDs in PyCID can help suggest how to improve the fairness of AI systems because [ECL + 21] argue that an RI on a sensitive attribute can be interpreted as problematic from a fairness perspective. A decision is considered counterfactually unfair if a change to a sensitive attribute, such as race or gender, would change the decision [KLRS17]. Therefore, an RI on a sensitive attribute indicates that counterfactual unfairness is incentivised; specifically, it implies that all optimal policies are counterfactually unfair. To mitigate this, [ECL + 21] propose redesigning the gradepredictor. By removing the predictor's access to knowledge about the student's high school (i.e., the edge HS → P ), there will no longer be an RI on a sensitive attribute. The following code trims the edge and shows that now no node admits an RI in the modified CID ( Figure 5 (Right)): cid.remove_edge('HS', 'P') cid.draw_property(lambda node: \ pycid.admits_ri(cid, 'P', node))

Value of Control (VoC) and Instrumental Control Incentives (ICI)
We now turn to Value of Control (VoC) and Instrumental Control Incentives (ICI) and show that implementing the latter in PyCID can help design safer AI systems. Intuitively, a variable has positive value of control (VoC) if a decision-maker could benefit from choosing that variable's value.
VoC Definition: For a CIM M , the value of control for a non-decision node X ∈ V \ D is max M g X denotes the CIM M after intervening on X with any CPD, g X , that respects the graph. 6. For a formal definition, we refer the reader to [ECL + 21]. 7. A requisite graph is also known as a minimal reduction, trimmed_graph, or d-reduction. VoC Graphical Criterion: A single decision CID, G , admits positive value of control for a node X ∈ V \ {D} if and only if there is a directed path X U in the requisite graph G req . Although VoC is a useful concept, it does not consider whether it is actually possible for an agent to control that variable. Therefore, [ECL + 21] introduce Instrumental Control Incentives, which can be intuitively understood as follows: if the agent got to choose D to influence X independently of how D influences other aspects of the environment, would that choice matter? In other words, is controlling X instrumentally useful for maximising utility? The graphical criteria for ICI in a single-decision CID is: ICI Graphical Criterion: A single decision CID, G , admits an instrumental control incentive on X ∈ V if and only if G has a directed path from the decision D to a utility node U ∈ U that passes through X.
To demonstrate how to find these incentives in PyCID, we introduce another example from [ECL + 21].
Content recommendation: An AI algorithm has the task of choosing posts (P) to show a user, to maximise the user's click rate (C). The designers want the algorithm to present content adapted to each user's original opinions (O) to optimize clicks; the algorithm does not know the user's true original opinions, so it instead relies on an approximate model (M). However, the designers are worried that the algorithm will use polarising content to influence user opinions (I) so that the user clicks more predictably: With RI, we showed that implementing CIDs in PyCID can aid the design of fairer systems; with ICI, we demonstrate how PyCID can be used to help design safer AI systems. First, we can use analogous functions to what we used for VoI and RIpycid.admits_voc_list(cid) and pycid.admits_ici_list(cid, 'P') -to find that O, M, I, and C can have positive VoC whilst I, P, and C admit ICI. From this, because I (influenced user opinions) admits an instrumental control incentive, we discover that the content recommender may seek to influence that variable to attain utility.
[ECL + 21] offer an alternative content recommender design that avoids this undesirable behaviour. Instead of being rewarded for the true click-through rate, the content recommender is rewarded for the clicks it would be predicted to have, based on a separately trained model of the user's preferences. The modified CID for this changed model is shown in Figure 6 c) where the old utility node C (actual clicks) has become PC (predicted clicks):

Multi-agent (Causal) Influence Diagrams
In this section, we will show how to instantiate MA(C)IDs/MA(C)IMs in PyCID and demonstrate a selection of methods for analysing games (strategic interactions between self-interested players) including strategic relevance [KM03] and finding Nash equilibria (NE) [N + 50].
Recall from the Background section that a Multi-agent Causal Influence Diagram/Model (MACID/MACIM) is a simple multiagent extension of a CID/CIM [HFE + ]. For our purpose, all that's important is that there is now a set of N agents and so the decision and utility nodes are partitioned into {D i } i∈N and {U i } i∈N to correspond to their association with a particular agent i ∈ N. We also again underline that the only difference between statistical multi-agent influence diagrams/models (MAIDs/MAIMs) and MACIDs/MACIMs is that the edges represent every causal relationship between the random variables chosen to be endogenous variables in the model, as opposed to just associational relationships. Nevertheless, because MACIDs subsume MAIDs (in the sense of Pearl's causal hierarchy [Pea09]), everything we can do in a MAID, we can also do in a MACID. Therefore, for the two examples we present here, MAIDs and MACIDs can be viewed as the same.
To serve as our example, we shall use the Prisoner's Dilemma, which is probably the best known simultaneous and symmetric two-player game: Prisoner's Dilemma: Two prisoners, suspected of committing a robbery together, are isolated and urged to confess. Each is concerned only with getting the shortest possible prison sentence for himself and must decide whether to confess without knowing his partner's decision. Both prisoners, however, know the consequences of their decisions. Each year spent in prison can be represented as -1 utility and so the payoff matrix for this game (or Normal form) is given in Figure 7. MA(C)IDs and MA(C)IMs are instantiated as MACID objects with identical syntax to CID objects except for there being multiple agents and so we can draw them in the same way. Figure 8 (Left) shows that in PyCID, consistent with (C)IDs, decision nodes are drawn as rectangles and utility nodes are drawn as diamonds; however, because we now have more than one player, we reserve colouring to denote agent membership: each agent is assigned a unique colour. Chance nodes remain as grey circle ( Figure 11): macid = pycid.MACID( [ ("D1", "U1"), ("D1", "U2"), ("D2", "U1"), ("D2", "U2"), ], # specifies each agent's decision and utility nodes. The following command tells us that the second player (agent) receives expected utility = -3 (i.e., they will spend 3 years in prison) given that player 1 decides to defect and player 2 decides to cooperate. This agrees with the payoff matrix in Figure 7: Strategic relevance is a useful concept for analysing decisions made in games; it asks which other decisions' decision rules need to be already be known before we can optimise a particular decision rule. [KM03] introduced the graphical criterion s-reachability for determining this from the graph: S-reachability Graphical Criterion: Another decision node Using PyCID, lines 1 and 2 below evaluate to True, which tells us that each decision strategically relies on the other; each prisoner would be better off knowing the other prisoner's policy before deciding on their own action. To show this visually, line 3 plots the MACID's relevance graph [KM03] (Figure 8 Right):

pycid.RelevanceGraph(macid).draw()
We now turn to finding NE in games. We use π A to denote player i's set of decision rules for decisions A ⊆ D i , given a partial policy profile π −A over all of the other decision nodes in a MA(C)ID, M . We write U i M (π A , π -A ) to denote the expected utility for player i under the policy profile π = (π A , π -A ).
Definition: [KM03] A full policy profile π is a Nash equilibrium (NE) in a MA(C)IM M if, for every player i ∈ N, In the Prisoner's Dilemma, there is only one NE and this involves both players defecting. We can then find that the expected utility for each agent is -2 under this NE joint policy profile: PyCID can also be used to find subgame perfect equilibria (SPE) [Sel65]. A SPE is a NE where no player makes a non-credible threat -an action that, if the player is rational, they would never actually carry out.
Definition: [HFE + 21] A full policy profile π is a subgame perfect equilibrium (SPE) in a MA(C)IM M if π is an NE in every MAIM subgame 8 of M .
The Prisoner's Dilemma MAIM has no proper MAIM subgames and so the NE we found above is (trivially) also a SPE. Therefore, to demonstrate how PyCID distinguishes between NE and SPE, we use the following example: Taxi Competition: Two autonomous taxis, operated by different companies, are driving along a road with two hotels located next to one another -one expensive and one cheap. Each taxi must decide (one first, then the other) which hotel to stop in front of, knowing that it will likely receive a higher tip from guests of the expensive hotel. However, if both taxis choose the same location, this will reduce each taxi's chance of being chosen by that hotel's guests. The payoffs for each player are shown in Figure 9 and the MACIM for this example is instantiated in PyCID below   This MA(C)IM has three pure NE, which are found using macid.get_all_pure_ne(). We can also find the decision nodes in

Random (C)IDs and MA(C)IDs
PyCID has other features that can be useful for researchers. In particular, the library contains functions for instantiating random (MA)(C)IDs. This is useful for estimating the average properties of graphs, or for finding a counterexample to some conjecture. The first example below finds and plots a random 10-node, singleagent (C)ID with two decision nodes and three utility nodes. The second example finds and plots a random 12-node MA(C)ID with two agents. The first agent has one decision and two utility nodes, the second agent has three decisions and two utility nodes. In both these examples, we set the add_cpds flag to False to create non-parameterised (MA)(C)IDs. If one sets this flag to True, each chance and utility node is assigned a random CPD, and each decision node a domain to instantiate a (MA)CIM. One can also force every agent in the (MA)(C)ID to have sufficient recall; an agent has sufficient recall if the relevance graph restricted to include just that agent's decision nodes is acyclic. This can be useful for certain incentives analyses [vMCE]. The edge_density and max_in_degree parameters set the density of edges in the (MA)(C)ID's DAG as a proportion of the maximum possible number (n × (n − 1)/2) and the maximum number of edges incident to a node in the DAG. To find a (MA)(C)ID that meets all of the specified constraints, PyCID uses rejection sampling and so max_resampling_attempts specifies the number of samples to try before timing out:

Conclusions and Future Directions
PyCID is a Python library for solving and analysing single-agent and multi-agent (causal) influence diagrams. Several key classes -CausalBayesianNetwork, CID, and MACID -enable decision problems to be solved and the effects of causal interventions to be studied whilst PyCID's analysis functions can find graphical properties such as incentives in CIDs and reasoning patterns in MACIDs. This makes PyCID a customizable, but powerful library for testing research ideas and exploring applications. Moreover, implementing examples programmatically can substantiate the claims made by ID researchers about the benefit of their work; one can assess how different quantities vary over the parameter space or empirically verify complexity results [HFE + ]. Singleagent and multi-agent (causal) influence diagrams are an area of active research, so as theory develops, the PyCID library will also grow. Extensions will likely include: • Support for finding incentives in multi-decision CIDs [vMCE].
• Support for Structural Causal Models [Pea09] and therefore also quantitative RI and ICI.
• Support for multi-agent incentives.
In this paper, we have demonstrated the usefulness of PyCID by focusing on causal influence diagrams; however, this library is also well suited for working with statistical influence diagrams. The development team would like to invite researchers from any domain to use PyCID to test the package for diverse applications, to contribute new methods and functions, and to join our Causal Incentives Working Group: https://causalincentives.com/. The Py-CID repository is available on GitHub under our working group's organization: https://github.com/causalincentives/pycid.