When facilitating decisions it sometimes happens that some inputs to the preference model either cannot be assessed at all or can only be assessed within relatively large bounds of uncertainty (e.g. [1,12,13]). This can happen for a number of reasons: a lack of time, a politically sensitive problem context, or a lack of decision maker (DM) involvement, for example. Whatever the reason, in these cases the DM is unable or unwilling to express him or herself with the degree of precision required by conventional decision aids. We call decision problems which must be addressed under such conditions “low-involvement” decisions. The question is what, if any, decision support can be provided in such situations.
Stochastic multicriteria acceptability analysis (SMAA [25,19]) is a family of decision models that can be used with arbitrarily precise preference information. It addresses low-involvement decision-making by providing information about the types of preferences (if any) that would lead to the selection of each alternative.
In this paper we use a simulation experiment to evaluate the ability of SMAA to approximate results obtained using multi- attribute utility theory (MAUT) where preferences are repre- sented by a multiplicative utility function. In particular, we ask how closely results computed from a key output from SMAA (the acceptability index) can approximate those obtained using MAUT.
☆This manuscript was processed by Associate Editor Doumpos.
n Corresponding author at: Department of Statistical Sciences, University of Cape Town, Rondebosch 7701, South Africa.
E-mail address: ian.durbach@uct.ac.za (I.N. Durbach).
http://dx.doi.org/10.1016/j.omega.2015.10.015
0305-0483/& 2015 Elsevier Ltd. All rights reserved.
abstract
This paper considers problem contexts in which decision makers are unable or unwilling to assess trade- off information precisely. A simulation experiment is used to assess (a) how closely a rank order of alternatives based on partial information and stochastic multicriteria acceptability analysis (SMAA) can approximate results obtained using full-information multi-attribute utility theory (MAUT) with multi- plicative utility, and (b) which characteristics of the decision problem influence the accuracy of this approximation. We find that fairly good accuracy can be achieved with limited preference information, and is highest if either quantiles and probability distributions are used to represent uncertainty.
   & 2015 Elsevier Ltd. All rights reserved.
In doing so we hope to provide a broad indication of the losses that are possible if facilitators choose to use a low-involvement decision aid such as SMAA rather than compelling DMs to be more precise in their assessment of certain types of preference infor- mation – for example, using more detailed problem structuring. We also wish to test the robustness of the SMAA approach to various aspects of the decision process: the size of the decision problem, the way attribute evaluations are distributed, the underlying preference functions, the accuracy of assessed infor- mation, the amount of preference information gathered, and the way in which the acceptability index is constructed.
In addition to the conventional SMAA model, which uses probability distributions to represent uncertainty in the attribute evaluations, we also introduce and evaluate a number of ‘simpli- fied’ models which make use of summarised measures of uncer- tainty instead of a full probability distribution [6]. By assessing the accuracy of both conventional and simplified uncertainty models under a range of different conditions we hope to provide moti- vation for the use of simplified models in appropriate circum- stances. A similar approach has been used in Durbach and Stewart [9] to assess the effect of using simplified uncertainty formats in general decision-making, and we employ a similar simulation structure in the current paper.
Our view is that in nearly all cases it is preferable to resolve preferential uncertainty through discussion and problem structur- ing rather than by employing more ‘lenient’ decision aids, because of the additional insight and opportunities for learning. We focus on those circumstances in which the DM is unable or unwilling to participate fully in this process. In using a simulation experiment, we acknowledge that we can only evaluate the extent to which
  
14 I.N. Durbach, J.M. Calder / Omega 64 (2016) 13–23
using SMAA rather than MAUT might impact results. We cannot evaluate critical issues like whether the reduced time spent on problem structuring in SMAA is “worth” the reduction in decision quality, or the degree to which the problem structuring process, through the insight it generates for the DM, is useful as an end in itself. Simulation results are unable to provide general conclusions on the viability of different methods, but provide inputs to such discussion by identifying the potential trade-offs in accuracy that are implied when using a simplified model. Ultimately accuracy must be weighed against other factors to determine which decision model may be most appropriate for a problem.
The remainder of the paper is structured as follows. Section 2 provides a review of the relevant background literature and notation. Section 3 describes the structure of our simulation experiment. Section 4 presents the results in a direct fashion, delaying a more detailed discussion of these until Section 5, which also provides theoretical justification for some key findings using results from applied probability theory. Section 6 discusses implications of the simulation results for the use of SMAA as a prescriptive decision aid, and concludes.
2. Notation and background
Consider a decision problem consisting of I alternatives fa1;a2 ,..., aIg evaluated on J attributes fc1;c2;...;cJg. Let Zij be a random variable denoting the attribute evaluation of ai on cj, and U be a multi-attribute utility function mapping the attribute evaluations of alternative ai (denoted Zi) to a real value using a weight vector w. A joint density function fXðZÞ governs the generation of the Zij in the space X D RI J , and a second joint density function gðwÞ governs the generation of imprecise or unknown weights in the weight space W. Total lack of knowledge is usually represented by a uniform dis- tribution in W. If restrictions have been placed on W we denote the feasible weight space by W0.
The original SMAA method [20] analysed the combinations of attribute weights that result in each of a set of alternatives being selected when using an additive utility function. Subsequently, a number of SMAA variants have been developed. These differ in terms of the preference model used and thus the type of pre- ference information that is imprecisely known, but are all based upon Monte Carlo simulation from distributions which govern unknown preference parameters (and attribute evaluations). For example, SMAA variants are available for value function [20,17], outranking [10], reference point [21,5], prospect theory [18], Choquet integral [2], and AHP [7] methods. Comprehensive reviews are given by Tervonen and Figueira [25] and Lahdelma and Salminen [19].
Given a particular weight vector w, the global utility of each alternative can be computed and a rank ordering of alternatives obtained. SMAA-2 [17] is based on simulating a large number of random weight vectors from gðwÞ and observing the proportion and distinguishing features of weight vectors which result in each alternative obtaining a particular rank r (usually the “best” rank, r1⁄41), using an additive value function model. Let the set of weight vectors that result in alternative ai obtaining rank r be denoted by Wir. SMAA is based on an analysis of these sets of weights using a number of descriptive measures, the most important of which are:
Acceptability indices: The rank-r acceptability index bir measures the proportion of all simulation runs, i.e. weight vectors, that make alternative ai obtain rank r. A cumulative form of the acceptability index called the R-best ranks acceptability index is defined as BRi 1⁄4 PRr 1⁄4 1 bri and measures the proportion of all weight vectors for which alternative ai appears anywhere in the best R ranks. In
the discussion in Section 5 we make use of ordered acceptability indices, where we denote the alternative with the k-th largest rank-r acceptability index by arðkÞ , and its acceptability index by brðkÞ . Central weight vectors: The central weight vector wci is defined as the expected center of gravity of the favourable weight space Wi1. It gives a concise description of the “typical” preferences sup- porting the selection of a particular alternative ai, and in practice is computed from the empirical (element-wise) averages of all weight vectors supporting the selection of ai as the best alternative.
The exact number of Monte Carlo iterations that are required to achieve a given accuracy is discussed in Tervonen and Lahdelma [26]. To estimate the acceptability index within δ of the true value with 95% confidence, one requires 1:962=4δ2 iterations – so that 10000 iterations will usually be sufficient to achieve error bounds of 1%.
Uncertain attribute evaluations are conventionally treated in SMAA using probability distributions, with each simulation run drawing values at random from these distributions. Adapting SMAA to use other uncertainty formats, however, is generally straightforward, as described in Durbach and Davis [6]. Each uncertain attribute is simply replaced by a number of lower-level attributes which capture the uncertainty in the evaluations on that attribute, using one of many possible simplified uncertainty for- mats. In this paper, we test the effect of using three different for- mats: expected values; variances; and quantiles. This transforms the decision problem into one having the same appearance as a deterministic decision problem, which can be treated by any of the existing SMAA models with some minor modifications. We sometimes refer to these collectively as “SMAA models” although it should be clear that these are approximations of external uncertainty built on top of the same SMAA approach. The models are described in more detail in Section 3. Table 1 provides a summary of notation used in the paper.
3. Description of the simulation study
The general structure of the simulation1 is summarised in Fig. 1 and implements the following basic steps:
1. Form a hypothetical problem context, generating the relevant attribute evaluations.
2. Apply a multiplicative MAUT model to derive idealised or ‘true’ utilities and thus find the ‘true’ rank ordering of alternatives.
3. Calculate summarised measures of uncertainty based on the
generated data and incorporating observational errors.
4. Run different models using SMAA and the inputs from step 3, and then compare the model results against the ‘true’ utilities
and rank order obtained from step 2.
3.1. Generating attribute evaluations
We consider a decision problem involving I alternatives evaluated over J attributes. External uncertainty about attribute performance is captured by simulating attribute evaluations for alternative ai on attribute cj from a gamma distribution with mean μij, standard deviation σij, and skewness ξij. We denote this distribution f ij ðZ ij Þ. Each mean μij is drawn randomly from a uniform distribution between 0.2 and 0.8 for all alternatives on all attributes. Means are then stan- dardised across all the attributes to lie on the unit hypersphere i.e.
1 All codes used to run the simulations described in this section are openly available from http://dx.doi.org/10.5281/zenodo.30523.
 
Table 1
Summary of all parameters used.
I.N. Durbach, J.M. Calder / Omega 64 (2016) 13–23 15 [15, Theorem 6.11]. These conditions allow us to rewrite the global
utility of alternative ai as a function of its global value Vi, and more
precisely as a function of its value as derived from an additive
 Problem context
ai
cj
cðkÞ
Zij
fXðZÞ fijðZijÞ or fij
Attribute weights
W W0
SMAA
bir brðkÞ arðkÞ B1ðkÞ BRi
Alternative i
Attribute j
Attribute with k-th largest importance weight Performance of ai on cj
Joint probability distribution over all Zij Marginal probability distribution for Zij
Total weight space
Currently feasible weight space Favourable weight space for ai
Weight for cj
k-th largest attribute importance weight
Rank-r acceptability index
k-th largest rank-r acceptability index
Alternative with k-th largest rank-r acceptability index Random variable which when realised gives b1ðkÞ
Cumulative R-best ranks acceptability index
value model:
Wi wj wð1Þ
where wj denotes the relative importance weight of attribute cj, vj is the marginal value function for cj, and ζ determines whether the DM is constant risk averse ðζ 4 0Þ, risk neutral ðζ 1⁄4 0Þ or constant risk prone ðζo0Þ with respect to value. The precise form of the marginal value function vj is discussed in Section 3.2.1. To calculate the ‘true’ utility of an alternative ai, expectations are taken over the full set of K realisations. That is, we (a) compute, for each k, the value Vik and hence utility Uik of alternative ai using attribute
evaluations zi1k ; ...; ziJk ; (b) average these utilities over k. This gives 23
1
>: j j ij j1⁄41
Simulation: problem context
I Number of alternatives
J Number of attributes
K Number of simulated realisations of Zij 8
zijk Realisation k of Zij >
  ζ V
XK XJ i   1⁄4 ð1=KÞ exp4 ζ
k1⁄41 j1⁄41
XK XJ   exp4 ζ k1⁄41 j1⁄41
> μij Mean of attribute evaluations for ai on cj >
E1⁄2e
wjvjðzijkÞ5
ζo0
ζ 1⁄4 0
ζ40 ð2Þ
σij Standard deviation of attribute evaluations for ai on cj > ξij Skewness of attribute evaluations for ai on cj ><
UMAUT 1⁄4 Simulation: DM preferences i >
vj Marginal value function for cj >
τj Reference level for vj
λj Value of vj at reference level
Vi Global value of ai
Ui Global utility of ai
ζ Risk preferences defined over value ω ‘True’ attribute weights
Simulation: uncertainty modelling
XK XJ k 1⁄4 1j 1⁄4 1
wjvjðzijkÞ
ρ ε
z0ij
z′′ij
E1⁄2Zij , E^1⁄2Zij  VAR1⁄2Zij , VAR1⁄2Zij  Qr1⁄2Zij , Q^ rðZijÞ
φ
Proportion of importance ranks provided by DM Size of assessment errors
Sampled value from zij: used by ‘TDist’ model Sampled value from fij used by ‘EDist’ model
This ‘true’ measure of utility is based on perfect and complete information and, for the purposes of this simulation study, pro- vides a set of idealised results to which results from the different SMAA models can be compared.
3.2.1. Marginal value functions
Marginal value functions are assumed to be piecewise linear between the points ð0; 0Þ, ðτj ; λj Þ, and ð1; 1Þ. Convex, linear, and concave value functions can be generated by imposing τj 4 λj , τj 1⁄4 λj , and τj o λj respectively, allowing a wide range of preference to be simulated parsimoniously. Note that this is a simplified version of the approach used in Durbach and Stewart [9], where additional parameters were used to control the degree of curva- ture in the preference functions either side of τj. Marginal values are given by:
8> λ x
> j for 0rxrτj < τj
vjðxÞ1⁄4 ð3Þ >:λj þ1 λj ðx τjÞ for τj oxr1
^
Expected value of Zij before and after application of ε Variance of Z before and after application of ε
ij
5%, 50% and 95% quantiles of Zij for r 1⁄4 1; 2; 3 before and
after application of ε
Weight of VAR1⁄2Zij  relative to E1⁄2Zij 
823 > ζV XJ
>e i 1⁄4exp4 ζ wjvjðZijÞ5 > j1⁄41
> J
ζo0
ζ1⁄40
ζ40
<V 1⁄4XwvðZ Þ Ui1⁄4 >i jjij
ð1Þ
>j1⁄41 2 3 > XJ
45 > e ζVi 1⁄4 exp  ζ wvðZ Þ
E1⁄2Vi  1⁄4 ð1=KÞ >: E1⁄2e ζVi 1⁄4ð1=KÞ
23
>
wjvjðzijkÞ5
 PJ μ2 1⁄4 1; 8 i. This creates a set of non-dominated alternatives, at j1⁄41 ij
least in this ‘average’ sense. The standard deviations and skewnesses of the attribute evaluations are varied as parameters of the simulation (see Section 3.6 for details of chosen parameters values). A set of K 1⁄4 15 000 realisations is generated to represent the uncertain attribute evaluation of each alternative on each attribute. These evaluations, denoted zijk, are then standardised within each attribute (across all alternatives and realisations), to have a minimum of 0 and a maximum of 1.
3.2. Generating preference structures
We employ a multiplicative MAUT model to simulate the ‘true’ underlying preference structure of a DM, with Ui denoting the overall utility of alternative ai. For simplicity we assume that our simulated DM has a utility function that exhibits constant risk preferences over the scalar value attribute Vi. Specifically, this requires that the DM's preferences over the attribute space are compatible with an additive value function, that at least one attribute is utility independent of its complement, and that J Z 3
1 τj
  3.2.2. Inter-attribute weights
‘True’ attribute weights ω are simulated to be uniformly dis- tributed and sum to one, using the approach in Butler et al. [4]. This involves sorting J 1 randomly generated Un1⁄20;1  variates in ascending order, appending 0 and 1 to the extremes of this sequence, and extracting the differences between adjacent values.
3.3. Simulating the application of SMAA
The ‘true’ attribute weights ω provide a rank ordering of attributes from most important to least important. Let cðkÞ denote