When dealing with complex systems, all decision making occurs under some level of uncertainty. This is due to the physical attributes of the system being analyzed, the environment in which the system operates, and the individuals which operate the system. Techniques for decision making that rely on traditional probability theory have been extensively pursued to incorporate these inherent aleatory uncertainties. However, complex problems also typically include epistemic uncertainties that result from lack of knowledge. These problems are fundamentally different and cannot be addressed in the same fashion. In these instances, decision makers typically use subject matter expert judgment to assist in the analysis of uncertainty. The difficulty with expert analysis, however, is in assessing the accuracy of the expert's input. The credibility of different information can vary widely depending on the expert’s familiarity with the subject matter and their intentional (i.e., a preference for one alternative over another) and unintentional biases (heuristics, anchoring, etc.). This paper proposes the metric of evidential credibility to deal with this issue. The proposed approach is ultimately demonstrated on an example problem concerned with the estimation of aircraft maintenance times for the Turkish Air Force. 1. Introduction Real-world decision making is always performed under uncertainty. This uncertainty is present in the physical attributes of the system being analyzed, the environment in which it operates, and the individuals which operate the system. Decision makers must make decisions which best incorporate these uncertainties. With some problems, such as determining the probability of a terrorist attack on a given target, assigning probabilistic estimations to uncertain parameters is impossible due to the lack of statistical evidence upon which to base probabilistic estimates. Given these complex problems, decision makers often solicit subject matter expert opinion to provide estimates on uncertain parameters within a model. While this is a valid approach, soliciting expert opinions introduces additional uncertainty due to the varying degree of knowledge of the expert about the subject matter (i.e., one individual may truly be the world renowned expert in a field whereas others are merely seasoned practitioners). Additionally, as human beings, they have the potential for intentional and unintentional biases. The challenge when performing this type of analysis, in which expert judgment is essential to address uncertainty, is in assigning “weights” to the
References
[1]
N. Bohr, “Quantum postulate and the recent development of atomic theory,” Nature, vol. 121, no. 3050, pp. 580–591, 1928.
[2]
H. Kudak and P. Hester, “Application of Dempster-Shafer theory in aircraft maintenance time assessment: a case study,” Engineering Management Journal, vol. 23, no. 2, pp. 55–62, 2011.
[3]
J. C. Helton and D. E. Burmaster, “Treatment of aleatory and epistemic uncertainty in performance assessments for complex systems,” Reliability Engineering and System Safety, vol. 54, no. 2-3, pp. 91–94, 1996.
[4]
W. L. Oberkampf, “Uncertainty quantification using evidence theory,” in Proceedings from the Advanced Simulation & Computing Workshop, Albuquerque, NM, USA, 2005.
[5]
W. L. Oberkampf and J. C. Helton, “Evidence theory for engineering applications,” in Engineering Design Reliability Handbook, E. Nikolaidis, D. M. Ghiocel, and S. Singhal, Eds., CRC Press, Boca Raton, Fla, USA, 2005.
[6]
W. L. Oberkampf, S. M. DeLand, B. M. Rutherford, K. V. Diegert, and Alvin, “Error and uncertainty in modeling and simulation,” Reliability Engineering and System Safety, vol. 75, no. 3, pp. 333–357, 2002.
[7]
W. L. Oberkampf, K. V. Diegert, K. F. Alvin, and B. M. Rutherford, “Variability, uncertainty, and error in computational simulation,” in ASME Proceedings of the 7th AIAA/ASME Joint Thermophysics and Heat Transfer Conference, vol. 2, pp. 259–272, 1998.
[8]
H. Agarwal, J. E. Renaud, E. L. Preston, and D. Padmanabhan, “Uncertainty quantification using evidence theory in multidisciplinary design optimization,” Reliability Engineering and System Safety, vol. 85, no. 1–3, pp. 281–294, 2004.
[9]
J. Darby, Evaluation of Risk from Acts of Terrorism: The Adversary/Defender Model Using Belief and Fuzzy Sets (SAND2006-5777), Sandia National Laboratories, Albuquerque, NM, USA, 2006.
[10]
J. C. Helton, “Treatment of uncertainty in performance assessments for complex systems,” Risk Analysis, vol. 14, no. 4, pp. 483–511, 1994.
[11]
J. C. Helton, “Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty,” Journal of Statistical Computation and Simulation, vol. 57, no. 1–4, pp. 3–76, 1997.
[12]
F. O. Hoffman and J. S. Hammonds, “Propagation of uncertainty in risk assessments: the need to distinguish between uncertainty due to lack of knowledge and uncertainty due to variability,” Risk Analysis, vol. 14, no. 5, pp. 707–712, 1994.
[13]
M. E. Paté-Cornell, “Uncertainties in risk analysis: six levels of treatment,” Reliability Engineering and System Safety, vol. 54, no. 2-3, pp. 95–111, 1996.
[14]
S. N. Rai, D. Krewski, and S. Bartlett, “A general framework for the analysis of uncertainty and variability in risk assessment,” Human and Ecological Risk Assessment, vol. 2, no. 4, pp. 972–989, 1996.
[15]
W. D. Rowe, “Understanding uncertainty,” Risk Analysis, vol. 14, no. 5, pp. 743–750, 1994.
[16]
A. P. Dempster, “Upper and lower probabilities induced by a multivalued mapping,” The Annals of Statistics, vol. 38, no. 2, pp. 325–339, 1967.
[17]
G. Shafer, A Mathematical Theory of Evidence, Princeton University Press, Princeton, NJ, USA, 1976.
[18]
G. J. Klir and M. J. Wierman, Uncertainty Based Information: Elements of Generalized Information Theory, Physica, Heidelberg, Germany, 1998.
[19]
D. Padmanabhan and S. M. Batill, “Reliability based optimization using approximations with applications to multi-disciplinary system design,” in Proceedings of the 40th AIAA Sciences Meeting & Exhibit, AIAA-2002-0449, Reno, Nev, USA, 2002.
[20]
L. A. Zadeh, “Fuzzy sets,” Information and Control, vol. 8, no. 3, pp. 338–353, 1965.
[21]
G. J. Klir and B. Yuan, Fuzzy Sets and Fuzzy Logic: Theory and Applications, Prentice Hall, Upper Saddle River, NJ, USA, 1995.
[22]
R. R. Yager, “Arithmetic and other operations on Dempster-Shafer structures,” International Journal of Man-Machine Studies, vol. 25, no. 4, pp. 357–366, 1986.
[23]
D. Dubois and H. Prade, Possibility Theory: An Approach to Computerized Processing of Uncertainty, Plenum Press, New York, NY, USA, 1988.
[24]
A. Tversky and D. Kahneman, “Availability: a heuristic for judging frequency and probability,” Cognitive Psychology, vol. 5, no. 2, pp. 207–232, 1973.
[25]
A. Tversky and D. Kahneman, “Judgment under uncertainty: heuristics and biases,” Science, vol. 185, no. 4157, pp. 1124–1131, 1974.
[26]
A. Tversky and D. Kahneman, “Extensional versus intuitive reasoning: the conjunction fallacy in probability judgment,” Psychological Review, vol. 90, no. 4, pp. 293–315, 1983.
[27]
L. A. Zadeh, “Review of Shafer's mathematical theory of evidence,” Artifical Intelligence Magazine, vol. 5, pp. 81–83, 1984.
[28]
K. Sentz and S. Ferson, Combination of Evidence in Dempster-Shafer Theory (SAND2002-0835), Sandia National Laboratories, Albuquerque, NM, USA, 2002.
[29]
T. O. F-4E-1-9, “F-4E aircraft jet engine maintenance technical order manual,” U.S. Air Force, 1983.
[30]
D. Prelec, “A Bayesian truth serum for subjective data,” Science, vol. 306, no. 5695, pp. 462–466, 2004.
[31]
J. E. Matheson and R. L. Winkler, “Scoring rules for continuous probability distributions,” Management Science, vol. 22, no. 10, pp. 1087–1096, 1976.
