Bayesian Model Specification: Some problems related to model choice and calibration

View/ Open
Date
2011Author
Krnjajic, Milovan
Metadata
Show full item recordUsage
This item's downloads: 300 (view details)
Recommended Citation
Draper, David (2011) Bayesian Model Specification: Some problems related to model choice and calibration Proceedings of AMSA 2011, Novosibirsk, Russia
Abstract
In the development of Bayesian model specification for inference and prediction we
focus on the conditional distributions p([theta],[beta]) and p(D[theta],[beta]), with data D and background
assumptions [beta], and consider calibration (an assessment of how often we get the right
answers) as an important integral step of the model development. We compare several
predictive model-choice criteria and present related calibration results. In particular, we
have implemented a simulation study to compare predictive model-choice criteria LS[cv] ,
a log-score based on cross-validation, LS[fs], a full-sample log score, with deviance information
criterion, DIC. We show that for several classes of models DIC and LS[cv] are
(strongly) negatively correlated; that LS[fs] has better small-sample model discrimination
performance than either DIC, or LS[cv] ; we further demonstrate that when validating
the model-choice results, a standard use of posterior predictive tail-area for hypothesis
testing can be poorly calibrated and present a method for its proper calibration.