Bayesian Model Specification: Some problems related to model choice and calibration
MetadataShow full item record
This item's downloads: 288 (view details)
Draper, David (2011) Bayesian Model Specification: Some problems related to model choice and calibration Proceedings of AMSA 2011, Novosibirsk, Russia
In the development of Bayesian model specification for inference and prediction we focus on the conditional distributions p([theta],[beta]) and p(D[theta],[beta]), with data D and background assumptions [beta], and consider calibration (an assessment of how often we get the right answers) as an important integral step of the model development. We compare several predictive model-choice criteria and present related calibration results. In particular, we have implemented a simulation study to compare predictive model-choice criteria LS[cv] , a log-score based on cross-validation, LS[fs], a full-sample log score, with deviance information criterion, DIC. We show that for several classes of models DIC and LS[cv] are (strongly) negatively correlated; that LS[fs] has better small-sample model discrimination performance than either DIC, or LS[cv] ; we further demonstrate that when validating the model-choice results, a standard use of posterior predictive tail-area for hypothesis testing can be poorly calibrated and present a method for its proper calibration.