Abstract
Additive regression models with interactions have been considered extensively in the literature, using estimation methods such as splines or Gaussian process regression. At least two difficulties have hampered their application: (i) estimating the models can be difficult due to potentially many tuning (or smoothing) parameters, and (ii) model selection may be difficult due to a lack of adequate criteria. In this paper, we attempt to address these issues with a novel approach to estimating and selecting additive models with and without interaction effects.
Firstly, we extend the I-prior methodology (Bergsma, 2020) to multiple covariates, each of which may be multidimensional. For this purpose, we define a class of hierarchical interaction models assuming the regression function lies in a reproducing kernel Krein space (RKKS), and derive the (possibly indefinite) reproducing kernel for the models. The I-prior is an objective prior for a statistical parameter based on its Fisher information. In the present case, the I-prior for the regression function is Gaussian with covariance kernel proportional to its (positive definite) Fisher information, with support a subset of the assumed RKKS, i.e., the I-prior is proper. The I-prior methodology has some theoretical and practical advantages over competing methods such as Gaussian process regression and Tikhonov regularization. A practical (computational) advantage is that it permits an EM algorithm with simple E and M steps to find the maximum marginal likelihood estimators of the scale parameters (also known as tuning parameters), making their estimation easier than for competing methods.
A second innovation we introduce is a parsimonious specification of models with interac- tions. That is, each covariate is assigned a single (length) scale parameter, regardless of the number of interactions present in the model. Rather than assigning a new scale parameter to tensor product spaces, these inherit the scale parameters of the components through their product. This approach is mathematically justified and has two key advantages: (i) estimation of models with interactions is simplified due to the presence of fewer scale parameters, and (ii) model selection (among models with different interactions present) is simplified, in that simply the model with the highest marginal likelihood can be chosen.
The I-prior approach is suitable for both parametric and nonparametric regressions. Our simulations show comparatively good performance for model selection in basic multiple regression with interactions. The methodology is also illustrated with a real-data example. An R-package implementing our methodology is available (Jamil, 2019).
Haziq Jamil
Assistant Professor in Statistics
My research interests include statistical theory, methods and computation, with applications towards the social sciences.