Parameter Expansion for Dynamic Factor Analysis
Introduction
A persistent difficulty with variational Bayesian inference for factor models is slow convergence caused by strong posterior coupling between the loading matrix and the latent variables. This coupling arises from the rotational invariance of the factor model: for any non-singular matrix \(\mathbf{R}\), the transformation \(\mathbf{H} \to \mathbf{H}\mathbf{R}^{-1}\), \(\mathbf{z}_t \to \mathbf{R}\mathbf{z}_t\) leaves the likelihood unchanged. As a consequence, sequential VB updates zigzag slowly along ridge-lines of equal likelihood.
Parameter expansion (PX) alleviates this problem by introducing auxiliary parameters that decouple the variational factors, enabling faster convergence. The Parameter Expanded Variational Bayes (PX-VB) framework 1 optimizes auxiliary parameters within the variational framework, and the result is remapped to the original parameterization via a reduction step.
This document derives the PX-VB approach for the Dynamic Factor Analysis (DFA)
model implemented in sppcax. For the DFA model definition, prior distributions,
and ARD priors, see Bayesian Dynamic Factor Analysis. For the static-case VB-EM update
equations, see Factor Analysis and PCA.
Parameter Expanded VBEM (PX-VBEM)
The PX-VB framework 1 introduces auxiliary parameters \(\boldsymbol{\alpha}\) that expand the original model \(p(\tilde{\mathbf{x}}, D)\) into \(p_{\boldsymbol{\alpha}}(\mathbf{x}, D)\), where the original model is recovered at \(\boldsymbol{\alpha} = \boldsymbol{\alpha}_0\). Each PX-VB iteration consists of three steps:
Sequential VB updates with \(\boldsymbol{\alpha} = \boldsymbol{\alpha}_0\)
Minimize \(\text{KL}(q(\mathbf{x}) \| p_{\boldsymbol{\alpha}}(\mathbf{x}, D))\) over \(\boldsymbol{\alpha}\)
Reduce the expanded model back to the original via reparameterization
Rotation-Based Parameter Expansion for LGSSM
For state-space models, a natural parameter expansion is the rotation of the latent subspace 3. For any non-singular \(\mathbf{R} \in \mathbb{R}^{K \times K}\):
The rotation must be compensated in the dynamics:
Thus the expanded model uses transformed variables \(\tilde{\mathbf{z}}_t = \mathbf{R}\mathbf{z}_t\), \(\tilde{\mathbf{H}} = \mathbf{H}\mathbf{R}^{-1}\), and \(\tilde{\mathbf{F}} = \mathbf{R}\mathbf{F}\mathbf{R}^{-1}\).
Standard VB-EM Updates
The factorized approximate posterior is:
The VBE-step updates \(q(\mathbf{Z})\) via Kalman smoothing using the expected parameters from the other factors.
The VBM-step updates each factor in turn:
\(q(\mathbf{H}, \boldsymbol{\Psi})\): Normal-Gamma (MVNIG) posterior update using sufficient statistics from the E-step (see Factor Analysis and PCA for the static update equations)
\(q(\mathbf{F}, \mathbf{Q})\): MNIW or MVN posterior update using state transition sufficient statistics
\(q(\boldsymbol{\tau})\): Gamma posterior update (ARD hyperparameters)
PX-VB Rotation Step
After the standard VB-EM updates, we find the optimal rotation \(\mathbf{R}\) that minimizes the KL divergence. The rotation is applied to the posterior distributions as follows 3:
Rotating \(q(\mathbf{Z})\):
where the sufficient statistics transform as:
Rotating \(q(\mathbf{H})\):
Rotating \(q(\mathbf{F})\):
Finding the Optimal Rotation
The optimal rotation \(\mathbf{R}\) is found by minimizing the expected negative log-prior of the rotated model parameters:
This objective decomposes into four terms:
where:
\(\mathcal{L}_{\text{init}}\): expected negative log-prior of the rotated initial state \(\tilde{\mathbf{z}}_0 = \mathbf{R}^{-1}\mathbf{z}_0\)
\(\mathcal{L}_{\text{emission}}\): expected negative log-prior of the rotated emission matrix. The rotation uses a block-diagonal structure \(\mathbf{R}_{\text{block}} = \text{blkdiag}(\mathbf{R}, \mathbf{I})\) so that only the first \(K\) columns of the augmented emission matrix are rotated, preserving the input weights and bias columns:
\[\mathcal{L}_{\text{emission}} = \frac{1}{2} \sum_d \text{tr}\left(\boldsymbol{\Lambda}_d \, \mathbf{R}_{\text{block}}^\top \mathbb{E}[\tilde{\mathbf{h}}_d \tilde{\mathbf{h}}_d^\top] \mathbf{R}_{\text{block}}\right) - D \ln|\mathbf{R}|\]\(\mathcal{L}_{\text{dyn-prior}}\): expected negative log-prior of the rotated dynamics \(\tilde{\mathbf{F}} = \mathbf{R}^{-1}\mathbf{F}\mathbf{R}_{\text{block}}\)
\(\mathcal{L}_{\text{dyn-lik}}\): expected negative log-likelihood of the dynamics residuals under the rotated state noise
For the static case (FA/PCA), only \(\mathcal{L}_{\text{init}}\) and \(\mathcal{L}_{\text{emission}}\) are needed since there are no dynamics.
Numerical optimization: In sppcax, the rotation \(\mathbf{R}\) is found by
minimizing Eq. (5) using gradient descent with Anderson acceleration 7.
Starting from \(\mathbf{R} = \mathbf{I}\), the optimizer runs for a fixed number
of steps (default: 32, learning rate: 0.001). Anderson acceleration with memory \(m=1\)
is applied after the first gradient step to improve convergence. If the final loss exceeds
the initial loss, the optimizer falls back to the identity rotation.
Static Case Simplification
For static Factor Analysis (\(\mathbf{F} = \mathbf{0}\), \(\mathbf{Q} = \mathbf{I}\)), the rotation only affects \(\mathbf{H}\) and \(\mathbf{Z}\). The dynamics terms vanish, and the loss simplifies to:
PX-VBEM Algorithm Summary
Algorithm 1: PX-VBEM for Dynamic Factor Analysis
See Also
For practical examples comparing EM, VB-EM, and PX-VB:
Testing PX-VBEM for Bayesian Factor Analysis — Parameter expansion for Factor Analysis
Testing PX-VBEM for Dynamic Factor Analysis — Parameter expansion for Dynamic Factor Analysis
References
- 1(1,2)
Qi, Y., & Jaakkola, T. S. (2006). Parameter expanded variational Bayesian methods. In Advances in Neural Information Processing Systems 19, pp. 1097-1104.
- 3(1,2)
Luttinen, J., Raiko, T., & Ilin, A. (2014). Linear state-space model with time-varying dynamics. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2014), LNCS 8725, pp. 338-353.
- 7
Luttinen, J., & Ilin, A. (2010). Transformations in variational Bayesian factor analysis to speed up learning. Neurocomputing, 73, 1093-1102.