Louizos et al, ICLR 2016, [link]
tags: representation learning - iclr - 2016
Given input data , the goal is to learn a representation of , that factorizes out nuisance or sensitive variables, , while retaining task-relevant content . Working in the
VAE (Variational Autoencoder) framework, this is modeled as a generative process:
where the prior on latent is explicitly made invariant to the variables to filter out, . Introducing decoder , this model can be trained using the standard variational lower bound objective () .
In order to make the learned representations relevant to a specific task, the authors propose to incorporate label knowledge during the feature learning stage. This is particularly useful if the task target label is correlated with the sensitive information , otherwise unsupervised learning could yield random representations in order to get rid of only. In practice, this is done by considering two distinct independent sources of information for the latent: , the label for data point (categorical variable in the classification scenario) and a continuous variable that contains the remaining data variations which do not depend on (nor ). This yields the following generative process:
In the fully supervised setting, when the label is known, the ELBO can be extended to handle this two-stage latent variables scenario in a simple manner:
In the semi-supervised scenario, the label y can be missing for some of the samples. In which case, the ELBO can once again be extended to consider as another latent variable with encoder and prior .
So far, the model incorporates explicit statistical independence between the sensitive variables to protect, and the latent information to capture from the input data, . The authors additionally propose to regularize the marginal posterior using the Maximum Mean Discrepancy : The
MMD is a distance between probability distributions that compare mean statistics, and is often combined with the kernel trick for efficient computation.
we want the latent representation to not learn any information about : This constraint can be enforced by making the distributions and close to one another, as measured by their
MMD. This introduces a second loss term to minimize, .
The is defined as a difference of mean statistics over two distributions. Empirically, given two sample sets , and a feature map operation , the squared distance can be written as
where the last line is the result of the kernel trick and is an arbitrary kernel function. In practice, the MMD loss term is computed over each batch: A naive implementation would require computing the full kernel matrix of for each pair of samples in the batch, where each scalar product requires operations, where is the dimensionality of . Instead, following , they use Random Kitchen Sinks  which is a low-rank method that allows to compute the MMD loss more efficiently.
The authors consider three experimental scenarios to test the proposed Variational Fair Autoencoder (
The models are evaluated on (i) their performance on the target task and (ii) their ignorance of the sensitive data. The latter is evaluated by training a classifier to prediction from and evaluating its performance.
On the Fairness task, the proposed method seems to be better at ignoring the sensitive information, although its trade-of between accuracy and fairness is not always the best. The main baseline is the model presented in , which also exploits a moment matching penalty term, although a simpler one, and does not use the
In the Domain Adaptation task, the authors mostly compare to the
DANN  which proposes a simple adversarial training technique to align different domain representations. The
VFAE results are on-par, or even slightly better in most scenarios.