Misra et al, CVPR 2017, [link]
tags: few-shot learning - cvpr - 2017
The main idea of the proposed method is to learn how to composite visual concepts (here: object categories and attributes) at the feature-level. Each visual concept is represented by a linear classifier that has been trained to recognize it. The model then learns to compose such linear classifier by feeding them to a Transformation Network, , which in turn outputs another classifier, which should capture the composition of the two input concepts.
Figure: Overview of the Transformation Network. At training time, we assume access to a limited set of combinations of the primitives, each modeled by a linear classifier. The transformation network takes these classifiers as input and composes them to produce a classifier for their combination.
The input concepts are split in two categories (e.g., objects and attributes) which can be combined. For each of them, a linear binary classifier (SVM) is trained to detect the concept, based on some pre-trained feature extractor . The goal is to learn the composition operation, which could be applied to even new combinations of the concepts, unseen during training. is parametrized as a 3 layer Multi-Layer Perceptron and trained with the following loss:
where is a standard binary classification loss (here, cross-entropy). is the label of the image for the considered task and should be 1 if and only if the image contains the complex context .
The feature extractor used in the experiments is a
VGG-M-1024 network pretrained on ImageNet. It is kept fixed and not re-trained, only the Transformation Network and linear classifiers are. Experiments are performed on the MITStates dataset and evaluated in terms of predicting (top-k accuracy and mAP) the presence or absence of unseen combinations of (object, attribute) pairs. The model is compared against the following baselines, using the same pre-trained backbone feature extractor:
Interestingly the Individual-Obj baseline that considers only the object category performs quite well in terms of average precision. This suggest the object category is more significant than the attribute prediction. Finally, the proposed model outperforms the baselines but still produces poor overall performance in terms of classification accuracy, so it is not clear how much is really due to generalization to new concept combinations. Ablation and qualitative experiments do show that the model learns some meaningful information about composition, however the gain seems to depend a lot on the actual object and attribute combined.