A Neighbourhood Encoding Framework for Deep Mining Heterogeneous Texts in Recipe-image Retrieval

Cross-modal retrieval usually fills the semantic gap between different modalities by sharing subspaces. However, existing methods rarely consider that the data in a certain modality may be heterogeneous when mapping multimodal data into a shared subspace. In addition, most existing methods focus on semantic associations between different modalities, while few approaches consider the semantic associations within a single modality. To address the above two deficiencies, we propose a Neighbourhood Encoding (NE) framework that mines the semantic association of data in the same modality, solves the problem of data heterogeneity by improving the semantic expression of a single modality. To verify the effectiveness of the proposed framework, we use two types of recurrent neural networks to instantiate the framework. Experiments show that the instantiated approaches outperform existing advanced methods in both text-to-image and image-to-text retrieval directions.