Direção de pesquisa : Matthieu CORD

Deep multimodal embeddings and grounding

Nowadays Artificial Intelligence (AI) is omnipresent in our society. The recent development of learning methods based on deep neural networks also called "Deep Learning" has led to a significant improvement in visual and textual representation models. In this thesis, we aim to further advance image representation and understanding. Revolving around Visual Semantic Embedding (VSE) approaches, we explore different directions: We present relevant background covering images and textual representation and existing multimodal approaches. We propose novel architectures further improving retrieval capability of VSE and we extend VSE models to novel applications and leverage embedding models to visually ground semantic concept. Finally, we delve into the learning process and in particular the loss function by learning differentiable approximation of ranking based metric.

Publicações 2018-2020