The task of retrieving video content relevant to
natural language queries plays a critical role in
ectively handling internet-scale datasets. Most
of the existing methods for this caption-to-video
retrieval problem do not fully exploit
cross-modal cues present in video. Furthermore,
they aggregate per-frame visual features with
limited or no temporal information. In this
paper, we present a multi-modal transformer to
jointly encode the dirent modalities in video,
which allows each of them to attend to the
others. The transformer architecture is also
leveraged to encode and model the temporal
information. On the natural language side, we
investigate the best practices to jointly
optimize the language embedding together with the
multi-modal transformer. This novel framework
allows us to establish state-of-the-art results
for video retrieval on three datasets. More
details are available at
http://thoth.inrialpes.fr/research/MMT.