ATLANTIS - Multimodal Corpora for Crossmodal, Grounded Language Learning, Brigitte Krenn and Friedrich Neubarth, Austrian Research Institut for Artificial Intelligence (OFAI)

• Motivation: Why task descriptions?

• Comparability to language input for early language learning in infants

• Parallelism between speech and action

• Situatedness

• Goals:

• Multimodal object reference resolution

• Crossmodal learning of word-object mappings

• 2 Corpora: Multimodal Task Description Corpus, Action Verb Corpus

• Data collection experiments

• Technical setup

• Multimodal annotations

• Learning of Word-Object Mappings

• Learning from small multimodal task-oriented datasets (learner-tutor scenarios)

• The effect of signals from different modalities: head-eye gaze, manual manipulation

• Exemplary discussion of different learning approaches