chatcrowd(ViL): A Dialog-based Platform
for Visual Layout Composition

NAACL 2019
Paola Cascante-Bonilla  Xuwang Yin  Vicente Ordonez  Song Feng
Abstract

In this paper we introduce Chat-crowd, an interactive environment for visual layout composition via conversational interactions. Chat-crowd supports multiple agents with two conversational roles: agents who play the role of a designer are in charge of placing objects in an editable canvas according to instructions or commands issued by agents with a director role. The system can be integrated with crowdsourcing platforms for both synchronous and asynchronous data collection and is equipped with comprehensive quality controls on the performance of both types of agents. We expect that this system will be useful to build multimodal goal-oriented dialog tasks that require spatial and geometric reasoning.

image_aspect_ratio

Grounded by Visual Layouts

A multi-modal dialog simulation system with a focus on spatial reasoning. Two sample tasks are included, 2D-shape layout and COCO image layout.

group

Multiple Crowdsourcing Mode

Integrated with crowdsourcing platforms for both synchronous and asynchronous data collection.

trending_up

Quality Control

Equipped with comprehensive quality controls on the performance of both types of Instructors.

computer Chat-crowd repository
vibration Check our paper
visibility View sample dialogues of our tasks

Try our demo

Task: 2D-shape Layout (please select one taskId)

Play Instructor Play Painter

Task: COCO Layout (please select one taskId)

Play Instructor Play Painter

Check our 3 minute video presentation



[Bibtex]
@inproceedings{cascante-bonilla-etal-2019-chat,
  title = "Chat-crowd: A Dialog-based Platform for Visual Layout Composition",
  author = "Cascante-Bonilla, Paola and
  Yin, Xuwang and
  Ordonez, Vicente and
  Feng, Song",
  booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics (Demonstrations)",
  month = jun,
  year = "2019",
  address = "Minneapolis, Minnesota",
  publisher = "Association for Computational Linguistics",
  url = "https://www.aclweb.org/anthology/N19-4024",
  doi = "10.18653/v1/N19-4024",
  pages = "138--142",
  abstract = "In this paper we introduce Chat-crowd, an interactive environment for visual layout composition via conversational interactions. Chat-crowd supports multiple agents with two conversational roles: agents who play the role of a designer are in charge of placing objects in an editable canvas according to instructions or commands issued by agents with a director role. The system can be integrated with crowdsourcing platforms for both synchronous and asynchronous data collection and is equipped with comprehensive quality controls on the performance of both types of agents. We expect that this system will be useful to build multimodal goal-oriented dialog tasks that require spatial and geometric reasoning.", }