Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models

Victor Dibia

Microsoft HAX Team

Abstract

Systems that support users in the automatic creation of visualizations must address several subtasks - understand the semantics of data, enumerate relevant visualization goals, and generate visualization specifications. In this work, we pose visualization generation as a multi-stage generation problem and argue that well-orchestrated pipelines based on large language models (LLMs) and image generation models (IGMs) are suitable for addressing these tasks. This talk presents LIDA, a novel tool for generating grammar-agnostic visualizations and infographics. LIDA comprises of 4 modules - A SUMMARIZER that converts data into a rich but compact natural language summary, a GOAL EXPLORER that enumerates visualization goals given the data, a VISGENERATOR that generates, refines, executes, and filters visualization code, and an INFOGRAPHER module that yields data-faithful stylized graphics using IGMs. LIDA provides a Python API, and a hybrid user interface (direct manipulation and multilingual natural language) for interactive charts, infographics, and data story generation.

Overall, the talk will cover:

  • The design of a tool (LIDA) that concretely applies LLMs to the task of automated visualization generation
  • Challenges with evaluating LLM-based systems and how they are addressed with LIDA
  • Current research challenges and opportunities for LLM-enabled visualization tools.

Project Page: https://microsoft.github.io/lida/



Profile

Victor Dibia is a Principal Research Software Engineer at the Human-AI eXperiences (HAX) team, Microsoft Research, where he focuses on Generative AI. His research interests span human-computer interaction, computational social science, and applied machine learning. Victor's work has been published at conferences such as ACL, EMNLP, AAAI, and CHI, earning multiple best paper awards and garnering attention from media outlets like the Wall Street Journal and VentureBeat. He is an IEEE Senior member, a Google Certified Professional in Data Engineering and Cloud Architect, and a Google Developer Expert in Machine Learning. Victor holds a Ph.D. in Information Systems from the City University of Hong Kong and a Masters in Information Networking from Carnegie Mellon University.

Week 24 2022/2023

Thursday 18th May 2023
3:00-4:00pm

Microsoft Teams - request a link via email

For queries or meeting link, contact Dr. Ignatius Ezeani (i.ezeani@lancaster.ac.uk)