Search Explore

Show HN: Microsoft releases Flint, a visualization language for AI agents

https://microsoft.github.io/flint-chart/#/

chenglong-hn · 1 hours ago

Data visualizations are the bridge between user and data.

But building AI agents that can generate visualizations reliably can be very tricky:

- simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability

We figured out it is a limitation on the language issue (not just AI capability thing) -- current visualization languages are a bit too low-level for AI agents, requiring them to explicitly make visual decisions that are supposed to be handled by a good compiler. Flint is a visualization intermediate language to address this issue, allow AI agents to solve this last-mile human-agent interaction problem. It provides a simple semantic-type based specification, and contains a layout optimization engine that can produce good-looking charts (filled with derived low-level details) from simple high-level specs. The result is also very human understandable and adaptable. Flint powers data formulator for generating visualizations (another open source project from microsoft https://data-formulator.ai/).

Flint is available open source, and we built a MCP server that you can directly plug flint in your favorite agent app to play with data.

14 comments

      • theK · 1 hours ago

        > requiring them to explicitly make visual decisions that are supposed to be handled by a good compiler

        Isnt graphviz there for the same reason?

        Edit: I see it is using JSON as the declaration language, I am OK with llms being "good at json" but a syntax also consumable by humans it is not!

        • chenglong-hn · 1 hours ago

          In fact, Json as a common language for human in visualization has been around for a while! The benefit of declarative grammar is that users can effective manipulate specs through UI (drag and drop, clicks).

          Btw, Flint is intentionally designed to allow agent skip low-level params like scale, axe, zero, step size etc (which are extremely crucial for "GOOD-looking") and they are dynamically optimized by the compiler. So AI agents can have a easier time.

          • theK · 52 minutes ago

            > Json as a common language for human in visualization has been around for a while

            Plant, Mermaid, Graphviz are all declarative textual representations designed for human authoring, JSON is made for tools. Its not a criticism just a statement that if interop across agent and human was intended this is not the simplest option.

            • chenglong-hn · 36 minutes ago

              right, in fact many small models still struggle with following json, some new forms are also needed

        • animal_spirits · 1 hours ago

          It compiles into Echarts, but echarts already has a JSON co figuration spec

          • chenglong-hn · 1 hours ago

            It's more like a simple high-level spec to make it easier. The idea is that you don't have to fill position / axes details just to make the chart work. The compiler has a bit of magic of using semantic types to optimize what parameters will be set in ECharts.

            In some composite chart examples, the good-looking echart spec is like 5x longer than the simple Flint one!

          • FailMore · 1 hours ago

            The charts are very nice, and I think the visualisation layer for LLMs is a very interesting problem.

            I’ve been building https://smalldocs.org for this exact reason. It’s an office suite for AI agents - but my main use case is giving a cli based LLM the canvas to express itself - charts, mermaid diagrams, etc. I’ve extended it a bit further to be a format for all types of work so the agent can embed slides and spreadsheets in a document.

            Sample document: https://smalldocs.org/blogs/what-is-a-smalldoc

            Source: https://github.com/espressoplease/smalldocs

            • giancarlostoro · 1 hours ago

              > mermaid diagrams

              I'm terrible at diagrams, so I gave GPT very generic descriptions of one of our project, to convert in to that mermaid style, then for Lucid I pasted it in there, and had a visualization of what I needed. Worked out nicely.

            • altmanaltman · 37 minutes ago

              interesting how you don't discuss literally anything about the project actually posted and spam your thing. Not pointing you out, seen many other comments like this on HN but always felt a bit weird about them

            • neomantra · 1 hours ago

              This is cool to see from a research team. A few weeks ago I was exploring a similar idea with ntcharts, where a user or LLM can specify a chart in a Golang or JSON object...

              and then that spec would be rendered either to a Bubble TUI via NTCharts or to HTML/SVG via ECharts. That Echarts HTML could be naturally served by a Golang http service.

              But Flint goes much deeper with semantic layers and settings optimizations. Perhaps a NTChart, or whatever terminal chart, could be a rendering target? I'll add it to the list to explore...

              https://github.com/NimbleMarkets/ntcharts/blob/spec/spec/REA...

              • chenglong-hn · 50 minutes ago

                This is fun! We started thinking it would just be an engineering task in the beginning, but doing a solid intermediate language turned out to be a research project (the paper will be out soon).

                Also, I find NTChart very fun, maybe we should add NT chart to the list of compilation backend for Flint so it works in the library. Putting a reminder here: https://github.com/microsoft/flint-chart/issues/45

              • santiagobasulto · 58 minutes ago

                Forget AI agents, this DSL is better even for humans. Cool project!

                • chenglong-hn · 40 minutes ago

                  For AI agents and Humans :)

                • kveykva · 55 minutes ago

                  Is there a specific explanation about how this is better or different than vega itself? https://vega.github.io/vega/docs/specification/

                  My understanding is that Vega was already an expressive DSL for visualizations and its probably already well spread through LLM training data.

                  • chenglong-hn · 47 minutes ago

                    Vega was a high-level language in the past for human, but now they can be a bit too low-level for AI agents! AI agents have to write a lot of low-level params just to make charts looking good, and the result is that programs are hard to write reliably for AI agents.

                    Flint is a higher-level abstraction, with simpler much shorter spec, and the compiler derives low-level decisions so that charts are looking good.

                    So: flint lets agent write short program that achieving good looking charts that had to be done with lengthy program in the past.

                  • dvt · 51 minutes ago

                    This is pretty crazy, literally built something almost exactly like this for a project I'm working on (a local-first AI agent that does work on folders while you sleep). Basically going from JSON "Lego blocks" to full reports (including charting, though a subset of what Flint offers). And with post-generation validation and retry steps.

                    Functions extremely well and the result is a very clear (and consitent) human-readable "output layer." Cool idea, fun to see people converging on similar concepts in the space.

                    • chenglong-hn · 46 minutes ago

                      That's awesome!

                      I find that besides training better models, designing new language for agents is also a super viable paths to improve their performance!

                    • nrub · 26 minutes ago

                      > simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability

                      N of only a few of us working on an analytics agent, I don't think we've been finding this to be the case. We've been impressed with just how good LLMs (even smaller open weight models) are at using Python and R for visualization. Often any shortcomings go away if we iterate a bit to about ambiguity. Are there any threads of research that could better support this claim or highlight where issues might be?

                      • mbreese · 8 minutes ago

                        A simpler spec can be used by a simpler agent. So, maybe that's the use-case here... use by smaller/cheaper agents that run in parallel as opposed to large models running one visualization at a time.

                        Or at least, maybe that's the idea?

                        IME, Claude and ChatGPT do just fine generating ggplot models, but extensive customization can get a bit hairy.

                      • cpard · 14 minutes ago

                        There’s an emerging pattern in agentic systems and this project is a great example.

                        A deterministic layer like a compiler or generator of code with some kind of IR that the LLM generates and feeds it with.

                        I feel we will be seeing this more and more in the near future.

                        • pwarner · 10 minutes ago

                          When I first saw Claude generating PPT decks by writing Python code instead of making the XML directly, it was sort of an "aha moment" for me. This seems to be the path for many things. It also feels slightly limiting, and like a hack LONG term, but 100% correct approach for a while.

                          • ajrouvoet · 3 minutes ago

                            A well designed intermediary enables both validation and control over the output independent of the AI. This changes the interaction model between human and AI from delegation to collaboration.

                          • rbalicki · 13 minutes ago

                            "For AI agents". I understand why everything needs to be marketed in this way, but it's just ... an easy-to-generate language for expressing charts. That's impressive! That's useful.

                            • khurs · 9 minutes ago

                              > Flint is built by Microsoft Research in collaboration with the IDEAS Lab, Renmin University of China.

                              Interesting to see a Chinese University collab for once.