Flametui - plotting and recording flamegraphs with eBPF... in the TUI!

[koe]

Hi all, I wanted to start a small thread where I shared progress towards a fun project I’ve been working on: flametui! Hope that this is appropriate for this forum. The idea behind this project is to measure a simple flamegraph using eBPF, resolve the symbols, and then display it in the terminal.

I have my own profiling backend that uses eBPF to measure the stack traces from userspace and the kernel on all CPUs at a user defined rate (in hertz). We parse the resolve the symbols from the stacktraces partially while measuring, and partially after the fact. Optionally, a user may skip my (janky) profiler and just supply their own recording they made with e.g. perf. We show the flamegraph in the tui, and allow the user to explore it! Here’s a little gif of how that looks, where userspace and kernel traces are colored differently (based on your terminals colorscheme, or atleast on my machine):

We use libbpf for working with eBPF and the wonderful libvaxis for the purpose of plotting.

The reason I made this program is to solve a simple pain point I was having at work: I wanted to measure a flamegraph on an embedded platform, but couldn’t be bothered with perf-ing, manually copying the big data file over, then parsing it with the collection of perl scripts to view the svg. Too many steps, too much effort. This project can be cross compiled to e.g. aarch64, simply deployed, and presto!

That being said, I am aware that making a profiler is probably a fools errand, as perf is always going to better than whatever I make. That being said, for informal profiling and educational purposes, this has been great fun and satisfying to see come to life.

You can find flametui here: GitHub - nielsdekoeijer/flametui

Currently, it is in v0.0.1-alpha, but I’ve been having so much fun working on it that I was keen to share it anyway. Feel free to try it out! It’s janky so apologies in advance if you run into any struggles.

Future goals are as follows:

• Plotting AND measuring at the same time: I want to see the Flamegraph grow as I measure. Moreover, I am keen to explore what I will dub “time-varying” flamegraphs. Essentially, I am curious to experiment with flamegraphs that show a “window” in time. This way they can be used as a live-monitoring tool. I can easily see how we can get into a sort of measurement uncertainty principal though, as this would likely be quite expensive to do. I’m sure it will look cool nevertheless
• De-jankyify the codebase: Currently, its a codebase only a mother can love. I think I can improve it greatly by refactoring for better readability and optimizing. As of alpha, it more or less works but there are bugs. I also really want to improve the error handling. Currently, if we can’t quite find a symbol, we just print “unmapped”. That’s not very helpful.
• Add better interactions: I want to add arrow keys and vim keybindings (naturally). Plus, I’d like the user to be able to read all the availible controls by pressing “?”.

Anyway, thanks for taking the time to check out my project

[unnamed]

looks geat!
my current setup for profiling is this abomination:

zig build
perf record -g -F 999 zig-out/bin/executable
perf script -F +pid >/tmp/test.perf
firefox-developer-edition https://profiler.firefox.com/from-url/http%3A%2F%2Flocalhost%3A8080%2Ftest.perf/
(
cd /tmp/
deno x -Ay jsr:@joakimunge/denoliver@3.0.0 -c
)

so i think i could get a lot of use from a simple tool that you just run the binary with and get instant performance feedback.

[koe]

This is encouraging, thanks for your kind words. It’s been fun and challenging to work on, hopefully something useful rolls out of it eventually.

[unnamed]

@koe

hopefully something useful rolls out of it eventually.

Have you thought about zig build and git integration?
For instance i sometimes want to know, how some changes i made compare to the previous version, before i commit them.

I think there is so much cool stuff you could explore, that i wouldn’t mind if it just shells out to perf if you think your backend is not enough. (though plz find someway to make this work without the user having to download the FlameGraph scripts)

Ultimately i think there is so much performance simply left on table, because profiling is too much work (even if just to setup) that you end up only doing it once the software is already too slow.
That leads to death by a thousand cuts and i think the late feedback is also bad for learning how to write performant software, so this doesn’t happen in the first place.

[koe]

Really cool ideas. I appreciate you also sharing how you are utilizing perf earlier, it is useful for understanding the requirements I might want to satisfy. In that regard, keep the comments coming if you want .

Some questions:

zig build integration

How do you envision this specifically? My immediate thoughts are that one may want to add a “profile” step, where a profiler runs on a fixed binary or a set of binaries. After measuring, either collapsed stack traces save to disk (viewable by any flamegraph tool) or the TUI could open. I.e. 1) zig build profile 2) wait a few seconds 3) inspect results, or store for later for regression analysis.

git integration

Here I have little ideas, unless you mean github actions-esque approaches, where profiling can run in CI/CD? What do you have in mind?

I think there is so much cool stuff you could explore, that i wouldn’t mind if it just shells out to perf if you think your backend is not enough

Personally, my goal is to have it such that I can get to 80% of what perf can do, some reasonable subset. perf in my mind remains the culmination of loads of wisdom and experience, so displacing it is probably somewhat of a fools errand. Having something minimally pragmatic and useful is the first goal, and we’re kind of close to it.

In general, I plan to ingest and export various file formats, one of which being collapsed stack traces (what you get from one of Brendan Greggs perl scripts). This decouples us from my potentially dodgy backend. In a perfect world, you’d additionally be able to do as well: perf script | flametui or something similar.

Ultimately i think there is so much performance simply left on table, because profiling is too much work (even if just to setup) that you end up only doing it once the software is already too slow.
That leads to death by a thousand cuts and i think the late feedback is also bad for learning how to write performant software, so this doesn’t happen in the first place.

100% agree. A funny aside is that flametui can now record and plot in real-time. This also means I can use flametui on itself, which has already helped me catch some performance regressions .

[unnamed]

@koe

How do you envision this specifically?

1. Make an API you can use in your build.zig to define your own profiling steps
   • recording to file
   • running the ui
2. have some way of annotating tests to be benchmarks (name starts with"bench:" or smth.)
3. In the future you could have flametui build which replace zig build:

• for codebases that havent added the necessary hooks
• profile the build itself

@koe

Here I have little ideas, unless you mean github actions-esque approaches, where profiling can run in CI/CD? What do you have in mind?

Git captures the code changes over time. If flametui knew how to build a specific commit and run the benchmarks, it could not only tell you how much time the code takes, but how much that changed.

• Did the optimization i implemented work?
  • run flametui --test="bench:feature_a" --commit="head,master"
    => runtime: 15% faster, fn foo took 500ms, now 55ms (spends 5ms on hashing for the new cache)
• Are we regressing in performance?
  • run flametui --test="bench:feature_b" --graph-history
    => graph is almost flat, except for fix: handle recursive definitions properly which increased runtime by 30%
• I want to be notified when performance dropped by more than X%
  • add "bench:feature_c": { "reference": "<current_commit>", "max_increase": "X%" } to flametui.json
  • optionally add precommit-hook
    => get a warning/error if you made something slow

[koe]

1. Make an API you can use in your build.zig to define your own profiling steps
   • recording to file
   • running the ui
2. have some way of annotating tests to be benchmarks (name starts with"bench:" or smth.)
3. In the future you could have flametui build which replace zig build:

• for codebases that havent added the necessary hooks
• profile the build itself

Great suggestions, noted.

Git captures the code changes over time. If flametui knew how to build a specific commit and run the benchmarks, it could not only tell you how much time the code takes, but how much that changed.

• Did the optimization i implemented work?
  • run flametui --test=“bench:feature_a” --commit=“head,master”
    => runtime: 15% faster, fn foo took 500ms, now 55ms (spends 5ms on hashing for the new cache)
• Are we regressing in performance?
  • run flametui --test=“bench:feature_b” --graph-history
    => graph is almost flat, except for fix: handle recursive definitions properly which increased runtime by 30%
• I want to be notified when performance dropped by more than X%
  • add “bench:feature_c”: { “reference”: “<current_commit>”, “max_increase”: “X%” } to flametui.json, optionally add precommit-hook
    => get a warning/error if you made something slow

I like the ideas here too. Fully integrating with zig build or git may not be an immediate goal upon first thought, but the comparison use case is an interesting one. Accordingly, I think I will add APIs for this such that these workflows can be implemented downstream (i.e. through build.zig or script) easily. Relevant is adding support for: Differential Flame Graphs