gix-blame performance with imara-diff 0.1 and 0.2 – Performance analyses of gitoxide

Note

This post is a modernized version of an earlier post of mine in my main blog.

This post’s “published” date reflects the date the original post was published.

Recently, we started the process of upgrading gitoxide’s dependency on imara-diff from 0.1.8 to 0.2.0 (tracked in this issue). Because imara-diff’s API has changed significantly, the changes are currently behind a feature flag. What I’ve been wondering, though, is whether this update has any impact on gix-blame’s performance as gix-blame spends a lot of time diffing two versions of a file.

Running the benchmark

In order to collect some data, I compiled two versions of the gix binary via cargo build --release --features blame-experimental and cargo build --release. Then I used hyperfine to run gix blame on a set of 9 files in my local copy of the gitoxide repo.

env GIT_DIR="$HOME/github/Byron/gitoxide/.git" BASELINE_EXECUTABLE="$HOME/bin/gix-blame-2026-01-25-3b6650a66" COMPARISON_EXECUTABLE="$HOME/bin/gix-blame-experimental-2026-01-25-3b6650a66" ./run_benchmark.py

run_benchmark.py, the script used to benchmark both versions of gix blame

#!/usr/bin/env python
# /// script
# requires-python = ">=3.13"
# ///

import os

def run(path, i):
    os.system(
        f'hyperfine --parameter-list command "$BASELINE_EXECUTABLE","$COMPARISON_EXECUTABLE" --parameter-list path {path} "{{command}} blame {{path}}" --command-name baseline --command-name comparison --export-json benchmark-{i}.json'
    )

for i, path in enumerate(
    [
        "CHANGELOG.md",
        "STABILITY.md",
        "README.md",
        "Cargo.toml",
        "gix-blame/src/file/function.rs",
        "gix-path/src/env/mod.rs",
        "gix-index/tests/index/file/write.rs",
        "gix-object/src/lib.rs",
        "gix-odb/src/store_impls/loose/write.rs",
    ]
):
    run(path, i + 1)

This will create a couple of Markdown files, benchmark-1.json through benchmark-9.json.

Loading the data

Then, in order to work with these files, we’re going to use numpy and pandas to load them into data.

A few helper functions for loading the data

import numpy as np
import pandas as pd

import matplotlib as mpl

import seaborn as sns

import json

def load_results(filename):
    with open(filename) as f:
        return json.load(f)["results"]

def extract_data_points(result):
    command = result["command"]
    path = result["parameters"]["path"]

    return {"command": command, "path": path, "time": result["times"]}

filenames = [f"benchmark-{i}.json" for i in range(1, 10)]
results = [load_results(filename) for filename in filenames]
results = [extract_data_points(result) for result in np.concatenate(results)]

data = pd.concat(
    [pd.DataFrame(result) for result in results],
    ignore_index=True,
)

And finally, we’re going to create 2 plots that will give us an idea of how both versions of gix blame compare with respect to performance. Looking in particular at the boxplot, it seems that performance got better for files that are changed frequently, such as CHANGELOG.md and README.md.

Creating a boxplot

Code

FIGURE_WIDTH = 8
FIGURE_ASPECT = 1.6

sns.set_theme(
    palette="Paired",
    rc={"figure.figsize": (FIGURE_WIDTH, FIGURE_WIDTH / FIGURE_ASPECT)},
)

boxplot = sns.boxplot(x="time", y="path", hue="command", data=data)
sns.despine(offset=10, trim=True)
mpl.pyplot.tight_layout()
mpl.pyplot.show()

Creating a stripplot

mpl.pyplot.figure(figsize=(FIGURE_WIDTH * 0.75, FIGURE_WIDTH / FIGURE_ASPECT))

stripplot = sns.stripplot(
    x="time",
    y="path",
    hue="command",
    dodge=True,
    size=2.5,
    jitter=0.2,
    legend=False,
    data=data,
)

mpl.pyplot.show()

Conclusion

It seems that the version using imara-diff 0.2 has a slight advantage over the version using imara-diff 0.1 when it comes to some of the files that have changed a lot over the course of gitoxide’s history, such as CHANGELOG.md or README.md. Cargo.toml, on the other hand, doesn’t fit that diagnosis. For files that changed less frequently, the situation is much closer, so I don’t want to draw too many conclusions.

Code

summary = (
    data
    .groupby(["path", "command"])
    .agg(
        mean=("time", "mean"),
        stddev=("time", "std"),
    )
    .reset_index()
)

summary["mean_ms"] = summary["mean"] * 1000
summary["stddev_ms"] = summary["stddev"] * 1000

wide = summary.pivot(
    index="path",
    columns="command",
    values=["mean_ms", "stddev_ms"],
)

table = pd.DataFrame({
    "baseline_mean_ms": wide["mean_ms"]["baseline"],
    "baseline_stddev_ms": wide["stddev_ms"]["baseline"],
    "comparison_mean_ms": wide["mean_ms"]["comparison"],
    "comparison_stddev_ms": wide["stddev_ms"]["comparison"],
})

table["Δ ms"] = table["comparison_mean_ms"] - table["baseline_mean_ms"]
table["Δ %"] = table["Δ ms"] / table["baseline_mean_ms"] * 100
table["ratio"] = table["comparison_mean_ms"] / table["baseline_mean_ms"]

table = table.sort_values("Δ %", ascending=False)

display_table = pd.DataFrame({
    "path": table.index.to_list(),
    "baseline mean ± sd": [
        f"{mean:.2f} ± {sd:.2f}"
        for mean, sd in zip(table["baseline_mean_ms"], table["baseline_stddev_ms"])
    ],
    "comparison mean ± sd": [
        f"{mean:.2f} ± {sd:.2f}"
        for mean, sd in zip(table["comparison_mean_ms"], table["comparison_stddev_ms"])
    ],
    "Δ ms": table["Δ ms"].to_list(),
    "Δ %": table["Δ %"].to_list(),
    "ratio": table["ratio"].to_list(),
})

delta = display_table["Δ %"]

worse_rows = delta[delta > 0].index
better_rows = delta[delta < 0].index

display_table.style.format({
    "Δ ms": "{:+.2f}",
    "Δ %": "{:+.2f}%",
    "ratio": "{:.3f}×",
}).background_gradient(
    subset=pd.IndexSlice[worse_rows, ["Δ %"]],
    cmap="Reds",
).background_gradient(
    subset=pd.IndexSlice[better_rows, ["Δ %"]],
    cmap="Greens_r",
).hide(axis="index")

path	baseline mean ± sd	comparison mean ± sd	Δ ms	Δ %	ratio
gix-object/src/lib.rs	94.33 ± 2.62	94.93 ± 2.32	+0.59	+0.63%	1.006×
gix-odb/src/store_impls/loose/write.rs	93.75 ± 2.27	94.22 ± 2.30	+0.46	+0.50%	1.005×
Cargo.toml	102.49 ± 1.87	102.87 ± 2.39	+0.39	+0.38%	1.004×
gix-index/tests/index/file/write.rs	69.46 ± 2.11	69.61 ± 1.88	+0.15	+0.21%	1.002×
gix-path/src/env/mod.rs	49.05 ± 1.76	49.13 ± 1.57	+0.08	+0.17%	1.002×
gix-blame/src/file/function.rs	48.02 ± 1.39	47.88 ± 1.80	-0.13	-0.28%	0.997×
STABILITY.md	53.76 ± 1.81	53.09 ± 1.93	-0.66	-1.24%	0.988×
README.md	113.28 ± 2.89	110.43 ± 3.14	-2.85	-2.52%	0.975×
CHANGELOG.md	158.94 ± 3.23	150.66 ± 3.58	-8.28	-5.21%	0.948×