Incremental Source Generators Done Right: Prove the Cache Hits

Incremental Source Generators Done Right: Prove the Cache Hits

This is part 3 of 4. Part 1 established the equality contract: every value flowing through an IIncrementalGenerator pipeline must compare by value, or the cache misses. Part 2 covered the pipeline shapes that keep those comparisons cheap. Both parts end with a claim: do this and your generator caches correctly. Throughout this post, the generator under test is the AutoToStringGenerator we assembled at the end of part 2 — the one that turns [AutoToString] on a partial class into a generated ToString override.

Claims are cheap. I have reviewed generators whose authors swore they were incremental, complete with record models and EquatableArray<T> wrappers, and a single test proved they re-ran the entire pipeline on every unrelated edit. One overlooked tuple, one captured symbol, and the whole thing silently degrades to the full-rebuild behavior we measured in the predecessor article.

If you do not write that test, your generator’s incrementalism is folklore. This post is the test.

Asking the Driver

Roslyn can tell you exactly why each pipeline step ran. The GeneratorDriver records this when you opt in via GeneratorDriverOptions with trackIncrementalGeneratorSteps: true. Tracking is off by default because it costs memory; you enable it only in tests.

The setup is ordinary Roslyn hosting: build a CSharpCompilation, wrap your generator in a driver, run it.

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;

internal static class GeneratorTestHost
{
    public static CSharpCompilation CreateCompilation(params string[] sources)
    {
        var syntaxTrees = sources
            .Select(source => CSharpSyntaxTree.ParseText(
                source,
                new CSharpParseOptions(LanguageVersion.Latest)))
            .ToArray();

        return CSharpCompilation.Create(
            assemblyName: "GeneratorTests",
            syntaxTrees: syntaxTrees,
            references: Basic.Reference.Assemblies.Net90.References.All,
            options: new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary));
    }

    public static GeneratorDriver CreateDriver(IIncrementalGenerator generator)
        => CSharpGeneratorDriver.Create(
            generators: [generator.AsSourceGenerator()],
            driverOptions: new GeneratorDriverOptions(
                IncrementalGeneratorOutputKind.None,
                trackIncrementalGeneratorSteps: true));
}

Two details matter here. First, the references: Basic.Reference.Assemblies gives you a complete, deterministic reference set without touching the machine’s SDK installation. If you refuse the extra package, MetadataReference.CreateFromFile(typeof(object).Assembly.Location) works for trivial cases, but you will chase missing-reference diagnostics the moment your test source uses System.Linq. Take the package.

Second, AsSourceGenerator(): the Create overload that accepts GeneratorDriverOptions takes ISourceGenerator instances, so incremental generators get wrapped. This is the supported adapter, not a hack.

After running, GetRunResult() returns a GeneratorDriverRunResult whose per-generator results expose TrackedSteps — a dictionary keyed by the names you assigned with WithTrackingName in part 2. If you skipped naming your steps, go back and name them. Unnamed steps are invisible to this whole approach.

GeneratorDriver driver = GeneratorTestHost.CreateDriver(new AutoToStringGenerator());
driver = driver.RunGenerators(compilation);

GeneratorDriverRunResult runResult = driver.GetRunResult();
var trackedSteps = runResult.Results[0].TrackedSteps;
// trackedSteps["Models"], ...

Every step you tagged with WithTrackingName shows up in this dictionary. The AutoToStringGenerator names exactly one step, Models, so that is the one key we care about here — a more complex pipeline with named steps after each Combine or Collect gives you more keys, and you assert on all of them.

Note that RunGenerators returns a new driver carrying the cache state. Discard the return value and every subsequent run starts cold — the single most common reason this test passes when it should fail.

The Cache-Hit Test

The test protocol has four steps: run the generator once to populate the cache, apply an unrelated change to the compilation, run again with the same driver instance, and assert that no tracked step reports a cache miss.

Each IncrementalGeneratorRunStep in TrackedSteps exposes Outputs, an array of (object Value, IncrementalStepRunReason Reason) tuples. The reasons are the verdict:

  • Cached — the inputs compared equal, the step was skipped entirely. Best case.
  • Unchanged — the step re-executed, but produced an output equal to the cached one, so downstream steps were spared. Acceptable, slightly wasteful.
  • Modified / New — the cache missed. This is what the test forbids.
  • Removed — the input disappeared, so the step re-ran to drop its output. Also a re-run.

Here is the complete test, against the AutoToStringGenerator and its Models step:

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Xunit;

public class IncrementalityTests
{
    private const string Source = """
        using AutoToString;

        namespace Demo;

        [AutoToString]
        public partial class Customer
        {
            public string Name { get; set; } = "";
            public int Age { get; set; }
        }
        """;

    [Fact]
    public void UnrelatedChange_DoesNotInvalidateCache()
    {
        var compilation = GeneratorTestHost.CreateCompilation(Source);
        GeneratorDriver driver = GeneratorTestHost.CreateDriver(new AutoToStringGenerator());

        // Run 1: populate the cache.
        driver = driver.RunGenerators(compilation);

        // Unrelated change: a new syntax tree the generator does not care about.
        var updatedCompilation = compilation.AddSyntaxTrees(
            CSharpSyntaxTree.ParseText(
                "namespace Demo; public class Unrelated { }",
                new CSharpParseOptions(LanguageVersion.Latest)));

        // Run 2: same driver instance, carrying cache state.
        driver = driver.RunGenerators(updatedCompilation);
        GeneratorDriverRunResult result = driver.GetRunResult();

        AssertAllStepsCached(result, "Models");
    }

    private static void AssertAllStepsCached(
        GeneratorDriverRunResult result,
        params string[] stepNames)
    {
        var trackedSteps = result.Results[0].TrackedSteps;

        foreach (var stepName in stepNames)
        {
            Assert.True(
                trackedSteps.ContainsKey(stepName),
                $"Step '{stepName}' was not tracked. Check WithTrackingName.");

            foreach (var step in trackedSteps[stepName])
            {
                foreach (var (_, reason) in step.Outputs)
                {
                    Assert.True(
                        reason is IncrementalStepRunReason.Cached
                               or IncrementalStepRunReason.Unchanged,
                        $"Step '{stepName}' reported '{reason}'. Cache miss.");
                }
            }
        }
    }
}

Adding an independent syntax tree is the sharpest unrelated change, because it modifies the Compilation — which invalidates anything foolishly bound to CompilationProvider — without touching any syntax your ForAttributeWithMetadataName predicate matches. A whitespace edit inside an existing file is a good second variant; run both.

This test is cheap to run and brutal to pass. That is exactly the point.

The Test That Catches the ImmutableArray Bug

Part 1 spent an entire section on why ImmutableArray<T> breaks caching: it compares by reference, so two arrays with identical contents are not equal. Here is that bug meeting its test.

Suppose the ClassModel from part 2 had been written the obvious way:

// Compiles. Looks incremental. Is not.
internal sealed record ClassModel(
    string? Namespace,
    string Name,
    ImmutableArray<string> Properties);

Run the test above against a generator producing this model, and the second run fails:

IncrementalityTests.UnrelatedChange_DoesNotInvalidateCache [FAIL]
  Assert.True() Failure
  Step 'Models' reported 'Modified'. Cache miss.
  Expected: True
  Actual:   False

The transform re-ran (the compilation changed, so ForAttributeWithMetadataName re-evaluates its matches), produced a ClassModel with contents identical to the cached one — and the record’s synthesized equality called ImmutableArray<string>.Equals, which compared references of two distinct arrays and said not equal. Reason: Modified. Everything downstream re-ran too.

The fix is the EquatableArray<T> wrapper from part 1 — the model exactly as part 2 ships it:

internal sealed record ClassModel(
    string? Namespace,
    string Name,
    EquatableArray<string> Properties);

Same test, green. No benchmark, no binlog archaeology, no profiler session — a one-line model change validated by a unit test in milliseconds. This is why the test comes first and the patterns second: the patterns are only trustworthy because the test enforces them.

The Anti-Pattern Catalog

Every entry below is a real failure mode I have hit or reviewed. Each one passes code review and fails the cache-hit test.

1. Holding Compilation or INamedTypeSymbol in the model. Symbols are reference-equal and tied to a specific compilation; every edit produces a new compilation and new symbols.

// Broken: symbol identity changes on every edit.
public sealed record Target(INamedTypeSymbol Type);

// Fixed: flatten to strings in the transform, drop the symbol.
public sealed record Target(string FullyQualifiedName, string Namespace);

2. Anonymous types and tuples carrying arrays out of Select. Tuples compare element-wise, which is fine for strings — and silently wrong the moment one element is an array.

// Broken: the string[] element compares by reference.
var models = targets.Select(static (t, _) => (t.TypeName, Members: t.Members.ToArray()));

// Fixed: a named record with an equatable collection.
var models = targets.Select(static (t, _) =>
    new Target(t.TypeName, new EquatableArray<string>(t.Members)));

3. Building the source string inside the transform. The transform runs on every keystroke that re-evaluates the provider; RegisterSourceOutput runs only on cache misses. Generate text in the latter.

// Broken: allocates and formats megabytes of source per keystroke.
transform: static (ctx, _) => SourceEmitter.Emit(ExtractModel(ctx));

// Fixed: transform extracts the model; emission happens after the cache.
transform: static (ctx, _) => ExtractModel(ctx);
// ...
context.RegisterSourceOutput(models, static (spc, model) =>
    spc.AddSource($"{model.TypeName}.g.cs", SourceEmitter.Emit(model)));

4. Capturing a SyntaxNode in the model and re-reading it later. Syntax nodes are reference-equal and pin the entire tree in memory — a documented way to leak whole compilations inside the IDE.

// Broken: node identity changes on every edit; the tree cannot be collected.
public sealed record Target(ClassDeclarationSyntax Declaration);

// Fixed: extract what you need, flatten the location for diagnostics.
public sealed record Target(string TypeName, LocationInfo? DiagnosticLocation);

public sealed record LocationInfo(
    string FilePath, TextSpan TextSpan, LinePositionSpan LineSpan)
{
    public Location ToLocation()
        => Location.Create(FilePath, TextSpan, LineSpan);

    public static LocationInfo From(SyntaxNode node)
        => new(
            node.SyntaxTree.FilePath,
            node.Span,
            node.GetLocation().GetLineSpan().Span);
}

Do not reach for Location as the fix — it carries the same disease. A Location holds a reference to its SyntaxTree and compares that tree by reference, so storing one pins the tree in memory and breaks equality after every edit to the file. Flatten it to file path and spans in the transform, and rebuild the real Location with Location.Create only when you report the diagnostic in RegisterSourceOutput.

5. Non-static lambdas capturing generator state. A capturing lambda closes over this or a local; the captured state is invisible to the cache and a correctness bug waiting for a second compilation. Mark every pipeline lambda static and the compiler enforces it.

// Broken: captures the field; state leaks across runs.
// Inside the generator class:
private readonly List<string> _seen = [];
// Inside Initialize:
transform: (ctx, _) => { _seen.Add(ctx.TargetSymbol.Name); return Extract(ctx); }

// Fixed: static lambda, no capture possible.
transform: static (ctx, _) => Extract(ctx)

6. Combining CompilationProvider without projecting first. Covered in depth in part 2: Combine with the raw compilation makes every edit a cache miss for the combined step, because the compilation is never equal to its predecessor.

// Broken: invalidated on every keystroke.
var combined = targets.Combine(context.CompilationProvider);

// Fixed: project to the stable value you actually need.
var assemblyName = context.CompilationProvider
    .Select(static (c, _) => c.AssemblyName ?? string.Empty);
var combined = targets.Combine(assemblyName);

Six patterns, one test. You do not need to memorize the catalog if the test is in CI — it will find the seventh pattern I have not hit yet, too.

Wire It into CI

The incrementality test is a plain xUnit fact; it needs no special runner, no MSBuild integration, no IDE. It runs in the generator’s test project alongside everything else. The project file needs the Roslyn C# compiler package and the reference assemblies:

<ItemGroup>
  <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.14.0" />
  <PackageReference Include="Basic.Reference.Assemblies" Version="1.8.0" />
  <PackageReference Include="Verify.SourceGenerators" Version="2.5.0" />
  <PackageReference Include="Verify.Xunit" Version="30.4.0" />
  <ProjectReference Include="..\..\src\AutoToString.Generators\AutoToString.Generators.csproj" />
</ItemGroup>

The cache-hit test proves the generator runs rarely. It says nothing about whether the generated code is correct — that is the job of snapshot testing, and Verify.SourceGenerators does it with almost no ceremony. The one piece of ceremony it does demand is a module initializer that registers its converters — without it, Verify throws instead of producing a snapshot:

internal static class VerifyInit
{
    [ModuleInitializer]
    public static void Init() => VerifySourceGenerators.Initialize();
}

[Fact]
public Task GeneratedSource_MatchesSnapshot()
{
    var compilation = GeneratorTestHost.CreateCompilation(Source);
    GeneratorDriver driver = GeneratorTestHost.CreateDriver(new AutoToStringGenerator());
    driver = driver.RunGenerators(compilation);

    return Verify(driver);
}

Verify serializes every generated hint name, source text, and diagnostic into .verified.cs files committed to the repository. A change in output shows up as a reviewable diff in the pull request instead of a surprise in a consumer’s build. The two tests are complementary and both belong in the default dotnet test run: one guards performance, the other guards output.

Three Parts in Three Sentences

Part 1: every value in the pipeline must implement value equality, and ImmutableArray<T> does not — flatten symbols to records and wrap collections in EquatableArray<T>. Part 2: filter early with ForAttributeWithMetadataName, transform late, project CompilationProvider before combining, and name every step with WithTrackingName. Part 3: none of that counts until a test runs the driver twice with trackIncrementalGeneratorSteps: true and asserts every tracked step reports Cached or Unchanged.

Wired right, a non-trivial generator on a 200-file project completes most keystrokes in single-digit milliseconds, and the multi-second build tax from the predecessor article simply does not apply to you. Wired wrong, you are shipping that tax to every consumer, on every target framework, forever — and telling yourself it is fine because the interface name says Incremental.

The difference is not magic and it is not tribal knowledge. It is one equality contract, one pipeline shape, and one test that refuses to take your word for it. And once all three hold, one question remains: whether the generator survives being packaged and shipped into projects you will never see.

The interface makes it possible. The test makes it true.

Comments

VG Wort