My Analyzer Shipped Without Checking Its Own Source

I maintain NetEvolve.Analyzer, a Roslyn analyzer that tells other people’s C# code to use is null instead of == null, keep one type per file, and stop nesting #region blocks nobody reads. It ships four separate builds to cover four Roslyn API versions. It has automatic code fixes. It has decent test coverage.

It had never run against its own source code.

That is not a hypothetical embarrassment I am inventing for a hook. I merged PR #49 on August 5, 2026, specifically to fix it, and the first thing that happened when I pointed the analyzer at itself was 39 warnings across three rules I had written and shipped. The tool that tells you your code has problems had problems it had never told itself about.

Why an Analyzer Skips Its Own Medicine

The reason is mechanical, not laziness. NetEvolve.Analyzer multi-targets four Roslyn API versions (4.4.0, 4.7.0, 4.14.0, 5.6.0) because consumers on older SDKs need an analyzer that loads against their compiler, not the newest one. Each version is its own .csproj, compiling the same shared source under a different RoslynApiVersion and, until now, the same AssemblyName.

If you want a Roslyn analyzer project to analyze itself, the obvious move is to reference itself as an Analyzer item. Roslyn analyzers run as MSBuild Analyzer items, not ProjectReference items, specifically so the compiler can load the diagnostic DLL without pulling it into the build’s dependency graph as a regular assembly reference. But a project cannot reference its own output as an analyzer before that output exists. That is a build-ordering cycle, and MSBuild will not resolve it for you.

The fix in PR #49 uses the fact that there are four sibling projects compiling the same source:

<!-- NetEvolve.Analyzer.Build.props -->
<ItemGroup Label="Dogfooding: apply this analyzer's own rules to its own source"
           Condition=" '$(MSBuildProjectName)' != 'NetEvolve.Analyzer.Dogfood' ">
  <ProjectReference Include="NetEvolve.Analyzer.Dogfood.csproj"
                    OutputItemType="Analyzer"
                    ReferenceOutputAssembly="false" />
</ItemGroup>

OutputItemType="Analyzer" tells MSBuild to feed the referenced project’s output DLL into the compiler as an analyzer, not as a regular assembly reference. ReferenceOutputAssembly="false" stops the compiler from also linking against it as a library. The Condition excludes the dogfood project from referencing itself. Each of the four real variants gets checked by a sibling build, never by itself, so there is no cycle: NetEvolve.Analyzer.Roslyn4_4 depends on NetEvolve.Analyzer.Dogfood, never the reverse.

Building the Roslyn 4.4 variant with this wired up surfaced 39 warnings against three rules: NE0007 (use <see langword> for keywords in doc comments instead of <c>true</c>), NE0008 (use <see cref> for native types instead of <c>string</c>), and NE0009 (methods with a CancellationToken parameter must check it before doing real work). Thirty-nine warnings, in a codebase whose entire job is telling other people to fix exactly this class of problem, and the build stayed green throughout because these are warnings, not errors. Nothing forced anyone to look at them. That is precisely the failure mode dogfooding exists to catch: a tool that would flag the issue in your codebase but never flags it in its own, because nobody wired it up to check.

The Silent Failure Referencing Yourself Creates

Getting the diagnostics to fire was the easy half. The code fixes, the actual value proposition of an analyzer with a lightbulb icon, quietly stopped working, and nothing told me.

The first attempt referenced the existing NetEvolve.Analyzer.Roslyn4_14 build (the 4.14 baseline) as the Analyzer item for the sibling projects. Diagnostics appeared in the editor exactly as expected. Code fixes did not. No error, no warning, no red squiggle explaining why: the lightbulb simply never appeared for rules that unambiguously should have offered one.

The cause is Visual Studio’s MEF composition for analyzer and code-fix providers. Every one of the four real variants shares the assembly name NetEvolve.Analyzer, because that is the name consumers expect in their PackageReference. When two same-named assemblies from different paths (the NuGet-facing build and the one loaded as a dogfooding analyzer) end up in the same solution’s MEF catalog, Visual Studio’s composition silently drops the CodeFixProvider exports for the duplicate. DiagnosticAnalyzer exports survive; CodeFixProvider exports do not. You get the warning, not the fix, and there is no log line pointing at MEF as the culprit.

The instinctive fix, override the assembly name on the reference itself, does not work either:

<!-- Does not work: Visual Studio's Solution Build Manager builds each
     project once and will not build a second, differently-named
     instance of the same project file. -->
<ProjectReference Include="NetEvolve.Analyzer.Roslyn4_14.csproj"
                  Properties="AssemblyName=NetEvolve.Analyzer.Dogfood" />

dotnet build resolves its own reference graph and can build parameterized instances of the same project under different property values. Visual Studio’s Solution Build Manager does not work that way: it builds each project in the solution once, under whatever AssemblyName its .csproj already declares. A Properties= override on the reference is invisible to it.

So PR #49 adds a fifth project, NetEvolve.Analyzer.Dogfood.csproj, whose only job is to compile the shared source under a genuinely distinct assembly name:

<Project>
  <PropertyGroup>
    <RoslynApiVersion>5.6.0</RoslynApiVersion>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <!-- BaseIntermediateOutputPath/BaseOutputPath must be set before
       Sdk.props is imported (MSB3539), so this project can't use the
       <Project Sdk="Microsoft.NET.Sdk"> shorthand like its siblings. -->
  <PropertyGroup>
    <BaseIntermediateOutputPath>obj\$(MSBuildProjectName)\</BaseIntermediateOutputPath>
    <BaseOutputPath>bin\$(MSBuildProjectName)\</BaseOutputPath>
  </PropertyGroup>

  <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk" />
  <Import Project="NetEvolve.Analyzer.Build.props" />

  <PropertyGroup>
    <!-- Overrides the AssemblyName set by Build.props: referenced only
         as an Analyzer item, never packed or shipped. -->
    <AssemblyName>NetEvolve.Analyzer.Dogfood</AssemblyName>
  </PropertyGroup>

  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk" />
</Project>

It targets Roslyn 5.6.0, the newest baseline, so its diagnostics reflect the latest analyzer API surface. It sets IsPackable=false (added in a follow-up commit after the first push, because the project’s default packing behavior would otherwise have put an internal build artifact into the NuGet package) so it never ships. And it exists purely as an Analyzer reference target for the other four, none of which build it as a library, none of which risk a cycle, because none of them are it.

If your Roslyn analyzer multi-targets several API versions and shares an assembly name across builds, and you want it to check its own source, do not reuse one of the shipping variants as the analyzer reference. Give the self-check a dedicated project with its own AssemblyName. It costs one small .csproj and saves you a debugging session where the analyzer looks like it is half-broken and nothing in the IDE tells you why.

What Got Fixed, and What Is Deliberately Still Open

The first commit in PR #49 only wired up the reference and confirmed the build stayed green with 39 warnings outstanding: the PR description at that point explicitly said the findings were “not fixed in this PR, follow-up.” That held for exactly one more commit. Once the dedicated Dogfood project made the code fixes actually fire, applying NE0007 across the doc comments was one command, and standardizing on <see cref> (NE0008) and honoring cancellation tokens at the start of async methods (NE0009) landed in the same pass:

// Before, in CSharpKeywords.cs: NE0007 and NE0008 both firing on the same doc comment
// (em dash in the real source replaced with a comma here for house style; wording otherwise verbatim)
/// prefix (<c>@class</c>). Native type names (<c>bool</c>, <c>byte</c>, <c>char</c>, ...) are
/// intentionally excluded, referencing a type by its bare name in a doc comment is idiomatic and not
/// the keyword-in-<c>&lt;c&gt;</c> mistake this list exists to catch. <c>void</c> is kept, since it

// After, applied by the code fix
/// prefix (<c>@class</c>). Native type names (<see cref="bool"/>, <see cref="byte"/>, <see cref="char"/>, ...) are
/// intentionally excluded, referencing a type by its bare name in a doc comment is idiomatic and not
/// the keyword-in-<c>&lt;c&gt;</c> mistake this list exists to catch. <see langword="void"/> is kept, since it

// Before, in RequireCancellationCheckCodeFixProvider.cs: NE0009 firing on its own fix provider
private static async Task<Document> InsertCheckAsync(
    Document document,
    MethodDeclarationSyntax method,
    CancellationToken cancellationToken
)
{
    var root = (await document.GetSyntaxRootAsync(cancellationToken).ConfigureAwait(false))!;
    // ...
}

// After
private static async Task<Document> InsertCheckAsync(
    Document document,
    MethodDeclarationSyntax method,
    CancellationToken cancellationToken
)
{
    cancellationToken.ThrowIfCancellationRequested();

    var root = (await document.GetSyntaxRootAsync(cancellationToken).ConfigureAwait(false))!;
    // ...
}

Twenty-some files across Documentation/, Maintainability/, and Usage/ picked up one or both fixes: CSharpKeywords.cs, NullCheckOperand.cs, the various CodeFixProvider classes, RequireCancellationCheckAnalyzer.cs. Coverage landed at 94.42%, every changed line covered, 8 CI checks green. Not a rewrite. A tool correcting itself with its own tooling, which is a more convincing demonstration than any amount of README copy about “reliable code fixes.” Whatever in the original 39 the code fixes could not resolve mechanically is still open, follow-up work, same as the PR description always said.

The Follow-Up Bug Dogfooding Was Supposed to Catch

It caught one within hours. PR #52, merged the same day as #49, is a direct consequence of pointing NE0008 at the analyzer’s own multi-targeted source.

NE0008 hardcoded DateOnly and TimeOnly into its list of always-recognized BCL type names, next to string and Guid, and flagged <c>DateOnly</c> in a doc comment regardless of what the consuming project actually targets. Both types were introduced in .NET 6. A project targeting netstandard2.0 has no such type to cref, so rewriting the doc comment to <see cref="DateOnly"/> there is not a style improvement, it is a compile error waiting to happen.

Two distinct gaps, not one. First, the analyzer flagged the doc comment even when the current compilation could not resolve the type at all: a plain hardcoded-name check, no verification against the compiler in front of it. Second, and specific to multi-targeted projects: the analyzer runs once per target framework and can correctly decide, per compilation, whether the type exists there. But the code fix, once triggered, rewrote the doc comment for every target framework in one shot. If one sibling TFM in the project’s TargetFrameworks list predates the type, the rewritten <see cref="DateOnly"/> fails to resolve on that sibling and breaks the build for a framework the fix never even looked at.

That second failure mode is the same shape as the MEF collision earlier in this post: a check that is correct for the compilation directly in front of it, wrong for a sibling target it never considers, because nothing forced it to look sideways. NetEvolve.Analyzer itself multi-targets four Roslyn API versions; running NE0008 against that source is exactly the scenario where a hardcoded BCL-type-name list, blind to which framework it is currently compiling against, was always going to fall over.

The fix moves DateOnly/TimeOnly out of the unconditional well-known-types list into ConditionalBclTypeMinimumVersions, a name-to-minimum-version map (not hardcoded to .NET 6, so a future conditionally-available BCL type with a different introduction version is not misclassified the same way). The analyzer only treats the type as recognized once Compilation.GetTypeByMetadataName confirms it actually resolves in the current compilation:

// NativeTypeCrefAnalyzer.cs, conceptually: don't trust the name list alone
if (ConditionalBclTypeMinimumVersions.ContainsKey(typeName)
    && compilation.GetTypeByMetadataName($"System.{typeName}") is null)
{
    // Not in scope for this compilation: this TFM predates the type. Don't flag.
    return;
}

The code fix gets a second, separate check: before rewriting to <see cref="...">, it inspects the project’s full TargetFrameworks list and withholds the rewrite if any sibling framework predates the type, even though the diagnostic itself still fires correctly for the TFM that does have it. Getting that list into the analyzer required its own small fix: MSBuild’s $(TargetFrameworks) uses semicolons to separate values, but Roslyn’s AnalyzerConfig format treats an unescaped ; as a comment leader and silently truncates the property at the first entry. NetEvolve.Analyzer.props now re-joins the list with commas before exposing it as NetEvolveAnalyzerTargetFrameworks, a CompilerVisibleProperty.

Five new test cases pin down the boundary: a diagnostic fires against net8.0 reference assemblies, none fires against netstandard2.0, the same source analyzed independently against two TFMs shows no cached state leaking between them, the fix is withheld when a multi-targeted project names an incompatible sibling framework, and it still applies when every listed TFM supports the type. Coverage: 94.37%, 38 new lines.

The general lesson, if you are writing any analyzer rule that references a specific BCL type by name: check Compilation.GetTypeByMetadataName, not a hardcoded string list, and if your rule also ships a code fix, check the project’s full TargetFrameworks, not just the one compilation the analyzer happened to run against. DateOnly/TimeOnly versus .NET 6 is one instance. Any BCL type introduced after your oldest supported TFM is the same bug waiting for the next dogfooding pass, or the next multi-targeted consumer who reports it, to find.

Why This Should Change How You Evaluate the Analyzer

None of this changes what the analyzer does for a consumer: PackageReference it as a dev dependency, it adds zero runtime assemblies, it runs against net8.0, net9.0, and net10.0 targets, and its file-organization rules can be opted out of via MSBuild properties if you disagree with one-type-per-file as a hard rule. That part was already true.

What changes is what you can infer about whether the rules are any good. A linter that has never been pointed at its own several-thousand-line codebase is a linter whose maintainer has only ever seen it work on other people’s problems. The gap between “passes my unit tests” and “survives contact with a real codebase, including mine” is exactly where analyzer rules turn out to be too aggressive, miss obvious cases, or (as happened here) ship a code-fix provider that silently does not fire under a condition nobody tested for. Dogfooding does not prove the rules are correct. It proves someone actually ran them somewhere that matters to them, which is a lower bar than correctness and a higher bar than most tooling clears.

The MEF pitfall is also a useful data point independent of this analyzer specifically. If you are building or evaluating any Roslyn analyzer that multi-targets several API versions under a shared AssemblyName, and you see diagnostics appear without their code fixes, check whether two same-named analyzer assemblies from different paths have landed in the same MEF catalog before you assume the CodeFixProvider itself is broken. It is a two-hour investigation if you know to look there, and an open-ended one if you do not.

NetEvolve.Analyzer is a young project: no meaningful star count, a small commit history, one contributor. I would not point at PR #49 as evidence of “community momentum,” because there is none to point at yet, and pretending otherwise would be dishonest about a repository I run myself. What I would point at is the shape of the fixes, which is my own read, not a metric. I have seen this exact class of bug, an analyzer rule that knows a BCL type by name but not by which target frameworks actually have it, play out before in more established Roslyn analyzer projects, usually surfaced by a confused user filing an issue rather than by the maintainer finding it first. PR #52 is that same failure mode caught within hours of PR #49 landing, by the maintainer’s own dogfooding pass, on the maintainer’s own multi-targeted codebase, not by somebody else’s bug report eighteen months from now. Multi-targeting plus a newer BCL type is not a one-off mistake; it is a recurring class of bug for any analyzer that spans more than one target framework or Roslyn version. Seeing that pattern show up here this early, and getting closed the same day, is the good sign. Not that the project has traction. That whoever runs it went looking for the multi-targeting failure mode before it became somebody else’s confusing support ticket.