Live Unit Testing

I've been dabbling on and off with Visual Studio's Live Unit Testing feature. Introduced in 2017, it's arguably a sherlocking of NCrunch, with all that entails: a first party implements something that used to be a third party's business model, and also gets to integrate it better (since, y'know, they're the first party). I hope the NCrunch folks are doing well.

In a recent project, I've tried to be more judicious about unit tests, and Live Unit Testing helps not only with that, but also with a Test-Driven Development-like approach. So what is it?

First, of course, there's continuous integration, where the entire suite of tests gets run on a separate machine, typically after source control push or at an interval. You get the added benefit that you know the code works exactly the same as on your local machine (did you forget to check something in? Is your code platform-dependent in a way you didn't expect? Is there a hardcoded path somewhere?). Tools like Jenkins, or these days, perhaps GitHub Actions.

Then there's already the Test Explorer pane, which enumerates tests and their current status. A test that ran successfully is green. A test that failed is red. Tests that haven't run recently aren't as saturated. You can make so-called Playlists, where you only want to run a subset of tests, perhaps for performance reasons or because you already know you aren't currently working on a section affecting the other tests.

A Visual Studio Test Explorer screenshot, showing 39 tests total, 6 of which have recently successfully run, and 4 of which failed on their non-recent run. There's also good use of hierarchy here: first, the assembly containing the tests, then the namespace they're in, then the class, then the concrete test method, and finally multiple inputs (test cases) for the same test.

It also integrates with CodeLens, a Visual Studio feature where a little line gets inserted in your text editor for additional metadata. The most recent committer, the amount of references to the code (is it perhaps unused?), and, if any, the result of the tests.

CodeLens above a method named GeneratePassword. Notice how it tells us that only two out of six tests are passing.

Wait, which ones are failing, you ask? You can simply click on "2/6 passing" to get a detailed popup. And you can even use that popup to run those tests, or run them in a debugger, if you'd like.

CodeLens's test popup, which is a more detailed and also slightly interactive view.

Live Unit Testing similar to Test Explorer in that its primary UI is a pane right in the IDE. Indeed, much of it looks similarly, occasionally to the point of confusion, because it's not the same.

The main Live Unit Testing pane. We get the same hierarchy and similar status indication. One key difference: there are play/pause/stop buttons.

Test Explorer interactively requires you to run tests. It encourages it, such as with the CodeLens inline icons, but it won't do it for you.

Live Unit Testing, on the other hand, runs your tests continuously in the background. In fact, it also compiles the code in the background, in its own little sandboxed environment. Your regular compilation isn't affected by this, nor are the results in Test Explorer.¹

I mentioned TDD at the beginning, and in a scenario like this library, this really rocks. You write stubs of code, you write the tests, they all fail, then you start implementing, and you see one test after another begin succeeding — nor not. It's TDD as it should be, with only a few seconds of lag.

But it's also been largely unchanged since 2017, and I really thought Visual Studio 2022 (two major releases later) would've addressed a few issues. Here's my two big ones.

Poor diagnostics

As mentioned, LUT runs in its separate environment. It does not compile to bin/obj, instead creating its own hierarchy inside the hidden .vs folder. This is good in that it avoids interfering with your regular work. None of building, running the debugger, etc. care about what LUT is currently doing or not doing.

But what if LUT fails? The Error List won't tell you, because that is for the regular toolchain. The Output window will give you a hint, but only if you've explicitly switched to its Live Unit Testing section. Not only does the drop-down not give you a clue that there could be new lines in there; it also likes to automatically switch back to the Build section instead, which doesn't mention LUT at all (again, sort of by design).²

The Output window. This list of sections is… not VS's finest hour in UI.

If you're lucky enough to correctly guess that LUT isn't working right, here's what the Output window, with default settings, will tell you:

LUT's log.

Build errors

Now that we're a few levels deeper in the rabbit hole, we can look in the diag folder, which in turn may contain a log file, which in turn sometimes contains a build error:

[12:41:14.463 Verbose] - BuildManager - D:\PasswordRulesSharp\PasswordRulesSharp\CharacterClass.cs(14,55): error CS0246: The type or namespace name 'NotNullWhenAttribute' could not be found (are you missing a using directive or an assembly reference?)

(Why didn't you just tell me that?)

Now, that's a very actionable error. But keep in mind that none of this is surfaced in UI. In fact, that very moment, the Live Unit Testing pane will still happily show you all-green checkmark circles for your tests, making it look as though not only are all tests passing, but also, that all tests have run recently. The first is true; the latter is not. Unlike Test Explorer, Live Unit Testing doesn't reliably show you whether a result may be stale.

Poor file path behavior

But sometimes, it also leads a few levels deeper, like LUT complaining about a missing file. What file? The log doesn't say. Does it work in a different project? Indeed.

Remember how LUT uses its own, semi-hidden hierarchy in .vs? Well, that combined with Windows's traditional limit of file paths of 255 characters³ can easily mean that, if your project hierarchy is just short enough to work, but LUT adds its own internal hierarchy on top, it can fail. Again, the key problem here is not that this happens, but that you'll be spending the next two hours figuring out 1. that LUT hasn't actually been doing anything, 2. that this isn't a build error, as a regular build does work, 3. that it works with a fresh project, 4. so it must be something weird between LUT and this particular project.

Broken standard template

I've had multiple occasions where LUT refuses to run any tests with a freshly-created test project. The cause appears to be that the project, by default, references a Test SDK that isn't binary-compatible with the Visual Studio IDE. Absolutely nothing in the diagnostics seems to hint at that. Instead, you try, randomly, to update your NuGet packages, and suddenly, it works.

Suggestions

Two easy fixes:

• if anything non-successful occurs, have a yellow bar at the top of the LUT pane that tells me so!
• the Test Explorer tests that haven't run in a while have a dimmed icon and text. The same should be true of Live Unit Testing's, especially if (see above) something appears to be wrong.
• the Error List already has Build + IntelliSense option. Let me choose to also surface LUT errors here, please.

Contradictory information

Remember that CodeLens UI? Well, here's a fun one. Another piece of UI LUT nicely adds is a gutter to the side that shows you which lines of code failed in a recent test.

The GeneratePassword method again; this time also with LUT hints to the left-hand side.

Except, hang on a second. CodeLens says 2 out of 6 tests are passing, and LUT shows only green checkmarks?

Again, this is a downside to LUT being its own thing. If you stop LUT, its gutter disappears. If not, it runs continuously (unless, of course, it has silently decided to fail) — whereas CodeLens, instead, relies on Test Explorer, which doesn't automatically run at all.

So in other words, LUT's information cannot always be trusted to be recent, because when failures (other than, of course, failing tests) occur, the UI gives you no indication of that. And Test Explorer's information also cannot be trusted to be recent, because you need to run it manually, but at least that one shows you when a result is stale.⁴

In conclusion

Live Unit Testing seems to have improved in terms of performance and quality, but it needs work. Even for a relatively simple project like the one in my screenshots, I ran into a multitude of issues that almost made me give up on the feature.

Coda: The Mac

The Mac version of Visual Studio doesn't offer this feature at all. Nor does it offer CodeLens.

It does have a Test Results pane, a bit like Test Explorer (but without the ability to create playlists), and also offers to run a test in a debugger.

I'm not sure if there's any Microsoft-internal roadmap at all to get it to feature parity with Windows at least in aspects that aren't platform-specific (the Mac version will likely never get a WPF designer, say) — but if so, it feels like it's easily five years away. It's steadily gotten better, but they sure have their work cut out for them.