We have the unfortunate honour of supporting multiple versions of the .NET Framework across our entire software suite.

All of our client side software (i.e. the stuff that is installed on client machines, both desktop and server) tends to target .NET Framework 4.0. Historically, this was the last version available on Windows XP, and up until relatively recently, it was important that we continue to support that ancient and terrifying beast. You know how it works, old software, slow clients, we’ve all been there.

Meanwhile, all of our server side software tends to target .NET Framework 4.5. We really should be targeting the latest .NET Framework (as the deployment environments are fully under our control), but it takes effort to roll that change out (new environments mostly) and it just hasn’t been a priority. Realistically our next major move might be to just start using .NET Core inside Docker images instead, so it might never really happen.

One of the main challenges with supporting both .NET 4.0 and .NET 4.5 is trying to build libraries that work across both versions. Its entirely possible, but is somewhat complicated and unless you understand a bunch of fundamental things, can appear confusing and magical.

Manual Targeting Mode Engaged

The first time we had to support both .NET 4.0 and .NET 4.5 for a library was for Serilog.

Serilog by itself works perfectly fine for this sort of thing. As of version 1.5.9, it supported both of the .NET Framework versions that we needed in a relatively painless manner.

As is almost always the way, we quickly ran into a few situations where we needed some custom components (a new Sink, a custom JSON formatter, some custom destructuring logic), so we wrapped them up in a library for reuse. Targeting the lowest common denominator (.NET 4.0), we assumed that we would be able to easily incorporate the library wherever we needed.

The library itself compiled file, and all of its tests passed.

We incorporated it into one the pieces of software that targeted .NET 4.0 and it worked a treat. All of the custom components were available and worked exactly as advertised.

Then we installed it into a piece of software targeting .NET 4.5 and while it compiled perfectly fine, it crashed and burned at runtime.

What had actually happened was that Serilog changed a few of its public interfaces between 4.0 and 4.5, replacing some IDictionary<TKey, TValue> types with IReadOnlyDictionary<TKey, TValue>. When we created the library, and specifically targeted .NET 4.0, we thought that it would use that version of the framework all the way through, even if we installed it in something targeting .NET 4.5. Instead, when the Serilog Nuget package was installed into the destination via the dependency chain, it came through as .NET 4.5, while our poor library was still locked at 4.0.

At runtime the two disagreed and everything exploded.

Use The Death Dot

The easiest solution would have been to mandate the flip to .NET 4.5. We actually first encountered this issue quite a while ago, and couldn’t just drop support for .NET 4.0, especially as this was the first time we were putting together our common logging library. Building it such that it would never work on our older, still actively maintained components would have been a recipe for disaster.

The good news is that Nuget packages allow for the installation of different DLLs for different target frameworks. Unsurprisingly, this is exactly how Serilog does it, so it seemed like the sane path to engage on.

Targeting specific frameworks is pretty straightforward. All you have to do is make sure that your Nuget package is structured in a specific way:

• lib
• net40
• {any and all required DLLs for .NET 4.0}
• net45
• {any and all required DLLs for .NET 4.5}

In the example above I’ve only included the options relevant for this blog post, but there are a bunch of others.

When the package is installed, it will pick the appropriate directory for the target framework and add references as necessary.  Its actually quite a neat little system.

For us, this meant we needed to add a few more projects to the common logging library, one for .NET 4.0 and another for .NET 4.5. The only thing inside these projects was the CustomJsonFormatter class, because it was the only thing we were interacting with that had a different signature depending on the target framework.

With the projects in place (and appropriate tests written), we constructed a custom nuspec file that would take the output from the core project (still targeting .NET 4.0), and the output from the ancillary projects and structure them in the way that Nuget wanted. We also had to add Nuget package dependencies into the nuspec file as well, because once you’re using a hand-rolled nuspec file, you don’t get any of the nice automatic things that you would if you were using the automatic package creation from a .csproj file.

Target Destroyed

With the new improved Nuget package in place, we could reference it from both .NET 4.0 and .NET 4.5 projects without any issue.

At least when the references were direct anyway. That is, when the Nuget package was referenced directly from a project and the classes therein only used in that context.

When the package was referenced as part of a dependency chain (i.e. project references package A, package A references logging package) we actually ran into the same problem again, where the disconnect between .NET 4.0 and .NET 4.5 caused irreconcilable differences in the intervening libraries.

It was kind of line async or generics, in that once you start doing it, you kind of have to keep doing it all the way down.

Interestingly enough, we had to use a separate approach to multi-targeting for one of the intermediary libraries, because it did not actually have any code that differed between .NET 4.0 and .NET 4.5.  We created a shell project targeting .NET 4.5 and used csproj trickery to reference all of the files from the .NET 4.0 project such that both projects would compile to basically the same DLL (except with different targets).

I don’t recommend this approach to be honest, as it looks kind of like magic unless you know exactly what is going on, so is a bit of a trap for the uninitiated.

Regardless, with the intermediary packages sorted everything worked fine.

Summary

The biggest pain throughout this whole process was the fact that none of the issues were picked up at compilation time. Even when the issues did occur, the error messages are frustratingly vague, talking about missing methods and whatnot, which can be incredibly hard to diagnose if you don’t immediately know what it actually means.

More importantly, it wasn’t until the actual code was executed that the problem surfaced, so not only was the issue only visible at runtime, it would happen during execution rather than right at the start.

We were lucky that we had good tests for most of our functionality so we realised before everything exploded in the wild, but it was still somewhat surprising.

We’re slowly moving away from .NET 4.0 now (thank god), so at some stage I will make a decision to just ditch the whole .NET 4.0/4.5 multi-targeting and mark the new package as a breaking change.

Sometimes you just have to move on.

Unfortunately for us, we had to move our TeamCity build server into another AWS account recently, which is never a pleasant experience, though it is often eye opening.

We had to do this for a number of reasons, but the top two were:

• We’re consolidating some of our many AWS accounts, to more easily manage them across the organisation.
• We recently sold part of our company, and one of the assets included in that sale was a domain name that we were using to host our TeamCity server on.

Not the end of the world, but annoying all the same.

Originally our goal was to do a complete refresh of TeamCity, copying it into a new AWS account, with a new URL and upgrading it to the latest version. We were already going to be disrupted for a few days, so we thought we might as well make it count. We’re using TeamCity 8, and the latest is 10, which is a pretty big jump, but we were hopeful that it would upgrade without a major incident.

I should have remembered that in software development, hope it for fools. After the upgrade to TeamCity 10, the server hung on initializing for long enough that we got tired of waiting (and I’m pretty patient).

So we abandoned the upgrade, settling for moving TeamCity and rehosting it at a different URL that we still had legal ownership of.

That went relatively well. We needed to adapt some of our existing security groups in order to correctly grant access to various resources from the new TeamCity Server/Build Agents, but nothing we hadn’t dealt with a hundred times before.

Our builds seemed to be working fine, compiling, running tests and uploading nuget packages to either MyGet or Octopus Deploy as necessary.

As we executed more and more builds though, some of them started to fail.

Failures Are Opportunities To Get Angry

All failing builds were stopping in the same place, when uploading the nuget package at the end of the process. Builds uploading to Octopus Deploy were fine (its a server within the same AWS VPC, so that’s not surprising), but a random sampling of builds uploading packages to MyGet had issues.

Investigating, the common theme of the failing builds was largish packages. Not huge, but at least 10 MB. The nuget push call would timeout after 100s, trying a few times, but always experiencing the same issue.

With 26 MB of data required to be uploaded for one of our packages (13 MB package, 13 MB symbols, probably should optimize that), this meant that the total upload speed we were getting were < 300 KBps, which is pretty ridiculously low for something literally inside a data centre.

The strange thing was, we’d never had an issue with uploading large packages before. It wasn’t until we moved TeamCity and the Build Agents into a new AWS account that we started having problems.

Looking into the network configuration, the main differences I could determine were:

• The old configuration used a proxy to get to the greater internet. Proxies are the devil, and I hate them, so when we moved into the new AWS account, we put NAT gateways in place instead. Invisible to applications, a NAT gateway is a far easier way to give internet access to machines that do not need to be exposed on the internet directly. 
• Being a completely different AWS account means that there is a good chance those resources would be spun up on entirely different hardware. Our previous components were pretty long lived, so they had consistently been running on the same stuff for months.

At first I thought maybe the NAT gateway had some sort of upload limit, but uploading large payloads to other websites was incredibly fast. With no special rules in place for accessing the greater internet, the slow uploads to MyGet were an intensely annoying mystery.

There was another thing as well. We wrap our usages of Nuget.exe in Powershell functions, specifically to ensure we’re using the various settings consistently. One of the settings we were setting by default with each usage of push, was the timeout. It wasn’t set to 100 seconds though, it was set to 600.

Bugs, Bugs, Bugs

A while back I had to upgrade to the latest Nuget 3.5 release candidate in order to get a fix for a bug that was stopping us from deploying empty files from a package. Its a long story, but it wasn’t something we could easily change. Unfortunately, the latest release candidate also has a regression in it where the timeout for the push is locked at 100 seconds, no matter what you do.

Its been fixed since, but there isn’t another release candidate yet.

Rolling back to a version that allows the timeout to work correctly, stops the other thing from working.

That whole song and dance is how software feels sometimes.

With no obvious way to simply increase the timeout, and because all other traffic seemed to be perfectly fine, it was time to contact MyGet support.

They responded that its something they’ve seen before, but they do not know the root cause. It appears to be an issue with the way that AWS is routing traffic to their Azure hosts. It doesn’t happen all the time, but when it does, it tanks performance. They suggested recycling the NAT gateway to potentially get it on new hardware (and thus give it a chance at getting access to better routes), but we tried that and it didn’t make a difference. We’ve since sent them some detailed fiddler and network logs to help them diagnose the issue, but I wouldn’t be surprised if it was something completely out of their control.

On the upside, we did actually have a solution that was already working.

Our old proxy.

It hadn’t been shut down yet, so we configured the brand new shiny build agents to use the old proxy and lo and behold, packages uploaded in a reasonable time.

This at least unblocked our build pipeline so that other things could happen while we continue to investigate.

Conclusion

Disappointingly, that’s where this blog post ends. The solution that we put into place temporarily with the old proxy (and I really hate proxies) is a terrible hack, and we’re going to have to spend some significant effort fixing it properly because if that proxy instance dies, we could be returned to exactly the same place without warning (if the underlying issue is something to do with routing that is out of control).

Networking issues like the one I’ve described above are some of the most frustrating, especially because they can happen when you least expect it.

Not only are they basically unfathomable, there is very little you can do to actively fix the issue other than moving your traffic around trying to find a better spot.

Of course, we can also optimise the content of our packages to be as small as possible, hopefully making all of our artifacts small enough to be uploaded with the paltry amount of bandwidth available.

Having a fixed version of Nuget would be super nice as well.

Like any normal, relatively competent group of software developers, we write libraries to enable reuse, mostly so we don’t have to write the same code a thousand times. Typically each library is small and relatively self contained, offering only a few pieces of highly cohesive functionality that can be easily imported into something with little to no effort. Of course, we don’t always get this right, and there are definitely a few “Utility” or “Common” packages hanging around, but we do our best.

Once you start building and working with libraries in .NET, the obvious choice for package management is Nuget. Our packages don’t tend to be released to the greater public, so we don’t publish to Nuget.org directly, we just use the built-in Nuget feed supplied by TeamCity.

Well we did, up until recently, when we discovered that the version of TeamCity we were using did not support the latest Nuget API and Visual Studio 2015 didn’t like the old version of the Nuget feed API. This breaking change was opportune, because while it was functional, the TeamCity Nuget feed had been bugging us for a while. Its difficult to add packages that weren’t created by a Buidl Configuration and packages are intrinsically tied to the Artefacts of a Build Configuration, so they can be accidentally deleted when cleaning up. So we decided to move to an actual Nuget server, and we chose Myget because we didn’t feel like rolling our own.

That preface should set the stage for the main content of this post, which is about dealing with the difficulties in debugging when a good chunk of your code is in a library that was prepared earlier.

Step In

To me, debugging is a core part of programming.

Sure you can write tests to verify functionality and you can view your resulting artefact(webpage, application, whatever), but at some pointits incredibly helpful to be able to step through your code, line by line, in order to get to the bottom of something (usually a bug).

I’m highly suspicious of languages and development environments that don’t offer an easy way to debug, and of developers that rely on interpreting log statements to understand their code. I imagine this is probably because of a history of using Visual Studio, which has had all of these features forever, but trying to do anything but the most basic actions without a good IDE is hellish. Its one of the many reasons why I hate trying to debug anything in a purely command line driven environment (like some flavours of Linux or headless Windows).

I spend most of my time dealing with two languages, C# and Powershell, which are both relatively easy to get a good debugging experience. Powershell ISE is functional (though not perfect) and Visual Studio is ridiculously feature rich when it comes to debugging.

To tie all of this back in with the introduction though, debugging into C# libraries can be somewhat challenging.

In order to debug C# code you need debugging information in the form of PDB files. These files either need to be generated at compile time, or they need to be generated from the decompiled DLL before debugging. PDB files give the IDE all of the information it needs in order to identify the constituent components of the code, including some information about the source files the code was compiled from. What these files don’t give, however, is the actual source code itself. So while you might have the knowledge that you are in X function, you have no idea what the code inside that function is, nor what any of the variables in play are.

Packaging the PDB files along with the compiled binaries is easy enough using Nuget, assuming you have the option enabled to generate them for your selected Build Configuration.

Packaging source files is also pretty easy.

The Nuget executable provides a –symbols parameter that will automatically create two packages, the normal one and another one containing all of the necessary debug information + all of your source files as of the time of compile.

What do you do with it though?

Source Of All Evil

Typically, if you were uploading your package to Nuget.org, the Nuget executable would automatically upload the symbols package to SymbolSource. SymbolSource is a service that can be configured as a symbol server inside Visual Studio, allowing the application to automatically download debugging information as necessary (including source files).

In our case, we were just including the package as an artefact inside TeamCity, which automatically includes it in the TeamCity Nuget feed. No symbol server was involved.

In fact, I have a vague memory of trying to include both symbol and non-symbol packages and having it not work at all.

At the time, the hacky solution was put in place to only include the symbols package in the artefacts for the Build Configuration in TeamCity, but to rename it to not include the word symbols in the filename. This kept TeamCity happy enough, and it allowed us to debug into our libraries when we needed to (the PDB files would be automatically loaded and it was a simple matter to select the source file inside the appropriate packages directory whenever the IDE asked for them).

This worked up until the point where we switched to Myget.

Packages restored from Myget no longer had the source files inside them, leaving us in a position where our previous approach no longer worked. Honestly, our previous approach was a little hacky, so its probably for the best that it stopped working.

Sourcery

It turns out that Myget is a bit smarter than TeamCity when it comes to being a Nuget feed.

It expects you to upload both the normal package and the symbols package, and then it deals with all the fun stuff on the server. For example, it will automatically interpret and edit the PDB files to say that the source can be obtained from the Myget server, rather than from wherever the files were when it was compiled and packaged.

Our hacky solution to get Symbols and Source working when using TeamCity was getting in the way of a real product.

Luckily, this was an easy enough problem to solve while also still maintaining backwards compatibility.

Undo the hack that enforced the generation of a single output from Nuget (the renamed symbols package), upload both files correctly to Myget and then upload just a renamed symbols package to TeamCity separately.

Now anything that is still using TeamCity will keep working in the same way as it always was (manual location of source files as necessary) and all of the projects that have been upgraded to use our new Myget feed will automatically download both symbols and source files whenever its appropriate (assuming you have the correct symbol server setup in Visual Studio).

Conclusion

Being able to debug into the source files of any of your packages automatically is extremely useful, but more importantly, it provides a seamless experience to the developer, preventing the waste of time and focus that occurs when flow gets interrupted in order to deal with something somewhat orthogonal to the original task.

The other lesson learned (which I should honestly have engraved somewhere highly visible by now) is that hacked solutions always have side effects that you can’t foresee. Its one of those unfortunate facts of being a software developer.

From a best practices point of view, its almost always better to fully understand the situation before applying any sort of fix, but you have to know how to trade that off against speed. We could have chosen to implement a proper Nuget server when we first ran across the issue with debugging our libraries, rather than hacking together a solution to make TeamCity work approximately like we wanted, but we would have paid some sort of opportunity cost in order to do that.

At the least, what we should have done was treat symbols package rename for TeamCity in a isolated fashion, so the build still output the correct packages.

This would have made it obvious that something special was happening for TeamCity and for TeamCity only, making it easier to work with a completely compliant system at a later date.

I suppose the lesson is, if you’re going to put a hack into place, make sure its a well constructed hack.

Which seems like a bit of an oxymoron.

Another short one this time, as I still haven’t been up to anything particularly interesting recently. This should change in the coming weeks as we start work on a brand new service, which will be nice. We learnt a lot from the first one, but we still have some unanswered questions like database schema versioning and deploying/automating functional tests as part of a continuous integration environment, which should make for some good topics.

Anyway this weeks post is a simple one about a component I ran across the other day while refactoring an application.

Its called Fluent Command Line Parser and it fills one of those niche roles that you come across all the time, but never really spend any time thinking about, which is supplying command line arguments to an application.

Yes Commander?

Most of the time when you see an application that takes input from the command line, you’ll probably see assumptions about the order of elements supplied (like args[0] is the database, args[1] is the username, etc). Typically there is little, if any, validation around the arguments, and you would be lucky if there was help or any other documentation to assist the people who want to use the application from the command line. There are many reasons for this, but the most common is usually effort. It takes effort to put that sort of thing in, and unless the command line tool is going to be publically consumed, that effort probably isn’t necessary.

Fluent Command Line Parser (to be referred to as FCLP from here on) makes this sort of thing much much easier, reducing the effort to the level of “Why not?”.

Essentially you configure a parser object to recognise short and long tokens to indicate a particular command line argument, and then use the presence of those tokens to fill properties in a strongly typed object. The parser can be configured to provide descriptions of the tokens, whether or not they are required, callbacks to be run when the token is encountered and a few other things that I haven’t really looked into yet.

FCLP also offers the ability to set a help callback, executed whenever /? or /help is present, which is nice. The help callback automatically ties in with the definitions and descriptions for the tokens that have been configured, so you don’t need to document anything twice.

Enough rambling about the code, lets see an example:

public static void Main(string[] args)
{
    var parser = new FluentCommandLineParser<Arguments>();
    parser
        .Setup(a => a.Address)
        .As('a', "address")
        .WithDescription("The URL to connect to the service.");
        .Required();

    parser
        .Setup(a => a.Username)
        .As('u', "username")
        .WithDescription("The username to use for authentication.");
    
    parser
        .Setup(a => a.Password)
        .As('p', "password")
        .WithDescription("The password to use for authentication.");

    var parsedArgs = parser.Parse(args);

    // Use parsedArgs in some way.
}

private class Arguments
{
    public string Address { get; set; }
    public string Username { get; set; }
    public string Password { get; set; }
}

As you can see its pretty clear what’s going on here. Its simply to add new arguments as well, which is nice.

The Commander is Down!

Its not all puppies and roses though. The FCLP library is very simple and is missing the ability to do at least 1 important thing that I needed, which was a shame.

It only supports simple/primitive objects (like strings, integers, booleans, enums and collections of those primitives) for the properties being set in your parsed arguments object. This means that if you want to supply a Uri on the command line you have to add an extra step or an extract function on your parsed arguments class that gets the string value passed in as a Uri. Its not a total deal breaker, its just unfortunate, because it seems like something you should be able to do with a callback or something similar, but the context (the object containing the parsed arguments) is just not available at that point in the workflow.

I might take a stab at adding that ability to the library (its open source) if I get a chance. I feel like you can probably accomplish it by adding another method to override the default parsing of the object from the entered string (supplying a Func<string, T> to transform the string with custom logic).

Conclusion

If you’re looking for a nice simple library to make your command line parsing code a bit clearer, this is a good one to grab. It really does help to improve the quality of a typically messy area of an application, which makes it a lot easier for someone coming in later to understand. Just add a reference to the FluentCommandLineParser package via Nuget (you are using Nuget aren’t you? No reason not to) and off you go.

Remember, every little bit like this helps, because we don’t write code for ourselves. We write it for the people who come after us. Even if that person is you 6 months from now, when you’ve forgotten everything.

This will hopefully be a short post, as I haven’t been up to anything particularly interesting recently. We deployed the service we’d been working on, and everything appears to be going swimmingly. The log aggregation that we setup is doing its job (allowing us high visibility into the operations of the service) and we’re slowly introducing new users, making sure they have a great experience.

As a result of the recent deployment, we’re in a bit of a lull, so I took this opportunity to work on our development infrastructure a little bit.

Well, I didn’t start out with the intent to work on our infrastructure. I just wanted to update this component we have that seeds demonstration data into our new service. When I got into the code though, I found that we had a bunch of shared code that was duplicated. Mostly code for dealing with the models inherent in the service and with authentication to the service. The approach that had been taken was to simply copy the code around, which is not generally a good idea in the long run. The demonstration data component didn’t have the copied code (yay!) but it was referencing the libraries of another solution directly from their output directory, which meant you had to remember to compile that other solution before the demo data, which is a bit crap.

All in all, it was in need of a clean up.

Fairly straight forward, split the common code into a different project and repository, create a new TeamCity build configuration to make it output a Nuget Package and reference it like that.

Its never straightforward.

Free As In Beer

We currently use TeamCity as our Nuget server, with the outputs of various build configurations being added to the Nuget server when they are compiled. The downside of this is that you can’t actually publish packages to the TeamCity Nuget server from outside of TeamCity. Another downside of the TeamCity Nuget server is that the packages are tied to the builds that made them. We have had issues in the past with packages disappearing from the the Nuget server because the build that created the package was cleaned up. It makes sense when you think about it, and we should definitely have pinned those builds to keep them forever, but its still another thing that we need to remember to do, which is vulnerable to human failure.

The free version of TeamCity is pretty great. 3 build agents and 20 build configurations are usually enough for small teams, and the product itself is pretty amazing. When you hit that build configuration limit though, you need to pay a few hundred dollars. That’s fine, its a solid product and I have no problems in giving JetBrains (the company that make TeamCity) money for their efforts.

Getting money out of some organizations though, that can be a giant pain. Which it was.

Time for an alternate approach. Time to setup our own Nuget server.

Nuget makes this pretty easy. Just create a new, blank ASP.NET web application and add a reference to the Nuget Server package. Easy.

When you’re building a piece of infrastructure though, you want to go to a bit of trouble to make it nicely encapsulated. So I leveraged some work we’d done as part of the new service and created a deployable package for our Nuget server, an Octopus project to deploy it and an environment for it to be deployed to via AWS CloudFormation. While I was there I added the logging components to make sure we have records of what is being uploaded and downloaded from the server.

Yak Shaved, Moving On

Now that I had a Nuget server separate from TeamCity, I could split out the common components into their own repository and construct a build/publish script to publish to the Nuget server from my local machine (with the intent to turn it into a TeamCity build configuration later, when we finally pay money).

I’ve done this before so I just grabbed my .NET Library Repository template and added the new solution with the project and its tests. The build script in the template automatically picks up the solution file, restores its packages, automatically generates a version, compiles it with MSBuild and then publishes a package for the single non-test project. Its pretty simple, I just use it to get reusable libraries up and running ASAP.

I did have a few difficulties with untangling the common code from the rest, which is probably why it was copied in the first place. The easiest way to go about this sort of thing is to create a brand new project and namespace and just migrate the minimum amount of code that is needed by the other components. If you try and take everything at once, you’re going to have a bad time.

Now with the magic of Nuget I just need to add a reference to the new package in the various places that need it and delete the duplicated code, running the tests (where they exist) to ensure that I haven’t broken anything.

Summary

Its important for the health of your code base to execute this sort of clean up when you see an issue like duplicated code.  Some people would say that some duplicated code isn't too bad, and I would generally agree. However, you need to be able to tell a very good story for why the code is duplicated, and not just accept that it is. Its usually not too hard to extract a library and create a Nuget package, assuming you have the infrastructure setup to support it. Which you should. If you don’t, you should set it up.

If you’ve never heard the team “Shaving a Yak” before, you can blame Scott Hanselman for that one. He did a blog post on the concept and I have never seen anything that better describes an average day in the life of a software developer. I use the term a lot whenever I’m doing something I never expected to be doing due to some crazy causality chain, which happens more than I would like.