And thus we come to the end of this particular saga, of rewriting the our AWS Lambda ELB Logs Processor in C# using .NET Core.
To summarise the posts in this series:
- I went through some of the basics around how to use C# and .NET Core to implement an AWS Lambda function
- I explained how we constructed the automated build, test and deployment pipeline for the .NET Core ELB Logs Processor
- I outlined how we tested the new .NET Core Lambda function, with a specific focus around how we handled the AWS Credentials
In conjunction with those posts, I’ve also uploaded a copy of the entire source code (build and deployment scripts included) to Solavirum.Logging.ELB.Processor.Lambda, mostly as a master reference point. Its a copy of our internal repository, quickly sanitized to remove anything specific to our organization, so I can’t guarantee that it works perfectly, but there should be enough stuff there to demonstrate the entire process working end to end.
As with all the rest of our stuff, the build script is meant to run from inside TeamCity, and the deployment script is meant to run from inside Octopus Deploy, so if you don’t have either of those things, you will have to use your imagination.
Of course, everything I’ve gone through is purely academic unless its running successfully in Production…
Seize The Means Of Production
Unsurprisingly, the difference between a sample file with a few hundred lines in it and a real file with hundreds of thousands of lines is significant, and that’s not even really taking into account the difference between my local development environment and AWS Lambda.
In an unprecedented result though, the .NET Core implementation worked just fine when I shipped it to our production environment.
Unfortunately it was around twenty times slower.
The .NET Core implementation processes a file in around 200 seconds, which is a hell of a lot slower.
Now, I can explain some of that away with the fact that the .NET Core implementation uses HTTP instead of TCP to communicate with Logstash (which was the primary reason I rewrote it in the first place), but it was still pretty jarring to see.
Interestingly enough, because executing Lambda functions is so cheap, the total cost increase was like $US8 a month, even though it was twenty times slower. so who really cares.
Overall, I’m pretty happy with how this whole .NET Core rewrite went, for a number of reasons:
- I got to explore and experiment with the newest .NET and C# framework, while delivering actual business value, which was great.
- In the process of developing this particular solution, I managed to put together some more reusable scripts and components for building other .NET Core components
- I finally switched the ELB Logs Processor to HTTP, which means that the overall feed of events into our ELK stack is more reliable (TCP was prone to failure)
- I managed to get a handle on fully automated testing of components that require AWS Credentials, without accidentally checking the credentials into source control