• Setting up my Ubiquiti Network

    For a while now, I’ve been having problems with my Virgin Media Super Hub 3 and the Wifi randomly dropping out. At first I attributed it to bad devices (old 2Ghz stuff), and wasn’t that bothered as I mostly used a wired connection on my Desktop PC. However, since moving house I’m unable to use a wired connection - my PC and the Fibre are in opposite corners of the house - and even with a brand new Wifi Card, I’ve been experiencing the same problems. Another issue was that I could only get 150Mbps over Wifi - when I’m paying for 200Mbps.

    I could have gone to Virgin Media support and requested a replacement, it would probably have been some hassle, but I’m sure they would have sorted it eventually.

    But, I still wanted something a bit better than what their standard Router/Wifi could offer, so it was time for an overhaul.

    After seeing some blogs on people implementing Ubiquiti products in their house, I thought I’d give it a go.

    unifi bits

    I didn’t buy all that, but it’s pretty looking stuff

    I’ll be the first to admit that I’m never the best at buying things online, and I’m no networking expert. So I ended up buying what I thought was enough bits - and technically it was - without proper research.

    What initially I bought was:

    • AC-PRO (Wifi)
    • USG (Router)
    • Cloud Key (a way to manage everything)

    The first problem I came across was that I didn’t have enough Ethernet cables in my house (thrown away during the move). So I borrowed a couple from the office, and liberated one from another device in the house.

    With just 4 Ethernet cables I just about managed to get everything setup, but it wasn’t pretty.

    Initially I setup the USG, and then added the AC-PRO. To do this I had to setup the Controller Software on my Desktop, then I got around to setting up the Cloud Key, and then realised that it worked as the Controller instead of what is on my Desktop, so had to start all over again.

    I really struggled to get everything on the same LAN and keep internet connected - at times I had to remove the Ethernet cable providing Internet so I could connect a Computer to the LAN to setup the Wifi, then with the Wifi setup I could disconnect the Ethernet cable to reconnect the Internet.

    Lesson 1

    Have enough Ethernet Cables before you start!

    Lesson 2

    Check how everything will connect together - I foolishly thought that the USG had built-in Wifi and the AC-PRO was a booster.

    The little research I did online said you could use just the three devices without a switch, but I don’t see how people managed.

    In the end I used the 3 spare ports on the VM Superhub (whilst in modem mode) as a switch for the AC-PRO, USG and Cloud Key.

    Lesson 3

    Setup the Cloud Key before anything else. Don’t download the Controller and start Adopting all the devices to then realise you can do it all on the Cloud Key.

    The Problems

    I was happy everything worked. I could get 200Mbps + speeds over Wifi again - something I wasn’t able to do with the SuperHub:

    Before

    before speed

    After

    after speed

    The problem I had now was that the AC-PRO was in a corner with everything else meaning I wasn’t getting the best range, ideally I wanted it the middle of my house. Moving it would require a power and Ethernet cable running the 10M+ to it, as well the the Power Adaptor, which would be ugly and not pass the Spouse Approval Test.

    I also had an abundance of things plugged in in that corner, so I needed a way to move it and make it pretty.

    Solution

    I decided to fork out a little more money and get a Ubiquiti switch with PoE (power over ethernet) coming out from it so that I could power the AC-PRO (and Cloud Key) without a power cable.

    As those are the only 2 requirements for PoE I got a:

    • US-8-60W

    to add into the mix.

    That provides 4 PoE ports, and is capable of powering a Cloud Key and AC-PRO.

    Now I have my AC-PRO connected via a Flat White CAT7 cable, and not looking ugly at all.

    AC-PRO

    I love how you cannot see the wire above the doorway unless you really look.

    The rest of the devices are wired up with Flat Black CAT7 cables (except the Tivo).

    The Gear

    End Result

    I’m really happy with the performance of everything, and the setup was really easy - except for my own failings above. Adding the switch in was just plug-in, go to the web interface and press “Adopt”.

    The devices I have connected at the moment are:

    VM Router (modem)
              |
              |--------------------USG
                                    |
                                    |
                                    |
                                □□□▣ □□□□
                              US-8-60W (switch)
                                    |  
                                    |
            ------------------------|--------------------
            |           |           |           |       |
          PS4         Tivo       Pi Hole    Cloud Key   |
        ▣□□□ □□□□   □▣□□ □□□□   □□□□ □□▣□   □□□□ □□□▣ |
                                                        |
                                                        |
                                                      AC-PRO  (wifi)
                                                    □□□□ □▣□□
    

    The Management via the Cloud Key / Controller is awesome. There are so many settings and it is so easy to control everything. I’ve not had a proper play yet, but so far my favourite feature is been able to assign Alias’s to devices so I know what everything is - most phones just show up as MAC addresses on the Superhub. Simple things like that always make me happy.

    Final thoughts

    I started writing this post a few months ago, but due to the stresses of moving house, it’s taken me 6 months to complete. But now I’ve had some time running with the above setup I can say that it is rock solid. I’ve had no problems, and no complaints from the family either - you know you got it right if they don’t complain.

    Changes since I started:

    • I’ve added a pi-hole to my network to block ads on mobile devices. This is something I wouldn’t have been able to do on the VM router, as I could not assign DNS to the DHCP clients, and manually changing it per device would not have been acceptable.
    • I’ve installed the Unifi Network app on my phone to help manage it when I’m away.
    • I’ve turned off the blue glow on the AC-PRO - it’s pretty, but it did make the house glow all night.

    Other than that, I’ve just been applying the odd updates and keeping an eye on things.

    If anyone is thinking about getting setup with this, feel free to reach out to discuss, I can share what little I know and maybe save you from my mistakes :)

  • Deploy a website for a Pull Request

    Some of my colleagues shared a great idea the other day for one of our internal repositories…

    Spin up a new website for each PR so you can see what the finished site will look like.

    I really like this idea, so I thought I’d change the Fable Xmas List to deploy a new version on each PR I submitted.

    Note: I’ve only done this for my repository, not Forks.

    Previous Setup

    The Xmas List code is built and deployed by Azure Pipelines from my Public Azure DevOps to a static website in an AWS S3 Bucket.

    The previous process was to only trigger a Build on pushes to master and if everything succeeded then a release was triggered automatically to push the latest code into the Bucket.

    The live site lives on:

    https://s3-eu-west-1.amazonaws.com/xmaslist/index.html

    Plan

    The plan is to deploy each Pull Request to another bucket with a naming convention of:

    https://s3-eu-west-1.amazonaws.com/xmaslist-pr-branch-name/index.html

    I could have use subfolder in another bucket, but I thought I’d keep it simple here.

    The Pipeline for pushes to master will remain unchanged.

    Implementing

    To get this to work, you will need to change the Build and Release pipelines.

    Build

    The first thing you will need to do is get the name of the Pull Request branch in the Release. At the moment this is only available in the Build via the SYSTEM_PULLREQUEST_SOURCEBRANCH variable.

    I’ll use the UPPER_CASE version of the variable names when working in PowerShell and the $(Title.Case) version when working in the Task.

    To pass the value of a variable from the Build to the Release you will have to add it into the Pipeline Artifact. As I only had a single value to pass, I just used a text file with the value of the variable in it.

    I added a PowerShell Task and used an inline script:

    PowerShell Build Task

    The script is:

    $Env:SYSTEM_PULLREQUEST_SOURCEBRANCH > $(Build.Repository.LocalPath)\deploy\branch.txt
    

    deploy is the root of the published website

    To stop a file been added to the live site deployments, I set the Custom Conditions on the task to:

    Build Custom Condition

    and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/master'))
    

    It only writes the file if the source branch is not equal (ne) to master.

    The Build will now publish the Pipeline artifact for Pull Requests with the name of the PR branch in a file called branch.txt.

    This is a little bit of a pain, but it is the only way I can find.

    Note: There is a variable in the Release Pipeline called Release.Artifacts.{alias}.SourceBranchName but in a Pull Request this is set to merge. That is because we build the PR branch of refs/pull/5/merge. There isn’t a Pull Request source branch name in Releases at this moment.

    Releases

    To enable a release on a Pull Request you first need to alter the triggers…

    Triggers

    Click on the Continuous Deployment Trigger icon

    Release triggers

    and then Enable the Pull Request Trigger and set the source branch:

    Pull Request trigger

    To keep things simple I created a duplicate Stage of the live stage and called it PR Deployment(s) and changed it’s pre-deployment conditions to run on Pull requests:

    Pre-deployment conditions

    Stages

    With the duplicate stage setup, I needed to add some extra logic to change the bucket path on AWS.

    Again, as I was keeping things simple, I just duplicated and changed the stage. I could have created Task Group and made the Tasks conditional, but this way is easier to know what each stage does.

    To get the Branch name available to the Agent I needed to get the contents of the branch.txt file from the Pipeline Artifact that was created by the build.

    I added a PowerShell task with an Inline script with the following:

    $p = $Env:AGENT_RELEASEDIRECTORY + '\' + $Env:RELEASE_PRIMARYARTIFACTSOURCEALIAS + '\drop\branch.txt'
    $PRBranch = Get-Content $p -Raw 
    del "$p"
    
    Write-Host $PRBranch #for debugging
    
    Write-Host "##vso[task.setvariable variable=PRBranch;]$PRBranch"
    

    This gets the path to branch.txt into a variable called $p, reads the entire contents into a variable called $PRBranch, and deletes branch.txt so it isn’t published.

    The line Write-Host "##vso[task.setvariable variable=PRBranch;]$PRBranch" will set a variable called $(PRBranch) in the Build agent, so that I can access it in the AWS tasks later.

    The final piece is to use this in the S3 tasks:

    S3 path with PR Branch Name

    Note: $(BucketName) is set to xmaslist.

    The last thing I added was to write out the URL of the website at the end of the Process so I can just grab it from the logs and try without having to remember the address.

    Summary

    This is a really nice way to test out any changes on a version of your site before merging a pull request, even it is only for your own PR’s. This will be much more powerful if a team is working on the repository.

    There will be many different ways to achieve this, especially if you are using Infrastructure as Code (e.g. ARM Templates on Azure), but this works even on simple static sites.

  • Santa's Xmas List in F# and Fable

    This post is part of the F# Advent Calendar 2018. Many thanks to Sergey Tihon for organizing these.

    So this year I decided to write something for the F# Advent Calendar, and even though I picked a date far enough in the future, panic still set in. I’m not one for “ideas on demand”, and after a bit of deliberating about Xmas themed games, I finally settled on something that let me explore my favourite parts of F# at the moment:

    • Domain Modelling
    • Testing
    • Event Sourcing
    • Fable & Elmish

    The Concept

    My initial design was for something a bit more complicated, but I scaled it down into simple Web App where Santa can:

    • Record children’s names
    • Who’s been Naughty and who’s been Nice
    • What presents Nice children are getting
    • See an overall list of all the presents he needs to sent to the elves

    Screen shot

    Click here to have a play

    The app is written in F#, using Fable, Elmish and Fulma (which I also used to write Monster Splatter) and all the associated tooling in SAFE stack. I did consider writing a back-end for it, but decided to keep things simple.

    The Domain Model

    A common problem with any Model-View-X architecture is that everything that isn’t POCO (Model) or UI (View) related ends up X, so I look for ways to make sure the Domain logic can be quickly broken out and separated from X.

    With Elmish, this was very easy. I began my modelling the Domain and the Operations that can be performed on it:

    type Item = {
      Description: string
    }
    
    type NaughtyOrNice =
      | Undecided
      | Nice of Item list
      | Naughty
    
    type Child = {
      Name: string
      NaughtyOrNice: NaughtyOrNice
    }
    
    type SantasItem = {
      ItemName: string
      Quantity: int
    }
    
    type Model = {
      CurrentEditor: CurrentEditorState //Not shown
      ChildrensList: Child list
      SantasList: SantasItem list
    }
    
    type AddChild = string -> Model -> Model * EventStore.Event
    type AddItem = string -> Item -> Model -> Model * EventStore.Event
    type ReviewChild = string -> NaughtyOrNice -> Model -> Model * EventStore.Event
    

    There’s a few things above, so let’s go through the types:

    1. The Model holds a list of Child records and SantaItem records.
    2. A child has a name and a Naughty or Nice status. If they are Nice, they can also have a list of Items.
    3. Santa’s items have a quantity with them.
    4. I didn’t separate the UI stuff (CurrentEditor) from the Domain model, this was just to keep things simple.

    And the functions:

    1. AddChild takes in a string for the childs name as well as the current model and returns an updated model and Event (see below)
    2. AddItem takes in a child’s name, an item, and the current state and also returns an updated model and Event.
    3. ReviewChild also takes in a child’s name and if they are naughty or nice, as well as the current state, and guess what, returns an updated model and Event.
    4. The Event is explained in the Event Sourcing section below, but is simple a Union Case representing what just happened.

    There’s no need to go into implementation of the Domain, it’s pretty basic, but it is worth pointing out that Adding an item to a Nice child, also adds an item to SantasList, or increments the quantity of an existing item.

    Reuse-Reuse-Reuse

    The main take away here is that the Domain module contains pure F#, no Fable, no Elmish, just my Domain code. This means if I wanted to run it on my F# Services I could use the exact same file and be guaranteed the exact same results.

    Full source can be seen here.

    Testing

    I just said I could be guaranteed the exact same results if I ran this code on my Services… but how?

    Fable transpiles my F# into JavaScript and runs it in the browser, how could I know this works the same in .NET Core when run on the server?

    The answer is Testing - the only way you can be sure of anything in software.

    Using the Fable Compiler’s tests as inspiration and the Fable bindings for Jest, I’ve created a suite of tests that can be run against the generated JavaScript and the compiled .NET code.

    As of writing there is a Bug with Fable 2 and the Jest Bindings, but you can work around them.

    The trick is to use the FABLE_COMPILER compiler directive to produce different code under Fable and .NET.

    For example the testCase function is declared as:

    let testCase (msg: string) (test: unit->unit) =
      msg, box test
    

    in Fable, but as:

    open Expecto
    
    let testCase (name: string) (test: unit -> unit) : Test =
      testCase name test
    

    in .NET Code.

    Full source can be seen here.

    What this gives me is a test can now be written once and run many times depending how the code is compiled:

    testCase "Adding children works" <| fun () ->
        let child1 = "Dave"
        let child2 = "Shaw"
    
        let newModel =
            addChild child1 defaultModel
            |> addChild child2
    
        let expected = [
            { Name = child1; NaughtyOrNice = Undecided }
            { Name = child2; NaughtyOrNice = Undecided } ]
    
        newModel.ChildrensList == expected
    

    What’s really cool, is how you can run these tests.

    The JS Tests took 2 different NPM packages to get running:

    • fable-splitter
    • Jest

    Both of these operated in “Watch Mode”, so I could write a failing test, Ctrl+S, watch it fail a second later. Then write the code to make it pass, Ctrl+S again, and watch it pass. No building, no run tests, just write and Save.

    As the .NET tests are in Expecto, I can have the same workflow for them too with dotnet watch run.

    I have all 3 tasks setup in VS Code and can set them running with the “Run Test Task” command. See my tasks.json and packages.json files for how these are configured.

    Test Terminals

    I have a CI/CD Pipeline setup in Azure Dev Ops running these tests on both Windows and Ubuntu build agents. That takes 25 written tests to 100 running tests.

    Event Sourcing

    As I decided to avoid building a back-end for this I wanted a way to maintain the state on the client by persisting it into Local Storage in the browser.

    Instead of just serializing the current Model into JSON and storing it, I thought I’d try out storing each of the users actions as an Event and then playing them back when the user (re)loads the page.

    This isn’t a pure event sourcing implementation, but one that uses events instead of CRUD for persistence. If you want to read a more complete introduction to F# and Event Sourcing, try Roman Provazník’s Advent Post.

    Most of the application is operating on the “View / Projection” of the events, instead of the Stream of events.

    To model each event I create a simple discriminated union for the Event and also used type aliases for all the strings, just to make it clearer what all these strings are:

    type Name = string
    type Item = string
    type Review = string
    
    type Event =
      | AddedChild of Name
      | AddedItem of Name * Item
      | ReviewedChild of Name * Review
    

    These are what are returned from the Domain model representing what has just changed. They are exactly what the user input, no normalising strings for example.

    The “Event Store” in this case is a simple ResizeArray<Event> (aka List<T>), and each event is appended onto it.

    Every time an event is appended to the Store, the entire store is persisted into Local Storage. Fable has “bindings” for access local storage which mean you only need to call:

    //Save
    Browser.localStorage.setItem(key, json)
    
    //Load
    let json = Browser.localStorage.getItem(key)
    

    Local Storage

    For serialization and deserialization I used Thoth.Json and just used the “Auto mode” on the list of Events.

    When the page is loaded all the Events are loaded back into the “Event Store”, but now we need to some how convert them back into the Model and recreate the state that was there before.

    In F# this is actually really easy.

    let fromEvents : FromEvents =
      fun editorState events ->
    
        let processEvent m ev =
          let updatedModel, _ =
            match ev with
            | EventStore.AddedChild name -> m |> addChild name
            | EventStore.ReviewedChild (name, non) -> m |> reviewChild name (stringToNon non)
            | EventStore.AddedItem (name, item) -> m |> addItem name { Description = item }
          updatedModel
    
        let state0 =
          createDefaultModel editorState
    
        (state0, events)
        ||> List.fold processEvent
    

    It starts by declaring a function to process each event, which will be used by the fold function.

    The processEvent function takes in the current state m and the event to process ev, matches and deconstructs the values from ev and passes them to the correct Domain function, along with the current model (m) and returns the updated model (ignoring the returned event as we don’t need them here).

    Next it creates state0 using the createDefaultModel function - you can ignore the editorState, as I mentioned above, it has leaked in a little.

    Then it uses a fold to iterate over each event, passing in the initial state (state0) and returning a new state. Each time the fold goes through an event in the list, the updated state from the previous iteration is passed in, this is why you need to start with an empty model, which is then built up on with the events.

    Summing Up

    There’s a lot more I could have talked about here:

    • How I used Fulma / Font Awesome for the Styling.
    • How I used Fable React for the UI.
    • How I used Azure Pipelines for the CI/CD Pipeline to S3.
    • How I never needed to run a Debugger once.
    • How I used FAKE for x-plat build scripts.

    But, I think this post has gone on too long already.

    What I really wanted to highlight and show off are the parts of F# I love. Along with that, the power of the SAFE-Stack for building apps that are using the same tech stacks people are currently using, like React for UI and Jest for Testing, but show how Fable enables developers to do so much more:

    • 100% re-usable code
    • Type safe code
    • Domain modelling using Algebraic Data Types
    • Event Sourcing
    • Familiarity with .NET
    • Functional Architecture (Elmish).

    I also wanted to share my solutions to some of the problems I’ve had, like running the tests, or setting up webpack, or using FAKE.

    It doesn’t do everything that the SAFE Demo applications do, but I hope someone can find it a useful starting point for doing more than just TODO lists. Please go checkout the source, clone it and have a play.

    If anyone has any questions or comments, you can find me on Twitter, or open an Issue in the Repo.

    Don’t forget to have a play ;)

  • Playing with Fable and the SAFE Stack

    I’ve recently started looking at Fable as way to use F# to write Web Apps.

    For the past 2 years I have had a game that I wrote in TypeScript as a playground for learning more about the language. However, not been a JavaScript or a game developer I think I had some fundamental problems with the app that I never managed to escape.

    Over the past few months Fable has kept appearing on my twitter stream and looked really interesting, especially as it can create React Web Apps, which is something I need to know more about.

    I began by using the SAFE-Dojo from CompositionalIT as a playground to learn and found it did a real good job of introducing the different parts of the SAFE-Stack.

    Using it as a reference, I managed to re-write my game in Fable in very little time.

    If you want to see it in action you can have a look here. It’s quite basic and doesn’t push the boundaries in away, but it’s inspired by my Daughter, and she loves to help me add features.

    Monster Splatter

    Play it now / View Code

    Why do I love SAFE?

    There are a number of awesome features of this whole stack that I want to shout about:

    Less Bugs

    With the old version, I found managing state really hard, there was a persistent bug where the user could click “hit” twice on the same monster and get double points.

    With Fable and Elmish, you have a really great way of managing state. Yes, it is another model-view-everything else approach. But the idea of the immutable state coming in and new state been returned is a great fit for functional programming.

    You are also coding in F# which can model Domains really well meaning you are less likely to have bugs.

    Less Code

    I’m always surprised by how small each commit is. I might spend 30 minutes or more messing with a feature, but when I come to commit it, it’s only ever a few lines of code. Even replacing the timer for the entire game was a small change.

    Fulma, or should I say Bulma

    The SAFE Stack introduced me to Fulma which is a set of Fable helpers for using Bulma.

    At first I struggled to get to grips with Fulma, but once I realised how it just represented the Bulma stylings, I found it much easier. Even someone as bad at UI as me, can create something that doesn’t look terrible.

    I mostly kept the Bulma documentation open when styling the app as it had better examples and I could translate them to Fulma in my head.

    It’s React

    React is quite a big thing at the moment, and something I’m looking to use at work. Having something that is React, but isn’t pure JS is great for me. It also supports Redux, so things like the Chrome React and Redux developer tools work with it.

    These are amazingly useful tools for debugging Web Apps, even ones this simple.

    Conclusion

    I’m going to keep looking for situations where I can use the SAFE-Stack. Next will have to be something more complicated - with multiple pages and a back-end with some persistence.

    This will give me a feel if it could be something I could use everyday - I’d really like to code this way all the time.

    I’m already looking to push F# at work, and this would be a great compliment.

    Play it now / View Code

  • You might not be seeing exceptions from SQL Server

    This post describes a problem I noticed whereby I wasn’t seeing errors from my SQL code appearing in my C#/.NET code.

    I was recently debugging a problem with a stored procedure that was crashing. I figured what caused the stored procedure to crash and replicated the crash in SQL Management Studio, but calling it from the application code on my development environment didn’t throw an exception. What was even stranger was that the bug report was from an exception thrown in the C# code, I had the stack trace to prove it.

    After a bit of digging through the code, I noticed a difference between my environment and production that meant I wasn’t reading all the results from the SqlDataReader.

    The C# was something like this:

    var reader = command.ExecuteReader();
    
    if (someSetting) //Some boolean I didn't have set locally.
    {
        if (reader.Read())
        {
            //reading results stuff.
        }
        reader.NextResult();
    }
    

    Changing someSetting to true in my development environment resulted in the exception been thrown.

    What’s going on?

    The stored procedure that was crashing looked something like this:

    create procedure ThrowSecond
    as
    
    --Selecting something, anything
    select name
    from sys.databases
    
    raiserror (N'Oops', 16, 1); --This was a delete violating a FK, but I've kept it simple for this example.
    

    It turns out that if SQL raises an error in a result set other than the first and you don’t try and read that result set, you won’t get an exception thrown in your .NET code.

    I’ll say that again, there are circumstances where SQL Server raises an error, and you will not see it thrown in your .NET Code.

    Beware transactions

    The worst part of this… if you are using transactions in your application code, e.g. using TransactionScope, you will not get an exception raised, meaning nothing will stop it calling Complete and committing the transaction, even though part of your operation failed.

    void Update()
    {
        using (var tx = TransactionScope())
        {
            DeleteExisting(); //Delete some data
            InsertNew(); //Tries to save some data, but SQL Errors, but the exception doesn't reach .NET
    
            tx.Complete();
        }
    }
    

    In the above hypothentical example if InsertNew() happens to call a stored procedure like before and is using C# like in the previous examples. It will delete the existing entry, but will not insert a new entry.

    When does it happen?

    To figure out when this does and doesn’t happen I wrote a number of tests.

    Using 3 different stored procedures and 4 different ways of calling it from C#.

    Stored Procedures

    create procedure ThrowFirst
    as
    raiserror (N'Oops', 16, 1);
    
    select name
    from sys.databases
    go
    
    create procedure ThrowSecond
    as
    select name
    from sys.databases
    
    raiserror (N'Oops', 16, 1);
    go
    
    create procedure Works
    as
    select name
    from sys.databases
    go
    

    CSharp

    void ExecuteNonQuery(SqlCommand cmd)
    {
        cmd.ExecuteNonQuery();
    }
    
    void ExecuteReaderOnly(SqlCommand cmd)
    {
        using (var reader = cmd.ExecuteReader())
        {
        }
    }
    
    void ExecuteReaderReadOneResultSet(SqlCommand cmd)
    {
        using (var reader = cmd.ExecuteReader())
        {
            var names = new List<String>();
            while(reader.Read())
                names.Add(reader.GetString(0));
        }
    }
    
    void ExecuteReaderLookForAnotherResultSet(SqlCommand cmd)
    {
        using (var reader = cmd.ExecuteReader())
        {
            var names = new List<String>();
            while (reader.Read())
                names.Add(reader.GetString(0));
            reader.NextResult();
        }
    }
    

    Results

    The results are as follows:

    Test Procedure Throws Exception
    ExecuteNonQuery ThrowFirst
    ExecuteNonQuery ThrowSecond
    ExecuteNonQuery Works n/a
    ExecuteReader Only ThrowFirst
    ExecuteReader Only ThrowSecond
    ExecuteReader Only Works n/a
    ExecuteReader Read One ResultSet ThrowFirst
    ExecuteReader Read One ResultSet ThrowSecond
    ExecuteReader Read One ResultSet Works n/a
    ExecuteReader Look For Another ResultSet ThrowFirst
    ExecuteReader Look For Another ResultSet ThrowSecond
    ExecuteReader Look For Another ResultSet Works n/a

    Explained

    The two problematic examples have a ❌ against them.

    Those are when you call ExecuteReader with the ThrowSecond stored procedure, and don’t go near the second result set.

    The only times where calling ThrowSecond will raise an exception in the .NET code is when using either, ExecuteNonQuery() (no good if you have results) or you call reader.NextReslt() even when you only expect a single result set.

    XACT_ABORT

    I tried setting SET XACT_ABORT ON but that made no difference, so I’ve left it out of the example.

    Conclusion

    I’m not sure what my conclusion is for this. I could say, don’t write SQL like this. Perform all your data-manipulation (DML) queries first, then return the data you want. This should stop errors from the DML been a problem because they will always be prior to the result set you try and read.

    However, I don’t like that. SQL Management Studio does raise the error and I wouldn’t want to advocate writing your SQL to suite how .NET works. This feels like a .NET problem, not a SQL one.

    I will say don’t write stored procedures that return results, and then write C# that ignores them. That’s just wasteful.

    The only other solution would be to ensure you leave an extra reader.NextResult() after reading all of your expected result sets. This feels a little unusual too, and would probably be removed by the next developer, who could be unaware of why it is there in the first place.

    So in the end, I don’t know what’s the best approach, if anyone has any thoughts/comments about this, feel free to contact me on twitter.

    Downloads

    You can download the fully runnable examples from here:

    They are LINQPad scripts that run against a LocalDB called “Dave”.

  • XMAS Pi Fun

    I’ve had a few days off over XMAS, so I decided to have a play with my Raspberry Pi and the 3D XMAS Tree from ThePiHut.com.

    With my (very) basic Python skills I managed to come up with a way of using a Status Board on one Pi to control the four different light settings on the XMAS Tree, running on another Pi (the “tree-berry”).

    (sorry about the camera work, I just shot it on my phone on the floor)

    All the source for this is on my GitHub, if you want to see it.

    How it works

    The “tree-berry” Pi has a Python SocketServer running on it, receiving commands from the client, another Python program running on the other Pi.

    The server is very rudimentary. Each light setting was initially written as a separate python script with different characteristics on how it runs: some have while True: loops, others just set the lights and pause. To save me from figuring out in how to “teardown” each setting and start a new one, I decided to fork a new process from the server, and then kill it before changing to the next setting. This makes it slow to change, but ensures I clean up before starting another program.

    The 2 consoles can be seen side by side here:

    Console Outputs

    There’s a lot I need to learn about Python, but this is only for a few weeks a year ;).

  • Drag and Drop Batch Files

    This is a little trick that can make dealing with batch files a real breeze.

    You can make a batch file support drag and drop.

    Drag and drop

    Here I’ve create a simple batch file that takes in a single argument tells you that it is listing the file then prints the contents using the TYPE command and then PAUSE’s.

    @echo off
    echo Listing the contents of %1
    echo.
    type %1
    echo.
    echo.
    pause
    

    This works because when you drop a file on a executable in Windows the first argument passed to that program is the name of the file you dropped on it. So in the above script %1 is the full path to whatever file you drop on the batch file.

    I’ve used this in a few different ways:

    1. SDelete: I have a batch file to call SDelete with 64 passes . I created a shortcut to the batch file with an icon (so it looks nice), that I use for deleting sensitive files at work.
    2. Restoring development databases: I have another a batch file to restore development database backups, first it unzips the archive and then runs restore via SQLCMD.

    I’m sure there are a lot more uses for this. If you want to process multiple files you can iterate through all the arguments.

    Thanks to bepe from XDA Developers who was the person who first showed me this technique in his ROM Kitchen videos many years ago.

  • Handling results from complex operations

    In this article I am going to look at a number of different approaches to model the results of a complex operation in C#. This is a technique I find useful when I have to perform some logic, and it can return different types of results, depending on the outcome. I’ll start with some naive approaches, before looking at two options that look more promising.

    I’ll be using the following logic for my “complex operation”:

    if length(s) is even
        return length(s);
    else
        return "Length is odd";
    

    With an additional a requirement that any exceptions in the “complex operation” will be handled in a suitable way.

    This is actually something I have had to implement at work in a number of cases; try an operation, then depending on the outcome handle it in the most appropriate way.

    In the examples I’ll be using Console.WriteLine for simplicity, but in the real world there could be database calls, UI updates, HTML rendering, service calls, whatever usually makes testing hard.

    The inputs to and outputs from every example will be the same.

    Inputs:

    • "Food" (returns 4)
    • "Foo" (returns Length is odd.)
    • null (returns NullReferenceException)

    All the code for the following is available in my GitHub repo ResultType-blog - these are Linqpad scripts, but can easily be modified to be C# by removing the first line.

    1. Just do it

    Here’s the most trivial approach to solve the issue:

    void Main()
    {
        ComplexOperation("Food");
        ComplexOperation("Foo");
        ComplexOperation(null);
    }
    
    void ComplexOperation(String input)
    {
        try
        {
            if (input.Length % 2 == 0)
                Console.WriteLine($"Even length: {input.Length}.");
            else
                Console.WriteLine("Length is odd.");
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex);
        }
    }
    

    This meets our requirements, but lets see if we can see a few issues with it.

    Firstly, it isn’t possible to test the Complex Operation on it’s own, you have no way to mock out the dependencies. Secondly, you’re mixing business logic, with side effects. Readers of Mark Seemann’s blog will know that this makes code harder to reason about.

    A common approach to solve this is to introduce a type to model the result of the complex operation. The remaining examples look at different approaches to do this.

    2. Implicit Results

    Let’s start with an Implicit Result type:

    class Result
    {
        public String FailureMessage { get; set;}
        public Int32? EvenLength { get; set; }
        public Exception Error { get; set;}
    }
    

    I call it Implicit because if I passed you an instance of it, you have no obvious way of knowing what the result is or how to figure out what happened during the complex operation. You could check that EvenLength is not null and assume success, but what’s to say I didn’t put set it to 0 and populate FailureMessage instead? It it mostly guess work and assumptions to know what this result contains.

    Here’s the new program and complex operation using this type:

    void Main()
    {
        var inputs = new[] { "Food", "Foo", null, };
    
        foreach (var result in inputs.Select(ComplexOperation))
        {
            if (result.Error != null)
                Console.WriteLine(result.Error);
            else if (result.FailureMessage != null)
                Console.WriteLine(result.FailureMessage);
            else
                Console.WriteLine($"Even length: {result.EvenLength}.");
        }
    }
    
    Result ComplexOperation(String input)
    {
        try
        {
            if (input.Length % 2 == 0)
                return new Result { EvenLength = input.Length, };
            else
                return new Result { FailureMessage = "Length is odd.", };
        }
        catch (Exception ex)
        {
            return new Result { Error = ex, };
        }
    }
    

    Now you see the implementation you know there is no funny business, but I made you read the complex operation to be sure.

    There are some other problems with this too:

    • The type is mutable, meaning something might change the result after the operation.
    • When the type is constructed, it is half-full, some members will have values, some won’t.
    • You have to know how to process a result, I’d find myself asking can an error also have a message?.
    • It violates the Open/closed principle because changes to Success, Failure or Error require a change to this one type.

    This does, however, separate the operation from the outputs, making the operation testable without it needing any additional depencies for the output. The operation is now Pure, which is why I can use Select on each input to return the result.

    3. Explicit Results

    I’ve harped on enough about how bad it it that the result is implied in the previous example. So let’s have a go at been more explicit.

    Here’s a result type with an enum to tell you what happened:

    class Result
    {
        public String FailureMessage { get; set;}
        public Int32? EvenLength { get; set; }
        public Exception Error { get; set; }
        public ResultType Type { get; set; }
    }
    
    enum ResultType
    {
        Success,
        Failure,
        Error,
    }
    

    Now when you get a result you can first check the Type and know it is an error.

    Here’s the application code for it:

    void Main()
    {
        var inputs = new[] { "Food", "Foo", null, };
    
        foreach (var result in inputs.Select(ComplexOperation))
        {
            switch (result.Type)
            {
                case ResultType.Success:
                    Console.WriteLine($"Even length: {result.EvenLength}.");
                    break;
                case ResultType.Failure:
                    Console.WriteLine(result.FailureMessage);
                    break;
                case ResultType.Error:
                    Console.WriteLine(result.Error);
                    break;
            }
        }
    }
    
    Result ComplexOperation(String input)
    {
        try
        {
            if (input.Length % 2 == 0)
                return new Result { EvenLength = input.Length, Type = ResultType.Success, };
            else
                return new Result { FailureMessage = "Length is odd.", Type = ResultType.Failure, };
        }
        catch (Exception ex)
        {
            return new Result { Error = ex, Type = ResultType.Error, };
        }
    }
    

    This still has some of the problems of the previous version: mutability, Open/closed principle, mixed bag of properties, and not knowing what to do with them. It also has the problem that nothing is forcing you to check the Type, I might just say It’s OK, this won’t fail, just get my the EvenLength - famous last words…

    So, whilst it is a little better, it can still lead to unreasonable code.

    4. Explicit - with factory methods

    To solve the problem of people creating a “mixed bag” of mutable properties, a factory method could be created on the type to initialise the result in the correct state depending on the outcome of the operation.

    class Result
    {
        public String FailureMessage { get; private set; }
        public Int32? EvenLength { get; private set; }
        public Exception Error { get; private set; }
        public ResultType Type { get; private set; }
    
        public static Result CreateFailure(String message)
        {
            return new Result { FailureMessage = message, Type = ResultType.Failure, };
        }
    
        public static Result CreateSuccess(Int32 value)
        {
            return new Result { EvenLength = value, Type = ResultType.Success, };
        }
    
        public static Result CreateError(Exception ex)
        {
            return new Result { Error = ex, Type = ResultType.Error, };
        }
    }
    

    This changes the complex operation to look like this:

    Result ComplexOperation(String input)
    {
        try
        {
            if (input.Length % 2 == 0)
                return Result.CreateSuccess(input.Length);
            else
                return Result.CreateFailure("Length is odd.");
        }
        catch (Exception ex)
        {
            return Result.CreateError(ex);
        }
    }
    

    The rest of the program is unchanged from the previous version.

    We now have a way to know that the Result with the type success will only have an EvenValue, however we still need to ignore the other properties that don’t relate to success. There’s still nothing forcing people to check the Type, and this requires additional factory methods for every state.

    I’ve seen a number of people stop at this level, and call it “good enough” to avoid having to go to the next level. You still have unreasonable code, and have to understand things in the operation.

    5. Exceptions for control flow

    This is another approach I have seen used, I do not like it, but I thought I would include it, as I almost used it years ago before using one of the approaches in the following sections.

    void Main()
    {
        var inputs = new[] { "Food", "Foo", null, };
    
        foreach (var input in inputs)
        {
            try
            {
                var result = ComplexOperation(input);
                Console.WriteLine($"Even length: {result}.");
            }
            catch (BusinessException be)
            {
                switch (be)
                {
                    case FailureException f:
                        Console.WriteLine(f.Message);
                        break;
                    case ErrorException e:
                        Console.WriteLine(e.InnerException);
                        break;
                    default:
                        throw;
                }
            }
        }
    }
    
    Int32 ComplexOperation(String input)
    {
        try
        {
            if (input.Length % 2 == 0)
                return input.Length;
            else
                throw new FailureException("Length is odd.");
        }
        catch (Exception ex) when (!(ex is BusinessException))
        {
            throw new ErrorException(ex);
        }
    }
    
    class FailureException : BusinessException
    {
        public FailureException(String message) : base(message) { }
    }
    
    class ErrorException : BusinessException
    {
        public ErrorException(Exception inner) : base(inner) { }
    }
    
    abstract class BusinessException : Exception
    {
        public BusinessException(String message) : base(message) { }
        public BusinessException(Exception inner) : base("Something bad happened", inner) { }
    }
    

    I hope by looking at this code you can see it isn’t an ideal appraoch.

    I’ve introduced the concept of a BusinessException that the program will handle in a try...catch block. All problems in the complex operation will throw some sort of exception derived from BusinessException, which the program will then type match on. I’ve used pattern matching here, but I’ve see other approaches such as a Dictionary<Exception, Action<Exception>> that has a list of exceptions and the delegate to call.

    Using exceptions like this is the equivelent of goto, many people have said it before, so I won’t go into detail on that aspect. I did notice when writing this how hard it is to not accidentally catch your own BusinessException, this is why I have an exception filter to not handle them twice: catch (Exception ex) when (!(ex is BusinessException)). I could imagine the case where one stray try...catch could cause a lot of problems.

    6. Type per Result

    I’ve now harped one enough about not knowing what to do with results. This example removes the ambiguity and uses a separate type for each result.

    class Success : Result
    {
        public Int32 EvenLength { get; }
        public Success(Int32 value) { EvenLength = value; }
    }
    class Failure : Result
    {
        public String FailureMessage { get; }
        public Failure(String message) { FailureMessage = message; }
    }
    
    class Error : Result
    {
        public Exception Exception { get; }
        public Error(Exception ex) { Exception = ex; }
    }
    
    abstract class Result
    {
    }
    

    Each result now has its own type. Each type only has properties relating to that type of result. The results are immutable.

    The base class in this case is empty, but it might capture the input, elapsed time, or anything else you need in every result.

    The program and complex operation now are much easier to reason about, and it is a lot harder to mix things up:

    void Main()
    {
        var inputs = new[] { "Food", "Foo", null, };
    
        foreach (var result in inputs.Select(ComplexOperation))
        {
            switch (result)
            {
                case Success s:
                    Console.WriteLine($"Even length: {s.EvenLength}.");
                    break;
                case Failure f:
                    Console.WriteLine(f.FailureMessage);
                    break;
                case Error e:
                    Console.WriteLine(e.Exception);
                    break;
            }
        }
    }
    
    Result ComplexOperation(String input)
    {
        try
        {
            if (input.Length % 2 == 0)
                return new Success(input.Length);
            else
                return new Failure("Length is odd.");
        }
        catch (Exception ex)
        {
            return new Error(ex);
        }
    }
    

    I’m using C# 7’s Pattern Matching feature in the program to compare each type of the result. When I get a match it is already cast into the correct type, so s will be an instance of Success, and Success only has properties relating to a successful outcome.

    To me this is very clear what I can do next after a complex operation and what happened in the operation.

    It is reassuring to know that if I have a Success type I can only see properties relating to a successful operation, I can pass the result to another method, that accepts an instance of Success knowing it can’t be called with an Error by mistake - the type safety in the language is on your side.

    Consider these 2 methods:

    void DisplaySuccess(Result r) { }
    void DisplaySuccess(Success s) { }
    

    In all previous examples you had to have the first version, and that would either assume you called it correctly, or would have to check that r is a success. The second method can only be called with an instance of Success, you cannot pass an Error to it, making it much harder to get wrong.

    There are a few negatives to this approach, in C#’s pattern matching the compiler doesn’t check you have every case matched, adding a new result type means I need to find all instances of result handlers and update them. If you have only one handler, then this isn’t so bad.

    Another consideration is that the result’s “next step” logic - what happens next - is separated from the type. Sometimes this could be desirable, other times you might want it contained in a single place, it depends on how your application is designed and what works best. The next exmaple looks at keeping the behaviour with the result.

    7. Types with Behaviour

    I’ve fleshed the following code out a little more, to highlight one of the drawbacks of the approach. In all previous examples, I’ve left out how you might test the entire operation - passing in test doubles for Console.WriteLine into the program from the composition root would be trivial.

    However, in this case I wanted to show the extra effort needed to keep things testable.

    First, we’ll look at the base result type:

    abstract class Result
    {
        public Result(IProcessor processor)
        {
            Processor = processor;
        }
    
        protected IProcessor Processor { get;}
    
        public abstract void Process();
    }
    
    interface IProcessor
    {
        void WriteMessage(Object message);
    }
    
    class Processor : IProcessor
    {
        public void WriteMessage(Object message) => Console.WriteLine(message);
    }
    

    There’s now an additional Process member on every result and every result needs access to a IProcessor which facilitates the injection of the dependencies for the Process method.

    This is what the calling program will use to handle the result:

    void Main()
    {
        var inputs = new[] { "Food", "Foo", null, };
    
        foreach (var result in inputs.Select(ComplexOperation))
        {
            result.Process();
        }
    }
    

    This looks very neat, I get a result, I call Process.

    The problems are getting the dependencies managed in an nice way. When deriving instances of the Result you need to write the code to pass IProcessor through:

    class Success : Result
    {
        public Int32 EvenLength { get; }
        public Success(IProcessor p, Int32 value) : base(p) { EvenLength = value; }
        public override void Process() => Processor.WriteMessage($"Even length: {EvenLength}.");
    }
    
    class Failure : Result
    {
        public String FailureMessage { get; }
        public Failure(IProcessor p, String message) : base(p) { FailureMessage = message; }
        public override void Process() => Processor.WriteMessage(FailureMessage);
    }
    
    class Error : Result
    {
        public Exception Exception { get; }
        public Error(IProcessor p, Exception ex) : base(p) { Exception = ex; }
        public override void Process() => Processor.WriteMessage(Exception);
    }
    

    Each result now has an implementation of the logic to handle it. If you want to know what happens given a success, I can just look at the Success type.

    But when you create an instance, you also need to pass in an IProcessor, so the complex operation will have to do this:

    IProcessor processor = new Processor();
    
    Result ComplexOperation(String input)
    {
        try
        {
            if (input.Length % 2 == 0)
                return new Success(processor, input.Length);
            else
                return new Failure(processor, "Length is odd.");
        }
        catch (Exception ex)
        {
            return new Error(processor, ex);
        }
    }
    

    This is quite a lot of ceremony, and now the complex operation has knowledge of the IProcessor. An instance of an IProcessor would have to be injected so that it can be passed into each result. The complex operation doesn’t depend on IProcessor though, just the results, making this a kind of transient dependency.

    This example isn’t perfect, but I have used it in a number of places where I wanted to keep the logic of what to do with a result with the result, and not separated out across the code base. Usually when there’s a lot of code related to handling the result.

    I also like that I am able to write code such as:

    var result = ComplexOperation(input);
    resut.Process();
    

    If you need to add a new result, type (e.g. Timeout) you can do so by just deriving a new type from Result and implementing all the logic there. The only other place that needs a change is the complex operation to return new Timeout(processor), the program doesn’t have to change.

    8. Bonus F# version

    I am a big fan of F# so I thought I would model the same problem in F#.

    I’ve deliberately kept it similar to the C# examples to avoid it getting too functional. This is quite close to example #6 above.

    type Result =
        | Success of int
        | Failure of string
        | Error of Exception
    
    (* Unchecked.defaultof<String> is used for null to make it crash - F# doesn't do null really. *)
    let inputs = [ "Food"; "Foo"; Unchecked.defaultof<String>; ]
    
    let complexOp (input: string) =
        try
            if input.Length % 2 = 0 then
                Success input.Length
            else
                Failure "Length is odd."
        with
        | ex -> Error ex
    
    let processResult r =
        match r with
        | Success s -> printfn "Even length: %d" s
        | Failure f -> printfn "%s" f
        | Error e -> printfn "%A" e
    
    let main () =
        let results =
            inputs
            |> Seq.map complexOp
    
        results |> Seq.iter processResult
    
    main ()
    

    I’ve modelled the result as a Discriminated Union with 3 cases, one for each outcome. The complex operation, like the C# version returns 1 of these 3 cases. What is nice in F# is that in processResult where I take in a single result and handle it, the pattern match must be complete. If I added another case to the Result type, the compiler will complain that is isn’t handled in the match.

    Conclusion

    This won’t be an exhaustive list of ways to handle results, but it does provide some different approaches to the problem that should help keep your code base a little cleaner. Options 6 and 7 are ones I would use in C#, the rest create unreasonable code that I would not like to have to think about. The complex operation is never going to be a few lines of code like in my scenario, it might be many classes working to do many different operations, building one final result. I like it when I don’t have to know the implementation details of an operation to know what the behaviour is for a given outcome.

    Above, I have only used Success, Failure and Error as the outcomes of my operation, but I could have modelled different states for success too: a MatchFound/NoMatch result could be suitable for a result type.

  • Controlling VS2017 Developer Console Start Directory

    At work I use ConEmu for my console, it’s a great console to work on Windows with. To keep things tidy I have all my code on my X:\ partition. In ConEmu I have different “Tasks” setup for different configurations of Visual Studio and pass /Dir X:\ as one of the task parameters so that a new Console’s current Dir is X:\.

    ConEnum Settings

    When running “Developer Command Prompt for VS 2017” on my work computer I noticed that the directory it was opening in wasn’t the current directory that ConEmu was setting, but C:\Dave\Source.

    **********************************************************************
    ** Visual Studio 2017 Developer Command Prompt v15.0.26228.9
    ** Copyright (c) 2017 Microsoft Corporation
    **********************************************************************
    
    C:\Dave\Source>
    

    After a bit of digging through the batch files I found the reason for this is because of this bit code in:

    C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\Tools\vsdevcmd\core\vsdevcmd_end.bat

    ...
    @REM Set the current directory that users will be set after the script completes
    @REM in the following order:
    @REM 1. [VSCMD_START_DIR] will be used if specified in the user environment
    @REM 2. [USERPROFILE]\source if it exists
    @REM 3. current directory
    if "%VSCMD_START_DIR%" NEQ "" (
        cd /d "%VSCMD_START_DIR%"
    ) else (
        if EXIST "%USERPROFILE%\Source" (
            cd /d "%USERPROFILE%\Source"
        )
    )
    ...
    

    As you can see, it has two chances to pick a different directory before using your current one.

    In my case, I had a folder at %USERPROFILE%\Source, which was empty, so I deleted it.

    The other alternative is to set the VSCMD_START_DIR environment variable for your user account to your preferred directory.

  • TFS 2015 does not support 2012 build agents

    This post is part PSA, part debugging story.

    The important bit:

    Team Foundation Server 2012 XAML Build Agents do not work with TFS 2015

    I discover this fact the weekend just gone whilst performing an upgrade to TFS 2015.3 from TFS 2012.4.

    The plan was to only upgrade the TFS Server and leave the build infrastructure running on TFS 2012. This seemed like a sound idea as I know Microsoft care about compatibility, and the upgrade was more complicated than your usual one. I figured it would just keep working and that I’d upgrade the build agents later, boy was I wrong.

    I may have even checked the documentation, which does not show a compatibility, but it isn’t explicitly called out, so I could have glanced over it.

    TFS build compatibility Look - No 2012

    The problems with TFS 2012 build agents against TFS 2015 manifested as two different errors when I queued a build without a Drop Location. Queuing a build with a drop location worked just fine.

    Error 1 - Build agents not using the FQDN

    The build infrastructure runs on a different domain to the Team Foundation Server.

    We have tfs-server.corp.com for TFS and build-server.corp-development.com for builds.

    The error manifested as:

    FQDN error message

    The error that appeared twice was not very helpful.

    An error occurred while copying diagnostic activity logs to the drop location. Details: An error occurred while sending the request.

    I eventually debugged this (details later) and found out that the last task on the build agent was trying to access tfs-server with no DNS suffix of .corp.com to publish some logs. As a temporary workaround I bobbed an entry in the hosts file entry to make tfs-server point to the actual IP of the TFS server.

    Error 2 - the bad request

    With the all the steps of the build resolving the server name, I came across the second error.

    Bad request error message

    The error message was still no more use than the last one:

    An error occurred while copying diagnostic activity logs to the drop location. Details: TF270002: An error occurred copying files from ‘C:\Users\tfsbuild\AppData\Local\Temp\BuildAgent\172\Logs\151436\LogsToCopy\ActivityLog.AgentScope.172.xml’ to ‘ActivityLog.AgentScope.172.xml’. Details: BadRequest: Bad Request

    An error occurred while copying diagnostic activity logs to the drop location. Details: An error occurred while sending the request.

    My debugging would lead me to see that this was caused by TFS returning an HTTP 400 (Bad Request) for the exact same step as the first error.

    It was at this point I figured something was really wrong and started searching for compatibility problems. In my effort to find a KB or update I re-checked the documentation and noticed the lack of support as well as finding an MSDN forum post from RicRak where they solved the problem by upgrading their agents off of TFS 2012.

    Solution

    My solution was to upgrade our entire build infrastructure (some 9/10 servers) to TFS 2015, and discovering you must install VS2015 on the servers too to get the Test Runner to work.

    One day of diagnosis and testing to get to the point of knowing TFS 2015 build agents would solve the problem and still build our codebase. Another half-day was spend upgrading all the servers.

    Diagnostics

    How do you figure out when something like this goes wrong? TFS diagnostic logging did not provide any more information than minimum logging did. The error only appeared at the very end of a build, it wasn’t related to a step in the XAML workflow, nor any variables in the build process.

    The solution (as always) came from Charlie Kilian on Stack Overflow.

    I stopped the Build Service and opened up TFSBuildServiceHost.exe.config and added the following section:

    <system.diagnostics>
        <sources>
            <source name="System.Net" tracemode="includehex" maxdatasize="1024">
                <listeners>
                    <add name="System.Net"/>
                </listeners>
            </source>
        </sources>
        <switches>
            <add name="System.Net" value="Verbose"/>
        </switches>
        <sharedListeners>
            <add name="System.Net"
                type="System.Diagnostics.TextWriterTraceListener"
                initializeData="C:\Logs\network.log" />
        </sharedListeners>
        <trace autoflush="true"/>
    </system.diagnostics>
    

    Then restarted the build service and ran the smallest build I could to produce minimal logs.

    The log folder looked something like this:

    Log files on disk

    The network.log file had a few errors, but nothing fatal looking, so I looked in the other files for errors and finally found this line:

    System.Net Error: 0 : [4916] Exception in HttpWebRequest#13319471:: - The remote name could not be resolved: 'tfs-server'.
    

    That was proceeded by:

    System.Net Verbose: 0 : [4928] HttpWebRequest#13319471::HttpWebRequest(http://tfs-server:8080/tfs/DefaultCollection/_apis/resources/containers/122598?itemPath=logs%2FActivityLog.AgentScope.172.xml#752534963)
    

    Here you can see the server name without the necessary DNS suffix during some HTTP POST to _apis/resources/containers.

    This was the point I added the hosts file entry and then got the next error.

    For the second error I repeated the diagnostic logging steps and this time found the following errors (searching for Bad Request):

    System.Net Information: 0 : [16628] Connection#50276392 - Received status line: Version=1.1, StatusCode=400, StatusDescription=Bad Request.
    

    By tracing the ID (in this case 16628) back up the file I found it was a call to the same endpoint, but this time a PUT:

    System.Net Information: 0 : [16628] HttpWebRequest#9100089 - Request: PUT /tfs/DefaultCollection/_apis/resources/containers/122603?itemPath=logs%2FActivityLog.AgentScope.59.xml HTTP/1.1
    

    This was the point I gave up thinking this could be fixed by a configuration change.

    Conclusion

    I wish I had read something like this before I planned the weekend. I did do testing, but because testing TFS in live is risky I had most of the test instance network isolated and that required a lot of configurations; I just thought this error was just configuration based, lesson well and truly learned.

    It would have been nice to see this called out more explicitly on MSDN. In my opinion these are two bugs that Microsoft decided not to fix in the TFS 2012 product life-cycle.

    On the plus side, I learned some really neat debugging skills I didn’t know before.

    Remember, if you’re upgrading from TFS 2012, plan to upgrade your build agents at the same time!

subscribe via RSS