Quite a bit has changed since my end of December post. The first two months of 2020 were a blur. {Redacted} went into a full change-over, a new CTO, and a new org structure early in the year, just as I had received word from a regional credit union that they wanted me to be their new, and only, Principal Software Engineer. I had to make a excruciating decision; to leave a company and people I loved, to move over to a new-idea-to-the-org position. Instead of building things and fixing things, I would be responsible for fixing the org’s developers and development processes.

One of the things I will take away is {Redacted}’s dedication to making the customer right, if there was a mistake or error.

Knowing that we would always make the client whole allowed for experimentation and mistakes. Engineers are free there to do the best they can. There are many orgs who strive to be the very thing that {Redacted} has been.

Still, careers should not stagnate, and there was no place for me to go at {Redacted}. I had reached a space that the only place to go was if my superior retired or quit.

It has been three and a half months since I left, and I still miss everyone terribly.

Well, with the whirlwind of changes that occurred in February, naturally, a pandemic ensued which caused me to get a whopping three weeks of face-time with my new team. There are some outstanding people at my credit union, and with this past three months, I have been working to carve out my new-to-the-org role.

Some notes from the first

1. Larger organizations have larger org problems. Each subteam has its’ own micro-climate.
2. Empathy, empathy, empathy. You simply have to assume everyone is trying their best.
3. TDD is nowhere near as ubiquitous as I assumed that it was.

On item three, I have been a TDD practitioner in spurts and starts since 2005, and I was later than I should have been to the game. It has simply been a part of a majority of the code I have dealt with. When folks describe it as new, it is a bit of a shock.

The elephant in the room clearly has to be the same thing everyone is dealing with. The big CV19. Here’s the high points.

• My kids have been homeschooling since March.
• My wife was furloughed from the Y, but still teaches the occasional ZOOM yoga course. We are blessed, in that the income from my new job covers what she was bringing in.
• I do Crossfit at home, either on ZOOM, or simply by myself, and work from home completely. My outings are walking the dog, or going to the grocery store. I have visited a few wineries for club pickups, and recommend everyone go visit a local winery and support small business!

Stay safe!

Sometimes, you can see problems coming.

A vendor of ours provides security information monthly in a series of text files. Typically, those text files are in the 10s of megabytes large. Those files get viewed  cleaned up, and imported into our system by portfolio managers monthly.

The system to do the loading is homegrown, not ETL based (someone thought users should ‘upload’ the files) into an ASP.NET MVC app, and then render UIs with the data. :eyeroll:

Naturally, as our data requirements got larger the files got bigger. And bigger.

This last month, the system eventually exploded. A 1 GB text file proved to be too much for our cute little “not-quite-an-ETL” tool. The portfolio managers needed a solution, and in order to quickly get ’em one, I present (with relevant bits redacted) a 30 minute “lets play with F#” script to solve the problem.

Nothing too fancy. The function simply takes a file path, creates a couple of functions to help it along, and then loads up the file and splits it up into distinct new files. Whole thing took 30 minutes to do. Yes, the complexity is O(n-squared), but when you’ve got panicked users, and all of a half hour to hit it, getting it ‘working’ first is the best way to go.

Yesterday afternoon, I had a bit of time on my hands, and a report that someone desperately needed redone. The original author was a support engineer who was trying to learn Ruby and didn’t realize that Ruby hash-maps are case sensitive. In the interest of time, I corrected the issue by downcasing all the keys, and reran the script (which ran successfully, but inelegantly), but I looked at the problem, and once again, found a perfectly lovely use case for the language I love so much, F#.

First off, some of the basics.

open System

// these may not be strictly necessary, but they do help describe
// precisely what data I'm dealing with
module UpperCaseString = 
    type UpperCaseString = UpperCaseString of string
    let private upper (a:string) =
        a.ToUpper()

    let create (a:string) = 
        if String.IsNullOrEmpty(a) then None
        else Some (UpperCaseString (upper a))
    let value (UpperCaseString s) =
        s
open UpperCaseString

module FloatingDecimalBetweenZeroAndOne = 
    type FloatingDecimalBetweenZeroAndOne = FloatingDecimalBetweenZeroAndOne of decimal

    let create (a:decimal) = 
        if a > Decimal.One || a < Decimal.Zero then              None          else              Some (FloatingDecimalBetweenZeroAndOne a)          let value (FloatingDecimalBetweenZeroAndOne f) =               f     let make a = (create a |> Option.get)    
   
open FloatingDecimalBetweenZeroAndOne 

Generally, I’ll do this for most business application coding nowadays. Creating a simple type that encapsulates correctness from the get-go (including basic validation) makes it so I don’t have to try to figure out weird results. I know the sorts of things I’m dealing with, and I deal with them from the beginning. Also, given the fundamental problem of the original script I had (the case sensitive hash-maps), ensuring I had a type that enforced upper-casing seemed worthwhile.

Next, the domain.

type Model = Model of UpperCaseString
    with static member Value (Model (UpperCaseString s)) = s
type Ticker = Ticker of UpperCaseString
    with static member Value (Ticker (UpperCaseString s)) = s

type Holding = { Ticker : Ticker; Allocation : FloatingDecimalBetweenZeroAndOne }
type ModelAllocations = { Model : Model; Holdings : Holding list }
type ModelWeight = { Model : Model; Weight : FloatingDecimalBetweenZeroAndOne }

type SleevedHolding = { Ticker : Ticker; Model : Model; Allocation : FloatingDecimalBetweenZeroAndOne } 
    with override x.ToString () = sprintf "%s,%s,%5M" (Ticker.Value x.Ticker) (Model.Value x.Model) (FloatingDecimalBetweenZeroAndOne.value x.Allocation) 


// This is the simplest description of what we're doing.
// taking a list of holdings, a list of models and their holdings
// a list of weights to apply to the model, and returning a set of 'sleeved' holdings.
type SleeveProcess = Holding list -> ModelAllocations list -> ModelWeight list -> SleevedHolding list

The problem the original script was attempting to solve was to assign an appropriate ‘Model’ to a holding. A Model is, for sake of brevity, simply an identity (representing the “name” of a standard financial benchmark (like the S&P 500), and a Ticker an identity representing a stock ticker.

A Holding represents an individual stock in a of a portfolio, with the Allocation representing the percentage of the portfolio. ModelAllocations represent a model, and the list of holdings it has (what stocks are in the S&P 500, etc.) ModelWeight represents the approximate weight a portfolio is associated to a given model or benchmark.

E.G. S&P 500 -> 50% , and Russell 3000 -> 50%.

let makeModel str =
    UpperCaseString.create str |> Option.get |> Model  

let makeTicker str =
    UpperCaseString.create str |> Option.get |> Ticker

let holding str all =
    { Ticker = makeTicker str; Allocation = make all }

let makeSleeve t m a = { Ticker = t; Model = m; Allocation = a }
let modelWeight str a = { Model = makeModel str; Weight = make a }
let modelAllocation str h = { Model = makeModel str; Holdings = h }

One of the problems of heavy use of custom types is that you’ll typically want ‘shorthand’ functions for creating the types as necessary. It can be a pain, but difficult to explain results after the fact is worse.

Finally, the meat of it.

let sleeveHoldings holdings models weights =
    let findHolding t = 
        models |> 
             List.collect (fun n -> n.Holdings 
                                        |> List.filter (fun n -> n.Ticker = t) 
                                        |> List.map (fun x -> (n.Model, x.Allocation)))
    let sleeve h = 
        let t = h.Ticker
        let m = findHolding t // a list of models that hold the security
        match m with 
        | [] -> 
            weights |> 
                List.map (fun w -> 
                    let weightTimesAllocation = (value w.Weight * value h.Allocation)
                    makeSleeve t w.Model (make weightTimesAllocation))
        | [(x,_)] -> [makeSleeve t x h.Allocation]
        | xs -> 
            let weightsMap = weights |> List.map (fun n -> (n.Model, value n.Weight)) |> Map.ofList
            let total = xs |> List.sumBy (fun (m, w) -> weightsMap.[m] * value (w))
            xs |> List.map (fun (m, w) -> 
                                let weightTimesAllocationOverTotal = (value w * weightsMap.[m]) / total
                                makeSleeve t m (make (weightTimesAllocationOverTotal * value h.Allocation)))
        

    holdings |> List.collect sleeve

First, we define a function to find the models with a given holding, and return that model and the allocation. Then we define a function to sleeve an individual holding. Finally, we call that function over the list of holdings passed in.

A simple set of tests:

let testHoldings = [
        holding  "A" 0.20m;
        holding  "B" 0.20m;
        holding  "C" 0.20m;
        holding  "D" 0.20m; 
        holding  "E" 0.20m; ]
        
let testWeights =  [
        modelWeight "MODEL1" 0.5m; 
        modelWeight "MODEL2" 0.3m;
        modelWeight "MODEL3" 0.2m;
    ]

let testModelAllocations = [
    modelAllocation "MODEL1" [
            holding  "A" 1.0m;
    ];
    modelAllocation "MODEL2" [
            holding  "A" 0.5m;
            holding  "B" 0.3m;
            holding  "D" 0.2m;
    ];
    modelAllocation "MODEL3" [
            holding  "B" 0.5m;
            holding  "C" 0.3m;
            holding  "D" 0.2m;
    ]
]

sleeveHoldings testHoldings testModelAllocations testWeights |> List.map (fun m -> m.ToString ())

val it : string list =
  ["A,MODEL1,0.1538461538461538461538461538";
   "A,MODEL2,0.0461538461538461538461538462";
   "B,MODEL2,0.0947368421052631578947368421";
   "B,MODEL3,0.1052631578947368421052631579"; "C,MODEL3, 0.20";
   "D,MODEL2,0.120"; "D,MODEL3,0.080"; "E,MODEL1,0.100"; "E,MODEL2,0.060";
   "E,MODEL3,0.040"]

At the financial services firm I work for, this concept of attributing holdings is called ‘sleeving.’

We have a reporting application at {Redacted}, one I’ve spent more than a few hours maintaining. A colleague of mine and I (the junior developer I spoke of in a prior post) met up with the business owner of the reporting application, to talk about rebuilding the application in more functional way.

This application is not a terribly complicated one. In short, it takes data from multiple data sources, and re-aggregates it into a series of reports. It has however, had multiple developers come and “peek in”, drop code into it, and leave.  In the current C# implementation, it’s 5 distinct assemblies. There are 8 test assemblies to go with it.  It has a lot of business value riding on it being robust and correct, so the tests do seem warranted.

A word to the wise though… simple things that are more complicated than they should be AND that have a ton of tests are REFACTORING GOLD. Nothing feels better than taking something complex and unwinding it to it’s core essentials, and nothing feels better than doing it in F#!

We took a page out of For Fun and Profit’s DDD page and focused for nearly an hour on the objects and processes involved in this report. Naturally, the process seemed extremely solvable in a functional way.

Unfortunately, due to time constraints on the rest of the day we were only able to start with some very high level type definitions, but those type definitions described our problem in such a way that the business owner was able to see and understand.

We did the whole thing with an instance of VS Code and Ionide, and had types describing the objects and functions involved, all with just a simple “domain” setup in the process. Did we implement anything particularly? Well, kinda yeah, this is perfectly compile-able F# code, which as opposed to Gherkin or some other spec-based “code”, does not need a secondary interpreter.

That’s the crux of this application and function. I think we may be able to get this down to a few pages of code… as opposed to the novella you’d call it now. #FeelingHopeful

As we last spoke about Fractions, we had a simple task after getting a Fraction object. We wanted to reduce them. I was taught to rit educe a fraction by factorizing the numerator and denominator, and then crossing out the common factors. Then, multiplying the numbers together to get the reduced value. E.G.

 60        2 * 2 * 3 * 5        2 * 2 * 3 * 5       2 
----  =  -----------------  =  ---------------  =  ---
 90        2 * 3 * 3 * 5        2 * 3 * 3 * 5       3

The example above is a simple one.

So the question is, how do we best factor a number?

First, we define the mechanisms we require.  We need a way to define that a number is a factor of another.

let isFactor i pf =
    i % pf = 0I

What does this method buy us? First, it gives us a simple True/False flag defining whether or not a number is a factor of another. Namely, if the module of i and pf is 0I  (specifically, zero, as a System.Numerics.BigInteger), then we know that pf is a factor of i.

Next, we define a recursive function that takes two parameters, start and i. The idea of this function is to return a sequence of numbers from start to i, counting by twos. The start value is an option type, so if not present, it assumes 2 is a valid option, and starts the sequence there, then goes with every odd number. Note: I’m accepting as a given that this will do some checks against numbers it’s unnecessary to do that against.

Finally, I define my internal recursive loop to accumulate the resulting found factors. An interesting thing we’ve got here is some issues of partial application for function calls. The “findFunc” function is a partially applied isFactor call, with the i value already passed in. The reason I did this was to make the Seq.tryFind call easy to use, because the Seq.tryFind call is shaped like this:

('a -> bool) -> seq 'a -> Option 'a

But to get to my “bool”, I needed more than just a single ‘a parameter.  That’s why partial application was so valuable here! Instead of changing the function signature to meet my needs, I simply made a quick, easy to reference function by supplying some arguments ahead of time.

The rest of the function is fairly apparent. The loop recurses over the sequence, finding factors and then appending them to as the head of the accumulator (f::acc), until the Seq.tryFind call eventually returns None, and the accumulated list of factors is returned.

The last bits of the function just wrap up return values. In this case, we check for negative numbers, or a 0I being passed in.  Negative numbers are fairly easy to factor as they only require normal factorization of the inverse with -1I appended, and 0I simply returns an empty list.

With that, we get our results (mapped to ints, for ease of readability.)

getFactors 64000I |> List.map int;;
val it : int list = [5; 5; 5; 2; 2; 2; 2; 2; 2; 2; 2; 2]

getFactors 6423453000I |> List.map int;;
val it : int list = [8599; 83; 5; 5; 5; 3; 3; 2; 2; 2]

getFactors 9536923853063I |> List.map int64;;
val it : int64 list = [354041L; 178393L; 151L]

getFactors 953692385306332I |> List.map int64;;
// this one's still chugging.

The last thing to do here is to figure out how to avoid unnecessary iterations but that’s a topic for another blog.