hyperexponential logo

hx did Advent of Code 2021

Technology, Coding, Advent Of Code5Mins
2022-03-25Written by Tom Steavenson

Winter was coming. The nights were drawing in. The energy crisis meant we couldn’t heat our homes, and Omicron meant we couldn’t leave them. What comfort could we find in these dark times? It’s a good thing Advent of Code was coming around.

In December 2021, hx took on Advent of Code (AoC) in force. There was a thirst for learning languages and taking on a challenge. hx’ers from across the company: actuaries, engineers, managers, our founders and CEO, all got stuck in! We worked with a veritable smorgasbord of languages: ones we knew, one we didn’t, ones that challenged us to think about problems in different ways, and ones that challenged us by not being the sort of language or tool you might expect to attempt some coding challenges in!

hx’s product, Renew, is a platform for building and exercising pricing models in speciality insurance. In Renew, actuaries develop and test pricing models in Python, releasing them for use by underwriters who interact with the models via a web interface (without seeing any of the code), all in the browser!

While Python is on the rise as a language for developing models, there’s other tools and languages at play in the industry. Our in-house team of actuaries were keen to use this opportunity to experiment with the alternatives, sniffing out the competition!

One of our actuaries, Jonathan Bowden, attempted Advent of Code in R, “a language and environment for statistical computing and graphics”. They made a good go of things, completing days 1 through 14, while learning the language for the first time.

They summarised their experience with:

“I tried not doing this with functions to begin with, BIG MISTAKE. Using functions in a functional language is really important, who knew!”

And

“It really isn't a general purpose language.”

Expanding on that they observed that:

“Sometimes the things you need are right there and have more simple, sensible function names than Python (e.g. colSum, rowSum, rollSum etc).”

and:

They also found that there just wasn’t packages available for things that there are in Python. This all speaks to R being more domain specific and less flexible that Python. Put simply there’s a set of things that it makes really easy, but at the expense of making other things a lot harder. For actuaries using Renew, writing pricing models in Python, the availability of the Python ecosystem, and the general-purpose nature of the language, means they’re not limited when pushing the boundaries, doing something unorthodox, doing things that haven’t been done before.

They found R wasn’t as ergonomic as they were used to with Python:

“Some things are split across multiple packages. Where functionality in Python is all in one place (usually in Pandas or Numpy)”

”There's a tiny bit more syntax fluff than Python”

”It was difficult to find the right/best way to load data from a file. A lot of my attempts ended up being quite slow, whereas with Python I’d simply call the ‘read_csv’ function from Pandas”

”Functions can't have multiple outputs, more annoying than you'd think as you have to make chains of multiple functions to cover separate needs for outputs”

”R doesn't support multi-line docstrings”

All in all, it appeared Jonathan had a bit of hard time of it, getting R to do things it didn’t excel at. But they persevered through to day 14 and if nothing else, have reaffirmed how much more they can achieve using Python for modelling pricing scenarios!

Continuing with the theme of “checking out the competition to Python in actuarial spheres”, Josh Boddy, had a crack at AoC in Julia. Julia promises to be a faster, compiled alternative to Python, with syntax friendly to mathematical computation, while still remaining dynamically typed with minimal syntax.

Ultimately Josh only did 3 days in Julia: days 1, 5, and 7, finding that keeping up with the daily challenges, while learning a new language from scratch, was a bit much to fit in their free time. From doing those problems they did make the observations that:

“Julia’s syntax is quite similar to Python’s.”

“Learning to parse data provided in a non-standard format, in a new language takes up a lot of time and faff. Perhaps it’s a good call to convert your input data to a familiar format before entering into your program.”

“I especially liked Julia’s dot syntax, making mutli-step operations on collections of data easy to express in a minimal fashion, and making it easy to chain operations together.”

Switching back over to more familiar ground in Python, Josh was able to finish the whole of Advent of Code! 🎉

Completing this year’s AoC, as well as completing it in previous years, they were able to make the observation that:

“I’d say the first 15 or so days were easier than last years, and the last 10 or so were more difficult than last year.”

This seems to corroborate with the fact that a reasonable proportion of us got to day 15, and no further!

Completing Advent of Code is a impressive feat. Hats off to you Josh. 🎩

Rounding off the “checking out Python’s competition in the actuarial space”, our co-founder and CPO, Michael Johnson, made a brave attempt at tackling Advent of Code within Microsoft Excel, the much used and much loved and loathed tool of actuarial past.

So, how did that go?

“I optimistically tried to do AoC with both Python and Excel in parallel, to show our engineers the raw unbridled power of an actuary armed with a spreadsheet.”

“That idea fell flat pretty quickly - by day 3, the rose tinted glasses had come off and I’d remembered there's a very good reason that the actuarial profession is moving away from Excel to languages like Python. Turns out that building complex processes is much easier in a real programming language - who'd have thought it? Answer: Us. We thought it. And then we built Renew.”

Over in engineering, we lacked the uniting purpose driving our actuarial team, and approached Advent of Code in all sorts of languages, all for our own reasons. Amongst these were attempts in: Go, C, Haskell, Common Lisp, Zig, SQL, and Lua. Our CEO and co-founder, Amrit Santhirasenan, also got involved, completing some of the days in Clojure.

Go, a language for “getting things done”. Something that Daryl Finlay, one of our founding engineers, proved by using it to complete Advent of Code, and win our private leaderboard by being the first to do it!

Haskell: much revered, feared, and then loved by those who can make it round the learning curve. Statically typed and purely functional: solving problems in Haskell forces you to think about things in a different way: step by step, how you might process the data, requiring more abstract expressions of how data within the problem relate.

Fela Maslen, one of our frontend engineers, completed Advent of Code in Haskell, solving some problems first in C as a prototype, before thinking how to re-express their solution in the purely functional realm.

Being one of our Advent of Code finishers, Fela was able to comment on the full experience:

“My experience was that it was easy until day 15, then hard mode was activated. Some days were particularly hard, like 24, 19 and a few others.”

We seem to have general agreement that day 15 represented a step change in difficulty for the daily challenges!

One backend engineer, Tom Steavenson, attempted Advent of Code in Common Lisp, learning the language from scratch in the process. Tom Steavenson is me, so I’ll stop referring to myself in the 3rd person...

I was excited to learn myself a Lisp. I had heard great testimonials from colleagues over the years about how learning a Lisp revolutionises your thinking, totally rewiring your brain. In Lisp, your code and your data taking the same form, a list. Adding in a macro system, (I’ve been told) gives an incredible expressivity, unlocking an unparalleled ability to make abstractions.

Common Lisp isn’t a language implementation, it’s a specification. So first I had to find myself an implementation. After some quick reading on the internet, it seemed Steel Bank Common Lisp (SBCL) was the go-to for beginners. This provides a REPL, essential for exploring syntax, but that isn’t so ergonomic on its own - you can’t use arrow keys to see previous commands! After some google’ing I found SBCL linedit which fixed this issue.

I was also in need of a package manager. Having done Advent of Code in an unfamiliar language before, I knew how important it was to get the basics of a development setup in place on day 1. After some more searching online, quicklisp looked to be an ergonomic and low hassle choice.

Things got off to a great start. I was impressed with the expressivity of looping over a list (perhaps that’s to be expected in a language where everything is a list). The day 1 problem was made trivial by the ability to “slide a window along a list” with loop for (a b c) in things . The first iteration of the loop takes the 1st, 2nd, and 3rd element of things as a, b, and c respectively. The second iteration take the 2nd, 3rd and 4th, and so on. The problem involved iterating over the elements in a list, while considering their neighbours. This construct made the solution trivial.

Carrying on through the problems, I found it was easy to express functional patterns with great support for folds, lambdas and control over when an expression is evaluated. I found the “Common Lisp Object System” (CLOS) a refreshing and novel take on the concepts of classes and objects.

A big pain point for me was the compiler/runtime not necessarily being that helpful with the errors. I’ve got somewhat accustomed to how helpful Rust’s compiler errors are, and while it can be tricky at times in Rust, I found if I missed a bracket somewhere rewriting the line was quicker than trying to debug it. That being said, I didn’t hate all the brackets. Once I got used to them and read some style guides on how to format them, they were strangely readable.

I managed to keep pace with Advent of Code until day 14, then catching COVID then Christmas prevented me from keeping up any further. Reflecting on the experience I don’t know that I quite got the “mental rewiring” that learning a Lisp is reported to provide. But I didn’t go anywhere near building up layers of abstraction with the macro system. There was also aspects of the language that may have been more novel in decades past, but now have found their way into modern languages. And coming from Rust I could see some of these...

Everything is an expression. Every line of code evaluates to something. This provides excellent composability of expressions and is a feature built into Rust.

There is very strong control for what values are in scope. This makes code a lot easier to reason about and harder to break as you control what data a line of code has access to. Scopes are a strong feature in Rust too. I have to admit I almost prefer how let bindings in Common Lisp create a scope beneath them that the bindings apply in, over how in Rust the binding applies until the end of the current scope. It somewhat focuses the mind down on only what is beneath the current line.

Some time after stopping my efforts at Advent of Code in Common Lisp, I found myself reading the Learn X in Y minutes page on Lua. “What a delightful little language” I thought: It’s a super minimal language with some ergonomic basics for looping and control flow; there’s only one collection type; oh, and functions are first class objects! 😍

I was intrigued to give it a go and see if the very few language constructs results in a practical simplicity or whether I’d get frustrated with not having constructs geared towards specific purposes.

Given I had some solutions written up in Common Lisp, I attempted some ports of those solutions to get used to the syntax. Not having to work out a solution to a problem while learning new syntax was a huge boon to the initial experience.

It’s my intention to perform a few more ports until I’m comfortable with the syntax and basic constructs and then continue in Lua with the problems I didn’t do in Common Lisp. I hope to find time later this year to come back to this.

Those of us who took on Advent of Code in a language totally new to us found the experience of learning new syntax, getting setup and used to the language tooling, and learning about novel language paradigms and constructs, quite a challenge to do while keeping up with the daily duties. While the first few days aren’t so hard, the difficulty can ramp up quicker than we’re able to get a good handle on the language. With the limited time to complete the challenge you might have in a day, a conflict can emerge between taking the time to understand how things work, and getting the answer to your day’s problem.

Another observation that was made was that parsing the problem input could take up a lot of the time we wanted to spend on tackling the problem. The problem inputs are not usually in a commonly know format so parsing it would require some custom parsing code, rather than say: call a library to read the input from CSV.

One way in which tackling Advent of Code problems diverges from developing real applications is that you don’t have to deal with invalid inputs. A huge part in developing software in any language is learning how to deal with error cases. Learning a language with Advent of Code, doesn’t necessarily give you exposure to this.

Discussing some of these problems we came up with some ways to help make Advent of code a better experience for learning a new language...

Take it at your own pace. While there is a daily puzzle, and getting to the answer first will maximise your score on any private leaderboard (or even the global one), there’s nothing stopping you using them as a backlog of exercises of increasing difficulty as you work through a book, or some other learning resource on your chosen language.

Change the input format. To avoid custom parsing, you can convert your problem input to a well known format. This might require a quick crash course in sed or finding a useful feature for block editing text in your editor or IDE, but can pay dividends over the course of a few problems.

Do the first few days in a familiar language. In a language you know well, the first 10 or so days of Advent of Code should go by without too much hassle. There’ll still be some interesting problems to solve, but they’ll remain small enough to be achievable while keeping pace with the problems. With a few solutions down in a language you know well, you can port your solutions to a new language. You can initially perform almost line by line ports of your solution, focusing solely on getting up to speed with the syntax. This can help you map similar concepts between the languages. Once your comfortable with the syntax you can start thinking more in terms of re-writes of your solution, and how it might be better to solve the problems in the new language. If this all goes well you can have a strong basis to continue onto problems you haven’t tackled yet in the new language.

Tim Whelan, our VP of Engineering, started their Advent of Code in Common Lisp, collaborating with Tom Steavenson, and later moved over to attempting some of the problems in Zig.

Zig is an attempt at a more modern update on C, applying some of the lessons learned in the last 50 years, while keeping the simplicity of no hidden control flow or memory allocations, and introducing a system for compile time code execution, “comptime”.

With a strong background in C, there was a lot Tim enjoyed about Zig:

“I discovered a language I felt pretty much at home in (unsurprising as it's pretty close to C in a lot of ways).”

”The comptime facility - analogous in some ways to Common Lisp's macro system (and that of Rust) is particularly powerful!”

”Having gone through the process of learning Rust and Zig relatively closely together - at the risk of incurring the wrath of any Rustaceans (at hx there are a few) - I felt more at home with Zig.”

As a learner they found some difficulty with the flexibility of the language specification, citing “overloaded use of keywords” as a source of confusion.

Amrit Santhirasenan, our co-founder and CEO, doesn’t always get the chance to dive into some coding in his day job. He relished the opportunity to exercise those coding muscles so fine tuned during their Computer Science degree and took up Advent of Code in Clojure.

Clojure is a functional, dynamically typed Lisp dialect built on top of the Java Virtual Machine (JVM). It’s inventor, Rich Hickey, is a thoughtful and captivating speaking. It well worth finding some of their talks on YouTube!

Speaking on their experience, Amrit had to say:

“I loved learning about the power of Clojure's sequence abstractions”

”The Threading Macro is awesome!”

”Lisp is awesome! Although the pain of realising you built the wrong abstraction for Part 2 in Part 1 felt harder to bulldoze through in a functional language like Clojure (exacerbated when you are old and crusty like me)”

This last point speaks to any language that allows you to make robust expressions of abstractions. You’re made to think about your abstractions, as you can’t so easily hack around them. While a hindrance in getting the answer to an Advent of Code problem, it’s exactly what we want when building applications in the wild!

To round off the multifaceted menagerie of languages, in a spirit of seeking challenge and adventure, brave Adomas Boruta, one of our founding engineers, took on Advent of Code in SQL! Completing problems up to day 14!

They chose SQLite as a SQL server, loading the daily problem input into the SQL database and then trying to solve the day’s problem in a SQL query.

“I chose SQLite for two reasons: it would challenge me to use something that is closest to standard SQL; and setup for running it is much simpler since it doesn’t have client-server architecture.”

”The only thing outside SQL was preparing the database - for that I wrote a very simple python script that put each days input into separate table with single column line TEXT in it. After that all parsing and solving was done in SQL.”

Commenting on their use of SQL to solve the problems:

“As much as possible, I tried to solve problems with only SELECT statements.”

”Throughout the challenges, I explored multiple possibilities: creating temporary tables or views; and using Common Table Expression (CTE). Overall, I think almost all of the problems could be solved with single query and made it readable if you used CTEs.”

Diving deeper into their use of SQL:

“The next big thing is using and abusing recursive queries. They were very powerful whenever there was a need for a loop or variable length input parsing. A really good example was day 10. The problem involved parsing brackets and iterating through all the input lines as long as they have brackets in them.”

Speaking to some of the difficulties they encountered:

“One key limitation of SQLite was lack of data structures. As a result, a lot of solutions involved custom encoding data into a string then passing it to the next row to recursive query and decoding it to do the next bit of computation. It would have been much easier in more powerful SQL dialects like PostgreSQL which can use native arrays or even JSON.”

Reflecting on the experience they noted:

“By no means do I think SQL is appropriate for these kinds of challenges, but it was a good learning experience. It was interesting to see how much you can use a system in completely unintended way.”

”I would probably suggest using more powerful SQL dialect [than SQLite] since it would make life much much easier.”

So that was our Advent of Code.

There was: frustration, despair, dogged perseverance, salvation, elation, and joy. A formative experience for us all!

Some hx’ers: Daryl Finlay, Fela Maslen, and Josh Boddy, managed to complete all of this year’s challenges, which is no mean feat!

We very much look forward to taking on the challenge next year!

Interested in finding out more about hx?

hyperexponential logo