Postcodes

After a pretty practical previous post about records and collections, this post is less likely to give anyone ideas about how they might tackle a problem in their own project, and doesn’t have any feature requests for Microsoft either. It’s an area I’ve found really fun though.

An introduction to postcodes in the UK

I realise that most of the readers of this blog post will probably not be in the UK. Most countries have postal codes of some description, but the details vary a lot by country. UK postcodes are quite different in scale to US zipcodes, for example. A UK postcode is quite fine-grained – often just a part of a single street – so knowing a house/flat number an a postcode is usually enough to get to a precise address.

Importantly for my election site, every postcode is in a single constituency – and indeed that’s what I want to use them for. My constituencies page allows you to start typing a constituency name or postcode, and it filters the results as you type. I suspect that a significant proportion of the UK population knows their postcode but not the name of their constituency (particularly after the boundary changes in 2023) so it’s helpful to be able to specify either.

Wikipedia has a lot more information about UK postcodes but I’ll summarize it briefly here. Note that I only care about modern postcodes – the Wikipedia has details about the history of how things evolved, but my site doesn’t need to care about legacy postcodes.

A postcode consists of an outcode (or outward code) followed by incode (or inward code). The outcode is an area followed by a district, and the incode is a sector followed by a unit. As an example, my postcode is RG30 4TT: broken down into:

• Outcode: RG30
  • Area: RG
  • District: 30
• Incode: 4TT
  • Sector: 4
  • Unit: TT

Incodes are nice and uniform: they’re always a digit followed by two letters. (And those two letters are within a 20-character alphabet.) Outcodes are more “interesting” as they fall into one of the seven formats below, where ‘9’ represents “any digit” and ‘A’ represents “a letter” (although the alphabet varies):

• AA9
• AA99
• A9
• A99
• A9A
• AA9A

Note how the first two formats for outcodes only vary by whether they have one or two digits – and remember that an incode always starts with a digit. This isn’t a problem when parsing text that is intended to represent a complete postcode (as you can tell the length of the outcode by assuming the final three characters are the incode) – but when I need to parse an incomplete postcode, it can be ambiguous. For example, “RG31 4FG” and “RG3 1AA” are both valid postcodes, and as I don’t want to force users to type the space, “RG31” should display constituencies for both (“Reading West and Mid Berkshire” and “Reading Central” respectively).

Requirements

The requirements for my use of postcodes in the election site is pretty straightforward:

• Ingest data from an external source (the Office of National Statistics was what I found)
• Store everything we need in a compact format, ideally in a single Firestore document (so 1MB)
• Keep the data in memory, still in a reasonably compact format
• Provide an API with input of “a postcode prefix” and return “the set of all constituencies covered by postcodes starting with that prefix” sufficiently quickly for it to feel “instant”

Interestingly, Wikipedia refers to “a table of all 1.7 million postcodes” but the file I ingest currently has 2,712,507 entries. I filter out a few, but I still end up with 2,687,933 distinct postcodes. I don’t know whether the Wikipedia number is just a typo, or whether there’s a genuine discrepancy there.

Spoiler alert: the network latency easily dominates the latency for making a request to the API. Using airport wifi in Munich (which is pretty good for free wifi) I see a roundtrip time of about 60ms, and the page feels like it’s updating pretty much instantaneously as I type. But the time within ASP.NET Core is less than a millisecond (and that’s including filtering by constituency name as well).

Storage and in-memory formats

Logically, the input data is just a sequence of “postcode, constituency code” entries. A constituency code is 9 characters long. That means a very naive representation of 2712507 entries, each taking 16 characters (expecting an average of 7 characters for the postcode, and 9 characters for the constituency code) would be just under 42MB, and that’s before we take into account splitting the data into the entries. Surely we can do better.

There’s a huge amount of redundancy here – we’re bound to be able to do better. For a start, my naive calculation assumed using a whole byte per character, even though every character we need is in the range A-Z, 0-9 (so an alphabet of 36 characters) – and several of the characters within those values are required to just be digits.

Grouping by outcode

But before we start writing complex code to represent the entry string values in fewer bytes, there’s a lot more redundancy in the actual data. It’s not like postcodes are randomly allocated across all constituencies. Most postcodes within the same sector (e.g. “RG30 4”) will be in the same constituency. Even when we group by outcode (e.g. “RG30”) there are relatively few constituencies represented in each group.

At the time of writing, there are 3077 outcodes:

• 851 are all in 1 constituency
• 975 are in 2 constituencies
• 774 are in 3
• 348 are in 4
• 98 are in 5
• 21 are in 6
• 8 are in 7
• 2 are in 8

We also want to diffentiate between invalid and valid postcodes – so we can think of “no constituency at all” as an additional constituency in each of the lines above.

In each outcode, there are 4000 possible incodes, which are always the same (0AA, 0AB .. 9ZY, 9ZZ). So for each outcode, we just need an array of 4000 values.

A simple representation with a byte per value would be 4000 bytes per map, with 3077 such maps (one per outcode) – that’s 12,308,000 bytes.

Compressing the maps

Can we do better by using the fact that we don’t really need a whole byte for each value? For the 851 outcodes in a single constituency, each of those values would be 0 or 1. For the 975 outcodes in two constituencies, each value would be 0, 1 or 2 – and so on. Even without splitting things across bytes, we could store those maps as:

• 851 maps with a single bit per value (500 bytes per map)
• 975 + 774 maps with two bits per value (1000 bytes per map)
• The remaining (348 + 98 + 21 + 8 + 2) maps with four bits per value (2000 bytes per map)

That’s a total of (851*500) + ((975 + 774) * 1000) + ((348 + 98 + 21 + 8 + 2) * 2000) = 3,128,500 bytes. That sounds great – it’s saved three quarters of the space needed for the maps! But it’s still well over the maximum size of a Firestore document. And, of course, for each outcode we also need the outcode text, and the list of constituencies represented in the map. That shouldn’t be much, but it doesn’t help an already-blown size budget.

At this point, I tried multiple options, including grouping by sector instead of outcode (as few outcodes actually have all ten possible sectors, and within a sector there are likely to be fewer constituencies, so we’ll have more sectors which can be represented by one or two bits per value). I tried a hand-rolled run-length encoding and various options. I did manage to get things down quite a lot – but the code became increasingly complicated. Fundamentally, what I was doing was performing compression, and trying to apply some knowledge about the data to achieve a decent level of compression.

Around this point, I decided to stop trying to make the code smarter, and instead try making the code simpler and lean on existing code being smart. The simple 4000-byte maps contain a lot of redundancy. Rather than squeezing that redundancy out manually, I tried general-purpose compression – using DeflateStream and InflateStream. The result was beyond what I’d hoped: the total size of the compressed maps is just 500,602 bytes. That leaves plenty of room for the extra information on a per-outcode basis, while staying well under the 1MB document limit.

I would note that when I wrote about storage options, I mentioned the possibility of using Google Cloud Storage instead of Firestore – at which point there’d really be no need for this compression at all. While it would be nice to remove the compression code, it’s a very isolated piece of complexity – which is only actually about 15 lines of code in the end.

Processing a request

So with the storage requirements satisfied, are we done? Pretty much, it turns out. I keep everything uncompressed in memory, with each per-outcode map storing a byte per incode – leading to just over 12MB in memory, as shown earlier. That’s fine. Each per-outcode map is stored as an OutcodeMapping.

The process of converting a prefix into a set of possible constituencies is fiddly simply due to the ambiguity in the format. I don’t return anything for a single character – it’s reasonable to require users to type in two characters of a postcode before we start doing anything with it.

It’s easy to store a dictionary of “first two characters” to “list of possible OutcodeMapping entries”. That list is always reasonably short, and can be pruned down very quickly to “only the entries that actually match the supplied prefix”. For each matching OutcodeMapping, we then need to find the constituencies that might be represented by the incode part of whatever’s left of the supplied prefix.

For example, if the user has typed “RG30” then we’ll quickly filter down to “RG3” and “RG30” as matching outcodes. From that, we need to say:

• Which constituencies are represented by incodes for RG3 which start with “0”?
• Which constituencies are represented by incodes for RG30? (We’ve already consumed the whole of the supplied prefix for that outcode.)

By splitting it into outcode and then incode, it makes life much simpler because the incode format is so simpler. For example, if the user had typed in “RG30 4” then the questions above become:

• Which constituencies are represented by incodes for RG3 which start with “04”?
  • There can’t be any of those, because the second character of an incode is never a digit.
• Which constituencies are represented by incodes for RG30 which start with “4”?

We always have zero, one, two or three characters to process within an outcode mapping:

• Zero characters: the full set of constituencies for this outcode
• One character: the set of constituencies for that “sector” (which we can cache really easily; there are only ten sectors per outcode)
• Two characters: the fiddliest one – look through all 20 possibilities for that “half unit” in the map.
• Three characters: a single postcode, so just look it up in the map; the result will be either nothing (if the postcode is invalid) or a single-entry result

I suspect that there’s more efficiency that could be squeezed out of my implementation here – but it’s already fast enough that I really don’t care. At some point I might do a load test to see how many requests I can process at a time while still keeping the latency low… but I’d be very surprised if this caused a problem. (And hey, Cloud Run will just spin up some more instances if necessary.)

Conclusion

There are two things I love about this topic. Firstly, it’s so isolated. You don’t need to know anything about the rest of the site, election polls, parties etc to understand this small domain. You need to know about postcodes and constituencies, but that’s it. Aside from anything else, that makes it a pleasant topic to write about. Secondly, the results seem so wildly impressive, with relatively little effort. In around half a megabyte of data, we’re storing information about over 2.7 million postcodes, and the final lookup is dirt cheap.

There’s an interesting not-quite-contradiction in the approaches taken in the post, too.

Firstly, even though the data is sort of mapping a 6/7 character input (postcode) to a 9 character output (constituency code), it’s very far from random. There are only 650 constituencies, and the constituencies are obviously very “clustered” with respect to the postcodes (so two postcodes which only differ in the fifth or later character are very likely to belong to the same constiteuncy). The approach I’ve taken uses what we know about the data as a first pass – so that aspect is very domain-specific. Just compressing a file of “postcode constituency-code” lines only gets us down to just over 7MB.

In contrast, when we’ve got as far as “for each outcode, we have a map of 4000 entries, each to a number between 0 and 8 inclusive” (with awareness that for many outcodes, the numbers are either in the range 0-1 or 0-2 inclusive), trying to be “clever” didn’t work out terribly well. The code became increasingly complicated as I tried to pack as much information as possible into each byte… whereas when I applied domain-agnostic compression to each map, the results were great.

I don’t think it’s particularly rare to need to find that balance between writing code to try to take advantage of known data characteristics, and relying on existing domain-agnostic algorithms. This is just one of the clearest examples of it that I’ve come across.

Records and Collections

This post is to some extent a grab-bag of points of friction I’ve encountered when using records and collections within the election site.

Records recap

This may end up being the most generally useful blog post in this series. Although records have been in C# since version 10, I haven’t used them much myself. (I’ve been looking forward to using them for over a decade, but that’s a different matter.)

Having decided to make all my data models immutable, using records (always sealed records in my case) to implement those models in C# was pretty much a no-brainer. Just specify the properties you want using the same format as primary constructors, and the compiler does a bunch of boilerplate work for you.

As a simple example, consider the following record declaration:

public sealed record Candidate(int Id, string Name, int? MySocietyId, int? ParliamentId);

That generates code roughly equivalent to this:

public sealed class Candidate : IEquatable<Candidate>
{
    public int Id { get; }
    public string Name { get; }
    public int? MySocietyId { get; }
    public int? ParliamentId { get; }

    public Candidate(int id, string name, int? mySocietyId, int? parliamentId)
    {
        Id = id;
        Name = name;
        MySocietyId = mySocietyId;
        ParliamentId = parliamentId;
    }

    public override bool Equals(object? obj) => obj is Candidate other && Equals(other);

    public override int GetHashCode()
    {
        // The real code also uses EqualityContract, skipped here.
        int idHash = EqualityComparer<int>.Default.GetHashCode(Id);
        int hash = idHash * -1521134295;
        int nameHash = EqualityComparer<string>.Default.GetHashCode(Name);
        hash = (hash + nameHash) * -1521134295;
        int mySocietyIdHash = EqualityComparer<int?>.Default.GetHashCode(MySocietyId);
        hash = (hash + mySocietyIdHash) * -1521134295;
        int parliamentIdHash = EqualityComparer<int?>.Default.GetHashCode(ParliamentId);
        hash = (hash + parliamentIdHash) * -1521134295;
        return hash;
    }

    public bool Equals(Candidate? other)
    {
        if (ReferenceEquals(this, other))
        {
            return true;
        }
        if (other is null)
        {
            return false;
        }
        // The real code also uses EqualityContract, skipped here.
        return EqualityComparer<int>.Default.Equals(Id, other.Id) &&
            EqualityComparer<string>.Default.Equals(Name, other.Name) &&
            EqualityComparer<int?>.Default.Equals(MySocietyId, other.MySocietyId) &&
            EqualityComparer<int?>.Default.Equals(ParliamentId, other.ParliamentId);
    }

    public static bool operator==(Candidate? left, Candidate? right) =>
    {
        if (ReferenceEquals(left, right))
        {
            return true;
        }
        if (left is null)
        {
            return false;
        }
        return left.Equals(right);
    }

    public static bool operator!=(Candidate? left, Candidate? right) => !(left == right);

    public override string ToString() =>
        $"Candidate {{ Id = {Id}, Name = {Name}, MySocietyId = {MySocietyId}, ParliamentId = {ParliamentId} }}";

    public void Deconstruct(out int Id, out string Name, out int? MySocietyId, out int? ParliamentId) =>
        (Id, Name, MySocietyId, ParliamentId) = (this.Id, this.Name, this.MySocietyId, this.ParliamentId);
}

(This could be written a little more compactly using primary constructors, but I’ve kept to “old school” C# to avoid confusion.)

Additionally, the compiler allows the with operator to be used with records, to create a new instance based on an existing instance and some updated properties. For example:

var original = new Candidate(10, "Jon", 20, 30);
var updated = original with { Id = 40, Name = "Jonathan" };

That’s all great! Except when it’s not quite…

Record equality

As shown above, the default implementation of equality for records uses EqualityComparer<T>.Default for each of the properties. That’s fine when the default equality comparer for the property type is what you want – but that’s not always the case. In our election data model case, most of the types are fine – but ImmutableList<T> is not, and we use that quite a lot.

ImmutableList<T> doesn’t override Equals and GetHashCode itself – so it has reference equality semantics. What I really want is to use an equality comparer for the element type, and say that two immutable lists are equal if they have the same count, and the elements are equal when considered pairwise. That’s easy enough to implement – along with a suitable GetHashCode method. It could easily be wrapped in a type that implements IEqualityComparer<ImmutableList<T>>code>, although it so happens I haven’t done that yet.

Unfortunately, the way that records work in C#, there’s no way of specifying an equality comparer to be used for a given property. If you implement the Equals and GetHashCode methods directly, those are used instead of the generated versions (and the Equals(object) generated code will still use the version you’ve implemented) but it does mean you’ve got to implement it for all the properties. This in turn means that if you add a new property in the record, you need to remember to modify Equals and GetHashCode (something I’ve forgotten to do at least once) – whereas if you’re happy to use the default generated implementation, adding a property is trivial.

What I’d really like would be some way of indicating to the compiler that it should use a specified type to obtain the equality comparer (which could be assumed to be stateless) for a property. For example, imagine we have these types:

// Imagine this is in the framework...
public interface IEqualityComparerProvider
{
    static abstract IEqualityComparer<T> GetEqualityComparer<T>();
}

// As is this...
[AttributeUsage(AttributeTargets.Property)]
public sealed class EqualityComparerAttribute : Attribute
{
    public Type ProviderType { get; }

    public EqualityComparer(Type providerType)
    {
        ProviderType = providerType;
    }
}

Now I could implement the interface like this:

public sealed class CollectionEqualityProvider : IEqualityComparerProvider
{
    public static IEqualityComparer<T> GetEqualityComparer<T>()
    {
        var type = typeof(T);
        if (!type.IsGenericType)
        {
            throw new InvalidOperationException("Unsupported type");
        }
        var genericTypeDefinition = type.GetGenericTypeDefinition();
        if (genericTypeDefinition == typeof(ImmutableList<>))
        {
            // Instantiate and return an appropriate equality comparer
        }
        if (genericTypeDefinition == typeof(ImmutableDictionary<,>))
        {
            // Instantiate and return an appropriate equality comparer
        }
        // etc...
        throw new InvalidOperationException("Unsupported type");
    }
}

It’s unfortunate that the comments would involve further reflection – but it would certainly be feasible.

We could then declare a record like this:

public sealed record Ballot(
    Constituency Constituency,
    [IEqualityComparerProvider(typeof(CollectionEqualityProvider))] ImmutableList<Candidacy> Candidacies);

… and I’d expect the compiler to generate code such as:

public sealed class Ballot
{
    private static readonly IEqualityComparer<ImmutableList<Candidacy>> candidaciesComparer;

    // Skip code that would be generated as it is today.

    public bool Equals(Candidate? other)
    {
        if (ReferenceEquals(this, other))
        {
            return true;
        }
        if (other is null)
        {
            return false;
        }
        return EqualityComparer<Constituency>.Default.Equals(Constituency, other.Constituency) &&
            candidaciesComparer.Equals(Candidacies, other.Candidacies);
    }

    public override int GetHashCode()
    {
        int constituencyHash = EqualityComparer<Constituency>.Default.GetHashCode(Constituency);
        int hash = constituencyHash * -1521134295;
        int candidaciesHash = candidaciesComparer.GetHashCode(Candidacies);
        hash = (hash + candidaciesHash) * -1521134295;
        return hash;
    }
}

I’m sure there are other ways of doing this. The attribute could instead specify the name of a private static read-only property used to obtain the equality comparer, removing the interface. Or the GetEqualityComparer method could be non-generic with a Type parameter instead (leaving the compiler to generate a cast after calling it). I’ve barely thought about it – but the important thing is that the requirement of having a custom equality comparison for a single property becomes independent of all the other properties. If you already have a record with 9 properties where the default equality comparison is fine, then adding an 10th property which requires more customization is easy – whereas today, you’d need to implement Equals and GetHashCode including all 10 properties.

(The same could be said for string formatting for the properties, but it’s not an area that has bitten me yet.)

The next piece of friction I’ve encountered is also about equality, but in a different direction.

Reference equality

If you remember from my post about data models, within a single ElectionContext, reference equality for models is all we ever need. The site never needs to fetch (say) a constituency result from the 2024 election from one context by specifying a Constituency from a different context. Indeed, if I ever found code that did try to do that, it would probably indicate a bug: everything within any given web request should refer to the same ElectionContext.

Given that, when I’m creating an ImmutableDictionary<Constituency, Result>, I want to provide an IEqualityComparer<Constituency> which only performs reference comparisons. While this seems trivial, I found that it made a pretty significant difference to the time spent constructing view-models when the context is reloaded.

I’d expected it would be easy to find a reference equality comparer within the framework – but if there is one, I’ve missed it.

Update, 2025-03-27T21:04Z, thanks to Michael Damatov

As Michael pointed out in comments, there is one in the framework: System.Collections.Generic.ReferenceEqualityComparer – and I remember finding it when I first discovered I needed one. But I foolishly dismissed it. You see, it’s non-generic:

public sealed class ReferenceEqualityComparer :
    System.Collections.Generic.IEqualityComparer<object>,
    System.Collections.IEqualityComparer

That’s odd and not very useful, I thought at the time. Why would I only want IEqualityComparer<object> rather than a generic one?

Oh Jon. Foolish, foolish Jon.

IEqualityComparer<T> is contravariant in T – so there’s an implicit reference conversion from IEqualityComparer<object> to IEqualityComparer<X> for any class type X.

I have now removed my own generic ReferenceEqualityComparer<T&gt type… although it’s meant I’ve had to either cast or explicitly specify some type arguments where previously the types were inferred via the type of the comparer.

End of update

I’ve now made a habit of using reference equality comparisons everywhere within the data models, which has made it worth adding some extension methods – and these probably don’t make much sense to add to the framework (although they could easily be supplied by a NuGet package):

public static ImmutableDictionary<TKey, TValue> ToImmutableReferenceDictionary<TSource, TKey, TValue>(
    this IEnumerable<TSource> source,
    Func<TSource, TKey> keySelector,
    Func<TSource, TValue> elementSelector) where TKey : class =>
    source.ToImmutableDictionary(keySelector, elementSelector, ReferenceEqualityComparer<TKey>.Instance);

public static ImmutableDictionary<TKey, TSource> ToImmutableReferenceDictionary<TSource, TKey>(
    this IEnumerable<TSource> source,
    Func<TSource, TKey> keySelector) where TKey : class =>
    source.ToImmutableDictionary(keySelector, ReferenceEqualityComparer<TKey>.Instance);

public static ImmutableDictionary<TKey, TValue> ToImmutableReferenceDictionary<TKey, TValue>(
    this IDictionary<TKey, TValue> source) where TKey : class =>
    source.ToImmutableDictionary(ReferenceEqualityComparer<TKey>.Instance);

public static ImmutableDictionary<TKey, TValue> ToImmutableReferenceDictionary<TKey, TValue, TSourceValue>(
    this IDictionary<TKey, TSourceValue> source, Func<KeyValuePair<TKey, TSourceValue>, TValue> elementSelector) where TKey : class =>
    source.ToImmutableDictionary(pair => pair.Key, elementSelector, ReferenceEqualityComparer<TKey>.Instance);

(I could easily add similar methods for building lookups as well, of course.) Feel free to take issue with the names – while they’re only within the election repo, I’m not going to worry too much about them.

Interlude

Why not make reference equality the default?

I could potentially kill two birds with one stone here. If I often want reference equality, and “deep” equality is relatively hard to achieve, why not just provide Equals and GetHashCode methods that make all my records behave with reference equality comparisons?

That’s certainly an option – but I do lean on the deep equality comparison for testing purposes: if I load the same context twice for example, the results should be equal, otherwise there’s something wrong.

Moreover, as record types encourage deep equality, it feels like I’d be subverting their natural behaviour by specifying reference equality comparisons. While I’m not expecting anyone else to ever see this code, I don’t like writing code which would confuse readers who come with expectations based on how most code works.

End of interlude

Speaking of extension methods for commonly-used comparers…

Ordinal string comparisons

String comparisons make me nervous. I’m definitely not an internationalisation expert, but I know enough to know it’s complicated.

I also know enough to be reasonably confident that the default string comparisons are ordinal for Equals and GetHashCode, but culture-sensitive for CompareTo. As I say, I’m reasonably confident in that – but I always find it hard to validate, so given that I almost always want to use ordinal comparisons, I like to be explicit. Previously I’ve specified StringComparer.Ordinal (or StringComparer.OrdinalIgnoreCase just occasionally) but – just as above with the reference equality comparer – that gets irritating if you’re using it a lot.

I’ve therefore create another bunch of extension methods, just to make it clear that I want to use ordinal string comparisons – even if (in the case of equality) that would already be the default.

I won’t bore you with the full methods, but I’ve got:

• OrderByOrdinal
• OrderByOrdinalDescending
• ThenByOrdinal
• ThenByOrdinalDescending
• ToImmutableOrdinalDictionary (4 overloads, like the ones above for ToImmutableReferenceDictionary)
• ToOrdinalDictionary (4 overloads again)
• ToOrdinalLookup (2 overloads)

(I don’t actually use ToOrdinalLookup much, but it feels sensible to implement all of them.)

Would these be useful in the framework? Possibly. I can see why they’re not there – string is “just another type” really… but I bet a high proportion of uses of LINQ end up with strings as keys in some form or another. Possibly I should suggest this for MoreLINQ – although having started the project over 15 years ago, I haven’t contributed to it for over a decade…

Primary constructor and record “call hierarchy” niggle in VS

I use “call hierarchy” in Visual Studio all the time. Put your cursor on a member, then Ctrl-K, Ctrl-T and you can see everything that calls that member, and what calls the caller, etc.

For primary constructor and record parameters, “find references” works (Ctrl-K, Ctrl-R) but “call hierarchy” doesn’t. I’m okay with “call hierarchy” not working for primary constructor parameters, but as the record parameters become properties, I’d expect to see the call hierachy for them just as I can with any other property.

More frustrating though is the inability to see the call hierarchy for “calling the constructor”. Given that the declaration of the class/record sort of acts as the declaration of the constructor as well, I’d have thought that putting your cursor on the class/record declaration (in the name) would work. It’s not that it’s ambiguous – Visual Studio just complains that “Cursor must be on a member name”. You can get at the calls by expanding the source file entry in Solution Explorer, but it’s weird only to have to do that for this one case.

Feature requests (for the C# language, .NET, and Visual Studio)

In summary, I love records, and love the immutable collections – but some friction could be reduced with the introduction of:

• Some way of controlling (on a per-property basis) which equality comparer is used in the generated code
• Equality comparers for immutable collections, with the ability to specify the element comparisons to use
• An IEqualityComparer<T> implementation which performs reference comparisons
• “Call Hierarchy” showing the calls to the constructors for primary constructors and records

Conclusion

Some of the niggles I’ve found with records and collections are at least somewhat specific to my election site, although I strongly suspect that I’m not the only developer with immutable collections in their records, with a desire to use them in equality comparisons.

Overall, records have served me well so far in the site, and I’m definitely pleased that they’re available, even if there are still possible improvements to be made. Similarly, it’s lovely to have immutable collections just naturally available – but some help in performing comparisons with them would be welcome.

Storage

Since my last post about the data models, I’ve simplified things very slightly – basically the improvements that I thought about while writing the post have now been implemented. I won’t go into the details of the changes, as they’re not really important, but that’s just to explain why some examples might look like things are missing.

I have two storage implementations at the moment:

• Firestore, with separate named databases for the test, staging and production environments
• JSON files on disk, for development purposes so that I don’t need to hit Firestore every time I start the site locally. (The “test” Firestore database doesn’t get much use – but it’s nice to be able to easily switch to use it for testing any Firestore storage implementation changes before they hit staging.)

As I mentioned before, the data models are immutable in memory. I have an interface (IElectionStorage) used to abstract the storage aspects (at least most of them) so that hardly any code (and no code in the site itself) needs to know or care which implementation is in use. The interface was originally quite fine-grained, with separate methods for different parts of ElectionContext. It’s now really simple though:

public interface IElectionStorage
{
    Task StoreElectionContext(ElectionContext context, CancellationToken cancellationToken);
    Task<ElectionContext> LoadElectionContext(CancellationToken cancellationToken);
}

This post will dive into how that’s implemented – primarily focusing on the Firestore side as that’s rather more interesting.

Storage classes

When choosing to store data models as JSON or XML, I usually end up following one of three broad patterns:

• Produce and consume the JSON/XML directly (using JObject or XDocument etc) explicitly in code
• Separate “storage” and “usage” – introduce a parallel set of types that roughly mirrors the data models used by the rest of the code; these storage-specific classes are only used as a simpler way of performing serialization and deserialization
• Serialize and deserialize the data model used by the rest of the code directly

I’ve had success with all three of these approaches – my church A/V system uses the last of these, for example. The first (most explicit) approach is one I sometimes use for XML, but Json.NET (and System.Text.Json) makes it easy to serialize and deserialize types to/from JSON that I rarely use it there. (I’m aware that XML serialization options exist, but I’ve never found them nearly as easy to use as JSON serialization.)

The middle option – having a separate set of classes just for serialization/deserialization – feels like a sweet spot though. It does involve duplication – when I add a new property to one of the core data models, I have to add it to my storage classes too. But that happens relatively rarely, and it makes things significantly more flexible, and keeps all storage concerns out of the core data model. It helps me to resist designing the data model around what would be easy to store, too.

Note on JSON library choice
For my election site, I’m using Json.NET (aka Newtonsoft.Json). I have no doubt that System.Text.Json has some benefits in terms of performance, but I’m personally still more comfortable with Json.NET, having used it for years. None of the JSON code is performance-sensitive, so I’ve gone with the familiar.

The fact that the data models are immutable encourages the choice of using separate storage classes, too. While both Json.NET and System.Text.Json support deserializing to records, the deserialization code for Firestore is a little more limited. I could write a custom converter to use constructors (and I have added a few custom converters) but if we separate storage and usage, it’s fine to just make the storage classes mutable.

Initially – for quite a long time, in fact – I had separate storage classes for JSON (decorated with [JsonProperty] where necessary) and for Firestore (decorated with [FirestoreData] and [FirestoreProperty]). That provided me the flexibility to use different storage representations for files and for JSON. It became clear after a while that I didn’t actually need this flexibility though – so now there’s just a single set of types used for both storage implementations. There’s still the opportunity to use different types for one particular piece of data should I wish to – and for a while I had a shared representation for everything apart from the postcode mapping.

Storage representations

File storage

On disk, each collection within the context has its own JSON file – and there’s a single timestamp.json file for the timestamp of the whole context. So the files are ballots-2029.json, by-elections.json, candidates.json, constituencies.json, data-providers.json, electoral-commission-parties.json, notional-results-2019.json, parties.json, party-changes.json, polls.json, postcodes.json, projection-sets.json, results-2024.json, results-2029.json and timestamp.json. It’s currently just under 4MB. The test environment on disk is used for some automated testing as well as local development, so it’s checked into source control. That’s useful as history for how the data has changed.

Firestore storage – initial musings

Firestore is a little more complicated. A Firestore database consists of documents in collections – and a document can contain its own collections too. So a path to a Firestore document (relative to the database root) is something along the lines of collection1/document1/collection2/document2. So, how should we go about storing the election data in Firestore?

There are four important requirements:

• It must be reasonably fast to load the entire context on startup
• It must be fast to load a new context after changes, using whatever in-memory caching we want
• We must be consistent: when a new context is available, loading it should load everything from the new context. (We don’t want to start loading a new context before it’s finished being stored, for example.)
• I’m a cheapskate: it must be cheap to run, even if the site becomes popular

First, let’s consider the “chunkiness” of our documents. Firestore pricing is primarily based (at least in our case) on the number of document reads we perform. One option would be to make each object in each collection its own document – so there’d be one document per constituency, one per party, one per candidate, one per result, etc. That would involve reading thousands of documents when starting the server, which could end up being relatively expensive – and I’d expect it to be slower than reading the same amount of data in a much smaller number of documents.

At the other end of the chunkiness spectrum, we could try storing the whole context in a single document. That wouldn’t work due to the Firestore storage limits: each document (not including documents in nested collections) has a maximum size of about 1MB. (See the docs for details of how the document size is calculated.)

Handling the data in Firestore roughly the same way as on disk works pretty well – putting each collection in its own document. The only document I’d be nervous about here is the collection of projection sets. Each projection set has details of the projections for up to 650 constituencies (more commonly 632, as few projection sets include projections for Northern Ireland). Right now, that’s fine – but if there are a lot of projection sets before the election, we could hit the 1MB document limit. I’m sure there would be ways I could optimize how much data is stored – there’s a lot of redundancy in party IDs, field names etc – but I’d prefer not to get into that if I could avoid it. So instead, I’ve chosen to store each projection set in its own document.

So, how should we store contexts? Initially I just used a static set of document names (typically an all-values document within a Firestore collection named after the collection in the context – so results-2024/all-values, results-2029/all-values etc) but that doesn’t satisfy the consistency requirement… at least not in a trivial way. Firestore supports transactions, so in theory I could commit all the data for a new context at the same time, and then read within a transaction as well, to effectively get a consistent snapshot at a point in time. I’m sure that would work, but it would be at least a little more fiddly to use transactions than to not do so.

Another option would be to have a collection per context – so whenever I made a change at all, I’d create a new collection, with a complete set of documents within that collection to represent the context. We’d still need to work out how to avoid starting to read a context while it’s being stored, but at least we wouldn’t be modifying any documents, so it’s easier to reason about. Again, I’m sure that would work – but it feels really wasteful when we consider one really important point about the data:

Almost all the data changes at a glacial pace.

There are rarely new parties, new constituencies, new by-elections etc. The data for the 2019 and 2024 elections isn’t going to change at this point, modulo schema changes etc. Existing polls and projection sets are almost never modified (although it can happen, if I had a bug in the processing code or some bad metadata). In other words, the vast majority of the data in a “new” context is the same as in the “old” context.

Manifest-based storage

Thinking about this, I hit on the idea of having a “manifest” for a context, and storing that in one document. Each collection in the context is still be a single document, but the name of that document would be based on a hash (SHA-256 in the current implementation) – and the manifest records which documents are in the context. When storing a new context, if a document with the same name that we would write already exists, we can just skip it. (I’m assuming that there won’t be any hash collisions – which feels reasonable.) So for example, the common operation of “add a poll” only needs to write a new document for “all the polls” (which will admittedly be largely the same as the previous “all the polls” document) and then write a new manifest which refers to that new document. For seat projections, there’s a list of documents instead of a single document.

Loading a context from scratch requires loading the manifest and then loading all the documents it refers to. But importantly, loading a new context can be really efficient:

• See whether there’s a new manifest. If there isn’t, we’re done.
• Otherwise, load the new manifest.
• For each document in the new manifest, see whether that’s the same document that was in the old manifest. If it is, the data hasn’t changed so we can use the old data model, adapted to update any cross-references. (More on that below.) If it’s actually a new document, we need to deserialize it as normal.

This leaves “garbage” documents in terms of old manifests, and old documents which are only referenced by old manifests. These don’t actually do any harm – the amount of storage taken is really pretty minimal – but they do make it slightly more difficult to examine the data in the Firestore console. Fortunately, it’s simple to just prune storage periodically: delete any manifests which were created more than 10 minutes ago (just in case any server is still in the process of loading data from recently-created-but-not-latest manifests), and delete any documents which aren’t referenced from the remaining manifests. This is the only time that code accessing storage needs to know which implementation it’s talking to – because the pruning operation only exists for Firestore.

Cross-referencing

As discussed in the previous post, there are effectively two layers of data models in the context: core and non-core. Core models are loaded first, and don’t refer to each other. Non-core models don’t refer to each other, but can refer to core models.

If any of the core collections has changed in the manifest, we need to create a new core context. If none of them has changed, we can keep the whole “old” core context as it is (i.e. reuse the existing ElectionCoreContext object). In that case, any non-core models which haven’t changed in storage can be reused as well – all their references will still be valid.

If the core context has changed – so we’ve got a new ElectionCoreContext – then any non-core models which haven’t changed in storage need to be recreated to refer to the new core context. (For example, if a candidate were to change name, then the displaying the 2024 election results should display that new name, even though the election results themselves haven’t changed.)

This is pretty easy to do in code, and in a generic way, by introducing two interfaces and some extension methods:

public interface ICoreModel<T>
{
    /// <summary>
    /// Returns the equivalent model (e.g. with the same ID) from a different core context.
    /// </summary>
    T InCoreContext(ElectionCoreContext coreContext);
}

public interface INonCoreModel<T>
{
    /// <summary>
    /// Returns a new but equivalent model which references elements from the given core context.
    /// </summary>
    T WithCoreContext(ElectionCoreContext coreContext);
}

public static class ContextModels
{
    [return: NotNullIfNotNull(nameof(list))]
    public static ImmutableList<T> InCoreContext<T>(
        this ImmutableList<T> list, ElectionCoreContext coreContext)
        where T : ICoreModel<T> => [.. list.Select(m => m.InCoreContext(coreContext))];

    [return: NotNullIfNotNull(nameof(list))]
    public static ImmutableList<T> WithCoreContext<T>(
        this ImmutableList<T> list, ElectionCoreContext coreContext)
        where T : INonCoreModel<T> => [.. list.Select(m => m.WithCoreContext(coreContext))];
}

We end up with quite a lot of “somewhat boilerplate” code to implement these interfaces. Result is a simple example of this, implementing WithCoreContext by calling InCoreContext on its direct core context dependencies, and WithCoreContext for each CandidateResult. All standalone data (strings, numbers, dates, timestamps etc) can just be copied directly:

public Result WithCoreContext(ElectionCoreContext coreContext) =>
    new(Constituency.InCoreContext(coreContext), Date, WinningParty.InCoreContext(coreContext),
        CandidateResults?.WithCoreContext(coreContext), SpoiltBallots, RegisteredVoters,
        IngestionTime, DeclarationTime);

There’s an alternative to this, of course. We could recreate the whole ElectionContext from scratch each time we load a new manifest. We could keep a cache of all the documents referenced by the manifest so that we wouldn’t need to actually hit storage again, and just deserialize. That would probably involve fewer lines of code – but it would possibly be more fiddly code to get right. It would also probably be somewhat less efficient in terms of memory and CPU, but I’m frankly not overly bothered about that. Even on election night, it’s unlikely that we’ll create new manifests more than once every 30 seconds.

Conclusion

It’s possible that all of this will still change, but at the moment I’m really happy with how the data is stored. There’s nothing particularly innovative about it, but the characteristics of the Firestore storage are really nice – even if the site ends up being really popular and Cloud Run spins up several instances, I don’t expect to end up paying significant amounts for Firestore across the entire lifetime of the site. To put it another way: the expected cost is so low that any effort in optimizing it further would almost certainly cost more in terms of time than it would achieve in savings. (Admittedly the “cost” of time for a hobby project where I’m enjoying spending the time anyway is hard to quantify.) Of course, I’m keeping an eye on my GCP expenditure over time to make sure that reality matches the theory here.

I believe I could implement the Firestore approach to storage in Google Cloud Storage (blob storage) almost trivially – especially given that serialization to JSON is already implemented for local storage. I could use separate storage buckets for different environments, in the same way that I use separate databases in Firestore. I have no particular reason to do so other than curiosity as to whether it would be as easy as I’m imagining. (I guess it might be interesting to look at any performance differences in terms of I/O as well.) Migrating away from Firestore entirely would mean I only ever had to deal with one serialization format (JSON) which in turn would potentially allow me to reconsider the design decision of having separate storage classes – at least for some types. That’s not currently enough of a motivation to move, but who knows what changes the next four years might bring.

Having looked at how we’re storing models, the next post will be about records and collections.

Data models (and view-models) and how they’re used

I was considering using the term “architecture” somewhere in the title of this post, but it feels too pompous for the scale of site. I could probably justify it, but it would give me the ick every time I used the term. But this post will basically describe how I’m approaching data within the election site, as well as what that data is.

In many systems, there are lots of uses for data, and the system needs to be designed with all of those in mind. For my election site, I only have to worry about two uses:

• The site itself, which is purely read-only
• Tooling to maintain the data, which is read-write

The tooling is only run by me, at home – so I don’t need to worry about concurrency in terms of multiple concurrent writes occurring. I do need to worry about concurrency in terms of making sure that the site always reads a consistent view of the data even if it checks for updates while I’m in the middle of writing – but I’ll cover that in a later post.

This post is only about the internal representation of the data, where I control basically everything. Most of the site data is sourced from external data sources, and I’ll cover how those are handled in a separate post later on as well.

How the site uses data

The code for the site involves the following projects (ignoring test projects for now). This list is in dependency order: so any project in the list may have dependencies on earlier projects, but not on later ones.

• Election2029.Common: utility code
• Election2029.Models: the core data models, which are all immutable (mostly sealed records)
• Election2029.Storage: code purely about storing the data, either on the file system or in Firestore, but with a common interface
• Election2029.ViewModels: immutable wrappers around the models to simplify the view code (no dependency on storage)
• Election2029.Web: the ASP.NET Core code, with Razor pages etc

The Razor pages don’t generally have code-behind, instead injecting a corresponding view-model which is just rendered as HTML in simple Razor syntax. The intention is to keep the view code really straightforward – I’m fine with having a loop here or there, but most of the view should be HTML rather than C#, if you see what I mean. Aside from anything else, putting any “real” logic in the view models makes them easier to test, which is how I’ve got a comprehensive test suite aspirations to have unit tests.

Each page is injected with “the current view-model for the state of the world, for that view”. This view-model is reused until the underlying data changes, which happens relatively rarely. It does change though, which means the view-models can’t be injected as singletons. I’ll revisit the data reloading mechanics in a later post. (I realise I’m saying this a lot. As posts go on, that should happen less, of course.) The result of page rendering is also cached, but only for 10 seconds, as a balance between data freshness and excessive work duplication. In theory I could probably say “cache until we have a new view-model” but that’s likely to be more complex than is worthwhile.

How the tooling uses data

The code for the tooling involves the following projects, as well as Election2029.Common, Election2029.Models, Election2029.Storage as for the site.

• Election2029.Tools.Common: utility code for all tools (as there are some tools beyond the “data manager” I’m discussing here)
• Election2029.Tools.DemocracyClub.Models: models and or Democracy Club data – an external source I import from
• Election2029.Tools.Parliament.Members: models and client code for the Parliament Members API – another external source
• Election2029.Tools.DataManager: a command-line tool for various data management tasks, including adding polls, updating data from external data sources etc

Importantly, this doesn’t use Election2029.ViewModels at all. Until I started writing this post, there was a dependency from the DataManager project to the ViewModels project, which made me slightly nervous… but having speculatively removed it, I found that there were only a few references, which we easily fixed. (Arguably it’s still useful for the DataManager to be able to perform some simple text formatting, e.g. “format the fieldwork dates for this poll” – I might move that into the models project in the future.)

The models

The models themselves are not just record definitions, although that’s what I’ll show here. I figure it’s reasonable for the data models to have additional members for:

• Simple convenience calculations. For example, working out the total number of votes cast for a single constituency result by summing the number of votes cast for each party.
• Lookups based on the rest of the data. For example, a “projection set” contains a list of projected results by constituency; we have a lookup from constituency to “results for that constituency”.

These could go in the view-models, but they’re typically used by multiple view-models, so it makes sense to put them in the models.

The models are grouped together. There’s an ElectionCoreContext which contains separate models which don’t refer to each other, then ElectionContext which contains everything – it has an ElectionCoreContext, and then other models which can refer to models within the core context, but not other non-core models. This means that creating an ElectionContext consists of:

• Creating each element of the core context independently
• Creating the core context from those elements
• Creating each non-core element, with the core context for reference
• Creating the overall context from the core context and the non-core elements

This makes the data model much easier to work with than if elements couldn’t refer to other models – for example, it means that a PartyChange can refer to Candidate and Party models, rather than just containing the ID of the candidate and the old/new party IDs. The core/non-core split means all of this can be done without any circular dependencies.

In all of the declarations below, I’m omitting any extra interfaces that the records implement, as well as any code within the record.

Quick terminology note: the “context” here is just my term for “the latest state of the world”. It’s got nothing to do with Entity Framework, and any resemblance to DDD bounded contexts is at least somewhat coincidental. (Or I suspect you could think of the whole ElectionContext as a single bounded context – the system is small enough that the one boundary goes around everything. I’m not sufficiently knowledgeable about DDD to understand how well the system fits with it.)

Core models

The ElectionCoreContext has the following declaration:

public sealed record ElectionCoreContext(
    ImmutableList<Constituency> Constituencies,
    ImmutableList<Candidate> Candidates,
    ImmutableList<Party> Parties,
    ImmutableList<ElectoralCommissionParty> ElectoralCommissionParties,
    ImmutableList<DataProvider> DataProviders);

The elements within those are:

public sealed record Constituency(string Name, string Code, string? Code2023, string HexRQ, ConstituencyLinks Links);
public sealed record ConstituencyLinks(string? WhoCanIVoteFor2024, string? WhoCanIVoteFor2029, int ParliamentId);

public sealed record Candidate(int Id, string Name, int? MySocietyId, int? ParliamentId);

public sealed record Party(
    string Id, string BriefName, string FullName, ImmutableList<string> ElectoralCommisionIds,
    string? ParliamentAbbr, string CssPrefix, ImmutableList<string> ColorsByStrength);

public sealed record ElectoralCommissionParty(string ElectoralCommissionId, string Name);

public sealed record DataProvider(
    string Id, string Name, bool Enabled,
    string DescriptionHtml, string? Link, string? DataDirectory);

A few notes on these:

• In general I’ve found that modeling records in terms of lists rather than sets or dictionaries makes it easier to keep data consistent… even if the order of a collection doesn’t logically matter (e.g. for constituencies), it’s really handy to have a canonical ordering so that equality tests are simpler, diffs are easy to read etc.
• ConstituencyLinks could be inlined into Constituency (and indeed it is, in storage). It’s inconsitent with how Candidate has the IDs from external data sources inlined. This may well change over time.
• The “core elements don’t refer to each other” aspect is somewhat polluted by a Party having a list of Electoral Commission IDs. In theory I could have increased the number of “tiers” within the model so that a party could have an ImmutableList<ElectoralCommissionParty> instead… but it turns out that I very rarely need to refer to Electoral Commission parties. (The background on this is that there’s a very large number of parties registered with the Electoral Commission. Most of the time we only need major parties, and we don’t want or need to distinguish between, say, “Green Party” and “Scottish Green Party” – or indeed the two Electoral Commission parties which are both called “Conservative and Unionist Party”.)
• The DataDirectory part of DataProvider is only used by the tooling… that feels a little odd, but it’s not particularly unreasonable.
• DataProvider (which is basically for polls and seat projections) has an Enabled flag to allow me to ingest data from a provider without displaying it, while I’m obtaining permission to use it on the site.

Non-core models

So far, so relatively simple. The full ElectionContext is rather bigger:

public sealed record ElectionContext(
    ElectionCoreContext CoreContext,
    PostcodeMapping PostcodeMapping,
    ImmutableList<ByElection> ByElections,
    ImmutableList<Ballot> Ballots2029,
    ResultSet Results2029,
    ResultSet Results2024,
    ImmutableList<NotionalResult> NotionalResults2019,
    ImmutableList<ProjectionSet> ProjectionSets,
    ImmutableList<Poll> Polls,
    ImmutableList<PartyChange> PartyChanges,
    ImmutableList<CurrentRepresentation> CurrentRepresentation)

The elements within those are:

public sealed record PostcodeMapping(ImmutableList<OutcodeMapping> OutcodeMappings);
public sealed record OutcodeMapping(string Outcode, ImmutableList<Constituency> Constituencies, ReadOnlyMemory<byte> Map);

public sealed record ByElection(LocalDate Date, Result Result);
public sealed record Result(
    Constituency Constituency, Party WinningParty, ImmutableList<Candidacy>? CandidateResults,
    int? SpoiltBallots, int? RegisteredVoters, Instant IngestionTime, Instant? DeclarationTime);

public sealed record Candidacy(Candidate Candidate, ElectoralCommissionParty Party, int? Votes);

public sealed record Ballot(Constituency Constituency, ImmutableList<Candidacy> Candidacies);

public sealed record ResultSet(ImmutableList<Result> Results, Instant? LastUpdated);

public sealed record NotionalResult(
    Constituency Constituency,
    ImmutableList<NotionalCandidacy> CandidateResults,
    int AbsoluteMajority,
    decimal Turnout);

public sealed record ProjectionSet(
    string Id, DataProvider Provider, string Name, string Abbreviation, LocalDate FieldworkStart,
    LocalDate FieldworkEnd, LocalDate PublicationDate, string? ArticleLink, string? DataLink,
    ImmutableList<Projection> Projections, ImmutableList<ProjectionDistribution> Distributions);
public sealed record Projection(Constituency Constituency, Party Party,
    ProjectionStrength Strength, string? Link,
    string? Description, ImmutableList<ProjectionMeasure>? VictoryChances, ImmutableList<ProjectionMeasure>? VoteShares);
public enum ProjectionStrength { Tossup, Lean, Likely, Safe };
public sealed record ProjectionMeasure(Party Party, decimal Percentage);

public sealed record Poll(
    string Id, DataProvider Provider, ImmutableList<PollValue> Values, string? PreviousId,
    LocalDate FieldworkStart, LocalDate FieldworkEnd, LocalDate PublicationDate,
    string? ArticleLink, string? DataLink);
public sealed record PollValue(Party Party, decimal Share, decimal? ShareChange);

public sealed record PartyChange(
    Candidate Candidate, Constituency Constituency,
    Party OldParty, Party? NewParty, LocalDate Date, string Description);

public sealed record CurrentRepresentation(Constituency Constituency, Candidate? Member, Party? Party);

More notes:

• ReadOnlyMemory<byte> isn’t necessarily immutable (e.g. it can wrap a byte[]); I might change this to ImmutableList<byte> at some point, although in practice it doesn’t cause any problems. The structure of postcode data is an interesting topic in its own right which – you’ve guessed it – I’ll cover in another post.
• For non-UK readers, a by-election is an election out of the normal cycle, caused by an MP resigning, dying, or being ejected via a petition. (By-elections happen in non-parliamentary-constituency contexts as well, but my site only deals with parliamentary constituencies.)
• Within the system, the term “ballot” is effectively “the list of candidates in a constituency”; in everyday usage it can also mean “a single vote” but I couldn’t think of a better term, and this is what’s used by Democracy Club.
• We could probably manage without the LastUpdated property of ResultSet, but it would still make sense to have as its own type, as it includes a simplified mapping from constituency to result. It’s not (yet) worth doing this for the 2019 notional results as they’re used in far fewer places.
• Speaking of the 2019 notional results… these are “special” because there were significant constituency boundary changes between 2019 and 2024. The notional results are a prediction of what the 2019 actual results would have looked like under the new boundaries. For significantly more detail, consult the “Rallings & Thrasher” document on the topic.
• The Poll.PreviousId property is used to compute the ShareChange. It should probably go away – it’s not used anywhere in the site. (One nice side-effect of writing these posts is the effect of a sort of code review. I may have removed it by the time I actually publish this post.)
• PartyChange isn’t as normalized as it might be. Arguably we should record the history of parties for a given person, and then reflect that in the constituency view by considering the history of “who was the MP at any given time, and which party were they representing at any given time” – this is definitely an area I might revisit in the next few years. (Once we’ve had the first actual by-election, which will be quite soon, it’ll be easier to reason about.)
• CurrentRepresentation is similarly a little odd, in that we should be able to derive the data from the combiation of 2024 election results, by-elections, and party changes. It happens to be convenient at the moment, but may well become less so over time.

The aspects about PartyChange and CurrentRepresentation make me wonder whether I might introduce a ConstituencyHistory model that isn’t stored anywhere, but is derived in the ElectionContext on construction.

Final thoughts

As noted earlier, both the models and the view-models are immutable. In other projects where I’ve attempted to use immutability, I’ve encountered quite a bit of friction – but the combination of this being a fully-read-only web site, and the functionality of C# records, has made it really pretty straightforward. The benefits I’ve always known about (making it easier to reason about what other code might do, and not having to worry about thread safety) are all still as valuable as ever… but in this particular project, the limitations and frustrations haven’t been a problem. C# records have some challenges and annoyances of their own, but nothing really problematic.

One aspect I’ll highlight now and then talk about later when posting specifically about some performance is that within a single ElectionContext, reference equality for models is all we ever need. If you have two ProjectionSet references that refer to different objects (but are part of the same ElectionContext), they’ll definitely be “different” projection sets. There are a very few models where that’s only coincidentally true, such as ProjectionMeasure, but those don’t have any logical identity. All the “primary” models really do have some form of identity, and within an ElectionContext reference equality is equivalent to logical identity.

Having described the data model and the MVVM (at least notionally) approach to the site, there are now plenty of options I can choose from for what to write about next. I think I may go into the storage side of things…

Technical overview

This post is mostly for scene-setting purposes. There’s nothing particularly remarkable here, but it’s useful to get the plain facts out of the way before we get into genuinely interesting design aspects.

Just as a reminder, go to https://election2029.uk if you want to see what any of this looks like at the moment.

The main thing to remember at all times is that the site has some relatively rare characteristics:

• It’s purely read-only: where there’s user input, it’s solely to change what’s displayed.
• There’s a small enough amount of data for everything to be in memory.
• Until election night itself, the data will evolve very slowly. Before the election is announced, it’s unlikely that there’d be more than two or three data changes per day. During the election campaign there may well be multiple new polls and MRPs each day. On election night, I want to serve the latest results as quickly as I reasonably can.

You may notice that most of the tech stack is based on Google Cloud. Yes, I’m biased. No, I didn’t bother looking to see whether it would have been easier or cheaper to build the same thing in Azure or AWS. Aside from anything else, I like being my own customer: I’m using libraries I’ve built for GCP, and if there’s something that I find frustrating, I’ll file a feature request and probably implement a fix. Similarly, I can pass feedback for other aspects (such as Cloud Run and Cloud Build) on to colleagues. I’m sure the site could have been built on any other cloud platform. My choices here aren’t intended to disparage other options – but I have to say, I’m pretty happy with the choices I have gone with.

Web: ASP.NET Core and Razor

The site uses Razor for all the HTML formatting, but doesn’t really use MVC or Razor Pages in a “normal” way. Very few pages have a codebehind. It’s something akin to MVVM, or at least “MVVM as I understand it” (which would probably horrify most practitioners). I’ll go into more detail on this in a later post, including how it all hangs together with dependency injection.

Currently I’m using .NET 9 – I expect to update to new versions of .NET as they’re released, but never use pre-release versions. This means it’s likely that the site will be running .NET 13 on election night.

I’m using a few JavaScript libraries for graphical presentation purposes, but that’s about it in terms of web technologies. There’s no “framework” as such – no Bootstrap, no Angular, no React etc. I’m sure they have their place, but I haven’t felt the need for anything beyond handcrafted CSS so far.

Hosting: Google Cloud Run

My 2024 election site ran on a GKE cluster, which I’ve now turned down in favour of hosting everything on Cloud Run. It took a while for me to really get onboard with the whole serverless notion, but I love it now. It does come with some design challenges, which I’ll come onto in another post, but overall I’m very happy with it. Aspects which would have at least taken some effort a few years ago (a vanity domain, and HTTPS certificates) are now very simple indeed – which makes all the difference for a hobby project. (It’s one thing to spend an entire day configuring a piece of infrastructure for a commercial project; it’s another thing entirely for a hobby.)

Currently I still have a Committed-Use Discount with Google Cloud Platform (which I purchased before seriously considering turning down my GKE cluster), but when that expires in 2027 I expect to be able to run all my sites for well under a dollar a day. It may be worth rewnewing my CUD but for a much lower value. I’ll have to see closer to the time.

Of course, if the site is successful I expect it to be rather more expensive during the election campaign itself – but it would have to be wildly successful to need to scale up to enough instances to make my wallet break a sweat.

Building/deployment: Google Cloud Build

While there are other options I could consider here, Cloud Run integrates very nicely with Cloud Build, which in turn integrates nicely with GitHub. I’ll go into a little more detail here when I write about the different environments, but it’s mostly boring, which is just the way I like it. I write code, commit it, push it, then deploy it when I want to.

Data/storage: Firestore

Firestore is a document database. Now when you hear “database” you probably think about querying – and that’s something I’m really not doing a lot of. It’s fair to say that the way I’m using Firestore could almost certainly have been implemented just as well using Google Cloud Storage (basically blob storage). I may even implement that as an option at some point just to see whether what it’s like in both code complexity and performance.

But for now, Firestore certainly works well enough for me, and the way I’m using it is almost free.

Source control: GitHub

The code is all stored in GitHub. To answer the next obvious question: no, it’s not in a public repo. There are a few reasons for this, some of them bureaucratic, but the most important one is probably that I keep copies of the underlying data for polls (mostly Excel files and PDFs) as published by the polling companies. This is data they make freely available to everyone, but I’m not sure they’d be happy with it being publicly distributed elsewhere.

Of course, there are other approaches I could have taken, such as storing the data files in GCS instead of in source control, etc. I could potentially revisit this decision in the future, particularly if the bureaucratic aspects go away over time. It’s definitely not a matter of the “secret sauce” of the site being so brilliant that I think it’s worth keeping to myself for financial reasons.

Data sources

This is the aspect that’s most likely to go stale, but currently I use the following data sources:

• Democracy Club for a lot of data about candidates, ballots, results etc.
• The Parliament Members API for data about party changes.
• The Office for National Statistics for postcode information.
• The Wikipedia page for polling for the next election to see when there’s a new poll to add
• Lots of polling companies, all of whom have generously given me permission to use their polls and seat projections.

Introduction

It’s been over 8 months since I started my UK Election 2029 site, and high time that I actually wrote an introduction post so that I can get into more detailed topics later.

In 2024, shortly after the UK general election was announced, I created a small site at https://jonskeet.uk/election2024, initially to keep track of Sam Freedman’s election predictions and compare them what actually happened. The scope of the site increased a little, but it was never intended to be particularly polished or for a wide audience. The total time between the first code push and the election was about 4 weeks.

On election day, however, I decided I’d enjoyed the experience so much that I registered the election2029.uk domain that day, created a new GitHub repo and a new GCP project, and pushed an initial holding page. See my earlier post about election night for more details about the 2024 site – I don’t expect to go back over any of the technical details except for the purpose of comparison.

For the 2029 site I have a much broader target audience, and a much longer timescale. I doubt that many “small” projects (in terms of the number of contributors and expected time spent) have deadlines quite as long as this. We don’t have a date for the next UK general election yet, but I’m reasonably hopeful that it will be in 2029 – I’m guessing May 2029, personally. If we expect interest in the next election start warming up in late 2028 or early 2029, that still gives me over 4 years from the first registration date until when I expect the site to actually need to be reasonably appealing.

The date of the election is an interesting deadline, as it’s simultaneously “very generous” (I’ve never worked to such long timescales) and really sharp (if it only comes together the day after the election, it’ll be pointless).

The site already (March 2025) has most of the functionality I’m expecting to include, so I hope most of the data schema is appropriate, even if the way that data is presented changes quite a lot. Even just within the last 8 months I’ve had a lot of fun on the technical side, and I’m hoping to share quite a lot of that via blog posts.

This post doesn’t go into any of the technical aspects, but I think it’s worth just going over what I’m trying to achieve.

Requirements

• Valuable and informative to both election geeks and the general public. (Thinking about what “the average voter” wants to know is likely to be one of the biggest challenges, I suspect.)
• Entirely free for users – no subscriptions, ads, or even cookies.
• Fast for users – I see no reason why any page shouldn’t load in the blink of an eye, and it should really use very small amounts of data. (External JavaScript libraries are likely to be bulk of data transfer, and those should only be needed if you want to see a visualization such as a map, graph or Sankey diagram.)
• Cheap for me – I shouldn’t end up having to worry about how much the site is costing on a daily basis, unless it becomes very popular, unexpectedly… and even in my wildest dreams that’s only likely to have a financial impact during the few weeks of the election campaign itself. (As a note of caution for this, I’m considering paying for advertising at some point, if I think I’ve got a site worth going to, but finding it hard to get traction. Obviously I’d love to get a mention on a popular podcast or similar, and then use word of mouth to get traffic.)
• Fun for me – if this starts being more of a chore than a joy (and assuming I don’t have a significant user base who would feel let down at that point) I can just shut it down. I think this is very unlikely to happen though.
• Factful – this is not intended to be a vehicle for my political leanings, nor a chance to practise political analysis. There’s always a potential risk of bias in terms of choosing which polls to include etc, but realistically I doubt that I’d actually exclude anything mainstream. (I won’t be doing 538-style pollster ratings.)

Future posts

I’m expecting to write posts on the following topics over time – with some posts including multiple topics, I suspect.

• High level architecture and technology choices
• Data models: ElectionContext and ElectionCoreContext
• Third party APIs and data transformations
• Storage choices (Firestore and files)
• Having fun with JavaScript
• Storing over 2 million postcodes in a 1MB document
• Immutability, records and performance
• Multiple environments in a hobby project
• Hacking a “background service” into Cloud Run

Do let me know in comments if there are any additional topics you’d like me to cover.