Style | StandardCards

Sheffgeeks Blogs

Sunday, 28. June 2020

benofbrown

Fresh new style

After many years (almost 7!) I’ve finally ditched the default jekyll style and created something a bit different.

After many years (almost 7!) I’ve finally ditched the default jekyll style and created something a bit different.

Frontend development isn’t my strong suit, so I’ve kept it pretty simple. The colour scheme is one I’ve been using on my terminal and on my desktop for a long time, Ethan Schoonover’s Solarized Dark.

There’s no JavaScript or images, not a lot of CSS, uses built in fonts, and it seems to work pretty well on my phone and my laptop so I’m happy with it.

Saturday, 27. June 2020

benofbrown

Real config on staging, featuring nginx and dante

I’m a firm believer in keeping staging as close to production as possible. It just makes sense to me, there’s too much room for error if things are different, and you’re not really testing what you think you are.

I’m a firm believer in keeping staging as close to production as possible. It just makes sense to me, there’s too much room for error if things are different, and you’re not really testing what you think you are.

Now this can get a bit tricky when it comes to configuration files, especially when they contain hostnames, doubly so if you’re using HTTPS, which of course you are aren’t you?

Platform Overview

OK, “platform” might be stretching it for my current setup. I look after the technical side of a pretty niche website, Ninehertz. It’s been around for a long time now, and still gets a decent amount of traffic but nothing too taxing.

For the staging side I use a virtual machine. Like a lot of people these days I manage it with Vagrant using the libvirt backend.

The site itself is served by nginx.

TLS Complications

If we’re going to use the exact same config on live and staging, then we’re going to need to use TLS (fka SSL), and if we’re going to use TLS, we need certificates for it. Now, we don’t really want genuine private keys on staging, access to those should be restricted, so we’re going to make some dummy certificates ourselves.

We could simply use self-signed certificates, but then we’d get annoying warnings in the browser. So what I do is create my own CA certificate.

This is really easy to do, and completely safe as long as you follow some precautions:

  • Use a different browser profile when you import the CA - I really can’t stress this enough.
    • Firefox has a profile manager you can use to create a separate profile, create one with a meaningful name and import the CA to that.
    • Chrome has the --user-data-dir argument which does a similar thing. A shell alias makes this easy to use.
  • Only add this CA to the system approved list on your VM - don’t do this on your host machine.

First we create a private key for the CA:

openssl genrsa -out CA.key 4096

Next the CA itself:

openssl req -x509 -days 365 -key CA.key -subj '/CN=dummy CA cert' -out CA.crt

Adjust the subject and days to taste, though I recommend making it obvious that this isn’t a real CA in the subject. 365 days is a year, so it will expire after that time, though it’s just a case of re-running the command to renew it.

Now we need to create a key and a Certificate Signing Request for the server to use. For this example we’ll use www.example.com and will create a cert that’s also valid for example.com:

openssl genrsa -out server.key 2048
openssl req -key server.key -new -subj '/CN=www.example.com' -out server.csr

Keep hold of these two files, server.key and server.csr. The key will be needed to use the certificate we create and the CSR can be used to renew the cert later.

Now to sign the certificate request to generate a certificate:

printf 'basicConstraints=CA:FALSE
keyUsage=nonRepudiation, digitalSignature, keyEncipherment
subjectAltName=DNS:www.example.com,DNS:example.com' | \
openssl x509 -req -days 365 -in server.csr -out server.crt -CA CA.crt -CAkey CA.key \
    -set_serial 1 \
    -extfile -

Some things to note:

  • set_serial 1 - this sets the serial number. When the certificate expires and you create a new one, make sure you set the new serial to something higher than this value.
  • subjectAltName etc, this is what allows the cert to be used for more than one hostname. It’s also required for the site to display in Chrome without warnings.

Now you have a brand new server.crt and server.key you can use in the nginx config in place of the real certs, just use the same paths for them so the config remains the same.

Approach 1: Hostfiles

The simplest way to approach this problem is to simply add entries to /etc/hosts or the equivalent on your host OS, to point records to your VM.

Downsides to this are:

  • You can’t look at live and staging at the same time to compare things
  • You can easily forget you’ve done that and confuse yourself later down the line
  • Your test site can access third party resources, some of which you only want live to be able to access

For these reasons I decided I needed something better - a proxy.

Approach 2: SOCKS proxy via SSH

You may or may not know this already, but ssh includes a pretty usable SOCKS proxy, you just invoke it with -D and a port number, then use that port as your proxy. Combined with adding hosts records redirecting to the loopback device this worked great.

But hang on! You said you’d moved away from using the hosts file!?

Indeed I had, but my main problem was editing /etc/hosts on the host OS, and this change is in the VM. As clients can use the SOCKS proxy to resolve hostnames too this is a nice solution, you know when you’re using the proxy that you’re hitting staging.

I used this approach for a little while, but I wasn’t completely happy with it for the following reasons:

  • You have to run the ssh command every time, which gets annoying
  • It still doesn’t control access to third party resources

With these concerns in mind I decided I needed a SOCKS proxy that would start automatically when the VM started, and had some access control. After some searching I found…

Approach 3: dante

Dante is a free SOCKS server, developed by Inferno Nettverk A/S. It’s included in Debian, my OS of choice, so was simple to install. It also supports allowing and denying access to resources, so I quickly set up some rules. I knew it would resolve the hostnames I was interested in to the local loopback address, so I just added a rule to allow traffic to that interface and deny all other traffic.

This worked really well, I didn’t need to ssh in any more, but something looked a bit strange. The fonts weren’t right.

I knew straight away why this was, Ninehertz uses Google fonts, which were now being blocked. For a while I lived with that, but I got curious and started thinking of a better way.

Approach 4: nginx proxy_pass for specific hostnames

My next approach was to add specific hostnames to the VM’s hosts file and direct them to the local nginx instance, with a small config snippet using proxy_pass to redirect the traffic. I’d also create certs for these hostnames, signed with my CA. With a bit of Ansible this was pretty easy to do so I got it up and running fairly quickly.

This worked great for Google Fonts, but I hit another problem when I loaded a page with an embedded SoundCloud embed, which was trying to fetch things from loads of subdomains. This wasn’t going to scale nicely at all - I really needed a way I could use wildcards.

Approach 5: nginx stream proxy_pass, with dante redirecting

I had a bit of a think and came up with what I thought was a cunning plan. I’m a bit of an nginx enthusiast and like to stay up to date with new features etc, even if I don’t really use them. One such feature popped in to my head, ssl_preread.

Combined with the nginx map directive, this allows us to read the target hostname from the TLS handshake (amongst other things) and route the traffic to different ports depending on the values it finds. We can even use wildcards here, which makes it even easier.

This uses the nginx stream module which operates at a low level, so we don’t even need to think about TLS handshakes, we can now forward the TCP traffic unencrypted to the hosts we want to allow.

For other hosts we’ll just forward it to a dummy HTTPS server, also in nginx, with an invalid certificate.

All I need to do now, is capture all requests to HTTPS on dante to this nginx stream server. Now we can’t use port 443 here without changing our own http server config, which would defeat the point of this, so we run the stream server on a different port.

Looking at the dante documentation it looked really easy to redirect this traffic to our arbitrary port, but when I put the config in place, the danted service failed to start. I’d missed a key part of the documentation, from Redirection:

Some additional functionality is however offered as modules that can be purchased separately. The redirect module can be used to modify traffic by redirecting it to a different location than the client requested.

OK, so dante can do it, but not without a proprietary module. This was a little disappointing, but if that’s their model then that’s fair enough.

Finished setup: nginx stream proxy_pass, dante, and iptables

Finally I arrived at the setup I’m using today. The final piece in the puzzle was to simply use iptables to redirect all http and https traffic to the local nginx instance, with the exception of traffic from nginx itself as that would cause a loop.

Here’s the nginx configuration for the stream module:

map $ssl_preread_server_name $backend {
  hostnames;
  .ninehertz.co.uk 127.0.0.1:443;

  fonts.googleapis.com $ssl_preread_server_name:443;
  fonts.gstatic.com $ssl_preread_server_name:443;
  .soundcloud.com $ssl_preread_server_name:443;
  .sndcdn.com $ssl_preread_server_name:443;
  default 127.0.0.1:8001;
}

resolver 127.0.0.1;

server {
  listen 8000;
  proxy_pass $backend;
  ssl_preread on;
}

server {
  listen 8002;
  proxy_pass 127.0.0.1:80;
}

I should probably mention that I use PowerDNS on this box and on live as a local resolver, that’s what resolver 127.0.0.1 is referring to. We need to set a resolver up here so nginx can look up the hostnames it needs to proxy the external traffic.

Most of the work here is done in the map section. $ssl_preread_server_name is the server name passed in the TLS negotiation. If it matches the hostnames listed and it’s one of the sites that are being staged on the VM then the variable $backend is set to the loopback device. If it’s a site we don’t host, but want to allow access to $backend is set to the original hostname.

We are using the hostnames option in the map (which needs to be listed before the list of values we’re checking) so we can use the special leading . form to match a domain and any subdomains at the same time. e.g. .ninehertz.co.uk matches ninehertz.co.uk, www.ninehertz.co.uk, foo.bar.ninehertz.co.uk etc.

Finally default is a special record which is used to set a default value, which will be for anything we want requests to to fail. These will be redirected to another nginx server block in the http scope, which is running on port 8001.

The first server block simply listens for connections on port 8000, and passes them to the $backend set by the map. We need to have ssl_preread on here or the special $ssl_preread_server_name variable won’t get set.

The next server block is for handling any requests on port 80, it just passes them all to the loopback address.

Here’s the config of the server that non-matching results are forwarded to:

server {
  listen 8001 default_server ssl;

  ssl_certificate ssl/default.crt;
  ssl_certificate_key ssl/default.key;

  return 403 "No\n";
}

This is very simple, we just have a cert signed by our CA and it’ll return 403 for all requests.

Dante wise it’s pretty simple, we pass anything going to to ports 80 or 443, and reject everything else.

Finally, here’s the iptables rules to make it all work:

iptables -t nat -A OUTPUT -m owner --uid-owner nginx -j ACCEPT
iptables -t nat -A OUTPUT -d 127.0.0.1/32 -j ACCEPT
iptables -t nat -A OUTPUT -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8002
iptables -t nat -A OUTPUT -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 8000

It simply accepts all requests from the nginx user, all requests to the local loopback, and redirects any remaining requests to the default https port of 443 to port 8000 on the loopback device.

And a nice feature of this approach is that we no longer need to touch /etc/hosts on either our host machine or the VM.

I hope you might find this useful, if you have any feedback drop me a message on any of the platforms listed at the bottom of each page.

Thursday, 23. April 2020

mbooth

Using the remote OSGi console with Equinox

You may be familiar with the OSGi shell you get when you pass the "-console" option to Equinox on the command line. Did you know you can also use this console over Telnet sessions or SSH sessions? This article shows you the bare minimum needed to do so.

You may be familiar with the OSGi shell you get when you pass the "-console" option to Equinox on the command line. Did you know you can also use this console over Telnet sessions or SSH sessions? This article shows you the bare minimum needed to do so.

Tuesday, 03. March 2020

caolan

Time Disorder

When presented with a series of events, many developers will first be tempted to sort the events by timestamp. This is dangerous because timestamps do not provide the strict ordering they might assume.

Out of order events can lead to infrequent but significant bugs: consider "add to basket then checkout" vs "checkout then add to basket".

Instead of timestamps, d

When presented with a series of events, many developers will first be tempted to sort the events by timestamp. This is dangerous because timestamps do not provide the strict ordering they might assume.

Out of order events can lead to infrequent but significant bugs: consider "add to basket then checkout" vs "checkout then add to basket".

Instead of timestamps, developers should prefer simple counters and proper conflict detection. Timestamps may still be useful, but should be approached with caution due to the complexities outlined below.

Time resolution is not infinite

What happens when two events have the same timestamp?

When writing to a log file, two entries with the same timestamp are not a problem because the lines in the log file provide the real order of the data. However, import those entries into a database and the original line order is lost. Now, when sorted by timestamp, two entries with the same value may be returned in an undefined order.

The chances of a duplicate timestamp are affected by:

  1. The resolution of the hardware clock and time APIs - running your code on another platform may significantly increase your chance of duplicates.
  2. The resolution of your timestamps - do you store seconds since epoch (resolution 1 second)? nanoseconds? a date string with HH:MM (resolution 1 minute)?
  3. The frequency of events

Events in close proximity may record identical timestamps causing them to appear shuffled. To make matters worse, this is most likely when a machine is under heavy load.

Clocks can go backwards

If, like me, you experience time in one direction, this is easy to forget. A clock is merely a device to measure time and as such requires calibration and adjustment.

Manual adjustments, like when a user naively changes timezone or corrects a slow clock, are the most likely cause of a jump backwards in time, but automatic changes can also be to blame.

If a developer generates timestamps or stores timezone data incorrectly, the automatic change from daylight saving time could jump events backwards by a whole hour. We have to be particularly careful in the UK, where GMT can happily masquerade as UTC for half the year.

Services like ntpd (Network Time Protocol Daemon) can also cause dramatic clock changes. Depending on configuration, a large drift in system time can cause ntpd to hop immediately to the correct time (possibly backwards). Devices like the Raspberry Pi are particularly vulnerable to this as they are frequently disconnected from a network and have no Real Time Clock.

There are clocks guaranteed to never run backwards, called 'monotonic' clocks, but a timestamp from a monotonic clock is often of little use between reboots, and useless to compare between machines. Generally, a monotonic clock is used to measure a time interval on a single machine.

Intervals can stretch and shrink

Jumps in time can cause problems, so services like ntpd often prefer to slow down or speed up the system clock until it gradually approaches the correct time (this is called 'slew' correction).

Google uses a similar approach for leap seconds, 'smearing' an extra second over a 24 hour period, instead of bamboozling software with a 61 second minute.

Even if you could start a timer on multiple machines at a known instant in time and stop them at another instant, they would likely measure a subtly different elapsed time. The longer the interval, the more apparent manufacturing tolerances will be. As an example, Adafruit advises this PCF8523 based RTC "may lose or gain a second or two per day".

Clocks are never in sync

A developer may be attracted to timestamps because they're easy to collect at multiple sites then insert into an ordered collection later. However, in addition to all of the above, they must now consider the disparity between multiple system clocks.

Replying to a chat message on one machine you might easily record a timestamp before the original if the original was recorded at a different machine.

Recommendations

Timestamps are complex. They're difficult to store and generate correctly, they're almost impossible to compare accurately across machines, and they cannot guarantee a strict causal ordering of events.

When you sort data by timestamp it almost always implies a causal relationship (e.g. implying a message happened before it's reply, or a form GET happened before a POST). Because of this, techniques that provide a strict (or at least causal) ordering of events should be preferred.

Use a counter

The most fool-proof alternative to timestamps is an incremental counter stored on a single machine. If there is only one instance of the software, or clients always submit to a central server, this is often the best choice.

Most databases provide an auto increment or sequence type that can provide a suitable value.

Consider distributed clocks

If you need to generate points in a sequence at multiple sites, then you may need a more complex series of counters like Lamport timestamps or a vector clock. Distributed clocks like this provide a partial causal ordering of events and a means to detect conflicts (i.e. events that are seen as concurrent because they extend a shared point in history).

If your clients generate timestamps locally but the data is only integrated by a central server (not shared peer-to-peer), your logical clock can be relatively simple requiring only two peers.

Handle conflicts

Distributed clocks will only help you detect concurrent events. Once detected, the problem of resolving conflicting events is often domain-specific. Using the appropriate clock or data structure should force you to handle these conflicts early on. Remember, the conflicts were always present with regular timestamps, they were just not being surfaced in your design.

Conflict detection and resolution can get as fancy and as complicated as you like, including employing tools like git to provide a full history. That said, it's so hard to imagine an architecture that started with simple timestamps and ended with git, that I'm going to suggest you try a distributed clock or simple counter first.

When are timestamps appropriate?

I'm only suggesting timestamps are a bad way to order causally linked events. Timestamps are still useful for:

  • Communication with humans - Logical clocks don't mean a lot to us. Adding a timestamp as part of the presentation (but not ordering) of data is often a good idea as it lets us place entries in a wider context outside of a single application.
  • Sampling - Data collected for statistical analysis is often collected ad-hoc from multiple sources and strictly ordering measurements in close proximity may not be important. Ask yourself: "If I shuffled a few events around would my conclusions still be sound?"
A frequent bugbear

I'm recording my arguments against ordering by timestamp here as a reference because it's a conversation I frequently have in architecture meetings. I hope this is a useful reference for you too, and if you have any relevant experience please do share it with me.

Friday, 14. February 2020

kitation

I'm not sick but I'm not well: Self-Injury Awareness Day 2020

Content warnings: Depression, anxiety, disassociation, self-harm (specifically cutting) and suicidal ideation

Five years ago, I wrote something for Self Injury Awareness Day. Since then, things have gotten both worse and better at the same time. I wanted to write down my “journey” since then, but also some thoughts about mental health vs mental illness in general, and why I find some man

Content warnings: Depression, anxiety, disassociation, self-harm (specifically cutting) and suicidal ideation

Five years ago, I wrote something for Self Injury Awareness Day. Since then, things have gotten both worse and better at the same time. I wanted to write down my “journey” since then, but also some thoughts about mental health vs mental illness in general, and why I find some many things in this space so frustrating.

Thursday, 06. February 2020

mbooth

Eclipse and Handling Content Types on Linux

Getting deep desktop integration on Linux.

Getting deep desktop integration on Linux.

Monday, 20. January 2020

mbooth

Eclipse 2019-12 Now Available From Flathub

How to get the latest release of Eclipse from Flathub.

How to get the latest release of Eclipse from Flathub.

Wednesday, 01. January 2020

kitation

2020: Let's start out optimistic and see how long it takes

I did start out writing a retrospective of 2019, but it was depressing and making me feel depressed every time I sat down to work on it, so I’ve ditched that and decided to look ahead for once. So here are my goals for the next year, and we’ll see how long it takes for the world to destroy them.

I did start out writing a retrospective of 2019, but it was depressing and making me feel depressed every time I sat down to work on it, so I’ve ditched that and decided to look ahead for once. So here are my goals for the next year, and we’ll see how long it takes for the world to destroy them.

Thursday, 21. November 2019

kitation

Accessibility for anxiety

Content warning: Mentions of panic, anxiety, depression and self-harm

I have anxiety. I’ve had it for since my teens. I have depression too, although it can often be hard to distinguish one from the other. It’s mostly controlled by medication, and a boatload of therapy. There are times though when it “flares up”.

I went to a conference this week, I had a “flare up”. It wasn’t pre

Content warning: Mentions of panic, anxiety, depression and self-harm

I have anxiety. I’ve had it for since my teens. I have depression too, although it can often be hard to distinguish one from the other. It’s mostly controlled by medication, and a boatload of therapy. There are times though when it “flares up”.

I went to a conference this week, I had a “flare up”. It wasn’t pretty. I want to talk about my anxiety, how it presents, and what events should do to help people who have similar issues and triggers to me.

Wednesday, 11. September 2019

tomwardill

Making a Chefs Knife

“I’ve been thinking about running a knife making workshop”. Those were the words in the email that got my attention. Someone was offering to run a workshop and had a PID controlled heated knife oven to do the heat cycle and tempering. Seemed like a good idea to me. This write up is mostly a ‘this is what I did’, the end product is usable, but not ideal. It’s a fi
“I’ve been thinking about running a knife making workshop”. Those were the words in the email that got my attention. Someone was offering to run a workshop and had a PID controlled heated knife oven to do the heat cycle and tempering. Seemed like a good idea to me. This write up is mostly a ‘this is what I did’, the end product is usable, but not ideal. It’s a first attempt and a learning process.

Tuesday, 03. September 2019

mbooth

Simple Hammer Repair

You may recall from my previous post about broken tools that I broke my favourite hammer by using the claw to break down what turned out to be some really sturdy cupboardry. In this post I document the manufacture of a new handle from scratch.

You may recall from my previous post about broken tools that I broke my favourite hammer by using the claw to break down what turned out to be some really sturdy cupboardry. In this post I document the manufacture of a new handle from scratch.


Don't Know My Own Strength

Breaking cheap tools so you don't have to.

Breaking cheap tools so you don't have to.

Friday, 30. August 2019

tomwardill

Creating an RSS Planet in 2019

There’s an IRC channel for our local geeks. Some of the people in it have blogs. In an idle conversation about the state of the modern internet, the idea of a webring came up and continued into ‘why aren’t blog planets’ a thing anymore? Wait, a what now? A planet is an aggregator of blogs usually centered around a topic or project. With the slow death of blogging and the ide
There’s an IRC channel for our local geeks. Some of the people in it have blogs. In an idle conversation about the state of the modern internet, the idea of a webring came up and continued into ‘why aren’t blog planets’ a thing anymore? Wait, a what now? A planet is an aggregator of blogs usually centered around a topic or project. With the slow death of blogging and the idea of commercial content, they seem to have gone away (with a few notable exceptions).

mbooth

Eclipse Module on F30 Addendum

Additional information about installing the Eclipse IDE module on F30.

Additional information about installing the Eclipse IDE module on F30.

Wednesday, 21. August 2019

mbooth

Eclipse is Now a Module on Fedora 30

How to install the Eclipse IDE module on Fedora 30!

How to install the Eclipse IDE module on Fedora 30!

Friday, 02. August 2019

caolan

Redux and TypeScript

I've been using both Redux and TypeScript for a while and, honestly, they don't play well together. Adding types for actions and state and making sure you've handled all the cases in your reducer leads to a lot of boilerplate. Here is a new approach I've been trying for TamaWiki which eliminates a lot of this boilerplate and friction.

The traditional approach

I'll start by describing a co

I've been using both Redux and TypeScript for a while and, honestly, they don't play well together. Adding types for actions and state and making sure you've handled all the cases in your reducer leads to a lot of boilerplate. Here is a new approach I've been trying for TamaWiki which eliminates a lot of this boilerplate and friction.

The traditional approach

I'll start by describing a common pattern first. This is roughly the recipe described on the Redux website, and perhaps the most widespread approach to adding type information to a Redux application.

// Action keys

enum TypeKeys {
    INCREMENT = "INCREMENT",
    DECREMENT = "DECREMENT",
    SET_AMOUNT = "SET_AMOUNT",
}

// Action types

interface IncrementAction {
    type: TypeKeys.INCREMENT,
    by: number
}

interface DecrementAction {
    type: TypeKeys.DECREMENT,
    by: number
}

interface SetAmountAction {
    type: TypeKeys.SET_AMOUNT,
    to: number
}

type Action =
    | IncrementAction
    | DecrementAction
    | SetAmountAction;

// State type

interface State {
    value: number
}

const initial: State = {
    value: 0
};

// Typed reducer

function reducer(state: State = initial, action: Action): State {
    switch (action.type) {
        case TypeKeys.INCREMENT:
            state.value += action.by;
            return state;
        case TypeKeys.DECREMENT:
            state.value -= action.by:
            return state;
        case TypeKeys.SET_AMOUNT:
            state.value = action.to;
            return state;
        default:
            return state;
    }
}

As your application grows it's quite possible you'll end up with a hundred or more actions. All of these pieces need constant maintenance and adding types for actions and their keys quickly becomes onerous.

As is common with all Redux applications, managing scopes in the big switch statement quickly becomes unwieldy too. I often end up breaking the code into separate handler functions, which helps, but also adds more boilerplate!

function handleIncrement(state: State, by: number): State {
    state.value += by;
    return state;
}

function handleDecrement(state: State, by: number): State {
    state.value -= by;
    return state;
}

function handleSetAmount(state: State, to: number): State {
    state.value = to;
    return state;
}

function reducer(state: State = initial, action: Action): State {
    switch (action.type) {
        case TypeKeys.INCREMENT: return handleIncrement(state, action.by);
        case TypeKeys.DECREMENT: return handleDecrement(state, action.by);
        case TypeKeys.SET_AMOUNT: return handleSetAmount(state, action.to);
        default:
            return state;
    }
}
Eliminating separate action keys

One quick win is to simply remove the separate TypeKeys enum. Our TypeScript enum is taking the place of the more traditional action type constants in vanilla Redux.

// Action type constants as recommended by Redux

const INCREMENT = "INCREMENT";
const DECREMENT = "DECREMENT";
const SET_AMOUNT = "SET_AMOUNT";

One of the main reasons Redux recommends this is to avoid making typos when creating actions. By importing a constant, you'll get some early warning if you mess up the name.

I think many people emulate this in TypeScript without much thought, but TypeScript will check this for you. By replacing the TypeKeys value with a string literal, TypeScript will still ensure Actions use the correct string at compile time.

// Action types

interface IncrementAction {
    type: "INCREMENT",
    by: number
}

interface DecrementAction {
    type: "DECREMENT",
    by: number
}

interface SetAmountAction {
    type: "SET_AMOUNT",
    to: number
}

type Action =
    | IncrementAction
    | DecrementAction
    | SetAmountAction;

// ...

function reducer(state: State = initial, action: Action): State {
    switch (action.type) {
        // These get type checked too!
        case "INCREMENT": return handleIncrement(state, action.by);
        case "DECREMENT": return handleDecrement(state, action.by);
        case "SET_AMOUNT": return handleSetAmount(state, action.to);
        default:
            return state;
    }
}
Bonus: detecting unhandled actions

This tip doesn't reduce boilerplate but does address one of my pet peeves with the Redux switch statement. How do you know each action has code in the reducer to handle it?

function reducer(state: State = initial, action: Action): State {
    switch (action.type) {
        case "INCREMENT": return handleIncrement(state, action.by);
        case "DECREMENT": return handleDecrement(state, action.by);
        default:
            return state;
    }
}

In the above code "SET_AMOUNT" is not handled. The only way to find that out currently is at runtime. Hopefully in our unit tests.

By using the never type, we can check all actions have a handler at compile time.

function assertNever(state: State, _: never): State {
    return state;
}

function reducer(state: State = initial, action: Action): State {
    switch (action.type) {
        case "INCREMENT": return handleIncrement(state, action.by);
        case "DECREMENT": return handleDecrement(state, action.by);
        case "SET_AMOUNT": return handleSetAmount(state, action.to);
        default:
            // Check all action types have been handled at compile time,
            // but return current state if called at runtime.
            return assertNever(state, action);
    }
}

The one complication is that Redux has internal actions like @@INIT you're not meant to handle. So at runtime we're likely to accidentally execute assertNever() as the default handler.

To handle this, we only perform the action type check at compile time (by comparing action to the type never). At runtime, assertNever() will safely return the current state.

Aggressive cleanup: generating action types from handlers

OK, let's review where we are. Here's the code so far.

// Action types

interface IncrementAction {
    type: "INCREMENT",
    by: number
}

interface DecrementAction {
    type: "DECREMENT",
    by: number
}

interface SetAmountAction {
    type: "SET_AMOUNT",
    to: number
}

type Action =
    | IncrementAction
    | DecrementAction
    | SetAmountAction;

// State type

interface State {
    value: number
}

const initial: State = {
    value: 0
};

// Typed reducer

function handleIncrement(state: State, by: number): State {
    state.value += by;
    return state;
}

function handleDecrement(state: State, by: number): State {
    state.value -= by;
    return state;
}

function handleSetAmount(state: State, to: number): State {
    state.value = to;
    return state;
}

function assertNever(state: State, _: never): State {
    return state;
}

function reducer(state: State = initial, action: Action): State {
    switch (action.type) {
        case "INCREMENT": return handleIncrement(state, action.by);
        case "DECREMENT": return handleDecrement(state, action.by);
        case "SET_AMOUNT": return handleSetAmount(state, action.to);
        default:
            // Check all action types have been handled at compile time,
            // but return current state if called at runtime.
            return assertNever(state, action);
    }
}

Wow. OK. That's a lot of code to add and subtract some numbers. Here are a few things that stand out to me:

  • Action names exist in three places:
    • The action type itself
    • The handler that's named after it
    • The cases of the switch statement
  • Type information for action parameters are repeated:
    • In the action's type
    • In the action's handler function
  • Maintaining the union type for Action feels very manual.
  • Maintaining the dispatch to handlers in the switch statement feels very manual.

What I'd like is:

  • To no longer manually maintain a union type for all actions.
  • One place to define action parameter types and names.
  • A dispatcher to replace the switch statement so I don't have to touch it any more.
  • To make sure I've handled all the action types.

I think the logical place to collect a lot of this information is in the handlers themselves. I'm going to start by putting the handlers into an object because that gives us the opportunity to map over them in TypeScript.

const handlers = {
    increment(state: State, by: number): State {
        state.value += by;
        return state;
    },
    decrement(state: State, by: number): State {
        state.value -= by;
        return state;
    },
    setAmount(state: State, to: number): State {
        state.value = to;
        return state;
    }
};

We can then map over the handlers object to extract some type names.

type Actions = {[T in keyof typeof handlers]: {type: T}};

The above mapped type is equivalent to:

type Actions = {
    'increment': {type: 'increment'},
    'decrement': {type: 'decrement'},
    'setAmount': {type: 'setAmount'},
};

It's a break from the convention of ALL_CAPS action names in Redux, but otherwise works fine. The actions don't have any parameters yet, but we'll come to that.

To create a type for a specific action we can index into this Actions type.

type Action = Actions[keyof Actions];

Which is equivalent to:

type Action =
    | {type: 'increment'}
    | {type: 'decrement'}
    | {type: 'setAmount'};

So, how to add those parameters? Turns out TypeScript has a Parameters type that might be useful here.

We'll start by collecting all the parameters into a generic data parameter (all our handlers currently have the same number of parameters but that may not be true in future).

const handlers = {
    increment(state: State, data: {by: number}): State {
        state.value += data.by;
        return state;
    },
    decrement(state: State, data: {by: number}): State {
        state.value -= data.by;
        return state;
    },
    setAmount(state: State, data: {to: number}): State {
        state.value = data.to;
        return state;
    }
};

We can then access the data parameter in our mapped type.

type Actions = {
    [T in keyof typeof handlers]:
    { type: T, data: Parameters<typeof handlers[T]>[1] }
};

Which is equivalent to:

type Actions = {
    'increment': {type: 'increment', data: {by: number}},
    'decrement': {type: 'decrement', data: {by: number}},
    'setAmount': {type: 'setAmount', data: {to: number}},
};

Making Action equivalent to:

type Action =
    | {type: 'increment', data: {by: number}}
    | {type: 'decrement', data: {by: number}}
    | {type: 'setAmount', data: {to: number}};

That avoids repeating action names and parameters in two places because we're generating the action type definitions from the handler functions!

Now, let's see if we can get rid of that switch statement.

function reducer(state: State = initial, action: Action): State {
    if (handlers.hasOwnProperty(action.type)) {
        return handlers[action.type](
            state,
            (action as any).data
        );
    }
    // Internal redux action
    return state;
}

Since we know every Action has a matching handler (because it's generated from it), we can simply ignore the type information here and safely dispatch the action to it's associated handler.

We can remove the assertNever(state, action) safety check too because by definition every Action is handled.

How are things looking now?

// --- Here are the things you update ---

interface State {
    value: number
}

const initial: State = {
    value: 0
};

const handlers = {
    increment(state: State, data: { by: number }): State {
        state.value += data.by;
        return state;
    },
    decrement(state: State, data: { by: number }): State {
        state.value -= data.by;
        return state;
    },
    setAmount(state: State, data: { to: number }): State {
        state.value = data.to;
        return state;
    }
};

// --- You should never need to touch these again ---

type Actions = {
    [T in keyof typeof handlers]:
    { type: T, data: Parameters<typeof handlers[T]>[1] }
};

// Action type generated from Handler method names and data parameter
type Action = Actions[keyof Actions];

function reducer(state: State = initial, action: Action): State {
    if (handlers.hasOwnProperty(action.type)) {
        return handlers[action.type](
            state,
            (action as any).data
        );
    }
    // Internal redux action
    return state;
}

Looking at the parts you actually need to update, this looks a lot more manageable!

// This is what your actions look like

dispatch({
    type: 'increment',
    data: {
        by: 123
    }
})

Of course, you'll want to make sure dispatch() checks its parameter against your new Action type. One simple way would be to just wrap Redux's dispatch with your own dispatch function.

function dispatch(action: Action) {
    store.dispatch(action);
}
Bonus: check your store is JSON safe

Redux recommends you only put JSON serializable plain objects into your store to ensure things like time-travel debugging continue to work. If that's something you care about, we can also check this with TypeScript!

Here's a JSON type definition I've been using:

interface JsonObject { [name: string]: JsonValue }
interface JsonArray extends Array<JsonValue> { }
type JsonValue = (null | boolean | number | string | JsonObject | JsonArray);

type Json<T> = T extends JsonValue ? T : InvalidJson<T>;
// The InvalidJson type only exists to present nicer error messages
// than using the never type.
interface InvalidJson<_> { };

Wrapping any type with Json<...> will check at compile-time that you're only using JSON serializable data inside the type.

For example, we could use it in our Handler definitions to make sure action parameters are JSON safe.

const handlers = {
    increment(state: State, data: Json<{ by: number }>): State {
        state.value += data.by;
        return state;
    },
    // ...
};

If for some reason you used a property that wasn't valid JSON, like Date, you'd get an error when you attempt to access any properties on the object.

const handlers = {
    increment(state: State, data: Json<{ by: Date }>): State {
        state.value += data.by;
        return state;
    },
    // ...
};
Property 'by' does not exist on type 'InvalidJson<{ by: Date; }>'.
TL;DR
  • Split reducer into separate handler functions.
  • Generate action type definitions from those handler functions.
  • Optionally check for JSON compatibility in your store.

Any questions or feedback? Send me a comment by email and I'll try to include any useful information in this page.

Edit: (2018-08-03) Replaced the Handlers class with a plain object based on feedback from lobste.rs.

Edit: (2018-08-16) Removed the ReduxAction type and associated type guard after my friend Glen Mailer pointed out having assertNever() return the current state at runtime might be simpler and safer.

Monday, 22. July 2019

mbooth

The State of Java in Flathub

What's the deal with Java in Flathub?

What's the deal with Java in Flathub?

Thursday, 04. July 2019

caolan

TamaWiki 0.1.1

TamaWiki now has participant labels on cursors. A small change that makes it much nicer to use.

Your browser does not support the video tag. Changes
  • Add name labels to participant cursors.
  • Ask for a participant name before connecting to the document.
  • Hide the participants list by default.
Download
  • Linux (x86_64)
  • MacOS (x86_64

TamaWiki now has participant labels on cursors. A small change that makes it much nicer to use.

Your browser does not support the video tag. Changes
  • Add name labels to participant cursors.
  • Ask for a participant name before connecting to the document.
  • Hide the participants list by default.
Download
  • Linux (x86_64)
  • MacOS (x86_64)
  • Windows (x86_64)
Note
  • It has only been tested in Firefox and Chrome.
  • The document is temporary and will be lost when the server is stopped.

Friday, 21. June 2019

caolan

Collaborative Editor in Rust

I've been experimenting with techniques for collaborative editing in Rust recently and I'd like to share my first functional prototype with you.

Your browser does not support the video tag.

You'll probably want to make the video full screen to see it in action. There is no audio.

Download
  • Linux (x86_64)
  • MacOS (x86_64)
  • Windows (x86_64)
N

I've been experimenting with techniques for collaborative editing in Rust recently and I'd like to share my first functional prototype with you.

Your browser does not support the video tag.

You'll probably want to make the video full screen to see it in action. There is no audio.

Download
  • Linux (x86_64)
  • MacOS (x86_64)
  • Windows (x86_64)
Note
  • It has only been tested in Firefox and Chrome.
  • The document is temporary and will be lost when the server is stopped.
Implementation

The demo uses Rust and WebAssembly, and I implemented it using a conflict-free replicated data type (CRDT) based on LOGOOT, which I modified to work with variable-sized strings rather than individual characters or lines. I will explore the implementation and my reasons for this choice in a future blog post (you can subscribe via rss if that's of interest), but for now, I'm just happy to mark this small milestone.

Next steps

I plan to develop the code further and have been particularly interested in designing a wiki around collaborative editing.

It turns out designing a collaborative tool is not as simple as slapping a multi-user textarea on an existing product. It has implications for the user interface, workflows, data storage - it affects the whole project.

This fledgling project is called TamaWiki, after our cat Tama.

If you'd like to chat about it's development or just follow along, please join the mailing list.

Saturday, 09. March 2019

kitation

Hills Of Radiant Winds: Thoughts on Nier

Spoiler Warning: This post will have spoilers for Nier Ending A and B as well as major spoilers for Nier Automata. I haven’t finished Ending C or D of Nier at time of writing so please don’t spoil me either

I remember playing Nier a year or so after it came out and not getting along with it. Partially because the first sidequest I picked up was the one where you can break the thing you’r

Spoiler Warning: This post will have spoilers for Nier Ending A and B as well as major spoilers for Nier Automata. I haven’t finished Ending C or D of Nier at time of writing so please don’t spoil me either

I remember playing Nier a year or so after it came out and not getting along with it. Partially because the first sidequest I picked up was the one where you can break the thing you’re delivering and partially because I’d only played turn based JRPGs up to this point and my skills were lacking. Years later with all the Souls games under my belt and a strong desire to replay Nier Automata I thought I’d revisit it.

This game has possessed me.

Sunday, 20. January 2019

kitation

Discomfort and Player Experience Part 4: The Steam Reviews

A common pattern for exploratory studies is to do one study which is broad and shallow, and follow up those findings with a deep but narrow study. My original plan was to do a questionnaire with follow-up interviews; however the subject matter meant that I wasn’t likely to get ethical approval and I wasn’t in a good place myself. So instead I started off looking at reviews on Steam.

A common pattern for exploratory studies is to do one study which is broad and shallow, and follow up those findings with a deep but narrow study. My original plan was to do a questionnaire with follow-up interviews; however the subject matter meant that I wasn’t likely to get ethical approval and I wasn’t in a good place myself. So instead I started off looking at reviews on Steam.

Thursday, 27. December 2018

kitation

Discomfort and Player Experience Part 3: Research Questions

Author’s note: From this point on, I’ll be writing these sections in my own words. It will give me more space to explain why I did what I did, and allow me to make wider conclusions.

This post is about research questions. What are they? Why are they so hard to come up with?

Author’s note: From this point on, I’ll be writing these sections in my own words. It will give me more space to explain why I did what I did, and allow me to make wider conclusions.

This post is about research questions. What are they? Why are they so hard to come up with?

Monday, 12. November 2018

kitation

Interlude: Why Did I Want To Do This

These “Interlude” posts are an opportunity to talk about things around the project that didn’t go into the final paper. It’s also good to help me reflect on the whole project. This post is looking at why I wanted to study games in the first place and why this project in particular appealed to me.

These “Interlude” posts are an opportunity to talk about things around the project that didn’t go into the final paper. It’s also good to help me reflect on the whole project. This post is looking at why I wanted to study games in the first place and why this project in particular appealed to me.

Friday, 02. November 2018

kitation

Discomfort and Player Experience Part 2: Uncomfortable Interactions

Part 2 covers the second half of my literature review; looking at uncomfortable interactions, empathy and moral values.

Part 2 covers the second half of my literature review; looking at uncomfortable interactions, empathy and moral values.