For anyone else running into this in the future, all I had to do was to fix my Network Site urls under Network Admin > Sites and ensure that every sub-site’s mapped domain started with https://. For some reason it looks like Wordpress, by default, sets the protocol on new sub-site creation to http: when using sub-domain setup.