But also, Astro has the concept of a middleware, that allows to run custom logic in front of every request, which can be handy for things like auth, redirects, proxying and more.

The problem? The middleware is not run for static pages.

Why run the middleware on static pages?

In many cases this is not a problem, because you may not often need to run custom logic in front of static pages. After all if those pages needed custom logic per request, they’d probably be dynamic.

Pages behind auth are most likely dynamic so the content can be contextual to the logged-in user. As for redirects and proxies, they usually make sense when no actual page matches the URL, and Astro runs the middleware in both those cases.

However my use case is a bit different. I’d like to get a sense of what pages are visited on my site, and I don’t like the idea of client-side tracking, partly because of the privacy stigma, and partly because “you can’t trust the client”.

Unnecessary note on client-side tracking

At that point it seems that mostly everyone on the internet does client-side tracking, including every business I ever worked with. Yet I never encountered cases where the data got heavily manipulated.

The only anomalies I ever noticed are the occasional pen test with all sorts of injection attempts, but those typically just get ignored at ingestion because the data is broken, or at worst, it breaks queries because corrupted data was ingested (read: data type mismatch, hopefully we know how to avoid SQL injections by now).

However it seems that most people have something else to do than programmatically sending well formatted but fake tracking data on public endpoints for the mere pleasure of causing chaos for someone else.

Despite that, I’m still allergic to client-side tracking because on paper this is all still possible.

Unnecessary note on HTTP logs

You could say I can achieve all this with HTTP logs without bothering with an edge worker, and you’d be 100% right. However, Cloudflare only gives access to HTTP logs to Enterprise customers and this is not exactly an interesting option for me right now.

If not for that it would definitely be my favorite solution.

Running backend code in front of static pages

Back to the topic. Sadly there’s no generic way I found to run the Astro middleware in front of static pages. This means the solution is gonna be dependent on your adapter. In my case, I’m using the Cloudflare adapter.

We have two layers to deal with here. Out of the box the logic is as follows:

• Cloudflare Workers:
  • If there’s a static asset that matches the URL, serve that.
  • Otherwise, run the Astro worker.js.
• Astro worker.js:
  • If there’s a static asset that matches the URL, serve that.
  • Otherwise, run the user-provided middleware.

The Cloudflare part

In order to mitigate the first layer (Cloudflare), we need to configure the runtime to run the worker code in front to some or all static assets. This is done in wrangler.jsonc using the run_worker_first directive (relevant docs):

{
  "assets": {
    "binding": "ASSETS",
    "directory": "./dist",
    "run_worker_first": [
      "/",
      "/en",
      "/en/*",
      "/fr",
      "/fr/*",
    ],
}

In this case I force the worker to run for the index, as well as /en, /fr and anything under.

This means for other assets like JS/CSS/images, we still skip the worker, but for the static pages I have that match those paths, Cloudflare will run the edge worker.

The Astro part

Now we have Cloudflare run the Astro worker in front of static pages, but it’s still not enough, because the Astro Cloudflare adapter skips our middleware anyway when a static asset matches.

if (app.manifest.assets.has(requestPathname)) {
  return env.ASSETS.fetch(request.url.replace(/\.html$/, ''))
}

In order to solve that, we can’t use the Astro middleware anymore. Instead we need to configure a custom entry point for the worker. This means we now control the top-level worker code and run our logic there, regardless what the adapter decides to do.

This is done with workerEntryPoint option in astro.config.mjs:

import { defineConfig } from 'astro/config'
import cloudflare from '@astrojs/cloudflare'

export default defineConfig({
  // ...

  adapter: cloudflare({
    workerEntryPoint: {
      path: 'src/worker.ts',
    },
  }),

  // ...
})

Where src/worker.ts is a custom Cloudflare Worker entry file as documented here:

import type { SSRManifest } from 'astro'
import { App } from 'astro/app'

import { handle } from '@astrojs/cloudflare/handler'

type Env = {
  [key: string]: unknown
  ASSETS: {
    fetch: (req: Request | string) => Promise<Response>
  }
}

export function createExports(manifest: SSRManifest) {
  const app = new App(manifest)

  const fetch: ExportedHandlerFetchHandler<Env> = async (request, env, ctx) => {
    const url = new URL(request.url)
    const { pathname, search } = url

    // Do anything before Astro handles the request

    const response = await handle(manifest, app, request, env, ctx)

    // Do anything after

    return response
  }

  return {
    default: {
      fetch,
    } satisfies ExportedHandler<Env>,
  }
}

What about development?

The workerEntryPoint is great for production, but astro dev won’t pick up on that. So if you need any of this logic to also run in development, you need to abstract it and also include it in middleware.ts.

This works fine even for static pages because Astro do run the middleware when generating static pages, it just outputs a warning if you try to access things like request headers.

In my case, I chose to move all my production logic in worker.ts, so I don’t rely on the middleware whatsoever in production. I use a conditional export like follows in order to keep the middleware only in development, where it mimics what worker.ts otherwise does in production.

import type { MiddlewareHandler } from 'astro'

const handler: MiddlewareHandler = async (context, next) => {
  // ...
}

export const onRequest = import.meta.env.DEV ? handler : undefined

Wrapping up

In short: configure run_worker_first on Cloudflare so it runs the worker in front of static pages, then use a custom workerEntryPoint with the Astro Cloudflare adapter so you get full control over the worker, and can run code outside of the middleware (which does not run for static pages).

nodejs 24.13.1

Then your Cloudflare deploy fails with:

Initializing build environment...
Success: Finished initializing build environment
Cloning repository...
Found a .tool-versions file in repository root. Installing dependencies.
Restoring from dependencies cache
Restoring from build output cache
Failed: error occurred while installing tools or dependencies

There’s a couple threads on Cloudflare Community about this and similar .tool-versions issues. They get closed after 15 days without any answer, with the oldest one from 2023 and still no solution to this day.

The fact Cloudflare Workers (and Cloudflare Pages) look at .tool-versions is undocumented, it so happens that it chokes on even the most basic .tool-versions possible, so it essentially means the mere presence of this file in your project will break your build on Cloudflare, without any way to turn off this behavior (like forcing Cloudflare to ignore that file). As reported in this issue, the SKIP_DEPENDENCY_INSTALL environment variable does not help with this behavior.

The solution

So what’s left? Well, I had to remove .tool-versions from my repository.

Instead, I moved it to .tool-versions.template, and when setting up the repo in any environment that actually supports .tool-versions, I just cp .tool-versions.template .tool-versions (with .tool-versions being in .gitignore).

Setting the correct Node.js version on Cloudflare

As for Cloudflare, in order to set the proper versions, I’m using the NODE_VERSION and PNPM_VERSION build environment variables. (Also NPM_VERSION and YARN_VERSION depending on your package manager of choice).

NODE_VERSION=24.13.1
PNPM_VERSION=10.29.2

This is, of course, also undocumented, but it works!

Squamish

I’m well settled in Squamish now. Two years already!

As an introvert and working remote, it was a bit of a slow start rebuilding social circles from scratch. It really takes being intentional about it. Facebook groups and the Oak app are pretty sweet, and believe it or not, talking to strangers IRL also works! The good thing is that the more people you know, the more people you meet, so it gets pretty effortless over time.

Mountains 🏔️

In 2025 I’ve been out in the mountains like crazy, and hit 130,000 meters of elevation gain all sports combined (ski touring, trail running, gravel and mountain biking). It’s my biggest elevation gain year lifetime, 30,000 meters more than 2024. 💪

That includes scrambling Sky Pilot, climbing Star Check (3 times lol), Cloudburst, Goat Ridge scramble, Blackcomb Peak ridge scramble, a marathon to Castle Towers, Gentian Peak and Panorama Ridge, Ossa Mountain, Rainbow Mountain, the Armchair Traverse (Mount Cook and Mount Weart), Locomotive Mountain and its loop scramble, Coliseum Mountain, and finishing with the Howe Sound Crest Trail (with extra Harvey, Brunswick and West Lion lol get rekt). What a year!

About part-time work

If you’re wondering how I go in the mountains that much, you need to know I’ve been working part-time as a software contractor since 2021. Going part-time was the best thing I’ve ever done in my career (other than connecting a ping-pong table to the internet 🏓) and I’m incredibly grateful to have worked with companies who gave me that opportunity.

Maybe I should write more about this because it’s still really uncommon in the industry, but believe it or not, those years have also been the most productive of my career.

It’s definitely a combination of factors. Despite working only 20 hours a week:

• I’m naturally more experienced now than I was years ago.
• I have been intentional about working with small companies (read: less than 10 people) where I can have a sizable impact, without being slowed down by corporate friction, processes and politics.
• Parkinson’s law is a thing (“work expands so as to fill the time available”).
• And of course since a few years and increasingly, using AI.

Still, this means that there’s a sweet spot for time-to-productivity ratio, especially when it comes to creative work like building software, and in many cases I’m not convinced that it’s at or above 40 hours.

Travel ✈️

Mexico 🇲🇽

Went to Mexico for a work retreat.

Montreal

Did a pit stop by Montreal to catch up with friends and the 🔥 food scene there.

Toulouse

Did a quick pit stop in Toulouse, which I visited for the first time, since my flight landed there.

Andorra 🇦🇩

Visited friends in Andorra, also first time for me there. There’s some sick mountain biking and trail running. Lots of ridges to scramble, everywhere, and all the mountains are connected by trails, so it feels like there’s really no limits to the terrain!

Camargue 🦩

Another pit stop at Saintes-Maries-de-la-Mer to cut a long drive in half. Also first time here, very pretty!

Metz & Nancy

Finally some family time around Metz and Nancy.

Cakes 🍰

I also happened to turn 30 on that trip. While I don’t believe the specific number means much at all, I still see no reason to not treat myself to 3 different cakes. 😜

Nova Scotia 🦀

I bundled all the above destinations together in July. Nova Scotia was a separate trip.

A good friend of mine was getting married there, and I got to stand at his wedding. Took that as an opportunity to go explore around the Cape Breton and particularly ride the Cabot Trail on my friend’s road bike that I borrowed. Around 300 km and 3,000 meters of elevation that I split in two days. It was beautiful.

Shortest move ever

At the end of the summer, my landlord sold the place I was renting, and I had to move. I got lucky and found a place in the same building just across the floor the same day I got the notice. Still can’t believe this happened. 🙏

This made for a pretty easy and quite ridiculous move, with me dragging my stuff across the hallway.

The big jump 🏃

Remember how I was writing about part-time work earlier? Well all good things have an end, and this was not an option anymore at the company I had been working until then.

Going full-time right before ski season didn’t feel exactly right, and it really felt like a sign for me to 1. take some proper time off and 2. get out of my comfort zone and try something a bit scarier.

The time off consisted of a fuckton of climbing, then skiing. Being in BC, we eventually got a few of the usual warm storms where it rains all the way to the top of mountains and destroy the skiing conditions until it snows again. This gave me some solid windows to start building again.

We got together with my friend Ben and started EveTools, where we make apps for developers.

The first app is Flame, and its goal is to remove as much friction as possible from the Firebase local development experience, especially when it comes to dealing with the emulators.

We’ve been working with Firebase quite extensively for over 3 years now, so we have a pretty good understanding of the pain points of the current experience. There’s a lot to do, and I hope we can have a positive impact on that platform.

At the time of writing it’s in private beta, but we’re planning on opening it up pretty soon! If you find about Flame via this post, shoot me an email and I’ll send you a discount! 🫶

Two interesting realizations since this happened.

Buildingception

First, building a product, and doing so end to end (read: not just the tech), reveals a lot more pain points and problems to be solved. Some specific to the stack we work with, and some broader ones too. And the good news is, a lot of them can be solved with more tech!

In other words, the more I build, the more ideas I get about what to build next.

Granted it’s not the best way to source ideas if you’re optimizing for maximum revenue, but it seems to be doing a decent job at optimizing for maximum motivation, and I do enjoy that for the time being.

So much for part-time

Second, is that despite advocating for the benefits of part-time work for the past 4 years, including higher up in this article, I’m actually having a really good time working every single day right now, weekends included. Working on your own thing does hit different.

I hope my fellow coastal BC skiers don’t read this, but I’m lowkey grateful for this rain-into-dry-spell event a few weeks ago, so that I feel no guilt and FOMO to be so much on my laptop.

Ultimately, I still believe in part-time work, and in fact, being an independent developer is my current best bet on building that lifestyle back one day or another. But it also seems like a futile dream until I generate any sustainable income.

Let’s see where the snow takes me…

On coffee ☕️

I’m back to drinking caffeine, after years of being on decaf. 🤷‍♀️

Seems to pair well with the business hustling for now. We’ll see if I build up a tolerance again. If it stops being a net positive I’ll readjust.

And weirdly, after being into specialty coffee for years (read: small batch micro roaster beans I weigh and grind before brewing), I’m back with my moka pot and sous vide Italian pre-ground. Who would have thought?

A few words on 2024

I know, I know, I skipped the routine update last year. To be fair there was no important update to write about, which is probably why I didn’t feel the need to write. Still it included many cool adventures and trips that I might as well slide in this post.

First there was the northern lights. Took the camera out for the occasion.

Of course, the usual local mountain adventures. Including skiing Panorama Ridge, scrambling West Lion and Black Tusk, Mount Currie, running a 60K to Diamond Head and Mamquam Lake, Sigurd Peak, biking the Triple Crown (Cypress, Grouse and Seymour), and finishing with a marathon to Overlord Mountain!

Went to Jackson Hole for a work retreat, and took some extra time off there to explore the mountains and do a lot of road and gravel biking, including riding around Yellowstone, with a colleague who’s an absolute triathlon machine. Those few weeks are probably the most volume I’ve done in a continuous stretch to this day, although the Andorra trip from last year was a close contender.

Also went sailing in Majorca with a few friends!

And finished by some trail running in the Vosges.

On reflecting

Putting together all those photos is pretty time consuming, but honestly it feels valuable to take that time to reflect on the past and really appreciate everything that I experienced.

It’s easy to get caught up in the fast paced train of life and never take a step back to contemplate the big picture. Those “now page” updates help me do exactly that.

macOS has a setting where it shows fixed scrollbars to people using a mouse, while people using a trackpad have tiny “floating” scrollbars that are entirely hidden when not actively scrolling.

You can find it in Appearance > Show scroll bars and either force it to always on or always off (or keep it auto).

Story time

While developing a macOS app (in my case with Electron, although this problem is not specific to Electron), I encountered a wild bug.

Some users reported having scrollbars visible in different parts of the app, and looking quite “off” visually:

However on my side it looked just fine:

I could not understand what was causing this. After all, it was happening across the same version of Electron, Chromium, of the app itself, and of macOS!

It’s only when another developer reported having the bug that I jumped on a call with him to debug the issue.

While messing with the dev tools, one thing jumped out to me: inside his dev tools, scrollbars were visible at all times:

While on my side I only had floating scrollbars while I was scrolling, and they would disappear entirely otherwise:

Same version of macOS as well!

What was going on?

It turns out the culprit was that my colleague was using a mouse, while I was using the trackpad of my laptop.

What the mouse vs. trackpad have to do with this? Meet this setting:

By default, macOS shows scrollbars at all times when using a mouse, but uses floating scrollbars when using a trackpad!

Now I could force this setting to “Always” to reproduce the issue locally, and update the app to look OK even for people who use a mouse. 🙃

In many places, we had unnecessary overflow: scroll in places where overflow: auto would do, and a few places where we had to hide scrollbars entirely with overflow: hidden.

It was easy to make those silly mistakes when all the devs working on the app were using a trackpad. 😂

Bonus: dark mode scrollbars

For the places where we did want scrollbars, there was a remaining issue: we force our app in dark mode, but the scrollbars were showing in light mode nevertheless!

The way we force our Electron app in dark mode is by doing:

import { nativeTheme } from 'electron'

nativeTheme.themeSource = 'dark'

However in order for native scrollbars to show in dark mode, we also had to add the following to our HTML file:

<body style="color-scheme: dark"></body>

(This is not the place where we actually needed scrollbars, but for lack of a better screenshot, here’s what it would look like with both directions scrollbars:)

• Signing Electron apps with GitHub Actions
• Installing an Apple certificate on macOS runners for Xcode development
• Setting up hosted macOS GitHub Actions workflows for Electron builds

But what to do if your GitHub workflow just hangs forever at the Electron signing step?

This appears to be a symptom of not properly configuring the macOS Keychain in the CI environment. Sadly there could be a bunch of reasons for that and the hanging doesn’t really tell us which one specifically is a problem.

There’s two issues I found with that hanging problem, and the solutions there may or may not solve the problem for you.

What I’m gonna suggest though is really double checking the code you’re using to import the signing keys based on all the links I shared above, and see if you’re missing something.

If it can help, here’s the code that ended up working for me:

name: Release

on:
  push:
    branches:
      - main

jobs:
  release:
    runs-on: macos-latest

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - uses: swift-actions/setup-swift@v2

      - name: Import certificate
        env:
          PRIVATE_KEY: ${{ secrets.PRIVATE_KEY }}
          PRIVATE_KEY_PASSWORD: ${{ secrets.PRIVATE_KEY_PASSWORD }}
          CERTIFICATE: ${{ secrets.CERTIFICATE }}
        run: |
          # Create a temporary keychain
          security create-keychain -p "" build.keychain

          # Set it as default for the user session
          security default-keychain -s build.keychain

          security unlock-keychain -p "" build.keychain

          # Set it to lock in 1 hour (should be long enough, and probs longer than the macOS default that could be too short)
          security set-keychain-settings -t 3600 -l ~/Library/Keychains/build.keychain

          mkdir -p ~/certificates
          cd ~/certificates

          echo "$PRIVATE_KEY" | base64 --decode > "My App.p12"
          echo "CERTIFICATE" | base64 --decode > "My App.cer"

          security import "My App.p12" -k build.keychain -P "$PRIVATE_KEY_PASSWORD" -T /usr/bin/codesign
          security import "My App.cer" -k build.keychain -T /usr/bin/codesign

          # Check certificates
          security find-identity -v -p codesigning build.keychain

          # Add keychain to search list
          security list-keychains -d user -s build.keychain
          security set-key-partition-list -S apple-tool:,apple:,codesign: -s -k "" build.keychain

      # ...

Here, PRIVATE_KEY is a Base64-encoded version of the .p12 private key, password protected by PRIVATE_KEY_PASSWORD.

CERTIFICATE is a Base64-encoded version of the .cer file that Apple provided for your application.

Hope that helps!

This is neat, but what if you want to do do the same thing from your own app?

I’ve had to do this recently, so this blog post will compile everything I learnt about and especially the undocumented quirks I encountered and worked around.

kCMIOHardwarePropertyAllowScreenCaptureDevices

The very first thing you need is to enable kCMIOHardwarePropertyAllowScreenCaptureDevices.

This is a “hardware property” (whatever that means) that, when set, allows the current process to access USB-connected mobile devices for screen recording.

You can find many flavors of how to do this online, and here’s mine anyway:

import CoreMediaIO

// Sets the "hardware" prop that allows to discover USB mobile devices for screen recording.
func allowScreenCaptureDevices() {
  let element: CMIOObjectPropertyElement
  if #available(macOS 12.0, *) {
    element = CMIOObjectPropertyElement(kCMIOObjectPropertyElementMain)
  } else {
    element = CMIOObjectPropertyElement(kCMIOObjectPropertyElementMaster)
  }

  var prop = CMIOObjectPropertyAddress(
    mSelector: CMIOObjectPropertySelector(kCMIOHardwarePropertyAllowScreenCaptureDevices),
    mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal),
    mElement: element)

  var allow: UInt32 = 1
  let dataSize: UInt32 = 4
  let zero: UInt32 = 0

  CMIOObjectSetPropertyData(
    CMIOObjectID(kCMIOObjectSystemObject), &prop, zero, nil, dataSize, &allow)
}

This will allow you to discover USB mobile devices as part of your usual AVCaptureDevice.DiscoverySession.

Now while this code uses a relatively verbose low-level old C interface (because it’s the only way to do this right now), it’s fairly straightforward. It’s spiritually equivalent to doing kCMIOHardwarePropertyAllowScreenCaptureDevices = 1 (no shit).

But this hardware property is not as innocent as it looks, and I’m about to infodump on you everything I found out about it. Brace yourselves (or skip to the next section until you encounter weird issues and need to come back here 😂).

It’s not instant

When you set kCMIOHardwarePropertyAllowScreenCaptureDevices, the effect is not instant, meaning if you do a AVCaptureDevice.DiscoverySession right after, you’re basically guaranteed to not see the connected USB mobile devices.

This is not necessarily a problem. For example if your Swift process is long-running, you set that prop first thing on boot, but you actually list the devices later on upon user interaction, everything will be fine.

However if you’re working with a CLI (i.e. my-cli list-devices / my-cli record-device <device>), or simply need access to the mobile devices immediately upon starting the app, this is not gonna cut it.

After setting the prop, the devices are gonna take up to a few seconds to “show up”, and you can listen to the AVCaptureDeviceWasConnected notification from the NotificationCenter to know about it. There’s a good example for that in this Gist.

// See "The get devices warmup side-effect" below for why this is necessary...
let _ = AVCaptureDevice.devices()

NotificationCenter.default
  .addObserver(
    forName: NSNotification.Name.AVCaptureDeviceWasConnected, object: nil, queue: nil
  ) { (notif) -> Void in
    let device = notif.object! as! AVCaptureDevice
    // ...
  }

This works, but it also means there’s no way to tell immediately that no device is currently connected. This is a problem if you want to implement my-cli list-devices. Your best option is to time out after a few seconds, but it’s not ideal because of the added delay when no device is connected…

The get devices warmup side-effect

This one is super sneaky and I wasted a lot of time on it. It turns out that if you don’t call an API to list the devices, i.e. the deprecated AVCaptureDevice.devices, or now a proper AVCaptureDevice.DiscoverySession, the AVCaptureDeviceWasConnected notification will never arrive.

So you need to start a DiscoverySession first, expecting to get 0 devices back (because you just set the hardware prop and its effect is not instant), just to “warm up” the system, so that it will actually send the notification.

In the Gist I linked earlier, the print("\(AVCaptureDevice.devices().count)") line is actually significant and the code will not work without it:

func start() {
  print("\(AVCaptureDevice.devices().count)")

  NotificationCenter.default
    .addObserver(
      forName: NSNotification.Name.AVCaptureDeviceWasConnected, object: nil, queue: nil
    ) { (notif) -> Void in
      self.iosDeviceAttached(device: notif.object! as! AVCaptureDevice)
    }
}

It’s not just the innocent debug print that it seems. The fact it calls AVCaptureDevice.devices is what allows to warm up the system and for the notification to actually be sent later on. Without it, the notification will never arrive.

I like to make it a bit more explicit with:

// We don't need the data but this appears to be required to "warm up"
// the system. If we don't make the system call to get devices first,
// we can't discover new devices with `AVCaptureDeviceWasConnected`. 🤷
let _ = AVCaptureDevice.devices()

It’s rate limited?

This one also made me pull my hair out for a while. So I start my app that sets the above hardware prop, listen to device connected notifications, and can see the iPhone available for screen recording.

Then I iterate on my code, maybe add some logging or write come code to actually start capturing the video feed, and then restart the app.

And then, not only setting kCMIOHardwarePropertyAllowScreenCaptureDevices takes a few long blocking seconds to complete, but on top of that I don’t ever get any device connected notification despite the iPhone being plugged in!

It appears to me that setting this prop is somehow rate limited. I would need to wait around a minute before launching my CLI again in order for it to behave “normally” (where setting the prop is near-instant, and I do get a notification for the plugged-in devices).

However, and that’s where it gets interesting, I noticed that if any other process on the computer also sets that same hardware property (i.e. QuickTime), and that process stays running in the background, then my CLI would reliably work every single time, even if I launch it many times in a short time span. So it’s like that “rate limit” is really an issue if my CLI is the only process on the system to set that prop.

So what did I do? I made a my-cli background command that literally only sets the kCMIOHardwarePropertyAllowScreenCaptureDevices prop, then sleeps indefinitely. Ran that in the background, then could do my-cli list-devices and so on as much as I wanted.

Wasn’t gonna cut it for me for production, but at least that was useful during development to allow me to iterate quickly.

Actual device recording

I won’t go in much details here because there’s actually no quirks on that side of things. It’s just your typical AVFoundation recording which is very well covered online already.

First we get the external devices via a DiscoverySession:

let devices = AVCaptureDevice.DiscoverySession(
  deviceTypes: [.external],
  // Muxed type seems to be a decent way to distinguish USB connected
  // mobile devices from other external devices like e.g. "OBS Virtual Camera".
  mediaType: .muxed,
  position: .unspecified
).devices

This returns a list of AVCaptureDevice. Alternatively if we have the ID of a mobile device already:

let device = AVCaptureDevice(uniqueID: "...")

Then we make an input from that device:

let deviceInput = try AVCaptureDeviceInput(device: device)

And the rest is the usual AVCaptureSession protocol.

Wrapping up

If you encountered any if the quirks above, I hope that it helped you work around them and hopefully you didn’t waste as much time on this as I did. Happy device recording!

This is a nice feature when you’re looking at the logs after a run is completed.

However when you’re trying to debug a timing-based issue live, this makes it really annoying because the logs are buffered the entire time the process runs and only spit out at the end! And then we can’t see which package logged what first because the logs got grouped by package instead of being in a merged stream.

I ended up dissecting the source of Turborepo to figure out what causes this behavior inside GitHub Actions. I found it.

It’s triggered by the GITHUB_ACTIONS environment variable being present.

So running unset GITHUB_ACTIONS before running turbo run turns off this behavior!

In theory that should suffice but I had the following in my notes so I’ll also leave it here in case it helps:

unset $(env | grep RUNNER | cut -d= -f1)
unset $(env | grep GITHUB | cut -d= -f1)

Triggering a full index

BTW to trigger a full index, either change the Algolia extension config from the Firebase console, and make sure that the Full Index existing documents field is set to true.

If it’s already on true (so that won’t trigger an actual config change) and you don’t want to change anything else in the config, you can set it to false, wait 5 minutes for it to deploy, then set it to true again (lol). Or, better, go in Google Cloud Console, in Cloud Task, and for the ext-firestore-algolia-search-executeFullIndexOperation task queue, select Actions > Force a task run.

Errors during full index

I was looking at the ext-firestore-algolia-search-executeFullIndexOperation Cloud Function logs to monitor the index operation, and after a few minutes it choked with a 400 error (but I’ve also seen 403).

The logs are really unhelpful because the request body gets logged entirely first, but in my case it’s too big truncates the log line, and so if this function logs the response body (which I’m not even sure), it’s not accessible because the line got truncated in the request part.

It took me longer than I’m willing to admit to figure that, but I found out that there’s a API Monitoring section on Algolia where we can Search API Logs and I could see the 400 and 403 errors there!

In my case the errors were:

{
  "message": "Record at the position ... objectID=... is too big size=.../10000 bytes. Please have a look at https://www.algolia.com/doc/guides/sending-and-managing-data/prepare-your-data/in-depth/index-and-records-size-and-usage-limitations/#record-size-limits",
  "status": 400
}

{
  "message": "You have exceeded your Record quota. You’ll need to change your plan for more capacity, or delete records. See more details at https://www.algolia.com/account/billing/details?applicationId=...",
  "status": 403
}

That’s much more useful.

Turns out my staging env had too many records already for its plan (free plan) so I couldn’t sync more during the full index, and in another case the record I was trying to write was too big for the free plan max record size.

I wasn’t having the error in prod because we use a paid plan there.

So the solution was simple. Upgrade staging to a paid plan so I can do my testing!

After that I triggered a full index again and it worked just fine. 🙏

Then you need to sign (left) and notarize (right) your app!

Signing consists in buying a developer membership with Apple, which will let you create a key that you can use to codesign your app with.

Notarizing consists in uploading your app to an Apple service that scans it for malware. If it passes the process, your app gets a “stamp of approval” that is bundled with your app and also mirrored on Apple’s Gatekeeper servers.

In the case of Electron, here’s some relevant docs this article is based on:

• Electron: Signing & notarizing macOS builds
• Electron Forge: Signing a macOS app

Get a signing certificate

The first step is to generate a signing keypair, and get a signing certificate from Apple, which requires you to subscribe to Apple’s developer program.

Then, follow create a certificate signing request.

In my experience, it doesn’t seem that the User Email Address you input matters.

As for Common Name, it seems to only affect how the private and public key are named in Keychain Access.

By saving the request to disk, you will get a CertificateSigningRequest.certSigningRequest file.

This will also create a Common Name.p12 and Common Name.pem entry in your Keychain Access. The .p12 is the private key, and the .pem is the public key, that are going to be associated with the certificate you’re requesting.

You should now upload the .certSigningRequest file to your Apple Developer account, in Certificates, IDs & Profiles. Choose the Developer ID Application certificate type.

This will give you a certificate developerID_application.cer that you need to import in Keychain Access (by simply opening it).

Get the intermediate certificate

In Apple’s PKI, your developer certificate is signed by an intermediate certificate, that’s itself signed by one of Apple’s root certificates.

The Apple root certificates should already be present out of the box in your System Roots keychain, but the intermediate ones are not, and need to be downloaded for the certificate chain to be complete and for codesign to work.

When you created your Developer ID certificate, you were likely prompted between “G2 Sub-CA” and “Previous Sub-CA”. At the time of writing, on the Apple PKI page, those are respectively Developer ID - G2 (Expiring 09/17/2031 00:00:00 UTC) and Developer ID - G1 (Expiring 02/01/2027 22:12:15 UTC). So go ahead and download the appropriate one and install it to your login keychain just like your developer certificate.

$ openssl x509 -in "developerID_application.cer" -noout -issuer
issuer=CN=Developer ID Certification Authority, OU=G2, O=Apple Inc., C=US

Warning: unable to build chain to self-signed root for signer

Manually sign your app

You’re now in a place where you can manually sign your app:

codesign --sign 'Developer ID Application: MyApp (ID)' MyApp.app

To find the identify to pass to --sign:

security find-identity -v -p codesigning

-v will show only valid identities. -p is for selecting a specific policy, here we care about codesigning.

Signing your app with Electron Forge

Add osxSign to your forge.config.js:

module.exports = {
  packagerConfig: {
    osxSign: {
      identity: 'Developer ID Application: MyApp (ID)'
    }
  }
}

If you only have one valid code signing identity configured on your Mac, you can omit the identity parameter. You still need to pass an empty object osxSign: {}.

Notarizing your app with Electron Forge

Add osxNotarize to your forge.config.js. There’s a few ways to configure it documented here.

The documentation is pretty clear and complete so I won’t bother repeating anything here. 🙂

Debugging osxSign and osxNotarize

If you encounter issues where Electron is not properly signing or notarizing your app, you can debug the signing and notarizing process that way:

DEBUG=electron-osx-sign,electron-notarize* npx electron-forge package

This will output detailed logs that should help you identify the culprit.

Conclusion

That should be all you need to have your app approved by Apple so that you can share it with the world. 🌎

Happy building! 🚀

In this post, I’m gonna focus on a macOS app. I’m not sure how much of this applies to Windows. I’ll update this post when we eventually port the app to Windows!

How does the auto updater works?

On macOS, Electron auto updater uses the Squirrel.Mac framework, a “Cocoa framework for updating macOS apps”.

So ultimately, when it comes to the distribution of auto updates, your source of truth is gonna be Squirrel.

Electron has a built-in autoUpdater API that lets you setFeedURL, checkForUpdates, and quitAndInstall.

On boot, you configure the auto updater with a mysterious, undocumented “feed URL”, and then you check for updates periodically, and when an update is found, you can prompt the user to install the update.

What about update-electron-app?

If you read the Electron reference on updating applications, they mention a update-electron-app package, that identifies as “a drop-in module that adds auto updating capabilities to Electron apps”.

This module implements the logic described in the previous section for you, so you just have to call one function on boot and let it deal with periodic checking, and prompting the user to install the update. Cool.

However it’s only meant to work with Electron’s public update service, or static file storage that we’ll talk about later

The typical usage looks like this when using Electron’s public update service:

const { updateElectronApp, UpdateSourceType } = require('update-electron-app')

updateElectronApp({
  updateSource: {
    type: UpdateSourceType.ElectronPublicUpdateService,
    repo: 'github-user/repo'
  }
})

Using update.electronjs.org?

That public update service is hosted by Electron and serves the obscure “feed URL” that we encountered earlier.

In order to use it, you need to point it to a public GitHub repository where you publish releases of your app.

Their service can then respond to auto update requests by checking if there’s a newer release. The app binary is downloaded directly from GitHub releases.

You can also host your own update server. There’s actually a few options you can chose from, and they all comply to this undocumented feed format we still know nothing about.

When using the autoUpdater module, you can configure it like this:

const { autoUpdater } = require('electron')

autoUpdater.setFeedURL({
  url: 'https://server/path/to/feed'
})

That URL seems arbitrary and typically contains the process.platform, maybe process.arch, and your program’s version.

As we saw before, a custom dynamic server won’t work with update-electron-app so you’ll have to implement the logic yourself. Luckily, it’s not very complicated.

What’s behind this feed URL and format?

This format is actually defined by the Squirrel framework.

In case of a dynamic server like in the previous section, the request is as an arbitrary GET request to the URL you configured. It’s important for that URL to include the current app version because your server is expected to respond based on whether or not a new version is available for the given version.

In case no update is available, you should return a 204 No Content.

If an update is available, you should return a 200 OK with the following JSON response:

{
  "url": "https://server/path/to/release.zip",
  "name": "Optional Release Name",
  "notes": "Optional release notes",
  "pub_date": "2024-05-03T12:34:56Z"
}

Now this makes a bit more sense. You can easily make your own server that implements this protocol. Actually, you can probably get away with adding just another endpoint to your existing app. 😎 No need to depend on a third-party service or to self-host and maintain another app. 😅

Static updates format

What’s a bit lesser known is that you don’t even need a dynamic server at all. You can implement auto updates with static files only. 🪶

There’s hints of that in update-electron-app that has a static storage option, as well as Squirrel’s docs that mention a static JSON format.

With update-electron-app, it would look like this:

const { updateElectronApp, UpdateSourceType } = require('update-electron-app')

updateElectronApp({
  updateSource: {
    type: UpdateSourceType.StaticStorage,
    baseUrl: 'https://server/path/to/feed'
  }
})

As for the native autoUpdater, you need to pass the little documented serverType: 'json':

const { autoUpdater } = require('electron')

autoUpdater.setFeedURL({
  url: 'https://server/path/to/feed.json',
  serverType: 'json'
})

In both cases, the feed URL typically contains process.platofrm and maybe process.arch again, but that seems to be really up to you.

It is supposed to respond with the following schema:

{
  "currentRelease": "1.2.3",
  "releases": [
    {
      "version": "1.2.1",
      "updateTo": {
        "version": "1.2.1",
        "url": "https://server/path/to/1.2.1.zip",
        "name": "Optional Release Name",
        "notes": "Optional release notes",
        "pub_date": "2024-05-02T12:34:56Z"
      }
    },
    {
      "version": "1.2.3",
      "updateTo": {
        "version": "1.2.3",
        "url": "https://server/path/to/1.2.3.zip",
        "name": "Optional Release Name",
        "notes": "Optional release notes",
        "pub_date": "2024-05-03T12:34:56Z"
      }
    }
  ]
}

From this static response, Squirrel is able to determine whether it needs to update, and where to fetch the update from.

Don’t get confused by the updateTo naming. releases contains all the releases of your software, and updateTo just contains some metadata about that release, with the url being the only really important part.

I haven’t tested this, but my guess is that all you really need is the entry containing the currentRelease, e.g.:

{
  "currentRelease": "1.2.3",
  "releases": [
    {
      "version": "1.2.3",
      "updateTo": {
        "version": "1.2.3",
        "url": "https://server/path/to/1.2.3.zip",
        "name": "Optional Release Name",
        "notes": "Optional release notes",
        "pub_date": "2024-05-03T12:34:56Z"
      }
    }
  ]
}

That should be enough for Squirrel to know there’s an update available. I’m not sure keeping the entire history of older releases adds any value.

Auto generating the static update files

From the above section, you should have everything you need to manually craft that updates feed and push it on your static file server with your ZIP updates.

However, if you use Electron Forge, there’s (again little documented) ways to generate this static structure automatically!

update-electron-app hints at @electron-forge/publisher-s3, but there’s also @electron-forge/publisher-gcs, allowing you to generate and upload that static update structure respectively to AWS S3 or Google Cloud Storage.

They both work the same but the documentation of the S3 plugin is more complete when it comes to auto updating.

You need not only to add the S3 or GCS publisher, but also configure @electron-forge/maker-zip with the undocumented option macUpdateManifestBaseUrl.

During the “make” step, Electron will build the ZIP file for the release, but with that option, it will also fetch your current static “update feed”, update the currentRelease, and add a new release entry to the releases array, then output that updated RELEASES.json file next to your ZIP files.

Then the S3 or GCS publisher will know to put that new update feed in the right place in your bucket.

In forge.config.js, it looks like this:

module.exports = {
  makers: [
    {
      name: '@electron-forge/maker-zip',
      config: arch => ({
        macUpdateManifestBaseUrl: `https://my-bucket.s3.amazonaws.com/custom/folder/darwin/${arch}`
      })
    }
  ],
  publishers: [
    {
      name: '@electron-forge/publisher-s3',
      config: {
        bucket: 'my-bucket',
        folder: 'custom/folder',
        public: true
      }
    }
    // {
    //   name: '@electron-forge/publisher-gcs',
    //   config: {
    //     bucket: 'my-bucket',
    //     folder: 'custom/folder',
    //     public: true
    //   }
    // }
  ]
}

In the case of macUpdateManifestBaseUrl, like for update-electron-app in JSON mode, it will automatically append /RELEASES.json, so in the above example, if arch is arm64, the complete feed URL would be https://my-bucket.s3.amazonaws.com/custom/folder/darwin/arm64/RELEASES.json.

module.exports = {
  makers: [
    {
      name: '@electron-forge/maker-zip',
      config: () => ({
        macUpdateManifestBaseUrl: 'https://my-bucket.s3.amazonaws.com/custom/folder/darwin/universal'
      })
    }
  ]
}

Conclusion

Will you use Electron’s hosted update service? Or self-host an open-source update server? Or instead implement your own dynamic endpoint? Or maybe you’ll just push static updates on S3, GCS, or your own file server?

Regardless what you chose, you should now have all the elements you need to implement auto updates in your Electron app on macOS the way that suits you best! Cheers. ✌️

What you can do however is using the Firebase Admin SDK to create a custom token for the client SDK, and use the client SDK to make the call. 🙃

What does this looks like?

const admin = require('firebase-admin')
const { initializeApp } = require('firebase/app')
const { getAuth, signInWithCustomToken } = require('firebase/auth')
const { getFunctions, httpsCallable } = require('firebase/auth')

admin.initializeApp({
  // Your admin config
})

initializeApp({
  // Your client config
})

const token = await admin.auth().createCustomToken('admin')

await signInWithCustomToken(getAuth(), token)

const result = await httpsCallable(getFunctions(), 'myCallableFunction').call({})

Here we created a custom token for a virtual user with UID admin (it doesn’t need to exist in Firebase Auth). We can verify that in the function:

const functions = require('firebase-functions')

exports.myCallableFunction = functions.https.onCall(async (data, context) => {
  if (!context.auth) {
    throw new functions.https.HttpsError(
      'unauthenticated',
      'User is not authenticated'
    )
  }

  if (context.auth.uid !== 'admin') {
    throw new functions.https.HttpsError(
      'permission-denied',
      'User is not authorized'
    )
  }
})

Why callable instead of HTTP function?

With a HTTP function, we could have made a simple fetch request to the endpoint.

Why use a callable function then?

In my case, it was because callable functions have auth built-in, whereas you’re responsible to implement your own auth for HTTP functions. I found that using a https.onCall function with a custom Firebase token was more elegant than configuring some kind of internal “API key”.

Invoking the callable function manually

It turns out it’s also quite easy to invoke a callable function without the Firebase SDK, via a plain HTTP call.

With cURL, calling a function that doesn’t have token authentication is as simple as:

curl \
    -X POST \
    -H 'Content-Type: application/json' \
    'https://region-project.cloudfunctions.net/myCallableFunction' \
    --data '{"data": {}}'

For the authentication, we need an Authorization: Bearer header, but we can’t directly use the custom token we generated above. We need to exchange it for an ID token first (this happened transparently in the previous example).

We could use the client SDK to do that for us but at that point we might as well use the client SDK to call the function as well. 😅

Just for educational purpose, and building off the earlier example, it would look like:

const { getAuth, signInWithCustomToken, getIdToken } = require('firebase/auth')

await signInWithCustomToken(getAuth(), token)

console.log(await getIdToken(getAuth().currentUser))

We could then use that token in the cURL request:

curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $token" \
    'https://region-project.cloudfunctions.net/myCallableFunction' \
    --data '{"data": {}}'

But if we’re calling the function via fetch, it’s probably that we don’t want to use the client SDK. Then, exchanging the token would look like this:

curl \
    -X POST \
    -H 'Content-Type: application/json' \
    "https://identitytoolkit.googleapis.com/v1/accounts:signInWithCustomToken?key=$firebaseApiKey" \
    --data "{\"token\": \"$customToken\", \"returnSecureToken\": true}"

This returns an idToken that we can use as Authorization: Bearer in the invocation of the callable function as seen above.

But your test just times out:

thrown: "Exceeded timeout of 5000 ms for a test.
Add a timeout value to this test to increase the timeout, if this is a long-running test. See https://jestjs.io/docs/api#testname-fn-timeout."

So you go on and add a longer timeout value to the test, but then you hit another level of timeout:

@firebase/firestore: Firestore: Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds.
This typically indicates that your device does not have a healthy Internet connection at the moment. The client will operate in offline mode until it is able to successfully connect to the backend.

By any chance, are you using jest-environment-jsdom? Something like this in your jest.config.js:

module.exports = {
  testEnvironment: 'jsdom'
  // testEnvironment: 'jest-environment-jsdom'
}

If so, look no further. Firestore doesn’t like what jest-environment-jsdom does to the global object and makes it hang forever.

It took me long enough to figure that out, so I didn’t manage to figure out why exactly it’s the case. So far my understanding is that it’s related to the fetch API somehow, because if you set the undocumented useFetchStreams option to false in the Firebase client, then it falls back to XMLHttpRequest (which jsdom implements) and things work again.

user.firestore({ useFetchStreams: false, merge: true })

My advice would be to run the Firestore tests in the default Node.js environment instead of the jsdom environment. This may be by using a dedicated jest.config.js, or simply running your Firestore tests separately from the rest of your frontend test suite and passing --env node to override the value from jest.config.js:

npx jest --env node firestore.test.js

Last resort, the useFetchStreams hack above should do it. 😄

Google Cloud Functions used to bundle Chromium in their base images, but it’s been a few years it’s no longer the case. That’s where packages like chrome-aws-lambda come in handy, by bundling Chromium directly inside a npm package, and exposing a function that extracts the Chromium binary and returns the path:

const chromium = require('chrome-aws-lambda')

const path = await chromium.executablePath

However that’s Chromium, and you may have reasons to want Google Chrome instead (mainly, proprietary codecs).

A totally unrelated note about AWS Lambda

This article is about Google Cloud Functions, but if you’re on AWS Lambda, the above option is your best bet. Because of the Lambda total size limit of 250 MB (all layers combined), it’s really hard to get a binary of Chrome that fits in there.

That’s why chrome-aws-lambda uses LambdaFS under the hood, to aggressively compress the Chrome installation with Brotli and make it fit in that limited space.

But again with that build, you won’t have proprietary codecs. I tried to trim down a Chrome Linux build and compress it with the same technique but never managed to make it fit on AWS Lambda. Recent Chrome versions are just too big.

There’s another option, which is to compile Chromium yourself with proprietary codecs. I never found any prebuilt binaries of Chromium that include proprietary codecs (maybe because of license issues redistributing them 🙃) so you’re on your own here.

Remotion successfully does that for Remotion Lambda. Here’s their instructions to compile Chromium with proprietary codecs for Lambda.

Fair warning: it gets hairy, fast.

Back to Google Cloud Functions

Google Cloud Functions is more generous as for bundle size, so we don’t need to resort to those tricks, and we can include a complete, uncompressed, Google Chrome installation.

Google publishes Chrome for Testing, builds specifically made for headless usage.

We can just download the latest build from there as part of the gcp-build script in our package.json.

{
  "scripts": {
    "gcp-build": "curl -s -O 'https://storage.googleapis.com/chrome-for-testing-public/124.0.6367.91/linux64/chrome-linux64.zip' && unzip chrome-linux64.zip && rm chrome-linux64.zip"
  }
}

You will then have the Chrome binary in chrome-linux64/chrome, that you can pass to the tool of your choice.

With Puppeteer

Courtesy of this post, with Puppeteer, you don’t need to download Chrome manually, since it provides a nifty script to do just that.

Actually, Puppeteer’s postinstall script automatically downloads the latest version of Chrome for Testing for your platform.

The caveat is that this script by default installs it to ~/.cache/puppeteer, which in the case of Google Cloud Build, is not gonna be preserved in the final image. So we need to instruct Puppeteer to install Chrome in a directory that Cloud Build will keep.

This can be done with the following .puppeteerrc.js:

module.exports = {
  cacheDirectory: `${__dirname}/.cache/puppeteer`
}

But even then, there’s another caveat. Puppeteer’s postinstall script will only run after it gets installed. However, because of build caching, you will get in a state where node_modules is restored, with Puppeteer already installed (so postinstall will not run), but the .cache/puppeteer directory will also not be restored.

To mitigate that, we need to make sure to install Chrome systematically. Again we can leverage the gcp-build for that:

{
  "scripts": {
    "gcp-build": "npx puppeteer browsers install chrome"
  }
}

The good thing is that this script knows to not re-download Chrome if it’s already found in the cache directory, so when the postinstall script does run, the extra gcp-build command will be a no-op.

Yes, you’re probably missing a .transacting(trx) call, but what’s going on exactly?

Knex maintains a connection pool to your database, which you configure with the pool min and max options. If max is 5, then Knex will keep up to 5 connections to your database in the pool.

If you’re attempting to make a query and the pool is full, then it’ll wait that one of the connection frees up in order to use it.

Sometimes however, it will timeout doing so. One common case is a deadlock when mixing queries inside and outside a transaction.

Let’s say that you start 5 transactions, but within those transactions, you’re performing some queries outside of the transaction:

await knex.transaction(trx => {
  await trx.raw('...')
  await knex.raw('...')
})

Here, the knex.raw statement will execute outside of the transaction (despite being in the function, because it doesn’t use trx). This means that it will use its own connection from the pool, on top of the one knex.transaction already uses.

If you have enough of those running in parallel, you can hit a case where there’s no available connection to execute the knex.raw bit. So it’s waiting that a connection frees up. But no connection get freed up because all the transactions are waiting for the knex.raw bit to complete in order to commit!

See the deadlock here?

So the solution is to make sure that all the queries you perform inside the transaction actually use that transaction. In the above example, it’s very obvious, but it can get trickier when you call a function that calls another function that calls a method that makes a query but didn’t accept a trx parameter and so ends up needing its own connection. 😬

Now you know what to look for. 👀

Then, maybe also like me, you didn’t even know that it had a built-in microphone in the first place, and even less that macOS was automatically switching to that microphone when you connect your headphones!

Luckily, I didn’t sound like shit on calls for too long, because my friends quickly told me “bro, ur mic sounds like shit”. 💩

Forcing the internal microphone

Now we know what’s wrong, let’s fix it. The idea is that when I connect my bluetooth headphones, I want the audio output to go to them, but I don’t want to switch my default microphone.

We can achieve that with the Audio MIDI Setup app.

Create an Aggregate Device (from the + icon at the bottom-left corner) that has only one input: your MacBook microphone. Then set this aggregate device as “default for sound input” from the right click menu.

Tada! Now connecting your headphones will leave the aggregate device alone, meaning you’ll keep using the good microphone that comes with your laptop, without thinking about it. 👌

import admin from 'firebase-admin'
import { applicationDefault } from 'firebase-admin/app'

admin.initializeApp({
  projectId: 'my-project',
  credential: applicationDefault()
})

const auth = admin.auth()

const user = await auth.getUserByEmail('foo@bar.com')

console.log(user)

After all, it works just fine with other Firebase APIs like Firestore.

But in the above case, you’d be getting the following error (spread onto lines for readability):

FirebaseAuthError: //cloud.google.com/docs/authentication/.

If you are getting this error with curl or similar tools, you may need
to specify 'X-Goog-User-Project' HTTP header for quota and billing
purposes.

For more information regarding 'X-Goog-User-Project' header, please
check https://cloud.google.com/apis/docs/system-parameters.

Raw server response:

{
  "error": {
    "code": 403,
    "message": "Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the identitytoolkit.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.",
    "errors": [
      {
        "message": "Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the identitytoolkit.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.",
        "domain": "usageLimits",
        "reason": "accessNotConfigured",
        "extendedHelp": "https://console.developers.google.com"
      }
    ],
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "SERVICE_DISABLED",
        "domain": "googleapis.com",
        "metadata": {
          "service": "identitytoolkit.googleapis.com",
          "consumer": "projects/123456"
        }
      }
    ]
  }
}
}

So what’s going on? Well applicationDefault() works with the application default credentials as created by gcloud auth application-default login, which live in ~/.config/gcloud/application_default_credentials.json.

In my case, those credentials didn’t have access to Firebase Auth for a reason I did not try to understand.

However, what did have access to Firebase Auth is the application default credentials as created by firebase login, which live in ~/.config/firebase/*_application_default_credentials.json.

Firebase’s applicationDefault(), despite being a method of the Firebase SDK, does not know about the Firebase application default credentials, and instead only uses the Google Cloud credentials. 😅

However it supports reading the credentials file from the GOOGLE_APPLICATION_CREDENTIALS environment variable, so we can run the script like this:

GOOGLE_APPLICATION_CREDENTIALS=~/.config/firebase/*_application_default_credentials.json node script.js

Something didn’t work the way it should? Woops, sorry, can’t do much about that, I have no trace of what happened. 🤷

Not ideal.

For a while, the alternative was to replace console.log statements by fetch requests to something that will actually persist logs. Fine, but still not ideal.

Thankfully they introduced Logpush for Workers back in 2022, which finally gave us a way to forward worker logs to a number of destinations, including Amazon S3, Google Cloud Storage, Datadog, Elasticsearch, BigQuery and more.

But none of those options was Google Cloud Logging. And I like to centralize my logs in Google Cloud Logging. Bummer.

Leveraging the HTTP destination

One of those options though is an arbitrary HTTP destination.

With that, I should be able to integrate any log backend I want.

What if I made a Cloudflare Worker to handle the logs of my other workers? That log drain worker probably shouldn’t drain logs to itself to avoid an infinite recursion, but I could fallback in one of the other integrations just for this one.

Configuring a Logpush handler

In order to use Cloudflare Logpush, you need to be under the Cloudflare Enterprise plan. However there’s an exception for the Cloudflare Workers logs! Then all you need is the Workers Paid plan.

You can configure a log handler from your dashboard, in Analytics & Logs > Logs > Add Logpush job. Select Workers trace events as a dataset, select the fields you care about (more on that later), and configure your HTTP endpoint.

This UI looks like a recent addition! When I originally worked on this, Logpush was only configurable by using the Cloudflare HTTP API.

For the record, and because it gives you more control over the Logpush settings, here’s how you would do this with the API.

First, you need an API token, which you can create from My Profile > API Tokens.

While the API docs often reference the usage of API keys with X-Auth-Email and X-Auth-Key headers, those API keys have complete permissions over your account, and I would recommend against using them if you have a better alternative.

The better alternative: API tokens, which lets you scope permissions. In our case, we want to create a custom token with permissions of Zone.Logs.Edit. That token can then be used in a Authorization: Bearer header.

Here’s how you would list existing Logpush jobs:

curl \
    -H "Authorization: Bearer $TOKEN" \
    'https://api.cloudflare.com/client/v4/accounts/my-account-id/logpush/jobs'

Where my-account-id is your account ID, that you can find for example in the Workers & Pages > Overview page on the right.

To create a job:

curl \
    -H "Authorization: Bearer $TOKEN" \
    -H 'Content-Type: application/json' \
    'https://api.cloudflare.com/client/v4/accounts/my-account-id/logpush/jobs' \
    --data '{
  "name": "test",
  "output_options": {
    "field_names": ["DispatchNamespace", "Entrypoint", "Event", "EventTimestampMs", "EventType", "Exceptions", "Logs", "Outcome", "ScriptName", "ScriptTags", "ScriptVersion"],
    "timestamp_format": "rfc3339"
  },
  "destination_conf": "https://my.worker.workers.dev",
  "dataset": "workers_trace_events",
  "enabled": true
}'

Where the API shines compared to the UI, is that you can configure a number of extra options like max_upload_bytes, max_upload_interval_seconds and max_upload_records, to make sure Logpush makes requests within acceptable limits for your endpoint.

In our case, the Logpush handler is also a Cloudflare worker so the max body size will be between 100 MB and 500 MB depending on your plan. But also, Cloudflare workers have a memory limit of 128 MB so that’s something to take into account as well. Oh and keep in mind this memory limit is per-isolate meaning that multiple requests could hit the same isolate. So adjust accordingly, but I don’t have a silver bullet for this one. 🙃

In order to update a job, you’ll need the id that was returned by the create request, or simply fetch it with the list request. It’s gonna be like the create request but you append the log ID in the end, e.g. logpush/jobs/12345, it’s a PUT request, and all fields are optional.

To delete a job, same but it’s a DELETE request with no body.

The Logpush HTTP destination protocol

I didn’t find documentation about what the HTTP destination is supposed to accept, so here’s what I figured out:

• It sends a POST request to the configured URL.
• The body is gzipped.
• The uncompressed body is a newline-delimited JSON of “events”, e.g.:

{"DispatchNamespace":"","Entrypoint":"","Event":{"RayID":"87ed87f80cf22d84","Request":{"URL":"https://test.workers.dev/","Method":"GET"},"Response":{"Status":200}},"EventTimestampMs":1714878560011,"EventType":"fetch","Exceptions":[],"Logs":[{"Level":"log","Message":["bar","foo"],"TimestampMs":1714878560016}],"Outcome":"ok","ScriptName":"test","ScriptTags":[],"ScriptVersion":{"ID":"1e7519b3-08e2-441d-ae10-7c8c6d3b7e17","Message":"","Tag":""}}
{"DispatchNamespace":"","Entrypoint":"","Event":{"RayID":"87ed87fb49cb2d84","Request":{"URL":"https://test.workers.dev/","Method":"GET"},"Response":{"Status":200}},"EventTimestampMs":1714878560532,"EventType":"fetch","Exceptions":[],"Logs":[{"Level":"log","Message":["bar","foo"],"TimestampMs":1714878560532}],"Outcome":"ok","ScriptName":"test","ScriptTags":[],"ScriptVersion":{"ID":"1e7519b3-08e2-441d-ae10-7c8c6d3b7e17","Message":"","Tag":""}}

When you configure the HTTP destination, you get a chance to choose which of those fields are included. You’ll get a different set of fields depending on the kind of dataset you’re dealing with, but in the scope of this article we’re focusing on worker logs.

For reference, here’s the list of supported zone-scoped datasets and account-scoped datasets. Zone-scoped datasets like DNS logs are tied to a specific “zone” (a specific domain), while account-scoped datasets like worker logs are global to your account (workers don’t belong to a particular zone).

Writing the worker

The worker will need to decompress the gzipped body, split it into lines and send the individual logs to the Google Cloud Logging API.

Calling Google Cloud APIs from Cloudflare Workers is a bit of a challenge because the Node.js SDK is not compatible with the workers environment, so we need to reimplement the whole authentication process. But it’s a problem we’ve already solved in the past so it should be no big deal. 😎

First let’s start with the base of the worker, including decompressing the body:

export default {
  async fetch (request) {
    if (request.method !== 'POST') {
      return new Response('', { status: 405 })
    }

    const ds = new DecompressionStream('gzip')
    const stream = request.body.pipeThrough(ds)
    const body = await new Response(stream).text()
    const logs = body.split('\n')

    for (const json of logs) {
      if (json.trim() === '') {
        continue
      }

      const log = JSON.parse(json)

      console.log(log)
    }

    return new Response()
  }
}

To do that, we use the native DecompressionStream, that we can then convert to text with await new Response(stream).text().

Calling the Google Cloud Logging API

Now let’s see how we can call the Google Cloud Logging API. Again, we can’t use the Google Cloud Node.js SDK, so we need to call the REST API manually. Everything about authentication is explained in this post so I won’t cover that again. Read that article to understand how to deal with Google Cloud API authentication from a Cloudflare worker!

When it comes to Google Cloud Logging specifically, you’ll need an aud of https://logging.googleapis.com/ in your JWT. The rest of this post will assume you generated a token variable thanks to the aforementioned article.

In order to write logs, we need to call the entries:write endpoint.

const res = await fetch(
  `https://logging.googleapis.com/v2/entries:write`,
  {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${token}`
    },
    body: JSON.stringify({
      entries: [
        {
          logName: `projects/my-project-id/logs/my-log-id`,
          resource: {
            type: 'generic_node',
            labels: {
              // project_id: '...',
              // location: '...',
              // namespace: '...',
              // node_id: '...'
            }
          },
          // severity: 'DEFAULT',
          timestamp: '2024-05-05T17:38:47.512Z',
          jsonPayload: {
            foo: 'bar
          }
        }
      ]
    })
  }
)

Replace my-project-id by your project ID. my-log-id can be anything.

Here I chose to use a generic_node resource type, but there’s quite a lot of other choices so feel free to use what makes the most sense to you.

The resource type that you chose will have a number of associated labels that you can feed. In this example I included the generic_node labels. project_id doesn’t really need to be set because it will be automatically populated from the project ID in your logName. The other ones are also optional. Put what makes the most sense for your data!

There’s a number of other fields you can set on each log entry in the entries array, but I kept it simple for this example.

You can for example tune the severity, e.g. to distinguish WARNING and ERROR logs appropriately.

Formatting Logpush logs to Google Cloud Logging entries

Let’s look in a bit more details at a worker log received from Logpush:

{
  "DispatchNamespace": "",
  "Entrypoint": "",
  "Event": {
    "RayID": "87ed87f80cf22d84",
    "Request": {
      "URL": "https://test.workers.dev/",
      "Method": "GET"
    },
    "Response": {
      "Status": 200
    }
  },
  "EventTimestampMs": 1714878560011,
  "EventType": "fetch",
  "Exceptions": [],
  "Logs": [
    {
      "Level": "log",
      "Message": [
        "bar",
        "foo"
      ],
      "TimestampMs": 1714878560016
    }
  ],
  "Outcome": "ok",
  "ScriptName": "test",
  "ScriptTags": [],
  "ScriptVersion": {
    "ID": "1e7519b3-08e2-441d-ae10-7c8c6d3b7e17",
    "Message": "",
    "Tag": ""
  }
}

And here’s the associated docs.

As we can see, we get one entry per “event” which in this case, is a whole HTTP request completing.

Then for this particular HTTP request, we’ve got an array of Logs that the worker outputted during its runtime.

console.log('foo', 'bar')

From there, it’s up to you how you translate that to Google Cloud Logging entries. You could:

1. Use the whole “event” as a single log entry and dig in the Logs property to see the actual logs. Then you could set the log severity based on the response status code, e.g. ERROR if the status is >=400.
2. Store the HTTP request “event” without logs in a separate entry, then map the Logs array to individual log entries. Then you could map the Level property to a log severity and have more granularity that way.

For this post, I’ll take the lazy approach and just shove the whole thing in the jsonPayload. 😄

Building off the worker base from earlier:

const entries = []

for (const json of logs) {
  if (json.trim() === '') {
    continue
  }

  const log = JSON.parse(json)

  entries.push({
    logName: `projects/my-project-id/logs/my-log-id`,
    resource: {
      type: 'generic_node',
      labels: {
        namespace: log.ScriptName
      }
    },
    severity: log.Event.Response.Status >= 400 ? 'ERROR' : 'DEFAULT',
    timestamp: new Date(log.EventTimestampMs).toISOString(),
    jsonPayload: log
  })
}

Then as we saw before, we can push those entries to Google Cloud Logging:

const res = await fetch(
  `https://logging.googleapis.com/v2/entries:write`,
  {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${token}`
    },
    body: JSON.stringify({
      entries
    })
  }
)

Make your workers use Logpush!

The final step is to enable Logpush on your workers. By default, even if you have Logpush destinations enabled, they won’t be used unless explicitly enabled at the worker level as well.

You can do that from the UI in your worker page, in Logs > Event logs Workers Logpush. If you use the Wrangler CLI, make sure to also set logpush = true in your wrangler.toml!

Final thoughts

Getting your Cloudflare Workers logs onto Google Cloud Logging is not easy, and using a Cloudflare worker for the integration layer makes it even harder, but it’s also kinda cool if you ask me. 😏

You should now have everything you need to implement that, from the details of using the Cloudflare API to create Logpush jobs and tune it in a way you can’t do from the UI, implementing a HTTP Logpush destination with gzip support, parsing the Logpush payload, all the way to translating it for Google Cloud Logging and push it using the raw HTTP API in an environment where the official SDK is not supported.

I hope you learnt a thing or two thanks to this post, and that your logs are being happily ingested now! 🫶

Essentially, in Safari 16.1+ (and now Firefox 109+), there are more aggressive restrictions on third-party cookies that mess with the way Firebase Auth signInWithRedirect is implemented.

By default, your app could be running on https://myapp.com but signInWithRedirect would redirect to https://myapp.firebaseapp.com/__/auth and then back to your app in order to handle the auth. The message passing with third-party cookies between those two hosts is no longer possible in Safari, Firefox, and soon Chrome.

Firebase docs now document 5 options to solve that.

1. If you host your app on Firebase, make sure your Firebase config authDomain point to your custom domain and not myapp.firebaseapp.com. Because Firebase hosts your app, it will automatically handle the special __/auth path.
2. Use signInWithPopup which doesn’t depend on third-party cookies.
3. If your frontend is not hosted on Firebase, proxy requests from https://myapp.com/__auth/* to https://myapp.firebaseapp.com/__/auth/* so there’s no cross-domain concerns.
4. Download the relevant files from https://myapp.firebaseapp.com/__/auth/* and “self-host” them on your app.
5. Handle provider auth by yourself.

In my case, I’m not hosting the website on Firebase, and I don’t want to use signInWithPupup, so the proxy looks like a solid option.

In a Next.js app, it’s as easy as adding the following to next.config.js:

module.exports = {
  async rewrites () {
    return {
      beforeFiles: [
        {
          source: '/__/auth/:path*',
          destination: `https://myapp.firebaseapp.com/__/auth/:path*`
        }
      ]
    }
  }
}

1. Clone the source table.
2. Perform the maintenance.
3. Make sure they’re in sync if needs be.
4. When the maintenance is over, in a transaction, drop the source table, and rename the clone to the original name.

It can get a bit more complicated than that if you have foreign keys, but I won’t cover that in this article.

However, another way it gets more complicated is when you have views that depend on the table you want to swap:

BEGIN;
DROP TABLE example;
ALTER TABLE example_swap RENAME TO example;

ERROR: cannot drop table example because other objects depend on it
DETAIL: view example_view depends on table example

In this case, we need to update example_view (and all other views that depend on example) to reference the example_swap table before we perform the actual swap.

If this is a one-off swap, fine, but if you’re doing the swap as part of some automated maintenance task, that won’t do it.

Automatically swapping dependent views

In my case, the dependent views don’t change very often (if at all), so I went with a static list of the views that depend on the table I need to swap.

Then I use the following script to swap the views:

CREATE OR REPLACE FUNCTION pg_temp.replace_view_table(view_schema text, view_name text, old_table text, new_table text) RETURNS void AS $$
DECLARE
    view_definition text;
BEGIN
    SELECT definition INTO view_definition
    FROM pg_views
    WHERE schemaname = view_schema
    AND viewname = view_name;

    view_definition := REPLACE(view_definition, old_table, new_table);

    EXECUTE 'CREATE OR REPLACE VIEW ' || view_schema || '.' || view_name || ' AS ' || view_definition;
END;
$$ LANGUAGE plpgsql;

This function will redefine the view to point to the new swap table. It does a basic search and replace in the SQL definition of the view, so you need to make sure the table name doesn’t conflict with anything else in there.

You can now perform the swap as follows:

BEGIN;
SELECT pg_temp.replace_view_table('public', 'example_view', 'example', 'example_swap');
DROP TABLE example;
ALTER TABLE example_swap RENAME TO example;
COMMIT;

The renaming of the table will automatically propagate to the dependent views, they won’t keep referencing the now gone example_swap table, they’ll properly point to example! 🥳

But either way, you have another problem: Yarn downloads all your dependencies on every single build. That’s pretty time consuming. It doesn’t seem that dependencies are getting cached at all.

Vercel recommends setting a ENABLE_ROOT_PATH_BUILD_CACHE=1 environment variable to make build times faster in monorepos.

It sounds great, but in my experience it didn’t do anything, and I’m not the only one.

It seems that regardless of ENABLE_ROOT_PATH_BUILD_CACHE, Vercel doesn’t cache Yarn’s cache folder .yarn/cache, and Yarn 3 and greater will download everything again if this directory is not present, regardless of the state of node_modules.

So the key is to force Yarn’s cache folder to be inside a directory that Vercel actually caches.

I tried setting it inside the root node_modules by doing YARN_CACHE_FOLDER=../../node_modules/.yarn-cache yarn workspaces focus, which worked at first, but quickly encountered some issues when the cache was reused across different build states.

Luckily, thanks to this comment on a totally unrelated issue, I discovered I could put the cache in .next/cache instead, and I never had issues since then!

{
  "installCommand": "YARN_CACHE_FOLDER=.next/cache/yarn-cache yarn workspaces focus"
}