2 Comments

Wrapping a TypeScript Function for Background Execution

In my last post, I showed how to write a higher-order function that could wrap an existing function without losing the original function’s types.

Today, I’m going to show how you can use that same technique to wrap an existing function for a different result–to execute it in a background process using the workerpool npm module.

Background Worker Process

Since Node.js is single-threaded, you have to be careful about performing any kind of lengthy synchronous processing, as this process will block the entire application. In many cases, you’ll want to use a robust job queue (e.g. Kue) to handle background tasks. But there are plenty of situations where a more lightweight solution will suffice–and there are countless libraries that can start up a background process (or pool of processes) and manage the interprocess communication for you.

For the purposes of this post, I’m going to use the workerpool library. Specifically, I’ll be creating a dedicated worker (one of the options for how to use workerpool).

The Function

Here’s a function that synchronously generates random data, writes the data to a file, and returns the path to that file on disk:


import * as crypto from 'crypto';
import * as fs from 'fs';
import * as Path from 'path';

export function createRandomDataFile(numBytes: number): string {
  // https://stackoverflow.com/a/44078785
  const uniqueId = Math.random().toString(36).substring(2)
    + (new Date()).getTime().toString(36);

  const path = Path.join(process.cwd(), uniqueId);
  const buffer = crypto.randomBytes(numBytes);

  fs.writeFileSync(path, buffer);

  return path;
}

Depending on the size of the file being generated, this function could block the event loop for many seconds–clearly, not acceptable.

Background Process

When using a dedicated worker with workerpool, you pass the name (which can be any label) from the foreground process (the main event loop) to a background worker process which has a set of registered functions it can execute.

Here is our worker.ts file with the createRandomDataFile function registered:


import * as workerpool from 'workerpool';
import { createRandomDataFile } from 'random-data';

workerpool.worker({
  createRandomDataFile,
});

Foreground Process

To invoke our function from the main process, we need to tell workerpool about the worker.ts file and then tell it to invoke the function by name:


import * as workerpool from 'workerpool';

const pool = workerpool.pool(__dirname + '/worker.ts');

export function createRandomDataFile(numBytes: number): Promise<string> {
  return pool.exec('createRandomDataFile', [numBytes]);
}

This isn’t really that bad. The biggest downside is that the type signature of the function is now being defined in two places, and there’s nothing to enforce that they stay in sync. This is probably not that big of a deal for this simple example, but if the arguments included a more complicated object type, it could get out of sync and lead to an error that’s hard to catch.

Higher-Order Wrapper

Using the types I talked about in Generic Higher-Order Functions in TypeScript, here’s a function that wraps a given function so that when called, it dispatches the call to the worker process:


import * as workerpool from 'workerpool';

const pool = workerpool.pool(__dirname + '/worker.ts');

function makeBackgroundable<T extends (...args: any[]) => any>(func: T): (...funcArgs: Parameters<T>) => Promise<ReturnType<T>> {
  const funcName = func.name;

  return (...args: Parameters<T>): ReturnType<T> => {
    return pool.exec(funcName, args);
  };
}

Note that the function returned from makeBackgroundable wraps the return value in a Promise because the new function is now asynchronous, while the original function was not.

Finally, we can create a “backgrounded” function from the original function and call it from some other file (e.g. main.ts).


import { makeBackgroundable } from 'backgroundable'
import { createRandomDataFile } from 'random-data';

const backgroundedCreateRandomDataFile =
  makeBackgroundable(createRandomDataFile);

async function run() {
  const path = await backgroundedCreateRandomDataFile(1024 * 1024 * 1024);
  // ...
}

The type signature of the backgroundedCreateRandomDataFile function will exactly match that of the original (with the exception of the return value being wrapped in a Promise), and it will immediately reflect any changes made to the signature of the original as well. Just what we wanted.