I recently worked on an Express API that needed to support a file upload endpoint. After some research and experimenting my pair and I decided to use busboy for our implementation. Here, I’ll walk through the minimal solution we came up with to create a simple Express middleware that saves an uploaded file to a temp file and makes the location of the file available to a later request handler.
A number of libraries do this type of thing, and they’re all a lot more full-featured than what’s described in this post. We liked how multer worked. However, we ran into some issues during our initial research and decided against using it. (My guess is that those issues were the result of developer error and not problems with the library. Unfortunately, I don’t recall now exactly what the issues were).
This post documents an implementation that uses busboy in much the same way multer does.
Express Middleware
The desired outcome here is to have an Express middleware function that saves the uploaded file to disk and makes the location of the temporary file available to the endpoint handler function. It then cleans up the temp file when the request is done being handled.
The assumption is that the endpoint code will do something with the file (inspect it, copy it to S3 or another filesystem, etc.) during the handling of the request.
const express = require('express')
const app = express()
app.post('/packages', uploadedFile('package'), function (req, res, next) {
const filePath = req.uploadedPackage;
...
});
The uploadedFile function takes the name of the multipart section that is expected to hold the file data. It returns an Express middleware function.
Busboy
The way busboy works is by piping the incoming request into it and then handling events for the different multipart form sections it encounters.
In this case, we were only interested in a single file attachment. (The details of the package were all contained in the zipped-up file being uploaded, so there was no need to handle a separate multipart form section containing metadata about the file).
I’ll provide the full middleware function code and then call out a few important points below.
const os = require('os');
const fs = require('fs/promises');
const {createWriteStream} = require('fs');
const {v4: uuidv4} = require('uuid');
const busboy = require('busboy');
const path = require('path');
const uploadedFile = (multipartName) => (req, res, next) => {
let fileTmpPromise;
try {
const fileTmpPath = path.join(os.tmpdir(), uuidv4());
const bb = busboy({
headers: req.headers,
});
bb.on('error', err => {
// Send this error along to the global error handler
next(err);
});
bb.on('file', (name, file) => {
if (name === multipartName) {
fileTmpPromise = new Promise((resolve, reject) => {
const writeStream = createWriteStream(fileTmpPath)
.on('error', reject)
.on('finish', resolve);
file.on('error', reject);
file.pipe(writeStream);
});
} else {
file.resume();
}
});
bb.on('close', async () => {
try {
if (!fileTmpPromise) {
res.status(400).json({ message: `Uploaded file '${multipartName}' is missing` });
return;
}
// Wait for the temp file to be fully written to disk before proceeding
await fileTmpPromise;
// Make the file path available for later handlers
req[`uploaded${multipartName.charAt(0).toUpperCase()}${multipartName.slice(1)}`] = fileTmpPath;
res.on('finish', async () => {
try {
await fs.rm(fileTmpPath, {force: true});
} catch (err) {
// Oops, the tmp file didn't get removed
}
});
next();
} catch (err) {
// Send this error along to the global error handler
next(err);
} finally {
bb.removeAllListeners();
}
});
req.pipe(bb);
} catch (err) {
res.status(400).json({ message: `Could not retrieve file from request - ${err.message}`});
}
};
Waiting for the File
One issue we ran into while working on this was moving on to the next middleware (calling next()) before the uploaded file was fully saved to disk. We’re handling that by using a Promise (fileTmpPromise) that’s supposed to resolve when the file is ready. The key to getting that to work correctly was to resolve when the writeStream finished:
const writeStream = createWriteStream(fileTmpPath)
.on('error', reject)
.on('finish', resolve);
It took us a while to find the right place to resolve the promise that would ensure that the file was done being saved. By resolving when the writeStream fired it’s finish event, the file is always saved and ready after the fileTmpPromise is awaited.
And then the place to actually await that promise is inside the bb.close event handler, as that gets triggered once all of the multipart sections have been encountered.
Remove the Temp File
To remove the temporary file at the end of the request, a standard Express response finish event is used:
res.on('finish', async () => {
try {
await fs.rm(fileTmpPath, {force: true});
} catch (err) {
// Oops, the tmp file didn't get removed
}
});
Error Handling
There are a lot of places where something can go wrong when handling a multipart form file upload like this. The code above tries to put some kind of error handling in place for all of the different locations, but it doesn’t do any logging/reporting. If you’re going to try something like this implementation, you’d likely want some pretty robust logging in place to help identify where things are going wrong.
One thing to specifically point out is the call to next(err); in some of the error handlers. This allows your default error handling to take over instead of doing something custom (assuming you’ve got an Express default error handler in place).
Uploading a File
And finally, to test out your endpoint you can use Axios and form-data to upload an attached file.
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');
const uploadFile = async () => {
const form = new FormData();
const fileStream = fs.createReadStream("local_file.zip");
form.append('package', fileStream, 'package.zip');
await axios.post('http://localhost:3000/api/packages', form, {
headers: {
...form.getHeaders(),
'Content-Type': 'multipart/form-data',
},
});
};
uploadFile();
I encourage you to give multer a try before using something custom like this. But if you find yourself wanting something that multer’s not providing, this example shows you how you can use busboy directly.