Enhancing Payload CMS: Implement S3 Versioning Support

Managing file uploads in Payload CMS often involves integrating cloud storage solutions like Amazon S3. This integration addresses many needs but has limitations, especially with versioning. If you use AWS S3 to store file assets with versioning enabled, or if you utilize Payload CMS’s built-in versioning, the default S3 adapter in the Payload Plugin Cloud Storage package may fall short in several ways:

• Lack of Versioning Support: The default adapter doesn’t natively support AWS S3’s versioning. This makes it challenging to retrieve or store different versions of an asset efficiently.
• Custom URL Generation: Generating URLs that reference specific versions of a file can be difficult.
• Handling Payload Content Versions: Ensuring the correct version of an asset is served according to Payload CMS’s versioning can be tricky.

Here, I’ll show you how to create a custom S3 adapter that addresses these gaps. By the end, you’ll have a solution that ensures seamless integration of AWS S3 versioning with Payload CMS and ensures Payload CMS versioning works correctly with S3 Bucket assets.

Needed Adjustments

When you store files in an S3 bucket with versioning enabled or manage content versions in Payload CMS, you must handle and retrieve the specific file versions your application needs. However, the default S3 adapter in Payload CMS doesn’t fully integrate with Payload’s versioning system out of the box. To seamlessly manage versions in both S3 and Payload CMS, you’ll likely need to tweak the adapter code. The necessary adjustments include:

• Retrieving Objects by Version ID: The default adapter doesn’t handle retrieving an object from an S3 bucket based on its version ID. This limitation can cause inconsistencies when serving the correct version of an asset.
• Storing S3 Version IDs in Payload CMS: There’s no built-in mechanism to automatically capture and store S3 Version IDs within Payload collections. This gap makes it difficult to map a specific version of a stored asset to Payload’s versioning system.
• Generating Version-Specific URLs: The default S3 adapter generates URLs but lacks support for appending version IDs. However, this is essential for rendering the correct file version.
• Deletion: The default deletion handler from the cloud storage plugin doesn’t distinguish between different file versions. It deletes files based solely on their key. The custom solution reuses the underlying plugin’s handleDelete method. You can expand this solution when S3 version-aware deletion becomes necessary.

Custom S3 Adapter Solution

To bridge these gaps, we’ll create a custom S3 adapter that:

• Handles Version-Specific Retrieval: Retrieves specific versions of files from S3 using stored version IDs.
• Captures and Stores Version IDs: Automatically manages S3 Version IDs during uploads and stores them in Payload CMS.
• Provides Accurate URL Generation: Generates URLs that reference the correct version of a file stored in S3.
• Integrates with Payload CMS Versioning: Ensures that Payload’s content versions correctly interact with file versions stored in S3.

This custom adapter ensures smooth, version-accurate file management within Payload CMS, giving you more control over asset versions both in S3 and within your CMS content.

Detailed Features

1. Custom URL Generation with Versioning

When working with versioned files, your URLs must point to the correct versions. This custom function generates URLs that include S3 Version IDs where applicable:

const getCustomGenerateURL = ({ bucket }) => ({ filename, prefix = '', versionId = '' }) => {
const url = `${bucket}/${path.posix.join(prefix, filename)}`;
return versionId ? `${url}?versionId=${versionId}` : url;
};

This function generates a URL based on the bucket, prefix, and filename, and appends the versionId if it’s available. This ensures that the URL will point to the specific version of the file, giving you full control over which file version is rendered or accessed.

2. Capturing and Storing S3 Version IDs

When you upload files to an S3 bucket with versioning enabled, capturing the S3 Version ID and storing it in Payload CMS ensures that you can always retrieve the correct version of an asset.
The handleUpload function efficiently manages this by storing the VersionId returned from S3 after a successful upload:

const getCustomHandleUpload = ({ acl, bucket, getStorageClient, prefix = '' }) => {
return async ({ data, file }) => {
const fileKey = path.posix.join(data.prefix || prefix, file.filename);
const fileBufferOrStream = file.tempFilePath ? fs.createReadStream(file.tempFilePath) : file.buffer;

const uploadParams = {
ACL: acl,
Body: fileBufferOrStream,
Bucket: bucket,
ContentType: file.mimeType,
Key: fileKey,
};

const storageClient = getStorageClient();
let uploadResponse;

if (file.buffer.length < 50 * 1024 * 1024) { 
uploadResponse = await storageClient.putObject(uploadParams); 
} else { 
const parallelUploadS3 = new Upload({ client: storageClient, params: uploadParams, partSize: 50 * 1024 * 1024, queueSize: 4, });
uploadResponse = await parallelUploadS3.done(); 
} 
if (uploadResponse?.VersionId) { 
data.s3VersionId = uploadResponse.VersionId; 
} 
return data; 
}; 
};

This characterizes how the custom adapter keeps track of S3 Version IDs and stores them within your Payload CMS, ensuring seamless version management.

3. Reliable File Deletion with Versioning

When removing files, it’s crucial to account for both versioned and non-versioned files. Our custom deletion mechanism ensures that files are securely deleted:

const getCustomHandleDelete = ({ bucket, getStorageClient }) => async ({ doc: { prefix = '' }, filename }) => {
const storageClient = getStorageClient();
await storageClient.deleteObject({
Bucket: bucket,
Key: path.posix.join(prefix, filename),
});
};

This deletion logic is consistent with the underlying handleDelete method from the cloud storage plugin, ensuring files are deleted based on their key and prefix.

4. Enhanced Static File Retrieval

The staticHandler function in this custom adapter effectively retrieves the correct version of the file from S3, utilizing prefixes and version IDs for accurate retrieval. It’s imperative to ensure that the VersionId is utilized when retrieving the object, so the correct version is served in your Payload admin UI and that this retrieval aligns with the content versioning of Payload CMS:

export const getHandler = ({ bucket, collection, getStorageClient }) => {
return async (req, res, next) => {
try {
const { prefix, s3VersionId } = await getFilePrefix({ collection, req });

const object = await getStorageClient().getObject({
Bucket: bucket,
Key: path.posix.join(prefix, req.params.filename),
VersionId: s3VersionId, // Ensure the correct version is retrieved
});

res.set({
'Accept-Ranges': object.AcceptRanges,
'Content-Length': object.ContentLength,
'Content-Type': object.ContentType,
ETag: object.ETag,
});

if (object?.Body) {
return object.Body.pipe(res);
}

return next();
} catch (err) {
req.payload.logger.error(err);
return next();
}
};
};

This handler retrieves the correct file version from S3, ensuring accurate asset rendering or processing in your application. This accuracy is crucial when serving files through the Payload admin UI. The handler must grab the correct version to ensure consistent file serving that aligns with Payload CMS’s internal versioning system.

Integration With Payload CMS

To integrate the custom adapter into your Payload CMS setup, you can configure it within your Payload config file as follows:

const storageAdapter = customS3Adapter({
acl: 'private',
config: {
credentials: {
accessKeyId: process.env.S3_ACCESS_KEY_ID,
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
},
region: process.env.S3_REGION,
},
bucket: process.env.S3_BUCKET,
});

export default buildConfig({
plugins: [
cloudStorage({
collections: {
assets: {
adapter: storageAdapter,
},
},
}),
],
});

Remember to also track the S3 Version IDs within your Payload collections by adding a read-only field in your collection configuration:

const Assets: CollectionConfig = {
slug: 'assets',
labels: {
singular: 'Asset',
plural: 'Assets',
},
access: {
create: () => true,
read: () => true,
update: () => true,
delete: () => true,
},
versions: {
drafts: false,
},
upload: {
staticURL: '/uploads',
},
fields: [
{
name: 's3VersionId',
label: 'S3 Version ID',
type: 'text',
required: false,
admin: {
readOnly: true,
},
},
],
};

This field captures and retains the S3 Version ID for each asset, providing a clear record of the specific version of each stored file.

Custom S3 Adapter for Payload CMS

With a custom S3 adapter for Payload CMS, you can effectively leverage AWS S3’s versioning alongside Payload CMS’s content versioning. As a result, you can manage your assets accurately and keep them up-to-date. Additionally, it offers essential customization for file retrieval and serving, which helps maintain consistency between your S3 storage and CMS.

Moreover, this approach proves particularly valuable for content-heavy applications requiring precise version management. With this custom adapter, you’ll be well-equipped to handle complex versioning needs in both storage and CMS environments.