Serverless Git LFS for Game Development

📅 6 January 2018, updated 15 September 2023 🔖 cloud-software ⏲️ 3 minutes to read

For Estranged, I needed a simple, cheap way of storing binary files. All solutions I tested required me to host a server, or me to pay someone to host a server. I wanted to avoid the flat fee for a constantly running server, and use something completely serverless with a pay-for-what-you-use model.

I settled on using a GitHub private git repository (free) and an LFS (large file storage) backend using Amazon Lambda, Amazon S3 and Amazon API Gateway.

This write-up is a follow up to my older YouTube video covering the manual setup. This guide uses a template for a 1-click deployment of all resources mentioned in the video.

Cost

The costs can be broken down as follows (at the time of writing):

Amazon S3: $0.023 per GB of data stored, $0.090 per GB of data downloaded
Amazon API Gateway: $3.50 per million API calls received
Amazon Lambda: $0.00001667 per GB-second, $0.0000002 per request

The most expensive items here are going to be the storage and data transfer costs in Amazon S3. A quick usage example:

Storing 100GB of data costs (0.023 * 100GB) = $2.30
Downloading a quarter of that data costs (0.090 * 25GB) = $2.25
Total cost for a month = $4.55

The free tier of Amazon API Gateway and Amazon Lambda will likely completely cover your Git LFS backend.

Prerequisites

An Amazon Web Services account. Register here: aws.amazon.com
A Git client, like SourceTree: sourcetreeapp.com
Git LFS installed on your local machine: git-lfs.github.com
An existing private Git repository: BitBucket provides free private repositories

Instructions

Log into the AWS Console
Follow this magic link to set up the stack:
- https://console.aws.amazon.com/cloudformation/home#/stacks/new?region=eu-west-1&templateURL=https://ae-infrastructure-eu-west-1.s3-eu-west-1.amazonaws.com/git-lfs/4.0.0/git-lfs.yaml
Give the stack a helpful name. This will also be used to name the individual resources it creates.
Fill in a shared username and password for your Git LFS endpoint. A generated password works well here.
Click Next, check the checkbox about creating IAM roles, then hit Create.

You can delete this stack and all of its resources in the future by heading over to the CloudFormation console, and clicking Delete Stack. This will not delete the storage as a precautionary measure, but you can do this manaualy via the S3 console. Take note of the "Outputs" tab inside CloudFormation, you need the LfsEndpoint output for .lfsconfig below.

Create .lfsconfig at the root of your repository:

[lfs]
url = <LfsEndpoint from Stack Outputs tab>

You also need to tell Git which files to handle as binary and upload to LFS using .gitattributes at the root of your repository. Below is an example of such a file for Unreal Engine 4:

*.uasset filter=lfs diff=lfs merge=lfs -text
*.umap filter=lfs diff=lfs merge=lfs -text
*.bmp filter=lfs diff=lfs merge=lfs -text
*.tga filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.jpg filter=lfs diff=lfs merge=lfs -text
*.jpeg filter=lfs diff=lfs merge=lfs -text
*.icns filter=lfs diff=lfs merge=lfs -text
*.ico filter=lfs diff=lfs merge=lfs -text
*.dll filter=lfs diff=lfs merge=lfs -text
*.pdb filter=lfs diff=lfs merge=lfs -text
*.psd filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text

Commit .lfsconfig and .gitattributes to the root of your repository, and push to your origin.

Testing

Commit a binary file with one of the extensions in .gitattributes. If you copied the example from above, you can try to commit and push a JPEG file.
Open the S3 console and check the contents of your newly created bucket. It should contain a single object with a strange looking name - if you download it and add the extension back, you should be able to open the file.
Clone the repository to another location or another computer to confirm you can read the files.

Security

You shouldn't use the single username/password approach with a large team or public repository since it will mean multiple users share the same credential.

To instead use BitBucket or GitHub as the authentication provider, see the readme for Estranged.Lfs.

Article Updates

24/07/2022 - Updated CloudFormation stack to use version 4.0.0 of Estranged.Lfs which adds support for Azure blob storage
23/04/2021 - Updated CloudFormation stack to use version 3.1.0 of Estranged.Lfs which adds support for Azure blob storage
18/07/2020 - Updated CloudFormation stack to use version 3.0.0 of Estranged.Lfs, removed username/password from LFS URL it provides, updates security section to mention GitHub and BitBucket authentication

🏷️ lfs amazon git file repository stack estranged storage s3 data cloudformation github api costs console

Alan Edwardes