Serverless Git LFS for Game Development

📅 6 January 2018, updated 1 August 2024 🔖 cloud-software ⏲️ 5 minutes to read

For Estranged, I needed a simple, cheap way of storing binary files. All solutions I tested required me to host a server, or me to pay someone to host a server. I wanted to avoid the flat fee for a constantly running server, and use something completely serverless with a pay-for-what-you-use model.

I settled on using a GitHub private git repository (free) and an LFS (large file storage) backend using Amazon Lambda, Amazon S3 and Amazon API Gateway. This write-up is a follow up to my older YouTube video covering the manual setup. This guide uses a template for a 1-click deployment of all resources mentioned in the video.

Cost

The costs can be broken down as follows (at the time of writing):

Amazon S3: $0.023 per GB of data stored, $0.090 per GB of data downloaded
Amazon API Gateway: $3.50 per million API calls received
Amazon Lambda: $0.00001667 per GB-second, $0.0000002 per request

The most expensive items here are going to be the storage and data transfer costs in Amazon S3. A quick usage example:

Storing 100GB of data costs (0.023 * 100GB) = $2.30
Downloading a quarter of that data costs (0.090 * 25GB) = $2.25
Total cost for a month = $4.55

The free tier of Amazon API Gateway and Amazon Lambda will likely completely cover your Git LFS backend.

Prerequisites

An Amazon Web Services account. Register here: aws.amazon.com
A Git client, like SourceTree: sourcetreeapp.com
Git LFS installed on your local machine: git-lfs.github.com
An existing private Git repository: GitHub provides free private repositories

Setting up a CloudFormation Stack

The instructions below deploy to eu-west-1 (Ireland). If you'd like to deploy to another region (e.g. closer to you to reduce latency), see Using Another Region.

Log into the AWS Console
Follow this magic link to set up the stack. Alternatively, create a new stack, and manually paste in the following S3 URL:

https://ae-infrastructure-eu-west-1.s3.eu-west-1.amazonaws.com/git-lfs/5.0.0/git-lfs.yaml

Give the stack a helpful name. This will also be used to name the individual resources it creates.
Fill in a shared username and password for your Git LFS endpoint. A generated password works well here.
Click Next, check the checkbox about creating IAM roles, then hit Create.

To remove the resources later, delete the Stack via the CloudFormation console. This will not delete the storage as a precautionary measure, you must do this manually via the S3 console.

Take note of the "Outputs" tab once the CloudFormation Stack has completed, you need the LfsEndpoint output for .lfsconfig below.
Create .lfsconfig at the root of your repository:

[lfs]
url = <LfsEndpoint from Stack Outputs tab>

You also need to tell Git which files to handle as binary and upload to LFS using .gitattributes at the root of your repository. Below is an example of such a file for Unreal Engine 4:

*.uasset filter=lfs diff=lfs merge=lfs -text
*.umap filter=lfs diff=lfs merge=lfs -text
*.bmp filter=lfs diff=lfs merge=lfs -text
*.tga filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.jpg filter=lfs diff=lfs merge=lfs -text
*.jpeg filter=lfs diff=lfs merge=lfs -text
*.icns filter=lfs diff=lfs merge=lfs -text
*.ico filter=lfs diff=lfs merge=lfs -text
*.dll filter=lfs diff=lfs merge=lfs -text
*.pdb filter=lfs diff=lfs merge=lfs -text
*.psd filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text

Commit .lfsconfig and .gitattributes to the root of your repository, and push to your origin.

Testing

Commit a binary file with one of the extensions in .gitattributes. If you copied the example from above, you can try to commit and push a JPEG file.
Open the S3 console and check the contents of your newly created bucket. It should contain a single object with a strange looking name - if you download it and add the extension back, you should be able to open the file.
Clone the repository to another location or another computer to confirm you can read the files.

Security

You shouldn't use the single username/password approach with a large team or public repository since it will mean multiple users share the same credential.

To instead use BitBucket or GitHub as the authentication provider, see the readme for Estranged.Lfs.

Updating the CloudFormation Stack

The instructions below deploy to eu-west-1 (Ireland). If you are deploying to another region, see Using Another Region.

Navigate to the CloudFormation Console
Select the CloudFormation Stack you want to update
Press the "Update" button on the top right
When prompted, select Replace existing template
Paste the following Amazon S3 URL into the box (the latest version of the Stack):

https://ae-infrastructure-eu-west-1.s3.eu-west-1.amazonaws.com/git-lfs/5.0.0/git-lfs.yaml

Hit next, and you will be presented with the LFS username/password (there's no need to change these)
Hit next, and you will be presented with stack options (again, no need to change these)
Hit next, and AWS will ask you to review the changes
Check the checkbox at the bottom of the page about creating IAM credentials, and press Submit

Using Another Region

The automatic links only work for eu-west-1 (Ireland 🇮🇪), because Amazon needs the CloudFormation template to be in a bucket in the same region. To deploy to a different region:

Create a new S3 bucket in the region you want to deploy to
Upload both git-lfs.yaml and Estranged.Lfs.Hosting.Lambda.zip to your own bucket in the region you'd like to deploy to
When following the instructions on this post, use your own URL to the CloudFormation template, for example:

https://<bucket-name>.s3.<aws-region>.amazonaws.com/git-lfs.yaml

Article Updates

31/07/2024 - Updated CloudFormation stack to use version 5.0.0 of Estranged.Lfs with .NET 8
24/07/2022 - Updated CloudFormation stack to use version 4.0.0 of Estranged.Lfs with .NET 6
23/04/2021 - Updated CloudFormation stack to use version 3.1.0 of Estranged.Lfs which adds support for Azure blob storage
18/07/2020 - Updated CloudFormation stack to use version 3.0.0 of Estranged.Lfs, removed username/password from LFS URL it provides, updates security section to mention GitHub and BitBucket authentication

🏷️ lfs stack amazon cloudformation region git file s3 estranged repository deploy github data console password

Alan Edwardes