Serverless Git LFS for Game Development
For Estranged, I needed a simple, cheap way of storing binary files. All solutions I tested required me to host a server, or me to pay someone to host a server. I wanted to avoid the flat fee for a constantly running server, and use something completely serverless with a pay-for-what-you-use model.
I settled on using a BitBucket private git repository (free) and an LFS (large file storage) backend using Amazon Lambda, Amazon S3 and Amazon API Gateway.
This write-up is a follow up to my older YouTube video covering the manual setup. This guide uses a template for a 1-click deployment of all resources mentioned in the video.
The costs can be broken down as follows (at the time of writing):
The most expensive items here are going to be the storage and data transfer costs in Amazon S3. A quick usage example:
- Storing 100GB of data costs (0.023 * 100GB) = $2.30
- Downloading a quarter of that data costs (0.090 * 25GB) = $2.25
- Total cost for a month = $4.55
The free tier of Amazon API Gateway and Amazon Lambda will likely completely cover your Git LFS backend.
- An Amazon Web Services account. Register here: aws.amazon.com
- A Git client, like SourceTree: sourcetreeapp.com
- Git LFS installed on your local machine: git-lfs.github.com
- An existing private Git repository: BitBucket provides free private repositories
- Log into the AWS Console
- Follow this magic link to set up the stack:
- Give the stack a helpful name. This will also be used to name the individual resources it creates.
- Fill in a shared username and password for your Git LFS endpoint. A generated password works well here.
- Click Next, check the checkbox about creating IAM roles, then hit Create.
You can delete this stack and all of its resources in the future by heading over to the CloudFormation console, and clicking Delete Stack. This will not delete the storage as a precautionary measure, but you can do this manaualy via the S3 console. Take note of the "Outputs" tab inside CloudFormation, you need the
LfsEndpoint output for
.lfsconfig at the root of your repository (to see the implications of that read security below):
url = <LfsEndpoint from Stack Outputs tab>
- You also need to tell Git which files to handle as binary and upload to LFS using
.gitattributes at the root of your repository. Below is an example of such a file for Unreal Engine 4:
*.uasset filter=lfs diff=lfs merge=lfs -text
*.umap filter=lfs diff=lfs merge=lfs -text
*.bmp filter=lfs diff=lfs merge=lfs -text
*.tga filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.jpg filter=lfs diff=lfs merge=lfs -text
*.jpeg filter=lfs diff=lfs merge=lfs -text
*.icns filter=lfs diff=lfs merge=lfs -text
*.ico filter=lfs diff=lfs merge=lfs -text
*.dll filter=lfs diff=lfs merge=lfs -text
*.pdb filter=lfs diff=lfs merge=lfs -text
*.psd filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text
.gitattributes to the root of your repository, and push to your origin.
- Commit a binary file with one of the extensions in
.gitattributes. If you copied the example from above, you can try to commit and push a JPEG file.
- Open the S3 console and check the contents of your newly created bucket. It should contain a single object with a strange looking name - if you download it and add the extension back, you should be able to open the file.
- Clone the repository to another location or another computer to confirm you can read the files.
You cannot use the hardcoded single username/password approach above with a public git repository. As
.lfsconfig can be seen by everyone with access to the repository, it will allow all of your storage to be read/written to by anyone on the internet.
You should rotate the username/password on a regular basis - but for tighter security per-user, you will need to add custom security code to the Lambda itself. See the source on GitHub for more information: https://github.com/alanedwardes/Estranged.Lfs