Serverless Git LFS for Game Development

Posted January 6th, 2018, last updated July 18th, 2020 in cloud-software

For Estranged, I needed a simple, cheap way of storing binary files. All solutions I tested required me to host a server, or me to pay someone to host a server. I wanted to avoid the flat fee for a constantly running server, and use something completely serverless with a pay-for-what-you-use model.

I settled on using a GitHub private git repository (free) and an LFS (large file storage) backend using Amazon Lambda, Amazon S3 and Amazon API Gateway.

This write-up is a follow up to my older YouTube video covering the manual setup. This guide uses a template for a 1-click deployment of all resources mentioned in the video.

Cost

The costs can be broken down as follows (at the time of writing):

The most expensive items here are going to be the storage and data transfer costs in Amazon S3. A quick usage example:

  • Storing 100GB of data costs (0.023 * 100GB) = $2.30
  • Downloading a quarter of that data costs (0.090 * 25GB) = $2.25
  • Total cost for a month = $4.55

The free tier of Amazon API Gateway and Amazon Lambda will likely completely cover your Git LFS backend.

Prerequisites

  1. An Amazon Web Services account. Register here: aws.amazon.com
  2. A Git client, like SourceTree: sourcetreeapp.com
  3. Git LFS installed on your local machine: git-lfs.github.com
  4. An existing private Git repository: BitBucket provides free private repositories

Instructions

  1. Log into the AWS Console
  2. Follow this magic link to set up the stack:
  3. Give the stack a helpful name. This will also be used to name the individual resources it creates.
  4. Fill in a shared username and password for your Git LFS endpoint. A generated password works well here.
  5. Click Next, check the checkbox about creating IAM roles, then hit Create.

You can delete this stack and all of its resources in the future by heading over to the CloudFormation console, and clicking Delete Stack. This will not delete the storage as a precautionary measure, but you can do this manaualy via the S3 console. Take note of the "Outputs" tab inside CloudFormation, you need the LfsEndpoint output for .lfsconfig below.

  1. Create .lfsconfig at the root of your repository:
[lfs]
url = <LfsEndpoint from Stack Outputs tab>
  1. You also need to tell Git which files to handle as binary and upload to LFS using .gitattributes at the root of your repository. Below is an example of such a file for Unreal Engine 4:
*.uasset filter=lfs diff=lfs merge=lfs -text
*.umap filter=lfs diff=lfs merge=lfs -text
*.bmp filter=lfs diff=lfs merge=lfs -text
*.tga filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
*.jpg filter=lfs diff=lfs merge=lfs -text
*.jpeg filter=lfs diff=lfs merge=lfs -text
*.icns filter=lfs diff=lfs merge=lfs -text
*.ico filter=lfs diff=lfs merge=lfs -text
*.dll filter=lfs diff=lfs merge=lfs -text
*.pdb filter=lfs diff=lfs merge=lfs -text
*.psd filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text
  1. Commit .lfsconfig and .gitattributes to the root of your repository, and push to your origin.

Testing

  1. Commit a binary file with one of the extensions in .gitattributes. If you copied the example from above, you can try to commit and push a JPEG file.
  2. Open the S3 console and check the contents of your newly created bucket. It should contain a single object with a strange looking name - if you download it and add the extension back, you should be able to open the file.
  3. Clone the repository to another location or another computer to confirm you can read the files.

Security

You shouldn't use the single username/password approach with a large team or public repository since it will mean multiple users share the same credential.

To instead use BitBucket or GitHub as the authentication provider, see the readme for Estranged.Lfs.

Article Updates

  • 18/07/2020 - Updated CloudFormation stack to use version 3.0.0 of Estranged.Lfs, removed username/password from LFS URL it provides, updates security section to mention GitHub and BitBucket authentication

Comments