Advanced Git LFS and Jenkins
I use Jenkins to build Estranged, with Git + LFS.
Recently, I upgraded Git, LFS, and Jenkins, and things stopped working. The checkout strategy I use for Estranged is:
- Check out the Git repo, do not perform an LFS pull
- Set up Git LFS to use a shared LFS cache
- Set up Git LFS credentials to obtain objects not in the cache
- Perform the LFS pull
This stopped working when I upgraded the components involved. Read on for the retrospective write-up of the problems and how I fixed them.
Jenkins, Don't Pull From LFS!
I explicitly wanted Jenkins to check out the Git repo without pulling LFS, because I had some configuration to apply which Jenkins couldn't do itself (like using a network level LFS cache).
However... it kept doing it. So, clearly something other than Jenkins is configured to pull from LFS.
Git has multiple levels of configuration, but fortunately it has great tooling to help understand where each bit of config is coming from:
$ git config --list --show-origin <removed> file:C:/Program Files/Git/etc/gitconfig filter.lfs.clean=git-lfs clean -- %f file:C:/Program Files/Git/etc/gitconfig filter.lfs.smudge=git-lfs smudge -- %f file:C:/Program Files/Git/etc/gitconfig filter.lfs.process=git-lfs filter-process file:C:/Program Files/Git/etc/gitconfig filter.lfs.required=true <removed> file:C:/Users/alan/.gitconfig filter.lfs.clean=git-lfs clean -- %f file:C:/Users/alan/.gitconfig filter.lfs.smudge=git-lfs smudge -- %f file:C:/Users/alan/.gitconfig filter.lfs.process=git-lfs filter-process file:C:/Users/alan/.gitconfig filter.lfs.required=true
The global system config, and my user config! OK, let's undo that (run the below commands outside of the git repo):
$ git lfs uninstall $ git lfs uninstall --system
A quick check of
git config --list --show-origin again shows the entries removed, and Jenkins no longer automatically pulls from LFS.
On the Windows build machine, setting up LFS manually in the root of the Git repo kept failing with a cryptic error:
$ git lfs install open .\NUL:\pre-push: The filename, directory name, or volume label syntax is incorrect. To resolve this, either: 1: run `git lfs update --manual` for instructions on how to merge hooks. 2: run `git lfs update --force` to overwrite your hook.
Weird! Why is it trying to write the
pre-push hook to the Windows equivalent of
/dev/null? Well, the clue was in the config. Looking at our friend
git config again:
$ git config --list --show-origin file:.git/config core.hookspath=NUL:
Jenkins did that when checking out the Git repo. There's probably a reason for it, but I didn't have the energy to investigate, so I adjusted the commands my build runs to:
$ git config --unset core.hooksPath $ git lfs install Updated Git hooks. Git LFS initialized.
All fixed! We can use
git config --list --show-origin to verify.
Shared LFS Cache
As I said at the top of the post, I use a home-grown serverless Git LFS implementation on Amazon S3. It's been working great for storage, but if you need to clean your workspace every build, you end up dumping your local LFS cache.
If you have many gigabytes of LFS data (as I do for game development), a fresh check out can cost you a lot of money - in my case just under $5.
Since my build machine is on the same network as my development machines, I wanted to share the cache via a Samba/CIFS share.
So, how to get LFS to share the cache across all machines? My initial thought was to create a symbolic link from the
.git/lfs/objects directory to a network drive. This kind of works, but it's a hack (I found LFS breaks in weird and wonderful ways).
Fortunately, Git LFS has a specific config setting for this: https://github.com/git-lfs/git-lfs/pull/2023 (it's worth reading the thread to understand its limitations).
Here's the command to set the config variable (needs to be run in the repo):
$ git config lfs.storage \\share\lfs
That's it! Now LFS will use the network share as the local LFS cache, saving you bandwidth (and time) when running builds which need your LFS objects on the network.