Advanced Git LFS and Jenkins

📅 16 November 2022 🔖 cloud-software ⏲️ 2 minutes to read

I use Jenkins to build Estranged, with Git + LFS.

Recently, I upgraded Git, LFS, and Jenkins, and things stopped working. The checkout strategy I use for Estranged is:

Check out the Git repo, do not perform an LFS pull
Set up Git LFS to use a shared LFS cache
Set up Git LFS credentials to obtain objects not in the cache
Perform the LFS pull

This stopped working when I upgraded the components involved. Read on for the retrospective write-up of the problems and how I fixed them.

Jenkins, Don't Pull From LFS!

I explicitly wanted Jenkins to check out the Git repo without pulling LFS, because I had some configuration to apply which Jenkins couldn't do itself (like using a network level LFS cache).

However... it kept doing it. So, clearly something other than Jenkins is configured to pull from LFS.

Git has multiple levels of configuration, but fortunately it has great tooling to help understand where each bit of config is coming from:

$ git config --list --show-origin
<removed>
file:C:/Program Files/Git/etc/gitconfig filter.lfs.clean=git-lfs clean -- %f
file:C:/Program Files/Git/etc/gitconfig filter.lfs.smudge=git-lfs smudge -- %f
file:C:/Program Files/Git/etc/gitconfig filter.lfs.process=git-lfs filter-process
file:C:/Program Files/Git/etc/gitconfig filter.lfs.required=true
<removed>
file:C:/Users/alan/.gitconfig   filter.lfs.clean=git-lfs clean -- %f
file:C:/Users/alan/.gitconfig   filter.lfs.smudge=git-lfs smudge -- %f
file:C:/Users/alan/.gitconfig   filter.lfs.process=git-lfs filter-process
file:C:/Users/alan/.gitconfig   filter.lfs.required=true

The global system config, and my user config! OK, let's undo that (run the below commands outside of the git repo):

$ git lfs uninstall
$ git lfs uninstall --system

A quick check of git config --list --show-origin again shows the entries removed, and Jenkins no longer automatically pulls from LFS.

NUL Hooks

On the Windows build machine, setting up LFS manually in the root of the Git repo kept failing with a cryptic error:

$ git lfs install
open .\NUL:\pre-push: The filename, directory name, or volume label syntax is incorrect.
To resolve this, either:
  1: run `git lfs update --manual` for instructions on how to merge hooks.
  2: run `git lfs update --force` to overwrite your hook.

Weird! Why is it trying to write the pre-push hook to the Windows equivalent of /dev/null? Well, the clue was in the config. Looking at our friend git config again:

$ git config --list --show-origin
file:.git/config        core.hookspath=NUL:

Jenkins did that when checking out the Git repo. There's probably a reason for it, but I didn't have the energy to investigate, so I adjusted the commands my build runs to:

$ git config --unset core.hooksPath
$ git lfs install
Updated Git hooks.
Git LFS initialized.

All fixed! We can use git config --list --show-origin to verify.

Shared LFS Cache

As I said at the top of the post, I use a home-grown serverless Git LFS implementation on Amazon S3. It's been working great for storage, but if you need to clean your workspace every build, you end up dumping your local LFS cache.

If you have many gigabytes of LFS data (as I do for game development), a fresh check out can cost you a lot of money - in my case just under $5.

Since my build machine is on the same network as my development machines, I wanted to share the cache via a Samba/CIFS share.

So, how to get LFS to share the cache across all machines? My initial thought was to create a symbolic link from the .git/lfs/objects directory to a network drive. This kind of works, but it's a hack (I found LFS breaks in weird and wonderful ways).

Fortunately, Git LFS has a specific config setting for this: https://github.com/git-lfs/git-lfs/pull/2023 (it's worth reading the thread to understand its limitations).

Here's the command to set the config variable (needs to be run in the repo):

$ git config lfs.storage \\share\lfs

That's it! Now LFS will use the network share as the local LFS cache, saving you bandwidth (and time) when running builds which need your LFS objects on the network.

🏷️ lfs git jenkins cache repository config network machine command estranged fix configuration level windows local

Alan Edwardes