Automating macOS Notarization for UE4

📅 23 November 2019 🔖 game-development ⏲️ 4 minutes to read

⚠️ This post was last updated in 2019, meaning its contents may be outdated.

One of the trickier requirements introduced with macOS Catalina was the requirement that all applications must be notarized, for any kind of distribution.

Xcode does have tools to do this for you, but when you're building software that is bootstrapped by its own build process (such as a game), or you automate the build and deployment of your app, you need to use the custom workflow.

This post will discuss the entire notarization process for an Unreal Engine 4 game, but the process is the same for any app bundle.

Notarization

You can read about that process here, but below is a summary:

All binaries (which includes dylib files) must be signed with codesign, using a Developer ID certificate. At signing time you must indicate that the app uses the hardened runtime, and specify any special entitlements that it will need to run:

# See the end of this post for an example entitlements.xml
$ codesign --sign "MyCertificateId" --entitlements "entitlements.xml" --options runtime --timestamp "MyBundle.app"
# Verify the signature
$ codesign --verify --deep --strict --verbose=2 "MyBundle.app"

Once the binaries are signed, they must be uploaded to Apple to be notarized:

$ xcrun altool --notarize-app -primary-bundle-id "MyId" --file "MyBinaries.zip" --username "MyUser" --password "MyPassword"

The process will take a while (on Apple's servers), and it can be checked for completion using this command:

# Your Request ID is a UUID, and will be returned by the upload call
$ xcrun altool --notarization-info "RequestId" --username "MyUser" --password "MyPassword"

After Apple notarizes the binaries, they must be stapled using the stapler tool:

$ xcrun stapler staple MyBundle.app
# Verify the app with Gatekeeper
$ spctl -vvv --assess --type exec "MyBundle.app"

Automating

If you're building a one-off app the above is fairly easy to do, if not slightly time consuming. However if you want to automate it (say with Jenkins) there are a few challenges to overcome:

The initial binary discovery (you could hard code these)
~~Zipping only the binaries (makes upload faster)~~ Zipping up the app bundle [1]
The upload & wait loop

[1] As of Februrary 2020 zipping up the binaries alone no longer works for me. I have modified my tool to instead submit a zip of the app bundle using the ditto tool to preserve all original file attributes.

In order to run this on Jenkins and automate these pieces, I wrote a C# libary as part of Estranged.Build. Here's an example of this tool being invoked at the end of a build process in a Jenkinsfile:

withCredentials([[
    $class: 'UsernamePasswordMultiBinding',
    credentialsId: 'AppleDeveloperCredentials',
    usernameVariable: 'USERNAME',
    passwordVariable: 'PASSWORD'
]]) {
    sh 'dotnet run --project depot/Tools/Estranged.Build.Notarizer' +
       ' --appPath "Insulam.app"' +
       ' --certificateId "Developer ID Application: Alan Edwardes (2DZRF6JQ1B)"' +
       ' --entitlements "Insulam.app=com.apple.security.cs.allow-dyld-environment-variables"' +
       ' --developerUsername $USERNAME --developerPassword $PASSWORD'
}

The tool will run through the entire process listed above, including the verification steps (with verbose logging). See the the readme for more information.

Entitlements

This is a complete XML file for use as input to the codesign tool, which grants the com.apple.security.cs.allow-dyld-environment-variables entitlement to the app you wish to sign:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>com.apple.security.cs.allow-dyld-environment-variables</key>
        <true/>
    </dict>
</plist>

See the full list of entitlements for the hardened runtime. Adding more entitlements is just a case of adding more keys to the dictionary - though my tool will create this file for you.

To see which entitlements an application has, use this command:

$ codesign -d --entitlements :- "MyBundle.app"

It will print the full XML document showing the entitlements for that app bundle.

Troubleshooting

I had may problems inititally with my app crashing after I had used the codesign tool, and here's some information that would have helped me in this process:

Your app may need entitlements. Try signing with all of them, then remove them individually to see which are required
If you suspect you have problems with the signature, the codesign --verify command (above) is invaluable
To validate whether the whole process worked, use spctl command (example above) is invaluable
If you deploy your app, but get the error MyBundle.app: a sealed resource is missing or invalid, this means a file changed since the bundle was signed. In my case, this was the Steam deployment, which I had set to exclude *.dSYM files (but doing this modified the app bundle and invalidated the signature).
When deploying your app, you also need to be careful that (for example) an in-place upgrade didn't leave any files hanging around from the previous version of your app. In my case I pushed this update out via Steam over a very old version of my app - but Steam left some files around in the app bundle folder. This also invalidated the app with the a sealed resource is missing or invalid error.
The mac that you build your app bundle on seems to trust the app implicitly. I haven't worked out why or where that is set, but it is also very annoying for testing gatekeeper, since it won't block the opening of your app if the notarization process failed. If you have a second mac to test with this can help, but I believe the spctl command (example above) can be trusted to tell you when Gatekeeper will accept the app:

$ spctl -vvv --assess --type exec "Insulam.app"
Insulam.app: accepted
source=Notarized Developer ID
origin=Developer ID Application: Alan Edwardes (5FPFY3YS9F)

Resources

Here are some resources that I found useful when completing this work:

🏷️ application file bundle tool binary entitlements notarization automate apple command zip deployment signed upload signature