Improve DX of linking prebuilds

[kraenhansen]

Problem

Loading addons via relative paths on Node.js

Early on, we had a goal of supporting packages on NPM with existing Node-API addons.
Many existing packages use simple names for their library targets (often just "addon"), which result in an addon.node file somewhere in the package directory.
These dynamic libraries are references by their relative path when loading (perhaps located via a helper like bindings), so it's easy for Node.js to disambiguate them.

Library files share a global namespace in apps

A problem arise in a mobile context where library files are included in an app bundle, are no longer loaded by a relative path and therefore share a global namespace (the directory of bundled libraries).

For Apple triplets we rename library files and frameworks and rebuild XCFrameworks from the renamed files.
This is inflexible and brittle, since we're making assumptions on the internal structure of the original XCFramework, the absence of dSYM files and location of Info.plist files, etc. There are also differences between macOS frameworks (different dylib subdirectory) that we need to handle. The unknown-unknowns and adapting to format changes going forward is concerning to me.

This is not ideal.

We have similar renames of Android library files, but that seem less problematic so far.

Rethinking the solution

It seems inflexible that we don't just support linking any XCFramework with a dylib containing a Node-API addon.

Well ... we sort of already do

A library author could already today do the following:

1. Opt-out of "linking" their XCFramework (by not placing or removing the magic react-native-node-api-module file)
2. Get the dynamic library into the app bundle themselves, by:
   1. Including their XCFramework via the vendored_frameworks of a podspec and
   2. Including their Android .so files via the jniLibs.srcDirs in a Gradle project.
3. Call requireNodeAddon with the original library name:
   1. Use otool -D <path_to_dylib> to get the argument for an Apple binary (strip the @rpath/*.framework/` part)
   2. Use llvm-readelf -d <path_to_so> | grep SONAME to get the argument for an Android binary (strip the lib and .so suffix)

(Note, the call signature of requireNodeAddon is likely to change in the future).

While a nice escape hatch, it does feel to me that ☝️ is a significantly degraded DX and I feel the host package's ability to discover and automatically include libraries into the app bundles is a significant DX improvement on this. For developers with a Node.js background, learning Gradle and CocoaPods is daunting - using linking via the host package skips this entirely.

Option 1) Copy library files as-is

Instead of renaming the library files as we copy them into the host package, we could copy them as-is (and hope for the best 🤞 )
In this case, we should check for duplicate names and fail early with clear instructions on how to rename at build time.
This would push the responsibility of resolving naming conflicts to the library authors.

Naming conflicts appear most frequently in Node.js examples (repeating names like "hello" and "addon") or in packages which copied their CMakeList.txt from other packages or (old) templates which declare an "addon" target. It may be less of an actual problem in practice.

This behavior to avoid renaming, could potentially be a per-package or per-library opt-in option, i.e. renaming by default and advanced library authors, to opt-out of this behavior.

Option 2) Stop copying library files to the host package altogether

If we choose to not rename libraries, it makes less sense to copy them into the host package and instead reference them in their original location inside the dependency package:

1. On Android it's possible for our host package to reference .so files from all over the file system, by passing absolute paths through jniLibs.srcDirs in our Gradle script.
2. On Apple however, it's not possible to reference XCFrameworks by absolute paths nor relatively outside of the podspec root (the vendored_frameworks is an array of glob patterns resolved relative to the podspec's root).

The only solution to not copying XCFrameworks to the host while still preserving the DX of "linking" them, would be by adjusting the app's Xcode project ourselves instead of relying on Cocoapods to do this. This seems a bit more involved and risky, but to would rely on the xcodeproj gem (which Cocoapod is using internally) or @bacons/xcode. The CocoaPods framework handling scripts are quite involved (extracting from XCFrameworks, code-resigning, etc.) - it's unclear if this complexity is needed for our use case. Taking on this responsibility would however also help us "future proof" against the inevitable chaos when React Native no longer use / support Cocoapods.

Option 3) Build with better names

Regardless if we choose to "copy library files as-is" or "stop copying library files", we could help library authors get a better library name to avoid downstream conflicts:

1. By warning / failing if their target doesn't contain their package name when running cmake-rn. The package names however, may not match target names (e.g., @company/foo → foo), and being too strict about naming could be inflexible for experimentation or unpublished packages.
2. Inject a CMake cache variable definition (ex PACKAGE_NAME) with name from the surrounding package.json (escaped to avoid special chars like @ and /) and encourage (through examples and gyp-to-cmake output?) setting this as OUTPUT_NAME to this injected package name through set_target_properties on their library target.

[shirakaba]

(Cross-post from Discord)

Here's the NativeScript.podspec I'm currently using in order to try your workaround.

require "json"

Pod::Spec.new do |s|
  s.name         = "NativeScript"
  s.version      = "0.2.0"
  s.summary      = ""
  s.homepage     = "https://github.com/NativeScript/napi-ios.git"
  s.author       = "DjDeveloperr", "Jamie Birch"
  s.source       = { git: '' }

  # This leads to these two symptoms:
  # - `-framework NativeScript.apple.node`
  # - `ld: framework 'NativeScript.apple.node' not found`
  # ... so I think we have to call it a .xcframework just to keep CocoaPods happy?
  # 
  # s.vendored_frameworks = "build/RelWithDebInfo/NativeScript.apple.node"

  s.vendored_frameworks = "build/RelWithDebInfo/NativeScript.xcframework"

  s.platform = :macos, '11.0'
end

As instructed, I'm currently doing this in my entrypoint:

module.exports = require("react-native-node-api").requireNodeAddon("-nativescript-macos-node-api--NativeScript");

... however, I'm getting this at launch:

Termination Reason:  Namespace DYLD, Code 1, Library missing
Library not loaded: @rpath/-nativescript-macos-node-api--NativeScript.framework/-nativescript-macos-node-api--NativeScript
Referenced from: <0F80EE2B-ADF0-303F-940A-797FD7C02D14> /Users/USER/Library/Developer/Xcode/DerivedData/sumappmacos-grdfefdgguvwbhbnojmtycuxzsja/Build/Products/Debug/sumappmacos.app/Contents/MacOS/sumappmacos.debug.dylib
Reason: tried: '/usr/lib/swift/-nativescript-macos-node-api--NativeScript.framework/-nativescript-macos-node-api--NativeScript' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/swift/-nativescript-macos-node-api--NativeScript.framework/-nativescript-macos-node-api--NativeScript' (no such file), '/Users/jamie/Library/Developer/Xcode/DerivedData/sumappmacos-grdfefdgguvwbhbnojmtycuxzsja/Build/Products/Debug/sumappmacos.app/Contents/Frameworks/-nativescript-macos-node-api--NativeScript.framework/-nativescript-macos-node-api--NativeScript' (no such file), '/Users/jamie/Library/Developer/Xcode/DerivedData/sumappmacos-grdfefdgguvwbhbnojmtycu
(terminated at launch; ignore backtrace)

Given that the actual library looks like this (no mention of -nativescript-macos-node-api--NativeScript):

└── RelWithDebInfo
    └── NativeScript.apple.node
        ├── Info.plist
        ├── macos-arm64_x86_64
        │   └── NativeScript.framework
        │       ├── Headers -> Versions/Current/Headers
        │       ├── NativeScript -> Versions/Current/NativeScript
        │       ├── Resources -> Versions/Current/Resources
        │       └── Versions
        │           ├── 0.1.0
        │           │   ├── Headers
        │           │   │   └── NativeScript.h
        │           │   ├── NativeScript
        │           │   ├── Resources
        │           │   │   └── Info.plist
        │           │   └── _CodeSignature
        │           │       └── CodeResources
        │           └── Current -> 0.1.0
        └── react-native-node-api-module-optout

... I'm wondering whether my entrypoint should rather be:

module.exports = require("react-native-node-api").requireNodeAddon("NativeScript");

However, even if I save that change, the app keeps searching under:

@rpath/-nativescript-macos-node-api--NativeScript.framework/-nativescript-macos-node-api--NativeScript

I dunno if this is a cache thing or what. If it is cache, how can I clear it?

[ammarahm-ed]

Hey, you probably need to fix the @rpath manually on the library.
https://stackoverflow.com/questions/33991581/install-name-tool-to-update-a-executable-to-search-for-dylib-in-mac-os-x

and

https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172

[kraenhansen]

With #281 we're now copying Xcframeworks and updating them in-place instead of recreating them from the framework directories.

This is not exactly "Option 1) Copy library files as-is", since we are actually modifying the files, but I consider the new implementation much more robust and as such, I'll close this discussion for now.

We might want to add an option to copy the frameworks as-is, in the future, but my current thinking is that this will be an opt-in option in some per-app configuration of react-native-node-api.