merge #16392: macOS toolchain update

This commit is contained in:
Kittywhiskers Van Gogh 2021-08-25 20:40:23 +05:30
parent a8e8a3d585
commit a358d2e593
13 changed files with 173 additions and 87 deletions

View File

@ -37,7 +37,7 @@ builder-image:
image: $CI_REGISTRY_IMAGE:builder-$CI_COMMIT_REF_SLUG image: $CI_REGISTRY_IMAGE:builder-$CI_COMMIT_REF_SLUG
variables: variables:
SDK_URL: https://bitcoincore.org/depends-sources/sdks SDK_URL: https://bitcoincore.org/depends-sources/sdks
OSX_SDK: "10.11" OSX_SDK: "10.14"
MAKEJOBS: -j4 MAKEJOBS: -j4
before_script: before_script:
- echo HOST=$HOST - echo HOST=$HOST

View File

@ -314,7 +314,7 @@ after_success:
env: >- env: >-
HOST=x86_64-apple-darwin16 HOST=x86_64-apple-darwin16
PACKAGES="cmake imagemagick libcap-dev librsvg2-bin libz-dev libbz2-dev libtiff-tools python3-dev python3-setuptools" PACKAGES="cmake imagemagick libcap-dev librsvg2-bin libz-dev libbz2-dev libtiff-tools python3-dev python3-setuptools"
OSX_SDK=10.11 OSX_SDK=10.14
RUN_UNIT_TESTS=false RUN_UNIT_TESTS=false
RUN_FUNCTIONAL_TESTS=false RUN_FUNCTIONAL_TESTS=false
GOAL="deploy" GOAL="deploy"

View File

@ -83,7 +83,7 @@ elif [ "$BUILD_TARGET" = "linux64_release" ]; then
elif [ "$BUILD_TARGET" = "mac" ]; then elif [ "$BUILD_TARGET" = "mac" ]; then
export HOST=x86_64-apple-darwin16 export HOST=x86_64-apple-darwin16
export BITCOIN_CONFIG="--enable-gui --enable-reduce-exports --disable-miner" export BITCOIN_CONFIG="--enable-gui --enable-reduce-exports --disable-miner"
export OSX_SDK=10.11 export OSX_SDK=10.14
export GOAL="all deploy" export GOAL="all deploy"
export RUN_UNITTESTS=false export RUN_UNITTESTS=false
export RUN_INTEGRATIONTESTS=false export RUN_INTEGRATIONTESTS=false

View File

@ -70,8 +70,8 @@ def build():
if args.macos: if args.macos:
print('\nCompiling ' + args.version + ' MacOS') print('\nCompiling ' + args.version + ' MacOS')
subprocess.check_call(['wget', '-N', '-P', 'inputs', 'https://bitcoincore.org/depends-sources/sdks/MacOSX10.11.sdk.tar.gz']) subprocess.check_call(['wget', '-N', '-P', 'inputs', 'https://bitcoincore.org/depends-sources/sdks/MacOSX10.14.sdk.tar.gz'])
subprocess.check_output(["echo 'bec9d089ebf2e2dd59b1a811a38ec78ebd5da18cbbcd6ab39d1e59f64ac5033f inputs/MacOSX10.11.sdk.tar.gz' | sha256sum -c"], shell=True) subprocess.check_output(["echo '328aff47e28c17093d59a72712a9b2e62cd8a8b87bbe03f91abb32960b413f0f inputs/MacOSX10.14.sdk.tar.gz' | sha256sum -c"], shell=True)
subprocess.check_call(['bin/gbuild', '-j', args.jobs, '-m', args.memory, '--commit', 'dash='+args.commit, '--url', 'dash='+args.url, '../dash/contrib/gitian-descriptors/gitian-osx.yml']) subprocess.check_call(['bin/gbuild', '-j', args.jobs, '-m', args.memory, '--commit', 'dash='+args.commit, '--url', 'dash='+args.url, '../dash/contrib/gitian-descriptors/gitian-osx.yml'])
subprocess.check_call(['bin/gsign', '-p', args.sign_prog, '--signer', args.signer, '--release', args.version+'-osx-unsigned', '--destination', '../gitian.sigs/', '../dash/contrib/gitian-descriptors/gitian-osx.yml']) subprocess.check_call(['bin/gsign', '-p', args.sign_prog, '--signer', args.signer, '--release', args.version+'-osx-unsigned', '--destination', '../gitian.sigs/', '../dash/contrib/gitian-descriptors/gitian-osx.yml'])
subprocess.check_call('mv build/out/dashcore-*-osx-unsigned.tar.gz inputs/dashcore-osx-unsigned.tar.gz', shell=True) subprocess.check_call('mv build/out/dashcore-*-osx-unsigned.tar.gz inputs/dashcore-osx-unsigned.tar.gz', shell=True)

View File

@ -31,7 +31,7 @@ remotes:
- "url": "https://github.com/dashpay/dash.git" - "url": "https://github.com/dashpay/dash.git"
"dir": "dash" "dir": "dash"
files: files:
- "MacOSX10.11.sdk.tar.gz" - "MacOSX10.14.sdk.tar.gz"
script: | script: |
set -e -o pipefail set -e -o pipefail
@ -107,7 +107,7 @@ script: |
BASEPREFIX=`pwd`/depends BASEPREFIX=`pwd`/depends
mkdir -p ${BASEPREFIX}/SDKs mkdir -p ${BASEPREFIX}/SDKs
tar -C ${BASEPREFIX}/SDKs -xf ${BUILD_DIR}/MacOSX10.11.sdk.tar.gz tar -C ${BASEPREFIX}/SDKs -xf ${BUILD_DIR}/MacOSX10.14.sdk.tar.gz
# Build dependencies for each host # Build dependencies for each host
for i in $HOSTS; do for i in $HOSTS; do

View File

@ -1,15 +1,136 @@
### MacDeploy ### # MacOS Deployment
For Snow Leopard (which uses [Python 2.6](http://www.python.org/download/releases/2.6/)), you will need the param_parser package: The `macdeployqtplus` script should not be run manually. Instead, after building as usual:
sudo easy_install argparse
This script should not be run manually, instead, after building as usual:
```bash
make deploy make deploy
```
During the process, the disk image window will pop up briefly where the fancy During the deployment process, the disk image window will pop up briefly
settings are applied. This is normal, please do not interfere. when the fancy settings are applied. This is normal, please do not interfere,
the process will unmount the DMG and cleanup before finishing.
When finished, it will produce `Dash-Core.dmg`. When complete, it will have produced `Dash-Qt.dmg`.
## SDK Extraction
`Xcode.app` is packaged in a `.xip` archive.
This makes the SDK less-trivial to extract on non-macOS machines.
One approach (tested on Debian Buster) is outlined below:
```bash
apt install clang cpio git liblzma-dev libxml2-dev libssl-dev make
git clone https://github.com/tpoechtrager/xar
pushd xar/xar
./configure
make
make install
popd
git clone https://github.com/NiklasRosenstein/pbzx
pushd pbzx
clang -llzma -lxar pbzx.c -o pbzx -Wl,-rpath=/usr/local/lib
popd
xar -xf Xcode_10.2.1.xip -C .
./pbzx/pbzx -n Content | cpio -i
find Xcode.app -type d -name MacOSX.sdk -execdir sh -c 'tar -c MacOSX.sdk/ | gzip -9n > /MacOSX10.14.sdk.tar.gz' \;
```
on macOS the process is more straightforward:
```bash
xip -x Xcode_10.2.1.xip
tar -C Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.14.sdk.tar.gz MacOSX.sdk
```
Our previously used macOS SDK (`MacOSX10.11.sdk`) can be extracted from
[Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg).
The script [`extract-osx-sdk.sh`](./extract-osx-sdk.sh) automates this. First
ensure the DMG file is in the current directory, and then run the script. You
may wish to delete the `intermediate 5.hfs` file and `MacOSX10.11.sdk` (the
directory) when you've confirmed the extraction succeeded.
```bash
apt-get install p7zip-full sleuthkit
contrib/macdeploy/extract-osx-sdk.sh
rm -rf 5.hfs MacOSX10.11.sdk
```
## Deterministic macOS DMG Notes
Working macOS DMGs are created in Linux by combining a recent `clang`, the Apple
`binutils` (`ld`, `ar`, etc) and DMG authoring tools.
Apple uses `clang` extensively for development and has upstreamed the necessary
functionality so that a vanilla clang can take advantage. It supports the use of `-F`,
`-target`, `-mmacosx-version-min`, and `--sysroot`, which are all necessary when
building for macOS.
Apple's version of `binutils` (called `cctools`) contains lots of functionality missing in the
FSF's `binutils`. In addition to extra linker options for frameworks and sysroots, several
other tools are needed as well such as `install_name_tool`, `lipo`, and `nmedit`. These
do not build under Linux, so they have been patched to do so. The work here was used as
a starting point: [mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4).
In order to build a working toolchain, the following source packages are needed from
Apple: `cctools`, `dyld`, and `ld64`.
These tools inject timestamps by default, which produce non-deterministic binaries. The
`ZERO_AR_DATE` environment variable is used to disable that.
This version of `cctools` has been patched to use the current version of `clang`'s headers
and its `libLTO.so` rather than those from `llvmgcc`, as it was originally done in `toolchain4`.
To complicate things further, all builds must target an Apple SDK. These SDKs are free to
download, but not redistributable. To obtain it, register for an Apple Developer Account,
then download [Xcode 10.2.1](https://download.developer.apple.com/Developer_Tools/Xcode_10.2.1/Xcode_10.2.1.xip).
This file is many gigabytes in size, but most (but not all) of what we need is
contained only in a single directory:
```bash
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
```
See the SDK Extraction notes above for how to obtain it.
The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries which are
created using these tools. The build process has been designed to avoid including the
SDK's files in Gitian's outputs. All interim tarballs are fully deterministic and may be freely
redistributed.
`genisoimage` is used to create the initial DMG. It is not deterministic as-is, so it has been
patched. A system `genisoimage` will work fine, but it will not be deterministic because
the file-order will change between invocations. The patch can be seen here: [cdrkit-deterministic.patch](https://github.com/bitcoin/bitcoin/blob/master/depends/patches/native_cdrkit/cdrkit-deterministic.patch).
No effort was made to fix this cleanly, so it likely leaks memory badly, however it's only used for
a single invocation, so that's no real concern.
`genisoimage` cannot compress DMGs, so afterwards, the DMG tool from the
`libdmg-hfsplus` project is used to compress it. There are several bugs in this tool and its
maintainer has seemingly abandoned the project.
The DMG tool has the ability to create DMGs from scratch as well, but this functionality is
broken. Only the compression feature is currently used. Ideally, the creation could be fixed
and `genisoimage` would no longer be necessary.
Background images and other features can be added to DMG files by inserting a
`.DS_Store` before creation. This is generated by the script `contrib/macdeploy/custom_dsstore.py`.
As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a requirement in
order to satisfy the new Gatekeeper requirements. Because this private key cannot be
shared, we'll have to be a bit creative in order for the build process to remain somewhat
deterministic. Here's how it works:
- Builders use Gitian to create an unsigned release. This outputs an unsigned DMG which
users may choose to bless and run. It also outputs an unsigned app structure in the form
of a tarball, which also contains all of the tools that have been previously (deterministically)
built in order to create a final DMG.
- The Apple keyholder uses this unsigned app to create a detached signature, using the
script that is also included there. Detached signatures are available.
- Builders feed the unsigned app + detached signature back into Gitian. It uses the
pre-built tools to recombine the pieces into a deterministic DMG.

View File

@ -8,7 +8,7 @@ set -e
INPUTFILE="Xcode_7.3.1.dmg" INPUTFILE="Xcode_7.3.1.dmg"
HFSFILENAME="5.hfs" HFSFILENAME="5.hfs"
SDKDIR="Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk" SDKDIR="Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk"
7z x "${INPUTFILE}" "${HFSFILENAME}" 7z x "${INPUTFILE}" "${HFSFILENAME}"
SDKNAME="$(basename "${SDKDIR}")" SDKNAME="$(basename "${SDKDIR}")"

View File

@ -1,9 +1,8 @@
OSX_MIN_VERSION=10.12 OSX_MIN_VERSION=10.12
OSX_SDK_VERSION=10.11 OSX_SDK_VERSION=10.14
OSX_SDK=$(SDK_PATH)/MacOSX$(OSX_SDK_VERSION).sdk OSX_SDK=$(SDK_PATH)/MacOSX$(OSX_SDK_VERSION).sdk
LD64_VERSION=253.9 darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK)
darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -mlinker-version=$(LD64_VERSION) darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++
darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -mlinker-version=$(LD64_VERSION) -stdlib=libc++
darwin_CFLAGS=-pipe darwin_CFLAGS=-pipe
darwin_CXXFLAGS=$(darwin_CFLAGS) darwin_CXXFLAGS=$(darwin_CFLAGS)

View File

@ -1,45 +1,55 @@
package=native_cctools package=native_cctools
$(package)_version=807d6fd1be5d2224872e381870c0a75387fe05e6 $(package)_version=3764b223c011574971ee3ae09ce968ba5dc2f00f
$(package)_download_path=https://github.com/theuni/cctools-port/archive $(package)_download_path=https://github.com/tpoechtrager/cctools-port/archive
$(package)_file_name=$($(package)_version).tar.gz $(package)_file_name=$($(package)_version).tar.gz
$(package)_sha256_hash=a09c9ba4684670a0375e42d9d67e7f12c1f62581a27f28f7c825d6d7032ccc6a $(package)_sha256_hash=3e35907bf376269a844df08e03cbb43e345c88125374f2228e03724b5f9a2a04
$(package)_build_subdir=cctools $(package)_build_subdir=cctools
$(package)_clang_version=3.7.1 $(package)_clang_version=6.0.1
$(package)_clang_download_path=http://llvm.org/releases/$($(package)_clang_version) $(package)_clang_download_path=https://releases.llvm.org/$($(package)_clang_version)
$(package)_clang_download_file=clang+llvm-$($(package)_clang_version)-x86_64-linux-gnu-ubuntu-14.04.tar.xz $(package)_clang_download_file=clang+llvm-$($(package)_clang_version)-x86_64-linux-gnu-ubuntu-14.04.tar.xz
$(package)_clang_file_name=clang-llvm-$($(package)_clang_version)-x86_64-linux-gnu-ubuntu-14.04.tar.xz $(package)_clang_file_name=clang-llvm-$($(package)_clang_version)-x86_64-linux-gnu-ubuntu-14.04.tar.xz
$(package)_clang_sha256_hash=99b28a6b48e793705228a390471991386daa33a9717cd9ca007fcdde69608fd9 $(package)_clang_sha256_hash=fa5416553ca94a8c071a27134c094a5fb736fe1bd0ecc5ef2d9bc02754e1bef0
$(package)_libtapi_version=3efb201881e7a76a21e0554906cf306432539cef
$(package)_libtapi_download_path=https://github.com/tpoechtrager/apple-libtapi/archive
$(package)_libtapi_download_file=$($(package)_libtapi_version).tar.gz
$(package)_libtapi_file_name=$($(package)_libtapi_version).tar.gz
$(package)_libtapi_sha256_hash=380c1ca37cfa04a8699d0887a8d3ee1ad27f3d08baba78887c73b09485c0fbd3
$(package)_extra_sources=$($(package)_clang_file_name) $(package)_extra_sources=$($(package)_clang_file_name)
$(package)_extra_sources += $($(package)_libtapi_file_name)
define $(package)_fetch_cmds define $(package)_fetch_cmds
$(call fetch_file,$(package),$($(package)_download_path),$($(package)_download_file),$($(package)_file_name),$($(package)_sha256_hash)) && \ $(call fetch_file,$(package),$($(package)_download_path),$($(package)_download_file),$($(package)_file_name),$($(package)_sha256_hash)) && \
$(call fetch_file,$(package),$($(package)_clang_download_path),$($(package)_clang_download_file),$($(package)_clang_file_name),$($(package)_clang_sha256_hash)) $(call fetch_file,$(package),$($(package)_clang_download_path),$($(package)_clang_download_file),$($(package)_clang_file_name),$($(package)_clang_sha256_hash)) && \
$(call fetch_file,$(package),$($(package)_libtapi_download_path),$($(package)_libtapi_download_file),$($(package)_libtapi_file_name),$($(package)_libtapi_sha256_hash))
endef endef
define $(package)_extract_cmds define $(package)_extract_cmds
mkdir -p $($(package)_extract_dir) && \ mkdir -p $($(package)_extract_dir) && \
echo "$($(package)_sha256_hash) $($(package)_source)" > $($(package)_extract_dir)/.$($(package)_file_name).hash && \ echo "$($(package)_sha256_hash) $($(package)_source)" > $($(package)_extract_dir)/.$($(package)_file_name).hash && \
echo "$($(package)_clang_sha256_hash) $($(package)_source_dir)/$($(package)_clang_file_name)" >> $($(package)_extract_dir)/.$($(package)_file_name).hash && \ echo "$($(package)_clang_sha256_hash) $($(package)_source_dir)/$($(package)_clang_file_name)" >> $($(package)_extract_dir)/.$($(package)_file_name).hash && \
echo "$($(package)_libtapi_sha256_hash) $($(package)_source_dir)/$($(package)_libtapi_file_name)" >> $($(package)_extract_dir)/.$($(package)_file_name).hash && \
$(build_SHA256SUM) -c $($(package)_extract_dir)/.$($(package)_file_name).hash && \ $(build_SHA256SUM) -c $($(package)_extract_dir)/.$($(package)_file_name).hash && \
mkdir -p toolchain/bin toolchain/lib/clang/3.5/include && \ mkdir -p toolchain/bin toolchain/lib/clang/$($(package)_clang_version)/include && \
mkdir -p libtapi && \
tar --no-same-owner --strip-components=1 -C libtapi -xf $($(package)_source_dir)/$($(package)_libtapi_file_name) && \
tar --no-same-owner --strip-components=1 -C toolchain -xf $($(package)_source_dir)/$($(package)_clang_file_name) && \ tar --no-same-owner --strip-components=1 -C toolchain -xf $($(package)_source_dir)/$($(package)_clang_file_name) && \
rm -f toolchain/lib/libc++abi.so* && \ rm -f toolchain/lib/libc++abi.so* && \
echo "#!/bin/sh" > toolchain/bin/$(host)-dsymutil && \
echo "exit 0" >> toolchain/bin/$(host)-dsymutil && \
chmod +x toolchain/bin/$(host)-dsymutil && \
tar --no-same-owner --strip-components=1 -xf $($(package)_source) tar --no-same-owner --strip-components=1 -xf $($(package)_source)
endef endef
define $(package)_set_vars define $(package)_set_vars
$(package)_config_opts=--target=$(host) --disable-lto-support $(package)_config_opts=--target=$(host) --disable-lto-support --with-libtapi=$($(package)_extract_dir)
$(package)_ldflags+=-Wl,-rpath=\\$$$$$$$$\$$$$$$$$ORIGIN/../lib $(package)_ldflags+=-Wl,-rpath=\\$$$$$$$$\$$$$$$$$ORIGIN/../lib
$(package)_cc=$($(package)_extract_dir)/toolchain/bin/clang $(package)_cc=$($(package)_extract_dir)/toolchain/bin/clang
$(package)_cxx=$($(package)_extract_dir)/toolchain/bin/clang++ $(package)_cxx=$($(package)_extract_dir)/toolchain/bin/clang++
endef endef
define $(package)_preprocess_cmds define $(package)_preprocess_cmds
cd $($(package)_build_subdir); ./autogen.sh && \ CC=$($(package)_cc) CXX=$($(package)_cxx) INSTALLPREFIX=$($(package)_extract_dir) ./libtapi/build.sh && \
sed -i.old "/define HAVE_PTHREADS/d" ld64/src/ld/InputFiles.h CC=$($(package)_cc) CXX=$($(package)_cxx) INSTALLPREFIX=$($(package)_extract_dir) ./libtapi/install.sh && \
sed -i.old "/define HAVE_PTHREADS/d" $($(package)_build_subdir)/ld64/src/ld/InputFiles.h
endef endef
define $(package)_config_cmds define $(package)_config_cmds
@ -52,6 +62,9 @@ endef
define $(package)_stage_cmds define $(package)_stage_cmds
$(MAKE) DESTDIR=$($(package)_staging_dir) install && \ $(MAKE) DESTDIR=$($(package)_staging_dir) install && \
mkdir -p $($(package)_staging_prefix_dir)/lib/ && \
cd $($(package)_extract_dir) && \
cp lib/libtapi.so.6 $($(package)_staging_prefix_dir)/lib/ && \
cd $($(package)_extract_dir)/toolchain && \ cd $($(package)_extract_dir)/toolchain && \
mkdir -p $($(package)_staging_prefix_dir)/lib/clang/$($(package)_clang_version)/include && \ mkdir -p $($(package)_staging_prefix_dir)/lib/clang/$($(package)_clang_version)/include && \
mkdir -p $($(package)_staging_prefix_dir)/bin $($(package)_staging_prefix_dir)/include && \ mkdir -p $($(package)_staging_prefix_dir)/bin $($(package)_staging_prefix_dir)/include && \

View File

@ -93,7 +93,6 @@ $(package)_config_opts_darwin += -device-option MAC_SDK_VERSION=$(OSX_SDK_VERSIO
$(package)_config_opts_darwin += -device-option CROSS_COMPILE="$(host)-" $(package)_config_opts_darwin += -device-option CROSS_COMPILE="$(host)-"
$(package)_config_opts_darwin += -device-option MAC_MIN_VERSION=$(OSX_MIN_VERSION) $(package)_config_opts_darwin += -device-option MAC_MIN_VERSION=$(OSX_MIN_VERSION)
$(package)_config_opts_darwin += -device-option MAC_TARGET=$(host) $(package)_config_opts_darwin += -device-option MAC_TARGET=$(host)
$(package)_config_opts_darwin += -device-option MAC_LD64_VERSION=$(LD64_VERSION)
endif endif
# for macOS on Apple Silicon (ARM) see https://bugreports.qt.io/browse/QTBUG-85279 # for macOS on Apple Silicon (ARM) see https://bugreports.qt.io/browse/QTBUG-85279

View File

@ -18,7 +18,7 @@ QMAKE_APPLE_DEVICE_ARCHS=x86_64
!host_build: QMAKE_CFLAGS += -target $${MAC_TARGET} !host_build: QMAKE_CFLAGS += -target $${MAC_TARGET}
!host_build: QMAKE_OBJECTIVE_CFLAGS += $$QMAKE_CFLAGS !host_build: QMAKE_OBJECTIVE_CFLAGS += $$QMAKE_CFLAGS
!host_build: QMAKE_CXXFLAGS += $$QMAKE_CFLAGS !host_build: QMAKE_CXXFLAGS += $$QMAKE_CFLAGS
!host_build: QMAKE_LFLAGS += -target $${MAC_TARGET} -mlinker-version=$${MAC_LD64_VERSION} !host_build: QMAKE_LFLAGS += -target $${MAC_TARGET}
QMAKE_AR = $${CROSS_COMPILE}ar cq QMAKE_AR = $${CROSS_COMPILE}ar cq
QMAKE_RANLIB=$${CROSS_COMPILE}ranlib QMAKE_RANLIB=$${CROSS_COMPILE}ranlib
QMAKE_LIBTOOL=$${CROSS_COMPILE}libtool QMAKE_LIBTOOL=$${CROSS_COMPILE}libtool

View File

@ -19,8 +19,8 @@ Additionally, the Mac OSX SDK must be downloaded and extracted manually:
```bash ```bash
$ mkdir -p depends/sdk-sources $ mkdir -p depends/sdk-sources
$ mkdir -p depends/SDKs $ mkdir -p depends/SDKs
$ curl https://bitcoincore.org/depends-sources/sdks/MacOSX10.11.sdk.tar.gz -o depends/sdk-sources/MacOSX10.11.sdk.tar.gz $ curl https://bitcoincore.org/depends-sources/sdks/MacOSX10.14.sdk.tar.gz -o depends/sdk-sources/MacOSX10.14.sdk.tar.gz
$ tar -C depends/SDKs -xf depends/sdk-sources/MacOSX10.11.sdk.tar.gz $ tar -C depends/SDKs -xf depends/sdk-sources/MacOSX10.14.sdk.tar.gz
``` ```
When building the dependencies, as described in [build-generic](build-generic.md), use When building the dependencies, as described in [build-generic](build-generic.md), use

View File

@ -69,49 +69,3 @@ Other commands:
./src/dashd -daemon # Starts the dash daemon. ./src/dashd -daemon # Starts the dash daemon.
./src/dash-cli --help # Outputs a list of command-line options. ./src/dash-cli --help # Outputs a list of command-line options.
./src/dash-cli help # Outputs a list of RPC commands when the daemon is running. ./src/dash-cli help # Outputs a list of RPC commands when the daemon is running.
Deterministic macOS DMG Notes.
Working macOS DMGs are created in Linux by combining a recent clang, the Apple binutils (ld, ar, etc) and DMG authoring tools.
Apple uses clang extensively for development and has upstreamed the necessary functionality so that a vanilla clang can take advantage. It supports the use of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary when building for macOS.
Apple's version of binutils (called cctools) contains lots of functionality missing in the FSF's binutils. In addition to extra linker options for frameworks and sysroots, several other tools are needed as well such as install_name_tool, lipo, and nmedit. These do not build under linux, so they have been patched to do so. The work here was used as a starting point: mingwandroid/toolchain4.
In order to build a working toolchain, the following source packages are needed from Apple: cctools, dyld, and ld64.
These tools inject timestamps by default, which produce non-deterministic binaries. The ZERO_AR_DATE environment variable is used to disable that.
This version of cctools has been patched to use the current version of clang's headers and its libLTO.so rather than those from llvmgcc, as it was originally done in toolchain4.
To complicate things further, all builds must target an Apple SDK. These SDKs are free to download, but not redistributable. To obtain it, register for a developer account, then download the Xcode 7.3.1 dmg.
This file is several gigabytes in size, but only a single directory inside is needed:
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file. To create a tarball suitable for Gitian input, there are two options:
Using macOS, you can mount the dmg, and then create it with:
$ hdiutil attach Xcode_7.3.1.dmg
$ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk
Alternatively, you can use 7zip and SleuthKit to extract the files one by one. The script contrib/macdeploy/extract-osx-sdk.sh automates this. First ensure the dmg file is in the current directory, and then run the script. You may wish to delete the intermediate 5.hfs file and MacOSX10.11.sdk (the directory) when you've confirmed the extraction succeeded.
apt-get install p7zip-full sleuthkit
contrib/macdeploy/extract-osx-sdk.sh
rm -rf 5.hfs MacOSX10.11.sdk
The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries which are created using these tools. The build process has been designed to avoid including the SDK's files in Gitian's outputs. All interim tarballs are fully deterministic and may be freely redistributed.
genisoimage is used to create the initial DMG. It is not deterministic as-is, so it has been patched. A system genisoimage will work fine, but it will not be deterministic because the file-order will change between invocations. The patch can be seen here: theuni/osx-cross-depends. No effort was made to fix this cleanly, so it likely leaks memory badly. But it's only used for a single invocation, so that's no real concern.
genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the libdmg-hfsplus project is used to compress it. There are several bugs in this tool and its maintainer has seemingly abandoned the project. It has been forked and is available (with fixes) here: theuni/libdmg-hfsplus.
The 'dmg' tool has the ability to create DMGs from scratch as well, but this functionality is broken. Only the compression feature is currently used. Ideally, the creation could be fixed and genisoimage would no longer be necessary.
Background images and other features can be added to DMG files by inserting a .DS_Store before creation. This is generated by the script contrib/macdeploy/custom_dsstore.py.
As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a requirement in order to satisfy the new Gatekeeper requirements. Because this private key cannot be shared, we'll have to be a bit creative in order for the build process to remain somewhat deterministic. Here's how it works:
Builders use Gitian to create an unsigned release. This outputs an unsigned dmg which users may choose to bless and run. It also outputs an unsigned app structure in the form of a tarball, which also contains all of the tools that have been previously (deterministically) built in order to create a final dmg.
The Apple keyholder uses this unsigned app to create a detached signature, using the script that is also included there. Detached signatures are available from this repository.
Builders feed the unsigned app + detached signature back into Gitian. It uses the pre-built tools to recombine the pieces into a deterministic dmg.