aarch64
yugabyte目前还不支持在aarch64架构下编译及使用，也没有提供相关的第三方编译好的依赖包。一切都需要自己在aarch64环境编译生成相应的包。
不过官方现在已经开始着手支持aarch64架构，可以后面期待一下。
https://github.com/yugabyte/yugabyte-db/issues/9154
系统环境
CentOS Linux release 7.6.1810 (AltArch)
Linux localhost 4.18.0-80.7.2.el7.aarch64 #1 SMP Thu Sep 12 16:13:20 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
下载源代码
使用最新仓库中的代码git clone https://github.com/yugabyte/yugabyte-db.git
安装系统依赖
sudo yum update
sudo yum groupinstall -y 'Development Tools'
sudo yum install -y ruby perl-Digest epel-release ccache git python2-pip python-devel python3 python3-pip python3-devel which
sudo yum install -y cmake3 ctest3 ninja-build
准备编译工具
sudo ln -s /usr/bin/cmake3 /usr/local/bin/cmake
sudo ln -s /usr/bin/ctest3 /usr/local/bin/ctest
准备第三方依赖
yugabyte编译所需要的linuxbrew包和thirdparty包，官方暂时还未提供aarch64架构下编译好的包，需要我们自己下载源代码编译。
linuxbrew
Linuxbrew是Linux下的Homebrew， 使用它，就可以用brew install安装更多的软件。
yugabyte使用linuxbrew toolchains编译时，会用到linuxbrew下的许多命令，而官方提供的包里的这些命令是在x86_64下编译的，在aarch64下并不能使用，所以需要我们下载官方源代码在aarch64下编译。
源码地址：
https://github.com/yugabyte/brew-build
但这里有一个问题，就是本身linuxbrew本身就并不支持aarch64架构，brew命令无法在aarch64架构下安装；而且linuxbrew并不提供大部分命令在aarch64下的安装包(也就是bottle)，所以无法使用brew install的方法来进行命令安装。
基于上述原因，yugabyte采用linuxbrew编译基本是不可能了，所以使用官方最新的native gcc或clang编译。
https://github.com/yugabyte/yugabyte-db/issues/8808
thirdparty
thirdparty提供了yugabyte编译所需要的第三方依赖，同样需要我们在aarch64下编译.
源码地址：
https://github.com/yugabyte/yugabyte-db-thirdparty
下载源代码
git clone https://github.com/yugabyte/yugabyte-db-thirdparty.git
编译
cd yugabyte-db-thirdparty
./build_thirdparty.sh
编译完成
编译完成后使用，使用installed目录去替换我们的/opt/yb_build/thirdparty下对应的installed目录
yugabyte-db编译
使用gcc9编译
./yb_build.sh release --gcc9
踩坑记录
因为yugabyte本身还不支持aarch64，所以编译过程会遇到许多问题，需要一一去解决。这里记录下遇到过的错误。
unwind-arch
错误信息：
Call Stack (most recent call first):
cmake_modules/YugabyteFindThirdParty.cmake:85 (ADD_THIRDPARTY_LIB)
CMakeLists.txt:671 (include)
解决：
# cmake_modules/FindLibUnwind.cmake
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^arm")
SET(LIBUNWIND_ARCH "arm")
elseif (CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "amd64")
SET(LIBUNWIND_ARCH "x86_64")
elseif (CMAKE_SYSTEM_PROCESSOR MATCHES "^i.86$")
SET(LIBUNWIND_ARCH "x86")
# add by tianv
elseif (CMAKE_SYSTEM_PROCESSOR STREQUAL "aarch64")
SET(LIBUNWIND_ARCH "aarch64")
endif()
编译选项
错误信息：
g++: error: unrecognized command line option ‘-msse4.2’
\-------------------------------------------------------------------------------
g++: error: unrecognized command line option ‘-mcx16’
g++: error: unrecognized command line option ‘-mno-avx’; did you mean ‘-Wno-abi’?
g++: error: unrecognized command line option ‘-mno-bmi’; did you mean ‘-Wno-abi’?
g++: error: unrecognized command line option ‘-mno-bmi2’; did you mean ‘-Wno-abi’?
g++: error: unrecognized command line option ‘-mno-fma’; did you mean ‘-Wno-hsa’?
g++: error: unrecognized command line option ‘-mno-abm’; did you mean ‘-Wno-abi’?
g++: error: unrecognized command line option ‘-mno-movbe’
g++: error: unrecognized command line option ‘-mno-fma’; did you mean ‘-Wno-hsa’?
cc1plus: error: unknown value ‘ivybridge’ for -march
cc1plus: note: valid arguments are: armv8-a armv8.1-a armv8.2-a armv8.3-a armv8.4-a native
解决
删掉x86特有的编译选项，将-march=ivybridge 修改为-march=armv8-a+crc
if (NOT APPLE)
# To enable 16-byte atomics support we should specify appropriate architecture.
- ADD_CXX_FLAGS("-march=ivybridge")
- ADD_CXX_FLAGS("-mcx16")
+ ADD_CXX_FLAGS("-march=armv8-a+crc")
atomicops
错误信息
src/yb/gutil/atomicops.h:102:2: error: #error You need to implement atomic operations for this architecture
#error You need to implement atomic operations for this architecture
^~~~~
解决
新增文件yb/gutil/atomicops-internals-aarch64.h, 然后在src/yb/gutil/atomicops.h中包含它：
#if defined(THREAD_SANITIZER)
#include "yb/gutil/atomicops-internals-tsan.h"
...
...
#elif defined(__aarch64__) || defined(__aarch64)
#include "yb/gutil/atomicops-internals-aarch64.h"
#else
#error You need to implement atomic operations for this architecture
#endif
cpu
错误信息
src/yb/gutil/cpu.cc:288:4: error: #error unknown architecture
#error unknown architecture
^~~~~
解决
// src/yb/gutil/cpu.cc
#elif defined(ARCH_CPU_ARM_FAMILY) && (defined(OS_ANDROID) || defined(__linux__))
cpu_brand_.assign(g_lazy_cpuinfo.Get().brand());
has_broken_neon_ = g_lazy_cpuinfo.Get().has_broken_neon();
#elif defined(__aarch64__)
cpu_brand_.assign("ARM64");
has_broken_neon_ = false;
#else
#error unknown architecture
#endif
CycleTimer
错误信息
src/yb/gutil/cycleclock-inl.h:214:2: error: #error You need to define CycleTimer for your O/S and CPU
#error You need to define CycleTimer for your O/S and CPU
^~~~~
解决
// src/yb/gutil/cycleclock-inl.h
#elif defined(__aarch64__)
// System timer of ARMv8 runs at a different frequency than the CPU's.
// The frequency is fixed, typically in the range 1-50MHz. It can be
// read at CNTFRQ special register. We assume the OS has set up
// the virtual timer properly.
inline int64 CycleClock::Now() {
int64 virtual_timer_value;
asm volatile("mrs %0, cntvct_el0" : "=r"(virtual_timer_value));
return virtual_timer_value;
}
spinlock
错误信息
src/yb/gutil/spinlock_linux-inl.h:74:19: error: ‘sys_futex’ was not declared in this scope
sys_futex(&x, FUTEX_WAKE, 1, 0) >= 0);
^~~~~~~~~
src/yb/gutil/spinlock_linux-inl.h:74:19: note: suggested alternative: ‘have_futex’
sys_futex(&x, FUTEX_WAKE, 1, 0) >= 0);
解决
arm linux 不支持sys_futex1系统调用，使用syscall替换。
if (have_futex && syscall(__NR_futex, &x, FUTEX_WAKE | futex_private_flag,
1, NULL, NULL, 0) < 0) {
futex_private_flag = 0;
}
NR_gettid
错误信息
src/yb/gutil/threading/thread_collision_warner.cc:54:18: error: ‘__NR_gettid’ was not declared in this scope
return syscall(__NR_gettid);
^~~~~~~~~~~
src/yb/gutil/threading/thread_collision_warner.cc:54:18: note: suggested alternative: ‘__getpgid’
return syscall(__NR_gettid);
^~~~~~~~~~~
__getpgi
解决
// src/yb/gutil/threading/thread_collision_warner.cc
static subtle::Atomic64 CurrentThread() {
#if defined(__APPLE__)
uint64_t tid;
CHECK_EQ(0, pthread_threadid_np(NULL, &tid));
return tid;
#elif defined(__aarch64__)
#include <unistd.h>
return getpid();
#elif defined(__linux__)
return syscall(__NR_gettid);
#endif
}
protoc-gen-insertions
错误信息
make[2]: *** [src/yb/util/CMakeFiles/gen_src_yb_util_version_info_proto.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
make[3]: *** [src/yb/util/histogram.pb.cc] Error 1
--insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
make[2]: *** [src/yb/util/CMakeFiles/gen_src_yb_util_histogram_proto.dir/all] Error 2
make[3]: *** [src/yb/fs/fs.pb.cc] Error 1
--insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
解决
https://github.com/yugabyte/yugabyte-db/issues/8928
增加--no-tcmalloc编译选项
./yb_build.sh release --gcc9 --no-tcmalloc
encryption_util
错误信息
src/yb/util/encryption_util.cc: In constructor ‘yb::enterprise::OpenSSLInitializer::OpenSSLInitializer()’:
src/yb/util/encryption_util.cc:201:44: error: statement has no effect [-Werror=unused-value]
CRYPTO_THREADID_set_callback(&ThreadId);
^
src/yb/util/encryption_util.cc: In destructor ‘yb::enterprise::OpenSSLInitializer::~OpenSSLInitializer()’:
src/yb/util/encryption_util.cc:206:42: error: statement has no effect [-Werror=unused-value]
CRYPTO_THREADID_set_callback(nullptr);
解决
因为thirdparty中的openssl包更新了版本，而openssl-1.0.x不再支持对应的函数。
class OpenSSLInitializer {
@@ -153,12 +155,16 @@ class OpenSSLInitializer {
crypto_mutexes.emplace_back(std::make_unique<std::mutex>());
}
CRYPTO_set_locking_callback(&LockingCallback);
+#if !defined(CRYPTO_THREADID_set_callback)
CRYPTO_THREADID_set_callback(&ThreadId);
+#endif
}
~OpenSSLInitializer() {
CRYPTO_set_locking_callback(nullptr);
+#if !defined(CRYPTO_THREADID_set_callback)
CRYPTO_THREADID_set_callback(nullptr);
+#endif
crc
错误信息
build/release-gcc-dynamic/src/yb/util/CMakeFiles/yb_util.dir/crc.cc.o: In function `yb::crc::InitCrc32cInstance()':
src/yb/util/crc.cc:50: undefined reference to `crcutil_interface::CRC::CreateCrc32c(bool, unsigned long, unsigned long, void const**)'
collect2: error: ld returned 1 exit status
解决
因为thirdparty包中的crcutil-440ba7babeff77ffad992df3a10c767f184e946e.tar.gz 不支持aarch64，用https://github.com/cloudera/crcutil.git
错误信息
/-------------------------------------------------------------------------------
| COMPILATION FAILED
|-------------------------------------------------------------------------------
src/postgres/src/port/pg_crc32c_armv8_choose.c: In function ‘pg_crc32c_armv8_available’:
src/postgres/src/port/pg_crc32c_armv8_choose.c:59:13: error: implicit declaration of function ‘pg_comp_crc32c_armv8’; did you mean ‘pg_comp_crc32c_sse42’? [-Werror=implicit-function-declaration]
result = (pg_comp_crc32c_armv8(0, &data, sizeof(data)) ==
^~~~~~~~~~~~~~~~~~~~
pg_comp_crc32c_sse42
src/postgres/src/port/pg_crc32c_armv8_choose.c:60:7: error: implicit declaration of function ‘pg_comp_crc32c_sb8’; did you mean ‘pg_comp_crc32c_sse42’? [-Werror=implicit-function-declaration]
pg_comp_crc32c_sb8(0, &data, sizeof(data)));
^~~~~~~~~~~~~~~~~~
pg_comp_crc32c_sse42
src/postgres/src/port/pg_crc32c_armv8_choose.c: In function ‘pg_comp_crc32c_choose’:
src/postgres/src/port/pg_crc32c_armv8_choose.c:88:3: error: ‘pg_comp_crc32c’ undeclared (first use in this function); did you mean ‘pg_crc32c’?
pg_comp_crc32c = pg_comp_crc32c_armv8;
^~~~~~~~~~~~~~
pg_crc32c
src/postgres/src/port/pg_crc32c_armv8_choose.c:88:3: note: each undeclared identifier is reported only once for each function it appears in
src/postgres/src/port/pg_crc32c_armv8_choose.c:88:20: error: ‘pg_comp_crc32c_armv8’ undeclared (first use in this function); did you mean ‘pg_comp_crc32c_sse42’?
pg_comp_crc32c = pg_comp_crc32c_armv8;
^~~~~~~~~~~~~~~~~~~~
pg_comp_crc32c_sse42
src/postgres/src/port/pg_crc32c_armv8_choose.c:90:20: error: ‘pg_comp_crc32c_sb8’ undeclared (first use in this function); did you mean ‘pg_comp_crc32c_sse42’?
pg_comp_crc32c = pg_comp_crc32c_sb8;
^~~~~~~~~~~~~~~~~~
pg_comp_crc32c_sse42
src/postgres/src/port/pg_crc32c_armv8_choose.c:92:9: error: implicit declaration of function ‘pg_comp_crc32c’; did you mean ‘pg_comp_crc32c_sse42’? [-Werror=implicit-function-declaration]
return pg_comp_crc32c(crc, data, len);
^~~~~~~~~~~~~~
pg_comp_crc32c_sse42
cc1: all warnings being treated as errors
Input files:
build/release-gcc-dynamic-ninja/postgres_build/src/port/pg_crc32c_armv8_choose.c
Output file (from -o): pg_crc32c_armv8_choose.o
\-------------------------------------------------
解决
# CMakeLists.txt
if (NOT APPLE)
# To enable 16-byte atomics support we should specify appropriate architecture.
- ADD_CXX_FLAGS("-march=ivybridge")
- ADD_CXX_FLAGS("-mcx16")
+ ADD_CXX_FLAGS("-march=armv8-a+crc")
错误信息
/-------------------------------------------------------------------------------
| COMPILATION FAILED
|-------------------------------------------------------------------------------
/opt/rh/devtoolset-9/root/usr/libexec/gcc/aarch64-redhat-linux/9/ld: build/debug-gcc9-dynamic/postgres_build/src/backend/access/transam/twophase.o: in function `ReadTwoPhaseFile':
/home/yugabyte-db/src/postgres/src/backend/access/transam/../../../../../../../src/postgres/src/backend/access/transam/twophase.c:1310: undefined reference to `pg_comp_crc32c_sse42'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/aarch64-redhat-linux/9/ld: build/debug-gcc9-dynamic/postgres_build/src/backend/access/transam/twophase.o: in function `RecreateTwoPhaseFile':
/home/yugabyte-db/src/postgres/src/backend/access/transam/../../../../../../../src/postgres/src/backend/access/transam/twophase.c:1655: undefined reference to `pg_comp_crc32c_sse42'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/aarch64-redhat-linux/9/ld: build/debug-gcc9-dynamic/postgres_build/src/backend/access/transam/xlog.o: in function `WriteControlFile':
/home/yugabyte-db/src/postgres/src/backend/access/transam/../../../../../../../src/postgres/src/backend/access/transam/xlog.c:4473: undefined reference to `pg_comp_crc32c_sse42'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/aarch64-redhat-linux/9/ld: build/debug-gcc9-dynamic/postgres_build/src/backend/access/transam/xlog.o: in function `UpdateControlFile':
/home/yugabyte-db/src/postgres/src/backend/access/transam/../../../../../../../src/postgres/src/backend/access/transam/xlog.c:4737: undefined reference to `pg_comp_crc32c_sse42'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/aarch64-redhat-linux/9/ld: build/debug-gcc9-dynamic/postgres_build/src/backend/access/transam/xlog.o: in function `ReadControlFile':
/home/yugabyte-db/src/postgres/src/backend/access/transam/../../../../../../../src/postgres/src/backend/access/transam/xlog.c:4583: undefined reference to `pg_comp_crc32c_sse42'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/aarch64-redhat-linux/9/ld: build/debug-gcc9-dynamic/postgres_build/src/backend/access/transam/xlog.o:/home/yugabyte-db/src/postgres/src/backend/access/transam/../../../../../../../src/postgres/src/backend/access/transam/xlog.c:5176: more undefined references to `pg_comp_crc32c_sse42' follow
collect2: error: ld returned 1 exit status
解决
删掉'-DUSE_SSE42_CRC32C=1',
--- a/python/yb/build_postgres.py
+++ b/python/yb/build_postgres.py
@@ -262,7 +262,6 @@ class PostgresBuilder(YbBuildToolBase):
'-Wimplicit-function-declaration',
'-Wno-error=unused-function',
'-DHAVE__BUILTIN_CONSTANT_P=1',
- '-DUSE_SSE42_CRC32C=1',
'-std=c11',
'-Werror=implicit-function-declaration',
'-Werror=int-conversion',
atomic
错误信息
build/release-gcc-dynamic/src/yb/util/CMakeFiles/yb_util.dir/physical_time.cc.o: In function boost::atomics::detail::emulated_operations<16ul, false>::store(unsigned __int128 volatile&, unsigned __int128, boost::memory_order)':
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210210192532-45c97f45f1-centos7-linuxbrew/installed/uninstrumented/include/boost/atomic/detail/ops_emulated.hpp:47: undefined reference toboost::atomics::detail::lockpool::scoped_lock::scoped_lock(void const volatile)'
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210210192532-45c97f45f1-centos7-linuxbrew/installed/uninstrumented/include/boost/atomic/detail/ops_emulated.hpp:47: undefined reference to boost::atomics::detail::lockpool::scoped_lock::~scoped_lock()'
build/release-gcc-dynamic/src/yb/util/CMakeFiles/yb_util.dir/physical_time.cc.o: In functionboost::atomics::detail::emulated_operations<16ul, false>::load(unsigned __int128 const volatile&, boost::memory_order)':
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210210192532-45c97f45f1-centos7-linuxbrew/installed/uninstrumented/include/boost/atomic/detail/ops_emulated.hpp:53: undefined reference to boost::atomics::detail::lockpool::scoped_lock::scoped_lock(void const volatile*)'
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210210192532-45c97f45f1-centos7-linuxbrew/installed/uninstrumented/include/boost/atomic/detail/ops_emulated.hpp:53: undefined reference toboost::atomics::detail::lockpool::scoped_lock::~scoped_lock()'
build/release-gcc-dynamic/src/yb/util/CMakeFiles/yb_util.dir/debug-util.cc.o: In function boost::atomics::detail::emulated_operations<16ul, false>::load(unsigned __int128 const volatile&, boost::memory_order)':
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210210192532-45c97f45f1-centos7-linuxbrew/installed/uninstrumented/include/boost/atomic/detail/ops_emulated.hpp:53: undefined reference toboost::atomics::detail::lockpool::scoped_lock::scoped_lock(void const volatile)'
/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210210192532-45c97f45f1-centos7-linuxbrew/installed/uninstrumented/include/boost/atomic/detail/ops_emulated.hpp:53: undefined reference to boost::atomics::detail::lockpool::scoped_lock::~scoped_lock()'
build/release-gcc-dynamic/src/yb/util/CMakeFiles/yb_util.dir/debug-util.cc.o: In functionboost::atomics::detail::emulated_operations<16ul, false>::compare_exchange_weak(unsigned __int128 volatile&, unsigned __int128&, unsigned __int128, boost::memory_order, boost::memory_order)':
解决
在src/yb/util/CMakeLists.txt 增加boost_atomic
set(UTIL_LIBS
boost_system
boost_thread
boost_atomic
cds
icui18n
icuuc
crcutil
crypt_blowfish
gflags
glog
gutil
histogram_proto
libev
pb_util_proto
protobuf
version_info_proto
encryption_proto
zlib
${OPENSSL_CRYPTO_LIBRARY}
${OPENSSL_SSL_LIBRARY})
在cmake_modules/YugabyteFindThirdParty.cmake 增加atomic
# Find Boost static libraries.
set(Boost_USE_STATIC_LIBS ON)
find_package(Boost COMPONENTS system thread atomic REQUIRED)
show_found_boost_details("static")
nmmintrin
错误信息：
src/yb/common/key_encoder.h:37:10: fatal error: nmmintrin.h: No such file or directory
#include <nmmintrin.h>
解决
// src/yb/common/key_encoder.h
#ifndef __aarch64__
#include <emmintrin.h>
#include <smmintrin.h>
#endif
...
...
...
static bool SSEEncodeChunk(const uint8_t** srcp, uint8_t** dstp) {
#ifdef __aarch64__
return false;
#else
...
...
...
#endif //__aarch64__
}
is_lock_free
https://github.com/yugabyte/yugabyte-db-thirdparty/issues/75
[email protected] atomic/platform.hpp 不支持aarch64，导致is_lock_free函数总是返回失败。
暂时先将其注释掉
src/yb/consensus/replica_state.cc:92: CHECK(leader_state_cache_.is_lock_free());
src/yb/util/lockfree.h:103: CHECK(head_.is_lock_free());
src/yb/tserver/tserver_shared_mem.h:36: LOG_IF(FATAL, !catalog_version_.is_lock_free())
src/yb/tablet/operations/operation_driver.cc:92: DCHECK(op_id_copy_.is_lock_free());
编译完成
[root@localhost yugabyte-db]# cd build
[root@localhost build]# ls
latest latest_build_root release-gcc9-dynamic venv yb2.7 yb2.7.tar.gz yugabyte-bash-common