Aarch64 build issues

[tipabu]

I recently noticed the ARM CI for one of my projects that uses ISA-L was failing during build with

  CC       erasure_code/aarch64/libisal_la-gf_nvect_dot_prod_sve.lo
erasure_code/aarch64/gf_nvect_dot_prod_sve.c:33:10: fatal error: arm_sve.h: No such file or directory
   33 | #include <arm_sve.h>
      |          ^~~~~~~~~~~

I eventually figured out that arm_sve.h is provided by gcc and seems to have been first included in 10.1; if nothing else, it seems we should update README.md which currently lists 4.7 as the minimum required version.

Looking at my CI, sure enough, it was using 9.4 -- the default provided by Ubuntu 20.04. OK, that's old enough to be out of standard support, fair enough. It's fairly easy for me to control base OS version (easier than controlling specific gcc version, anyway). When I moved to Ubuntu 22.04 (which is still in standard support, and provides gcc 11.4), building succeeds -- but running tests under valgrind (3.18.1) blows up the first time it tries to dlopen isa-l:

93 - {.with_args = test_create_and_destroy_backend}: isa_l_rs_vand (idx=0) ... --36727-- Warning: DWARF2 CFI reader: unhandled DW_OP_ opcode 0x92

valgrind: m_debuginfo/readdwarf.c:2761 (copy_convert_CfiExpr_tree): Assertion 'srcix >= 0 && srcix < VG_(sizeXA)(srcxa)' failed.

host stacktrace:
==36727==    at 0x58040FEC: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x58041143: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x580412AB: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x580C1617: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x580C17B3: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x580C6DC3: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x5807A887: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x5806FA37: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x5809EB37: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x580AAF43: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x5809AA63: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x5809636B: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x580987BF: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0x580DEC8F: ??? (in /usr/libexec/valgrind/memcheck-arm64-linux)
==36727==    by 0xFFFFFFFFFFFFFFFF: ???

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable syscall 222 (lwpid 36727)
==36727==    at 0x401D7AC: __mmap64 (/misc/../sysdeps/unix/sysv/linux/mmap64.c:58)
==36727==    by 0x401D7AC: mmap (/misc/../sysdeps/unix/sysv/linux/mmap64.c:46)
==36727==    by 0x40079CB: _dl_map_segments (/elf/./dl-map-segments.h:140)
==36727==    by 0x40079CB: _dl_map_object_from_fd (/elf/./elf/dl-load.c:1258)
==36727==    by 0x4008ACF: _dl_map_object (/elf/./elf/dl-load.c:2327)
==36727==    by 0x400C727: dl_open_worker_begin (/elf/./elf/dl-open.c:534)
==36727==    by 0x4A11FCB: _dl_catch_exception (/elf/./elf/dl-error-skeleton.c:208)
==36727==    by 0x400BEF7: dl_open_worker (/elf/./elf/dl-open.c:782)
==36727==    by 0x4A11FCB: _dl_catch_exception (/elf/./elf/dl-error-skeleton.c:208)
==36727==    by 0x400C2FB: _dl_open (/elf/./elf/dl-open.c:883)
==36727==    by 0x495C443: dlopen_doit (/dlfcn/./dlfcn/dlopen.c:56)
==36727==    by 0x4A11FCB: _dl_catch_exception (/elf/./elf/dl-error-skeleton.c:208)
==36727==    by 0x4A120A3: _dl_catch_error (/elf/./elf/dl-error-skeleton.c:227)
==36727==    by 0x495BEF7: _dlerror_run (/dlfcn/./dlfcn/dlerror.c:138)
==36727==    by 0x495C4E7: dlopen_implementation (/dlfcn/./dlfcn/dlopen.c:71)
==36727==    by 0x495C4E7: dlopen@@GLIBC_2.34 (/dlfcn/./dlfcn/dlopen.c:81)
==36727==    by 0x489260F: liberasurecode_backend_open (org/openstack/liberasurecode/src/erasurecode.c:138)
==36727==    by 0x489260F: liberasurecode_instance_create (org/openstack/liberasurecode/src/erasurecode.c:243)
==36727==    by 0x10CB1F: test_create_and_destroy_backend (org/openstack/liberasurecode/test/liberasurecode_test.c:541)
==36727==    by 0x10C3EB: main (org/openstack/liberasurecode/test/liberasurecode_test.c:2392)
client stack range: [0x1FFEFFD000 0x1FFF000FFF] client SP: 0x1FFEFFF810
valgrind stack range: [0x1002CB8000 0x1002DB7FFF] top usage: 19584 of 1048576

Moving all the way to Ubuntu 24.04 (gcc 13.3, valgrind 3.22.0), everything works. It's not clear to me whether it was the newer gcc or the newer valgrind that fixed the dlopen issue...

[tipabu]

Couple more piece of info:

• My CI is pulling from master; the break happened somewhere between Jan 23 and Feb 1, which makes sense given how gf_nvect_dot_prod_sve.c was introduced in 4ff13c4, merged as part of Optimize gf N-vect dot product SVE functions #367 on Jan 30.
• Rolling back to v2.31.1 while building on Ubuntu 22.04 does not cause the same valgrind failure in dlopen -- my tests all pass.

[pablodelara]

@liuqinfei could you take a look at this? The minimum gcc version will need to be bumped because of arm_sve.h (and possibly because of arm_sme.h). But there are also Valgrind issues to look into.

[tipabu]

I did some more digging -- looks like the second part is specifically a valgrind issue. I managed to take some gcc-10.2.1-built artifacts that would throw a similar error with valgrind 3.15.0, drop them on a more-recent OS, and have the same command running happily under valgrind 3.22.0.

[liuqinfei]

Yes to update the requirement for gcc version in README.md. gcc v10.1 or later.
I did not reproduce the second issue you mentioned. It works fine on my platform openeuler22.03LTS-SP3 Kunpeng-920. Please show the steps, thanks. @tipabu

[root@node1 raw]# yum list installed | grep valgrind
valgrind.aarch64 1:3.16.0-5.oe2203sp3 @update
valgrind-devel.aarch64 1:3.16.0-5.oe2203sp3 @update
[root@node1 raw]# gcc --version
gcc (GCC) 10.3.1

============================================================================
Testsuite summary for libisal 2.31.1
============================================================================
# TOTAL: 17
# PASS:  17
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

[tipabu]

Just repro'ed on a fresh ubuntu 22.04 VM; history looks like

    1  sudo apt update
    2  sudo apt install -y git zlib1g-dev build-essential autoconf automake libtool valgrind
    3  git clone https://github.com/intel/isa-l.git
    4  cd isa-l/
    5  ./autogen.sh && ./configure && make && sudo make install
    6  cd ..
    7  git clone https://github.com/openstack/liberasurecode.git
    8  cd liberasurecode/
    9  ./autogen.sh && ./configure && make && make valgrind-test
   10  history

[tipabu]

Even simpler -- skip liberasurecode; compile a trivial program instead:

vagrant@vagrant:~$ cat << EOF > test.c
#include <dlfcn.h>

int main() {
  void * isal = dlopen("libisal.so.2", RTLD_LAZY | RTLD_LOCAL);
  return 0;
}
EOF
vagrant@vagrant:~$ gcc test.c -ldl
vagrant@vagrant:~$ ./a.out  # works fine
vagrant@vagrant:~$ valgrind ./a.out  # blows up

[liuqinfei]

Even simpler -- skip liberasurecode; compile a trivial program instead:

vagrant@vagrant:~$ cat << EOF > test.c
#include <dlfcn.h>

int main() {
  void * isal = dlopen("libisal.so.2", RTLD_LAZY | RTLD_LOCAL);
  return 0;
}
EOF
vagrant@vagrant:~$ gcc test.c -ldl
vagrant@vagrant:~$ ./a.out  # works fine
vagrant@vagrant:~$ valgrind ./a.out  # blows up

However, i can not reproduce the issue you mentioned. I compared the two isa-l version, master and v2.31.1. The output is the same.
[root@node1 test]# gcc test.c -ldl
[root@node1 test]# ./a.out
[root@node1 test]# valgrind ./a.out
==1627026== Memcheck, a memory error detector
==1627026== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1627026== Using Valgrind-3.16.0 and LibVEX; rerun with -h for copyright info
==1627026== Command: ./a.out
==1627026==

valgrind: Fatal error at startup: a function redirection
valgrind: which is mandatory for this platform-tool combination
valgrind: cannot be set up. Details of the redirection are:
valgrind:
valgrind: A must-be-redirected function
valgrind: whose name matches the pattern: strlen
valgrind: in an object with soname matching: ld-linux-aarch64.so.1
valgrind: was not found whilst processing
valgrind: symbols from the object with soname: ld-linux-aarch64.so.1
valgrind:
valgrind: Possible fixes: (1, short term): install glibc's debuginfo
valgrind: package on this machine. (2, longer term): ask the packagers
valgrind: for your Linux distribution to please in future ship a non-
valgrind: stripped ld.so (or whatever the dynamic linker .so is called)
valgrind: that exports the above-named function using the standard
valgrind: calling conventions for this platform. The package you need
valgrind: to install for fix (1) is called
valgrind:
valgrind: On Debian, Ubuntu: libc6-dbg
valgrind: On SuSE, openSuSE, Fedora, RHEL: glibc-debuginfo
valgrind:
valgrind: Note that if you are debugging a 32 bit process on a
valgrind: 64 bit system, you will need a corresponding 32 bit debuginfo
valgrind: package (e.g. libc6-dbg:i386).
valgrind:
valgrind: Cannot continue -- exiting now. Sorry.

[root@node1 test]# valgrind --version
valgrind-3.16.0

[tipabu]

I see the same thing on openEuler 22.03 (LTS-SP3) -- but if I follow the advice given in the error message and dnf install glibc-debuginfo, then it blows up:

[root@66af20730159 /]# cat /etc/openEuler-release
openEuler release 22.03 (LTS-SP3)
[root@66af20730159 /]# valgrind ./a.out
==7258== Memcheck, a memory error detector
==7258== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==7258== Using Valgrind-3.16.0 and LibVEX; rerun with -h for copyright info
==7258== Command: ./a.out
==7258==
--7258-- Warning: DWARF2 CFI reader: unhandled DW_OP_ opcode 0x92

valgrind: m_debuginfo/readdwarf.c:2486 (copy_convert_CfiExpr_tree): Assertion 'srcix >= 0 && srcix < VG_(sizeXA)(srcxa)' failed.

host stacktrace:
==7258==    at 0x5803EF10: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x5803F063: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x5803F1D3: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x580BC927: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x580BCAC3: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x580C15CF: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x58078227: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x5806C5DF: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x5809A827: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x580A6943: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x58096C47: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x5809235B: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x580949C3: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0x580DC2FF: ??? (in /usr/lib64/valgrind/memcheck-arm64-linux)
==7258==    by 0xFFFFFFFFFFFFFFFF: ???

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable syscall 222 (lwpid 7258)
==7258==    at 0x401D7EC: __mmap64 (mmap64.c:59)
==7258==    by 0x401D7EC: mmap (mmap64.c:47)
==7258==    by 0x4006BAB: _dl_map_segments (dl-map-segments.h:145)
==7258==    by 0x4006BAB: _dl_map_object_from_fd (dl-load.c:1336)
==7258==    by 0x400A51B: _dl_map_object (dl-load.c:2407)
==7258==    by 0x400E573: dl_open_worker_begin (dl-open.c:533)
==7258==    by 0x49E0EA7: _dl_catch_exception (dl-error-skeleton.c:212)
==7258==    by 0x400DD97: dl_open_worker (dl-open.c:795)
==7258==    by 0x49E0EA7: _dl_catch_exception (dl-error-skeleton.c:212)
==7258==    by 0x400E167: _dl_open (dl-open.c:897)
==7258==    by 0x492E053: dlopen_doit (dlopen.c:56)
==7258==    by 0x49E0EA7: _dl_catch_exception (dl-error-skeleton.c:212)
==7258==    by 0x49E0F83: _dl_catch_error (dl-error-skeleton.c:231)
==7258==    by 0x492DB17: _dlerror_run (dlerror.c:138)
==7258==    by 0x492E12F: dlopen_implementation (dlopen.c:71)
==7258==    by 0x492E12F: dlopen@@GLIBC_2.34 (dlopen.c:81)
==7258==    by 0x40069B: main (in /a.out)
client stack range: [0x1FFEFFE000 0x1FFF000FFF] client SP: 0x1FFEFFF240
valgrind stack range: [0x1002CBC000 0x1002DBBFFF] top usage: 17664 of 1048576


Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.

If that doesn't help, please report this bug to: www.valgrind.org

In the bug report, send all the above text, the valgrind
version, and what OS and version you are using.  Thanks.

Again, if I roll back isa-l to the v2.31.1 tag (being careful to clear out any 2.32 stuff in /usr/lib before running make install), it doesn't blow up. 😕

[tipabu]

We might want to rename this, now that the arm_sve.h build issue is resolved. Or even close it, as it seems increasingly likely that the remaining issue is a valgrind issue and we can just advise people to upgrade. Still kinda weird that it only seems to come up with newer isa-l builds, though...