Build error (Lua 5.3, 5.4, OpenSSL 3.5)

[scossu]

Installing luaossl via luarocks, on Arch Linux, Lua 5.4, openssl 3.5.0:

 $ luarocks install --local luaossl
Installing https://luarocks.org/luaossl-20220711-0.src.rock

luaossl 20220711-0 depends on lua (5.4-1 provided by VM: success)
Applying patch config.h.diff...
Hunk 1 found at offset 2...
gcc -O2 -fPIC -I/usr/include -c src/openssl.c -o src/openssl.o -D_REENTRANT -D_THREAD_SAFE -DCOMPAT53_PREFIX=luaossl -D_GNU_SOURCE -I/usr/include -I/usr/include
src/openssl.c: In function ‘auxL_testoption’:
src/openssl.c:1176:27: error: initialization of ‘int (*)(void)’ from incompatible pointer type ‘int (*)(const char *, const char *)’ [-Wincompatible-pointer-types]
 1176 |         int (*optcmp)() = (nocase)?
      |                           ^
src/openssl.c:1186:26: error: too many arguments to function ‘optcmp’; expected 0, have 2
 1186 |                 if (0 == optcmp(optlist[i], optname))
      |                          ^~~~~~ ~~~~~~~~~~
src/openssl.c: At top level:
src/openssl.c:2835:70: error: initialization of ‘void * (*)(void)’ from incompatible pointer type ‘void * (*)(const SSL_CTX *, int)’ {aka ‘void * (*)(const struct ssl_ctx_st *, int)’} [-Wincompatible-pointer-types]
 2835 |         [EX_SSL_CTX_ALPN_SELECT_CB] = { CRYPTO_EX_INDEX_SSL_CTX, -1, &SSL_CTX_get_ex_data, &SSL_CTX_set_ex_data },
      |                                                                      ^
src/openssl.c:2835:70: note: (near initialization for ‘ex_type[0].get_ex_data’)
In file included from src/openssl.c:77:
/usr/include/openssl/ssl.h:2221:7: note: ‘SSL_CTX_get_ex_data’ declared here
 2221 | void *SSL_CTX_get_ex_data(const SSL_CTX *ssl, int idx);
      |       ^~~~~~~~~~~~~~~~~~~
src/openssl.c:2835:92: error: initialization of ‘int (*)(void)’ from incompatible pointer type ‘int (*)(SSL_CTX *, int,  void *)’ {aka ‘int (*)(struct ssl_ctx_st *, int,  void *)’} [-Wincompatible-pointer-types]
 2835 |         [EX_SSL_CTX_ALPN_SELECT_CB] = { CRYPTO_EX_INDEX_SSL_CTX, -1, &SSL_CTX_get_ex_data, &SSL_CTX_set_ex_data },
      |                                                                                            ^
src/openssl.c:2835:92: note: (near initialization for ‘ex_type[0].set_ex_data’)
/usr/include/openssl/ssl.h:2220:12: note: ‘SSL_CTX_set_ex_data’ declared here
 2220 | __owur int SSL_CTX_set_ex_data(SSL_CTX *ssl, int idx, void *data);
      |            ^~~~~~~~~~~~~~~~~~~
src/openssl.c:2836:76: error: initialization of ‘void * (*)(void)’ from incompatible pointer type ‘void * (*)(const SSL_CTX *, int)’ {aka ‘void * (*)(const struct ssl_ctx_st *, int)’} [-Wincompatible-pointer-types]
 2836 |         [EX_SSL_CTX_TLSEXT_SERVERNAME_CB] = { CRYPTO_EX_INDEX_SSL_CTX, -1, &SSL_CTX_get_ex_data, &SSL_CTX_set_ex_data },
      |                                                                            ^
src/openssl.c:2836:76: note: (near initialization for ‘ex_type[1].get_ex_data’)
/usr/include/openssl/ssl.h:2221:7: note: ‘SSL_CTX_get_ex_data’ declared here
 2221 | void *SSL_CTX_get_ex_data(const SSL_CTX *ssl, int idx);
      |       ^~~~~~~~~~~~~~~~~~~
src/openssl.c:2836:98: error: initialization of ‘int (*)(void)’ from incompatible pointer type ‘int (*)(SSL_CTX *, int,  void *)’ {aka ‘int (*)(struct ssl_ctx_st *, int,  void *)’} [-Wincompatible-pointer-types]
 2836 |         [EX_SSL_CTX_TLSEXT_SERVERNAME_CB] = { CRYPTO_EX_INDEX_SSL_CTX, -1, &SSL_CTX_get_ex_data, &SSL_CTX_set_ex_data },
      |                                                                                                  ^
src/openssl.c:2836:98: note: (near initialization for ‘ex_type[1].set_ex_data’)
/usr/include/openssl/ssl.h:2220:12: note: ‘SSL_CTX_set_ex_data’ declared here
 2220 | __owur int SSL_CTX_set_ex_data(SSL_CTX *ssl, int idx, void *data);
      |            ^~~~~~~~~~~~~~~~~~~
src/openssl.c:2837:79: error: initialization of ‘void * (*)(void)’ from incompatible pointer type ‘void * (*)(const SSL_CTX *, int)’ {aka ‘void * (*)(const struct ssl_ctx_st *, int)’} [-Wincompatible-pointer-types]
 2837 |         [EX_SSL_CTX_CUSTOM_EXTENSION_ADD_CB] = { CRYPTO_EX_INDEX_SSL_CTX, -1, &SSL_CTX_get_ex_data, &SSL_CTX_set_ex_data },
      |                                                                               ^
src/openssl.c:2837:79: note: (near initialization for ‘ex_type[2].get_ex_data’)
/usr/include/openssl/ssl.h:2221:7: note: ‘SSL_CTX_get_ex_data’ declared here
 2221 | void *SSL_CTX_get_ex_data(const SSL_CTX *ssl, int idx);
      |       ^~~~~~~~~~~~~~~~~~~
src/openssl.c:2837:101: error: initialization of ‘int (*)(void)’ from incompatible pointer type ‘int (*)(SSL_CTX *, int,  void *)’ {aka ‘int (*)(struct ssl_ctx_st *, int,  void *)’} [-Wincompatible-pointer-types]
 2837 |         [EX_SSL_CTX_CUSTOM_EXTENSION_ADD_CB] = { CRYPTO_EX_INDEX_SSL_CTX, -1, &SSL_CTX_get_ex_data, &SSL_CTX_set_ex_data },
      |                                                                                                     ^
src/openssl.c:2837:101: note: (near initialization for ‘ex_type[2].set_ex_data’)
/usr/include/openssl/ssl.h:2220:12: note: ‘SSL_CTX_set_ex_data’ declared here
 2220 | __owur int SSL_CTX_set_ex_data(SSL_CTX *ssl, int idx, void *data);
      |            ^~~~~~~~~~~~~~~~~~~
src/openssl.c:2838:81: error: initialization of ‘void * (*)(void)’ from incompatible pointer type ‘void * (*)(const SSL_CTX *, int)’ {aka ‘void * (*)(const struct ssl_ctx_st *, int)’} [-Wincompatible-pointer-types]
 2838 |         [EX_SSL_CTX_CUSTOM_EXTENSION_PARSE_CB] = { CRYPTO_EX_INDEX_SSL_CTX, -1, &SSL_CTX_get_ex_data, &SSL_CTX_set_ex_data },
      |                                                                                 ^
src/openssl.c:2838:81: note: (near initialization for ‘ex_type[3].get_ex_data’)
/usr/include/openssl/ssl.h:2221:7: note: ‘SSL_CTX_get_ex_data’ declared here
 2221 | void *SSL_CTX_get_ex_data(const SSL_CTX *ssl, int idx);
      |       ^~~~~~~~~~~~~~~~~~~
src/openssl.c:2838:103: error: initialization of ‘int (*)(void)’ from incompatible pointer type ‘int (*)(SSL_CTX *, int,  void *)’ {aka ‘int (*)(struct ssl_ctx_st *, int,  void *)’} [-Wincompatible-pointer-types]
 2838 |         [EX_SSL_CTX_CUSTOM_EXTENSION_PARSE_CB] = { CRYPTO_EX_INDEX_SSL_CTX, -1, &SSL_CTX_get_ex_data, &SSL_CTX_set_ex_data },
      |                                                                                                       ^
src/openssl.c:2838:103: note: (near initialization for ‘ex_type[3].set_ex_data’)
/usr/include/openssl/ssl.h:2220:12: note: ‘SSL_CTX_set_ex_data’ declared here
 2220 | __owur int SSL_CTX_set_ex_data(SSL_CTX *ssl, int idx, void *data);
      |            ^~~~~~~~~~~~~~~~~~~
src/openssl.c: In function ‘ex_getdata’:
src/openssl.c:2991:22: error: too many arguments to function ‘type->get_ex_data’; expected 0, have 2
 2991 |         if (!(data = type->get_ex_data(obj, type->index)))
      |                      ^~~~              ~~~
src/openssl.c:2832:17: note: declared here
 2832 |         void *(*get_ex_data)();
      |                 ^~~~~~~~~~~
src/openssl.c: In function ‘ex_setdata’:
src/openssl.c:3019:21: error: too many arguments to function ‘type->get_ex_data’; expected 0, have 2
 3019 |         if ((data = type->get_ex_data(obj, type->index)) && data->state) {
      |                     ^~~~              ~~~
src/openssl.c:2832:17: note: declared here
 2832 |         void *(*get_ex_data)();
      |                 ^~~~~~~~~~~
src/openssl.c:3029:22: error: too many arguments to function ‘type->set_ex_data’; expected 0, have 3
 3029 |                 if (!type->set_ex_data(obj, type->index, data))
      |                      ^~~~              ~~~

[...]

/usr/include/openssl/pem.h:395:37: note: expected ‘int (*)(const void *, unsigned char **)’ but argument is of type ‘int (*)(void)’
  395 | int PEM_ASN1_write_bio(i2d_of_void *i2d, const char *name, BIO *bp,
      |                        ~~~~~~~~~~~~~^~~
src/openssl.c: In function ‘de5_string_to_key’:
src/openssl.c:12977:9: warning: ‘DES_string_to_key’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
12977 |         DES_string_to_key(luaL_checkstring(L, 1), &key);
      |         ^~~~~~~~~~~~~~~~~
In file included from src/openssl.c:80:
/usr/include/openssl/des.h:193:28: note: declared here
  193 | OSSL_DEPRECATEDIN_3_0 void DES_string_to_key(const char *str, DES_cblock *key);
      |                            ^~~~~~~~~~~~~~~~~
src/openssl.c: In function ‘de5_set_odd_parity’:
src/openssl.c:12992:9: warning: ‘DES_set_odd_parity’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
12992 |         DES_set_odd_parity(&key);
      |         ^~~~~~~~~~~~~~~~~~
/usr/include/openssl/des.h:176:28: note: declared here
  176 | OSSL_DEPRECATEDIN_3_0 void DES_set_odd_parity(DES_cblock *key);
      |                            ^~~~~~~~~~~~~~~~~~

Error: Build error: Failed compiling object src/openssl.o

Also tried with Lua 5.3 + luarocks 2.3.0.

Is this an upstream OpenSSL library incompatibility?

[a-schaefers]

tried with luajit on archlinux which is OpenSSL 3.5.0 currently and luarocks compiled via luajit (lua = luajit on PATH) same thing, so fairly sure 5.1 gonna have same thing

also tried with openssl 1.1 package from the arch repo

commands tried:

export CRYPTO_DIR="/usr"
export CRYPTO_LIBDIR="/usr/lib"
export CRYPTO_INCDIR="/usr/include/openssl-1.1"
export PKG_CONFIG_PATH="/usr/lib/pkgconfig"

luarocks make --only-deps --tree=lua_modules project-0.1-1.rockspec \
  CRYPTO_INCDIR=/usr/include/openssl-1.1 \
  CRYPTO_LIBDIR=/usr/lib \
  OPENSSL_INCDIR=/usr/include/openssl-1.1 \
  OPENSSL_LIBDIR=/usr/lib

nope

export CRYPTO_DIR="/usr"
export CRYPTO_LIBDIR="/usr/lib"
export CRYPTO_INCDIR="/usr/include"
export PKG_CONFIG_PATH="/usr/lib/pkgconfig"

luarocks make --only-deps --tree=lua_modules project-0.1-1.rockspec \
  CRYPTO_INCDIR=/usr/include \
  CRYPTO_LIBDIR=/usr/lib \
  OPENSSL_INCDIR=/usr/include \
  OPENSSL_LIBDIR=/usr/lib

and nopers

[jprjr]

I think the core issue is the default C standard for GCC changed with the release of GCC 15.

GCC 15+ defaults to C23, and C23 removes the K&R style function declarations where an empty list of parameters means any number of parameters.

Throughout openssl.c there are function pointers with an empty list of parameters, for example

luaossl/src/openssl.c

Line 1178 in 8e9622c

int (*optcmp)() = (nocase)?

This declares a function pointer that returns an int. In K&R C and up through C17 - can take any number of arguments. But starting with C23 - this takes zero arguments (it's equivalent to declaring a function with void as the only parameter). My understanding is this change was done to bring C more in-line with C++ (where an empty list has always meant no parameters).

So then on the following lines - said function pointer gets used with arguments and this results in a compiler error, since it was just declared as taking zero arguments.

When you compile luaossl locally using Make - it compiles fine, because the GNUMakefile sets a compiler flag to set the standard to gnu99. But when you install with LuaRocks - no such flag is set, it defaults to whatever C standard the compiler is using.

So, one option is to figure out how to get luarocks to set the C standard. The other, probably better / more future-proof option, is to add parameters to these function pointer declarations.

[jprjr]

Doing some more digging - even if changes are made in luaossl - OpenSSL itself relies on some K&R-style C declarations in ocsp.h macros.

luarocks doesn't currently support setting a C standard, I'm going to open an issue with LuaRocks to see if that's a possibility.

One work-around is to override the CC environment variable to set the C standard. The following will make LuaRocks use similar flags to compiling with make:

CC="gcc -std=gnu99" luarocks install luaossl

[scossu]

One work-around is to override the CC environment variable to set the C standard. The following will make LuaRocks use similar flags to compiling with make:

CC="gcc -std=gnu99" luarocks install luaossl

Thanks @jprjr , this works for me, even though the deprecation warnings are still there (a separate issue related to OpenSSL, I suppose).

[daurnimator]

Doing some more digging - even if changes are made in luaossl - OpenSSL itself relies on some K&R-style C declarations in ocsp.h macros.

Indeed, looking at my local /usr/include/openssl/ocsp.h, it has:

    252 #  define PEM_write_bio_OCSP_RESPONSE(bp,o) \
    253     PEM_ASN1_write_bio((int (*)())i2d_OCSP_RESPONSE,PEM_STRING_OCSP_RESPONSE,\
    254                         bp,(char *)(o), NULL,NULL,0,NULL,NULL)

Which resulted in the error:

luaossl/src/openssl.c:12440:14: error: passing argument 1 of ‘PEM_ASN1_write_bio’ from incompatible pointer type [-Wincompatible-pointer-types]
12440 |         if (!PEM_write_bio_OCSP_RESPONSE(bio, resp))
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      |              |
      |              int (*)(void)
/usr/include/openssl/pem.h:395:37: note: expected ‘int (*)(const void *, unsigned char **)’ but argument is of type ‘int (*)(void)’
  395 | int PEM_ASN1_write_bio(i2d_of_void *i2d, const char *name, BIO *bp,
      |                        ~~~~~~~~~~~~~^~~

even though the deprecation warnings are still there (a separate issue related to OpenSSL, I suppose).

Annoyingly OpenSSL has deprecated a few things without providing replacements.
To complicate matters, some of the new openssl APIs are so significantly different from the old APIs that it's too hard to support both.
So to bring in new OpenSSL we're going to have to drop support for some old versions.

[daurnimator]

I've pushed fixes for the 2 occurences of this issue in luaossl itself (94860c5, d4a6fec)

But the issue remains due to OpenSSL itself/luarocks.

[jprjr]

Opened an issue with OpenSSL re: the semantics change in C23: openssl/openssl#27938

[jprjr]

Something I just learned in the luarocks issue I opened is the concept of build-time dependencies for custom build systems - one of which is https://github.com/osch/luarocks-build-extended

I made modifications to the current published rockspec for luaossl and it compiled - basically have it depend on luarocks-build-extended and add the needed CFLAGS to set the C standard version. Right now I just did it for Unix platforms - I'm not familiar enough with MSVC to know what flags to add, if any. They're probably not going to default to C23 anytime soon.

Attaching the modified rockspec but the changes are:

• add rockspec_format = "3.0" to the top-level variables
• add build_dependencies = { "luarocks-build-extended" } to top-level.
• change build type to extended
• under build -> platforms -> unix -> modules -> _openssl, add a variables table with CFLAG_EXTRAS = { "-std=gnu99" }

demo.rockspec.txt

[offray]

I was also unable to build luaossl on Arch Linux since months ago and I solved it just a couple of days. I reported the issue in the Milua web framework repository as luaossl was the dependency that I was unable to build in order to have Milua working on Arch (based) systems. But, maybe, the solution is relevant here and is related with installing it like this:

CFLAGS="-Wno-error=incompatible-pointer-types" luarocks install --local luaossl 

(For more details read the Milua issue above)

I hope this helps and thanks for the work on luaossl.

Cheers,

[daurnimator]

@offray note that on archlinux, luaossl is available in the repos. You can run e.g. sudo pacman -S lua-luaossl (or even lua-http)