From b98fb6063599fa0c33ac4f8b1ca836b957f2ea96 Mon Sep 17 00:00:00 2001 From: Matthias Schiffer Date: Sat, 26 Apr 2025 22:37:16 +0200 Subject: [PATCH] perl: fix parallel build race condition in target build We have received reports of builds of perl occasionally failing when building with many parallel jobs, with a log like the following: LD_LIBRARY_PATH=[...]/perl/perl-5.40.0 ./miniperl -Ilib make_ext.pl \ dist/constant/pm_to_blib MAKE="make" LIBPERL_A=libperl.so File/Path.pm did not return a true value at [...]/hostpkg/usr/lib/perl5/5.40.0/ExtUtils/MakeMaker.pm line 13. BEGIN failed--compilation aborted at [...]/hostpkg/usr/lib/perl5/5.40.0/ExtUtils/MakeMaker.pm line 13. Compilation failed in require at Makefile.PL line 3. BEGIN failed--compilation aborted at Makefile.PL line 3. Unsuccessful Makefile.PL(dist/constant): code=65280 at make_ext.pl line 532. The failing extension (dist/constant in the above log) would differ between runs. The cause of the issue is the `-Ilib` in the command line of miniperl. In the host build, `./miniperl -I lib` will use the following include path: [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/AutoLoader/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/Carp/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/PathTools [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/PathTools/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/ExtUtils-Install/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/ExtUtils-MakeMaker/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/ExtUtils-Manifest/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/File-Path/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/ext/re [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/Term-ReadLine/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/Exporter/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/ext/File-Find/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/Text-Tabs/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/dist/constant/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/version/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/Getopt-Long/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/Text-ParseWords/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/cpan/ExtUtils-PL2Bat/lib [..]/build_dir/hostpkg/perl/perl-5.40.0/lib . Various dependencies of the extension build scripts (Makefile.PL) - including File-Path, which failed to be loaded in the error log - are included in the path by buildcustomize.pl, as these extensions are only installed to `lib` as the build proceeds. However, in a target build, miniperl is just a symlink to the previously built host perl. As the host perl does not implicitly load `buildcustomize.pl`, we get the following include path for `./miniperl -Ilib`: lib [..]/staging_dir/hostpkg/usr/lib/perl5/site_perl/5.40.0/x86_64-linux [..]/staging_dir/hostpkg/usr/lib/perl5/site_perl/5.40.0 [..]/staging_dir/hostpkg/usr/lib/perl5/5.40.0/x86_64-linux [..]/staging_dir/hostpkg/usr/lib/perl5/5.40.0 The host perl's install location is used as the default include path which provides File-Path etc. for the target build; however, as more and more libraries get installed into `lib` during the extension build, they may get loaded from there instead, as `lib` is at the beginning of the include path. When multiple extensions are built in parallel, a Makefile.PL may attempt to load File/Path from `lib` after the file has been created, but before its contents have been written fully, resulting in the build to fail. In fact, we should not load anything from `lib` during the target build, as it is the staging directory for the target, including native extensions built for the target architecture - with one exception: The build scripts expect to find target information in the `Config` module, so simply removing `lib` from the include path completely would break the build. Solve the issue by creating an alternative lib directory `lib_build`, symlinking `Config.pm` and its dependencies in it, and replacing the `-Ilib` argument with `-Ilib_build` using a wrapper script around the host perl executable. This is similar to the approach seen in perl's own obsolete/broken cross compile scripts (`Cross/Makefile`). Signed-off-by: Matthias Schiffer --- lang/perl/Makefile | 15 +++++++++++++++ lang/perl/files/base.config | 2 +- lang/perl/files/hostperl-wrapper | 14 ++++++++++++++ 3 files changed, 30 insertions(+), 1 deletion(-) create mode 100644 lang/perl/files/hostperl-wrapper diff --git a/lang/perl/Makefile b/lang/perl/Makefile index 6a6dd5ea86..6ad27818e1 100644 --- a/lang/perl/Makefile +++ b/lang/perl/Makefile @@ -92,6 +92,21 @@ endef # Target perl define Build/Configure + # We don't want to pass -Ilib to host perl in the target build (as lib + # contains the target libraries, and files may currently be written + # while being imported in parallel builds). We do however need the + # target versions of the Config modules at the beginning of the include + # path for the build scripts' use. + # + # Create an alternative lib_build directory that will be added to the + # include path instead of lib (using hostperl-wrapper), containing only + # the config modules. + $(INSTALL_DIR) $(PKG_BUILD_DIR)/lib_build + ln -sf ../lib/Config.pm ../lib/Config_heavy.pl ../lib/Config_git.pl $(PKG_BUILD_DIR)/lib_build/ + + install -m0755 files/hostperl-wrapper $(PKG_BUILD_DIR)/hostperl-wrapper + sed -i "s'@HOST_PERL@'$(HOST_PERL_PREFIX)/bin/perl'g" $(PKG_BUILD_DIR)/hostperl-wrapper + $(PERL_CMD) files/perlconfig.pl -Dowrt:target_cc='$(TARGET_CC)' \ -Dowrt:gccversion=$(CONFIG_GCC_VERSION) \ -Dowrt:target_cross='$(TARGET_CROSS)' \ diff --git a/lang/perl/files/base.config b/lang/perl/files/base.config index 7d8b88b200..1cadfc10ac 100644 --- a/lang/perl/files/base.config +++ b/lang/perl/files/base.config @@ -650,7 +650,7 @@ hint='recommended' hostcat='cat /etc/hosts' hostgenerate="$owrt:host_perl_prefix/bin/generate_uudmap" hostosname='' -hostperl="$owrt:host_perl_prefix/bin/perl" +hostperl="./hostperl-wrapper" html1dir=' ' html1direxp='' html3dir=' ' diff --git a/lang/perl/files/hostperl-wrapper b/lang/perl/files/hostperl-wrapper new file mode 100644 index 0000000000..e31ef1cf8f --- /dev/null +++ b/lang/perl/files/hostperl-wrapper @@ -0,0 +1,14 @@ +#!@HOST_PERL@ + +foreach (@ARGV) { + # Stop parsing options if we encounter a non-option argument or -- + last if $_ eq '--' || $_ !~ m/^-/; + + # Modify first option of the form -Ilib, -I../lib, ... to refer to lib_build instead + if ($_ =~ m@-I(.*/)?lib@) { + $_ .= '_build'; + last; + } +} + +exec '@HOST_PERL@', @ARGV