Repository navigation
Can linuxdeploy support a cross-compilation environment ? #258
Description
After an hour of trying, I couldn't figure out how to connect to Libera.Chat for the very first time and was unable to register a username/password so my fallback is to post an issue here.
I want to use linuxdeploy in a cross-compilation environment where my build server is a Ubuntu 20.04.5 LTS and my target server is a CCLinux 3.x (crosscontrol.com v700 / v1200) which is a variant build by CrossControl on top of Yocto Poky Kirkstone Linux.
Linux v700 5.4.47-2.2.0+g5ec03d06f54e #1 SMP PREEMPT Tue Feb 23 11:36:20 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
I'm also approaching CrossControl support with this question.
I'd like to be able to create AppImages so I can do a single-file deploy without having to try to figure out installing all the dependencies an app may have.
I'm successfully using your tool to deploy AppImages to another target, this time a Linux Mint Victoria server (based on Ubuntu 22.04.2 LTS Jammy Jellyfish) but since these are both x86_64 platforms it's an easier job since there is no cross-compilation needed.
My build VM
ccs@ccs-VirtualBox:/Qt/Projects/remotedesktop$ uname -a20.04.1-Ubuntu SMP Wed Jan 25 09:41:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Linux ccs-VirtualBox 5.15.0-60-generic #66
My Mint target
ccs@ccs-VirtualBox:/Qt/Projects/remotedesktop$ ssh vcu$ uname -a
vcuadmin@vcuproject2030:
Linux vcuproject2030 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
vcuadmin@vcuproject2030:~$ ^C
My CrossControl v700 target based on https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-processors/i-mx-8-applications-processors:IMX8-SERIES
Linux v700 5.4.47-2.2.0+g5ec03d06f54e #1 SMP PREEMPT Tue Feb 23 11:36:20 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
ccs@v700:~/Downloads$ cat /proc/cpuinfo
processor : 0
BogoMIPS : 16.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd04
CPU revision : 2
processor : 1
BogoMIPS : 16.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd04
CPU revision : 2
Activity
NeroBurner commentedon Nov 2, 2023
TheAssassin commentedon Dec 5, 2023
I gave up my efforts on cross compiling a while ago as using QEMU and Docker produce less of a headache usually.
Please post a log. I suspect that to make your use case work, you'd need to be able to specify a compatible
lddbinary.An ARM version of linuxdeploy will be available soon, though.
mdear commentedon Dec 13, 2023
I'm cross-compiling on an x86_64 build VM for an i.MX8 cortexa-35-poky-linux platform.
I've decided that AppImage for production images makes sense for our project and I'm looking for support for this use case.
Our production environment is simple and our boards (CrossControl v700/v1200) are resource-constrained and I have no plans to run Docker or QEMU emulation as speed really matters (automotive application requiring real-time responsiveness).
mdear commentedon Dec 13, 2023
I've also posted this question in the AppImage forum, FYI :
https://github.com/orgs/AppImage/discussions/1303
As AppImage is, as I understand it, a dependency of linuxdeploy I imagine this support must be extended by them as well.
Any idea on the roadmap for this feature? Any workarounds to getting this working roughly? I just finished cross-compiling a test executable and I'm eager to package it and try it out on our target.
CrossControl says they don't have a version of ldd that runs on x86_64 against a i.MX 8 ARM binary/shared library. Next step is for me to take a look at more commonly available cross-compilation toolchains. PS. I already have some of the "ldd" functionality using "readelf -d " or "objjdump -p " and I grep for "NEEDED" to display missing libraries.
TheAssassin commentedon Dec 13, 2023
I just looked up your platform. It's just another ARMv8 (aarch64) board. All the AppImage tools that provide such builds should run fine there, there is no reason why not.
At the moment, cross compilation is not supported in linuxdeploy, though. You can run the tool in a chroot, Docker container or whatever other QEMU binfmt_misc enabled environment. A "cross build" mode is not available; you'd need to convince the tool of running some
lddequivalent that traces the libraries properly. Same goes for all the input plugins (e.g., linuxdeploy-plugin-qt). Contributions welcome. As a workaround, you might run just linuxdeploy itself with QEMU in a suitable Docker container.Building the final AppImage can be done on foreign architectures easily. appimagetool and also linuxdeploy-plugin-appimage (which uses the former internally) both accept a runtime parameter in which you can specify any runtime for any architecture. This means you can use, e.g., linuxdeploy-plugin-appimage for x86_64 with an aarch64 AppImage runtime.
I also maintain most of AppImage toolchain, so we can collaborate on adding whatever is necessary. On the AppImage side, I suppose no work needs to be done. You should test some generic AArch64 AppImage on your boards, though, just to make sure the runtime itself works.
TheAssassin commentedon Dec 13, 2023
Given you run on some embedded platform, do you even need the single-file-bundle feature? Terminology wise, we need to separate AppImages from their payload, the so called AppDirs. AppDirs are basically portable bundles. All the AppImage format adds to that is that they are then shipped as single files.
linuxdeploy calls itself an AppDir build tool, i.e., it creates those portable AppDir bundles. AppImage is just one of the possible output formats.
I'm not saying you shouldn't use AppImages, though. What a use case! AppImages running in cars!
mdear commentedon Dec 13, 2023
lol! I really appreciate the replies, @TheAssassin . I'm a new transplant from spending decades in the telecommunications industry (where single-file deployment is commonplace for networking devices such as switches and routers), to the embedded vehicle space. I seek to do most of the hard work once, on a cross-compilation build machine, and only then push out single-file deployments to multiple targets. The idea is to make the deployments cookie-cutter and to remove the possibility of human error. As long as the latency of AppImage unpacking is still reasonably time-efficient (as cold reboot to in-service latency is important to keep as low as possible) then I think there are advantages in using this strategy.
I'm more than happy to contribute pull requests, I just need a pointer to get started (I imagine I wouldn't run linuxdeploy and its' plugins and appimage as AppImage binaries but probably in a developers mode). I'm thinking of using as a workaround "readelf -d | grep NEEDED" or "objdump -p " run on x86_64 cross compilation VM to suss out the dependency web, since I don't appear to have a version of "ldd" that runs on x86_64 against an ARM binary.
PS. I've also posted on the NXP community to see if a version of "ldd" can be made available:
https://community.nxp.com/t5/i-MX-Processors/Looking-for-quot-ldd-quot-that-runs-on-x86-64-platform-against/m-p/1773463#M217246
mdear commentedon Dec 13, 2023
I'm also expecting that by running my AppImage on my ARM platform that it may unpack its embedded file system to a well-known persistent location (other than /tmp) so that subsequent runs are more performant. I imagine it may be possible to configure this kind of deployment.
TheAssassin commentedon Dec 13, 2023
The dependency lookup algorithm based on
lddis very much limited, but as long as the tool is used in the classic scenario, it works exceptionally well. We can (and probably should) replace this eventually.lddis relatively foolproof since it asks the loader what it needs to load, after all; areadelfapproach does not compare symbols for instance and will be less reliable.I think the strategy here must be to either provide an ldd (output) compatible script for your specific use case or implement alternative methods in the C++ code. The former seems more straightforward at first; a simple shell or Python script that prints output similar to
lddshould actually do the trick along with an environment variableLINUXDEPLOY_LDD(or similar) to specify the path (honestly, at first, simply setting$PATHto include that tool should work just fine).There is a
NO_CLEANUPenvironment variable in the runtime that pretty much prevents the files from being cleaned up; upon successive runs, the AppImage should not reextract fles that exist. It's a little prone to errors (most commonly when running more than one instance in parallel) and could use some love. I encourage you to experiment with that. Contributions welcome there as well.mdear commentedon Dec 13, 2023
I may also consider running QEMU on my build VM to do the ARM packaging, but I'm also trying to simplify the toolchain as much as possible, so will consider less complex alternatives.
mdear commentedon Dec 13, 2023
Also, the guys over at CrossControl have promised to send a script that emulates ldd output on x86_64 against ARM binaries, I'll take a look at it and see if it's reasonably designed, and if so, will post it here (with their permission of course).
mdear commentedon Dec 13, 2023
Also, to make things more interesting, I'm working in Qt_6.4 as it's a new project with no need for older frameworks (I saw some of your recent modifications in the linuxdeploy-qt plugin for ARMv8 and they appear to be coded against Qt5, I imagine it wouldn't be too difficult for me to extend support)
mdear commentedon Dec 13, 2023
I should also ask what in your opinion is the easiest way to encode some metadata in an AppImage that indicates a version number, and a way for code to extract that version number from the unpacked file system. This will allow us to keep our AppImages organized and will allow us to show a version number on the GUI.
Likely this would be to introduce a JSON or YAML file in the AppDir structure and give apps a hint as to what directory it ends up being unpacked in, likely via an environment variable. I imagine this would probably be very simple to accomplish.
mdear commentedon Jan 24, 2024
Ok, circling back to this task, I have an aarch64 QEMU environment ready with sshfs enabled, goal is to package a Qt demo app on an x86_86 build VM to run on a CCLinux v700/v1200 aarch64. Here goes ...
mdear commentedon Feb 10, 2024
linuxdeploy_strace_Feb9_2024.txt
file_aarch64_Qt_lib_Feb10_2024.TXT
gtest_all_cc_build_strace_hang_Feb10_2024.TXT
I'm running linuxdeploy but it's failing without giving enough information to debug properly. I have a patch and I need to ask you please to build it for me because my build is hanging.
I've made it far enough to create a Yocto Poky Linux Kirkstone binary with all libraries required to build linuxdeploy, my "cmake ." has passed successfully, but my "cmake --build ." hangs .
mdear commentedon Feb 10, 2024
I tried disabling testing and the builds appear to be progressing now, perhaps the builds just happen so slowly in the nested virtualization that they get clobbered by some kind of watchdog.
mdear commentedon Feb 11, 2024
After disabling the Google Test, a few files did compile, but the compilation ultimately hung.
I also tried the "cmake --build ." with a "-j 4" suffix and I just got multiple compilations hanging and failing instead of just one.
gtestDisabledCompileStillHangsFeb11_2024.txt
mdear commentedon Feb 12, 2024
I gave the QEMU aarch64 ARM machine six CPUs and 8GB of memory (since top was showing default settings were too low), the build appears to be progressing now. More news as it happens.
Update : The build succeeded! I'll now work to produce a pull request that will fix the issue I'm having (linuxdeploy trying to do a "readelf" on a directory).
mdear commentedon Feb 14, 2024
I was sending in a directory into the "--library" parameter incorrectly, which led to the failure. Is there a way you could warn the user that directories are not allowed, since the error message wasn't explicit, I had to read code and build/debug to figure out what I had done wrong.
#275
mdear commentedon Feb 16, 2024
Making progress - I ran linuxdeploy and it gathered dependencies as expected.
As the linuxdeploy Qt plugin needs tools from package qtdeclarative-tools, and after building the meta-qt6 layer against branch 6.4.1 I found it failed (according to https://endoflife.date/qt it's end-of-support) so I'm trying to build under Qt 6.5 Long Term Support in the hopes that I'll be able to get a successful build under Yocto Poky Kirkstone branch.
https://forum.qt.io/topic/154563/yocto-build-failed-cannot-resolve-git-codereview-qt-project-org-qt-tqtc-qmlcompilerplus-git
mdear commentedon Feb 25, 2024
I just wanted everybody to know that I finally succeeded in creating an AppImage that runs on my ARMv8 target, the demo Dice app that comes with Qt 6.5.3. The linuxdeploy/appimagetool toolchain got me 95% of the way there, but I still had to do some manual packaging steps. I've raised issues across multiple repos with my findings and suggestions.
@TheAssassin , thanks for maintaining this excellent toolchain! I'll be continuing to learn more and deep diving on the qt/appimage plugins to better understand them, and will direct any pull request should I become confident enough to generate them.
mdear commentedon Mar 3, 2024
Ok, I succeeded in creating an AppImage that removes dependencies on enough underlying native libs that it is able to run on multiple underlying Linux deployments (this is a requirement of ours, as we need to be best-effort forward-compatible with current and future OS installations).
On the advice of @TheAssassin I built my own version of Yocto Poky Kirkstone Linux against a qemuarm64 ARMv8 platform.
We rely on CClinux from CrossControl, CClinux2 is built on top of Yocto Poky Dunfell, CClinux3 is built on top of Yocto Poky Kirkstone.
My previous AppImage ran on CClinux3 but not on CClinux2, meaning that I wasn't capturing all the dependencies.
The linuxdeploy toolchain got me around 95% of the way there, the rest I had to do manual file placement (issues raised across the various linuxdeploy repos to track each issue I found).
I used appimagetool manually to do final packaging once I manually got the AppDir tuned properly.
All this effort begs the question whether the linuxdeploy toolchain may benefit from an optional portability packaging mode that ensures best-effort future portability of the AppImage to different underlying OS versions within the same OS family.
Since I had trouble compiling the linuxdeploy-qt plugin, I wasn't able to validate the PR https://github.com/linuxdeploy/linuxdeploy-plugin-qt/pull/157 (I executed these steps manually as described below).
UPDATE I was able to compile the qt plugin after all, please see linuxdeploy/linuxdeploy-plugin-qt#162 (comment)
Link to the final AppImage:
https://www.dropbox.com/scl/fi/b77f7nkaomv9koko7my7s/Dice-aarch64.AppImage?rlkey=u1elt7fdb4534mfo60cjnszcv&dl=0
Summary of changes:
apprun-hooks/linuxdeploy-plugin-qt-hook.shwith a code block to set needed QT/QML variables. I setLD_LIBRARY_PATHjust to ensure that it didn't inherit it from the calling shell (not sure that was necessary, especially since you manually set the rpath of deployed libs to$ORIGIN).Here's a list of ARMv8 libraries under Yocto Poky Kirkstone which have their interpreter explicitly set, which indicates that for completeness I perhaps should have packaged and reset the interpreter of these libs (however, just by resetting
libc.so.6my Qt QML app ran just fine):I manually installed the following:
usr/qml/QtWayland
usr/plugins/wayland-decoration-client
usr/plugins/wayland-graphics-integration-client
usr/plugins/wayland-shell-integration
libqwayland* into usr/plugins/platforms
libqvnc* into usr/plugins/platforms (just so I could run apps and then connect to them remotely)
qml/QtSensors/* into usr/qml
plugins/generic into usr/plugins
The following into usr/lib:
libQt6Wayland*
libQt6Wl*
libglapi.so.0
ld-linux-aarch64.so.1 // Ensure this dynamic linker is used rather than the underlying platform's linker
libdl.so.2 // This one surprised me. I did say I was not a deep Linux / glibc guru, right? Apparently, the system depends on this lib even though it doesn't appear in any "ldd" output (a hidden dependency apparently). It is part of the standard glibc library.
Introduce into AppDir the line "
cd $this_dir" after the line where$this_diris assigned. This was necessary due to the fact that I didn't seem to be able to use the$ORIGINvariable when setting the main application's interpreter viapatchelf --set-interpreter.