DWARF Overpopulation
Aug. 7th, 2016 08:46 pmA new preview release of Swift 3 is out this month. I tried compiling it with debugging flags turned on, left my computer running, and came back several hours later to a failed build.
FAILED: lib/libLTO.so
(...cut giant parameter list...)
clang-3.7: error: unable to execute command: Killed
clang-3.7: error: linker command failed due to signal (use -v to see invocation)
Huh. I tried it again while browsing the web in the background. A few hours later, my system started swapping like crazy. KDE froze for minutes at a time. top showed several ld processes each using over 1GB of memory. I think it died when one of them hit the 32-bit limit at 2GB.
So, WTF? Let's check out the sizes of some of these libraries.
560968756 Aug 6 19:22 libclangSema.a 620546286 Aug 6 19:49 libclangStaticAnalyzerCheckers.a 660270964 Aug 6 19:19 libLTO.so
And let's see what's in some of these archives.
~/swift/src/build/Ninja-DebugAssert/llvm-freebsd-x86_64/lib% ar -tv libclangStaticAnalyzerCheckers.a rw-r--r-- 0/0 164312 Dec 31 16:00 1969 AllocationDiagnostics.cpp.o rw-r--r-- 0/0 7163456 Dec 31 16:00 1969 AnalyzerStatsChecker.cpp.o rw-r--r-- 0/0 7142344 Dec 31 16:00 1969 ArrayBoundChecker.cpp.o rw-r--r-- 0/0 7290936 Dec 31 16:00 1969 ArrayBoundCheckerV2.cpp.o rw-r--r-- 0/0 9251672 Dec 31 16:00 1969 BasicObjCFoundationChecks.cpp.o rw-r--r-- 0/0 7271432 Dec 31 16:00 1969 BoolAssignmentChecker.cpp.o rw-r--r-- 0/0 7159208 Dec 31 16:00 1969 BuiltinFunctionChecker.cpp.o ...
There are many object files in the 7MB range. It adds up. Looking at one of them with elfdump, nearly all of the space is used by DWARF debugging info.
entry: 730 sh_name: .debug_str sh_type: SHT_PROGBITS sh_flags: sh_addr: 0 sh_offset: 26152 sh_size: 3267448 sh_link: 0 sh_info: 0 sh_addralign: 1 sh_entsize: 1 entry: 734 sh_name: .debug_info sh_type: SHT_PROGBITS sh_flags: sh_addr: 0 sh_offset: 3296469 sh_size: 1252146 sh_link: 0 sh_info: 0 sh_addralign: 1 sh_entsize: 0 entry: 735 sh_name: .rela.debug_info sh_type: SHT_RELA sh_flags: sh_addr: 0 sh_offset: 5007208 sh_size: 2054928 sh_link: 748 sh_info: 734 sh_addralign: 8 sh_entsize: 24
How big are the original source code files?
4944
So that gets turned into 7MB when debugging flags are turned on, and it all goes over 2GB when multiplied by the many files in a large package. This causes the build to fail.