大型软件的编译过程是什么样的?
大型软件的编译过程是什么样的?
thought
用户问的是大型软件的编译过程是什么样的。这是一个相当广泛但也很实际的问题。我需要给出一个清晰、深入但又不会过于学术化的回答,涵盖从源代码到可执行文件的完整流程,特别是大型软件(如浏览器、操作系统内核、LLVM等)特有的挑战和解决方案。 让我从几个维度来组织这个回答: 1. 编译的基本阶段(预处理、编译、汇编、链接) 2. 大型软件编译的特殊挑战和解决方案(构建系统、增量编译、分布式编译、缓存等) 3. 实际例子 我需要做一些搜索来补充细节,特别是关于现代构建系统(Bazel、Buck等)和分布式编译的内容。 让我搜索几个相关主题。
网页搜索查询: "large software build process compilation stages build system", 结果数: 8
{
  "results": [
    {
      "title": "'build-system' tag wiki - Stack Overflow",
      "url": "https://stackoverflow.com/questions/9364913/build-system-wiki-description",
      "description": "Build systems are software tools designed to automate the process of program compilation. Build systems come in various forms, and are used for a variety of"
    },
    {
      "title": "What are build systems? : r/C_Programming - Reddit",
      "url": "https://www.reddit.com/r/C_Programming/comments/1dkdwht/what_are_build_systems",
      "description": "A build system is basically a layer of automation over compilation, linking, and potentially other tasks like making a package. For the simple"
    },
    {
      "title": "A Comprehensive Comparison of Modern Build Systems | Blog",
      "url": "https://www.acldigital.com/blogs/all-about-build-system-and-its-types",
      "description": "ACL Digital is a trusted AI-led, Digital & Systems Engineering Partner driving innovation by designing and building intelligent systems across the chip to cloud technology stack. # Modern Build Systems: A Comparative Analysis of GNU Make, CMake, Ninja, and Meson. The primary goal of a build system is to simplify the build process, making it easier for developers to compile their software while reducing errors, build time, and inconsistencies. In the present day, there has been an emergence of several modern build systems, including GNU Make, CMake, Ninja, and Meson. The primary objective of the Meson build system is to streamline the build process and enhance cross-platform compatibility by generating build files compatible with various build tools and IDEs. Additionally, Meson is a simple build system to build complex and large source files and integration support in CMake is good. ACL Digital, part of the ALTEN Group, is a trusted AI-led, Digital & Systems Engineering Partner driving innovation by designing and building intelligent systems across the full technology stack — from chip to cloud."
    },
    {
      "title": "2020/01/27/ Build systems 101 part 1 · GitBook - GitLab",
      "url": "https://felipepiovezan.gitlab.io/blog/build_system_basics",
      "description": "On a high level, building a project is a two-step process: first, a set of build files is converted into build artifacts. Then, a subset of those build"
    },
    {
      "title": "Lecture Notes: Build Systems | Introduction to Software Development Tooling",
      "url": "https://bernsteinbear.com/isdt/lecture-notes/3-bld",
      "description": "In this module, you’ll learn about *build systems*, a type of tool that automates the process of *building* code—that is, converting it into a form that can be run and/or distributed1. You can think of build systems as supercharged shell scripts: like shell scripts, their purpose is to encapsulate a complex and intricate sequence of steps within a single command that’s easy to remember and is guaranteed to produce the same result each time it’s run. Build systems differ from the shell in a key conceptual way, though: when you run a script, you tell the shell exactly what commands to run and in what order. This may seem like a matter of semantics (and to some degree it is—a complex enough shell script could in theory do exactly what a build system does), but it has a huge real-world effect on how efficiently you can work."
    },
    {
      "title": "What Is a Build System? - ITU Online IT Training",
      "url": "https://www.ituonline.com/tech-definitions/what-is-a-build-system",
      "description": "It can compile code, link binaries, run tests, generate artifacts, and prepare output for release. That is why teams use automated build systems"
    },
    {
      "title": "How do build systems handle parallel compilation of dependent files?",
      "url": "https://www.quora.com/How-do-build-systems-handle-parallel-compilation-of-dependent-files",
      "description": "B, C, D, E, G, and H depend upon A. One can process B, C, D, E, and H in parallel, one A is computed. When all of those computations have been"
    },
    {
      "title": "A Dive to the Build Process - What Goes On When You Press the Play Button - RANDOM BITS",
      "url": "https://zakuarbor.github.io/blog/building-code",
      "description": "# A Dive to the Build Process - What Goes On When You Press the Play Button. On my previous post What does the Build Team Do, I covered various topics about builds and devOps. Today, I want to go more in depth on what goes on when you press the run button which I briefly brushed over in the previous post. ## What is a Build. You may recall when working on Java, C, or C++ you ran a Build by pressing pressing Build, Run, or Play button. As you may know, a build is just simply the process of converting your source code (i.e. `.c` or `.java`) to a binary/exectuable (i.e. An executable to put it briefly is the **ready to run** form of a program (hence why we call it an **executable**). Before your source code is translated to assembly or machine code, it first goes through a program which we call the **preprocessor**."
    }
  ]
}
网页搜索查询: "distributed compilation build system Bazel Buck ccache", 结果数: 8
{
  "results": [
    {
      "title": "Distributed Builds  |  Bazel",
      "url": "https://bazel.build/basics/distributed-builds",
      "description": "# Distributed Builds Stay organized with collections Save and categorize content based on your preferences. this scale, it’s simply impossible to complete a build in a reasonable amount. of time on a single machine: no build system can get around the fundamental. The only way to make this work. is with a build system that supports distributed builds wherein the units of. work being done by the system are spread across an arbitrary and scalable. units (more on this later), this would allow us to complete any build of any. The simplest type of distributed build is one that only leverages *remote. Distributed build with remote caching. A distributed build showing remote caching. with the remote cache to see if that artifact already exists there. For a remote caching system to work, the build system must guarantee that builds. That is, for any build target, it must be possible. remote execution, in which the actual work of doing the build can be spread."
    },
    {
      "title": "Bazel: { Fast, Correct } — Choose One - JWillikers",
      "url": "https://www.jwillikers.com/bazel-fast-correct-choose-one",
      "description": "When it comes to build systems and package management, there really is no silver bullet. It’s also more correct than most of the competition. Another sign of its popularity is the number of clones its spawned which includes Buck2, Pants, and Please.[1] Of those, only Bazel and Buck2 treat C++ as a first-class citizen, and from Buck2’s documentation you can find it says, \"There are not yet mechanisms to build in release mode.\"[2] *Wait.* *Really?* Enough of the clones. I’ll just focus on Bazel here, since my primary target is C++ and I’d like to be able to make release builds. When you build a project with Bazel, there’s something you’ll notice immediately when you’re used to build systems like CMake, Meson, and Autotools. Where this does fix the issue, the root of the problem is typically a change to a default value that already exists in the build system’s cache and is therefore not updated, or a change to the external environment which, unbeknown to the build system, renders its cache invalid."
    },
    {
      "title": "The next generation of Bazel builds - by Julio Merino",
      "url": "https://blogsystem5.substack.com/p/bazel-next-generation",
      "description": "Today marks the 10th anniversary of Bazel’s public announcement so this is the perfect moment to reflect on what the next generation of build systems in the Bazel ecosystem may look like. I write this with the inspiration that comes from attending the first ever conference on Buildbarn, one of the many remote execution systems for Bazel. The predecessor to Bazel, Blaze, is a build system designed at Google for Google’s monorepo scale. What’s worse is that the Bazel server process is stateful: at start up, Bazel goes through the expensive steps of computing the analysis graph from disk and, to prevent redoing this work in every build, Bazel keeps this graph in its in-memory “analysis cache”. This was one of my original critiques when Bazel went open source in 2015: at that time, I really wished for a declarative build system like Bazel to replace the mess that was and is Make plus the GNU Autotools, but the fact that Bazel was written in Java meant that it would never do this (mostly for non-technical reasons)."
    },
    {
      "title": "Why use Bazel or Buck instead of cmake, xmake, meson, etc.? - Reddit",
      "url": "https://www.reddit.com/r/cpp/comments/192jeuc/why_use_bazel_or_buck_instead_of_cmake_xmake",
      "description": "Bazel is a power-tool designed to provide repeatable builds at scale. It's the open-source version of Google's internal build tool, Blaze. It"
    },
    {
      "title": "Does Using Bazel Help Speed Up Continuous Integration Builds?",
      "url": "https://arxiv.org/html/2405.00796v1",
      "description": "In addition to a variety of build scheduling and test selection heuristics studied in the past, new artifact-based build technologies like Bazel have built-in support for advanced performance optimizations such as parallel build and incremental build (caching of build results). In this study, we collected 383 Bazel projects from GitHub, then studied their parallel and incremental build usage of Bazel in popular CI services (GitHub Actions, CircleCI, Travis CI, or Buildkite), and compared the results with Maven projects. We conducted 3,500 experiments on 383 Bazel projects and analyzed the build logs of a subset of 70 buildable projects to evaluate the performance impact of Bazel’s parallel builds. Compared to sequential builds, the median speedups for long-build duration projects are 2.00x, 3.84x, 7.36x, and 12.80x, at parallelism degrees 2, 4, 8, and 16, respectively, even though, compared to a clean build, applying incremental build achieves a median speedup of 4.22x (with a build system tool-independent CI cache) and 4.71x (with a build system tool-specific cache) for long-build duration projects."
    },
    {
      "title": "C++ Build System Tutorial: Using Bazel for Scalable, Reliable Builds",
      "url": "https://www.youtube.com/watch?v=mvxrDN9mtTU",
      "description": "Building C++ at scale is notoriously complex. In this instructional video, you will learn how Bazel simplifies the process by helping you"
    },
    {
      "title": "A Tour Around Buck2, Meta's New Build System - Tweag",
      "url": "https://tweag.io/blog/2023-07-06-buck2",
      "description": "## A Tour Around Buck2, Meta's New Build System. Buck2 is a from-scratch rewrite of Buck, a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. We were thrilled to have the opportunity to work with Meta on Buck2 to help make the tool useful and successful in the open-source use case. We also have a strong focus and a lot of experience with the Bazel build system. The codebase is structured in a very modular fashion, such that most components can also be published as independent packages on crates.io, and are independently useful. For example, starlark-rust is used by Buck2 to parse and evaluate Starlark, however, it is also used for Buck2 IDE integration, and could be useful for Bazel IDE integration as well. Most of it is implemented in Java, and while some components can be used independently, e.g., the remote worker, most are generally not developed or published as standalone components."
    },
    {
      "title": "Lightning Talk: Bazel, Buck, and Friends. How to Scale your Build?",
      "url": "https://www.youtube.com/watch?v=S4pcHhb44mk",
      "description": "Developers want to focus on developing new features and solving problems without distraction. It is the build system's job to enable this:"
    }
  ]
}
网页搜索查询: "incremental compilation shared library linking large C++ project", 结果数: 8
{
  "results": [
    {
      "title": "incremental compilation via shared library - Google Groups",
      "url": "https://groups.google.com/g/comp.lang.dylan/c/7k9gtT7jWYs",
      "description": "Someone at an Arizona university is writing an IC library based on the intermediate code used by the Purdue LCC compiler project. Should be cool when they"
    },
    {
      "title": "Splitting large C++ project - Software Engineering Stack Exchange",
      "url": "https://softwareengineering.stackexchange.com/questions/360452/splitting-large-c-project",
      "description": "For linking, putting code into a shared library should help. Some build systems use link-time optimisations, that might add to build times and"
    },
    {
      "title": "C++ Compilation: 10 Tips for Faster Builds - YouTube",
      "url": "https://www.youtube.com/watch?v=BirLgxwCC0s",
      "description": "... Incremental Linking 15:47 Chapter 10: External Tools. ... Share. Save. Report. Comments. 29. PIMPL:"
    },
    {
      "title": "Software Development with C++: Shared Libraries - YouTube",
      "url": "https://www.youtube.com/watch?v=yxZqhAfdODI",
      "description": "What Happens After The Compiler in C++ (How Linking Works) - Anders Schau Knatten - C++ on Sea 2023. cpponsea•15K views · 12:27 · Go to channel"
    },
    {
      "title": "How long are your compilation & linking times? Incremental ... - Reddit",
      "url": "https://www.reddit.com/r/cpp/comments/bosqhx/how_long_are_your_compilation_linking_times",
      "description": "Incremental build probably takes about 5-7 minutes, 80-90% of which is spent on linking the humongously huge browser.dll . The resulting build"
    },
    {
      "title": "Complex C++ Compilation from the Command Line - Unrealistic",
      "url": "https://unrealistic.dev/posts/complex-c-compilation-from-the-command-line",
      "description": "We can create the shared library with GCC, as well, using the -fpic switch while creating object files and the -shared switch when linking. g++"
    },
    {
      "title": "Compiling and linking a library - IBM",
      "url": "https://www.ibm.com/docs/en/xl-c-and-cpp-linux/16.1.0?topic=library-compiling-linking",
      "description": "This section describes how to compile your source files into object files for inclusion in a library, how to link a library into the main program,"
    },
    {
      "title": "Linking Libraries - University College London",
      "url": "https://github-pages.ucl.ac.uk/research-computing-with-cpp/06tooling/sec03LinkingLibraries.html",
      "description": "You can compile a library without creating an executable if you want a library to be a separate project: simply use the add_library command! You don't need to"
    }
  ]
}
thought
Let me also search for more specific topics related to the build process - like preprocessor, compiler stages, linking, and concrete examples of large software builds.
网页搜索结果数: 8, 查询: "compiler stages preprocessing compilation assembly linking ELF executable format"
{
  "results": [
    {
      "title": "Executable and Linkable Format - Talks",
      "url": "https://talks.harshkapadia.me/elf/content.html",
      "description": "To compile a program, i.e., to create an executable binary, the high level steps are: Preprocessing; Compilation; Assembling; Linking. To load the program into"
    },
    {
      "title": "Compiling Our C Code: A Step-by-Step Journey With All Stages",
      "url": "https://medium.com/@fatmabeyzaaydogan/compiling-our-c-code-a-step-by-step-journey-with-all-stages-3f9010b832c9",
      "description": "5. Linking Stage ( main.elf ) · main.elf : This is now a fully executable file. · In Linux, it typically uses the ELF (Executable and Linkable"
    },
    {
      "title": "Journey of a C Program to Linux Executable in 4 Stages",
      "url": "https://www.thegeekstuff.com/2011/10/c-program-to-an-executable",
      "description": "The four stages for a C program to become an executable are the following: Pre-processing; Compilation; Assembly; Linking. In Part-I of this"
    },
    {
      "title": "Examining the Compilation Process. Part 1. - Linux Journal",
      "url": "https://www.linuxjournal.com/content/examining-compilation-process-part-1",
      "description": "Broadly speaking, the compilation process is broken down into 4 steps: preprocessing, compilation, assembly, and linking. We'll discuss each"
    },
    {
      "title": "What is the assembler and linker stage do in the compilation process",
      "url": "https://www.reddit.com/r/C_Programming/comments/vvi9sd/what_is_the_assembler_and_linker_stage_do_in_the",
      "description": "I've just studied the compilation process has 4 stages: Preprocessing Compiling Assembling Linking So what does stage 3 and 4 actually do ?"
    },
    {
      "title": "E Preprocess only; do not compile, assemble or link - Facebook",
      "url": "https://www.facebook.com/MyBitCast/posts/c-program-compilation-stepsyou-compile-c-program-and-get-executables-have-you-ev/818232024865839",
      "description": "Preprocessing This is the very first stage through which a source code passes. In this stage the following tasks are done: Macro substitution"
    },
    {
      "title": "[PDF] Compilation, Assembling, Linking and Program Execution",
      "url": "https://passlab.github.io/ITSC3181/notes/lecture02_CompilationAssemblingLinkingProgramExecution.pdf",
      "description": "27 Compile Multiple Files and Link to One Executable • Split the sum_full.c into two files – sum.c that only contains the definition of sum method • Also the “#define REAL float” line on top – Remove the sum definition from sum_full.c, but still keep sum method declaration (referred too as function signature) – Compile both together and generate sum executable • Compile in one step: gcc sum_full.c sum.c -o sum – The command compiles each *.c file one by one into object files and then link the two object files into one executable • Compile in multiple steps: compile each .c file one by one and link together 28 Compile in One Step 29 Compile in Multiple Steps 30 Try readelf 31 Try objdump for both object file and executable 32 “objdump -D” to disassembly: convert binary object code back to symbolic assembly code 33 nm: list symbols from object files • T: define a symbol • U: undefined symbol – Linker to link • Address are relative 34 Static Linking • If multiple program want to use read_timer functions – They all include the full definition in their source code • Duplicate: If the function changes, we need to change each file – Separate reader_timer in a new file, compile and statically linked with other object files to create executables • Duplicate the same object in multiple executables."
    },
    {
      "title": "gcc - Breaking down the C compilation process into preprocessing ...",
      "url": "https://stackoverflow.com/questions/77108297/breaking-down-the-c-compilation-process-into-preprocessing-compilation-assembl",
      "description": "I've tried to break down each step (preprocessing, compilation, assembling, and linking) in terms of input/output in order to grasp what is going on under the"
    }
  ]
}
网页搜索结果数: 8, 查询: "Chromium build process how long does it take LLVM build time"
{
  "results": [
    {
      "title": "If LLVM is so slow is anything being done about it? - Page 3",
      "url": "https://discourse.llvm.org/t/if-llvm-is-so-slow-is-anything-being-done-about-it/75389?page=3",
      "description": "An optimized build of llc typically takes around 48s (around 13m in user time), whereas a debug (-g) build takes ~9m (249m in user time)!"
    },
    {
      "title": "Big Project Build Times–Chromium | Random ASCII - WordPress.com",
      "url": "https://randomascii.wordpress.com/2020/03/30/big-project-build-times-chromium",
      "description": "Expect this to take about 13 hours on a four-core machine with lots of memory and a fast drive. All of my test builds use the”-j 4” option to"
    },
    {
      "title": "Chromium Build Times - Solomon Kinard - Medium",
      "url": "https://medium.com/@solomonkinard/m2-chromium-build-time-6a930f57663b",
      "description": "Chromium Build Times M2 MacBook Pro compilation took 1 hour and 49 minutes ... the build in 6:17. M4 is a single-die SoC, while M5"
    },
    {
      "title": "Compilation Time is Too Long",
      "url": "https://groups.google.com/a/chromium.org/g/chromium-dev/c/kwfR0vwXmKg",
      "description": "# Compilation Time is Too Long. I'm compiling Chromium using my modified clang, but it seems every round of build took 2 full days, which is longer than what I searched in the internet. I use \"gn args out/mybuild\" to config the building arguments and use ninja -C out/mybuild -j72 to build the system, the arguments are showing as follows:. When the build doesn't have a target, it tries to build all targets and that takes more time. Thanks for your response, I use ninja -C out/mybuild chrome -j72 to build the system, and it shows 50K+ tasks in total. I do find that the linking takes a lot of time, the linking flag is referring to this post (). The Clang I used is modified and add some sanitizers, I wonder if it's reasonable to take such a long time. to the...@chromium.org, Yizhuo Zhai, Chromium-dev. to Chromium-dev, Yoshisato Yanagisawa, Yizhuo Zhai, Chromium-dev, the...@chromium.org."
    },
    {
      "title": "google chrome - How to build Chromium faster? - Stack Overflow",
      "url": "https://stackoverflow.com/questions/37134466/how-to-build-chromium-faster",
      "description": "### your communities. ### more stack exchange communities. Communities for your favorite technologies. Stack Overflow for Teams is now called **Stack Internal**. ##### Collectives™ on Stack Overflow. Following only the instructions here -  I have been able to successfully build and get a Chromium executable which I can then run. So, I have been playing around with the code (adding new buttons to the browser etc.) for learning purposes. So each time I make a change (like adding a new button in the settings toolbar) and I use the ninja command to build it takes over 3 hours to finish before I can run the executable. (At the moment, I have literally just followed the instructions in the above mentioned link and no other optimizations to speed it up.). You can also peruse faster linux builds and see if any of those tips apply to you. For building on Linux, you can see how to build faster at:. You can edit the build arguments on a build directory by:."
    },
    {
      "title": "How long does it takes to compile Firefox and Chromium ? : r/Gentoo",
      "url": "https://www.reddit.com/r/Gentoo/comments/qbd4l6/how_long_does_it_takes_to_compile_firefox_and",
      "description": "0 merge time: 20 minutes and 25 seconds. and chromium about 2h30 last time I did so (I tend to use the binary google-chrome now instead). That's"
    },
    {
      "title": "Chrome should build faster on Windows [40551319]",
      "url": "https://issues.chromium.org/40551319",
      "description": "# Chrome should build faster on Windows [40551319] - Chromium. ### Chrome should build faster on Windows. br...@chromium.org created issue #1. This bug tracks all efforts to make building Chrome for Windows faster. https://chromium.googlesource.com/chromium/src/+/master/docs/windows_build_instructions.md#Faster-builds. However Chrome for Windows still takes an annoyingly long time to build, especially for those without access to goma. So, I'm making this the master bug for tracking most efforts to make Chrome build faster. crbug.com/725639- Component builds restrict build parallelism, costing 60-120 seconds on a full build. crbug.com/679089- See if incremental linking can be enabled for x64 in goma/symbol_level=1 builds. crbug.com/495204- Make it possible to build chrome/win on linux, using clang-cl and gn. crbug.com/787903- Python CPU usage on chrome builds is significant. crbug.com/702518- Increase link parallelism for win buildbot with is_component_build=true - many compilation steps wait on link steps that they don't need to link on, this can be seen when looking at ninjatracing reports where the compile stages all 'drain' before many of the link steps, leading to excessive serialization and wasted time. https://chromium.googlesource.com/chromium/src.git/+/396e35d38515d055988c7ad3d3ea43d68769b91f."
    },
    {
      "title": "Full disclosure: I work for Google on Chrome. A Chrome build is truly a computat... | Hacker News",
      "url": "https://news.ycombinator.com/item?id=14734171",
      "description": "| |  | | --- | | titzer on July 10, 2017  | parent | context | favorite | on: 24-core CPU and I can’t move my mouse   Full disclosure: I work for Google on Chrome. A Chrome build is truly a computational load to be reckoned with. Without the distributed build, a from-scratch build of Chrome will take at least 30 minutes on a Macbook Pro--maybe an hour(!). However, to actually have the C++ compiler do inlining at compile time (LTO be damned), we have to put the definitions of inline functions into header files, which greatly increases their size and processing time. Most of the projects I've worked on have been larger than Chrome, I've seen the compile time for BioShock Infinite go from 2 hours down to 15 minutes with serious work on header use, precompiled headers, and all the other tricks people use."
    }
  ]
}
网页搜索结果数: 8, 查询: "unity build jumbo build header optimization techniques large C++ codebase"
{
  "results": [
    {
      "title": "Unity build - Wikipedia",
      "url": "https://en.wikipedia.org/wiki/Unity_build",
      "description": "A unity build (also known as unified build, jumbo build or blob build) is a method used in C and C++ software development to speed up the compilation of"
    },
    {
      "title": "Comparing C/C++ unity build with regular build on a large codebase (2024) | Hacker News",
      "url": "https://news.ycombinator.com/item?id=47604768",
      "description": "| |  | | --- | | Comparing C/C++ unity build with regular build on a large codebase (2024) (hereket.com) | | 12 points by PaulHoule 46 days ago | hide | past | favorite | 6 comments |     |  |  | | --- | --- | | |  | | --- | | jjmarr 46 days ago  | next )   If you're doing single-core builds, you will get impressive speedups from unity builds. | | | |  | | --- | | torginus 46 days ago  | parent | next )   The problem is that due to how templates work, each compilation unit will end up with its own copy of templated function, which creates extra work, code bloat etc. But the linker is single threaded and notoriously slow - with LTO, I wouldnt be surprised it would take up as much time as the whole unity build, and the results are often suboptimal."
    },
    {
      "title": "Working with Jumbo/Unity Builds (Single Translation Unit) - Austin Morlan",
      "url": "https://austinmorlan.com/posts/unity_jumbo_build",
      "description": "Essentially it’s an alternate way of compiling C/C++ by putting all of the code (headers and source both) into a single file. In C and C++, code is split up into separate files: source files (**.c**) and header files (**.h**). When you run the compiler on a source file it compiles it down to an object file (**.o**) which is a translation unit. The compiler can only see what’s inside of that single C file and nothing else, so it takes it on faith that symbols referenced in the source file exist in some other translation unit. If **foo** is defined in **foo.c** and referenced in **bar.c**, the linker connects the symbols such that the reference in **boo.o** connects with the definition in **foo.o**. The compiler can only look at one source file at a time, but a source file can reference types and objects that exist outside of it. You have a source file **bar.c** that defines a function called **bar()**."
    },
    {
      "title": "Speeding up C++ builds with Unity Builds and reduced header ...",
      "url": "https://stackoverflow.com/questions/52423521/speeding-up-c-builds-with-unity-builds-and-reduced-header-dependencies",
      "description": "I did some research and found that there are basically three things recommended to reduce the build time: reducing header dependencies; using"
    },
    {
      "title": "Precompiled Headers and Unity (Jumbo) Builds in upcoming CMake",
      "url": "https://www.qt.io/blog/2019/08/01/precompiled-headers-and-unity-jumbo-builds-in-upcoming-cmake",
      "description": "The CMake MR 3611 adds support for Unity (Jumbo) Builds in CMake. With precompiled headers one could just change a few CMake lines and get a 20-40% speed"
    },
    {
      "title": "Comparing C/C++ unity build with regular build on a large codebase",
      "url": "https://www.reddit.com/r/cpp/comments/1e1v6np/comparing_cc_unity_build_with_regular_build_on_a",
      "description": "What happens with unity build when you change a single source file, does it need to recompile everything or are there possible optimizations?"
    },
    {
      "title": "C++ Weekly - (2x Faster!) What are Unity Builds (And How They Help)",
      "url": "https://www.youtube.com/watch?v=POYVF6urMwg",
      "description": "why do header files even exist? Low Level•601K views · 19:10. Go ... 20 Advanced Coding Tips For Big Unity Projects. Tesseract•310K views."
    },
    {
      "title": "JumboSupport: making unity builds easier in Clang - Page 2",
      "url": "https://discourse.llvm.org/t/jumbosupport-making-unity-builds-easier-in-clang/48269?page=2",
      "description": "The jumbo build just needs to append a bunch of files, that's fast. Compiling a module isn't. Well, compiling a module is just appending a bunch"
    }
  ]
}
thought
Let me also get some details on the linking stage and how it works for large projects, as well as LTO and related topics.
网页搜索结果数: 8, 查询: "static vs dynamic linking LTO link time optimization large software build"
{
  "results": [
    {
      "title": "Static vs Dynamic Linking for High-Performance / Low-Latency ...",
      "url": "https://www.reddit.com/r/cpp/comments/1mf3m3o/static_vs_dynamic_linking_for_highperformance",
      "description": "Static linking combined with LTO (link time optimization) also allows for more optimizations, for example: devirtualizing when only a single"
    },
    {
      "title": "[PDF] Linking Explained Static vs Dynamic vs LTO Guide in Modern C++.",
      "url": "https://simplifycpp.org/books/minibooklet/mini_booklet_Linking_Explained_Static_vs_Dynamic_vs_LTO_Guide.pdf",
      "description": "1.5.2 Dynamic Linking Relocations Dynamic linking requires a more flexible mechanism: • Some symbol addresses cannot be known until the shared library is loaded. Compare: • File size • Load time • Dependencies (via ldd) Ex.3 On Windows, link against both a static library and a DLL version of the same module. 2.4.1 Startup Time Dynamic executables must: • Load the dynamic loader (ld-linux or Windows loader) • Load required shared libraries • Process relocations • Bind symbols (lazy or eager) Static binaries skip nearly all of this, so startup time is often lower—especially in systems with many shared libraries. Example: cmake_minimum_required(VERSION 3.23) project(LtoExample CXX) set(CMAKE_CXX_STANDARD 20) add_library(core STATIC core.cpp) add_executable(app main.cpp) target_link_libraries(app PRIVATE core) # Simple LTO toggle option(ENABLE_LTO \"Enable Link-Time Optimization\" ON) if (ENABLE_LTO) include(CheckIPOSupported) check_ipo_supported(RESULT ipo_supported OUTPUT ipo_error) if (ipo_supported) message(STATUS \"IPO/LTO enabled\") set_property(TARGET core PROPERTY INTERPROCEDURAL_OPTIMIZATION TRUE) set_property(TARGET app PROPERTY INTERPROCEDURAL_OPTIMIZATION TRUE) else() message(WARNING \"IPO/LTO not supported: ${ipo_error}\") endif() endif() Advantages of using CMake IPO: • Automatically maps to -flto (GCC/Clang) and /GL (MSVC)."
    },
    {
      "title": "Link-time optimisation (LTO) | Lobsters",
      "url": "https://lobste.rs/s/mxwit7/link_time_optimisation_lto",
      "description": "A benefit of C++ LTO is that it removes the performance implications of inlining at the source level, since it can inline a function no matter where it’s implemented. That frees me to write [non-template] function/method implementations inline or not, as I see fit, for purposes of code clarity or compilation speed in debug builds. And you don’t have to deal with the major compilation slowdown produced by header-only libs, which occurs in all builds not just optimized ones. Technically, LTO can be made to work with DSOs, if we can guarantee that link-time DSOs are the same as run-time DSOs. However, this will be a lot of work and the gain will be unclear. The first LLVM pass that I wrote tried to do some of this automatically, pulling functions that did not touch mutable state out of a library into a separate IR file so that you could ship the library as a binary along with some functions that might make sense for inlining at install time."
    },
    {
      "title": "Static linked binaries are a generally *lot* smaller than a ...",
      "url": "https://news.ycombinator.com/item?id=39148420",
      "description": "Static linked binaries are a generally lot smaller than a dynamically linked library and its dependencies, especially with link-time optimizations and inlining."
    },
    {
      "title": "Linking Explained Static vs Dynamic vs LTO Guide - LinkedIn",
      "url": "https://www.linkedin.com/pulse/linking-explained-static-vs-dynamic-lto-guide-ayman-alheraki-unkvf",
      "description": "1. The conceptual and practical differences between static and dynamic linking, including how each impacts performance, portability, and"
    },
    {
      "title": "Dynamic link vs. static link efficiency - Stack Overflow",
      "url": "https://stackoverflow.com/questions/14286588/dynamic-link-vs-static-link-efficiency",
      "description": "You have two methods at your disposal. Build the library as a dynamic linked library; Build the library as a statically linked library. My"
    },
    {
      "title": "Rust static library + parts in C + ~~LTO~~ cross language optimization - help - The Rust Programming Language Forum",
      "url": "https://users.rust-lang.org/t/rust-static-library-parts-in-c-lto-cross-language-optimization/85722",
      "description": "# Rust static library + parts in C + ~~LTO~~ cross language optimization. how do i create a static library, where. i did get this to work (incl the inline annotations) using `rustc --crate-type=staticlib -Clinker-plugin-lto -Copt-level=2`. and something like `clang -c -flto=thin the_c_part.c`. but the issue was, the resulting \"object files\" are actually llvm bitcode, and the user of the library would have to compile with `clang -fuse-ld=lld -flto=thin`. so the question is: how to combine rust & C into a **static library** ~~*with* **LTO**~~? this last part was supposed to mean: as the rust & C are \"linked\" into a static library, they're supposed to be optimized together. Cross-language LTO is a new feature in the Rust compiler that enables LLVM's link time optimization to be performed across a mixed C/C++/Rust codebase. just like the C compiler can do *cross translation unit* optimization if you do `clang -O2 file1.c file2.c -o my_lib.a`."
    },
    {
      "title": "Static vs Dynamic Linking - C++ Forum",
      "url": "https://cplusplus.com/forum/beginner/250674",
      "description": "undefined NaN, NaN at NaN:NaNpm. Dynamic linking is great if you want the size of your executable to be as small as possible, with the possibility of having to include OS and third party library files when you distribute your program. undefined NaN, NaN at NaN:NaNpm. I wanted to get your thoughts regarding static vs dynamic linking. Last edited on undefined NaN, NaN at NaN:NaNpm. undefined NaN, NaN at NaN:NaNpm. That's pretty much what I managed to gather through independent research as well but needed someone to confirm it for me. The more libraries you use, the larger your executable can get. Edit: Added \"in our production environment\" to statement about rarity of dynamically (explicitly) loaded DLLs. Last edited on undefined NaN, NaN at NaN:NaNpm. undefined NaN, NaN at NaN:NaNpm. Shared libraries only make your program smaller if the library is already on the computer. undefined NaN, NaN at NaN:NaNpm. The choice often comes down to what you expect your users to have to install."
    }
  ]
}
网页搜索结果数: 8, 查询: "Google Blaze Bazel build graph dependency DAG action graph execution"
{
  "results": [
    {
      "title": "Grok Your Bazel Build: The Action Graph",
      "url": "https://jin.crypt.sg/articles/bazel-action-graph.html",
      "description": "# Grok Your Bazel Build: The Action Graph. A recent addition is the **Action Graph**. The action graph is different from the **target dependency graph**, which is generated from Bazel’s **loading** phase. You might know the target graph from `bazel query`:. If you’re looking for the target graph, check out this Bazel blog post on visualizing your build. The action graph contains a different set of information: file-level dependencies, full command lines, and other information Bazel needs to **execute** the build. If you are familiar with Bazel’s build phases, the action graph is the output of the **loading and analysis** phase and used during the **execution** phase. It only executes if it has to, that is, the action graph is the super set of what is actually executed. The action graph is generated by:. Let’s walk though an example of dumping the action graph of an Android application build. The `android_binary` target is `//examples/android/java/bazel:hello_world`. It’s defined in `examples/android/java/bazel/BUILD`:. Dumping action graph to 'action_graph.bin'."
    },
    {
      "title": "Bazel Build System - Augment Code",
      "url": "https://www.augmentcode.com/open-source/bazelbuild/bazel",
      "description": "# Bazel Build System. `src/main/java/com/google/devtools/build/lib/bazel/Bazel.java`. `src/main/java/com/google/devtools/build/lib/runtime/BlazeRuntime.java`. Bazel is a fast, scalable build system designed to handle projects of any size across multiple programming languages and platforms. The codebase implements a sophisticated client-server architecture that maintains state between builds for optimal performance. Bazel operates as a **client-server system** where a long-running server process stays in RAM between builds, allowing the system to maintain state and avoid redundant work. The entry point is the `Bazel` class, which initializes the runtime with a carefully ordered list of modules and services. ### Module System. For example, the `StandaloneModule` is placed after `WorkerModule` and `SandboxModule` to make standalone execution the default strategy. **Commands** are registered by modules and dispatched by `CommandDispatcher`, which parses options, applies invocation policies, and executes the appropriate command implementation. ### Build Execution. `src/main/java/com/google/devtools/build/lib/buildtool/BuildTool.java`. `src/main/java/com/google/devtools/build/lib/buildtool/AnalysisPhaseRunner.java`. `src/main/java/com/google/devtools/build/lib/buildtool/ExecutionTool.java`. `src/main/java/com/google/devtools/build/lib/analysis/BuildView.java`. The build system executes in three main phases:. `src/main/java/com/google/devtools/build/skyframe/AbstractInMemoryMemoizingEvaluator.java`. `src/main/java/com/google/devtools/build/skyframe/InMemoryMemoizingEvaluator.java`. `src/main/java/com/google/devtools/build/skyframe/ParallelEvaluator.java`. `src/main/java/com/google/devtools/build/skyframe/DirtyBuildingState.java`. `src/main/java/com/google/devtools/build/skyframe/IncrementalInMemoryNodeEntry.java`. `src/main/java/com/google/devtools/build/lib/skyframe/BzlLoadFunction.java`. `src/main/java/com/google/devtools/build/lib/skyframe/PackageFunction.java`. `src/main/java/com/google/devtools/build/lib/skyframe/ActionExecutionFunction.java`."
    },
    {
      "title": "Dynamic Dependencies #28346 - bazelbuild bazel - GitHub",
      "url": "https://github.com/bazelbuild/bazel/discussions/28346",
      "description": "An action runs a dependency resolver on the source files and writes the DAG to a json, then; dynamic_output takes the json as a dynamic argument, takes as"
    },
    {
      "title": "bazel: why do targets keep increasing if build graph is known",
      "url": "https://stackoverflow.com/questions/51484140/bazel-why-do-targets-keep-increasing-if-build-graph-is-known",
      "description": "If you are familiar with Bazel's build phases, the action graph is the output of the loading and analysis phase and used during the execution"
    },
    {
      "title": "Action Graph Query (aquery)  |  Bazel",
      "url": "https://bazel.build/query/aquery",
      "description": "# Action Graph Query (aquery) Stay organized with collections Save and categorize content based on your preferences. The `aquery` command allows you to query for actions in your build graph. It operates on the post-analysis Configured Target Graph and exposes. `aquery` is useful when you are interested in the properties of the Actions/Artifacts. generated from the Configured Target Graph. and their inputs/outputs/mnemonics. Notably, the aquery command runs on top of a regular Bazel build and inherits. the set of options available during a build. It supports the same set of functions that is also available to traditional. A simple example of the syntax for `aquery` is as follows:. `aquery` runs on top of a regular Bazel build and thus inherits the set of. Includes the content of the action command lines in the output (potentially large). Includes names of the action inputs and outputs in the output (potentially large). Whether to include Aspect-generated actions in the output."
    },
    {
      "title": "How to query build graph for actions and action details?",
      "url": "https://groups.google.com/g/bazel-discuss/c/_Uo5weYHdwc",
      "description": "I would like to see how a target is built down to each action, the input, output and command for each action. This is similar to to make -pn."
    },
    {
      "title": "Extracting build performance metrics  |  Bazel",
      "url": "https://bazel.build/advanced/performance/build-performance-metrics",
      "description": "# Extracting build performance metrics Stay organized with collections Save and categorize content based on your preferences. Improving the performance of individual builds has particular value. diagnosing and fixing issues in one build might help to resolve issues at the. An important step to improving the performance of builds is to understand where. how you can use these metrics to detect and fix build performance issues. There are a few main ways to extract metrics from your Bazel builds, namely:. For every build-like Bazel invocation, Bazel writes a trace profile in JSON. ## Execution Log. The execution log can help you to troubleshoot and fix. You can use these metrics for example to. you need additional information on the dependency graph of the executed actions. This information can be used to understand the drag that is added by a node on. The data helps you predict the impact of changes to the build and action graph."
    },
    {
      "title": "Understanding Bazel: An Introductory Overview",
      "url": "https://medium.com/@d.s.m/understanding-bazel-an-introductory-overview-0c9ddb1b1ce9",
      "description": "# Understanding Bazel: An Introductory Overview | by David Mavrodiev | Medium. In this article, you will understand what Bazel is, what it does, how it does it, and how you can use it. Bazel is a build tool that supports multiple languages and platforms. A fundamental aspect of Bazel is its ability to define precise build actions. Build actions are the individual tasks that must be executed to build software. Bazel defines these actions in a highly granular and controlled manner, allowing for precise build process management. For example, if a source file is changed, Bazel knows which binaries rely on this file and, therefore, need to be rebuilt. This dependency analysis is crucial for incremental builds, ensuring only the necessary parts of the project are rebuilt when changes are made. It features a `WORKSPACE.bazel` file, which defines the Main repository and references External repositories needed for the build."
    }
  ]
}
thought
Now I have enough material to give a comprehensive answer. Let me organize it. The question "大型软件的编译过程是什么样的" can be addressed at multiple levels: 1. The fundamental compilation stages (preprocessor, compiler, assembler, linker) 2. The build system layer that orchestrates these stages at scale 3. Performance optimization techniques specific to large codebases 4. Real examples (Chromium, LLVM, Linux kernel) Let me structure the answer: 1. 从单个文件说起:编译的四阶段(预处理 → 编译 → 汇编 → 链接) 2. 大型软件的挑战:不是阶段的线性放大 3. 构建系统的角色(Make → CMake/Ninja → Bazel/Buck) 4. 关键优化技术(增量编译、并行编译、分布式编译/缓存、Unity Build、预编译头等) 5. 链接阶段的特殊性 6. 实际案例(Chromium、LLVM) I'll also search for a couple more specific details.
网页搜索结果数: 6, 查询: "Linux kernel build process Kbuild recursive make config Kconfig"
{
  "results": [
    {
      "title": "Exploring the Linux kernel: The secrets of Kconfig/kbuild | Opensource.com",
      "url": "https://opensource.com/article/18/10/kbuild-and-kconfig",
      "description": "# Exploring the Linux kernel: The secrets of Kconfig/kbuild. The Linux kernel config/build system, also known as Kconfig/kbuild, has been around for a long time, ever since the Linux kernel code migrated to Git. As supporting infrastructure, however, it is seldom in the spotlight; even kernel developers who use it in their daily work never really think about it. To explore how the Linux kernel is compiled, this article will dive into the Kconfig/kbuild internal process, explain how the .config file and the vmlinux/bzImage files are produced, and introduce a smart trick for dependency tracking. The first step in building a kernel is always configuration. | config | Update current config utilizing a line-oriented program |. | menuconfig | Update current config utilizing a menu-based program |. Logically, Kconfig's infrastructure has two parts: one implements a new language to define the configuration items (see the Kconfig files under the kernel source), and the other parses the Kconfig language and deals with configuration actions."
    },
    {
      "title": "Kconfig make config — The Linux Kernel 5.10.0-rc1+ documentation",
      "url": "https://www.infradead.org/~mchehab/kernel_docs/kbuild/kconfig.html",
      "description": "Kconfig make config¶. This file contains some assistance for using make *config . Use “make help” to list all of the possible configuration targets."
    },
    {
      "title": "1.3: Kernel build system | raspberry-pi-os - GitHub Pages",
      "url": "https://s-matyukevich.github.io/raspberry-pi-os/docs/lesson01/linux/build-system.html",
      "description": "Linux uses recursive build. This means that each subfolder of the Linux kernel can define it's own Makefile and Kconfig . Most of the nested Makefiles are"
    },
    {
      "title": "Items and components for Linux kernel build - LinkedIn",
      "url": "https://www.linkedin.com/pulse/items-components-linux-kernel-build-david-zhu-sduyc",
      "description": "When working with kbuild in a Linux kernel build, you will interact with the kbuild system, Makefiles, Kconfig files, the .config file, and various tools and"
    },
    {
      "title": "Linux Kernel Makefiles — The Linux Kernel documentation",
      "url": "https://docs.kernel.org/kbuild/makefiles.html",
      "description": "The kbuild Makefile specifies object files for vmlinux in the $(obj-y) lists. If a kernel module is built from several source files, you specify that you want to build a module in the same way as above; however, kbuild needs to know which object files you want to build your module from, so you have to tell it by setting a $(-y) variable. In this example, the module name will be isdn.o. Kbuild will compile the objects listed in $(isdn-y) and then run `$(LD) -r` on the list of these files to generate isdn.o. Due to kbuild recognizing $(-y) for composite objects, you can use the value of a `CONFIG_` symbol to optionally include an object file as part of a composite object. Kbuild is not executing in the directory where the Makefile is located, so all custom rules shall use a relative path to prerequisite files and target files."
    },
    {
      "title": "Step 3 – Configuring the Linux kernel",
      "url": "https://subscription.packtpub.com/book/cloud-and-networking/9781803232225/2/ch02lvl1sec12/step-3-configuring-the-linux-kernel",
      "description": "The infrastructure that the Linux kernel uses to configure the kernel is known as the Kconfig system, and to build it, there is the Kbuild infrastructure."
    }
  ]
}
网页搜索结果数: 6, 查询: "ThinLTO vs full LTO difference compilation speed"
{
  "results": [
    {
      "title": "Consider ThinLTO vs LTO vs no LTO with respect to compile time ...",
      "url": "https://github.com/tikv/tikv/issues/4163",
      "description": "Thin LTO is much faster than full LTO, changes individual benchmark performance in the unpredictable ways LTO does. Surprisingly, it may"
    },
    {
      "title": "(PDF) ThinLTO: Scalable and incremental LTO - ResearchGate",
      "url": "https://www.researchgate.net/publication/314105281_ThinLTO_Scalable_and_incremental_LTO",
      "description": "Although ThinLTO [34] is more widely adopted in industry due to lower compile time [30,41,44], we chose full LTO to prove TRRIP is able to give performance"
    },
    {
      "title": "Recommendation for LTO type (thin or full) to use for Fedora",
      "url": "https://groups.google.com/g/clang-built-linux/c/N7OmA4ihJI8",
      "description": "ThinLTO is used by Android and is much less resource-heavy at build time. Full LTO exists as an option mostly to be able to a/b test results and bugs."
    },
    {
      "title": "Does LTO really have the same inlining opportunities as code in the ...",
      "url": "https://www.reddit.com/r/cpp/comments/1gxv4m9/does_lto_really_have_the_same_inlining",
      "description": "In general, in our measurements ThinLTO builds are almost as performant as Full LTO builds, but there's still a slight delta between them."
    },
    {
      "title": "ThinLTO — Clang 23.0.0git documentation",
      "url": "https://clang.llvm.org/docs/ThinLTO.html",
      "description": "## ThinLTO. *ThinLTO* compilation is a new type of LTO that is both scalable and incremental. However, monolithic LTO implements this by merging all input into a single module, which is not scalable in time or memory, and also prevents fast incremental compiles. In ThinLTO mode, as with regular LTO, clang emits LLVM bitcode after the compile phase. The ThinLTO bitcode is augmented with a compact summary of the module. By default, linkers that support ThinLTO are set up to launch the ThinLTO backends in threads. So the usage model is not affected as the distinction between the fast serial thin link step and the backends is transparent to the user. For more information on the ThinLTO design and current performance, see the LLVM blog post ThinLTO: Scalable and Incremental LTO. ### Clang/LLVM¶. For the latest ThinLTO support, build a recent version of clang and LLVM. * ELF ld.lld (as of LLVM 5.0): `-Wl,--thinlto-cache-dir=/path/to/cache`."
    },
    {
      "title": "Link-Time Optimisation (LTO) - Hacker News",
      "url": "https://news.ycombinator.com/item?id=38215535",
      "description": "For our (Rust) project, thin LTO doesn't cost much extra in compile time and gives a few percent speedup. But it's not a terribly optimized code"
    }
  ]
}
thought
I now have a comprehensive picture. Let me structure the answer carefully. I'll organize it as follows: 1. **从单个文件到可执行文件的四个阶段** - the classic view 2. **大型软件的编译不只是四阶段的线性放大** - the unique challenges 3. **构建系统的角色** - the orchestration layer 4. **关键的加速技术** - how they deal with scale 5. **链接——大型项目真正的瓶颈** - special attention to linking 6. **三个真实案例** - Chromium, LLVM, Linux Kernel 7. **总结** Let me write this out now.

大型软件的编译过程

大型软件的编译远不只是「把代码翻译成机器码」这么简单。把它拆成两层来理解会清晰很多:底层是编译器链(预处理→编译→汇编→链接)对单个文件做的事;上层是构建系统如何把这套流程在数万甚至数十万个文件上高效地调度起来。


一、底层:单个源文件经历了什么

以 C/C++ 为例(大多数大型系统软件的主力语言),一个 .c / .cpp 文件到可执行文件要经过四个阶段:

1. 预处理(Preprocessing)

处理 #include(头文件展开)、#define(宏替换)、#ifdef(条件编译)等指令。这一步的输出是「纯粹的 C/C++ 源码」——没有预处理指令,但体积会膨胀很多,因为一个 #include <iostream> 可能展开出数万行。

2. 编译(Compilation)

编译器(GCC/Clang/MSVC)将预处理后的源码翻译成汇编代码。这是最耗 CPU 的阶段,涉及词法分析、语法分析、语义分析、中间表示(IR)生成、优化等子步骤。编译器在这里做大量优化:内联、循环展开、死代码消除、常量折叠等。

3. 汇编(Assembly)

汇编器将汇编代码转换成机器码,输出目标文件.o / .obj)。目标文件里是二进制机器指令、数据段、符号表(记录了这个文件提供了哪些函数/变量,又引用了哪些外部符号)。

4. 链接(Linking)

链接器把成百上千个目标文件和库拼成一个可执行文件(ELF、PE、Mach-O)。核心工作:符号解析(把各 .o 文件里「我调用了 printf,但它在哪」的问号填上地址)和重定位(修正地址引用)。


二、大型软件的特殊挑战:为什么不是简单放大

单文件编译四阶段看起来很简单。但当项目有几万个源文件时,新的问题出现了:

  • 依赖图复杂度A.cpp 改了,哪些 .o 需要重新编译?哪些可执行文件需要重新链接?
  • 编译时间:Chromium 全量编译在普通 4 核机器上约需 13 小时 Big Project Build Times – Chromium,即使用 M2 MacBook Pro 也要近 2 小时
  • 链接时间:当链接器面对几千个 .o 和上百个库时,符号解析本身就变得极慢——甚至可能超过编译时间。
  • 内存压力:LTO(链接时优化)需要将整个程序的 IR 同时加载到内存,对大型项目可能耗上百 GB 内存。
  • 头文件地狱:C++ 的头文件展开机制导致每个翻译单元(translation unit)都要重新解析大量重复代码。

三、构建系统:编译的「大脑」

直接把命令行里的 gcc 一个一个调用来编译是不现实的。构建系统的作用是把整个编译过程建模成一棵依赖 DAG(有向无环图),然后自动决定执行什么、以什么顺序执行、什么可以并行执行。

构建系统的演进

代际 代表 特点
第一代 GNU Make 基于时间戳的增量判断;手写依赖规则容易出错;递归 Make 产生大量串行瓶颈
第二代 CMake + Ninja CMake 生成 Ninja 文件,Ninja 极速执行;依赖追踪更精确
第三代 Bazel(Google)、Buck2(Meta)、Pants 基于内容哈希的缓存;声明式规则;原生支持分布式编译和远程缓存

Bazel 为例,它在一次构建中分为三个核心阶段 Bazel: Action Graph

  1. Loading 阶段:解析 BUILD 文件,构建目标依赖图(target graph)。
  2. Analysis 阶段:生成 action graph——每个节点是一个具体动作(编译 foo.cpp、链接 bar.so),附带完整命令行和输入/输出文件。
  3. Execution 阶段:实际执行这些动作。Bazel 可以在这里将 action 分发到远程机器上执行。

Bazel 最关键的特性是基于内容哈希的缓存:如果 foo.cpp 的内容、编译器版本、编译选项、所有依赖头的哈希都跟上次一样,就直接从缓存拿结果,跳过编译。这跟 Make 的「看时间戳」有本质区别——更精确、更适合大规模分布式场景 Bazel: Distributed Builds


四、关键加速技术

大型项目的编译速度是核心痛点,工业界发展出了一整套技术栈:

4.1 并行编译

最基础的手段。make -jN 或 Ninja 会将无依赖关系的编译任务同时跑在多个核上。但并行度受限于依赖链的宽度:如果所有文件最终都依赖一个大头文件,那改了这个头文件后,并行编译前必须等它先处理完。

4.2 增量编译(Incremental Build)

只重编译「真正受影响」的文件。传统 Make 基于时间戳判断——但时间戳可能不可靠(切个分支、换个编译器就全乱了)。Bazel 这类系统用内容哈希做增量判断,可靠性大幅提升。一项对 383 个 Bazel 项目的研究显示,增量编译相比全量编译的中位加速达到 4.22x(通用 CI 缓存)到 4.71x(Bazel 专用缓存) Does Using Bazel Help Speed Up Continuous Integration Builds?

4.3 分布式编译与远程缓存

这是 Google/Meta 级别的终极方案:

  • 远程缓存:编译结果(.o 文件)存在共享服务器上。团队成员和 CI 之间可以共享——你编译了一个别人今天上午已经编译过的文件,直接下载 .o
  • 远程执行:编译任务本身分发到远程机器集群上执行。Google 内部用的 Blaze(Bazel 的前身)配合 Goma 远程编译服务,将 Chromium 的全量编译从 13 小时压到分钟级。

4.4 Unity Build / Jumbo Build

把多个 .cpp 文件 #include 进一个「超级翻译单元」一次性编译。好处是:减少重复的头文件解析、减少重复模板实例化、减少 .o 文件数量(从而减轻链接负担)。有报告称可以将编译速度提升 2 倍以上 C++ Weekly: Unity Builds。代价是:改一个文件就要重编整个合并单元;全局命名冲突(两个 .cpp 里定义了同名的 static 函数);不适合持续集成场景。

4.5 预编译头(PCH)

把不常变动的头文件(如标准库、大型第三方库)预先编译成二进制中间格式,后续编译时直接加载,跳过重复的解析开销。CMake 原生支持 PCH,通常能带来 20-40% 的编译加速 Qt Blog: Precompiled Headers and Unity Builds in CMake


五、链接——真正的瓶颈所在

在大型 C++ 项目里,链接经常比编译更让人头疼。一个 reddit 用户描述 Chromium 的增量构建:「80-90% 的时间都花在了链接巨大的 browser.dll 上」 r/cpp: How long are your compilation & linking times?

静态链接 vs 动态链接

静态链接 动态链接(.so/.dll
链接时间 长(全量处理) 短(符号只是占位引用)
运行时性能 略好(无间接跳转) 极轻微开销(PLT/GOT)
增量构建 改一个小文件 → 重新完整链接 Component build:只重链受影响的小 .so
分发便利性 单文件自包含 需附带一堆 .so

Chromium 开发时默认使用 component build 模式——将浏览器拆成几十个 .so/.dll,改了某个模块只重链对应的那个动态库。链接时间从「等十几分钟」变成「等几十秒」,极大提升了开发迭代速度。发布时则切回静态 + LTO 以获得最优性能。

LTO 与 ThinLTO

Link-Time Optimization(链接时优化) 允许编译器在链接阶段跨翻译单元做优化——比如跨文件内联、跨文件死代码消除。但全量 LTO 需要将整个程序的 IR 加载到内存中,对 Chromium 级别的项目内存消耗极恐怖。

ThinLTO 是 LLVM 给出的解决方案:编译阶段为每个 .o 附加一个紧凑的模块摘要,链接时先用摘要做全局分析,然后并行地对各个模块做后端优化 Clang: ThinLTO。ThinLTO 在保持接近全量 LTO 优化效果的同时,大幅降低了内存和时间开销,已被 Android 等大型项目采用。


六、三个真实案例

Chromium

  • 源码规模:约 25,000+ 个 C/C++ 源文件,超过 1,800 万行代码
  • 构建工具:GN(生成 Ninja 文件)+ Ninja;Google 内部用 Blaze
  • 全量编译:约 50,000+ 个编译任务,M4 MacBook Pro 约 6 分钟(M2 约 1 小时 49 分钟),普通 4 核机器约 13 小时 Chromium Build Times - Solomon Kinard
  • 增量编译:改一个 .cpp 通常重编 + 重链在几分钟以内(component build 下几十秒)
  • 关键策略:GN + Ninja 并行、component build(开发时)、ThinLTO(发布时)、Google 内部使用 Goma 远程编译

LLVM / Clang

  • 全量构建 LLVM 本身就是一个大型编译任务:优化构建 llc48 秒(用户态时间约 13 分钟分布式),Debug 构建约 9 分钟 LLVM Discourse: If LLVM is so slow
  • 用 CMake + Ninja,也支持 Bazel 构建
  • LLVM 自身的编译速度也是 LLVM 开发者持续优化的目标

Linux Kernel

  • 构建系统:Kbuild(基于 Make 的递归构建系统)+ Kconfig 配置系统 Exploring the Linux kernel: Kconfig/kbuild
  • 每个子目录有自己的 MakefileKconfig,顶层 make menuconfig 生成 .config,然后递归进入各子目录编译
  • 使用 make -j$(nproc) 做并行编译,全量编译在现代机器上约 10-30 分钟
  • 内核模块(.ko)的编译与主内核镜像(vmlinuxbzImage)是分开的流程

七、总结:一个全景视图

把整个流程串起来,一次大型软件的编译大致是这样的:

配置阶段
  │  CMake/GN/Kconfig 读取项目描述 → 生成构建文件
  ▼
构建系统启动(Bazel / Ninja / Make)
  │  解析依赖图 → 判断哪些需要重新编译 → 决定并行调度
  ▼
对每个需重编译的源文件:
  │  预处理器  →  编译器  →  汇编器  →  .o 文件
  │  (展开头文件)  (源码→IR→汇编)  (汇编→机器码)
  ▼
链接阶段
  │  链接器收集所有 .o + 库  →  符号解析 + 重定位
  │  (可选:LTO/ThinLTO 在此阶段做跨模块优化)
  ▼
可执行文件 / 动态库 / 内核镜像

整个过程的核心矛盾是:依赖图的复杂性与编译任务的计算密集性。解决思路则是:精确的依赖追踪(知道什么需要重编)+ 最大化并行度(多核/多机)+ 缓存复用(不重复劳动)+ 减少冗余解析(PCH、Unity Build)


尚需核实的点:本文中 Chromium、LLVM 的具体编译时间数据来自不同硬件平台和时间点,仅供量级参考,不是精确基准。实际时间高度依赖于 CPU 核心数、内存、磁盘 I/O 和具体编译选项。如果你的场景有特定的项目或编译器,可以进一步深入。

71045 tokens · $0.0099