How we brought our Rust CI from 20 minutes to less than 5 for Comper
How we brought our Rust CI from 20 minutes to less than 5 for Comper
AI Disclaimer
Ultimately it's just a list of things we did to optimize our Rust compilation, which is a problem that everyone that builds large Rust projects has. Our compilation jobs alone sometimes took 9 minutes. Now we reliably do it in just over 2 minutes. The tests then run for another 2.5.
Introduction
Comper currently contains 176,496 lines of Rust code across 975 files and 18 crates.
The test suite contains:
762 Rust unit tests
377 Rust integration tests
985 frontend unit and component tests
91 Playwright end-to-end tests
9 additional ignored Rust tests
Our merge request pipeline builds the frontend, runs the frontend tests, compiles every Rust test binary, runs three backend test shards, and runs two end-to-end test shards.
The complete pipeline now takes about 4 minutes and 40 seconds. It used to take around 20 minutes.
There was no single compiler flag that fixed it. The largest improvement came from running GitLab runners on our own Kubernetes cluster and deliberately using their local disks as persistent caches.
The biggest hack: node-local caches
GitLab normally restores a cache before a job and uploads it afterwards. This works well when a cache consists of a few large files. It works much less well for Rust directories containing tens of thousands of small files.
For every job GitLab had to:
Walk the cache directory
Package or compress it
Upload it
Download it in another job
Extract it again
At some point we were spending almost as much time moving Rust caches as compiling Rust.
Our GitLab runners run on a dedicated Kubernetes node pool. Every job pod mounts a directory from the local filesystem of its node:
[[runners]]
request_concurrency = 4
cache_dir = "/cache"
[runners.kubernetes]
namespace = "gitlab-runner"
[runners.kubernetes.node_selector]
nodepool = "gitlab-runners"
[[runners.kubernetes.volumes.host_path]]
name = "cache"
mount_path = "/cache"
host_path = "/var/lib/gitlab-runner/cache"This is an anti-pattern. Kubernetes pods are supposed to be disposable and independent of the node on which they run. Our cache is neither.
We accepted this because the cache is only an optimization. When a node disappears, the next build starts cold but still works. Our jobs are scheduled on a small dedicated runner pool, so they usually return to a node that already has the data.
Cargo and kache use separate directories on this mounted disk:
variables:
CARGO_HOME: "/cache/comper/cargo"
KACHE_CACHE_DIR: "/cache/comper/kache"
KACHE_MAX_SIZE: "100GiB"
CARGO_INCREMENTAL: "0"CARGO_HOME contains the registry and git dependencies. Kache contains compiler outputs.
We no longer cache the complete Cargo target/ directory. Kache gives us most of its useful contents without requiring GitLab to zip, transfer and unzip the whole directory.
Coverage is the exception. It still caches a limited set of Cargo fingerprints and dependencies.
Replacing sccache with kache
We originally used sccache. It worked, but its Rust cache keys included the compilation directory.
Every GitLab job gets a fresh checkout path. We also use lots of local Git worktrees. In both cases sccache only achieved about a 60% cross-directory hit rate.
We replaced it with kache, a new and mostly vibe-coded compiler cache. Kache's keys are independent of the checkout directory, so an artifact compiled in one worktree can be reused in another.
Our measurements were:
Cross-worktree build: 84 seconds with sccache, 40 seconds with kache
Warm CI compile: 59 seconds with sccache, 34 seconds with kache
CI wraps every rustc invocation with kache:
variables:
RUSTC_WRAPPER: "kache"
KACHE_S3_BUCKET: "comper-sccache"
KACHE_S3_ENDPOINT: "https://s3.sbg.io.cloud.ovh.net"
KACHE_S3_REGION: "sbg"
KACHE_S3_PREFIX: "kache"
KACHE_NAMESPACE: "comper"The node-local directory is the fast path. S3 shares artifacts between nodes and keeps the cache available when a node is replaced.
Kache requires a daemon for remote cache access:
.ensure_kache: &ensure_kache
- mkdir -p "${CARGO_HOME}" "${KACHE_CACHE_DIR}"
- command -v kache
- kache --version
- kache daemon startWe also tried bulk-prefetching the S3 cache at the start of every job. It downloaded too much and was not reliably faster, so we removed it.
Incremental compilation is disabled in CI
Incremental compilation is great when a developer repeatedly changes the same checkout. CI jobs start with a clean Cargo target directory.
In that situation incremental compilation added 15 to 20% overhead, created larger artifacts and prevented kache from caching some compiler outputs.
We explicitly disable it:
variables:
CARGO_INCREMENTAL: "0"Locally we make the opposite choice. More on that later.
Compile all tests once
Originally every backend test shard compiled the Rust workspace. Each shard also restored and uploaded its own caches.
That multiplied several minutes of compilation and cache transfer by the number of shards.
Now one job compiles all test binaries and creates a nextest archive:
build-test-binaries:
stage: test
image: ${CI_DOCKER_IMAGE}
needs: []
variables:
FF_USE_FASTZIP: "true"
ARTIFACT_COMPRESSION_LEVEL: "fastest"
script:
- *ensure_kache
- cargo build --profile test -p migration --bin migration-cli --locked
- >-
cargo-nextest nextest archive
--workspace
--cargo-profile test
--profile ci
--archive-file nextest-archive.tar.zst
artifacts:
paths:
- nextest-archive.tar.zstThe archive contains:
All Rust test binaries
comper-serverThe migration CLI
This is the only large Rust artifact transferred between jobs.
We initially built comper-server separately before creating the nextest archive. That caused Cargo to compile the root crate twice because feature unification differed between the normal build and the test build.
Removing one innocent-looking cargo build command saved a complete compilation of the application.
Our nextest fork
Our integration tests make HTTP requests to a real Comper server. Starting a separate server for every test would give excellent isolation, but it would also make the suite extremely slow.
Starting one server outside nextest was not ideal either. Every test would have to wait for it, including tests that did not need it. We would also lose nextest's normal process management, output capture and test scheduling.
We built a server-wrapper feature in our nextest fork. A server wrapper belongs to a group of tests rather than one test. Nextest starts it lazily when the first matching test is ready, probes it until it is healthy, captures its output and shuts it down when the run finishes.
Our configuration looks like this:
experimental = ["server-wrappers"]
[scripts.server-wrapper.test-server]
command = ["./scripts/start_test_server.sh"]
probe = {
url = "http://127.0.0.1:5150/_readiness",
interval = "100ms",
timeout = "500s"
}
capture-stdout = true
capture-stderr = true
[[profile.default.scripts]]
filter = "all()"
server-wrapper = "test-server"One Comper server is started per CI shard and shared by the tests in that shard. The tests still run in parallel, which also exercises concurrency inside the application.
At the time we introduced it, more than 150 integration tests ran in about 40 seconds with one shared server. Running every test with its own server would have taken around 15 minutes.
There are trade-offs. A test can corrupt shared in-process state and affect another test. Our database and fixtures have to support concurrent tests. A crashed server also affects the rest of the shard.
For our suite this is a good trade-off. The shared server is part of the system under test, and concurrent requests are closer to how it runs in production.
We proposed the feature upstream in nextest discussion #3330: Feature request: sidecar / server wrapper for client/server tests. The discussion also covers alternatives such as creating an on-disk seed and starting a fresh service for each test, process lifecycle questions, and use cases with heavyweight servers that can take 90 seconds to start.
The maintainers were interested in accepting some form of the feature. Until its lifecycle and isolation contracts are worked out, we are happy to maintain the fork.
Run the backend tests without Cargo
The backend tests are divided between three GitLab jobs.
Those jobs download the archive and run their assigned partition:
back-end-tests:
stage: test
image: ${CI_DOCKER_IMAGE}
parallel: 3
needs:
- job: build-test-binaries
artifacts: true
script:
- mkdir -p "${NEXTEST_ARCHIVE_DIR}"
- >-
cargo-nextest nextest run
--archive-file nextest-archive.tar.zst
--extract-to "${NEXTEST_ARCHIVE_DIR}"
--extract-overwrite
--workspace-remap .
--partition "count:${CI_NODE_INDEX}/${CI_NODE_TOTAL}"
--profile ci
--no-fail-fastThese jobs never invoke rustc.
They extract the binaries, start PostgreSQL, Keycloak and Mailhog, and run their part of the suite.
Reuse the binaries for end-to-end tests
Our Playwright jobs use the same precompiled server and migration CLI. There is no additional Rust build for E2E.
The browser suite runs in two GitLab shards. Each shard runs two Playwright workers:
const ciNodeIndex = process.env.CI_NODE_INDEX
const ciNodeTotal = process.env.CI_NODE_TOTAL
export default defineConfig({
fullyParallel: true,
retries: process.env.CI ? 1 : 0,
workers: process.env.CI ? 2 : undefined,
shard:
ciNodeIndex && ciNodeTotal
? {
current: Number(ciNodeIndex),
total: Number(ciNodeTotal),
}
: undefined,
})We also changed the fixtures to create one board per Playwright worker instead of one board per test. Creating and deleting a board took between 300 and 600 milliseconds. Repeating that for every test added up.
Navigation helpers use domcontentloaded where waiting for every image and asset provides no additional confidence.
Mold and the parallel rustc frontend
CI uses LLVM for code generation, mold for linking and rustc's parallel frontend:
variables:
RUSTFLAGS: "-C link-arg=-fuse-ld=mold -Z threads=8"The parallel frontend reduced a full compilation by about 25%.
Eight and sixteen threads performed about the same in our measurements. We use eight in CI to reduce peak memory when multiple jobs are running on the same node.
Mold only changes the linker. It does not change Rust's code-generation backend or reduce the quality of the generated code.
Linux and macOS development
Local development has different priorities. We care about the edit-compile-run loop more than runtime performance.
Linux uses wild and sixteen rustc frontend threads. Apple Silicon uses the platform linker and ten frontend threads:
[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=--ld-path=wild", "-Z", "threads=16"]
[target.aarch64-apple-darwin]
rustflags = ["-Zthreads=10"]We tested multiple combinations with the compile cache disabled:
config full build incremental
cranelift + wild 145s / 155s 8s
cranelift, no fast linker 161s 20s
LLVM + wild 158s / 169s 8s
LLVM, no fast linker 182s 23s
LLVM + mold 157s 8s
LLVM + wild + threads 121s 8s
LLVM + wild + threads, deps O3 285s 6sThe fast linker saves approximately 12 to 15 seconds on every rebuild. The parallel frontend mainly improves full builds.
We previously used Cranelift for local development. A year ago it made a large difference. With our current nightly toolchain, Cranelift and LLVM both produced an eight-second incremental result. Cranelift only saved about 8% on a full build.
We removed it.
Local development, CI and production now all use LLVM. The difference is in profiles, linkers and caching, not in code-generation quality.
Cache dependencies locally, keep our crates incremental
Wrapping every local rustc invocation with kache caused another problem.
Kache cannot cache incremental output, so it strips Cargo's incremental compilation flags. This meant every edit to one of our own crates paid for a full code-generation pass.
Kache's own exclude patterns did not solve this. Excluded crates still went through its passthrough path, which also removed the incremental flags.
Cargo only enables incremental compilation for workspace crates. Registry and git dependencies are not incremental. We use that distinction in a small dispatcher:
case "${CARGO_MANIFEST_DIR:-}" in
*/registry/src/*|*/git/checkouts/*) exec kache "$@" ;;
*) exec "$@" ;;
esacDependencies go through kache. Our workspace crates go directly to rustc and keep incremental compilation.
This also allows a new worktree to reuse dependency artifacts compiled in another worktree without slowing down the normal editing loop.
A warm full local build measured around 112 seconds. A one-line edit measured around 19 seconds before the final linker changes and eight seconds for an edit isolated to comper-api afterwards.
Splitting one giant crate into 18 crates
We also split the application into 18 Rust crates.
This was by far the largest change.
The original root crate contained HTTP controllers, database access, background workers, LLM pipelines, authentication, repository analysis and runtime infrastructure. Everything depended on everything else.
The root crate went from approximately 54,600 lines to 34,300 and eventually to 5,700 lines. The workspace now contains crates such as:
comper-apicomper-dbcomper-domaincomper-workerscomper-runtimecomper-authcomper-agentcomper-repo-intelligence
This helped local compilation by limiting invalidation when changing code near the top of the dependency graph. It also gave rustc more independent compilation units that can be built in parallel.
It did not produce the spectacular clean-CI improvement we expected.
CI still builds the complete workspace. Changing comper-db, for example, invalidates most crates above it. Splitting a large crate does not help when all the resulting crates still need to be rebuilt.
The architectural improvement was enormous. The direct clean-build improvement was much smaller. Linkers, caching and removing duplicated builds produced clearer benchmark results than the monumental crate split.
We do not optimize dependencies in development
A commonly recommended Cargo configuration is:
[profile.dev.package."*"]
opt-level = 3This makes dependencies run faster but also makes them much slower to compile.
In our benchmark it changed a cold build from 121 seconds to 285 seconds. The incremental build only changed from eight to six seconds.
Dependencies are not rebuilt when editing our own crates, so we would pay more than two additional minutes on a cold build for a two-second improvement that rarely matters.
Normal development therefore uses unoptimized dependencies. We have a separate release-profile task when we need the server itself to run quickly.
Production still gets full LLVM optimization
Production is built with Cargo's release profile:
build-release:
script:
- *ensure_kache
- cargo build --release --bin comper-server --locked
- cp target/release/comper-server ./comper-server
- cargo build --release -p migration --bin migration-cli --locked
- cp target/release/migration-cli ./migration-cliThis is a normal optimized LLVM release build. It uses mold for linking, but mold does not change LLVM's optimizations or the generated machine code before linking.
The development optimizations do not reduce production quality.
Smaller improvements
The large changes account for most of the reduction, but we removed smaller pieces of unnecessary work too:
Only one pipeline runs per change. An MR does not also get a redundant branch pipeline.
Backend jobs use
rules: changesand can be skipped for frontend-only changes.Coverage only runs for tags.
Merge request pre-commit checks only inspect changed files.
The CI image contains Chromium instead of Chromium, Firefox and WebKit.
The CI image is built with nix2container and split into approximately 80 reusable OCI layers.
Jobs use a content-hashed CI image and do not wait for the image-building job.
libeatmydatadisablesfsyncinside CI containers.The Mermaid validation bundle is stored by content hash and reused.
We transfer one nextest archive instead of the archive plus duplicate server binaries.
Artifact compression uses the fastest setting because the compiled binaries do not compress enough to justify the CPU time.
We removed Axum's
debug_handlerattributes from frequently changed API code. That saved another one or two seconds per check.The backend model drift check runs once rather than once per test shard.
Playwright browsers are already present in the CI image, so E2E jobs do not download Chromium.
None of these changes reduced the pipeline from 20 minutes to less than five by itself. Together they removed a lot of waiting.
The resulting pipeline
The critical path is now:
Start jobs immediately with a prebuilt CI image.
Read Cargo dependencies and kache artifacts directly from the runner node.
Compile all Rust test binaries once using LLVM, mold and the parallel frontend.
Transfer one nextest archive.
Run three backend test shards without a compiler.
Run two Playwright shards using the same precompiled server.
Run frontend tests, linting and other checks in parallel.
The full merge request pipeline, including the browser tests, now finishes in approximately 4 minutes and 40 seconds.
The biggest improvement did not come from making Rust dramatically faster. It came from compiling things once, keeping caches close to the CPU, and no longer transferring the same thousands of files between every job.
Comments
Post a Comment