Profile-guided Optimization

rustc supports doing profile-guided optimization (PGO). This chapter describes what PGO is, what it is good for, and how it can be used.

What Is Profile-Guided Optimization?

The basic concept of PGO is to collect data about the typical execution of a program (e.g. which branches it is likely to take) and then use this data to inform optimizations such as inlining, machine-code layout, register allocation, etc. Optimization levels -Copt-level=2 and above are recommended for use of profile guided optimization.

rustc supports profile guided optimization with two different kinds of profiling. A sampling profiler can generate a profile with very low runtime overhead, or you can build an instrumented version of the code that collects more detailed profile information. Both kinds of profiles can provide execution counts for instructions in the code and information on branches taken and function invocation.

There are different ways of collecting data about a program’s execution. One is to run the program inside a profiler (such as perf) and another is to create an instrumented binary, that is, a binary that has data collection built into it, and run that.

Differences Between Instrumentation and Sampling

Although both techniques are used for similar purposes, there are important differences between the two:

Profile data generated with one cannot be used by the other, and there is no conversion tool that can convert one to the other. So, a profile generated via -Cprofile-generate must be used with -Cprofile-use. Similarly, sampling profiles generated by external profilers must be converted and used with -Cprofile-sample-use.
Sampling profiles must be generated by an external tool. The profile generated by that tool must then be converted into a format that can be read by LLVM. The section on sampling profilers describes one of the supported sampling profile formats.

Profiling with Instrumentation

Generating a PGO-optimized program involves following a workflow with four steps:

Compile the program with instrumentation enabled (e.g. rustc -Cprofile-generate=/tmp/pgo-data -O main.rs)
Run the instrumented program (e.g. ./main) which generates a default_<id>.profraw file
Convert the .profraw file into a .profdata file using LLVM’s llvm-profdata tool
Compile the program again, this time making use of the profiling data (for example rustc -Cprofile-use=merged.profdata -O main.rs)

An instrumented program will create one or more .profraw files, one for each instrumented binary. E.g. an instrumented executable that loads two instrumented dynamic libraries at runtime will generate three .profraw files. Running an instrumented binary multiple times, on the other hand, will re-use the respective .profraw files, updating them in place.

These .profraw files have to be post-processed before they can be fed back into the compiler. This is done by the llvm-profdata tool. This tool is most easily installed via

rustup component add llvm-tools-preview

Note that installing the llvm-tools-preview component won’t add llvm-profdata to the PATH. Rather, the tool can be found in:

~/.rustup/toolchains/<toolchain>/lib/rustlib/<target-triple>/bin/

Alternatively, an llvm-profdata coming with a recent LLVM or Clang version usually works too.

The llvm-profdata tool merges multiple .profraw files into a single .profdata file that can then be fed back into the compiler via -Cprofile-use:

# STEP 1: Compile the binary with instrumentation
rustc -Cprofile-generate=/tmp/pgo-data -O main.rs

# STEP 2: Run the binary a few times, maybe with common sets of args.
# Each run will create or update `.profraw` files in /tmp/pgo-data
./main mydata1.csv
./main mydata2.csv
./main mydata3.csv

# STEP 3: Merge and post-process all the `.profraw` files in /tmp/pgo-data
llvm-profdata merge -o merged.profdata /tmp/pgo-data

# STEP 4: Use the merged `.profdata` file during optimization. All `rustc`
# flags have to be the same.
rustc -Cprofile-use=./merged.profdata -O main.rs

A Complete Cargo Workflow

Using this feature with Cargo works very similar to using it with rustc directly. Again, we generate an instrumented binary, run it to produce data, merge the data, and feed it back into the compiler. Some things of note:

We use the RUSTFLAGS environment variable in order to pass the PGO compiler flags to the compilation of all crates in the program.
We pass the --target flag to Cargo, which prevents the RUSTFLAGS arguments to be passed to Cargo build scripts. We don’t want the build scripts to generate a bunch of .profraw files.
We pass --release to Cargo because that’s where PGO makes the most sense. In theory, PGO can also be done on debug builds but there is little reason to do so.
It is recommended to use absolute paths for the argument of -Cprofile-generate and -Cprofile-use. Cargo can invoke rustc with varying working directories, meaning that rustc will not be able to find the supplied .profdata file. With absolute paths this is not an issue.
It is good practice to make sure that there is no left-over profiling data from previous compilation sessions. Just deleting the directory is a simple way of doing so (see STEP 0 below).

This is what the entire workflow looks like:

# STEP 0: Make sure there is no left-over profiling data from previous runs
rm -rf /tmp/pgo-data

# STEP 1: Build the instrumented binaries
RUSTFLAGS="-Cprofile-generate=/tmp/pgo-data" \
    cargo build --release --target=x86_64-unknown-linux-gnu

# STEP 2: Run the instrumented binaries with some typical data
./target/x86_64-unknown-linux-gnu/release/myprogram mydata1.csv
./target/x86_64-unknown-linux-gnu/release/myprogram mydata2.csv
./target/x86_64-unknown-linux-gnu/release/myprogram mydata3.csv

# STEP 3: Merge the `.profraw` files into a `.profdata` file
llvm-profdata merge -o /tmp/pgo-data/merged.profdata /tmp/pgo-data

# STEP 4: Use the `.profdata` file for guiding optimizations
RUSTFLAGS="-Cprofile-use=/tmp/pgo-data/merged.profdata" \
    cargo build --release --target=x86_64-unknown-linux-gnu

Troubleshooting

It is recommended to pass -Cllvm-args=-pgo-warn-missing-function during the -Cprofile-use phase. LLVM by default does not warn if it cannot find profiling data for a given function. Enabling this warning will make it easier to spot errors in your setup.
There is a known issue in Cargo prior to version 1.39 that will prevent PGO from working correctly. Be sure to use Cargo 1.39 or newer when doing PGO.

Profiling with Sampling

Sampling profilers are used to collect runtime information, such as hardware counters, while your application executes. They are typically very efficient and do not incur a large runtime overhead. The sample data collected by the profiler can be used during compilation to determine what the most executed areas of the code are.

Using the data from a sample profiler requires some changes in the way a program is built. Before the compiler can use profiling information, the code needs to execute under the profiler. The following is the usual build cycle when using sample profilers for optimization:

Build the code with source line table information. You can use all the usual build flags that you always build your application with. The only requirement is that DWARF debug info including source line information is generated. This DWARF information is important for the profiler to be able to map instructions back to source line locations. The accuracy of this DWARF information can be improved with the (unstable) -Zdebuginfo-for-profiling option. For example:

rustc -Cdebuginfo=line-tables-only -Zdebuginfo-for-profiling -O main.rs

Additionally emitted by debuginfo-for-profiling information helps the compiler distinguish different basic blocks during the optimization even if they have the same source line location, that consequently improves the accuracy of the profile-guided optimization.

Run the executable under a sampling profiler. The specific profiler you use does not really matter, as long as its output can be converted into the format that the LLVM optimizer understands.

Two such profilers are the Linux Perf profiler and Intel’s Sampling Enabling Product (SEP), available as part of Intel VTune. While Perf is Linux-specific, SEP can be used on Linux, Windows, and FreeBSD.

The LLVM tool llvm-profgen can convert output of either Perf or SEP. An external project, AutoFDO, also provides a create_llvm_prof tool which supports Linux Perf output.

When using Perf:

perf record -b -e BR_INST_RETIRED.NEAR_TAKEN:uppp ./main

If the event above is unavailable, branches:u is probably next-best.

Note the use of the -b flag. This tells Perf to use the Last Branch Record (LBR) to record call chains. While this is not strictly required, it provides better call information, which improves the accuracy of the profile data.

When using SEP:

sep -start -out code.tb7 -ec BR_INST_RETIRED.NEAR_TAKEN:precise=yes:pdir \
    -lbr no_filter:usr -perf-script brstack -app ./main

This produces a code.perf.data.script output which can be used with llvm-profgen’s --perfscript input option.

Convert the collected profile data to LLVM’s sample profile format. This is currently supported via the AutoFDO converter create_llvm_prof. Once built and installed, you can convert the perf.data file to LLVM using the command:

create_llvm_prof --binary=./main --out=main.prof

This will read perf.data and the binary file ./main and emit the profile data in main.prof. Note that if you ran perf without the -b flag, you need to use --use_lbr=false when calling create_llvm_prof.

Alternatively, the LLVM tool llvm-profgen can also be used to generate the LLVM sample profile:

llvm-profgen --binary=./main --output=main.prof --perfdata=perf.data

Please note, perf.data must be collected with -b flag to Linux perf for the above step to work.

When using SEP the output is in the textual format corresponding to llvm-profgen --perfscript. For example:

llvm-profgen --binary=./main --output=main.prof \
    --perfscript=main.perf.data.script

Build the code again using the collected profile. This step feeds the profile back to the optimizers. This should result in a binary that executes faster than the original one. Note that you are not required to build the code with the exact same arguments that you used in the first step. The only requirement is that you build the code with the same debug info options and -Cprofile-sample-use.

rustc -Cprofile-sample-use=main.prof -Zdebuginfo-for-profiling -O main.rs

Note that Sample-based PGO in rustc is mostly tested on x86-64 Linux platforms. It should work on other hardware architectures and operating systems but it’s not heavily tested yet.

Community Maintained Tools

As an alternative to directly using the compiler for Profile-Guided Optimization, you may choose to go with cargo-pgo, which has an intuitive command-line API and saves you the trouble of doing all the manual work. You can read more about it in cargo-pgo repository. For now, cargo-pgo supports only Instrumentation PGO.

For the sake of completeness, here are the corresponding steps using cargo-pgo for Instrumentation PGO:

# Install if you haven't already
cargo install --locked cargo-pgo

cargo pgo build
cargo pgo optimize

These steps will do the following just as before:

Build an instrumented binary from the source code.
Run the instrumented binary to gather PGO profiles.
Use the gathered PGO profiles from the last step to build an optimized binary.

The rustc book