Command Line Reference

Compiler Gym includes several command line tools in the compiler_gym.bin module. This document provides an overview of each module. Run any of the commands with --helpfull to examine the full usage options.


This program can be used to query and run the CompilerGym services.

Listing available environments

List the environments that are available using:

$ python -m compiler_gym.bin.service --ls_env

Querying the capabilities of a service

Query the capabilities of a service using:

$ python -m compiler_gym.bin.service --env=<env>

For example:

$ python -m compiler_gym.bin.service --env=llvm-v0


| Dataset                    | Num. Benchmarks [#f1]_   | Description                  |
| benchmark://anghabench-v0  | 1,042,976                | Compile-only C/C++ functions |
| benchmark://blas-v0        | 300                      | Basic linear algebra kernels |

Observation Spaces

| Observation space        | Shape                                        |
| Autophase                | `Box(0, 9223372036854775807, (56,), int64)`  |
| AutophaseDict            | `Dict(ArgsPhi:int<0,inf>, BB03Phi:int<0,...` |
| BitcodeFile              | `str_list<>[0,4096.0])`                      |

The output is tabular summaries of the environment’s datasets, observation spaces, reward spaces, and action spaces, using reStructuredText syntax (

To query the capabilities of an unmanaged service, use --service. For example, query a service running the llvm-v0 environment at localhost:8080 using:

$ python -m compiler_gym.bin.service --env=llvm-v0 --service=localhost:8080

To query the capability of a binary that implements the RPC service interface, use the --local_service_binary flag:

$ python -m compiler_gym.bin.service --env=llvm-v0 --local_service_binary=/path/to/service/binary

Running a Service

This module can also be used to launch a service that can then be connected to by other environments. Start a service by specifying a port number using:

$ python -m compiler_gym.bin.service --env=llvm-v0 --run_on_port=7777

Environment can connect to this service by passing the <hostname>:<port> address during environment initialization time. For example, in python:

>>> env = compiler_gym.make("llvm-v0", service="localhost:7777")

Or at the command line:

$ python -m compiler_gym.bin.random_search --env=llvm-v0 --service=localhost:7777


Run a CompilerGym environment with text interface controls.

$ python -m compiler_gym.bin.manual_env --env=<env> [--benchmark=<name>] [--observation=<space>] [--reward=<space>]

The benchmark to use can be specified using --benchmark=<name>.

CompilerGym Shell Tutorial

This program gives a basic shell through which many of commands from CompilerGym can be executed. CompilerGym provides a simple Python interface to various compiler functions, enabling programs to be compiled in different ways and to make queries about those programs. The goal is to have a simple system for machine learning in compilers.

Setting a Benchmark, Reward and Observation

The CompilerGym operates on a program or benchmark. If not set on the command line, the benchmark can be specified in the shell with:

compiler_gym:cbench-v1/qsort> set_benchmark <benchmark-name>

When a benchmark is set, the prompt will update with the name of the benchmark. Supposing that is “bench”, then the prompt would be:


The list of available benchmarks can be shown with, though this is limited to the first 200 benchmarks:

compiler_gym:bench> list_benchmarks

You can also see what datasets are available with this command:

compiler_gym:cbench-v1/qsort> list_datasets

The default reward and observation can be similarly set with:

compiler_gym:bench> set_default_reward <reward-name>
compiler_gym:bench> set_default_observation <observation-name>

And lists of the choices are available with:

compiler_gym:bench> list_rewards
compiler_gym:bench> list_observations

The default rewards and observations will be reported every time an action is taken. So, if, for example, you want to see how the instruction count of the benchmark program is affected by your actions, set the default reward to “IrInstructionCount”. Then the change in instruction count for each action will be reported.

Additionally, some of the search techniques require the default reward to be set, since they will try to optimise that reward.

Actions and the Action Stack

In CompilerGym an action corresponds to invoking an compiler operation (currently an LLVM opt pass) on the intermediate representation of the program. Each action acts on the result of the previous action and so on.

So, for example, to apply first the ‘tail call elimination’ pass, then the ‘loop unrolling’ pass we call two actions:

compiler_gym:bench> action -tailcallelim
compiler_gym:bench> action -loop-unroll

Each action will report its default reward. Note that multiple actions can be placed on a single line, so that the above is equivalent to:

compiler_gym:bench> action -tailcallelim -loop-unroll

You can choose a random action, by using just a ‘-’ as the action name:

compiler_gym:bench> action -

Since an empty line on the shell repeats the last action, you can execute many random actions by typing that line first then holding down return.

The actions are recorded in a stack, with the latest action on the top of the stack. You can view the action stack with stack command:

compiler_gym:bench> stack

This will show for each action if it had an effect (as computed by the underlying compiler), whether this terminated compiler, and what the per action and cumulative rewards are.

The last action can be undone by:

compiler_gym:bench> undo

All actions in the stack can be undone at once by:

compiler_gym:bench> reset

You can find out what the effect of each action would be by calling this command:

compiler_gym:bench> try_all_actions

This will show a table with the reward for each action, sorted by best first.

If you have a large stack of actions, many of which are not profitable, you can simplify the stack with this command:

compiler_gym:bench> simplify_stack

This will redo the entire stack, keeping only those actions which previously gave good rewards. (Note this doesn’t mean that the simplified stack will only have positive rewards, some negative actions may be necessary set up for a later positive reward.)

Current Status

For the current state of the program - after whatever actions have been called on it - you can make several queries.

The first is to get a reward. This might not be the same as the current default reward:

compiler_gym:bench> reward <reward-name>

You can see various observations with:

compiler_gym:bench> observation <observation-name>

Finally, you can print the equivalent command line for achieving the same behaviour as the actions through the standard system shell:

compiler_gym:bench> commandline


Some very basic search capabilities are supported, directly in the shell. Each of them just looks for another action to add.

First, is the random search through this command:

compiler_gym:bench> action -

Multiple steps can be taken by holding down the return key.

A hill-climbing search tries an action, but will only accept it if it yields a positive reward:

compiler_gym:bench> hill_climb <num-steps>

A simple greedy search tries all possible actions and takes the one with the highest reward, stopping when no action has a positive reward:

compiler_gym:bench> greedy <num-steps>


One useful command is:

compiler_gym:bench> breakpoint

Which drops into the python debugger. This is very useful if you want to see what is going on internally. There is a ‘self.env’ object that represents the environment that is definitely worth exploring.

And finally:

compiler_gym:bench> exit

Drops out of the shell. Ctrl-D should have the same effect.


Replay the best solution found from a random search.

$ python -m compiler_gym.bin.random_replay --env=llvm-ic-v0 --output_dir=/path/to/logs

Given a set of compiler_gym.bin.random_search logs generated from a prior search, replay the best sequence of actions found and record the incremental reward of each action.


Validate environment states.

Example usage:

$ cat << EOF |
cbench-v1/crc32,0,1.2,opt  input.bc -o output.bc
python -m compiler_gym.bin.validate --env=llvm-ic-v0 -

Use this script to validate environment states. Environment states are read from stdin as a comma-separated list of benchmark names, walltimes, episode rewards, and commandlines. Each state is validated by replaying the commandline and validating that the reward matches the expected value. Further, some benchmarks allow for validation of program semantics. When available, those additional checks will be automatically run.

Input Format

The correct format for generating input states can be generated using CompilerEnvStateWriter. For .. rubric:: Example

>>> env = gym.make("llvm-autophase-ic-v0")
>>> env.reset()
>>> env.step(env.action_space.sample())
>>> with CompilerEnvStateWriter(open("results.csv", "wb")) as writer:
...     writer.write_state(env.state)

Output Format

This script prints one line per input state. The order of input states is not preserved. A successfully validated state has the format:

✅  <benchmark_name>  <reproduced_reward>

Else if validation fails, the output is:

❌  <benchmark_name>  <error_details>