Command Line Reference
Compiler Gym includes several command line tools in the compiler_gym.bin
module. This document provides an overview of each module. Run any of the
commands with --helpfull
to examine the full usage options.
Command line tools:
compiler_gym.bin.service
This program can be used to query and run the CompilerGym services.
Listing available environments
List the environments that are available using:
$ python -m compiler_gym.bin.service --ls_env
Querying the capabilities of a service
Query the capabilities of a service using:
$ python -m compiler_gym.bin.service --env=<env>
For example:
$ python -m compiler_gym.bin.service --env=llvm-v0
Datasets
--------
+----------------------------+--------------------------+------------------------------+
| Dataset | Num. Benchmarks [#f1]_ | Description |
+============================+==========================+==============================+
| benchmark://anghabench-v0 | 1,042,976 | Compile-only C/C++ functions |
+----------------------------+--------------------------+------------------------------+
| benchmark://blas-v0 | 300 | Basic linear algebra kernels |
+----------------------------+--------------------------+------------------------------+
...
Observation Spaces
------------------
+--------------------------+----------------------------------------------+
| Observation space | Shape |
+==========================+==============================================+
| Autophase | `Box(0, 9223372036854775807, (56,), int64)` |
+--------------------------+----------------------------------------------+
| AutophaseDict | `Dict(ArgsPhi:int<0,inf>, BB03Phi:int<0,...` |
+--------------------------+----------------------------------------------+
| BitcodeFile | `str_list<>[0,4096.0])` |
+--------------------------+----------------------------------------------+
...
The output is tabular summaries of the environment’s datasets, observation spaces, reward spaces, and action spaces, using reStructuredText syntax (https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#tables).
To query the capabilities of an unmanaged service, use --service
. For
example, query a service running the llvm-v0
environment at
localhost:8080
using:
$ python -m compiler_gym.bin.service --env=llvm-v0 --service=localhost:8080
To query the capability of a binary that implements the RPC service interface,
use the --local_service_binary
flag:
$ python -m compiler_gym.bin.service --env=llvm-v0 --local_service_binary=/path/to/service/binary
Running a Service
This module can also be used to launch a service that can then be connected to by other environments. Start a service by specifying a port number using:
$ python -m compiler_gym.bin.service --env=llvm-v0 --run_on_port=7777
Environment can connect to this service by passing the <hostname>:<port>
address during environment initialization time. For example, in python:
>>> env = compiler_gym.make("llvm-v0", service="localhost:7777")
Or at the command line:
$ python -m compiler_gym.bin.random_search --env=llvm-v0 --service=localhost:7777
compiler_gym.bin.manual_env
Run a CompilerGym environment with text interface controls.
$ python -m compiler_gym.bin.manual_env --env=<env> [--benchmark=<name>] [--observation=<space>] [--reward=<space>]
The benchmark to use can be specified using --benchmark=<name>
.
CompilerGym Shell Tutorial
This program gives a basic shell through which many of commands from CompilerGym can be executed. CompilerGym provides a simple Python interface to various compiler functions, enabling programs to be compiled in different ways and to make queries about those programs. The goal is to have a simple system for machine learning in compilers.
Setting a Benchmark, Reward and Observation
The CompilerGym operates on a program or benchmark. If not set on the command line, the benchmark can be specified in the shell with:
compiler_gym:cbench-v1/qsort> set_benchmark <benchmark-name>
When a benchmark is set, the prompt will update with the name of the benchmark. Supposing that is “bench”, then the prompt would be:
compiler_gym:bench>
The list of available benchmarks can be shown with, though this is limited to the first 200 benchmarks:
compiler_gym:bench> list_benchmarks
You can also see what datasets are available with this command:
compiler_gym:cbench-v1/qsort> list_datasets
The default reward and observation can be similarly set with:
compiler_gym:bench> set_default_reward <reward-name>
compiler_gym:bench> set_default_observation <observation-name>
And lists of the choices are available with:
compiler_gym:bench> list_rewards
compiler_gym:bench> list_observations
The default rewards and observations will be reported every time an action is taken. So, if, for example, you want to see how the instruction count of the benchmark program is affected by your actions, set the default reward to “IrInstructionCount”. Then the change in instruction count for each action will be reported.
Additionally, some of the search techniques require the default reward to be set, since they will try to optimise that reward.
Actions and the Action Stack
In CompilerGym an action corresponds to invoking an compiler operation (currently an LLVM opt pass) on the intermediate representation of the program. Each action acts on the result of the previous action and so on.
So, for example, to apply first the ‘tail call elimination’ pass, then the ‘loop unrolling’ pass we call two actions:
compiler_gym:bench> action -tailcallelim
compiler_gym:bench> action -loop-unroll
Each action will report its default reward. Note that multiple actions can be placed on a single line, so that the above is equivalent to:
compiler_gym:bench> action -tailcallelim -loop-unroll
You can choose a random action, by using just a ‘-’ as the action name:
compiler_gym:bench> action -
Since an empty line on the shell repeats the last action, you can execute many random actions by typing that line first then holding down return.
The actions are recorded in a stack, with the latest action on the top of the stack. You can view the action stack with stack command:
compiler_gym:bench> stack
This will show for each action if it had an effect (as computed by the underlying compiler), whether this terminated compiler, and what the per action and cumulative rewards are.
The last action can be undone by:
compiler_gym:bench> undo
All actions in the stack can be undone at once by:
compiler_gym:bench> reset
You can find out what the effect of each action would be by calling this command:
compiler_gym:bench> try_all_actions
This will show a table with the reward for each action, sorted by best first.
If you have a large stack of actions, many of which are not profitable, you can simplify the stack with this command:
compiler_gym:bench> simplify_stack
This will redo the entire stack, keeping only those actions which previously gave good rewards. (Note this doesn’t mean that the simplified stack will only have positive rewards, some negative actions may be necessary set up for a later positive reward.)
Current Status
For the current state of the program - after whatever actions have been called on it - you can make several queries.
The first is to get a reward. This might not be the same as the current default reward:
compiler_gym:bench> reward <reward-name>
You can see various observations with:
compiler_gym:bench> observation <observation-name>
Finally, you can print the equivalent command line for achieving the same behaviour as the actions through the standard system shell:
compiler_gym:bench> commandline
Searching
Some very basic search capabilities are supported, directly in the shell. Each of them just looks for another action to add.
First, is the random search through this command:
compiler_gym:bench> action -
Multiple steps can be taken by holding down the return key.
A hill-climbing search tries an action, but will only accept it if it yields a positive reward:
compiler_gym:bench> hill_climb <num-steps>
A simple greedy search tries all possible actions and takes the one with the highest reward, stopping when no action has a positive reward:
compiler_gym:bench> greedy <num-steps>
Miscellaneous
One useful command is:
compiler_gym:bench> breakpoint
Which drops into the python debugger. This is very useful if you want to see what is going on internally. There is a ‘self.env’ object that represents the environment that is definitely worth exploring.
And finally:
compiler_gym:bench> exit
Drops out of the shell. Ctrl-D
should have the same effect.
compiler_gym.bin.random_search
Run a parallelized random search of an environment’s action space.
$ python -m compiler_gym.bin.random_search --env=<env> --benchmark=<name> [--runtime=<sec>]
This program runs a simple random agent on the action space of a single benchmark. The best reward, and sequence of actions that produced this, are logged to file.
For example, run a random search of the LLVM instruction count optimization space on a Dijkstra benchmark for 60 seconds using:
$ python -m compiler_gym.bin.random_search --env=llvm-ic-v0 --benchmark=cbench-v1/dijkstra --runtime=60
Started 16 worker threads for benchmark benchmark://cbench-v1/dijkstra (410 instructions) using reward IrInstructionCountOz.
=== Running for a minute ===
Runtime: a minute. Num steps: 470,407 (7,780 / sec). Num episodes: 4,616 (76 / sec). Num restarts: 0.
Best reward: 101.59% (96 passes, found after 35 seconds)
Ending jobs ... done
Step [000 / 096]: reward=0.621951
Step [001 / 096]: reward=0.621951, change=0.000000, action=AlwaysInlinerLegacyPass
...
Step [094 / 096]: reward=1.007905, change=0.066946, action=CfgsimplificationPass
Step [095 / 096]: reward=1.007905, change=0.000000, action=LoopVersioningPass
Step [096 / 096]: reward=1.015936, change=0.008031, action=NewGvnpass
Search strategy
At each step, the agent selects an action randomly and records the
reward. After a number of steps without improving reward (the “patience” of the
agent), the agent terminates, and the environment resets. The number of steps
to take without making progress can be configured using the
--patience=<num>
flag.
Use --runtime
to limit the total runtime of the search. If not provided,
the search will run indefinitely. Use C-c
to cancel an in-progress
search.
Execution strategy
The results of the search are logged to files. Control the location of these
logs using the --output_dir=/path
flag.
Multiple agents are run in parallel. By default, the number of agents is equal
to the number of processors on the host machine. Set a different value using
--nproc
.
compiler_gym.bin.random_replay
Replay the best solution found from a random search.
$ python -m compiler_gym.bin.random_replay --env=llvm-ic-v0 --output_dir=/path/to/logs
Given a set of compiler_gym.bin.random_search
logs generated from a
prior search, replay the best sequence of actions found and record the
incremental reward of each action.
compiler_gym.bin.validate
Validate environment states.
Example usage:
$ cat << EOF |
benchmark,reward,walltime,commandline
cbench-v1/crc32,0,1.2,opt input.bc -o output.bc
EOF
python -m compiler_gym.bin.validate --env=llvm-ic-v0 -
Use this script to validate environment states. Environment states are read from stdin as a comma-separated list of benchmark names, walltimes, episode rewards, and commandlines. Each state is validated by replaying the commandline and validating that the reward matches the expected value. Further, some benchmarks allow for validation of program semantics. When available, those additional checks will be automatically run.
Input Format
The correct format for generating input states can be generated using
CompilerEnvStateWriter
. For
.. rubric:: Example
>>> env = gym.make("llvm-autophase-ic-v0")
>>> env.reset()
>>> env.step(env.action_space.sample())
>>> with CompilerEnvStateWriter(open("results.csv", "wb")) as writer:
... writer.write_state(env.state)
Output Format
This script prints one line per input state. The order of input states is not preserved. A successfully validated state has the format:
✅ <benchmark_name> <reproduced_reward>
Else if validation fails, the output is:
❌ <benchmark_name> <error_details>