Let's get started

This guide runs one full iteration of statistical sampling on a prebuilt workload, from downloading it to the simulation result, and then shows how to iterate. Each step ends with a Check of what it should produce.

Overview of the workflow

graph TD
  W[Workload: prebuilt, or Boot + Load] --> I[Init Warm];
  I --> F[Functional Warming];
  F --> P[Partitioning];
  P --> RP[Run Partitions];
  RP --> R[Results];
  R --> C{Sampling error within the bound?};
  C -->|Yes| Fi[Finished];
  C ---->|No| T[Train the timekeeping model];
  T --> CleanPartition[Clean Up Partitions];
  CleanPartition --> F;

• Workload (1): a disk and a snapshot in which the workload already runs in steady state. Download a prebuilt one with image pull, or prepare your own with boot and load (see Your own workload).
• Init Warm (initialize, 3) warms the long-term microarchitectural state (caches, TLBs, branch predictors), once per experiment.
• Functional Warming (fw, 4) runs the workload over the population and takes a checkpoint per sample. It keeps the time of each vCPU with the timekeeping model (cfg/core_info.csv).
• Partitioning (partition, 5) splits the checkpoints into groups that run in parallel.
• Run Partitions (run-partition, 6) simulates every checkpoint in detail with Flexus.
• Results (result, 7) computes the sampling error and the sample size that meets the bound. If the error is within the bound, the experiment is finished.
• Train the timekeeping model (train, 8) fits the timekeeping model to the detailed results, so that the next functional warming keeps time more accurately.
• Clean Up Partitions (partition-cleanup, 8) removes the partitions and checkpoints of the iteration; the next one starts again from Functional Warming, with the suggested sample size.

0) Before you begin

QFlex is built and ./qflex --help works (see Installation). Run every command from the repository root, and outside the Nix shell if you built with Nix.

All the data lives in the working folder you created during the installation, <WORK_DIR> in the examples (--working-folder); with Docker, it is the folder you mounted:

• <WORK_DIR>/images/: disk images, and the downloaded workloads in images/prebuilt/<id>/;
• <WORK_DIR>/experiments/<experiment name>/: one folder per experiment.

Images are large: to keep them on another disk, make <WORK_DIR>/images a symbolic link.

1) Download a prebuilt workload

A prebuilt workload is a disk (root.qcow2) and a snapshot base (base.zstd) in which the workload already runs in steady state, plus a config.json describing the machine and the workload. IDs are <core count>/<workload>, e.g. 2c/djd or 16c/dc-ws.

• image list shows every workload with its size; with --working-folder, it also marks the downloaded ones.
• Without an ID, image pull lets you choose from a numbered list.
• An interrupted download resumes when you run image pull again.

Check. <WORK_DIR>/images/prebuilt/2c/djd/ holds root.qcow2, base.zstd and config.json (read-only), and image pull printed the machine: cpu_cores=4 measured_core_count=2 memory=64G network_card=none.

{
  "workload": "djd",
  "measured_core_count": 2,
  "cpu_cores": 4,
  "memory": "64G",
  "network_card": "none",
  "suggested_population_seconds": 10,
  "groups": [
    {"name": "djd", "cores": "0-1", "ipc": 2},
    {"name": "phantom", "cores": "2-3", "ipc": 4}
  ],
  ...
}

Core groups

The groups of config.json split the vCPUs of the VM; for 2c/djd, djd runs on vCPUs 0–1 and phantom on 2–3:

• A measured group runs a workload on simulated cores. A workload with more than one measured group, such as 16c/dc-ws (dc on 0–7, ws on 8–15), is consolidated.
• The phantom group is the second half of the vCPUs. It runs the client (load generator) of the workload and is not simulated in detail.
• The ipc is the starting estimate of the timekeeping model, with which functional warming keeps the time of each vCPU, until train fits the model to the detailed results.

The later steps work per group: Init Warm writes the IPC of each vCPU to cfg/core_info.csv, result computes the sampling error of each measured group, and train fits the model of each measured group.

2) Create the shared args file

Every command needs the same experiment options. Keep them in ./qflex.args, one argument per line, and pass them with xargs:

--prebuilt-workload 2c/djd
--working-folder <WORK_DIR>
--experiment-name djd-2c-001
--quantum-size-ns 2000
--llc-size-per-tile-mb 2
--parallel
--no-unique

• --core-count, --double-cores, --memory-gb, --network and --workload-name come from config.json. The snapshot only restores on the machine it was taken on, so qflex rejects values that do not match.
• --population-seconds defaults to the workload's suggested_population_seconds; set it lower (e.g. 1) for a quick test.
• --groups defaults to the workload's core groups; to override them, see Your own workload.
• --no-unique only silences a deprecation message: the experiment folder is always named --experiment-name.

3) Init Warm

The first command of an experiment sets it up from the prebuilt workload: it copies root.qcow2 into the experiment's run/ folder, links base.zstd, and records the workload in run/prebuilt.json. Later commands use this record, so they keep working even without --prebuilt-workload. --loadvm-name defaults to base.

Init Warm then generates the configuration, builds the experiment's WormCache plugin (with cargo, see Installation), and runs the workload until the long-term state is warm. It saves the result as the snapshot init_warmed.

Check. In <WORK_DIR>/experiments/djd-2c-001/:

• run/ has root.qcow2, base.zstd (a link), prebuilt.json, and at the end init_warmed.* (init_warmed.bxdb, .state.zstd, .uarch);
• cfg/ has parameter.rs, flexus_configuration.json, timing.cfg and core_info.csv, the timekeeping model: one row per vCPU at the IPC of its group, in instructions per ns (IPC × 2 GHz). For 2c/djd, vCPUs 0–1 are at 4.0 and the phantom vCPUs 2–3 at 8.0;
• lib/ has libknottykraken.so, libsemikraken.so and libworm_cache.so; bin/ has checkpoint_conversion and bxdb.

4) Functional Warming

Functional warming runs the workload from init_warmed over the population and takes --sample-size checkpoints, evenly spaced, keeping the microarchitectural state warm in between.

Check. run/ has snapshot_0 … snapshot_29 (.bxdb-meta, .state.zstd and .uarch each).

5) Partitioning

The checkpoints are split into partitions that are simulated in parallel. All partitions run at the same time, so choose at most the number of host cores you can spare.

Check. run/partition_0 … run/partition_15, each with its share of the checkpoints and a run_flexus.sh.

To partition differently, unpartition moves the checkpoints back to run/ and removes the partitions (add --force if they already hold simulation results, which it deletes):

6) Run Partitions

To run detailed simulation with 200-us detailed warming and 100-us measurement:

--warming-ratio and --measurement-ratio are in units of 100 µs of simulated time: 1 is 100 µs, 2 is 200 µs. Flexus counts time on the uncore clock (1 GHz, while the cores run at 2 GHz), and one unit is 100,000 uncore cycles. Each checkpoint is simulated for the warming, then measured; the statistics are recorded every 100 µs.

qflex first builds the index of the checkpoint database (run/init_warmed.bxdb/index.bxdb). Then each partition simulates its checkpoints one after the other: it converts the checkpoint to Flexus' format (snapshot_<n>-flexus/), runs the detailed simulation, and moves the logs to result_<n>/.

Check. The output of each partition goes to run/partition_<i>/log and err, not to the terminal:

The second command counts the simulated checkpoints; the step is done when it reaches the sample size.

To stop the run, press Ctrl+C or kill ./qflex: every partition and its QEMU stop with it.

cd <WORK_DIR>/experiments/djd-2c-001/run/partition_0 && ./run_flexus.sh 2 1

7) Results

result collects every measurement into timing.csv and computes the sampling error of each measured core group; --core-groups "0-7,8-15" analyzes other core ranges instead.

It first checks that the sample is complete, i.e. that it has as many sampling units as the --sample-size of functional warming: every checkpoint must have a result that reaches the end of the measurement. Otherwise it stops and lists the failed checkpoints with their partitions; check their log and err (see Run Partitions). --allow-incomplete analyzes the results anyway, with a smaller sample.

Check. The experiment folder has timing.csv and NEXT_SAMPLE_SIZE, the sample size that meets the error bound (--error, default 5% at --confidence 95%). The exit code tells whether this iteration is enough:

• 0: the bound is met, the experiment is finished;
• 255: the bound is not met yet, run the next iteration. This is not a failure.

result does not modify cfg/core_info.csv; train does.

8) Next iteration

Functional warming keeps the time of each vCPU with the timekeeping model, cfg/core_info.csv. The first iteration used a constant IPC per group (the estimates of the core groups); train fits an IPC model per core group to the detailed results in timing.csv, which makes the next functional warming more accurate. Then clean up and warm again with the suggested sample size:

Repeat until result exits with 0.

• train replaces cfg/core_info.csv (the previous one is kept as core_info.csv.<timestamp>) and writes the per-ASID models to run/asid_info.csv.
• partition-cleanup removes the partitions and the checkpoints, and drops them from the checkpoint database, keeping init_warmed. Collect anything you need from the partitions first.

host_core_idx,model_type,ipns,bx_private_icache_miss_coeff,bx_private_dcache_miss_coeff,bx_shared_cache_miss_coeff,bx_bp_miss_coeff,bx_drain_store_buffer_coeff,bx_instruction_coeff
0,constant,4.0,0.0,0.0,0.0,0.0,0.0,0.0

The experiment folder

Troubleshooting

xargs -a ./qflex.args -- ./qflex refresh-scripts

sudo sysctl -w vm.unprivileged_userfaultfd=1
echo 'vm.unprivileged_userfaultfd = 1' | sudo tee /etc/sysctl.d/99-userfaultfd.conf   # keep it after a reboot

Bring your own workload

To simulate a workload that is not prebuilt, create the image and the snapshot yourself with create-base-image, boot and load, then continue with Init Warm. See Your own workload.