bench.txt

"SPARTA WWW Site"_sws :c
:link(sws,index.html)

:line

SPARTA Benchmarks :h3

This page gives SPARTA performance on several benchmark problems, run
on different machines, both in serial and parallel.  When the hardware
supports it, results using the the accelerator options currently
available in the code are also shown.

All the information is provided below to run these tests or similar
tests on your own machine.  This includes info on how to build SPARTA,
how to launch it with the appropriate command-line arguments, and
links to input and output files generated by all the benchmark tests.
Note that input files and a few sample output files are also provided
in the {bench} directory of the SPARTA distribution.  See the
bench/README file for details.

Benchmark results: :h4

"Free"_#free_old = free molecular flow in a box, older results on a large BG/Q machine
"Collide"_#collide_old = collisional molecular flow in a box , older results on a large BG/Q machine
<IMG SRC = "images/new.gif"> "Free"_#free = same as above, with accelerator options and new machines
<IMG SRC = "images/new.gif"> "Collide"_#collide = same as above, with accelerator options and new machines
<IMG SRC = "images/new.gif"> "Sphere"_#sphere = flow around a sphere, with accelerator options and new machines :ul

Additional info: :h4

"Accelerator options"_#accelerate
"Machines and node hardware"_#machines
"How to build SPARTA and run the benchmarks"_#howto
"How to interpret the plots"_#plots :ul

:line

Free molecular flow in a box :h4,link(free_old)

This benchmark is for particles advecting in free molecular flow (no
collsions) on a regular grid overlaying a 3d closed box with
reflective boundaries.  The size of the grid was varied; the particle
counts is always 10x the number of grid cells.  Particles were
initialized with a thermal temperature (no streaming velocity) so they
move in random directions.  Since there is very little computation to
do, this is a good stress test of the communication capabilities of
SPARTA and the machines it is run on.

The input script for this problem is bench/in.free in the SPARTA
distribution.

This plot shows timings results in particle moves/sec/node, for runs
of different sizes on varying node counts of two different machines.
Problems as small as 1M grid cells (10M particles) and as large as 10B
grid cells (100B particles) were run.

Chama is an Intel cluster with Infiniband described "below"_#machines.
Each node of chama has dual 8-core Intel Sandy Bridge CPUs.  These
tests were run on all 16 cores of each node, i.e. with 16 MPI
tasks/node.  Up to 1024 nodes were used (16K MPI tasks).  Mira is an
IBM BG/Q machine at Argonne National Labs.  It has 16 cores per node.
These tests were run with 4 MPI tasks/core, for a total of 64 MPI
tasks/node.  Up to 8K nodes were used (512K MPI tasks).

The plot shows that a Chama node is about 2x faster than a BG/Q node.

Each individual curve in the plot is a strong scaling test, where the
same size problem is run on more and more nodes.  Perfect scalability
would be a horizontal line.  The curves show some initial super-linear
speed-up as the particle count/node decreased, due to cache effects,
then a slow-down as more nodes are added due to too-few particles/node
and increased communication costs.

Jumping from curve-to-curve as node count increases is a weak scaling
test, since the problem size is increasing with node count.  Again a
horizontal line would represent perfect weak scaling.

:c,image(images/bench_free_small.jpg,images/bench_free.jpg)

Click on the image to see a larger version.

:line

Collisional flow in a box :h4,link(collide_old)

This benchmark is for particles undergoing collisional flow.
Everything about the problem is the same as the free molecular flow
problem described above, except that collisions were enabled, which
requires extra computation, as well as particle sorting each timestep
to identify particles in the same grid cell.

The input script for this problem is bench/in.collide in the
SPARTA distribution.

As above, this plot shows timings results in particle moves/sec/node,
for runs of different sizes on varying node counts.  Data for the same
two machines is shown: "chama"_#machine (Intel cluster with Ifiniband
at Sandia) and mira (IBM BG/Q at ANL).  Comparing these timings to the
free molecule flow plot in the previous section shows the cost of
collisions (and sorting) slows down the performance by a factor of
about 2.5x.  Cache effects (super-linear speed-up) are smaller due to
the increased computational costs.

For collisional flow, problems as small as 1M grid cells (10M
particles) and as large as 1B grid cells (10B particles) were run.

The discussion above regarding strong and weak scaling also applies to
this plot.  For any curve, a horizontal line would represent perfect
weak scaling.

:c,image(images/bench_collide_small.jpg,images/bench_collide.jpg)

Click on the image to see a larger version.

:line
:line

Free benchmark :h4,link(free)

"in.free"_bench/in.free input script :ul

As described above, this benchmark is for particles advecting in free
molecular flow (no collisions) on a regular grid overlaying a 3d
closed box with reflective boundaries. The size of the grid was
varied; the particle counts is always 10x the number of grid
cells. Particles were initialized with a thermal temperature (no
streaming velocity) so they move in random directions. Since there is
very little computation to do, this is a good stress test of the
communication capabilities of SPARTA and the machines it is run on.

Additional packages needed for this benchmark: none

Comments:

In the data below, K = 1000 particles, so 1M = 1024*1000. :ul

:line

[Free single core and single node performance:]

Best timings for any accelerator option as a function of problem size.
Running on a single CPU or KNL core.  Running on a single CPU or KNL
node or a single GPU.  Only for double precision.

:image(bench/plot_free_core_best_small.jpg,bench/plot_free_core_best.jpg)
:image(bench/plot_free_node_best_small.jpg,bench/plot_free_node_best.jpg)

"Table for single core"_bench/plot_free_core_best.html
"Table for single node"_bench/plot_free_node_best.html :ul

:line

[Free strong and weak scaling:]

Fastest timing for any accelerator option running on multiple CPU or
KNL or a single GPUs, as a function of node count.  For strong scaling
of 2 problem sizes: 8M particles, 64M particles.  For weak scaling of
2 problem sizes: 1M particles/node, 16M particles/node.  Only for a
single GPU/node, only double precision.

Strong scaling means the same size problem is run on successively more
nodes.  Weak scaling means the problem size doubles each time the node
count doubles.  See a fuller description "here"_#interpret of how to
interpret these plots.

:image(bench/plot_free_strong_8M_best_small.jpg,bench/plot_free_strong_8M_best.jpg)
:image(bench/plot_free_strong_64M_best_small.jpg,bench/plot_free_strong_64M_best.jpg)
:image(bench/plot_free_weak_1M_best_small.jpg,bench/plot_free_weak_1M_best.jpg)
:image(bench/plot_free_weak_16M_best_small.jpg,bench/plot_free_weak_16M_best.jpg)

"Table for strong scaling of 8M particles"_bench/plot_free_strong_8M_best.html
"Table for strong scaling of 64M particles"_bench/plot_free_strong_64M_best.html
"Table for weak scaling of 1M particles/node"_bench/plot_free_weak_1M_best.html
"Table for weak scaling of 16M particles/node"_bench/plot_free_weak_16M_best.html :ul

:line

[Free performance details]:

Modes: per-core, per-node, strong scaling, weak scaling
Hardware: CPU, KNL, GPU options
Within plot: accelerator packages, one or multiple GPUs/node :ul

Mode | SPARTA Version | Hardware | Machine | Size | Plot | Table
core | 23Dec17 | SandyBridge | chama | 1K-16K | "plot"_bench/plot_free_chama_core_CPU.jpg | "table"_bench/plot_free_chama_core_CPU.html
core | 23Dec17 | Haswell | mutrino | 1K-16K | "plot"_bench/plot_free_mutrino_core_CPU.jpg | "table"_bench/plot_free_mutrino_core_CPU.html
core | 23Dec17 | Broadwell | serrano | 1K-16K | "plot"_bench/plot_free_serrano_core_CPU.jpg | "table"_bench/plot_free_serrano_core_CPU.html
core | 23Dec17 | KNL | mutrino | 1K-16K | "plot"_bench/plot_free_mutrino_core_KNL.jpg | "table"_bench/plot_free_mutrino_core_KNL.html
node | 23Dec17 | SandyBridge | chama | 32K-128M | "plot"_bench/plot_free_chama_node_CPU.jpg | "table"_bench/plot_free_chama_node_CPU.html
node | 23Dec17 | Haswell | mutrino | 32K-128M | "plot"_bench/plot_free_mutrino_node_CPU.jpg | "table"_bench/plot_free_mutrino_node_CPU.html
node | 23Dec17 | Broadwell | serrano | 32K-128M | "plot"_bench/plot_free_serrano_node_CPU.jpg | "table"_bench/plot_free_serrano_node_CPU.html
node | 23Dec17 | KNL | mutrino | 32K-128M | "plot"_bench/plot_free_mutrino_node_KNL.jpg | "table"_bench/plot_free_mutrino_node_KNL.html
node | 23Dec17 | K80 | ride80 | 32K-128M | "plot"_bench/plot_free_ride80_node_GPU.jpg | "table"_bench/plot_free_ride80_node_GPU.html
node | 23Dec17 | P100 | ride100 | 32K-128M | "plot"_bench/plot_free_ride100_node_GPU.jpg | "table"_bench/plot_free_ride100_node_GPU.html
strong | 23Dec17 | SandyBridge | chama | 8M | "plot"_bench/plot_free_chama_strong_8M_CPU.jpg | "table"_bench/plot_free_chama_strong_8M_CPU.html
strong | 23Dec17 | Haswell | mutrino | 8M | "plot"_bench/plot_free_mutrino_strong_8M_CPU.jpg | "table"_bench/plot_free_mutrino_strong_8M_CPU.html
strong | 23Dec17 | Broadwell | serrano | 8M | "plot"_bench/plot_free_serrano_strong_8M_CPU.jpg | "table"_bench/plot_free_serrano_strong_8M_CPU.html
strong | 23Dec17 | KNL | mutrino | 8M | "plot"_bench/plot_free_mutrino_strong_8M_KNL.jpg | "table"_bench/plot_free_mutrino_strong_8M_KNL.html
strong | 23Dec17 | K80 | ride80 | 8M | "plot"_bench/plot_free_ride80_strong_8M_GPU.jpg | "table"_bench/plot_free_ride80_strong_8M_GPU.html
strong | 23Dec17 | P100 | ride100 | 8M | "plot"_bench/plot_free_ride100_strong_8M_GPU.jpg | "table"_bench/plot_free_ride100_strong_8M_GPU.html
strong | 23Dec17 | SandyBridge | chama | 64M | "plot"_bench/plot_free_chama_strong_64M_CPU.jpg | "table"_bench/plot_free_chama_strong_64M_CPU.html
strong | 23Dec17 | Haswell | mutrino | 64M | "plot"_bench/plot_free_mutrino_strong_64M_CPU.jpg | "table"_bench/plot_free_mutrino_strong_64M_CPU.html
strong | 23Dec17 | Broadwell | serrano | 64M | "plot"_bench/plot_free_serrano_strong_64M_CPU.jpg | "table"_bench/plot_free_serrano_strong_64M_CPU.html
strong | 23Dec17 | KNL | mutrino | 64M | "plot"_bench/plot_free_mutrino_strong_64M_KNL.jpg | "table"_bench/plot_free_mutrino_strong_64M_KNL.html
strong | 23Dec17 | K80 | ride80 | 64M | "plot"_bench/plot_free_ride80_strong_64M_GPU.jpg | "table"_bench/plot_free_ride80_strong_64M_GPU.html
strong | 23Dec17 | P100 | ride100 | 64M | "plot"_bench/plot_free_ride100_strong_64M_GPU.jpg | "table"_bench/plot_free_ride100_strong_64M_GPU.html
weak | 23Dec17 | SandyBridge | chama | 1M/node | "plot"_bench/plot_free_chama_weak_1M_CPU.jpg | "table"_bench/plot_free_chama_weak_1M_CPU.html
weak | 23Dec17 | Haswell | mutrino | 1M/node | "plot"_bench/plot_free_mutrino_weak_1M_CPU.jpg | "table"_bench/plot_free_mutrino_weak_1M_CPU.html
weak | 23Dec17 | Broadwell | serrano | 1M/node | "plot"_bench/plot_free_serrano_weak_1M_CPU.jpg | "table"_bench/plot_free_serrano_weak_1M_CPU.html
weak | 23Dec17 | KNL | mutrino | 1M/node | "plot"_bench/plot_free_mutrino_weak_1M_KNL.jpg | "table"_bench/plot_free_mutrino_weak_1M_KNL.html
weak | 23Dec17 | K80 | ride80 | 1M/node | "plot"_bench/plot_free_ride80_weak_1M_GPU.jpg | "table"_bench/plot_free_ride80_weak_1M_GPU.html
weak | 23Dec17 | P100 | ride100 | 1M/node | "plot"_bench/plot_free_ride100_weak_1M_GPU.jpg | "table"_bench/plot_free_ride100_weak_1M_GPU.html
weak | 23Dec17 | SandyBridge | chama | 16M/node | "plot"_bench/plot_free_chama_weak_16M_CPU.jpg | "table"_bench/plot_free_chama_weak_16M_CPU.html
weak | 23Dec17 | Haswell | mutrino | 16M/node | "plot"_bench/plot_free_mutrino_weak_16M_CPU.jpg | "table"_bench/plot_free_mutrino_weak_16M_CPU.html
weak | 23Dec17 | Broadwell | serrano | 16M/node | "plot"_bench/plot_free_serrano_weak_16M_CPU.jpg | "table"_bench/plot_free_serrano_weak_16M_CPU.html
weak | 23Dec17 | KNL | mutrino | 16M/node | "plot"_bench/plot_free_mutrino_weak_16M_KNL.jpg | "table"_bench/plot_free_mutrino_weak_16M_KNL.html
weak | 23Dec17 | K80 | ride80 | 16M/node | "plot"_bench/plot_free_ride80_weak_16M_GPU.jpg | "table"_bench/plot_free_ride80_weak_16M_GPU.html
weak | 23Dec17 | P100 | ride100 | 16M/node | "plot"_bench/plot_free_ride100_weak_16M_GPU.jpg | "table"_bench/plot_free_ride100_weak_16M_GPU.html
:tb(s=|,ea=c)

:line
:line

Collide benchmark :h4,link(collide)

"in.collide"_bench/in.collide input script
"in.collide.kokkos_cuda"_bench/in.collide.gpu.steps variant for Kokkos/Cuda package :ul

As described above, this benchmark is for particles undergoing
collisional flow. Everything about the problem is the same as the free
molecular flow problem described above, except that collisions were
enabled, which requires extra computation, as well as particle sorting
each timestep to identify particles in the same grid cell.

Additional packages needed for this benchmark: none

Comments:

In the data below, K = 1000 particles, so 1M = 1024*1000. :ul

:line

[Collide single core and single node performance:]

Best timings for any accelerator option as a function of problem size.
Running on a single CPU or KNL core.  Running on a single CPU or KNL
or a single GPU.  Only for double precision.

:image(bench/plot_collide_core_best_small.jpg,bench/plot_collide_core_best.jpg)
:image(bench/plot_collide_node_best_small.jpg,bench/plot_collide_node_best.jpg)

"Table for single core"_bench/plot_collide_core_best.html
"Table for single node"_bench/plot_collide_node_best.html :ul

:line

[Collide strong and weak scaling:]

Fastest timing for any accelerator option running on multiple CPU or
KNL or a single GPUs, as a function of node count.  For strong scaling
of 2 problem sizes: 8M particles, 64M particles.  For weak scaling of
2 problem sizes: 1M particles/node, 16M particles/node.  Only for a
single GPU/node, only double precision.

Strong scaling means the same size problem is run on successively more
nodes.  Weak scaling means the problem size doubles each time the node
count doubles.  See a fuller description "here"_#interpret of how to
interpret these plots.

:image(bench/plot_collide_strong_8M_best_small.jpg,bench/plot_collide_strong_8M_best.jpg)
:image(bench/plot_collide_strong_64M_best_small.jpg,bench/plot_collide_strong_64M_best.jpg)
:image(bench/plot_collide_weak_1M_best_small.jpg,bench/plot_collide_weak_1M_best.jpg)
:image(bench/plot_collide_weak_16M_best_small.jpg,bench/plot_collide_weak_16M_best.jpg)

"Table for strong scaling of 8M particles"_bench/plot_collide_strong_8M_best.html
"Table for strong scaling of 64M particles"_bench/plot_collide_strong_64M_best.html
"Table for weak scaling of 1M particles/node"_bench/plot_collide_weak_1M_best.html
"Table for weak scaling of 16M particles/node"_bench/plot_collide_weak_16M_best.html :ul

:line 

[Collide performance details]:

Modes: per-core, per-node, strong scaling, weak scaling
Hardware: CPU, KNL, GPU options
Within plot: accelerator packages, one or multiple GPUs/node :ul

Mode | SPARTA Version | Hardware | Machine | Size | Plot | Table
core | 23Dec17 | SandyBridge | chama | 1K-16K | "plot"_bench/plot_collide_chama_core_CPU.jpg | "table"_bench/plot_collide_chama_core_CPU.html
core | 23Dec17 | Haswell | mutrino | 1K-16K | "plot"_bench/plot_collide_mutrino_core_CPU.jpg | "table"_bench/plot_collide_mutrino_core_CPU.html
core | 23Dec17 | Broadwell | serrano | 1K-16K | "plot"_bench/plot_collide_serrano_core_CPU.jpg | "table"_bench/plot_collide_serrano_core_CPU.html
core | 23Dec17 | KNL | mutrino | 1K-16K | "plot"_bench/plot_collide_mutrino_core_KNL.jpg | "table"_bench/plot_collide_mutrino_core_KNL.html
node | 23Dec17 | SandyBridge | chama | 32K-128M | "plot"_bench/plot_collide_chama_node_CPU.jpg | "table"_bench/plot_collide_chama_node_CPU.html
node | 23Dec17 | Haswell | mutrino | 32K-128M | "plot"_bench/plot_collide_mutrino_node_CPU.jpg | "table"_bench/plot_collide_mutrino_node_CPU.html
node | 23Dec17 | Broadwell | serrano | 32K-128M | "plot"_bench/plot_collide_serrano_node_CPU.jpg | "table"_bench/plot_collide_serrano_node_CPU.html
node | 23Dec17 | KNL | mutrino | 32K-128M | "plot"_bench/plot_collide_mutrino_node_KNL.jpg | "table"_bench/plot_collide_mutrino_node_KNL.html
node | 23Dec17 | K80 | ride80 | 32K-128M | "plot"_bench/plot_collide_ride80_node_GPU.jpg | "table"_bench/plot_collide_ride80_node_GPU.html
node | 23Dec17 | P100 | ride100 | 32K-128M | "plot"_bench/plot_collide_ride100_node_GPU.jpg | "table"_bench/plot_collide_ride100_node_GPU.html
strong | 23Dec17 | SandyBridge | chama | 8M | "plot"_bench/plot_collide_chama_strong_8M_CPU.jpg | "table"_bench/plot_collide_chama_strong_8M_CPU.html
strong | 23Dec17 | Haswell | mutrino | 8M | "plot"_bench/plot_collide_mutrino_strong_8M_CPU.jpg | "table"_bench/plot_collide_mutrino_strong_8M_CPU.html
strong | 23Dec17 | Broadwell | serrano | 8M | "plot"_bench/plot_collide_serrano_strong_8M_CPU.jpg | "table"_bench/plot_collide_serrano_strong_8M_CPU.html
strong | 23Dec17 | KNL | mutrino | 8M | "plot"_bench/plot_collide_mutrino_strong_8M_KNL.jpg | "table"_bench/plot_collide_mutrino_strong_8M_KNL.html
strong | 23Dec17 | K80 | ride80 | 8M | "plot"_bench/plot_collide_ride80_strong_8M_GPU.jpg | "table"_bench/plot_collide_ride80_strong_8M_GPU.html
strong | 23Dec17 | P100 | ride100 | 8M | "plot"_bench/plot_collide_ride100_strong_8M_GPU.jpg | "table"_bench/plot_collide_ride100_strong_8M_GPU.html
strong | 23Dec17 | SandyBridge | chama | 64M | "plot"_bench/plot_collide_chama_strong_64M_CPU.jpg | "table"_bench/plot_collide_chama_strong_64M_CPU.html
strong | 23Dec17 | Haswell | mutrino | 64M | "plot"_bench/plot_collide_mutrino_strong_64M_CPU.jpg | "table"_bench/plot_collide_mutrino_strong_64M_CPU.html
strong | 23Dec17 | Broadwell | serrano | 64M | "plot"_bench/plot_collide_serrano_strong_64M_CPU.jpg | "table"_bench/plot_collide_serrano_strong_64M_CPU.html
strong | 23Dec17 | KNL | mutrino | 64M | "plot"_bench/plot_collide_mutrino_strong_64M_KNL.jpg | "table"_bench/plot_collide_mutrino_strong_64M_KNL.html
strong | 23Dec17 | K80 | ride80 | 64M | "plot"_bench/plot_collide_ride80_strong_64M_GPU.jpg | "table"_bench/plot_collide_ride80_strong_64M_GPU.html
strong | 23Dec17 | P100 | ride100 | 64M | "plot"_bench/plot_collide_ride100_strong_64M_GPU.jpg | "table"_bench/plot_collide_ride100_strong_64M_GPU.html
weak | 23Dec17 | SandyBridge | chama | 1M/node | "plot"_bench/plot_collide_chama_weak_1M_CPU.jpg | "table"_bench/plot_collide_chama_weak_1M_CPU.html
weak | 23Dec17 | Haswell | mutrino | 1M/node | "plot"_bench/plot_collide_mutrino_weak_1M_CPU.jpg | "table"_bench/plot_collide_mutrino_weak_1M_CPU.html
weak | 23Dec17 | Broadwell | serrano | 1M/node | "plot"_bench/plot_collide_serrano_weak_1M_CPU.jpg | "table"_bench/plot_collide_serrano_weak_1M_CPU.html
weak | 23Dec17 | KNL | mutrino | 1M/node | "plot"_bench/plot_collide_mutrino_weak_1M_KNL.jpg | "table"_bench/plot_collide_mutrino_weak_1M_KNL.html
weak | 23Dec17 | K80 | ride80 | 1M/node | "plot"_bench/plot_collide_ride80_weak_1M_GPU.jpg | "table"_bench/plot_collide_ride80_weak_1M_GPU.html
weak | 23Dec17 | P100 | ride100 | 1M/node | "plot"_bench/plot_collide_ride100_weak_1M_GPU.jpg | "table"_bench/plot_collide_ride100_weak_1M_GPU.html
weak | 23Dec17 | SandyBridge | chama | 16M/node | "plot"_bench/plot_collide_chama_weak_16M_CPU.jpg | "table"_bench/plot_collide_chama_weak_16M_CPU.html
weak | 23Dec17 | Haswell | mutrino | 16M/node | "plot"_bench/plot_collide_mutrino_weak_16M_CPU.jpg | "table"_bench/plot_collide_mutrino_weak_16M_CPU.html
weak | 23Dec17 | Broadwell | serrano | 16M/node | "plot"_bench/plot_collide_serrano_weak_16M_CPU.jpg | "table"_bench/plot_collide_serrano_weak_16M_CPU.html
weak | 23Dec17 | KNL | mutrino | 16M/node | "plot"_bench/plot_collide_mutrino_weak_16M_KNL.jpg | "table"_bench/plot_collide_mutrino_weak_16M_KNL.html
weak | 23Dec17 | K80 | ride80 | 16M/node | "plot"_bench/plot_collide_ride80_weak_16M_GPU.jpg | "table"_bench/plot_collide_ride80_weak_16M_GPU.html
weak | 23Dec17 | P100 | ride100 | 16M/node | "plot"_bench/plot_collide_ride100_weak_16M_GPU.jpg | "table"_bench/plot_collide_ride100_weak_16M_GPU.html
:tb(s=|,ea=c)

:line
:line

Sphere benchmark :h4,link(sphere)

"in.sphere"_bench/in.sphere input script
"in.sphere.kokkos_cuda"_bench/in.sphere.gpu.steps variant for Kokkos/Cuda package :ul

This benchmark is for particles flowing around a sphere.

Comments:

In the data below, K = 1000 particles, so 1M = 1024*1000. :ul

:line

[Sphere single core and single node performance:]

Best timings for any accelerator option as a function of problem size.
Running on a single CPU or KNL core.  Running on a single CPU or KNL
node or a single GPU.  Only for double precision.

:image(bench/plot_sphere_core_best_small.jpg,bench/plot_sphere_core_best.jpg)
:image(bench/plot_sphere_node_best_small.jpg,bench/plot_sphere_node_best.jpg)

"Table for single core"_bench/plot_sphere_core_best.html
"Table for single node"_bench/plot_sphere_node_best.html :ul

:line

[Sphere strong and weak scaling:]

Fastest timing for any accelerator option running on multiple CPU or
KNL or a single GPUs, as a function of node count.  For strong scaling
of 2 problem sizes: 8M particles, 64M particles.  For weak scaling of
2 problem sizes: 1M particles/node, 16M particles/node.  Only for a
single GPU/node, only double precision.

Strong scaling means the same size problem is run on successively more
nodes.  Weak scaling means the problem size doubles each time the node
count doubles.  See a fuller description "here"_#interpret of how to
interpret these plots.

:image(bench/plot_sphere_strong_8M_best_small.jpg,bench/plot_sphere_strong_8M_best.jpg)
:image(bench/plot_sphere_strong_64M_best_small.jpg,bench/plot_sphere_strong_64M_best.jpg)
:image(bench/plot_sphere_weak_1M_best_small.jpg,bench/plot_sphere_weak_1M_best.jpg)
:image(bench/plot_sphere_weak_16M_best_small.jpg,bench/plot_sphere_weak_16M_best.jpg)

"Table for strong scaling of 8M particles"_bench/plot_sphere_strong_8M_best.html
"Table for strong scaling of 64M particles"_bench/plot_sphere_strong_64M_best.html
"Table for weak scaling of 1M particles/node"_bench/plot_sphere_weak_1M_best.html
"Table for weak scaling of 16M particles/node"_bench/plot_sphere_weak_16M_best.html :ul

:line

[Sphere performance details]:

Modes: per-core, per-node, strong scaling, weak scaling
Hardware: CPU, KNL, GPU options
Within plot: accelerator packages, one or multiple GPUs/node :ul

Mode | SPARTA Version | Hardware | Machine | Size | Plot | Table
core | 23Dec17 | SandyBridge | chama | 8K-16K | "plot"_bench/plot_sphere_chama_core_CPU.jpg | "table"_bench/plot_sphere_chama_core_CPU.html
core | 23Dec17 | Haswell | mutrino | 8K-16K | "plot"_bench/plot_sphere_mutrino_core_CPU.jpg | "table"_bench/plot_sphere_mutrino_core_CPU.html
core | 23Dec17 | Broadwell | serrano | 8K-16K | "plot"_bench/plot_sphere_serrano_core_CPU.jpg | "table"_bench/plot_sphere_serrano_core_CPU.html
core | 23Dec17 | KNL | mutrino | 8K-16K | "plot"_bench/plot_sphere_mutrino_core_KNL.jpg | "table"_bench/plot_sphere_mutrino_core_KNL.html
node | 23Dec17 | SandyBridge | chama | 32K-128M | "plot"_bench/plot_sphere_chama_node_CPU.jpg | "table"_bench/plot_sphere_chama_node_CPU.html
node | 23Dec17 | Haswell | mutrino | 32K-128M | "plot"_bench/plot_sphere_mutrino_node_CPU.jpg | "table"_bench/plot_sphere_mutrino_node_CPU.html
node | 23Dec17 | Broadwell | serrano | 32K-128M | "plot"_bench/plot_sphere_serrano_node_CPU.jpg | "table"_bench/plot_sphere_serrano_node_CPU.html
node | 23Dec17 | KNL | mutrino | 32K-128M | "plot"_bench/plot_sphere_mutrino_node_KNL.jpg | "table"_bench/plot_sphere_mutrino_node_KNL.html
node | 23Dec17 | K80 | ride80 | 32K-128M | "plot"_bench/plot_sphere_ride80_node_GPU.jpg | "table"_bench/plot_sphere_ride80_node_GPU.html
node | 23Dec17 | P100 | ride100 | 32K-128M | "plot"_bench/plot_sphere_ride100_node_GPU.jpg | "table"_bench/plot_sphere_ride100_node_GPU.html
strong | 23Dec17 | SandyBridge | chama | 8M | "plot"_bench/plot_sphere_chama_strong_8M_CPU.jpg | "table"_bench/plot_sphere_chama_strong_8M_CPU.html
strong | 23Dec17 | Haswell | mutrino | 8M | "plot"_bench/plot_sphere_mutrino_strong_8M_CPU.jpg | "table"_bench/plot_sphere_mutrino_strong_8M_CPU.html
strong | 23Dec17 | Broadwell | serrano | 8M | "plot"_bench/plot_sphere_serrano_strong_8M_CPU.jpg | "table"_bench/plot_sphere_serrano_strong_8M_CPU.html
strong | 23Dec17 | KNL | mutrino | 8M | "plot"_bench/plot_sphere_mutrino_strong_8M_KNL.jpg | "table"_bench/plot_sphere_mutrino_strong_8M_KNL.html
strong | 23Dec17 | K80 | ride80 | 8M | "plot"_bench/plot_sphere_ride80_strong_8M_GPU.jpg | "table"_bench/plot_sphere_ride80_strong_8M_GPU.html
strong | 23Dec17 | P100 | ride100 | 8M | "plot"_bench/plot_sphere_ride100_strong_8M_GPU.jpg | "table"_bench/plot_sphere_ride100_strong_8M_GPU.html
strong | 23Dec17 | SandyBridge | chama | 64M | "plot"_bench/plot_sphere_chama_strong_64M_CPU.jpg | "table"_bench/plot_sphere_chama_strong_64M_CPU.html
strong | 23Dec17 | Haswell | mutrino | 64M | "plot"_bench/plot_sphere_mutrino_strong_64M_CPU.jpg | "table"_bench/plot_sphere_mutrino_strong_64M_CPU.html
strong | 23Dec17 | Broadwell | serrano | 64M | "plot"_bench/plot_sphere_serrano_strong_64M_CPU.jpg | "table"_bench/plot_sphere_serrano_strong_64M_CPU.html
strong | 23Dec17 | KNL | mutrino | 64M | "plot"_bench/plot_sphere_mutrino_strong_64M_KNL.jpg | "table"_bench/plot_sphere_mutrino_strong_64M_KNL.html
strong | 23Dec17 | K80 | ride80 | 64M | "plot"_bench/plot_sphere_ride80_strong_64M_GPU.jpg | "table"_bench/plot_sphere_ride80_strong_64M_GPU.html
strong | 23Dec17 | P100 | ride100 | 64M | "plot"_bench/plot_sphere_ride100_strong_64M_GPU.jpg | "table"_bench/plot_sphere_ride100_strong_64M_GPU.html
weak | 23Dec17 | SandyBridge | chama | 1M/node | "plot"_bench/plot_sphere_chama_weak_1M_CPU.jpg | "table"_bench/plot_sphere_chama_weak_1M_CPU.html
weak | 23Dec17 | Haswell | mutrino | 1M/node | "plot"_bench/plot_sphere_mutrino_weak_1M_CPU.jpg | "table"_bench/plot_sphere_mutrino_weak_1M_CPU.html
weak | 23Dec17 | Broadwell | serrano | 1M/node | "plot"_bench/plot_sphere_serrano_weak_1M_CPU.jpg | "table"_bench/plot_sphere_serrano_weak_1M_CPU.html
weak | 23Dec17 | KNL | mutrino | 1M/node | "plot"_bench/plot_sphere_mutrino_weak_1M_KNL.jpg | "table"_bench/plot_sphere_mutrino_weak_1M_KNL.html
weak | 23Dec17 | K80 | ride80 | 1M/node | "plot"_bench/plot_sphere_ride80_weak_1M_GPU.jpg | "table"_bench/plot_sphere_ride80_weak_1M_GPU.html
weak | 23Dec17 | P100 | ride100 | 1M/node | "plot"_bench/plot_sphere_ride100_weak_1M_GPU.jpg | "table"_bench/plot_sphere_ride100_weak_1M_GPU.html
weak | 23Dec17 | SandyBridge | chama | 16M/node | "plot"_bench/plot_sphere_chama_weak_16M_CPU.jpg | "table"_bench/plot_sphere_chama_weak_16M_CPU.html
weak | 23Dec17 | Haswell | mutrino | 16M/node | "plot"_bench/plot_sphere_mutrino_weak_16M_CPU.jpg | "table"_bench/plot_sphere_mutrino_weak_16M_CPU.html
weak | 23Dec17 | Broadwell | serrano | 16M/node | "plot"_bench/plot_sphere_serrano_weak_16M_CPU.jpg | "table"_bench/plot_sphere_serrano_weak_16M_CPU.html
weak | 23Dec17 | KNL | mutrino | 16M/node | "plot"_bench/plot_sphere_mutrino_weak_16M_KNL.jpg | "table"_bench/plot_sphere_mutrino_weak_16M_KNL.html
weak | 23Dec17 | K80 | ride80 | 16M/node | "plot"_bench/plot_sphere_ride80_weak_16M_GPU.jpg | "table"_bench/plot_sphere_ride80_weak_16M_GPU.html
weak | 23Dec17 | P100 | ride100 | 16M/node | "plot"_bench/plot_sphere_ride100_weak_16M_GPU.jpg | "table"_bench/plot_sphere_ride100_weak_16M_GPU.html
:tb(s=|,ea=c)

:line
:line

Accelerator options :h4,link(accelerate)

SPARTA has an accelerator option implemented via the KOKKOS package,
"accelerator packages"_doc/Section_accelerate.html. The KOKKOS
packages support multiple hardware options.

For acceleration on a CPU:

CPU = reference implementation, no package, no acceleration
Kokkos/OMP = "Kokkos package"_doc/accelerate_kokkos.html with OMP option via OpenMP
Kokkos/serial = "Kokkos package"_doc/accelerate_kokkos.html with serial option for non-threaded operation on CPUs :ul

For acceleration on an Intel KNL:

CPU/KNL = reference implementation, no package, no acceleration
Kokkos/KNL = "Kokkos package"_doc/accelerate_kokkos.html with KNL option
Kokkos/serial = "Kokkos package"_doc/accelerate_kokkos.html with KNL/serial option :ul

For acceleration on an NVIDIA GPU:

Kokkos/Cuda = "Kokkos package"_doc/accelerate_kokkos.html with CUDA option :ul

:line

Machines and node hardware :h4,link(machines)

Benchmarks were run on the following machines and node hardware.

[chama] = Intel SandyBridge CPUs

1232-node cluster
node = dual-socket Sandy Bridge:2S:8C @ 2.6 GHz, 16 cores, no hyperthreading
interconnect = Qlogic Infiniband 4x QDR, fat tree :ul

[mutrino] = Intel Haswell CPUs or Intel KNLs

~100 CPU nodes
node = dual-socket Haswell 2.3 GHz CPU, 32 cores + 2x hyperthreading
~100 KNL nodes
node = Knights Landing processor, 68 cores + 4x hyperthreading
interconnect = Cray Aries Dragonfly :ul

[serrano] = Intel Broadwell CPUs

1122-node cluster
node = dual-socket Broadwell 2.1 GHz CPU E5-2695, 36 cores + 2x hyperthreading
interconnect = Omni-Path :ul

[ride80] = IBM Power8 CPUs with NVIDIA K80 GPUs

~10 nodes
node CPU = dual Power8 3.42 GHz CPU (Firestone), 16 cores + 8x hyperthreading
each node has 2 Tesla K80 GPUs (each K80 is "dual" with 2 internal GPUs)
interconnect = Infiniband :ul

[ride100] = IBM Power8 CPUs with NVIDIA P100 GPUs

~10 nodes
one node = dual Power8 3.42 GHz CPU (Garrison), 16 cores + 8x hyperthreading
each node has 2 Pascal P100 GPUs
interconnect = Infiniband :ul

:line

How to build SPARTA and run the benchmarks :h4,link(howto)

This table shows which accelerator packages were used on which
machines:

Machine | Hardware | CPU | Kokkos/OMP | Kokkos/KNL | Kokkos/Cuda
chama | SandyBridge | yes | yes | no | no
mutrino | Haswell/KNL | yes | yes | yes | no
serrano | Broadwell | yes | yes | no | no
ride80 | K80 | no | no | no | yes
ride100 | P100 | no | no | no | yes :tb(s=|,ea=c)

These are the software environments on each machine and the Makefiles
used to build SPARTA with different accelerator packages.

[chama]

Intel 17.0.2 icc compiler, GNU 4.9.2 g++ compiler, OpenMPI-Intel 2.0
module load intel/17.0.2.174; module load gnu/4.9.2; module load openmpi-intel/2.0
Makefiles: "Makefile.chama_cpu"_bench/Makefile.chama_cpu, "Makefile.chama_kokkos_omp"_bench/Makefile.chama_kokkos_omp, "Makefile.chama_kokkos_serial"_bench/Makefile.chama_kokkos_serial :ul

[mutrino]

Intel 17.0.2 icc compiler, Cray MPICH 7.5.2
module load intel/17.0.2; module load cray-mpich/7.5.2; module load craype-haswell   # for Haswell
module load intel/17.0.2; module load cray-mpich/7.5.2; module load craype-mic-knl   # for KNL
Makefiles: "Makefile.mutrino_cpu"_bench/Makefile.mutrino_cpu, "Makefile.mutrino_kokkos_omp"_bench/Makefile.mutrino_kokkos_omp, "Makefile.mutrino_kokkos_serial"_bench/Makefile.mutrino_kokkos_serial, "Makefile.mutrino_knl"_bench/Makefile.mutrino_knl, "Makefile.mutrino_kokkos_knl"_bench/Makefile.mutrino_kokkos_knl, "Makefile.mutrino_kokkos_serial_knl"_bench/Makefile.mutrino_kokkos_serial_knl  :ul

[serrano]

Intel 17.0.2 compiler, GNU 4.9.3 g++ compiler, OpenMPI-Intel 2.0
module load intel/17.0.2.174; module load gcc/4.9.3; module load openmpi-intel/2.0
Makefiles: "Makefile.serrano_cpu"_bench/Makefile.serrano_cpu, "Makefile.serrano_kokkos_omp"_bench/Makefile.serrano_kokkos_omp, "Makefile.serrano_kokkos_serial"_bench/Makefile.serrano_kokkos_serial :ul

[ride80]

GNU 4.9.3 g++ compiler, OpenMPI 1.10.5, Cuda 8.0.44
module load openmpi/1.10.6/gcc/4.9.3/cuda/8.0.44
Makefiles: "Makefile.ride80_kokkos_cuda"_bench/Makefile.ride80_kokkos_cuda :ul

[ride100]

GNU 4.9.3 g++ compiler, OpenMPI 1.10.5, Cuda 8.0.44
module load openmpi/1.10.6/gcc/4.9.3/cuda/8.0.44
Makefiles: "Makefile.ride100_kokkos_cuda"_bench/Makefile.ride100_kokkos_cuda :ul

If a specific benchmark requires a build with additional package(s)
installed, it is noted with the benchmark info below.

With the software environment initialized (e.g. modules loaded) and
the machine Makefiles copied into src/MAKE/MINE, building SPARTA is
straightforward:

cp Makefile.serrano_kokkos_omp sparta/src/MAKE/MINE   # for example
cd sparta/src
make yes-kokkos                                       # install accelerator package(s) supported by the Makefile
make serrano_kokkos_omp                               # target = suffix of Makefile.machine :pre

This should produce an executable named spa_machine,
e.g. spa_serrano_kokkos_omp.  If desired, you can copy the executable to a
directory where you run the benchmark.

IMPORTANT NOTE: Achieving best performance for the benchmarks (or your
own input script) on a particular machine with a particular
accelerator option, requires attention to the following issues.

mpirun command-line arguments which control how MPI tasks and threads
are assigned to nodes and cores. :ulb,l

SPARTA command-line arguments which invoke a specific accelerator
package and its options.  This may include options that are part of
the "package"_doc/package.html command, which can be specified in the
input script, or as below, invoked from the command line. :l

Some of the benchmarks use slightly-modified input scripts (indicated
below), depending on which package is used.  This is to boost
performance of a specific accelerator option. :l

Performance can be a strong function of problem size (see plots
below).  In addition, performance of an accelerator package can vary
with MPI tasks/node, MPI tasks/GPU, threads/MPI task, or hardware
threads/core (hyperthreading).  In the tables below we show which
choices gave best performance for specific problem sizes.  But you may
need to experiment for your simulation or machine. :l,ule

All of the plots below include a link to a table with details on all
of these issues.  The table shows the mpirun (or equivalent) command
used to produce each data point on each curve in the plot, the SPARTA
command-line arguments used to get best performance with a particular
package on that hardware, and a link to the logfile produced by the
benchmark run.

:line

How to interpret the plots :h4,link(plots)

All the plots below have particles or nodes on the x-axis, and performance
on the y-axis.  On all the plots, better performance is up and worse
performance is down.  For all the plots:

Data is normalized so that ideal performance (with respect to particle or
node count) would be a horizontal line.  :ulb,l

If a curve trends downward (moving to the right) it means scalability
is falling off.  For example, in the strong-scaling plots, this is
typically because the problem size/node is getting smaller as the
number of nodes increases. :l

If a point is missing from a curve, the simulation may have run out of memory or time,
or the number of requested nodes was greater than the number of nodes on the machine. :l

If a curve trends upward, scalability is increasing.  For example, in
the per-node plots for GPUs, simulations typically run faster (on a
per-particle basis) as the system size increases. :l,ule

Per-core and per-node plots:

The y-axis is millions of particle-timesteps/sec, running on one core or
an entire node. :ulb,l

To infer timesteps/sec, divide the y-axis value by the number of
particles in the simulation. :l

The inverse of the y-axis value is sec/particle/timestep. :l

To estimate how long a simulation with N particles for M timesteps will
take in CPU seconds, multiply the inverse by N*M times 1
million. :l,ule

Strong-scaling and weak-scaling plots:

Strong scaling means a problem of the same size is run on
successively more nodes.  :ulb,l

Weak scaling means the problem size is doubled each time the node
count doubles.  For example, if the problem size on 1 node is a
million particles, then the problem size on 512 nodes is ~1/2 billion
particles. :l

The y-axis is millions of particle-timesteps/sec/node. :l

To infer timesteps/sec, multiply the y-axis value by the number of
nodes and divide by the number of particles in the simulation. :l

The inverse of the y-axis value is sec-node/particle/timestep. :l

To estimate how long a simulation with N particles for M timesteps on P
nodes will take in CPU seconds, multiply the inverse by N*M
times 1 million and divide by P. :l,ule