cuda-samples/cpp/0_Introduction/simpleMultiGPU

Sample: simpleMultiGPU

Description

A multi-GPU reduction demo: the input vector is split across every GPU in the system, each GPU sums its slice concurrently, and the host combines the per-GPU results. Each GPU gets its own CUDA stream, so the H2D copy → reduction kernel → D2H copy pipelines run in parallel across devices. The device-side reduction uses cub::BlockReduce; the host finishes the sum across blocks and across GPUs and checks it against a CPU reference. The whole GPU phase is timed with CUDA events.

What You'll Learn

• Enumerating and driving multiple GPUs with cudaGetDeviceCount and cudaSetDevice
• Partitioning a workload across GPUs and running them concurrently, one CUDA stream per GPU
• Overlapping H2D copy, kernel, and D2H copy with cudaMemcpyAsync on independent streams
• Reducing within a block using cub::BlockReduce, then finishing the reduction on the host
• Timing GPU work with CUDA events (cudaEventRecord / cudaEventElapsedTime)
• Why pinned host memory (cudaMallocHost) is required for cudaMemcpyAsync to be asynchronous

Key Concepts

One Stream per GPU

The work is held in a std::vector<TGPUplan> with one entry per device, each carrying its own stream, buffers, and data slice. Selecting a device with cudaSetDevice and issuing async work on that device's stream lets all GPUs run at the same time:

GPU 0 stream: [H2D slice 0] → [reduceKernel] → [D2H partials 0]
GPU 1 stream: [H2D slice 1] → [reduceKernel] → [D2H partials 1]
...

The host launches every GPU's pipeline before synchronizing any of them, so the copies and kernels overlap across devices.

Multi-Level Reduction

The reduction happens in three stages. On each GPU, reduceKernel launches BLOCK_N blocks of THREAD_N threads: every thread accumulates a grid-strided partial sum, then cub::BlockReduce combines the per-thread partials within its block and thread 0 writes one value per block. The host then adds the BLOCK_N per-block partials into that GPU's sum, and finally adds the per-GPU sums together.

Pinned Host Memory

cudaMemcpyAsync only runs asynchronously when the host buffer is page-locked (pinned), so the OS cannot move the pages mid-transfer; with ordinary pageable memory the copy falls back to synchronous behavior. Both the input slice and the partial-sum buffer are allocated with cudaMallocHost.

Timing with CUDA Events

CUDA events are recorded on device 0's stream to bracket the GPU phase. cudaEventElapsedTime returns the milliseconds between the recorded events.

Key APIs

CUDA Runtime

• cudaGetDeviceCount — count the CUDA-capable GPUs in the system
• cudaSetDevice — select the active GPU for subsequent CUDA calls
• cudaStreamCreate / cudaStreamDestroy — per-GPU stream that orders each pipeline
• cudaMalloc / cudaFree — allocate and free device memory
• cudaMallocHost / cudaFreeHost — allocate and free pinned host memory
• cudaMemcpyAsync — non-blocking H2D and D2H transfers on a stream
• cudaStreamSynchronize — block the host until a GPU's stream is done
• cudaEventCreate / cudaEventRecord / cudaEventSynchronize / cudaEventElapsedTime / cudaEventDestroy — event-based timing

CCCL / CUB

• cub::BlockReduce — block-wide reduction inside the kernel

Requirements

Hardware

• Two or more NVIDIA GPUs with Compute Capability 7.5 or higher

Software

• CMake 3.20 or newer
• A C++17-capable host compiler

How to Build

See the top-level README for full build instructions, including how to build all samples or a single sample standalone.

How to Run

./simpleMultiGPU

No command-line arguments are required. The sample uses every GPU it detects, and exits without running if it finds fewer than two.

Expected Output

Starting simpleMultiGPU
CUDA-capable device count: 2
Generating input data...

Computing with 2 GPUs...
  GPU Processing time: 8.138752 (ms)

Computing with Host CPU...

Comparing GPU and Host CPU results...
  GPU sum: 16777294.000000
  CPU sum: 16777294.395033
  Relative difference: 2.354566E-08

Reading the output:

• The device count and processing time depend on your system
• The sums are the same every run because the input is filled from an unseeded rand()
• The sample passes when the GPU and CPU results agree to within a relative difference of 1e-5

Files

• simpleMultiGPU.cu — reduction kernel and the multi-GPU driver
• README.md — this file
• CMakeLists.txt — build configuration

See Also

• CUDA Programming Guide — CUDA Streams
• CUDA Programming Guide — CUDA Events
• CUDA Core Compute Libraries — What is CUB?