NASP/cuda-samples
84
0
Fork 0
mirror of https://github.com/NVIDIA/cuda-samples.git synced 2026-10-13 00:08:26 +08:00
Code Issues Packages Projects Releases Wiki Activity

CUDA 13.2 samples update (#432)

Browse Source 
- Added Python samples for CUDA Python 1.0 release
- Renamed top-level `Samples` directory to `cpp` to accommodate Python samples.
This commit is contained in:
Dheemanth
2026-05-13 15:13:18 -07:00
committed by GitHub
parent a4526d5229
commit aeab82ff30
2072 changed files with 15077 additions and 1970 deletions
Download Patch File Download Diff File Expand all files Collapse all files

102
python/2_CoreConcepts/blockwiseSum/README.md Normal file
View File

@@ -0,0 +1,102 @@
# Sample: Block-wise Array Sum (Python)
## Description
Demonstrates fundamental CUDA thread cooperation: thread/block indexing, strided loops, and block-wise reduction using shared memory. This sample shows three progressively complex kernel patterns using the **cuda.core API**:
1. **Simple indexing** - One thread per element
2. **Strided loop** - Each thread processes multiple elements
3. **Block partial sum** - Shared memory reduction within each block
## What You'll Learn
- How to calculate global thread ID from block and thread indices
- Strided loop pattern for processing arrays larger than grid size
- Block-level cooperation using shared memory and `__syncthreads()`
## Key Concepts
### Thread and Block Indexing
```
Global Thread ID = blockIdx.x * blockDim.x + threadIdx.x
Stride = blockDim.x * gridDim.x
```
### Strided Loop Pattern
Each thread processes multiple elements, enabling fixed grid size for arbitrary array lengths:
```c
for (size_t i = tid; i < N; i += stride) {
output[i] = input[i] * 2.0f;
}
```
## Key APIs
### From `cuda.core`:
- `Device` - Device management and context
- `Program` - Compile CUDA C++ kernels
- `ProgramOptions` - Kernel compilation options (architecture target)
- `LaunchConfig` - Configure grid/block dimensions and shared memory
- `launch()` - Execute kernel
- `EventOptions` - GPU timing configuration
### From CuPy:
- `cp.asarray()` - Transfer data to GPU
- `cp.zeros_like()` - Allocate GPU arrays
## Requirements
### Hardware:
- NVIDIA GPU with CUDA support
### Software:
- CUDA Toolkit 13.0 or newer
- Python 3.10 or newer
- See `requirements.txt` for Python packages
## Installation
```bash
pip install -r requirements.txt
```
## How to Run
```bash
python blockwiseSum.py
```
## Expected Output
```
Device: <Your GPU>
Compute Capability: sm_XX
Array size: 1,048,576 elements
Simple indexing: Test PASSED
Strided loop: Test PASSED
Block-wise sum: Test PASSED
Kernel time: X.XXX ms, Bandwidth: XXX.X GB/s
Done
```
## Files
- `blockwiseSum.py` - Python implementation with CUDA kernels
- `README.md` - This file
- `requirements.txt` - Sample dependencies
## See Also
- [cuda.core Documentation](https://nvidia.github.io/cuda-python/cuda-core/latest/)
- [CUDA Shared Memory](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#shared-memory)
- [CuPy Documentation](https://docs.cupy.dev/)

259
python/2_CoreConcepts/blockwiseSum/blockwiseSum.py Normal file
View File

@@ -0,0 +1,259 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Block-wise Array Sum with Threaded Access
Demonstrates thread/block indexing, strided loops, and block-wise reduction.
Key Concepts:
Global Thread ID = blockIdx.x * blockDim.x + threadIdx.x
Stride = blockDim.x * gridDim.x
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
from cuda_samples_utils import verify_array_result
try:
import cupy as cp
import numpy as np
from cuda.core import (
Device,
EventOptions,
LaunchConfig,
Program,
ProgramOptions,
launch,
)
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Install with: pip install -r requirements.txt")
sys.exit(1)
KERNELS_CODE: str = r"""
// Each thread processes one element
extern "C" __global__
void simple_indexing(const float* input, float* output, size_t N) {
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
if (tid < N) {
output[tid] = input[tid] * 2.0f;
}
}
// Each thread processes multiple elements via strided access
extern "C" __global__
void strided_loop(const float* input, float* output, size_t N) {
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
size_t stride = (size_t)blockDim.x * gridDim.x;
for (size_t i = tid; i < N; i += stride) {
output[i] = input[i] * 2.0f;
}
}
// Block-wise partial sum with shared memory reduction
extern "C" __global__
void block_partial_sum(const float* input, float* partial_sums, size_t N) {
extern __shared__ float sdata[];
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
unsigned int local_tid = threadIdx.x;
size_t stride = (size_t)blockDim.x * gridDim.x;
// Each thread accumulates multiple elements (strided)
float sum = 0.0f;
for (size_t i = tid; i < N; i += stride) {
sum += input[i];
}
sdata[local_tid] = sum;
__syncthreads();
// Block-level tree reduction
for (int s = blockDim.x / 2; s > 0; s >>= 1) {
if (local_tid < s) {
sdata[local_tid] += sdata[local_tid + s];
}
__syncthreads();
}
if (local_tid == 0) {
partial_sums[blockIdx.x] = sdata[0];
}
}
"""
def run_sample(num_elements: int = 1024 * 1024, device_id: int = 0) -> bool:
"""
Run block-wise sum demonstration.
Parameters
----------
num_elements : int
Number of array elements
device_id : int
CUDA device ID
Returns
-------
bool
True if all tests passed
"""
threads_per_block = 256
num_blocks = 64
device = Device(device_id)
device.set_current()
stream = device.create_stream()
arch = f"sm_{device.arch}"
print(f"Device: {device.name}")
print(f"Compute Capability: {arch}")
print(f"Array size: {num_elements:,} elements\n")
try:
# Make CuPy use our stream
cp.cuda.ExternalStream(int(stream.handle)).use()
# Compile kernels
program = Program(
KERNELS_CODE, code_type="c++", options=ProgramOptions(arch=arch)
)
module = program.compile(target_type="cubin")
kernel_simple = module.get_kernel("simple_indexing")
kernel_strided = module.get_kernel("strided_loop")
kernel_sum = module.get_kernel("block_partial_sum")
# Test data
h_input = np.arange(num_elements, dtype=np.float32)
d_input = cp.asarray(h_input)
d_output = cp.zeros_like(d_input)
expected = cp.asarray(h_input * 2.0)
# Demo 1: Simple indexing (1 thread = 1 element)
full_blocks = (num_elements + threads_per_block - 1) // threads_per_block
config = LaunchConfig(grid=full_blocks, block=threads_per_block)
launch(
stream,
config,
kernel_simple,
d_input.data.ptr,
d_output.data.ptr,
cp.uint64(num_elements),
)
stream.sync()
print("Simple indexing: ", end="")
test1 = verify_array_result(d_output, expected)
# Demo 2: Strided loop (threads process multiple elements)
d_output.fill(0)
config = LaunchConfig(grid=num_blocks, block=threads_per_block)
launch(
stream,
config,
kernel_strided,
d_input.data.ptr,
d_output.data.ptr,
cp.uint64(num_elements),
)
stream.sync()
print("Strided loop: ", end="")
test2 = verify_array_result(d_output, expected)
# Demo 3: Block-wise sum with shared memory
d_ones = cp.ones(num_elements, dtype=cp.float32)
d_partial = cp.zeros(num_blocks, dtype=cp.float32)
shared_mem = threads_per_block * 4
config = LaunchConfig(
grid=num_blocks, block=threads_per_block, shmem_size=shared_mem
)
launch(
stream,
config,
kernel_sum,
d_ones.data.ptr,
d_partial.data.ptr,
cp.uint64(num_elements),
)
stream.sync()
# Each block sums num_elements/num_blocks elements (strided access).
# Requires num_elements % num_blocks == 0 for correct expected values.
assert (
num_elements % num_blocks == 0
), "num_elements must be divisible by num_blocks for block_partial_sum"
expected_partial = cp.full(
num_blocks, num_elements / num_blocks, dtype=cp.float32
)
print("Block-wise sum: ", end="")
test3 = verify_array_result(d_partial, expected_partial)
# Performance timing
event_opts = EventOptions(enable_timing=True)
iterations = 100
stream.sync()
start = stream.record(options=event_opts)
for _ in range(iterations):
launch(
stream,
config,
kernel_sum,
d_ones.data.ptr,
d_partial.data.ptr,
cp.uint64(num_elements),
)
end = stream.record(options=event_opts)
end.sync()
time_ms = (end - start) / iterations
bandwidth = (num_elements * 4) / (time_ms * 1e6)
print(f"\nKernel time: {time_ms:.3f} ms, Bandwidth: {bandwidth:.1f} GB/s")
return test1 and test2 and test3
finally:
# Explicit resource cleanup
cp.cuda.Stream.null.use()
stream.close()
def main() -> None:
"""Entry point."""
success = run_sample()
if success:
print("\nDone")
else:
print("\nSome tests failed")
sys.exit(1)
if __name__ == "__main__":
main()

6
python/2_CoreConcepts/blockwiseSum/requirements.txt Normal file
View File

@@ -0,0 +1,6 @@
# Block-wise Array Sum Requirements
cuda-python>=13.0.0
cuda-core>=0.6.0
cupy-cuda13x>=13.0.0
numpy>=2.3.2

140
python/2_CoreConcepts/cudaGraphs/README.md Normal file
View File

@@ -0,0 +1,140 @@
# cudaGraphs (Python)
## Description
This sample demonstrates how to capture a multi-stage kernel pipeline as a
CUDA graph with `cuda.core` and replay it with a single driver call.
The sample runs a three-stage elementwise pipeline
`r3 = (a + b) * c - a` in two modes:
1. **Individual launches** - one `launch(stream, ...)` per stage, repeated
for every iteration of the pipeline.
2. **CUDA graph replay** - the same three launches are recorded into a
`Graph` once and replayed with `graph.launch(stream)` on each
iteration.
Both paths are timed over N iterations and their results are verified
against a reference computation. The sample also re-launches the graph
after mutating the input buffers to show that the graph captures
pointers (not data), so the same graph can process new inputs without
rebuilding.
## What You'll Learn
- Creating a `GraphBuilder` from a stream with `stream.create_graph_builder()`
- Capturing launches with `begin_building()` and `end_building()`
- Completing a graph with `builder.complete()` and uploading it to a stream
- Replaying the graph with `graph.launch(stream)`
- Measuring the launch-overhead savings for small kernels
- Re-running the same graph against updated input data
## Key Libraries
- [`cuda.core`](https://nvidia.github.io/cuda-python/cuda-core/latest/) - Pythonic access to CUDA runtime, programs, and graphs
- `cupy` - input buffers and result verification
- `numpy` - scalar kernel arguments
## Key APIs
### From `cuda.core`
- `Stream.create_graph_builder()` - obtain a `GraphBuilder`
- `GraphBuilder.begin_building()` / `end_building()` - begin and finish recording launches issued against the builder
- `GraphBuilder.complete()` - produce an executable `Graph`
- `Graph.upload(stream)` - upload the graph structure to the device
- `Graph.launch(stream)` - replay the entire graph
- `launch(graph_builder, config, kernel, ...)` - record a kernel launch into the graph being built
### From `cuda_samples_utils`
- `print_gpu_info()` - print device name and compute capability
## Requirements
### Hardware
- NVIDIA GPU with Compute Capability 7.0 or higher
- Minimum GPU memory: 512 MB
### Software
- CUDA Toolkit 13.0 or newer (matches `cuda-python` 13.x)
- Python 3.10 or newer
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=0.6.0)
- `cupy-cuda13x` (>=13.0.0)
## Installation
Install the required packages from `requirements.txt`:
```bash
cd /path/to/cuda-samples/python/2_CoreConcepts/cudaGraphs
pip install -r requirements.txt
```
The `requirements.txt` installs:
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=0.6.0)
- `cupy-cuda13x` (>=13.0.0)
## How to Run
### Basic usage
```bash
cd cuda-samples/python/2_CoreConcepts/cudaGraphs
python cudaGraphs.py
```
### With custom parameters
```bash
# Larger vectors and more iterations
python cudaGraphs.py --elements 4096 --iters 2000
# Use a specific GPU
python cudaGraphs.py --device 1
```
Short vectors exaggerate the launch-overhead savings; larger vectors
will show the two approaches converging because per-launch overhead
becomes negligible next to kernel runtime.
## Expected Output
Speedup numbers vary with GPU and host CPU.
```
Device: <Your GPU Name>
Compute Capability: <X.Y>
Individual launches: 1000 iters in 0.0085s (8.49 us/iter)
Building CUDA graph...
Graph replay: 1000 iters in 0.0034s (3.41 us/iter)
Graph speedup: 2.49x
Graph replay on updated data verified (same graph, new buffer contents)
Done
```
**Note:** Device name, compute capability, and speedup will vary based on
your GPU and host CPU.
## Files
- `cudaGraphs.py` - Python implementation using `cuda.core` CUDA graphs
- `README.md` - This file
- `requirements.txt` - Sample dependencies
- `../../Utilities/cuda_samples_utils.py` - Common utilities (imported by this sample)
## See Also
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [`cuda.core` graphs API](https://nvidia.github.io/cuda-python/cuda-core/latest/api.html#cuda-graphs)
- Upstream `cuda.core` example: [`cuda_graphs.py`](https://github.com/NVIDIA/cuda-python/blob/main/cuda_core/examples/cuda_graphs.py)
- [CUDA Graphs programming guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs)

266
python/2_CoreConcepts/cudaGraphs/cudaGraphs.py Normal file
View File

@@ -0,0 +1,266 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
CUDA Graphs with cuda.core
CUDA graphs let you record a DAG of operations once, then replay the entire
graph with a single driver call. For workflows that issue many small kernels
this can significantly reduce CPU-side launch overhead.
This sample runs a three-stage elementwise pipeline (add -> multiply ->
subtract) in two modes:
1. Individually launched kernels on a stream.
2. A single CUDA graph that captures the same three launches and is
replayed with ``graph.launch(stream)``.
We then measure the wall-clock time of each mode across many iterations to
illustrate the graph replay advantage for short kernels, and demonstrate that
a graph can be relaunched against new data (the pointers are baked in, but
the contents of those buffers are not).
"""
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
try:
import cupy as cp
import numpy as np
from cuda.core import Device, LaunchConfig, Program, ProgramOptions, launch
from cuda_samples_utils import print_gpu_info # noqa: E402
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install from requirements.txt:")
print(" pip install -r requirements.txt")
sys.exit(1)
PIPELINE_KERNELS = r"""
extern "C" __global__
void vec_add(const float* A, const float* B, float* C, size_t N) {
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
size_t stride = (size_t)gridDim.x * blockDim.x;
for (size_t i = tid; i < N; i += stride) C[i] = A[i] + B[i];
}
extern "C" __global__
void vec_mul(const float* A, const float* B, float* C, size_t N) {
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
size_t stride = (size_t)gridDim.x * blockDim.x;
for (size_t i = tid; i < N; i += stride) C[i] = A[i] * B[i];
}
extern "C" __global__
void vec_sub(const float* A, const float* B, float* C, size_t N) {
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
size_t stride = (size_t)gridDim.x * blockDim.x;
for (size_t i = tid; i < N; i += stride) C[i] = A[i] - B[i];
}
"""
def run_pipeline_individual(stream, kernels, config, buffers, size, n_iters):
"""Run the 3-stage pipeline `n_iters` times with one launch per stage."""
add_k, mul_k, sub_k = kernels
a, b, c, r1, r2, r3 = buffers
stream.sync()
t0 = time.perf_counter()
for _ in range(n_iters):
launch(
stream, config, add_k, a.data.ptr, b.data.ptr, r1.data.ptr, np.uint64(size)
)
launch(
stream, config, mul_k, r1.data.ptr, c.data.ptr, r2.data.ptr, np.uint64(size)
)
launch(
stream, config, sub_k, r2.data.ptr, a.data.ptr, r3.data.ptr, np.uint64(size)
)
stream.sync()
return time.perf_counter() - t0
def build_graph(stream, kernels, config, buffers, size):
"""Capture the 3-stage pipeline into a CUDA graph and return it."""
add_k, mul_k, sub_k = kernels
a, b, c, r1, r2, r3 = buffers
graph_builder = stream.create_graph_builder()
graph_builder.begin_building()
launch(
graph_builder,
config,
add_k,
a.data.ptr,
b.data.ptr,
r1.data.ptr,
np.uint64(size),
)
launch(
graph_builder,
config,
mul_k,
r1.data.ptr,
c.data.ptr,
r2.data.ptr,
np.uint64(size),
)
launch(
graph_builder,
config,
sub_k,
r2.data.ptr,
a.data.ptr,
r3.data.ptr,
np.uint64(size),
)
graph_builder.end_building()
graph = graph_builder.complete()
graph.upload(stream)
return graph_builder, graph
def run_pipeline_graph(stream, graph, n_iters):
"""Launch the compiled graph `n_iters` times."""
stream.sync()
t0 = time.perf_counter()
for _ in range(n_iters):
graph.launch(stream)
stream.sync()
return time.perf_counter() - t0
def main() -> int:
import argparse
parser = argparse.ArgumentParser(description="CUDA Graphs demo with cuda.core")
parser.add_argument(
"--elements",
type=int,
default=1 << 12,
help="Elements per vector (default: 4096 - small to emphasize launch overhead)",
)
parser.add_argument(
"--iters",
type=int,
default=1000,
help="Number of pipeline iterations to time (default: 1000)",
)
parser.add_argument("--device", type=int, default=0, help="CUDA device id")
args = parser.parse_args()
device = Device(args.device)
device.set_current()
print_gpu_info(device)
stream = device.create_stream()
# Tell CuPy to order its allocations on our stream so buffer initialization
# below is serialized with the kernels we launch.
cp.cuda.ExternalStream(int(stream.handle)).use()
graph_builder = graph = None
try:
program_options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
program = Program(PIPELINE_KERNELS, code_type="c++", options=program_options)
module = program.compile("cubin")
add_k = module.get_kernel("vec_add")
mul_k = module.get_kernel("vec_mul")
sub_k = module.get_kernel("vec_sub")
kernels = (add_k, mul_k, sub_k)
N = args.elements
rng = cp.random.default_rng(seed=0)
a = rng.random(N, dtype=cp.float32)
b = rng.random(N, dtype=cp.float32)
c = rng.random(N, dtype=cp.float32)
r1 = cp.empty_like(a)
r2 = cp.empty_like(a)
r3 = cp.empty_like(a)
buffers = (a, b, c, r1, r2, r3)
expected = (a + b) * c - a
config = LaunchConfig(grid=(N + 255) // 256, block=256)
device.sync()
# Warm up compilation/caches, then measure individual launches.
run_pipeline_individual(stream, kernels, config, buffers, N, n_iters=5)
t_individual = run_pipeline_individual(
stream, kernels, config, buffers, N, n_iters=args.iters
)
assert cp.allclose(r3, expected, rtol=1e-5, atol=1e-5), (
"Individual pipeline produced incorrect results"
)
print(
f"\nIndividual launches: {args.iters} iters in {t_individual:.4f}s"
f" ({t_individual * 1e6 / args.iters:.2f} us/iter)"
)
# Capture the same pipeline as a graph and measure the replay.
print("\nBuilding CUDA graph...")
graph_builder, graph = build_graph(stream, kernels, config, buffers, N)
run_pipeline_graph(stream, graph, n_iters=5) # warm up
t_graph = run_pipeline_graph(stream, graph, n_iters=args.iters)
assert cp.allclose(r3, expected, rtol=1e-5, atol=1e-5), (
"Graph pipeline produced incorrect results"
)
print(
f"Graph replay: {args.iters} iters in {t_graph:.4f}s"
f" ({t_graph * 1e6 / args.iters:.2f} us/iter)"
)
if t_graph > 0:
print(f"Graph speedup: {t_individual / t_graph:.2f}x")
# Demonstrate that the graph replays against current buffer contents.
a[:] = cp.ones(N, dtype=cp.float32)
b[:] = cp.full(N, 2.0, dtype=cp.float32)
c[:] = cp.full(N, 3.0, dtype=cp.float32)
device.sync()
# r3 = (a + b) * c - a = (1 + 2) * 3 - 1 = 8
graph.launch(stream)
stream.sync()
assert cp.allclose(r3, 8.0), "Graph replay with new data produced wrong result"
print(
"\nGraph replay on updated data verified (same graph, new buffer contents)"
)
print("\nDone")
return 0
finally:
if graph is not None:
graph.close()
if graph_builder is not None:
graph_builder.close()
stream.close()
cp.cuda.Stream.null.use()
if __name__ == "__main__":
sys.exit(main())

5
python/2_CoreConcepts/cudaGraphs/requirements.txt Normal file
View File

@@ -0,0 +1,5 @@
# CUDA Graphs Sample Requirements
cuda-python>=13.0.0
cuda-core>=0.6.0
cupy-cuda13x>=13.0.0

136
python/2_CoreConcepts/fftSignalAnalysis/README.md Normal file
View File

@@ -0,0 +1,136 @@
# Sample: FFT Signal Analysis (Python)
## Description
Analyze signal frequencies using Fast Fourier Transform (FFT) on the GPU. This sample demonstrates CuPy's cuFFT for GPU-accelerated frequency analysis: generating composite signals, computing magnitude spectrum, detecting dominant frequencies via peak detection, and comparing GPU vs CPU FFT performance.
## What You'll Learn
- Using CuPy's `cp.fft.rfft()` for real-to-complex FFT on GPU
- Computing magnitude spectrum from FFT results
- Peak detection to identify dominant frequencies
- Comparing GPU (cuFFT) vs CPU (NumPy) FFT performance
- Uses cuda.core APIs for device management and CUDA event timing
## Key Concepts
- **FFT (Fast Fourier Transform)**: Efficiently computes the Discrete Fourier Transform
- **Magnitude Spectrum**: `|FFT(signal)| * 2 / N` gives amplitude at each frequency
- **rfft**: Real FFT - optimized for real-valued input signals
- **Peak Detection**: Finding local maxima to identify dominant frequencies
### Stream Interop
This sample demonstrates CuPy integration with cuda.core streams:
```python
# Create stream with cuda.core
stream = device.create_stream()
# Use with CuPy operations
cp.cuda.ExternalStream(int(stream.handle)).use()
```
## Key APIs
### From `cuda.core`:
- `Device` - Device management and context
- `EventOptions` - Configure events for GPU timing
- `stream.record()` - Record events for timing
### From CuPy:
- `cp.fft.rfft()` - Real-to-complex FFT (GPU-accelerated via cuFFT)
- `cp.fft.rfftfreq()` - Generate frequency bins for rfft
- `cp.cuda.ExternalStream()` - Interop with cuda.core streams
### From NumPy:
- `np.fft.rfft()` - CPU FFT for comparison
## Requirements
### Hardware:
- NVIDIA GPU with CUDA support
### Software:
- CUDA Toolkit 13.0 or newer
- Python 3.10 or newer
- See `requirements.txt` for Python packages
## Installation
```bash
pip install -r requirements.txt
```
## How to Run
```bash
python fftSignalAnalysis.py
```
## Expected Output
```
============================================================
FFT Signal Analysis
============================================================
Device: <Your GPU>
Compute Capability: sm_XX
Signal Parameters:
Samples: 1,048,576
Sample Rate: 44,100 Hz
...
------------------------------------------------------------
GPU FFT (cuFFT)
------------------------------------------------------------
Time: X.XXX ms
Detected Frequencies:
440.0 Hz (magnitude: X.XXXX)
...
------------------------------------------------------------
CPU FFT (NumPy)
------------------------------------------------------------
Time: XX.XXX ms
------------------------------------------------------------
PERFORMANCE SUMMARY
------------------------------------------------------------
GPU (cuFFT): X.XXX ms
CPU (NumPy): XX.XXX ms
Speedup: XXx
------------------------------------------------------------
VERIFICATION
------------------------------------------------------------
GPU vs CPU FFT magnitude: Test PASSED
Frequency Detection Accuracy:
440 Hz: ✓
...
Done
```
**Note:** Times and speedup vary by hardware.
## Files
- `fftSignalAnalysis.py` - Main sample using cuda.core and CuPy
- `README.md` - This file
- `requirements.txt` - Dependencies
## See Also
- [cuda.core Documentation](https://nvidia.github.io/cuda-python/cuda-core/latest/)
- [CuPy FFT Documentation](https://docs.cupy.dev/en/stable/reference/fft.html)
- [cuFFT Documentation](https://docs.nvidia.com/cuda/cufft/)

318
python/2_CoreConcepts/fftSignalAnalysis/fftSignalAnalysis.py Normal file
View File

@@ -0,0 +1,318 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
FFT Signal Analysis
Demonstrates how to analyze signal frequencies using Fast Fourier Transform (FFT):
- Generate composite signals with multiple frequency components
- Use CuPy's cuFFT for GPU-accelerated frequency analysis
- Detect dominant frequencies (peak detection)
- Compare GPU vs CPU FFT performance
Uses cuda.core APIs for device management and timing.
"""
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
from cuda_samples_utils import verify_array_result
try:
import cupy as cp
import numpy as np
from cuda.core import Device, EventOptions
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Install with: pip install -r requirements.txt")
sys.exit(1)
def generate_composite_signal(
num_samples: int,
sample_rate: float,
frequencies: list[float],
amplitudes: list[float],
) -> np.ndarray:
"""
Generate a composite signal with multiple frequency components.
Parameters
----------
num_samples : int
Number of samples in the signal
sample_rate : float
Sampling rate in Hz
frequencies : list[float]
List of frequency components in Hz
amplitudes : list[float]
List of amplitudes for each frequency component
Returns
-------
np.ndarray
Signal array
"""
t = np.arange(num_samples, dtype=np.float32) / sample_rate
signal = np.zeros(num_samples, dtype=np.float32)
for freq, amp in zip(frequencies, amplitudes):
signal += amp * np.sin(2 * np.pi * freq * t)
return signal
def find_dominant_frequencies(
fft_magnitude: cp.ndarray,
frequencies: cp.ndarray,
num_peaks: int = 5,
threshold_ratio: float = 0.1,
) -> list[tuple[float, float]]:
"""
Find dominant frequencies from FFT magnitude spectrum.
Uses CPU-based peak detection (transfers magnitude/frequencies via cp.asnumpy).
Suitable for small-to-medium spectra; for large-scale analysis, consider
GPU-native peak detection.
Parameters
----------
fft_magnitude : cp.ndarray
Magnitude of FFT (positive frequencies only)
frequencies : cp.ndarray
Frequency bins
num_peaks : int
Maximum number of peaks to return
threshold_ratio : float
Minimum peak height as ratio of max peak
Returns
-------
list[tuple[float, float]]
List of (frequency, magnitude) tuples for detected peaks
"""
# Find peaks above threshold
max_magnitude = float(cp.max(fft_magnitude))
threshold = max_magnitude * threshold_ratio
# Simple peak detection: find local maxima above threshold
magnitude_cpu = cp.asnumpy(fft_magnitude)
freq_cpu = cp.asnumpy(frequencies)
peaks = []
for i in range(1, len(magnitude_cpu) - 1):
if magnitude_cpu[i] > threshold:
if (
magnitude_cpu[i] > magnitude_cpu[i - 1]
and magnitude_cpu[i] > magnitude_cpu[i + 1]
):
peaks.append((freq_cpu[i], magnitude_cpu[i]))
# Sort by magnitude and return top peaks
peaks.sort(key=lambda x: x[1], reverse=True)
return peaks[:num_peaks]
def run_fft_analysis(
num_samples: int = 2**20,
sample_rate: float = 44100.0,
device_id: int = 0,
num_iterations: int = 10,
) -> bool:
"""
Run FFT signal analysis benchmark.
device_id and num_iterations are not exposed via CLI; modify defaults
or call this function directly for customization.
Parameters
----------
num_samples : int
Number of samples (power of 2 recommended for FFT)
sample_rate : float
Sampling rate in Hz
device_id : int
CUDA device ID
num_iterations : int
Number of iterations for timing
Returns
-------
bool
True if analysis succeeded
"""
print("=" * 60)
print("FFT Signal Analysis")
print("=" * 60)
# Initialize device
device = Device(device_id)
device.set_current()
stream = device.create_stream()
try:
print(f"\nDevice: {device.name}")
print(f"Compute Capability: sm_{device.arch}")
# Make CuPy use our cuda.core stream
cp.cuda.ExternalStream(int(stream.handle)).use()
# Define test signal: composite of multiple frequencies
test_frequencies = [440.0, 880.0, 1320.0, 2000.0, 5000.0] # Hz
test_amplitudes = [1.0, 0.5, 0.3, 0.7, 0.4]
print("\nSignal Parameters:")
print(f" Samples: {num_samples:,}")
print(f" Sample Rate: {sample_rate:,.0f} Hz")
print(f" Duration: {num_samples / sample_rate:.3f} seconds")
print(f" Input Frequencies: {test_frequencies} Hz")
print(f" Input Amplitudes: {test_amplitudes}")
# Generate composite signal on CPU
h_signal = generate_composite_signal(
num_samples, sample_rate, test_frequencies, test_amplitudes
)
# Transfer to GPU
d_signal = cp.asarray(h_signal)
# ---------------------------------------------------------------------
# GPU FFT (cuFFT via CuPy)
# ---------------------------------------------------------------------
print("\n" + "-" * 60)
print("GPU FFT (cuFFT)")
print("-" * 60)
event_opts = EventOptions(enable_timing=True)
# Warmup
d_fft_result = cp.fft.rfft(d_signal)
stream.sync()
# Timed runs
start = stream.record(options=event_opts)
for _ in range(num_iterations):
d_fft_result = cp.fft.rfft(d_signal)
end = stream.record(options=event_opts)
end.sync()
gpu_time_ms = (end - start) / num_iterations
print(f"Time: {gpu_time_ms:.3f} ms")
# Compute magnitude spectrum
d_magnitude = cp.abs(d_fft_result) * 2 / num_samples
d_frequencies = cp.fft.rfftfreq(num_samples, 1 / sample_rate)
# Find dominant frequencies
detected_peaks = find_dominant_frequencies(d_magnitude, d_frequencies)
print("\nDetected Frequencies:")
for freq, mag in detected_peaks:
print(f" {freq:8.1f} Hz (magnitude: {mag:.4f})")
# ---------------------------------------------------------------------
# CPU FFT (NumPy) for comparison
# ---------------------------------------------------------------------
print("\n" + "-" * 60)
print("CPU FFT (NumPy)")
print("-" * 60)
# Warmup
h_fft_result = np.fft.rfft(h_signal)
# Timed runs
cpu_start = time.perf_counter()
for _ in range(num_iterations):
h_fft_result = np.fft.rfft(h_signal)
cpu_end = time.perf_counter()
cpu_time_ms = (cpu_end - cpu_start) * 1000 / num_iterations
print(f"Time: {cpu_time_ms:.3f} ms")
# ---------------------------------------------------------------------
# Performance Summary
# ---------------------------------------------------------------------
print("\n" + "-" * 60)
print("PERFORMANCE SUMMARY")
print("-" * 60)
speedup = cpu_time_ms / gpu_time_ms
print(f"GPU (cuFFT): {gpu_time_ms:.3f} ms")
print(f"CPU (NumPy): {cpu_time_ms:.3f} ms")
print(f"Speedup: {speedup:.1f}x")
# ---------------------------------------------------------------------
# Verification
# ---------------------------------------------------------------------
print("\n" + "-" * 60)
print("VERIFICATION")
print("-" * 60)
# Compare GPU and CPU results
h_magnitude = (
cp.asarray(np.abs(h_fft_result).astype(np.float32)) * 2 / num_samples
)
print("GPU vs CPU FFT magnitude: ", end="")
success = verify_array_result(
d_magnitude,
h_magnitude,
rtol=1e-4,
atol=1e-6,
)
# Verify detected frequencies match input
print("\nFrequency Detection Accuracy:")
detected_freqs = [freq for freq, _ in detected_peaks]
all_found = True
for expected_freq in test_frequencies:
found = any(abs(f - expected_freq) < 10 for f in detected_freqs)
status = "✓" if found else "✗"
print(f" {expected_freq:6.0f} Hz: {status}")
all_found = all_found and found
success = success and all_found
return success
finally:
# Cleanup - always close resources
cp.cuda.Stream.null.use()
stream.close()
def main() -> None:
"""Entry point."""
success = run_fft_analysis()
if success:
print("\nDone")
else:
print("\nAnalysis completed with errors")
sys.exit(1)
if __name__ == "__main__":
main()

6
python/2_CoreConcepts/fftSignalAnalysis/requirements.txt Normal file
View File

@@ -0,0 +1,6 @@
# FFT Signal Analysis Requirements
cuda-python>=13.0.0
cuda-core>=0.6.0
cupy-cuda13x>=13.0.0
numpy>=2.3.2

250
python/2_CoreConcepts/greenContext/README.md Normal file
View File

@@ -0,0 +1,250 @@
# greenContext (Python)
## Description
This sample demonstrates how to use **green contexts** with
`cuda.core` to statically partition a GPU's streaming multiprocessors
(SMs) so that independent kernels can run on dedicated subsets of the
device.
This examples takes A long-running kernel that fills the GPU's SMs,
and a short but latency-sensitive "critical" kernel is launched shortly after.
Without green contexts, the critical kernel must wait for SMs to
free up. With green contexts, the GPU's SMs are partitioned so the
critical kernel has its own dedicated SMs and can start immediately.
Three timed scenarios are compared:
1. **Reference**: the critical kernel alone on the primary context,
with no competing work. Establishes the pure compute time of the
critical kernel when every SM on the device is available to it.
2. **Baseline**: both kernels run on the device's primary context,
on two non-blocking streams that contend for all SMs.
3. **Green contexts**: the SMs are split into two disjoint groups
(e.g. 112 + 16). Each kernel runs on a stream that belongs to its
own green context, so the critical kernel never waits for SMs
held by the long-running kernel.
The headline metric is the total wall time of the critical kernel
from launch to completion. In the baseline it is dominated by time
spent waiting behind the long-running kernel. With green contexts it
reflects the kernel's own compute time on its (smaller) SM
partition. The reference row lets you separate those two effects:
- `baseline - reference` is roughly the time the critical kernel
spent waiting for SMs in the baseline run (the cost that green
contexts eliminate).
- `green / reference` is the compute slowdown caused by running on
a smaller SM partition (the cost that green contexts introduce).
## What You'll Learn
- Querying a device's SM resources via `Device.resources.sm` and
reading `sm_count`, `min_partition_size`, `coscheduled_alignment`
- Splitting an `SMResource` into disjoint partitions with
`sm.split(SMResourceOptions(count=(A, B)))`
- Creating a green context from an SM partition via
`Device.create_context(ContextOptions(resources=[group]))`
- Creating a non-blocking stream on a green context with
`ctx.create_stream()`
- Using CUDA events with timing enabled to measure kernel wall time
across streams
- Cleaning up green contexts safely with `ctx.close()`
## Key Libraries
- [`cuda.core`](https://nvidia.github.io/cuda-python/cuda-core/latest/) - device management, SM partitioning, green contexts, compilation, and launching
- `numpy` - scalar kernel arguments
## Key APIs
### From `cuda.core`
- `Device.resources.sm` - the device's SM-type device resource
- `SMResource.split(SMResourceOptions(count=(A, B)))` - partition SMs
into disjoint groups (plus an optional remainder)
- `Device.create_context(ContextOptions(resources=[sm_group]))` -
create a green context provisioned with a specific SM partition
- `Context.is_green` / `Context.resources` - introspect a green
context
- `Context.create_stream()` - create a non-blocking stream that is
tied to the green context's SM partition
- `Context.close()` - destroy a green context (must not be the
thread's current context when closed)
- `Device.create_event(EventOptions(timing_enabled=True))` /
`Stream.record(event)` / `event2 - event1` - measure elapsed time
in milliseconds between two events on the device
- `Program(..., ProgramOptions(std="c++17", arch=f"sm_{device.arch}"))`
/ `program.compile("cubin", name_expressions=(...))` - compile the
delay and critical kernels in one TU
- `launch(stream, LaunchConfig(grid=..., block=...), kernel, ...)` -
submit a kernel on a specific stream
## Requirements
### Hardware
- Any NVIDIA GPU supported by green contexts.
- Green-context SM partitioning is designed for larger server GPUs
(H100, H200, B200, ...) but works on any supported GPU as long as
the SM count is large enough to split meaningfully.
### Software
- NVIDIA driver >= 12.4
- CUDA Toolkit 13.0 or newer.
- Python 3.10 or newer.
- `cuda-core` (`>=1.0.0`)
## Installation
Install the required packages from `requirements.txt`:
```bash
cd /path/to/cuda-samples/python/2_CoreConcepts/greenContext
pip install -r requirements.txt
```
## How to Run
### Basic usage
The auto-default split reserves a small partition (~16 SMs) for the
critical kernel and gives the rest to the long-running kernel. The
exact sizes are chosen by probing the driver with a dry-run `sm.split`,
escalating the alignment granularity in powers of two until the driver
accepts the pair. This handles architectures where the driver enforces
stricter alignment (e.g. TPC/GPC-pair alignment on Blackwell) than the
reported `min_partition_size`. When that happens the sample prints a
`Note:` line with the granularity it landed on.
```bash
cd cuda-samples/python/2_CoreConcepts/greenContext
python greenContext.py
```
### Match the CUDA programming guide example (112 + 16)
```bash
python greenContext.py --split 112,16
```
### Tune the workload
```bash
# Longer long-running kernel, larger host launch gap
python greenContext.py --delay-us 3000 --launch-gap-ms 2.0
# Smaller/lighter critical kernel so its own compute time is negligible
python greenContext.py --critical-n 65536 --critical-iters 128
# Symmetric split: maximum SMs for the critical kernel, long kernel is
# roughly 2x slower but the critical kernel runs close to its reference time.
python greenContext.py --split 64,64
# Use a specific GPU
python greenContext.py --device 1
```
### All options
```
--device CUDA device ID (default: 0)
--split SM split as 'LONG,CRITICAL', e.g. '112,16'.
Each side must be a multiple of the device's
min_partition_size, and the driver may enforce additional
architecture-specific alignment (e.g. TPC/partition-grid
alignment on Blackwell). Omit --split to auto-select a
driver-accepted split.
--delay-us Per-block busy-wait of the delay kernel, in us (default: 2000)
--delay-waves Number of waves of the delay kernel on the long
partition. Drives the default --delay-blocks (default: 16)
--delay-blocks Number of blocks for the delay kernel. Overrides
--delay-waves if set.
(default: --delay-waves * device SM count)
--critical-n Work size of the critical kernel (default: 4194304)
--critical-iters Inner math-loop iterations inside the critical kernel.
Higher values make the critical kernel's own compute
time more substantial relative to its wait time
(default: 1024)
--launch-gap-ms Host delay between launching the long and critical
kernels (default: 1.0 ms)
```
## Expected Output
The output depends on the GPU and the number of SMs.
On an RTX 4090 (128 SMs) with the default auto split:
```
[Green Context Sample using CUDA Core API]
Device: NVIDIA GeForce RTX 4090
Compute Capability: sm_89
Total SMs: 128
Min. SM partition size: 2
SM co-scheduled alignment: 2
SM split (long/critical): 112 / 16
Workload parameters:
delay kernel: 2048 blocks, 2000 us/block (~32.0 ms on 128 SMs)
critical kernel: 4194304 elements, 1024 inner iterations
host launch gap: 1.0 ms
Compiling kernels ...
Running reference scenario (critical kernel alone) ...
Running baseline scenario (primary context) ...
Running green context scenario ...
scenario SMs (long/crit) long (ms) crit total (ms) crit offset (ms)
-------------------------------------------------------------------------------------------------------
crit alone (primary ctx) -/128 - 0.425 -
baseline (primary ctx) 128/128 32.034 30.024 1.090
green ctx (112+16 SMs) 112/16 38.017 2.696 1.075
long (ms) : wall time of the delay kernel
crit total (ms) : launch-to-complete wall time of the critical kernel
crit offset (ms) : when the critical stream started, relative to the long stream start
Critical-kernel latency speedup (baseline vs green ctx): 11.14x
Green-ctx compute cost vs unconstrained (crit alone): 6.34x
Baseline time spent waiting for SMs (not computing): ~29.60 ms
Done
```
**What to look for:**
- The critical kernel alone (reference row) takes only a fraction of
a millisecond; almost all of the baseline's `crit total` is time
spent queued waiting for SMs, not compute.
- The **critical kernel's wall time drops sharply** in the
green-context scenario (from ~30 ms to a few ms in the example
above) because it no longer waits for SMs held by the long-running
kernel.
- The **long-running kernel's duration may increase** proportional
to the reduction in SMs available to it (128 -> 112 SMs ~= 14%
slower; 128 -> 64 SMs ~= 2x slower). This is an expected tradeoff:
you reserve SMs for a critical kernel by taking them away from the
background workload.
- The **compute cost** ratio (`green / reference`) shows how close
the critical kernel is to ideal linear scaling with its SM count.
A 112/16 split gives the critical kernel only 12.5% of the SMs and
costs it roughly 6-7x its reference time; a 64/64 split gives it
half the SMs and costs roughly 1.5-2x.
- The `crit offset` column is approximately `--launch-gap-ms` in
both full scenarios; it confirms the host launched the critical
kernel the same amount of time after the long kernel in both runs.
Exact timings vary with GPU model, driver version, clock state, and
other concurrent GPU work.
## Files
- `greenContext.py` - Python implementation using `cuda.core` green-context APIs
- `README.md` - This file
- `requirements.txt` - Sample dependencies
## See Also
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [Green Contexts in the CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#green-contexts)
- [`cuda.core` green-context test suite](https://github.com/NVIDIA/cuda-python/blob/main/cuda_core/tests/test_green_context.py) - the authoritative API reference

752
python/2_CoreConcepts/greenContext/greenContext.py Normal file
View File

@@ -0,0 +1,752 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Green Context Sample using CUDA Core API.
Three scenarios are timed with CUDA events and compared:
1. Reference: the critical kernel alone on the primary context,
with no competing work. Establishes the pure compute time of
the critical kernel with access to every SM on the device.
2. Baseline: both kernels run on the device's primary context,
on two non-blocking streams. They contend for all SMs.
3. Green contexts: SMs are split into two disjoint groups; each
kernel runs on a stream belonging to its own green context.
The headline metric is the total wall time of the critical kernel
from launch to completion on its stream. In the baseline it is
dominated by waiting behind the long-running kernel; with green
contexts it reflects only the kernel's own compute time on a
smaller SM partition. The reference row separates those effects.
Note: Parallel execution on the GPU is never guaranteed. Green
contexts remove one common source of contention (shared SMs) but
they are not a hard scheduling promise.
"""
import argparse
import sys
import time
from dataclasses import dataclass
from typing import List, Optional, Tuple
import numpy as np
from cuda.core import (
ContextOptions,
Device,
EventOptions,
LaunchConfig,
Program,
ProgramOptions,
SMResourceOptions,
launch,
)
# Two CUDA kernels:
# 1. The delay kernel spins until `cycles` SM clock ticks have elapsed.
# 2. The critical kernel does a small amount of useful work.
KERNEL_SRC = r"""
extern "C" __global__ void delay_kernel(unsigned long long cycles)
{
unsigned long long start = clock64();
while ((unsigned long long)(clock64() - start) < cycles) { }
}
extern "C" __global__ void critical_kernel(float *out, int n, int iters)
{
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) {
// Two dependent accumulators so the compiler cannot collapse the
// loop into a closed-form expression. `iters` is a runtime argument
// for the same reason.
float v = (float)i * 1e-6f + 1.0f;
float u = (float)i * 1e-7f + 0.5f;
for (int k = 0; k < iters; ++k) {
v = v * 1.000001f + u;
u = u * 0.999999f + v * 1e-7f;
}
out[i] = v + u;
}
}
"""
@dataclass
class ScenarioResult:
name: str
critical_total_ms: float
critical_sm_count: int
long_ms: Optional[float] = None
critical_offset_ms: Optional[float] = None
long_sm_count: Optional[int] = None
def print_sm_topology(device: Device) -> None:
sm = device.resources.sm
print("[Green Context Sample using CUDA Core API]")
print(f"Device: {device.name}")
print(f"Compute Capability: sm_{device.arch}")
print(f"Total SMs: {sm.sm_count}")
print(f"Min. SM partition size: {sm.min_partition_size}")
print(f"SM co-scheduled alignment: {sm.coscheduled_alignment}")
def _align_down(n: int, k: int) -> int:
if k <= 0:
return n
return (n // k) * k
def _driver_accepts_split(sm, long_count: int, critical_count: int) -> bool:
if long_count <= 0 or critical_count <= 0:
return False
try:
groups, _ = sm.split(
SMResourceOptions(count=(long_count, critical_count)),
dry_run=True,
)
except Exception:
return False
actual = tuple(g.sm_count for g in groups)
return actual == (long_count, critical_count)
def _find_working_split(
sm, prefer_critical: Optional[int] = None
) -> Optional[Tuple[int, int, int]]:
"""
Probe the driver for a (long, critical) split it actually accepts.
Escalates the alignment granularity from `min_partition_size` upward in
powers of two, requiring BOTH sides to be multiples of the current
granularity. This handles architectures where the driver's true
allocation granularity is larger than the reported
`min_partition_size` (e.g. TPC/GPC-pair alignment on Blackwell: on a
188-SM part `min_partition_size` is 8 but the driver actually requires
each side to be a multiple of 16).
Returns (long_count, critical_count, granularity) or None. The
granularity is the smallest power-of-two multiple of
`min_partition_size` at which both sides are aligned and the driver
accepts the pair.
"""
total = sm.sm_count
min_part = sm.min_partition_size
if min_part <= 0:
return None
if prefer_critical is None or prefer_critical <= 0:
prefer_critical = max(min_part, min(16, total // 8))
# Escalate granularity in powers of two. The upper bound is half of
# `total` because below that we cannot fit two partitions of size
# >= granularity.
granularity = min_part
while granularity * 2 <= total:
base = max(granularity, _align_down(prefer_critical, granularity))
candidates: List[int] = []
seen = set()
def push(c: int) -> None:
if c >= granularity and c <= total - granularity and c not in seen:
seen.add(c)
candidates.append(c)
# Walk outward from `base` (the preferred critical size, aligned
# down to the current granularity) in steps of granularity.
push(base)
max_steps = max(total // granularity, 1)
for step in range(1, max_steps + 1):
push(base + step * granularity)
push(base - step * granularity)
for critical in candidates:
long_count = _align_down(total - critical, granularity)
if long_count < granularity:
continue
if _driver_accepts_split(sm, long_count, critical):
return long_count, critical, granularity
granularity *= 2
return None
def _format_suggestion(sm, prefer_critical: Optional[int]) -> Optional[str]:
"""
Return a '--split A,B' string the driver is known to accept, or None
if we couldn't find one.
"""
found = _find_working_split(sm, prefer_critical=prefer_critical)
if found is None:
return None
long_count, critical_count, _granularity = found
return f"--split {long_count},{critical_count}"
def parse_split(arg: Optional[str], device: Device) -> Tuple[int, int]:
"""
Parse the --split "A,B" CLI argument and validate it against the device.
Returns (long_count, critical_count).
"""
sm = device.resources.sm
total = sm.sm_count
min_part = sm.min_partition_size
if arg is None:
# Auto: reserve a small aligned slice for the critical kernel and
# hand the rest (also aligned) to the long-running kernel. We
# can't trust `min_partition_size` alone: on some GPUs (e.g.
# 188-SM Blackwell) the driver requires stricter alignment than
# it reports. Escalate the granularity until the driver accepts
# a pair.
prefer_critical = max(min_part, min(16, total // 8))
found = _find_working_split(sm, prefer_critical=prefer_critical)
if found is None:
print(
"Error: could not find an SM split that the driver accepts "
f"on this device (total SMs={total}, "
f"min_partition_size={min_part})."
)
print(
" The driver enforces architecture-specific alignment "
"rules beyond min_partition_size; try passing an explicit "
"--split."
)
sys.exit(1)
long_count, critical_count, granularity = found
if granularity > min_part:
print(
f"Note: driver required stricter alignment than "
f"min_partition_size={min_part}; selected split uses "
f"granularity={granularity} SMs."
)
return long_count, critical_count
# User-provided split.
try:
parts = [int(x.strip()) for x in arg.split(",")]
except ValueError:
print(f"Error: --split must look like 'A,B', got: {arg!r}")
sys.exit(1)
if len(parts) != 2:
print(
"Error: --split must contain exactly two comma-separated "
f"integers, got: {arg!r}"
)
sys.exit(1)
long_count, critical_count = parts
errors = []
if long_count <= 0 or critical_count <= 0:
errors.append("both partition sizes must be positive")
if long_count % min_part != 0 or critical_count % min_part != 0:
errors.append(f"each size must be a multiple of min_partition_size={min_part}")
if long_count + critical_count > total:
errors.append(
f"sum {long_count + critical_count} exceeds device total of {total} SMs"
)
if errors:
print("Error: invalid --split value:")
for e in errors:
print(f" - {e}")
suggestion = _format_suggestion(
sm, prefer_critical=critical_count if critical_count > 0 else None
)
if suggestion is not None:
print(f"Tip: a driver-accepted split on this device is {suggestion}")
sys.exit(1)
# Confirm the driver itself accepts the split. The well-known alignment
# checks above are necessary but not sufficient on every architecture.
try:
groups, _ = sm.split(
SMResourceOptions(count=(long_count, critical_count)),
dry_run=True,
)
except Exception as e:
print(f"Error: driver rejected the requested split: {e}")
print(
" The sample's own alignment checks are not exhaustive on "
"every architecture; the driver enforces additional hardware "
"constraints (for example TPC/partition-grid alignment)."
)
suggestion = _format_suggestion(sm, prefer_critical=critical_count)
if suggestion is not None:
print(f"Tip: a driver-accepted split on this device is {suggestion}")
sys.exit(1)
actual = tuple(g.sm_count for g in groups)
if actual != (long_count, critical_count):
print(f"Error: driver adjusted the requested split to {actual}.")
suggestion = _format_suggestion(sm, prefer_critical=critical_count)
if suggestion is not None:
print(f"Tip: a driver-accepted split on this device is {suggestion}")
else:
print(" Pick a different --split, or omit it for the auto default.")
sys.exit(1)
return long_count, critical_count
def compile_kernels(device: Device):
options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
program = Program(KERNEL_SRC, code_type="c++", options=options)
module = program.compile(
"cubin",
name_expressions=("delay_kernel", "critical_kernel"),
)
return module.get_kernel("delay_kernel"), module.get_kernel("critical_kernel")
def microseconds_to_cycles(device: Device, microseconds: float) -> int:
"""
Convert microseconds to SM clock cycles, using the reported GPU clock rate.
clock_rate is in kHz, so 1 us = clock_rate_kHz / 1000 cycles.
"""
clock_khz = device.properties.clock_rate
return int(microseconds * clock_khz / 1000.0)
def _run_one(
device: Device,
name: str,
long_stream,
critical_stream,
long_sm_count: int,
critical_sm_count: int,
delay_kernel,
critical_kernel,
delay_cycles: int,
delay_blocks: int,
critical_out_ptr: int,
critical_n: int,
critical_iters: int,
launch_gap_s: float,
) -> ScenarioResult:
"""
Launch the delay kernel on `long_stream`, wait `launch_gap_s` on the host,
launch the critical kernel on `critical_stream`, and time both with events.
"""
# Create events with timing enabled.
opts = EventOptions(timing_enabled=True)
e_long_start = device.create_event(opts)
e_long_end = device.create_event(opts)
e_crit_start = device.create_event(opts)
e_crit_end = device.create_event(opts)
# 1024 threads/block ensures at most one delay block is resident per SM
# on current architectures, so grid size directly controls the number of
# waves: delay_blocks / sm_count_visible_to_stream.
delay_block = 1024
delay_cfg = LaunchConfig(grid=delay_blocks, block=delay_block)
critical_block = 256
critical_grid = (critical_n + critical_block - 1) // critical_block
critical_cfg = LaunchConfig(grid=critical_grid, block=critical_block)
# Start of timed region
long_stream.record(e_long_start)
launch(long_stream, delay_cfg, delay_kernel, np.uint64(delay_cycles))
long_stream.record(e_long_end)
time.sleep(launch_gap_s)
critical_stream.record(e_crit_start)
launch(
critical_stream,
critical_cfg,
critical_kernel,
critical_out_ptr,
np.int32(critical_n),
np.int32(critical_iters),
)
critical_stream.record(e_crit_end)
# Sync both streams so every event has completed and is measurable.
long_stream.sync()
critical_stream.sync()
# End of timed region
return ScenarioResult(
name=name,
long_ms=e_long_end - e_long_start,
critical_total_ms=e_crit_end - e_crit_start,
critical_offset_ms=e_crit_start - e_long_start,
long_sm_count=long_sm_count,
critical_sm_count=critical_sm_count,
)
def run_critical_alone(
device: Device,
critical_kernel,
critical_n: int,
critical_iters: int,
) -> ScenarioResult:
"""
Critical kernel alone on the primary context, no competing work.
Establishes the pure compute time with every SM on the device available.
"""
stream = device.create_stream()
out = device.allocate(critical_n * 4)
total_sm = device.resources.sm.sm_count
try:
opts = EventOptions(timing_enabled=True)
e_start = device.create_event(opts)
e_end = device.create_event(opts)
block = 256
grid = (critical_n + block - 1) // block
cfg = LaunchConfig(grid=grid, block=block)
stream.record(e_start)
launch(
stream,
cfg,
critical_kernel,
int(out.handle),
np.int32(critical_n),
np.int32(critical_iters),
)
stream.record(e_end)
stream.sync()
return ScenarioResult(
name="crit alone (primary ctx)",
critical_total_ms=e_end - e_start,
critical_sm_count=total_sm,
)
finally:
out.close()
def run_baseline(
device: Device,
delay_kernel,
critical_kernel,
delay_cycles: int,
delay_blocks: int,
critical_n: int,
critical_iters: int,
launch_gap_s: float,
) -> ScenarioResult:
"""Both kernels on the primary context, two non-blocking streams."""
long_stream = device.create_stream()
critical_stream = device.create_stream()
out = device.allocate(critical_n * 4)
total_sm = device.resources.sm.sm_count
try:
return _run_one(
device,
name="baseline (primary ctx)",
long_stream=long_stream,
critical_stream=critical_stream,
long_sm_count=total_sm,
critical_sm_count=total_sm,
delay_kernel=delay_kernel,
critical_kernel=critical_kernel,
delay_cycles=delay_cycles,
delay_blocks=delay_blocks,
critical_out_ptr=int(out.handle),
critical_n=critical_n,
critical_iters=critical_iters,
launch_gap_s=launch_gap_s,
)
finally:
out.close()
def run_green_context(
device: Device,
split: Tuple[int, int],
delay_kernel,
critical_kernel,
delay_cycles: int,
delay_blocks: int,
critical_n: int,
critical_iters: int,
launch_gap_s: float,
) -> ScenarioResult:
"""Each kernel on its own green context, with disjoint SM partitions."""
long_count, critical_count = split
sm = device.resources.sm
groups, _remainder = sm.split(SMResourceOptions(count=(long_count, critical_count)))
assert len(groups) == 2
long_group, critical_group = groups
# Create the large ctx last so it's closed first: order matters only for
# ensuring we never try to close a ctx that's currently the thread's
# active ctx.
ctx_long = device.create_context(ContextOptions(resources=[long_group]))
ctx_crit = None
out = None
try:
ctx_crit = device.create_context(ContextOptions(resources=[critical_group]))
long_stream = ctx_long.create_stream()
critical_stream = ctx_crit.create_stream()
out = device.allocate(critical_n * 4)
return _run_one(
device,
name=f"green ctx ({long_count}+{critical_count} SMs)",
long_stream=long_stream,
critical_stream=critical_stream,
long_sm_count=ctx_long.resources.sm.sm_count,
critical_sm_count=ctx_crit.resources.sm.sm_count,
delay_kernel=delay_kernel,
critical_kernel=critical_kernel,
delay_cycles=delay_cycles,
delay_blocks=delay_blocks,
critical_out_ptr=int(out.handle),
critical_n=critical_n,
critical_iters=critical_iters,
launch_gap_s=launch_gap_s,
)
finally:
if out is not None:
out.close()
# Streams must be released before their owning ctx; letting them go out
# of scope here is sufficient since no references escape this frame.
if ctx_crit is not None:
ctx_crit.close()
ctx_long.close()
def _fmt_ms(value: Optional[float], width: int) -> str:
if value is None:
return f"{'-':>{width}}"
return f"{value:>{width}.3f}"
def print_results(results: List[ScenarioResult]) -> None:
print()
header = (
f"{'scenario':<32}{'SMs (long/crit)':>20}"
f"{'long (ms)':>14}{'crit total (ms)':>18}{'crit offset (ms)':>19}"
)
print(header)
print("-" * len(header))
for r in results:
long_sm = "-" if r.long_sm_count is None else str(r.long_sm_count)
sms = f"{long_sm}/{r.critical_sm_count}"
print(
f"{r.name:<32}{sms:>20}"
f"{_fmt_ms(r.long_ms, 14)}{_fmt_ms(r.critical_total_ms, 18)}"
f"{_fmt_ms(r.critical_offset_ms, 19)}"
)
print()
print("long (ms) : wall time of the delay kernel")
print("crit total (ms) : launch-to-complete wall time of the critical kernel")
print(
"crit offset (ms) : when the critical stream started, relative to the"
" long stream start"
)
def report_speedup(
alone: ScenarioResult,
baseline: ScenarioResult,
green: ScenarioResult,
) -> None:
"""
Print three headline numbers that put the raw scenario timings in context:
"""
if baseline.critical_total_ms <= 0 or alone.critical_total_ms <= 0:
return
latency_speedup = baseline.critical_total_ms / max(green.critical_total_ms, 1e-9)
compute_cost = green.critical_total_ms / alone.critical_total_ms
wait_ms = max(0.0, baseline.critical_total_ms - alone.critical_total_ms)
print()
print(
f"Critical-kernel latency speedup (baseline vs green ctx): "
f"{latency_speedup:.2f}x"
)
print(
f"Green-ctx compute cost vs unconstrained (crit alone): {compute_cost:.2f}x"
)
print(f"Baseline time spent waiting for SMs (not computing): ~{wait_ms:.2f} ms")
def main():
parser = argparse.ArgumentParser(
description="Green Context sample using CUDA Core API",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--device", type=int, default=0, help="CUDA device ID (default: 0)"
)
parser.add_argument(
"--split",
type=str,
default=None,
help="SM split as 'LONG,CRITICAL', e.g. '112,16'. Default: auto.",
)
parser.add_argument(
"--delay-us",
type=int,
default=2000,
help=(
"Per-block busy-wait duration of the delay kernel, "
"in microseconds (default: 2000)"
),
)
parser.add_argument(
"--delay-waves",
type=int,
default=16,
help=(
"Number of waves of the delay kernel on the long partition. "
"Drives the default --delay-blocks (default: 16)."
),
)
parser.add_argument(
"--delay-blocks",
type=int,
default=None,
help=(
"Number of blocks launched for the delay kernel. "
"Overrides --delay-waves if set. "
"Default: --delay-waves * device SM count."
),
)
parser.add_argument(
"--critical-n",
type=int,
default=1 << 22,
help="Work size of the critical kernel (default: 4194304)",
)
parser.add_argument(
"--critical-iters",
type=int,
default=1024,
help=(
"Iterations of the inner math loop inside the critical kernel. "
"Higher values make the critical kernel's compute time more "
"substantial (default: 1024)."
),
)
parser.add_argument(
"--launch-gap-ms",
type=float,
default=1.0,
help=(
"Host delay between launching the long and critical kernels, "
"in ms (default: 1.0)"
),
)
args = parser.parse_args()
try:
device = Device(args.device)
device.set_current()
except Exception as e:
print(f"Error: failed to initialize CUDA device {args.device}: {e}")
return 1
print_sm_topology(device)
long_count, critical_count = parse_split(args.split, device)
print(f"SM split (long/critical): {long_count} / {critical_count}")
sm_count = device.resources.sm.sm_count
delay_blocks = args.delay_blocks or args.delay_waves * sm_count
delay_cycles = microseconds_to_cycles(device, args.delay_us)
launch_gap_s = max(0.0, args.launch_gap_ms / 1000.0)
# Rough estimate of the long kernel's duration on the full device. Mostly
# informational; the real value is reported after the run.
est_long_ms = (delay_blocks / sm_count) * (args.delay_us / 1000.0)
print("Workload parameters:")
print(
f" delay kernel: {delay_blocks} blocks, {args.delay_us} us/block "
f"(~{est_long_ms:.1f} ms on {sm_count} SMs)"
)
print(
f" critical kernel: {args.critical_n} elements, "
f"{args.critical_iters} inner iterations"
)
print(f" host launch gap: {args.launch_gap_ms} ms")
print()
print("Compiling kernels ...")
delay_k, crit_k = compile_kernels(device)
try:
print("Running reference scenario (critical kernel alone) ...")
alone = run_critical_alone(
device,
crit_k,
args.critical_n,
args.critical_iters,
)
print("Running baseline scenario (primary context) ...")
baseline = run_baseline(
device,
delay_k,
crit_k,
delay_cycles,
delay_blocks,
args.critical_n,
args.critical_iters,
launch_gap_s,
)
print("Running green context scenario ...")
green = run_green_context(
device,
(long_count, critical_count),
delay_k,
crit_k,
delay_cycles,
delay_blocks,
args.critical_n,
args.critical_iters,
launch_gap_s,
)
except Exception as e:
print(f"Error: scenario failed: {e}")
import traceback
traceback.print_exc()
return 1
print_results([alone, baseline, green])
report_speedup(alone, baseline, green)
print("\nDone")
return 0
if __name__ == "__main__":
sys.exit(main())

3
python/2_CoreConcepts/greenContext/requirements.txt Normal file
View File

@@ -0,0 +1,3 @@
cuda-python>=13.0.0
cuda-core>=0.7.0
numpy>=2.3.2

133
python/2_CoreConcepts/jitLtoLinking/README.md Normal file
View File

@@ -0,0 +1,133 @@
# JIT Compilation and Link-Time Optimization (Python)
## Description
This sample demonstrates how to build a kernel out of two independently
compiled translation units and link them at runtime with
`cuda.core.Linker`. This is the pattern a library would use to accept
user-supplied device code as a plug-in without recompiling its own
kernels from scratch.
The sample runs the same program in two linking modes:
1. **PTX linking** - each module is compiled with
`ProgramOptions(relocatable_device_code=True)` down to PTX, and the
`Linker` emits a final cubin. The two modules stay independently
compiled (no cross-module inlining).
2. **Link-Time Optimization (LTO)** - each module is compiled with
`ProgramOptions(link_time_optimization=True)` down to LTO IR, and the
`Linker` is configured with `LinkerOptions(link_time_optimization=True)`
so the optimizer runs again across both modules, typically matching
the code generation of a single-source build.
The "main" kernel `apply_transform` calls a `user_transform` device
function that lives in a separate source string, and the results of both
linking modes are verified against a NumPy reference.
## What You'll Learn
- Compiling multiple `Program` objects into PTX or LTO IR
- Linking independent object codes into a single cubin with `Linker`
- Choosing between `relocatable_device_code` and `link_time_optimization`
- How a library's main kernel can call into user-supplied device code
- When to prefer LTO over plain PTX linking
## Key Libraries
- [`cuda.core`](https://nvidia.github.io/cuda-python/cuda-core/latest/) - Pythonic access to CUDA runtime, programs, and the JIT linker
- `cupy` - input and output buffers on the GPU
- `numpy` - reference computation on the host
## Key APIs
### From `cuda.core`
- `ProgramOptions(relocatable_device_code=True)` + `Program.compile("ptx")` - produce relocatable PTX
- `ProgramOptions(link_time_optimization=True)` + `Program.compile("ltoir")` - produce LTO IR
- `Linker(*object_codes, options=LinkerOptions(...))` - create a JIT linker over multiple object codes
- `LinkerOptions(link_time_optimization=True)` - opt into LTO during linking
- `Linker.link("cubin")` - produce a loadable module
- `ObjectCode.get_kernel(name)` - fetch a kernel from the linked module
### From `cuda_samples_utils`
- `print_gpu_info()` - print device name and compute capability
## Requirements
### Hardware
- NVIDIA GPU with Compute Capability 7.0 or higher
### Software
- CUDA Toolkit 13.0 or newer (matches `cuda-python` 13.x)
- Python 3.10 or newer
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=0.6.0)
- `cupy-cuda13x` (>=13.0.0)
## Installation
Install the required packages from `requirements.txt`:
```bash
cd /path/to/cuda-samples/python/2_CoreConcepts/jitLtoLinking
pip install -r requirements.txt
```
The `requirements.txt` installs:
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=0.6.0)
- `cupy-cuda13x` (>=13.0.0)
## How to Run
### Basic usage
```bash
cd cuda-samples/python/2_CoreConcepts/jitLtoLinking
python jitLtoLinking.py
```
### With custom parameters
```bash
# Larger element count
python jitLtoLinking.py --elements 1048576
# Use a specific GPU
python jitLtoLinking.py --device 1
```
## Expected Output
```
Device: <Your GPU Name>
Compute Capability: <X.Y>
[1] PTX linking (no LTO)
[ptx] result verified against NumPy reference
[2] LTO linking (link-time optimization)
[lto] result verified against NumPy reference
Both PTX and LTO linked kernels produced matching results. Done
```
**Note:** Device name and compute capability will vary based on your GPU.
## Files
- `jitLtoLinking.py` - Python implementation using `cuda.core.Linker`
- `README.md` - This file
- `requirements.txt` - Sample dependencies
- `../../Utilities/cuda_samples_utils.py` - Common utilities (imported by this sample)
## See Also
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [`cuda.core` compilation API](https://nvidia.github.io/cuda-python/cuda-core/latest/api.html#cuda-compilation-toolchain)
- Upstream `cuda.core` example: [`jit_lto_fractal.py`](https://github.com/NVIDIA/cuda-python/blob/main/cuda_core/examples/jit_lto_fractal.py)
- [NVIDIA nvJitLink documentation](https://docs.nvidia.com/cuda/nvjitlink/index.html)

223
python/2_CoreConcepts/jitLtoLinking/jitLtoLinking.py Normal file
View File

@@ -0,0 +1,223 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
JIT Compilation and Link-Time Optimization with cuda.core
Real-world GPU code is rarely a single source string. Libraries ship a
"main" kernel that is compiled once, then link in user-supplied device
functions at runtime to customize behavior without recompiling the whole
program.
cuda.core exposes this pattern through ``Program`` (NVRTC compilation)
and ``Linker`` (JIT linking of multiple object codes). Two modes are
shown here:
* **PTX linking**: compile each translation unit with
``relocatable_device_code=True`` to PTX and link to a CUBIN.
The two modules remain independently compiled: no cross-module
inlining.
* **LTO (Link-Time Optimization)**: compile each translation unit
with ``link_time_optimization=True`` to LTO IR, then link with
``LinkerOptions(link_time_optimization=True)``. The linker reruns
the optimizer across both modules and can inline the device function
into the main kernel, typically matching a single-source build.
The same kernel math runs in both modes and is verified against a
NumPy reference.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
try:
import cupy as cp
import numpy as np
from cuda.core import (
Device,
LaunchConfig,
Linker,
LinkerOptions,
Program,
ProgramOptions,
launch,
)
from cuda_samples_utils import print_gpu_info # noqa: E402
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install from requirements.txt:")
print(" pip install -r requirements.txt")
sys.exit(1)
# --------------------------------------------------------------------------
# Module A: the "library" main kernel. It forwards each element through a
# user-supplied device function (resolved at link time) and writes the result.
# --------------------------------------------------------------------------
MAIN_SRC = r"""
// Forward declare the user-supplied hook. Its definition lives in a separate
// translation unit and is resolved by the Linker at runtime.
extern "C" __device__ float user_transform(float x);
extern "C" __global__
void apply_transform(const float* __restrict__ in,
float* __restrict__ out,
size_t N)
{
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
size_t stride = (size_t)gridDim.x * blockDim.x;
for (size_t i = tid; i < N; i += stride) {
out[i] = user_transform(in[i]);
}
}
"""
# --------------------------------------------------------------------------
# Module B: the user-supplied "plug-in" device function. A different
# implementation of ``user_transform`` here produces different results without
# rebuilding MAIN_SRC.
# --------------------------------------------------------------------------
USER_SRC = r"""
extern "C" __device__
float user_transform(float x)
{
// A deliberately non-trivial expression so LTO has something to inline /
// optimize across the module boundary.
float y = x * x + 3.0f * x - 1.0f;
return y > 0.0f ? y : 0.0f;
}
"""
def host_reference(x: np.ndarray) -> np.ndarray:
y = x * x + 3.0 * x - 1.0
return np.where(y > 0.0, y, 0.0).astype(np.float32)
def link_ptx(device):
"""Compile both modules to PTX and link them into a cubin (no LTO)."""
prog_opts = ProgramOptions(
std="c++17", arch=f"sm_{device.arch}", relocatable_device_code=True
)
main_obj = Program(MAIN_SRC, "c++", options=prog_opts).compile("ptx")
user_obj = Program(USER_SRC, "c++", options=prog_opts).compile("ptx")
linker = Linker(main_obj, user_obj, options=LinkerOptions(arch=f"sm_{device.arch}"))
return linker.link("cubin")
def link_lto(device):
"""Compile both modules to LTO IR and link with LTO enabled."""
prog_opts = ProgramOptions(
std="c++17", arch=f"sm_{device.arch}", link_time_optimization=True
)
main_obj = Program(MAIN_SRC, "c++", options=prog_opts).compile("ltoir")
user_obj = Program(USER_SRC, "c++", options=prog_opts).compile("ltoir")
linker_opts = LinkerOptions(
arch=f"sm_{device.arch}", link_time_optimization=True
)
linker = Linker(main_obj, user_obj, options=linker_opts)
return linker.link("cubin")
def run_one_mode(mode, module, stream, d_in, d_out, size, expected):
kernel = module.get_kernel("apply_transform")
config = LaunchConfig(grid=(size + 255) // 256, block=256)
launch(
stream,
config,
kernel,
d_in.data.ptr,
d_out.data.ptr,
np.uint64(size),
)
stream.sync()
actual = cp.asnumpy(d_out)
if not np.allclose(actual, expected, rtol=1e-5, atol=1e-5):
max_err = np.max(np.abs(actual - expected))
print(f" [{mode}] verification FAILED (max_err={max_err})")
return False
print(f" [{mode}] result verified against NumPy reference")
return True
def main() -> int:
import argparse
parser = argparse.ArgumentParser(
description="JIT + LTO linking of two device modules with cuda.core"
)
parser.add_argument(
"--elements", type=int, default=1 << 16,
help="Number of float32 elements (default: 65536)",
)
parser.add_argument("--device", type=int, default=0, help="CUDA device id")
args = parser.parse_args()
device = Device(args.device)
device.set_current()
print_gpu_info(device)
stream = device.create_stream()
cp.cuda.ExternalStream(int(stream.handle)).use()
try:
N = args.elements
rng = np.random.default_rng(seed=0)
host_in = rng.standard_normal(N).astype(np.float32)
expected = host_reference(host_in)
d_in = cp.asarray(host_in)
d_out = cp.empty(N, dtype=cp.float32)
device.sync()
print("\n[1] PTX linking (no LTO)")
ptx_module = link_ptx(device)
ok_ptx = run_one_mode("ptx", ptx_module, stream, d_in, d_out, N, expected)
d_out.fill(0)
device.sync()
print("\n[2] LTO linking (link-time optimization)")
lto_module = link_lto(device)
ok_lto = run_one_mode("lto", lto_module, stream, d_in, d_out, N, expected)
print()
if ok_ptx and ok_lto:
print("Both PTX and LTO linked kernels produced matching results. Done")
return 0
return 1
finally:
stream.close()
cp.cuda.Stream.null.use()
if __name__ == "__main__":
sys.exit(main())

5
python/2_CoreConcepts/jitLtoLinking/requirements.txt Normal file
View File

@@ -0,0 +1,5 @@
# JIT + LTO Linking Sample Requirements
cuda-python>=13.0.0
cuda-core>=0.6.0
cupy-cuda13x>=13.0.0

184
python/2_CoreConcepts/launchConfigTuning/README.md Normal file
View File

@@ -0,0 +1,184 @@
# Sample: Launch Configuration Tuning (Python)
## Description
Benchmark different CUDA kernel launch configurations to find the optimal block-size setting using `cuda.core` APIs. This sample demonstrates **performance tuning** by measuring execution time across various thread block sizes.
## What You'll Learn
- Compiling CUDA kernels at runtime with `cuda.core.Program`
- Launching kernels with different `LaunchConfig` settings
- Benchmarking kernel performance with precise timing
- Understanding how thread block size affects performance
- Tuning for memory-bound vs compute-bound kernels
## Key Concepts
### Launch Configuration with cuda.core
```python
# Configure kernel launch with specific thread block size
config = LaunchConfig(
grid=(grid_size,),
block=(block_size,),
shmem_size=shared_memory_bytes
)
# Launch kernel
launch(stream, config, kernel, *args)
stream.sync()
```
### Thread Block Sizing
Thread block size significantly impacts performance due to:
| Factor | Impact |
|--------|--------|
| **Occupancy** | More active warps can hide memory latency |
| **Registers** | More threads/block = fewer registers/thread |
| **Shared Memory** | Divided among blocks on each SM |
| **Warp Efficiency** | Block size should be multiple of 32 |
### Benchmarking Approach
```python
# Use CUDA events for accurate GPU timing (not CPU wall-clock)
start_event = device.create_event(options=EventOptions(enable_timing=True))
end_event = device.create_event(options=EventOptions(enable_timing=True))
stream.record(start_event)
for _ in range(n_iterations):
launch(stream, config, kernel, *args)
stream.record(end_event)
end_event.sync()
elapsed_ms = (end_event - start_event) / n_iterations
```
## Key APIs
### From `cuda.core`:
- `Device` - CUDA device management
- `Program` - Runtime kernel compilation (NVRTC)
- `ProgramOptions` - Compilation options (architecture target)
- `LaunchConfig` - Kernel launch configuration (grid/block dimensions)
- `launch` - Execute compiled kernel (accepts Buffer objects directly)
- `EventOptions` - GPU timing with CUDA events
- `ManagedMemoryResource` - Device-preferred unified memory
- `ManagedMemoryResourceOptions` - Set preferred_location for representative benchmarks
### From `numpy`:
- `np.from_dlpack()` - Zero-copy view of GPU buffers via DLPack
### Benchmarked Kernels:
- **vector_add** - Simple memory-bound kernel (C = A + B) - low sensitivity to block size
- **reduce_sum** - Shared memory reduction - high sensitivity to block size
## Requirements
### Hardware:
- NVIDIA GPU with CUDA support
- Minimum GPU memory: 512 MB
### Software:
- CUDA Toolkit 13.0 or newer
- Python 3.10 or newer
- See `requirements.txt` for Python packages
## Installation
```bash
pip install -r requirements.txt
```
## How to Run
```bash
python launchConfigTuning.py
```
## Expected Output
```
============================================================
Launch Configuration Tuning (cuda.core)
Finding the Best Block Size for Your Kernel
============================================================
Device: <Your GPU Name>
Compute Capability: X.X
Compiling CUDA kernels with cuda.core.Program...
Target architecture: sm_XX
✓ vector_add kernel compiled
✓ reduce_sum kernel compiled
============================================================
VECTOR ADDITION - Launch Configuration Tuning
============================================================
Problem size: 10,000,000 elements
Kernel: vector_add (C = A + B)
Testing thread configurations: [32, 64, 128, 256, 512, 1024]
------------------------------------------------------------
Block Size: 32 | Blocks: 312500 | Time: X.XXXX ± X.XXXX ms
Block Size: 64 | Blocks: 156250 | Time: X.XXXX ± X.XXXX ms
...
------------------------------------------------------------
✓ OPTIMAL: block_size=XXX (X.XXXX ms)
✗ WORST: block_size=XXX (X.XXXX ms)
Speedup: X.XXx
✓ Results verified correct!
...
============================================================
SAMPLE COMPLETE
============================================================
Key Takeaway: The optimal thread configuration depends on your
specific kernel characteristics. Always benchmark to find the best!
```
## Tuning Guidelines
### Start Here
- **128-256 threads/block** is a good starting point for most kernels
- Always use **multiples of 32** (warp size)
### Memory-Bound Kernels
- Less sensitive to thread configuration
- Focus on memory access patterns
- Higher thread counts help hide latency
### Compute-Bound Kernels
- More sensitive to thread configuration
- Watch for register pressure at high thread counts
- Profile with Nsight Compute
### Reduction Kernels
- Block size affects shared memory usage
- Power-of-2 sizes simplify reduction logic
- Often 256-512 threads works well
## Files
- `launchConfigTuning.py` - Python implementation using cuda.core
- `README.md` - This file
- `requirements.txt` - Sample dependencies
## See Also
- [cuda.core Documentation](https://nvidia.github.io/cuda-python/cuda-core/latest/)
- [cuda.core.LaunchConfig](https://nvidia.github.io/cuda-python/cuda-core/latest/generated/cuda.core.LaunchConfig.html)
- [CUDA Occupancy Calculator](https://docs.nvidia.com/cuda/cuda-occupancy-calculator/)
- [CUDA Best Practices Guide - Execution Configuration](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#execution-configuration-optimizations)
- [Nsight Compute Profiler](https://developer.nvidia.com/nsight-compute)

388
python/2_CoreConcepts/launchConfigTuning/launchConfigTuning.py Normal file
View File

@@ -0,0 +1,388 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Launch Configuration Tuning
Demonstrates how to find the optimal threads-per-block configuration for CUDA
kernels using cuda.core APIs. Benchmarks different thread layouts to answer:
"What is the best threads-per-block for my kernel?"
"""
import sys
try:
import numpy as np
from cuda.core import (
Device,
EventOptions,
LaunchConfig,
ManagedMemoryResource,
ManagedMemoryResourceOptions,
Program,
ProgramOptions,
launch,
)
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install from requirements.txt:")
print(" pip install -r requirements.txt")
sys.exit(1)
# =============================================================================
# CUDA Kernel Source Code
# =============================================================================
# Vector Addition Kernel - Simple memory-bound kernel (grid-stride loop)
VECTOR_ADD_KERNEL = r"""
extern "C" __global__
void vector_add(const float* __restrict__ a,
const float* __restrict__ b,
float* __restrict__ c,
int n) {
int idx = blockIdx.x * blockDim.x + threadIdx.x;
int stride = blockDim.x * gridDim.x;
for (int i = idx; i < n; i += stride) {
c[i] = a[i] + b[i];
}
}
"""
# Reduction Kernel - Sensitive to block size due to shared memory (grid-stride load)
REDUCTION_KERNEL = r"""
extern "C" __global__
void reduce_sum(const float* __restrict__ input,
float* __restrict__ partial_sums,
int n) {
extern __shared__ float sdata[];
unsigned int tid = threadIdx.x;
unsigned int stride = blockDim.x * gridDim.x;
// Load data into shared memory (grid-stride loop)
float sum = 0.0f;
for (unsigned int i = blockIdx.x * blockDim.x + tid; i < n; i += stride) {
sum += input[i];
}
sdata[tid] = sum;
__syncthreads();
// Perform reduction in shared memory
for (unsigned int s = blockDim.x / 2; s > 0; s >>= 1) {
if (tid < s) {
sdata[tid] += sdata[tid + s];
}
__syncthreads();
}
// Write result for this block
if (tid == 0) {
partial_sums[blockIdx.x] = sdata[0];
}
}
"""
# =============================================================================
# Utility Functions
# =============================================================================
def compile_kernel(device, kernel_code, kernel_name):
"""Compile a CUDA kernel using cuda.core.Program."""
arch = f"sm_{device.arch}"
options = ProgramOptions(arch=arch)
program = Program(kernel_code, code_type="c++", options=options)
compiled = program.compile(target_type="cubin")
return compiled.get_kernel(kernel_name)
def benchmark_kernel_1d(
device,
stream,
kernel,
args,
n_elements,
block_size,
n_iterations=100,
shared_mem_bytes=0,
):
"""
Benchmark a 1D kernel with given threads-per-block configuration.
Uses CUDA events for accurate GPU timing.
Returns timing statistics as a dictionary.
"""
grid_size = (n_elements + block_size - 1) // block_size
config = LaunchConfig(
grid=(grid_size,), block=(block_size,), shmem_size=shared_mem_bytes
)
# Warm-up run
launch(stream, config, kernel, *args)
stream.sync()
# Timed runs with CUDA events
event_opts = EventOptions(enable_timing=True)
start_event = device.create_event(options=event_opts)
end_event = device.create_event(options=event_opts)
stream.record(start_event)
for _ in range(n_iterations):
launch(stream, config, kernel, *args)
stream.record(end_event)
end_event.sync()
elapsed_ms = (end_event - start_event) / n_iterations
return {
"block_size": block_size,
"grid_size": grid_size,
"mean_time_ms": elapsed_ms,
"std_time_ms": 0.0, # Single measurement with events
}
def print_gpu_info(device):
"""Print GPU information relevant to launch configuration."""
print(f"\nDevice: {device.name}")
cc = device.compute_capability
print(f"Compute Capability: {cc.major}.{cc.minor}")
def allocate_managed_array(mr, stream, n_elements, dtype=np.float32):
"""Allocate device-preferred unified memory and return buffer with numpy view."""
n_bytes = n_elements * np.dtype(dtype).itemsize
buffer = mr.allocate(n_bytes, stream)
stream.sync()
# Zero-copy numpy view via DLPack (holds reference to buffer)
np_view = np.from_dlpack(buffer).view(dtype).reshape(n_elements)
return buffer, np_view
# =============================================================================
# Benchmark Demonstrations
# =============================================================================
def demo_vector_add_tuning(device, stream, mr, kernel):
"""Demonstrate launch configuration tuning for vector addition."""
print("\n" + "=" * 60)
print("VECTOR ADDITION - Launch Configuration Tuning")
print("=" * 60)
N = 10_000_000 # 10 million elements
print(f"\nProblem size: {N:,} elements")
print("Kernel: vector_add (C = A + B)")
# Allocate device-preferred unified memory via cuda.core
d_a, np_a = allocate_managed_array(mr, stream, N)
d_b, np_b = allocate_managed_array(mr, stream, N)
d_c, np_c = allocate_managed_array(mr, stream, N)
try:
# Initialize data via numpy views
np_a[:] = np.random.rand(N).astype(np.float32)
np_b[:] = np.random.rand(N).astype(np.float32)
stream.sync()
# Thread configurations to test (multiples of warp size = 32)
thread_configs = [32, 64, 128, 256, 512, 1024]
print(f"\nTesting thread configurations: {thread_configs}")
print("-" * 60)
results = []
for tpb in thread_configs:
result = benchmark_kernel_1d(
device,
stream,
kernel,
(d_a, d_b, d_c, np.int32(N)),
N,
tpb,
n_iterations=100,
)
results.append(result)
print(
f"Block Size: {tpb:4d} | Blocks: {result['grid_size']:6d} | "
f"Time: {result['mean_time_ms']:.4f} ms"
)
# Find optimal and worst configurations
best = min(results, key=lambda x: x["mean_time_ms"])
worst = max(results, key=lambda x: x["mean_time_ms"])
print("-" * 60)
print(
f"\n✓ OPTIMAL: block_size={best['block_size']} "
f"({best['mean_time_ms']:.4f} ms)"
)
print(
f"✗ WORST: block_size={worst['block_size']} "
f"({worst['mean_time_ms']:.4f} ms)"
)
print(f" Speedup: {worst['mean_time_ms']/best['mean_time_ms']:.2f}x")
# Verify result
stream.sync()
expected = np_a + np_b
if np.allclose(np_c, expected):
print("\n✓ Results verified correct!")
return results
finally:
d_a.close()
d_b.close()
d_c.close()
def demo_reduction_tuning(device, stream, mr, kernel):
"""Demonstrate launch config tuning for reduction (shared memory)."""
print("\n" + "=" * 60)
print("REDUCTION - Launch Configuration Tuning")
print("=" * 60)
N = 16_777_216 # 16M elements (power of 2)
print(f"\nProblem size: {N:,} elements")
print("Kernel: reduce_sum (parallel reduction)")
print("Note: Reduction uses shared memory - more sensitive to block size!")
# Allocate device-preferred unified memory via cuda.core
d_input, np_input = allocate_managed_array(mr, stream, N)
try:
np_input[:] = np.random.rand(N).astype(np.float32)
stream.sync()
thread_configs = [32, 64, 128, 256, 512, 1024]
print(f"\nTesting thread configurations: {thread_configs}")
print("-" * 60)
results = []
for tpb in thread_configs:
# Allocate partial sums array
n_blocks = (N + tpb - 1) // tpb
d_partial, _ = allocate_managed_array(mr, stream, n_blocks)
try:
# Shared memory size = block_size * sizeof(float)
shared_mem_bytes = tpb * 4
result = benchmark_kernel_1d(
device,
stream,
kernel,
(d_input, d_partial, np.int32(N)),
N,
tpb,
n_iterations=50,
shared_mem_bytes=shared_mem_bytes,
)
results.append(result)
print(
f"Block Size: {tpb:4d} | Blocks: {result['grid_size']:6d} | "
f"Time: {result['mean_time_ms']:.4f} ms"
)
finally:
d_partial.close()
best = min(results, key=lambda x: x["mean_time_ms"])
worst = max(results, key=lambda x: x["mean_time_ms"])
print("-" * 60)
print(f"\n✓ OPTIMAL: block_size={best['block_size']}")
print(
f" Speedup over worst: {worst['mean_time_ms']/best['mean_time_ms']:.2f}x"
)
return results
finally:
d_input.close()
# =============================================================================
# Main
# =============================================================================
def main():
"""
Complete demonstration of CUDA launch configuration tuning.
This sample shows:
1. Device initialization with cuda.core.Device
2. Kernel compilation with cuda.core.Program
3. Benchmarking different thread block configurations
4. Finding optimal threads-per-block for various kernel types
"""
print("=" * 60)
print("Launch Configuration Tuning (cuda.core)")
print("Finding the Best Block Size for Your Kernel")
print("=" * 60)
# Initialize CUDA device
device = Device(0)
device.set_current()
# Print GPU information
print_gpu_info(device)
# Create stream and device-preferred memory resource
stream = device.create_stream()
mr_options = ManagedMemoryResourceOptions(preferred_location=device.device_id)
mr = ManagedMemoryResource(mr_options)
try:
# Compile kernels
print("\nCompiling CUDA kernels with cuda.core.Program...")
arch = f"sm_{device.arch}"
print(f" Target architecture: {arch}")
vec_add_kernel = compile_kernel(device, VECTOR_ADD_KERNEL, "vector_add")
print(" ✓ vector_add kernel compiled")
reduction_kernel = compile_kernel(device, REDUCTION_KERNEL, "reduce_sum")
print(" ✓ reduce_sum kernel compiled")
# Run demonstrations
demo_vector_add_tuning(device, stream, mr, vec_add_kernel)
demo_reduction_tuning(device, stream, mr, reduction_kernel)
print("\n" + "=" * 60)
print("SAMPLE COMPLETE")
print("=" * 60)
print("\nKey Takeaway: The optimal thread configuration depends on your")
print("specific kernel characteristics. Always benchmark to find the best!")
print()
finally:
stream.close()
if __name__ == "__main__":
main()

6
python/2_CoreConcepts/launchConfigTuning/requirements.txt Normal file
View File

@@ -0,0 +1,6 @@
# Launch Configuration Tuning Sample Requirements
# Requires Python 3.10+, CUDA Toolkit 13.0+
cuda-python>=13.0.0
cuda-core>=0.6.0
numpy>=2.3.2

163
python/2_CoreConcepts/matrixMulSharedMem/README.md Normal file
View File

@@ -0,0 +1,163 @@
# Matrix Multiplication with Shared Memory (GEMM)
Demonstrates efficient matrix multiplication using nvmath-python APIs and custom CUDA kernels with tiling, shared memory, and loop unrolling.
## Overview
- Uses nvmath.linalg.advanced.Matmul for high-performance GEMM via cuBLASLt
- Compares with custom CUDA kernel using tiling and shared memory
- Shows how tiling reduces global memory bandwidth requirements
- Demonstrates shared memory for data reuse within thread blocks
- Uses loop unrolling to improve instruction-level parallelism
## What You'll Learn
- How to use nvmath stateful API for optimized matrix multiplication
- How to tile matrix operations for better cache locality
- Using shared memory to reduce redundant global memory accesses
- Loop unrolling techniques for GPU kernels
- Benchmarking and comparing kernel performance
## Key Libraries
- `nvmath-python` - NVIDIA math library with cuBLASLt access
- `cuda.core` - Modern CUDA Python API for custom kernel compilation
- `cupy` - GPU array library for Python
## Key APIs
### From `nvmath.linalg.advanced`:
- `Matmul()` - Stateful matrix multiplication with planning and execution phases
- `MatmulComputeType` - Compute type options for mixed-precision
### From `cuda.core`:
- `Device()` - CUDA device management and properties
- `Program()` - Runtime kernel compilation (NVRTC)
- `LaunchConfig()` - Kernel launch configuration (grid/block dimensions)
- `launch()` - Kernel execution on a stream
- `Stream.record_event()` / `Event.elapsed_time()` - GPU timing
## Requirements
### Hardware:
- NVIDIA GPU with Compute Capability 7.0 or higher
- Minimum GPU memory: 256 MB (for 1024×1024 matrices)
### Software:
- CUDA Toolkit 13.0 or newer
- Python 3.10 or newer
- See requirements.txt for package dependencies
## Installation
```bash
cd cuda-samples/python/2_CoreConcepts/matrixMulSharedMem
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```
## How to Run
```bash
python matrixMulSharedMem.py
```
## Expected Output
```
======================================================================
Matrix Multiplication with Shared Memory (GEMM)
Using nvmath and cuda.core APIs
======================================================================
Device: NVIDIA GeForce RTX 4090
Compute Capability: sm_89
Custom kernel compiled ✓
Matrix dimensions: A(1024x1024) × B(1024x1024) = C(1024x1024)
Custom kernel tile size: 16x16
----------------------------------------------------------------------
NVMATH MATMUL (cuBLASLt)
----------------------------------------------------------------------
Using nvmath.linalg.advanced.Matmul stateful API
Average time: X.XXX ms
Performance: XXXX.XX GFLOPS
----------------------------------------------------------------------
CUSTOM KERNEL (Tiled + Shared Memory + Loop Unrolling)
----------------------------------------------------------------------
Grid: (64, 64), Block: (16, 16)
Average time: X.XXX ms
Performance: XXX.XX GFLOPS
----------------------------------------------------------------------
VERIFICATION
----------------------------------------------------------------------
nvmath : PASSED (max error: X.XXe-XX)
Custom kernel : PASSED (max error: X.XXe-XX)
======================================================================
PERFORMANCE SUMMARY
======================================================================
Implementation Time (ms) GFLOPS
----------------------------------------------------------------------
nvmath (cuBLASLt) X.XXX XXXX.XX
Custom (shared mem + unroll) X.XXX XXX.XX
```
## Tiling Concept
```
Matrix A (M×K) Matrix B (K×N) Matrix C (M×N)
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ T00 │ T01 │...│ │ T00 │ T01 │...│ │ │ │ │
├─────┼─────┼───┤ ├─────┼─────┼───┤ ├─────┼─────┼───┤
│ T10 │ T11 │...│ × │ T10 │ T11 │...│ = │ │ Cij │ │
├─────┼─────┼───┤ ├─────┼─────┼───┤ ├─────┼─────┼───┤
│ ... │ ... │...│ │ ... │ ... │...│ │ │ │ │
└───────────────┘ └───────────────┘ └───────────────┘
Cij = Σ (A_tile_row × B_tile_col) for all tiles along K
```
## nvmath Stateful API
```python
import nvmath.linalg.advanced as nvmath_advanced
# Create matrices (CuPy arrays)
A = cp.random.rand(m, k).astype(cp.float32)
B = cp.random.rand(k, n).astype(cp.float32)
# Use stateful API for fine-grained control
with nvmath_advanced.Matmul(A, B) as mm:
mm.plan() # Find optimal algorithm
C = mm.execute() # Execute computation
```
## Memory Access Optimization (Custom Kernel)
| Implementation | Global Reads per C element | Reduction |
|---------------|---------------------------|-----------|
| Naive | 2 × K | (baseline)|
| Tiled (16×16) | 2 × K / 16 | 16× |
## Files
- `matrixMulSharedMem.py` - Python implementation comparing nvmath vs custom kernel
- `README.md` - This file
- `requirements.txt` - Sample dependencies
## See Also
- [nvmath-python Documentation](https://docs.nvidia.com/cuda/nvmath-python/)
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [cuda.core API Guide](https://nvidia.github.io/cuda-python/cuda-core/latest/)
- [CuPy Documentation](https://docs.cupy.dev/)

243
python/2_CoreConcepts/matrixMulSharedMem/matrixMulSharedMem.py Normal file
View File

@@ -0,0 +1,243 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Matrix Multiplication with Shared Memory (GEMM)
Demonstrates efficient matrix multiplication using:
- nvmath.linalg.advanced.Matmul for high-performance GEMM via cuBLASLt
- Custom CUDA kernel with tiling, shared memory, and loop unrolling
Uses cuda.core APIs with CuPy arrays via ExternalStream.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
try:
import cupy as cp
import numpy as np
import nvmath.linalg.advanced as nvmath_advanced
from cuda.core import (
Device,
EventOptions,
LaunchConfig,
Program,
ProgramOptions,
launch,
)
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Install with: pip install -r requirements.txt")
sys.exit(1)
TILE_SIZE: int = 16
MATMUL_KERNEL: str = r"""
#define TILE_SIZE 16
extern "C" __global__
void matmul_shared(const float* A, const float* B, float* C,
int M, int N, int K) {
__shared__ float As[TILE_SIZE][TILE_SIZE];
__shared__ float Bs[TILE_SIZE][TILE_SIZE];
int bx = blockIdx.x, by = blockIdx.y;
int tx = threadIdx.x, ty = threadIdx.y;
int row = by * TILE_SIZE + ty;
int col = bx * TILE_SIZE + tx;
float sum = 0.0f;
int numTiles = (K + TILE_SIZE - 1) / TILE_SIZE;
for (int t = 0; t < numTiles; t++) {
int aCol = t * TILE_SIZE + tx;
int bRow = t * TILE_SIZE + ty;
As[ty][tx] = (row < M && aCol < K) ? A[row * K + aCol] : 0.0f;
Bs[ty][tx] = (bRow < K && col < N) ? B[bRow * N + col] : 0.0f;
__syncthreads();
#pragma unroll
for (int k = 0; k < TILE_SIZE; k += 4) {
sum += As[ty][k] * Bs[k][tx];
sum += As[ty][k + 1] * Bs[k + 1][tx];
sum += As[ty][k + 2] * Bs[k + 2][tx];
sum += As[ty][k + 3] * Bs[k + 3][tx];
}
__syncthreads();
}
if (row < M && col < N) {
C[row * N + col] = sum;
}
}
"""
def run_matmul_benchmark(
m: int = 1024,
n: int = 1024,
k: int = 1024,
device_id: int = 0,
num_iterations: int = 10,
) -> bool:
"""Run matrix multiplication benchmark comparing nvmath vs custom kernel."""
print("=" * 60)
print("Matrix Multiplication with Shared Memory (GEMM)")
print("=" * 60)
# Initialize device and stream
device = Device(device_id)
device.set_current()
stream = device.create_stream()
print(f"\nDevice: {device.name}")
print(f"Compute Capability: sm_{device.arch}")
# Make CuPy use our cuda.core stream
cp.cuda.ExternalStream(int(stream.handle)).use()
# Compile custom kernel
arch = f"sm_{device.arch}"
program = Program(MATMUL_KERNEL, code_type="c++", options=ProgramOptions(arch=arch))
kernel = program.compile(target_type="cubin").get_kernel("matmul_shared")
print("Custom kernel compiled ✓")
# Setup
print(f"\nMatrix: A({m}x{k}) × B({k}x{n}) = C({m}x{n})")
total_ops = 2 * m * n * k
event_opts = EventOptions(enable_timing=True)
# Allocate matrices
rng = cp.random.default_rng(42)
d_A = rng.random((m, k), dtype=cp.float32)
d_B = rng.random((k, n), dtype=cp.float32)
d_C_custom = cp.zeros((m, n), dtype=cp.float32)
success = True
try:
# -------------------------------------------------------------------------
# nvmath GEMM (cuBLASLt)
# -------------------------------------------------------------------------
print("\n" + "-" * 60)
print("NVMATH (cuBLASLt) - plan once, execute many")
print("-" * 60)
with nvmath_advanced.Matmul(d_A, d_B, stream=int(stream.handle)) as mm:
mm.plan()
d_C_nvmath = mm.execute()
stream.sync()
start = stream.record(options=event_opts)
for _ in range(num_iterations):
d_C_nvmath = mm.execute()
end = stream.record(options=event_opts)
end.sync()
nvmath_ms = (end - start) / num_iterations
nvmath_gflops = (total_ops / 1e9) / (nvmath_ms / 1e3)
print(f"Time: {nvmath_ms:.3f} ms | {nvmath_gflops:.2f} GFLOPS")
# -------------------------------------------------------------------------
# Custom kernel (tiled + shared memory + unroll)
# -------------------------------------------------------------------------
print("\n" + "-" * 60)
print("CUSTOM KERNEL (tiled + shared memory + unroll)")
print("-" * 60)
block = (TILE_SIZE, TILE_SIZE)
grid = ((n + TILE_SIZE - 1) // TILE_SIZE, (m + TILE_SIZE - 1) // TILE_SIZE)
config = LaunchConfig(grid=grid, block=block)
launch(
stream,
config,
kernel,
d_A.data.ptr,
d_B.data.ptr,
d_C_custom.data.ptr,
np.int32(m),
np.int32(n),
np.int32(k),
)
stream.sync()
start = stream.record(options=event_opts)
for _ in range(num_iterations):
launch(
stream,
config,
kernel,
d_A.data.ptr,
d_B.data.ptr,
d_C_custom.data.ptr,
np.int32(m),
np.int32(n),
np.int32(k),
)
end = stream.record(options=event_opts)
end.sync()
custom_ms = (end - start) / num_iterations
custom_gflops = (total_ops / 1e9) / (custom_ms / 1e3)
print(f"Time: {custom_ms:.3f} ms | {custom_gflops:.2f} GFLOPS")
# -------------------------------------------------------------------------
# Verification
# -------------------------------------------------------------------------
print("\n" + "-" * 60)
print("VERIFICATION")
print("-" * 60)
d_C_ref = d_A @ d_B
# Host-side verification: cp.allclose triggers NVRTC failure on sm_120
# (ldexp_cexp undefined). Use asnumpy + np.allclose instead.
ref_host = cp.asnumpy(d_C_ref)
for name, d_C in [("nvmath", d_C_nvmath), ("custom", d_C_custom)]:
print(f"{name}: ", end="")
passed = np.allclose(cp.asnumpy(d_C), ref_host, rtol=1e-4, atol=1e-4)
print("Test PASSED" if passed else "Test FAILED")
success = success and passed
return success
finally:
cp.cuda.Stream.null.use()
stream.close()
def main() -> bool:
"""Entry point. Returns True if benchmark passed."""
return run_matmul_benchmark()
if __name__ == "__main__":
success = main()
if not success:
sys.exit(1)

7
python/2_CoreConcepts/matrixMulSharedMem/requirements.txt Normal file
View File

@@ -0,0 +1,7 @@
# Matrix Multiplication with Shared Memory (GEMM) Requirements
cuda-python>=13.0.0
cuda-core>=0.6.0
cupy-cuda13x>=13.0.0
numpy>=2.3.2
nvmath-python[cu13]>=0.3.0

139
python/2_CoreConcepts/memoryResources/README.md Normal file
View File

@@ -0,0 +1,139 @@
# Sample: Memory Resources and Buffers (Python)
## Description
This sample demonstrates the `cuda.core` memory management model: a
`MemoryResource` owns a pool of memory and hands out `Buffer` objects that
can be passed to kernels, copied between resources with
`Buffer.copy_to()`, and viewed as NumPy or CuPy arrays through DLPack. The
script exercises three common resources side-by-side:
1. **`DeviceMemoryResource`** - device-local GPU memory. Every `Device`
exposes a default pool via `Device.memory_resource`, and applications
can create additional pools explicitly.
2. **`PinnedMemoryResource`** - page-locked host memory, used here as the
input and output staging buffers around a GPU kernel (the canonical
pinned-H2D / compute / pinned-D2H pattern).
3. **`ManagedMemoryResource`** - unified memory that the driver migrates
between host and device on demand; host views see the GPU's writes
without an explicit copy.
The same `scale_and_bias` kernel runs on each resource and every result is
verified on the host.
## What You'll Learn
- Creating and using `DeviceMemoryResource`, `PinnedMemoryResource`, and
`ManagedMemoryResource`
- Allocating `Buffer` objects from a resource with a bound stream
- Copying between buffers across resources with `Buffer.copy_to()`
- Taking zero-copy NumPy or CuPy views of a `Buffer` via DLPack
- Releasing buffers with stream-ordered `close(stream)` semantics
## Key Libraries
- [`cuda.core`](https://nvidia.github.io/cuda-python/cuda-core/latest/) - Pythonic access to CUDA runtime, programs, and memory resources
- `cupy` - GPU array views of device buffers
- `numpy` - host array views of pinned and managed buffers
## Key APIs
### From `cuda.core`
- `Device.memory_resource` - default memory pool attached to a device
- `DeviceMemoryResource`, `PinnedMemoryResource`, `ManagedMemoryResource` - allocate buffers of the corresponding memory kind
- `MemoryResource.allocate(nbytes, stream=...)` - returns a `Buffer`
- `Buffer.copy_to(dst_buffer, stream=...)` - async, stream-ordered copy
- `Buffer.close(stream)` - stream-ordered deallocation
- `Buffer` supports `__dlpack__` for zero-copy views
### From CuPy and NumPy
- `cp.from_dlpack()` / `np.from_dlpack()` - zero-copy array view of a `Buffer`
### From `cuda_samples_utils`
- `print_gpu_info()` - print device name and compute capability
## Requirements
### Hardware
- NVIDIA GPU with Compute Capability 7.0 or higher
- Managed memory support (most discrete GPUs on Linux and Windows)
### Software
- CUDA Toolkit 13.0 or newer (matches `cuda-python` 13.x)
- Python 3.10 or newer
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=0.6.0)
- `cupy-cuda13x` (>=13.0.0)
## Installation
Install the required packages from `requirements.txt`:
```bash
cd /path/to/cuda-samples/python/2_CoreConcepts/memoryResources
pip install -r requirements.txt
```
The `requirements.txt` installs:
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=0.6.0)
- `cupy-cuda13x` (>=13.0.0)
## How to Run
### Basic usage
```bash
cd cuda-samples/python/2_CoreConcepts/memoryResources
python memoryResources.py
```
### With custom parameters
```bash
# Larger buffer size
python memoryResources.py --elements 1048576
# Use a specific GPU
python memoryResources.py --device 1
```
## Expected Output
```
Device: <Your GPU Name>
Compute Capability: <X.Y>
[1] DeviceMemoryResource + PinnedMemoryResource (staging)
Pinned staging, device kernel, and copy_to verified
[2] ManagedMemoryResource (unified memory)
GPU writes observed directly through the host-visible mapping
[3] Explicit DeviceMemoryResource
Explicit DeviceMemoryResource allocation verified
All memory resource demos passed.
```
**Note:** Device name and compute capability will vary based on your GPU.
## Files
- `memoryResources.py` - Python implementation using `cuda.core` memory resources
- `README.md` - This file
- `requirements.txt` - Sample dependencies
- `../../Utilities/cuda_samples_utils.py` - Common utilities (imported by this sample)
## See Also
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [`cuda.core` memory API](https://nvidia.github.io/cuda-python/cuda-core/latest/api.html#memory-management)
- Upstream `cuda.core` example: [`memory_ops.py`](https://github.com/NVIDIA/cuda-python/blob/main/cuda_core/examples/memory_ops.py)
- Upstream `cuda.core` example: [`memory_pool_resources.py`](https://github.com/NVIDIA/cuda-python/blob/main/cuda_core/examples/memory_pool_resources.py)

248
python/2_CoreConcepts/memoryResources/memoryResources.py Normal file
View File

@@ -0,0 +1,248 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Memory management with cuda.core: Buffers and Memory Resources
Demonstrates the Memory Resource / Buffer abstraction in cuda.core:
* ``DeviceMemoryResource`` - GPU-only memory (device pool)
* ``PinnedMemoryResource`` - page-locked host memory accessible by the GPU
* ``ManagedMemoryResource`` - unified memory that migrates between
host and device on demand
Each resource hands out ``Buffer`` objects that can be:
* passed to kernels as pointers
* copied between each other with ``buffer.copy_to(...)``
* viewed as NumPy or CuPy arrays via DLPack (``__dlpack__``)
The kernel below performs a fused scale + bias on both a device buffer
and a pinned buffer, then we copy the result across resources to confirm
each pathway works end-to-end.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
try:
import cupy as cp
import numpy as np
from cuda.core import (
Device,
DeviceMemoryResource,
LaunchConfig,
ManagedMemoryResource,
PinnedMemoryResource,
Program,
ProgramOptions,
launch,
)
from cuda_samples_utils import print_gpu_info # noqa: E402
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install from requirements.txt:")
print(" pip install -r requirements.txt")
sys.exit(1)
SCALE_BIAS_KERNEL = r"""
extern "C" __global__
void scale_and_bias(float* data, size_t N, float scale, float bias) {
const unsigned int tid = threadIdx.x + blockIdx.x * blockDim.x;
const unsigned int stride = blockDim.x * gridDim.x;
for (size_t i = tid; i < N; i += stride) {
data[i] = data[i] * scale + bias;
}
}
"""
def demo_device_and_pinned(device, stream, kernel, size):
"""Use pinned host memory as a staging area for a device-side kernel.
Canonical H2D / compute / D2H pattern:
host (pinned) -> device -> launch -> device -> host (pinned)
"""
print("\n[1] DeviceMemoryResource + PinnedMemoryResource (staging)")
dtype = np.float32
nbytes = size * dtype().itemsize
# The device's built-in memory resource is a good default for GPU memory.
device_mr = device.memory_resource
pinned_mr = PinnedMemoryResource()
pinned_in = pinned_mr.allocate(nbytes, stream=stream)
pinned_out = pinned_mr.allocate(nbytes, stream=stream)
device_buffer = device_mr.allocate(nbytes, stream=stream)
try:
# Wrap each Buffer as a typed array via DLPack (no copies).
pinned_in_view = np.from_dlpack(pinned_in).view(dtype=dtype)
pinned_out_view = np.from_dlpack(pinned_out).view(dtype=dtype)
# Initialize host-side input.
pinned_in_view[:] = np.arange(size, dtype=dtype)
original = pinned_in_view.copy()
# Stage H2D: pinned -> device.
pinned_in.copy_to(device_buffer, stream=stream)
# Launch kernel on the device buffer.
config = LaunchConfig(grid=(size + 255) // 256, block=256)
launch(
stream,
config,
kernel,
device_buffer,
np.uint64(size),
np.float32(3.0),
np.float32(-0.5),
)
# Stage D2H: device -> pinned.
device_buffer.copy_to(pinned_out, stream=stream)
stream.sync()
expected = original * 3.0 - 0.5
assert np.allclose(pinned_out_view, expected), "H2D -> kernel -> D2H mismatch"
print(" Pinned staging, device kernel, and copy_to verified")
finally:
device_buffer.close(stream)
pinned_out.close(stream)
pinned_in.close(stream)
def demo_managed(device, stream, kernel, size):
"""Allocate a managed (unified) buffer; kernel writes are visible on host."""
print("\n[2] ManagedMemoryResource (unified memory)")
dtype = np.float32
nbytes = size * dtype().itemsize
managed_mr = ManagedMemoryResource()
managed_buffer = managed_mr.allocate(nbytes, stream=stream)
try:
managed_view = np.from_dlpack(managed_buffer).view(dtype=dtype)
managed_view[:] = np.arange(size, dtype=dtype)
original = managed_view.copy()
# Before launching, make sure host writes have reached the GPU.
device.sync()
config = LaunchConfig(grid=(size + 255) // 256, block=256)
launch(
stream,
config,
kernel,
managed_buffer,
np.uint64(size),
np.float32(0.5),
np.float32(10.0),
)
stream.sync()
# No explicit copy: the same numpy view observes the GPU's writes.
assert np.allclose(managed_view, original * 0.5 + 10.0), (
"Managed memory result mismatch"
)
print(" GPU writes observed directly through the host-visible mapping")
finally:
managed_buffer.close(stream)
def demo_explicit_device_pool(device, stream, kernel, size):
"""Allocate from a user-created DeviceMemoryResource with default options."""
print("\n[3] Explicit DeviceMemoryResource")
dtype = np.float32
nbytes = size * dtype().itemsize
# Explicitly create a pool tied to this device. Use .close() to tear it down.
explicit_mr = DeviceMemoryResource(device)
buffer = explicit_mr.allocate(nbytes, stream=stream)
try:
view = cp.from_dlpack(buffer).view(dtype=cp.float32)
view[:] = cp.arange(size, dtype=cp.float32)
device.sync()
config = LaunchConfig(grid=(size + 255) // 256, block=256)
launch(
stream,
config,
kernel,
buffer,
np.uint64(size),
np.float32(1.0),
np.float32(100.0),
)
stream.sync()
expected = cp.arange(size, dtype=cp.float32) + 100.0
assert cp.allclose(view, expected), "Explicit device pool result mismatch"
print(" Explicit DeviceMemoryResource allocation verified")
finally:
buffer.close(stream)
explicit_mr.close()
def main():
import argparse
parser = argparse.ArgumentParser(
description="Demonstrate cuda.core memory resources (Buffer + MR)"
)
parser.add_argument(
"--elements",
type=int,
default=1 << 16,
help="Number of float32 elements per buffer (default: 65536)",
)
parser.add_argument("--device", type=int, default=0, help="CUDA device id")
args = parser.parse_args()
device = Device(args.device)
device.set_current()
print_gpu_info(device)
stream = device.create_stream()
try:
program_options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
program = Program(SCALE_BIAS_KERNEL, code_type="c++", options=program_options)
module = program.compile("cubin")
kernel = module.get_kernel("scale_and_bias")
demo_device_and_pinned(device, stream, kernel, args.elements)
demo_managed(device, stream, kernel, args.elements)
demo_explicit_device_pool(device, stream, kernel, args.elements)
print("\nDone")
return 0
finally:
stream.close()
if __name__ == "__main__":
sys.exit(main())

5
python/2_CoreConcepts/memoryResources/requirements.txt Normal file
View File

@@ -0,0 +1,5 @@
# Memory Resources Sample Requirements
cuda-python>=13.0.0
cuda-core>=0.6.0
cupy-cuda13x>=13.0.0

158
python/2_CoreConcepts/pageRank/README.md Normal file
View File

@@ -0,0 +1,158 @@
# Sample: PageRank Algorithm (Python)
## Description
Demonstrates GPU-accelerated PageRank computation for graph analysis using RAPIDS cuGraph, with cuda.core for device, stream, and GPU timing. This sample focuses on cuda.core integration with high-level libraries (cuGraph/cuDF); for custom kernel programming (Program, LaunchConfig, launch), see the blockwiseSum sample.
## What You'll Learn
- Graph representation using cuDF DataFrames for edge lists
- GPU-optimized PageRank via RAPIDS cuGraph library
- Performance comparison between cuGraph GPU and CPU reference implementation
- cuda.core device/stream management and GPU timing
## Key Libraries
- `cugraph` - RAPIDS GPU-accelerated graph analytics
- `cudf` - RAPIDS GPU DataFrame library
- `cuda.core` - Device, stream, and event APIs for GPU timing
- `cupy` - GPU array library (ExternalStream for cuDF/cuGraph)
- `numpy` - CPU reference implementation
## Key APIs
### From cuda.core:
- `Device(0)` - Create device, `device.set_current()`, `device.create_stream()`
- `EventOptions(enable_timing=True)` - GPU timing via `stream.record()`
- `cp.cuda.ExternalStream(stream.handle).use()` - Make cuDF/cuGraph use cuda.core stream
### From cuGraph:
- `cugraph.Graph(directed=True)` - Create directed graph structure
- `Graph.from_cudf_edgelist()` - Build graph from edge list DataFrame
- `cugraph.pagerank()` - GPU-accelerated PageRank algorithm
### From cuDF:
- `cudf.DataFrame()` - GPU DataFrame for edge lists
## Requirements
### Hardware:
- NVIDIA GPU with Compute Capability 7.0 or higher
- Minimum GPU memory: 512 MB (for 10K node graph)
### Software:
- CUDA Toolkit 13.0 or newer
- Python 3.10 or newer
- See requirements.txt for package dependencies
## Installation
```bash
cd /path/to/cuda-samples/python/2_CoreConcepts/pageRank
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```
## How to Run
```bash
python pageRank.py
```
## Algorithm
The PageRank formula iteratively computes node importance:
```
PR(v) = (1-d)/N + d * Σ PR(u)/out_degree(u)
```
Where:
- `d` = damping factor (typically 0.85)
- `N` = total number of nodes
- Sum is over all nodes `u` that link to `v`
## Expected Output
```
============================================================
PageRank Algorithm (using RAPIDS cuGraph)
============================================================
Device: NVIDIA GeForce RTX ...
Compute Capability: sm_XX
Graph Parameters:
Nodes: 10,000
Avg edges/node: 15
Total edges: ~150,000
Avg in-degree: 14.9
------------------------------------------------------------
GPU PageRank (RAPIDS cuGraph)
------------------------------------------------------------
Time: X.XXX ms
Top 5 nodes by PageRank:
1. Node XXXXX: 0.XXXXXX
...
------------------------------------------------------------
CPU PageRank (Reference)
------------------------------------------------------------
Time: XXXX.XXX ms
Iterations: XX
------------------------------------------------------------
PERFORMANCE SUMMARY
------------------------------------------------------------
GPU (cuGraph): X.XXX ms
CPU (Reference): XXXX.XXX ms
Speedup: XXXX.Xx
------------------------------------------------------------
VERIFICATION
------------------------------------------------------------
GPU vs CPU PageRank scores: Test PASSED
PageRank Properties:
Sum of scores: 1.000000 (should be ~1.0)
Sum check: ✓
Done
```
## Files
- `pageRank.py` - Python implementation using RAPIDS cuGraph
- `README.md` - This file
- `requirements.txt` - Sample dependencies
## Why cuGraph?
RAPIDS cuGraph provides production-grade, GPU-accelerated graph analytics:
- **Highly optimized** - Uses advanced GPU parallelization techniques
- **Scalable** - Handles graphs with billions of edges
- **Easy to use** - Simple Python API similar to NetworkX
- **Integrated** - Works seamlessly with cuDF, cuML, and other RAPIDS libraries
## Applications
- Web page ranking (original Google PageRank)
- Social network influence analysis
- Citation network analysis
- Recommendation systems
- Fraud detection in financial networks
## See Also
- [RAPIDS cuGraph Documentation](https://docs.rapids.ai/api/cugraph/stable/)
- [cuGraph GitHub Repository](https://github.com/rapidsai/cugraph)
- [RAPIDS Installation Guide](https://rapids.ai/start.html)

346
python/2_CoreConcepts/pageRank/pageRank.py Normal file
View File

@@ -0,0 +1,346 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
PageRank Algorithm
Demonstrates GPU-accelerated PageRank computation for graph analysis:
- Graph representation using edge lists and cuDF DataFrames
- GPU-optimized PageRank via RAPIDS cuGraph library
- Performance comparison: cuGraph GPU vs CPU reference
Uses RAPIDS cuGraph for production-grade graph analytics on GPU.
PageRank Algorithm:
PR(v) = (1-d)/N + d * sum(PR(u)/out_degree(u)) for all u linking to v
where d = damping factor (typically 0.85), N = number of nodes
"""
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
from cuda_samples_utils import print_gpu_info, verify_array_result # noqa: E402
try:
import cudf
import cugraph
import cupy as cp
import numpy as np
from cuda.core import Device, EventOptions, Stream
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Install with: pip install -r requirements.txt")
sys.exit(1)
def generate_random_graph(
num_nodes: int,
avg_edges_per_node: int = 10,
seed: int = 42,
) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Generate a random directed graph as edge list.
Parameters
----------
num_nodes : int
Number of nodes in the graph
avg_edges_per_node : int
Average number of outgoing edges per node
seed : int
Random seed for reproducibility
Returns
-------
tuple[np.ndarray, np.ndarray, np.ndarray]
(sources, destinations, out_degree) arrays
"""
rng = np.random.default_rng(seed)
sources_list: list[int] = []
destinations_list: list[int] = []
out_degree = np.zeros(num_nodes, dtype=np.int32)
for src in range(num_nodes):
# Random number of outgoing edges (Poisson distribution)
n_edges = max(1, rng.poisson(avg_edges_per_node))
n_edges = min(n_edges, num_nodes - 1)
# Random destinations (no self-loops); rejection sampling avoids O(N²) memory
dests: set[int] = set()
while len(dests) < n_edges:
d = int(rng.integers(0, num_nodes))
if d != src:
dests.add(d)
dests = np.array(list(dests), dtype=np.int32)
for dst in dests:
sources_list.append(src)
destinations_list.append(dst)
out_degree[src] = len(dests)
sources = np.array(sources_list, dtype=np.int32)
destinations = np.array(destinations_list, dtype=np.int32)
return sources, destinations, out_degree
def pagerank_cpu(
sources: np.ndarray,
destinations: np.ndarray,
out_degree: np.ndarray,
num_nodes: int,
damping: float = 0.85,
max_iterations: int = 100,
tolerance: float = 1e-6,
) -> tuple[np.ndarray, int]:
"""
Compute PageRank on CPU using iterative method.
Parameters
----------
sources : np.ndarray
Source nodes of edges
destinations : np.ndarray
Destination nodes of edges
out_degree : np.ndarray
Outgoing degree for each node
num_nodes : int
Number of nodes
damping : float
Damping factor (default: 0.85)
max_iterations : int
Maximum iterations
tolerance : float
Convergence tolerance
Returns
-------
tuple[np.ndarray, int]
(PageRank scores, iterations until convergence)
"""
# Build incoming edges list for each node
incoming: list[list[int]] = [[] for _ in range(num_nodes)]
for src, dst in zip(sources, destinations):
incoming[dst].append(src)
# Initialize PageRank uniformly
pr = np.ones(num_nodes, dtype=np.float32) / num_nodes
pr_new = np.zeros(num_nodes, dtype=np.float32)
base_score = (1.0 - damping) / num_nodes
for iteration in range(max_iterations):
# Handle dangling nodes (nodes with no outgoing edges)
dangling_sum = np.sum(pr[out_degree == 0])
dangling_contrib = damping * dangling_sum / num_nodes
for v in range(num_nodes):
# Sum contributions from incoming neighbors
incoming_sum = 0.0
for u in incoming[v]:
if out_degree[u] > 0:
incoming_sum += pr[u] / out_degree[u]
pr_new[v] = base_score + damping * incoming_sum + dangling_contrib
# Check convergence
diff = np.sum(np.abs(pr_new - pr))
pr, pr_new = pr_new, pr
if diff < tolerance:
return pr, iteration + 1
return pr, max_iterations
def run_pagerank_benchmark(
num_nodes: int = 10000,
avg_edges: int = 15,
max_iterations: int = 100,
) -> bool:
"""
Run PageRank benchmark comparing cuGraph GPU and CPU performance.
Parameters
----------
num_nodes : int
Number of nodes in the graph
avg_edges : int
Average edges per node
max_iterations : int
Maximum PageRank iterations
Returns
-------
bool
True if benchmark succeeded
"""
print("=" * 60)
print("PageRank Algorithm (using RAPIDS cuGraph)")
print("=" * 60)
# Initialize cuda.core device and stream
device = Device(0)
device.set_current()
stream: Stream = device.create_stream()
print()
print_gpu_info(device)
# Make CuPy/cuDF use our cuda.core stream
cp.cuda.ExternalStream(int(stream.handle)).use()
# Generate random graph
print("\nGraph Parameters:")
print(f" Nodes: {num_nodes:,}")
print(f" Avg edges/node: {avg_edges}")
sources, destinations, out_degree = generate_random_graph(
num_nodes, avg_edges, seed=42
)
total_edges = len(sources)
print(f" Total edges: {total_edges:,}")
print(f" Avg in-degree: {total_edges / num_nodes:.1f}")
# -------------------------------------------------------------------------
# GPU PageRank (cuGraph)
# -------------------------------------------------------------------------
print("\n" + "-" * 60)
print("GPU PageRank (RAPIDS cuGraph)")
print("-" * 60)
# Create cuGraph graph from edge list with store_transposed for optimal perf
gdf = cudf.DataFrame(
{
"src": sources,
"dst": destinations,
}
)
G = cugraph.Graph(directed=True)
G.from_cudf_edgelist(gdf, source="src", destination="dst", store_transposed=True)
event_opts = EventOptions(enable_timing=True)
try:
# Warmup
_ = cugraph.pagerank(G, alpha=0.85, max_iter=100, tol=1e-5)
stream.sync()
# Timed run using cuda.core events
start = stream.record(options=event_opts)
pr_result = cugraph.pagerank(G, alpha=0.85, max_iter=max_iterations, tol=1e-6)
end = stream.record(options=event_opts)
end.sync()
gpu_time_ms = end - start
print(f"Time: {gpu_time_ms:.3f} ms")
# Extract results sorted by vertex ID (to numpy for verification)
pr_df = pr_result.sort_values("vertex").reset_index(drop=True)
pr_gpu = pr_df["pagerank"].to_numpy()
# Show top PageRank nodes
top_k = 5
top_df = pr_result.nlargest(top_k, "pagerank")
print(f"\nTop {top_k} nodes by PageRank:")
for i, row in enumerate(top_df.to_pandas().itertuples()):
print(f" {i + 1}. Node {row.vertex:5d}: {row.pagerank:.6f}")
# -------------------------------------------------------------------------
# CPU PageRank
# -------------------------------------------------------------------------
print("\n" + "-" * 60)
print("CPU PageRank (Reference)")
print("-" * 60)
cpu_start = time.perf_counter()
pr_cpu, cpu_iters = pagerank_cpu(
sources, destinations, out_degree, num_nodes, max_iterations=max_iterations
)
cpu_end = time.perf_counter()
cpu_time_ms = (cpu_end - cpu_start) * 1000
print(f"Time: {cpu_time_ms:.3f} ms")
print(f"Iterations: {cpu_iters}")
# -------------------------------------------------------------------------
# Performance Summary
# -------------------------------------------------------------------------
print("\n" + "-" * 60)
print("PERFORMANCE SUMMARY")
print("-" * 60)
speedup = cpu_time_ms / gpu_time_ms
print(f"GPU (cuGraph): {gpu_time_ms:.3f} ms")
print(f"CPU (Reference): {cpu_time_ms:.3f} ms")
print(f"Speedup: {speedup:.1f}x")
# -------------------------------------------------------------------------
# Verification
# -------------------------------------------------------------------------
print("\n" + "-" * 60)
print("VERIFICATION")
print("-" * 60)
# Compare GPU and CPU results (cuGraph and CPU ref may converge differently)
print("GPU vs CPU PageRank scores: ", end="")
success = verify_array_result(
pr_gpu, pr_cpu, rtol=1e-2, atol=1e-4, verbose=True
)
# Verify PageRank properties
print("\nPageRank Properties:")
pr_sum = float(np.sum(pr_gpu))
print(f" Sum of scores: {pr_sum:.6f} (should be ~1.0)")
pr_min = float(np.min(pr_gpu))
pr_max = float(np.max(pr_gpu))
print(f" Min score: {pr_min:.6f}")
print(f" Max score: {pr_max:.6f}")
# Check that sum is approximately 1
sum_ok = abs(pr_sum - 1.0) < 0.01
print(f" Sum check: {'✓' if sum_ok else '✗'}")
success = success and sum_ok
return success
finally:
cp.cuda.Stream.null.use()
stream.close()
def main() -> None:
"""Entry point."""
success = run_pagerank_benchmark()
if success:
print("\nDone")
else:
print("\nBenchmark completed with errors")
sys.exit(1)
if __name__ == "__main__":
main()

8
python/2_CoreConcepts/pageRank/requirements.txt Normal file
View File

@@ -0,0 +1,8 @@
# PageRank Requirements (RAPIDS cuGraph)
cuda-python>=13.0.0
cuda-core>=0.6.0
cugraph-cu13>=25.0.0
cudf-cu13>=25.0.0
cupy-cuda13x>=13.0.0
numpy>=2.3.2

117
python/2_CoreConcepts/parallelHistogram/README.md Normal file
View File

@@ -0,0 +1,117 @@
# Sample: Parallel Histogram with Atomics (Python)
## Description
Compute histograms on the GPU using atomic operations to handle concurrent updates from multiple threads. This sample demonstrates the modern **cuda.core API** for kernel compilation and launch, comparing two approaches:
1. **Global Atomics** - All threads atomically update a single global histogram
2. **Privatized Histograms** - Each block uses shared memory, then merges to global
## What You'll Learn
- Compiling CUDA C kernels with `cuda.core.Program`
- Configuring kernel launches with `cuda.core.LaunchConfig`
- Launching kernels with `cuda.core.launch()`
- Using **atomic operations** (`atomicAdd`) for thread-safe updates
- Optimizing with **shared memory privatization**
- GPU timing with `cuda.core` Events
## Key Concepts
### Atomic Operations
When multiple threads update the same histogram bin, a race condition occurs. Atomic operations ensure thread-safe updates:
```cuda
atomicAdd(&histogram[data[i]], 1); // Thread-safe increment
```
### Global vs Privatized Atomics
| Approach | Pros | Cons |
|----------|------|------|
| Global | Simple | High contention on popular bins |
| Privatized | Significantly faster | Extra shared memory, synchronization |
## Key APIs
### From `cuda.core`:
- `Device` - Device management and context
- `Program` - Compile CUDA C source code
- `ProgramOptions` - Set architecture, optimization flags
- `LaunchConfig` - Configure grid and block dimensions
- `launch()` - Launch compiled kernel
- `Stream` - Async stream management
- `EventOptions` - Configure events for GPU timing
- `stream.record()` - Record events for timing
### From `cupy`:
- `cp.random.randint()` - Generate random data directly on GPU
- `cp.zeros()` - Allocate zeroed GPU arrays
### CUDA Atomic Functions (in kernel):
- `atomicAdd()` - Thread-safe addition
## Requirements
### Hardware:
- NVIDIA GPU with CUDA support
### Software:
- CUDA Toolkit 13.0 or newer
- Python 3.10 or newer
- See `requirements.txt` for Python packages
## Installation
```bash
pip install -r requirements.txt
```
## How to Run
```bash
python parallelHistogram.py
```
## Expected Output
```
============================================================
Parallel Histogram with Atomics (cuda.core)
============================================================
Device: <Your GPU>
Compute Capability: ComputeCapability(major=X, minor=Y)
Compiling CUDA kernels with cuda.core.Program...
Compiled for architecture: sm_XY
Generating 10,000,000 random values on GPU...
Verifying correctness...
Global atomics: PASSED
Privatized atomics: PASSED
Benchmarking (100 iterations)...
Global atomics: X.XXX ms
Privatized atomics: X.XXX ms
Speedup: XXx
Test PASSED
```
## Files
- `parallelHistogram.py` - Main sample using cuda.core
- `README.md` - This file
- `requirements.txt` - Dependencies
## See Also
- [cuda.core Documentation](https://nvidia.github.io/cuda-python/cuda-core/latest/)
- [CUDA Atomic Functions](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#atomic-functions)
- [CUDA Shared Memory](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#shared-memory)

237
python/2_CoreConcepts/parallelHistogram/parallelHistogram.py Normal file
View File

@@ -0,0 +1,237 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Parallel Histogram with Atomics using cuda.core
This sample demonstrates GPU histogram computation using atomic operations,
showcasing the modern cuda.core API for:
- Kernel compilation (Program, ProgramOptions)
- Kernel launch configuration (LaunchConfig)
- Stream management (Stream)
- Event timing (EventOptions)
Two histogram approaches are compared:
1. Global Atomics - All threads atomically update global memory
2. Privatized Histograms - Shared memory reduces global atomic contention
"""
import sys
try:
import cupy as cp
import numpy as np
from cuda.core import (
Device,
EventOptions,
LaunchConfig,
Program,
ProgramOptions,
launch,
)
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install: pip install -r requirements.txt")
sys.exit(1)
NUM_BINS = 256
# CUDA C source code for both histogram kernels
HISTOGRAM_KERNELS = r"""
// Global Atomics - simple but high contention on popular bins
extern "C" __global__
void histogram_global(const unsigned char* data, unsigned int* histogram, int n) {
int idx = blockIdx.x * blockDim.x + threadIdx.x;
int stride = blockDim.x * gridDim.x;
for (int i = idx; i < n; i += stride) {
atomicAdd(&histogram[data[i]], 1);
}
}
// Privatized - uses shared memory to reduce global atomic contention
extern "C" __global__
void histogram_privatized(const unsigned char* data, unsigned int* histogram, int n) {
__shared__ unsigned int local_hist[256];
int tid = threadIdx.x;
int idx = blockIdx.x * blockDim.x + threadIdx.x;
int stride = blockDim.x * gridDim.x;
// Initialize shared memory
for (int i = tid; i < 256; i += blockDim.x)
local_hist[i] = 0;
__syncthreads();
// Accumulate into shared memory (fast)
for (int i = idx; i < n; i += stride)
atomicAdd(&local_hist[data[i]], 1);
__syncthreads();
// Merge to global (fewer atomics)
for (int i = tid; i < 256; i += blockDim.x)
if (local_hist[i] > 0)
atomicAdd(&histogram[i], local_hist[i]);
}
"""
def main():
print("=" * 60)
print("Parallel Histogram with Atomics (cuda.core)")
print("=" * 60)
# Initialize device using cuda.core
device = Device(0)
device.set_current()
print(f"\nDevice: {device.name}")
print(f"Compute Capability: {device.compute_capability}")
# Create stream using cuda.core
stream = device.create_stream()
# Make CuPy use the same stream for correct ordering (avoids null-stream sync)
cp.cuda.Stream.from_external(stream).use()
try:
_run_histogram(device, stream)
finally:
cp.cuda.Stream.null.use() # Restore CuPy to default stream before closing
stream.close()
def _run_histogram(device, stream):
"""Run histogram computation and benchmarking."""
# Compile CUDA kernels using cuda.core.Program
print("\nCompiling CUDA kernels with cuda.core.Program...")
arch = f"sm_{device.arch}"
options = ProgramOptions(arch=arch)
program = Program(HISTOGRAM_KERNELS, code_type="c++", options=options)
object_code = program.compile("cubin")
kernel_global = object_code.get_kernel("histogram_global")
kernel_privatized = object_code.get_kernel("histogram_privatized")
print(f" Compiled for architecture: {arch}")
# Generate test data directly on GPU (more efficient than CPU->GPU copy)
n = 10_000_000
print(f"\nGenerating {n:,} random values on GPU...")
data_gpu = cp.random.randint(0, 256, size=n, dtype=cp.uint8)
hist_gpu = cp.zeros(NUM_BINS, dtype=cp.uint32)
# Compute reference histogram on CPU for verification
data_cpu = cp.asnumpy(data_gpu)
hist_cpu, _ = np.histogram(data_cpu, bins=NUM_BINS, range=(0, 256))
hist_cpu = hist_cpu.astype(np.uint32)
# Configure kernel launch using cuda.core.LaunchConfig
block_size = 256
grid_size = min((n + block_size - 1) // block_size, 1024)
config = LaunchConfig(grid=(grid_size,), block=(block_size,))
print("\nVerifying correctness...")
# Ensure CuPy allocations complete before kernel launch on our stream
stream.sync()
# Launch global atomics kernel (hist_gpu is already zeros from cp.zeros)
launch(
stream, config, kernel_global, data_gpu.data.ptr, hist_gpu.data.ptr, np.int32(n)
)
stream.sync()
hist_global = cp.asnumpy(hist_gpu)
global_ok = np.array_equal(hist_cpu, hist_global)
print(f" Global atomics: {'PASSED' if global_ok else 'FAILED'}")
# Reset histogram and launch privatized kernel (fill on same stream)
hist_gpu.fill(0)
launch(
stream,
config,
kernel_privatized,
data_gpu.data.ptr,
hist_gpu.data.ptr,
np.int32(n),
)
stream.sync()
hist_privatized = cp.asnumpy(hist_gpu)
privatized_ok = np.array_equal(hist_cpu, hist_privatized)
print(f" Privatized atomics: {'PASSED' if privatized_ok else 'FAILED'}")
if not (global_ok and privatized_ok):
sys.exit(1)
# Benchmark using cuda.core Events (explicit Event objects recorded on stream)
print("\nBenchmarking (100 iterations)...")
num_iterations = 100
event_opts = EventOptions(enable_timing=True)
start_event = device.create_event(options=event_opts)
end_event = device.create_event(options=event_opts)
# Benchmark global atomics
stream.record(start_event)
for _ in range(num_iterations):
hist_gpu.fill(0)
launch(
stream,
config,
kernel_global,
data_gpu.data.ptr,
hist_gpu.data.ptr,
np.int32(n),
)
stream.record(end_event)
end_event.sync()
time_global = (end_event - start_event) / num_iterations
# Benchmark privatized
stream.record(start_event)
for _ in range(num_iterations):
hist_gpu.fill(0)
launch(
stream,
config,
kernel_privatized,
data_gpu.data.ptr,
hist_gpu.data.ptr,
np.int32(n),
)
stream.record(end_event)
end_event.sync()
time_privatized = (end_event - start_event) / num_iterations
print(f" Global atomics: {time_global:.3f} ms")
print(f" Privatized atomics: {time_privatized:.3f} ms")
print(f" Speedup: {time_global / time_privatized:.1f}x")
print("\nTest PASSED")
if __name__ == "__main__":
main()

7
python/2_CoreConcepts/parallelHistogram/requirements.txt Normal file
View File

@@ -0,0 +1,7 @@
# Parallel Histogram with Atomics Sample Requirements
# Requires Python 3.10+, CUDA Toolkit 13.0+
cuda-python>=13.0.0
cuda-core>=0.6.0
numpy>=2.3.2
cupy-cuda13x>=13.0.0

119
python/2_CoreConcepts/parallelReduction/README.md Normal file
View File

@@ -0,0 +1,119 @@
# Sample: Parallel Reduction (Python)
## Description
Efficiently sum a large array on GPU using parallel reduction. This sample demonstrates:
1. **Custom CUDA kernel** showing reduction tree pattern and synchronization
2. **cuda.compute.reduce_into()** for production-ready reduction
## What You'll Learn
- **Reduction tree pattern**: Divide-and-conquer parallel algorithm
- **Thread synchronization**: Using `__syncthreads()` for coordination
- **Avoiding warp divergence**: Sequential thread IDs vs strided IDs
## Key Concepts
### Reduction Tree Pattern
Parallel reduction uses a tree-based approach where each iteration halves active elements:
```
Initial: [a0, a1, a2, a3, a4, a5, a6, a7]
Step 1: [a0+a4, a1+a5, a2+a6, a3+a7] threads 0-3 active
Step 2: [a0+a2+a4+a6, a1+a3+a5+a7] threads 0-1 active
Step 3: [sum of all] thread 0 only
```
This requires only `log2(N)` steps to reduce N elements.
### Avoiding Warp Divergence
```c
// Good: Sequential thread IDs (warps stay coherent)
if (tid < s) {
sdata[tid] += sdata[tid + s];
}
// Bad: Strided IDs (causes warp divergence)
if (tid % (2 * s) == 0) { // Don't do this!
sdata[tid] += sdata[tid + s];
}
```
## Requirements
### Hardware
- NVIDIA GPU with CUDA support
### Software
- CUDA Toolkit 13.0+
- Python 3.10+
- `cuda-python` (13.0.0+)
- `cuda-core` (>=0.6.0)
- `cuda-cccl` (1.0.0+)
- `cupy` (13.0.0+)
- `numpy` (>=2.3.2)
## Installation
```bash
pip install -r requirements.txt
```
## How to Run
```bash
python parallelReduction.py
```
## Expected Output
```
======================================================================
Parallel Reduction - Efficient GPU Array Summation
======================================================================
Device: <your GPU name>
Compute Capability: <version>
Array size: 1,048,576 elements (4.2 MB)
Expected sum: <value>
Compiling custom CUDA kernel...
======================================================================
PART 1: Custom Kernel (Educational)
======================================================================
Reduction tree kernel: <result>
Expected: <result>
Error: <small value>
Time: <varies> ms
======================================================================
PART 2: cuda.compute.reduce_into() (Production)
======================================================================
cuda.compute result: <result>
Expected: <result>
Error: <small value>
Time: <varies> ms
Test PASSED!
```
Note: Exact values vary due to random input data. `cuda.compute.reduce_into()` is typically faster than the custom kernel because it calls CUB's `DeviceReduce`, which uses highly tuned, architecture‑specific kernels and optimized memory access patterns.
## Files
- `parallelReduction.py` - Custom kernel + cuda.compute comparison
- `README.md` - This documentation
- `requirements.txt` - Python dependencies
## See Also
- [Mark Harris - Optimizing Parallel Reduction in CUDA](https://developer.download.nvidia.com/assets/cuda/files/reduction.pdf)
- [cuda.core Documentation](https://nvidia.github.io/cuda-python/cuda-core/latest/)

375
python/2_CoreConcepts/parallelReduction/parallelReduction.py Normal file
View File

@@ -0,0 +1,375 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Parallel Reduction using cuda.core and cuda.compute
Demonstrates efficient parallel summation of large arrays on GPU:
1. Custom CUDA kernel showing reduction tree pattern and synchronization
2. cuda.compute.reduce_into() for production-ready reduction
Key Concepts:
- Reduction tree pattern: Divide-and-conquer parallel algorithm
- Thread synchronization: Using __syncthreads() for coordination
- Sequential thread IDs: How to avoid warp divergence
- cuda.core Stream integration with CuPy via ExternalStream
"""
import math
import sys
from pathlib import Path
# Add Utilities to path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
try:
import cupy as cp
import numpy as np
from cuda.compute import OpKind, reduce_into
from cuda.core import (
Device,
Kernel,
LaunchConfig,
Program,
ProgramOptions,
Stream,
launch,
)
from cuda_samples_utils import print_gpu_info, verify_array_result
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install from requirements.txt:")
print(" pip install -r requirements.txt")
sys.exit(1)
# =============================================================================
# CUDA Kernel: Parallel Reduction (optimized - no warp divergence)
# =============================================================================
REDUCTION_KERNEL: str = r"""
extern "C" __global__
void reduce_sum(const float* __restrict__ input,
float* __restrict__ output, int n) {
/*
* Parallel reduction using grid-stride loop (canonical pattern) and
* sequential thread IDs for the reduction tree (avoids warp divergence).
*
* Grid-stride loop: each thread processes multiple elements
* for (i = tid; i < n; i += gridDim.x * blockDim.x)
*
* Reduction tree: sequential addressing keeps warps coherent.
*/
extern __shared__ float sdata[];
unsigned int tid = threadIdx.x;
unsigned int grid_stride = (unsigned int)gridDim.x * blockDim.x;
float sum = 0.0f;
for (unsigned int i = blockIdx.x * blockDim.x + threadIdx.x; i < n;
i += grid_stride) {
sum += input[i];
}
sdata[tid] = sum;
__syncthreads();
// Reduction in shared memory (sequential addressing - no divergence)
for (unsigned int s = blockDim.x / 2; s > 0; s >>= 1) {
if (tid < s) {
sdata[tid] += sdata[tid + s];
}
__syncthreads(); // Wait for all threads before next iteration
}
// Thread 0 writes block result
if (tid == 0) {
output[blockIdx.x] = sdata[0];
}
}
"""
def compile_kernel(device: Device) -> Kernel:
"""Compile the reduction kernel for the given device."""
arch = f"sm_{device.arch}"
options = ProgramOptions(arch=arch)
program = Program(REDUCTION_KERNEL, code_type="c++", options=options)
return program.compile(target_type="cubin").get_kernel("reduce_sum")
def reduction_stage_output_counts(n: int, block_size: int) -> list[int]:
"""Lengths of intermediate arrays for each multi-launch reduction stage."""
counts: list[int] = []
while n > 1:
num_blocks = math.ceil(n / block_size)
counts.append(num_blocks)
n = num_blocks
return counts
def reduce_custom(
stream: Stream,
kernel: Kernel,
d_input: cp.ndarray,
block_size: int = 256,
sync: bool = True,
work_buffers: list[cp.ndarray] | None = None,
) -> float | cp.ndarray:
"""
Perform parallel reduction using custom CUDA kernel.
Uses multiple kernel launches to reduce array to single value.
Each launch reduces by factor of block_size.
When sync=True (default), syncs and returns the scalar result.
When sync=False, returns the 1-element array without syncing;
caller must sync before reading (avoids host overhead in benchmarks).
work_buffers: optional list of device arrays, one per stage, with length
at least each stage's output count (from ``reduction_stage_output_counts``).
When provided, avoids per-call allocation (e.g. for benchmarking).
"""
n = len(d_input)
current = d_input
stage = 0
if work_buffers is not None:
expected_counts = reduction_stage_output_counts(n, block_size)
if len(work_buffers) != len(expected_counts):
msg = (
f"work_buffers length {len(work_buffers)} != "
f"{len(expected_counts)} stages"
)
raise ValueError(msg)
while n > 1:
num_blocks = math.ceil(n / block_size)
if work_buffers is not None:
d_output = work_buffers[stage]
if d_output.size < num_blocks:
msg = f"work_buffers[{stage}] size {d_output.size} < {num_blocks}"
raise ValueError(msg)
if d_output.size != num_blocks:
d_output = d_output[:num_blocks]
else:
d_output = cp.empty(num_blocks, dtype=cp.float32)
config = LaunchConfig(
grid=(num_blocks, 1, 1),
block=(block_size, 1, 1),
shmem_size=block_size * 4, # float = 4 bytes
)
launch(
stream,
config,
kernel,
current.data.ptr,
d_output.data.ptr,
np.int32(n),
)
current = d_output
n = num_blocks
stage += 1
if sync:
stream.sync()
return float(current[0])
return current
def benchmark_custom(
stream: Stream,
kernel: Kernel,
d_input: cp.ndarray,
num_runs: int = 10,
block_size: int = 256,
) -> tuple[float, float]:
"""Benchmark custom reduction kernel using cuda.core events."""
stage_counts = reduction_stage_output_counts(len(d_input), block_size)
work_buffers = [cp.empty(c, dtype=cp.float32) for c in stage_counts]
# Warmup run (with sync to get valid result)
_ = reduce_custom(
stream, kernel, d_input, block_size=block_size, work_buffers=work_buffers
)
event_opts = {"enable_timing": True}
start_event = stream.device.create_event(options=event_opts)
end_event = stream.device.create_event(options=event_opts)
times: list[float] = []
result = 0.0
for _ in range(num_runs):
stream.record(start_event)
d_result = reduce_custom(
stream,
kernel,
d_input,
block_size=block_size,
sync=False,
work_buffers=work_buffers,
)
stream.record(end_event)
end_event.sync()
result = float(d_result[0])
times.append(end_event - start_event)
return result, float(np.mean(times))
def benchmark_cuda_compute(
stream: Stream,
d_input: cp.ndarray,
num_runs: int = 10,
) -> tuple[float, float]:
"""Benchmark cuda.compute.reduce_into() using cuda.core events."""
h_init = np.array([0.0], dtype=np.float32)
# Warmup (includes JIT compilation)
d_warmup = cp.empty(1, dtype=cp.float32)
reduce_into(
d_in=d_input,
d_out=d_warmup,
op=OpKind.PLUS,
num_items=len(d_input),
h_init=h_init,
stream=stream,
)
stream.sync()
d_output = cp.empty(1, dtype=cp.float32)
event_opts = {"enable_timing": True}
start_event = stream.device.create_event(options=event_opts)
end_event = stream.device.create_event(options=event_opts)
times: list[float] = []
result = 0.0
for _ in range(num_runs):
stream.record(start_event)
reduce_into(
d_in=d_input,
d_out=d_output,
op=OpKind.PLUS,
num_items=len(d_input),
h_init=h_init,
stream=stream,
)
stream.record(end_event)
end_event.sync()
result = float(d_output[0])
times.append(end_event - start_event)
return result, float(np.mean(times))
def main() -> bool:
"""Main function demonstrating parallel reduction."""
print("=" * 70)
print("Parallel Reduction - Efficient GPU Array Summation")
print("=" * 70)
device = Device(0)
device.set_current()
stream = device.create_stream()
cp_stream = cp.cuda.ExternalStream(int(stream.handle))
print()
print_gpu_info(device)
array_size = 1 << 20 # 1M elements
h_input = np.random.rand(array_size).astype(np.float32)
expected_sum = float(np.sum(h_input))
print(f"\nArray size: {array_size:,} elements ({array_size * 4 / 1e6:.1f} MB)")
print(f"Expected sum: {expected_sum:.6f}")
print("\nCompiling custom CUDA kernel...")
kernel = compile_kernel(device)
try:
with cp_stream:
d_input = cp.asarray(h_input)
# ======================================================================
# Part 1: Custom Kernel
# ======================================================================
print("\n" + "=" * 70)
print("PART 1: Custom Kernel (Educational)")
print("=" * 70)
result, time_ms = benchmark_custom(stream, kernel, d_input)
print(f"\nReduction tree kernel: {result:>14.2f}")
print(f"Expected: {expected_sum:>14.2f}")
print(f"Time: {time_ms:>14.3f} ms")
# ======================================================================
# Part 2: cuda.compute (Production)
# ======================================================================
print("\n" + "=" * 70)
print("PART 2: cuda.compute.reduce_into() (Production)")
print("=" * 70)
result_cc, time_cc = benchmark_cuda_compute(stream, d_input)
print(f"\ncuda.compute result: {result_cc:>14.2f}")
print(f"Expected: {expected_sum:>14.2f}")
print(f"Time: {time_cc:>14.3f} ms")
# Verify both results using principled rtol/atol
with cp_stream:
d_expected = cp.array([expected_sum], dtype=cp.float32)
custom_ok = verify_array_result(
cp.array([result], dtype=cp.float32),
d_expected,
rtol=1e-5,
atol=1e-8,
verbose=False,
)
compute_ok = verify_array_result(
cp.array([result_cc], dtype=cp.float32),
d_expected,
rtol=1e-5,
atol=1e-8,
verbose=False,
)
if custom_ok and compute_ok:
print("\nTest PASSED!")
return True
else:
print("\nTest FAILED - Error too large!")
return False
finally:
stream.close()
if __name__ == "__main__":
sys.exit(0 if main() else 1)

7
python/2_CoreConcepts/parallelReduction/requirements.txt Normal file
View File

@@ -0,0 +1,7 @@
# Parallel Reduction Sample Requirements
cuda-python>=13.0.0
cuda-core>=0.6.0
cuda-cccl>=1.0.0
cupy-cuda13x>=13.0.0
numpy>=2.3.2

83
python/2_CoreConcepts/prefixSum/README.md Normal file
View File

@@ -0,0 +1,83 @@
# Prefix Sum (Scan)
Demonstrates parallel prefix sum (scan) algorithms using cuda.compute with cuda.core stream management.
## Overview
- Inclusive scan: `output[i] = [init_value] + input[0] + input[1] + ... + input[i]`
- Exclusive scan: `output[i] = init_value + input[0] + input[1] + ... + input[i-1]`
- Uses cuda.compute APIs for optimized CUB-based implementations
- Uses cuda.core APIs for device and stream management
- Demonstrates CuPy integration via `ExternalStream`
## Requirements
### Hardware
- NVIDIA GPU with CUDA support
### Software
- CUDA Toolkit 13.0+
- Python 3.10+
- `cuda-python` (13.0.0+)
- `cuda-core` (>=0.6.0)
- `cuda-cccl` (1.0.0+)
- `cupy-cuda13x` (13.0.0+)
- `numpy` (>=2.3.2)
## Usage
```bash
# Create and activate virtual environment
python -m venv venv
source venv/bin/activate # Linux/macOS
# venv\Scripts\activate # Windows
# Install dependencies
pip install -r requirements.txt
# Run sample
python prefixSum.py
```
## Key Concepts
| Scan Type | Formula | First Element |
|-----------|---------|---------------|
| Inclusive | `output[i] = [init_value] + Σ input[0..i]` | `[init_value] + input[0]` |
| Exclusive | `output[i] = init_value + Σ input[0..i-1]` | `init_value` (typically `0`, the identity for sum) |
### Stream Management
This sample demonstrates proper stream usage across libraries:
```python
# Create stream with cuda.core
stream = device.create_stream()
# Wrap for CuPy compatibility (requires int handle)
cp_stream = cp.cuda.ExternalStream(int(stream.handle))
# Use with CuPy operations
with cp_stream:
d_input = cp.asarray(data)
d_output = cp.empty_like(d_input)
# Pass to cuda.compute
inclusive_scan(
d_in=d_input,
d_out=d_output,
op=OpKind.PLUS,
init_value=None,
num_items=len(d_input),
stream=stream,
)
```
## Applications
- Stream compaction
- Radix sort
- Histogram computation
- Polynomial evaluation

199
python/2_CoreConcepts/prefixSum/prefixSum.py Normal file
View File

@@ -0,0 +1,199 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Prefix Sum (Scan)
Demonstrates parallel prefix sum algorithms using cuda.compute:
- Inclusive scan: output[i] = [init_value] + input[0] + ... + input[i]
- Exclusive scan: output[i] = init_value + input[0] + ... + input[i-1]
Uses cuda.compute APIs for optimized CUB-based scan operations.
Uses cuda.core APIs for device and stream management.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
try:
import cupy as cp
import numpy as np
from cuda.compute import OpKind, exclusive_scan, inclusive_scan
from cuda.core import Device, EventOptions
from cuda_samples_utils import print_gpu_info, verify_array_result
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install from requirements.txt:")
print(" pip install -r requirements.txt")
sys.exit(1)
def main() -> bool:
"""Run prefix sum sample. Returns True if all tests passed."""
print("=" * 60)
print("Prefix Sum (Scan) - Using cuda.compute")
print("=" * 60)
device = Device(0)
device.set_current()
stream = device.create_stream()
cp_stream = cp.cuda.ExternalStream(int(stream.handle))
ok = True
try:
print()
print_gpu_info(device)
h_input = np.array([3, 1, 4, 1, 5, 9, 2, 6], dtype=np.int32)
init_value = np.array([0], dtype=np.int32)
# =========================================================================
# Inclusive Scan
# =========================================================================
print("\n" + "-" * 60)
print("INCLUSIVE SCAN")
print("-" * 60)
print(
"Formula: output[i] = [init_value] + input[0] + input[1] + ... + input[i]"
)
with cp_stream:
d_input = cp.asarray(h_input)
d_output = cp.empty_like(d_input)
print(f"\nInput: {h_input.tolist()}")
inclusive_scan(
d_in=d_input,
d_out=d_output,
op=OpKind.PLUS,
init_value=None,
num_items=len(h_input),
stream=stream,
)
stream.sync()
print(f"Output: {cp.asnumpy(d_output).tolist()}")
with cp_stream:
expected = cp.asarray(np.cumsum(h_input))
ok &= verify_array_result(d_output, expected, rtol=0, atol=0)
# =========================================================================
# Exclusive Scan
# =========================================================================
print("\n" + "-" * 60)
print("EXCLUSIVE SCAN")
print("-" * 60)
print("Formula: output[i] = init_value + input[0] + ... + input[i-1]")
with cp_stream:
d_output = cp.empty_like(d_input)
print(f"\nInput: {h_input.tolist()}")
exclusive_scan(
d_in=d_input,
d_out=d_output,
op=OpKind.PLUS,
init_value=init_value,
num_items=len(h_input),
stream=stream,
)
stream.sync()
print(f"Output: {cp.asnumpy(d_output).tolist()}")
with cp_stream:
expected = cp.asarray(np.concatenate([init_value, np.cumsum(h_input)[:-1]]))
ok &= verify_array_result(d_output, expected, rtol=0, atol=0)
# =========================================================================
# Large Array Performance
# =========================================================================
print("\n" + "-" * 60)
print("PERFORMANCE (10M elements)")
print("-" * 60)
N = 10_000_000
with cp_stream:
d_large_in = cp.ones(N, dtype=np.int32)
d_large_out = cp.empty_like(d_large_in)
inclusive_scan(
d_in=d_large_in,
d_out=d_large_out,
op=OpKind.PLUS,
init_value=None,
num_items=N,
stream=stream,
)
stream.sync()
event_opts = EventOptions(enable_timing=True)
start_event = device.create_event(options=event_opts)
end_event = device.create_event(options=event_opts)
num_iterations = 10
stream.record(start_event)
for _ in range(num_iterations):
inclusive_scan(
d_in=d_large_in,
d_out=d_large_out,
op=OpKind.PLUS,
init_value=None,
num_items=N,
stream=stream,
)
stream.record(end_event)
end_event.sync()
elapsed_ms = (end_event - start_event) / num_iterations
print(f"Inclusive scan: {elapsed_ms:.3f} ms")
print(f"Throughput: {N / elapsed_ms / 1e6:.1f} M elements/ms")
# =========================================================================
# Summary
# =========================================================================
print("\n" + "=" * 60)
print("KEY CONCEPTS")
print("=" * 60)
print("• Inclusive: output[i] includes input[i]")
print("• Exclusive: output[i] excludes input[i], starts with init_value")
print("• cuda.compute provides CUB-based optimized implementations")
print("• cuda.core Stream integrates with CuPy via ExternalStream")
print("• Applications: stream compaction, radix sort, histograms")
print("=" * 60)
return ok
finally:
cp.cuda.Stream.null.use()
stream.close()
if __name__ == "__main__":
success = main()
if not success:
sys.exit(1)

8
python/2_CoreConcepts/prefixSum/requirements.txt Normal file
View File

@@ -0,0 +1,8 @@
# Prefix Sum Sample Requirements
# Requires Python 3.10+, CUDA Toolkit 13.0+
cuda-python>=13.0.0
cuda-core>=0.6.0
cuda-cccl>=1.0.0
cupy-cuda13x>=13.0.0
numpy>=2.3.2

206
python/2_CoreConcepts/processCheckpoint/README.md Normal file
View File

@@ -0,0 +1,206 @@
# processCheckpoint (Python)
## Description
This sample demonstrates how to use the **CUDA process checkpoint API**
via `cuda.core.checkpoint.Process` to suspend, capture, and restore the
CUDA state of a running Linux process.
CUDA process checkpointing is the driver-level primitive that powers
CRIU + `cuda-checkpoint` integration.
The sample:
1. Allocates a GPU buffer and fills it with a deterministic pattern
via a small kernel.
2. Reads the buffer back to host and computes a SHA-256 hash.
3. Runs the full checkpoint lifecycle on its own process:
`lock → checkpoint → restore → unlock`.
4. Reads the buffer back again and verifies that the hash is
unchanged, proving that GPU memory contents survived the round
trip.
The sample prints the CUDA process state after each step so the
full state machine is visible:
```
lock() checkpoint() restore() unlock()
running ---------> locked ------------> checkpointed -----------> locked ---------> running
```
## What You'll Learn
- Creating a `cuda.core.checkpoint.Process` for the current process
by PID and observing its `.state` transitions.
- Running the full `lock → checkpoint → restore → unlock` cycle with
a lock timeout.
- The fact that `restore()` leaves the process in the `locked` state;
you must still call `unlock()` to return to `running`.
- Verifying that GPU memory is preserved across the checkpoint
round-trip by comparing SHA-256 hashes of the buffer before and
after.
- The rough cost of each step (checkpoint and restore dominate and
scale with the device-memory footprint being captured).
## Key Libraries
- [`cuda.core`](https://nvidia.github.io/cuda-python/cuda-core/latest/)
- device management, memory allocation, kernel compilation and
launch, and the `checkpoint.Process` wrapper.
- [`cuda.bindings`](https://nvidia.github.io/cuda-python/cuda-bindings/latest/)
- used directly for a pageable `cuMemcpyDtoH`.
## Key APIs
### From `cuda.core.checkpoint`
- `checkpoint.Process(pid)` - create a handle to a CUDA process by
PID. Accepts `os.getpid()` for the self-checkpoint case shown
here.
- `Process.state` - one of `"running"`, `"locked"`, `"checkpointed"`,
or `"failed"`.
- `Process.lock(timeout_ms=…)` - block further CUDA API calls on the
process; completes already-submitted work. Always pass a non-zero
timeout to avoid deadlocks.
- `Process.checkpoint()` - copy device memory to host-side driver
allocations and release GPU resources. Process state becomes
`checkpointed`.
- `Process.restore(gpu_mapping=None)` - re-acquire GPU resources and
copy memory back to device. Leaves the process in the `locked`
state.
- `Process.unlock()` - return the process to `running`.
- `Process.restore_thread_id` - thread ID that `restore()` must be
called from in the target process (not used in the self-checkpoint
case here).
### From `cuda.core`
- `Device.set_current()` / `Device.memory_resource.allocate(...)` /
`Stream`, `LaunchConfig`, `Program`, `launch` - standard device,
compile, and launch primitives used to produce the buffer
contents.
### From `cuda.bindings.driver`
- `cuMemcpyDtoH(host_ptr, device_handle, nbytes)` - synchronous D2H
copy into a pageable host buffer.
## Requirements
### Hardware
- Any NVIDIA GPU supported by CUDA process checkpointing. CUDA
checkpointing is currently limited to x86-64 Linux.
### Software
- Linux (the CUDA checkpoint API is Linux-only).
- NVIDIA driver with CUDA process checkpoint support.
- CUDA Toolkit 13.0 or newer.
- Python 3.10 or newer.
- `cuda-core >= 1.0.0`.
## Installation
Install the required packages from `requirements.txt`:
```bash
cd /path/to/cuda-samples/python/2_CoreConcepts/processCheckpoint
pip install -r requirements.txt
```
## How to Run
### Basic usage
```bash
python processCheckpoint.py
```
### Larger GPU footprint to see checkpoint time scale
```bash
python processCheckpoint.py --buffer-mib 512
```
### Use a specific GPU
```bash
python processCheckpoint.py --device 1
```
### All options
```
--device CUDA device ID (default: 0)
--buffer-mib GPU buffer size in MiB (default: 16)
--lock-timeout-ms Timeout passed to Process.lock in ms (default: 5000)
```
## Expected Output
On an RTX 4090 with a 16 MiB buffer:
```
[Process Checkpoint Sample using CUDA Core API]
PID: 748330
Device: NVIDIA GeForce RTX 4090
Compute Capability: sm_89
Buffer size: 16 MiB
Lock timeout: 5000 ms
Compiling kernel ...
Writing deterministic pattern to GPU buffer ...
Buffer hash (before): b045f7975dc23352
Running checkpoint lifecycle on self ...
step duration (ms) state after
--------------------------------------------------
initial - running
lock 0.578 locked
checkpoint 268.369 checkpointed
restore 235.024 locked
unlock 1.648 running
--------------------------------------------------
total 505.618
Buffer hash (before): b045f7975dc23352
Buffer hash (after): b045f7975dc23352
PASS: GPU buffer contents survived checkpoint/restore.
Done
```
**What to look for:**
- The **four state transitions** are all observable: `running →
locked → checkpointed → locked → running`. Note that `restore()`
leaves the process in `locked`, not `running`.
- The **checkpoint and restore steps dominate** the wall-clock time
(hundreds of ms even for a small buffer) - they copy GPU memory to
and from driver-managed host allocations. Increasing
`--buffer-mib` visibly increases the checkpoint time.
- The `lock` and `unlock` steps are essentially free (sub-ms) - they
just flip the process state.
- The SHA-256 **hashes before and after match**, proving the GPU
memory contents survived the round trip.
Exact timings vary with GPU model, driver version, system load, and
the size of the device memory footprint being captured.
## Files
- `processCheckpoint.py` - Python implementation using `cuda.core.checkpoint`
- `README.md` - This file
- `requirements.txt` - Sample dependencies
## See Also
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [`NVIDIA/cuda-checkpoint`](https://github.com/NVIDIA/cuda-checkpoint)
- the CUDA checkpoint/restore utility, the CRIU plugin, and C
reference programs (`r570-features.c`, `r580-migration-api.c`).
- [Checkpointing CUDA Applications with CRIU](https://developer.nvidia.com/blog/checkpointing-cuda-applications-with-criu/)
- NVIDIA technical blog post on the broader CRIU workflow.

248
python/2_CoreConcepts/processCheckpoint/processCheckpoint.py Normal file
View File

@@ -0,0 +1,248 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS `AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Process Checkpointing Sample using CUDA Core API.
The sample allocates a GPU buffer, fills it with a deterministic
pattern via a kernel, hashes the contents, runs the full
lock/checkpoint/restore/unlock cycle on its own PID, and re-hashes
the buffer afterwards to verify that the GPU memory contents
survived the round trip.
"""
import argparse
import hashlib
import os
import sys
import time
from dataclasses import dataclass
from typing import List
import numpy as np
from cuda.bindings import driver as cudrv
from cuda.core import (
Device,
LaunchConfig,
Program,
ProgramOptions,
checkpoint,
launch,
)
# Small fill kernel: deterministic, non-trivial pattern so the before/after
# hashes would disagree on any bit flip.
KERNEL_SRC = r"""
extern "C" __global__ void fill_pattern(float *out, unsigned long long n)
{
unsigned long long i = (unsigned long long)blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) {
float v = (float)(i & 0xFFFFu) * 1e-3f + 1.0f;
float u = (float)((i >> 16) & 0xFFFFu) * 1e-4f + 0.5f;
// A handful of dependent ops per element. Deterministic given i.
for (int k = 0; k < 8; ++k) {
v = v * 1.000001f + u;
u = u * 0.999999f + v * 1e-6f;
}
out[i] = v + u;
}
}
"""
@dataclass
class StepTiming:
label: str
duration_ms: float
state_after: str
def _cu_check(result) -> None:
err = result[0]
if int(err) != 0:
raise RuntimeError(f"CUDA driver call failed: {err}")
def compile_fill_kernel(device: Device):
options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
program = Program(KERNEL_SRC, code_type="c++", options=options)
module = program.compile("cubin", name_expressions=("fill_pattern",))
return module.get_kernel("fill_pattern")
def hash_device_buffer(device_buffer, host: np.ndarray) -> str:
_cu_check(
cudrv.cuMemcpyDtoH(
host.ctypes.data,
device_buffer.handle,
host.nbytes,
)
)
return hashlib.sha256(host.tobytes()).hexdigest()[:16]
def _time_call(fn, *args, **kwargs) -> float:
t0 = time.monotonic()
fn(*args, **kwargs)
return (time.monotonic() - t0) * 1000.0
def run_lifecycle(proc: checkpoint.Process, lock_timeout_ms: int) -> List[StepTiming]:
"""
Drive the full `lock -> checkpoint -> restore -> unlock` cycle on
`proc` and return per-step timings with the state observed after
each step.
Note on state after `restore()`: the driver leaves the process in
the `locked` state. You must still call `unlock()` to return to
`running`.
"""
timings: List[StepTiming] = [StepTiming("initial", 0.0, proc.state)]
ms = _time_call(proc.lock, timeout_ms=lock_timeout_ms)
timings.append(StepTiming("lock", ms, proc.state))
ms = _time_call(proc.checkpoint)
timings.append(StepTiming("checkpoint", ms, proc.state))
ms = _time_call(proc.restore)
timings.append(StepTiming("restore", ms, proc.state))
ms = _time_call(proc.unlock)
timings.append(StepTiming("unlock", ms, proc.state))
return timings
def print_timings(timings: List[StepTiming]) -> None:
print()
header = f"{'step':<14}{'duration (ms)':>18}{'state after':>18}"
print(header)
print("-" * len(header))
total = 0.0
for t in timings:
if t.label == "initial":
dur = "-"
else:
dur = f"{t.duration_ms:.3f}"
total += t.duration_ms
print(f"{t.label:<14}{dur:>18}{t.state_after:>18}")
print("-" * len(header))
print(f"{'total':<14}{total:>18.3f}{'':>18}")
def main():
parser = argparse.ArgumentParser(
description="CUDA process checkpoint sample using cuda.core",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument(
"--device", type=int, default=0, help="CUDA device ID (default: 0)"
)
parser.add_argument(
"--buffer-mib",
type=int,
default=16,
help="GPU buffer size in MiB (default: 16)",
)
parser.add_argument(
"--lock-timeout-ms",
type=int,
default=5000,
help="Timeout passed to Process.lock in ms (default: 5000)",
)
args = parser.parse_args()
if sys.platform != "linux":
print("Error: CUDA process checkpointing is Linux-only.")
return 1
if args.buffer_mib <= 0:
print("Error: --buffer-mib must be positive")
return 1
print("[Process Checkpoint Sample using CUDA Core API]")
print(f"PID: {os.getpid()}")
device = Device(args.device)
device.set_current()
print(f"Device: {device.name}")
print(f"Compute Capability: sm_{device.arch}")
print(f"Buffer size: {args.buffer_mib} MiB")
print(f"Lock timeout: {args.lock_timeout_ms} ms")
print()
print("Compiling kernel ...")
fill_kernel = compile_fill_kernel(device)
buffer_bytes = args.buffer_mib * 1024 * 1024
n_elements = buffer_bytes // 4 # float32
stream = device.create_stream()
device_buffer = device.memory_resource.allocate(buffer_bytes, stream=stream)
try:
print("Writing deterministic pattern to GPU buffer ...")
block = 256
grid = (n_elements + block - 1) // block
cfg = LaunchConfig(grid=grid, block=block)
launch(stream, cfg, fill_kernel, device_buffer, np.uint64(n_elements))
stream.sync()
host = np.empty(n_elements, dtype=np.float32)
hash_before = hash_device_buffer(device_buffer, host)
print(f"Buffer hash (before): {hash_before}")
print()
print("Running checkpoint lifecycle on self ...")
proc = checkpoint.Process(os.getpid())
timings = run_lifecycle(proc, args.lock_timeout_ms)
print_timings(timings)
hash_after = hash_device_buffer(device_buffer, host)
print()
print(f"Buffer hash (before): {hash_before}")
print(f"Buffer hash (after): {hash_after}")
if hash_before != hash_after:
print()
print("FAIL: GPU buffer contents changed across checkpoint/restore.")
return 1
print()
print("PASS: GPU buffer contents survived checkpoint/restore.")
finally:
device_buffer.close(stream)
print()
print("Done")
return 0
if __name__ == "__main__":
sys.exit(main())

3
python/2_CoreConcepts/processCheckpoint/requirements.txt Normal file
View File

@@ -0,0 +1,3 @@
cuda-python>=13.0.0
cuda-core>=0.7.0
numpy>=2.3.2

137
python/2_CoreConcepts/reduction/README.md Normal file
View File

@@ -0,0 +1,137 @@
# Sample: Fast Array Sum using Shared Memory (Python)
## Description
Two-stage parallel reduction: each GPU block sums its chunk in **shared memory** (tree reduction, two elements per thread), writes one partial sum per block; the host combines partial sums for the final result.
**Stack:** `cuda-core` for `Device`, stream, events, `Program` / `launch()`. **CuPy** allocates device memory and copies; `launch()` takes device pointers as `ndarray.data.ptr` (Python `int`). Copies run on the same CUDA stream as the kernel via `cp.cuda.Stream.from_external(stream)` (cuda.core `Stream` implements the CUDA stream protocol) and `with cp_stream:`.
## What you will learn
- Shared-memory block reduction and sequential-addressing tree reduction
- `LaunchConfig` with dynamic shared memory and `launch()` with pointer arguments
- Aligning CuPy transfers with a `cuda.core` stream (`Stream.from_external`)
- GPU timing with `EventOptions` / `device.create_event()`
## Key libraries
| Library | Role |
|------------|------|
| `cuda-core`| Device, stream, events, compile, launch |
| `cupy` | `cp.empty`, `cp.asarray`, `cp.asnumpy`, `Stream.from_external` |
| `numpy` | Host data and CPU reference sum |
## Key APIs (quick reference)
- **cuda.core:** `Device`, `create_stream`, `Program` / `ProgramOptions`, `LaunchConfig`, `launch`, `EventOptions`, `create_event`
- **CuPy:** `cp.empty`, `cp.asarray`, `cp.cuda.Stream.from_external(stream)`, `with cp_stream:`, `cp.asnumpy`
## Requirements
- NVIDIA GPU, CUDA-capable driver; **CUDA Toolkit 13+** (for toolchain alignment with `cuda-core`)
- **Python 3.10+**
```bash
pip install -r requirements.txt
```
## How to run
```bash
python reduction.py
```
Defaults: 2²⁴ elements, 256 threads/block, `float`, 100 benchmark iterations.
**Change data type** (selects `blockReduceKernel_int` / `_float` / `_double`):
```bash
python reduction.py --type float # default; 32-bit float
python reduction.py --type double # 64-bit float
python reduction.py --type int # 32-bit integer (exact equality check)
```
Combine with other flags as needed, e.g. `python reduction.py --type int --n 1048576`.
Other main flags: `--n`, `--threads`, `--iterations`. Full list: `python reduction.py --help`.
## Output
Example run (`python reduction.py`, defaults) on **Tesla T10**, compute capability **7.5**:
```
======================================================================
Fast Array Sum using Shared Memory - Two-Stage Reduction
======================================================================
Demonstrates: Efficient parallel reduction using shared memory
Device Information:
Name: Tesla T10
Compute Capability: sm_7.5
Configuration:
Array size: 16,777,216 elements
Data type: float
Memory: 64.00 MB
Threads per block: 256
Two-Stage Reduction Strategy:
Stage 1: GPU block reduction
- Number of blocks: 32768
- Elements per block: 512
- Output: 32768 partial sums
Stage 2: CPU final reduction
- Combine 32768 partial sums → 1 final result
Compiling CUDA kernel...
Kernel 'blockReduceKernel_float' compiled successfully
> Generating random input data...
> Computing reference result on CPU...
CPU time: 2.428208 seconds
> Warming up GPU...
Warm-up completed
> Benchmarking Stage 1 (GPU block reduction)...
Running 100 iterations...
> Running Stage 2 (CPU final reduction)...
======================================================================
Performance Results
======================================================================
Stage 1 (GPU block reduction):
Average time: 0.338404 ms
Throughput: 198.31 GB/s
Stage 2 (CPU final reduction):
Time: 0.078073 ms
(32768 partial sums)
Total time: 0.416477 ms
Speedup vs CPU: 5830.35x
> Validating results...
GPU result: 2147639808.00000000
CPU result: 2147639929.62027407
Test PASSED
======================================================================
Summary
======================================================================
Key optimizations:
- Load 2 elements per thread: 8,388,608 global reads (50% savings)
- Shared memory for reduction: ~10-20x faster than global memory
- Parallel block outputs: 32768 independent writes
Result: 198.31 GB/s throughput
======================================================================
Two-Stage Reduction completed successfully!
======================================================================
```
## Files
`reduction.py` · `requirements.txt` · `README.md`

485
python/2_CoreConcepts/reduction/reduction.py Normal file
View File

@@ -0,0 +1,485 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# distribution and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Fast Array Sum using Shared Memory - Two-Stage Reduction
Demonstrates efficient parallel reduction using shared memory and
two-stage approach to avoid atomic operation bottlenecks.
Key Features:
- Block-level reduction using shared memory
- Each thread loads 2 elements to reduce global memory traffic
- Sequential addressing tree reduction pattern
- No atomic operations - eliminates serialization bottleneck
- Device memory via CuPy; ``launch()`` takes pointers as ``ndarray.data.ptr``
- CuPy uses ``cp.cuda.Stream.from_external(stream)``.
"""
import argparse
import os
import sys
import time
try:
import cupy as cp
import numpy as np
from cuda.core import (
Device,
EventOptions,
LaunchConfig,
Program,
ProgramOptions,
launch,
)
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install dependencies:")
print(" pip install -r requirements.txt")
sys.exit(1)
# Import utilities
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "Utilities"))
from cuda_samples_utils import verify_array_result # noqa: E402
# Two-stage block reduction kernel
REDUCTION_KERNEL = """
/*
* Block-level reduction kernel using shared memory
*
* Strategy:
* - Each block processes blockSize * 2 elements
* - Uses shared memory for fast intra-block reduction
* - Outputs one partial sum per block (no atomics)
*
* Key optimizations:
* - Load 2 elements per thread (reduces global memory traffic by 50%)
* - Tree reduction with sequential addressing (avoids divergence)
* - Shared memory instead of atomic operations (eliminates bottleneck)
*
* Note: This sample provides separate implementations for each data type
* for clarity. Production code typically uses templates with SharedMemory<T>
* or reinterpret_cast to avoid duplication. See NVIDIA reduction guide for
* template-based approaches.
*/
extern "C" __global__ void blockReduceKernel_int(
const int *__restrict__ input,
int *__restrict__ blockSums,
unsigned int n)
{
extern __shared__ int sdata_int[];
unsigned int tid = threadIdx.x;
unsigned int blockSize = blockDim.x;
unsigned int gid = blockIdx.x * (blockSize * 2) + tid;
// Load 2 elements per thread
int sum = 0;
if (gid < n) sum += input[gid];
if (gid + blockSize < n) sum += input[gid + blockSize];
sdata_int[tid] = sum;
__syncthreads();
// Tree reduction with sequential addressing
for (unsigned int s = blockSize / 2; s > 0; s >>= 1) {
if (tid < s) {
sdata_int[tid] += sdata_int[tid + s];
}
__syncthreads();
}
// Write block result
if (tid == 0) {
blockSums[blockIdx.x] = sdata_int[0];
}
}
extern "C" __global__ void blockReduceKernel_float(
const float *__restrict__ input,
float *__restrict__ blockSums,
unsigned int n)
{
extern __shared__ float sdata_float[];
unsigned int tid = threadIdx.x;
unsigned int blockSize = blockDim.x;
unsigned int gid = blockIdx.x * (blockSize * 2) + tid;
// Load 2 elements per thread
float sum = 0.0f;
if (gid < n) sum += input[gid];
if (gid + blockSize < n) sum += input[gid + blockSize];
sdata_float[tid] = sum;
__syncthreads();
// Tree reduction with sequential addressing
for (unsigned int s = blockSize / 2; s > 0; s >>= 1) {
if (tid < s) {
sdata_float[tid] += sdata_float[tid + s];
}
__syncthreads();
}
// Write block result
if (tid == 0) {
blockSums[blockIdx.x] = sdata_float[0];
}
}
extern "C" __global__ void blockReduceKernel_double(
const double *__restrict__ input,
double *__restrict__ blockSums,
unsigned int n)
{
extern __shared__ double sdata_double[];
unsigned int tid = threadIdx.x;
unsigned int blockSize = blockDim.x;
unsigned int gid = blockIdx.x * (blockSize * 2) + tid;
// Load 2 elements per thread
double sum = 0.0;
if (gid < n) sum += input[gid];
if (gid + blockSize < n) sum += input[gid + blockSize];
sdata_double[tid] = sum;
__syncthreads();
// Tree reduction with sequential addressing
for (unsigned int s = blockSize / 2; s > 0; s >>= 1) {
if (tid < s) {
sdata_double[tid] += sdata_double[tid + s];
}
__syncthreads();
}
// Write block result
if (tid == 0) {
blockSums[blockIdx.x] = sdata_double[0];
}
}
"""
def reduce_cpu(data):
"""Compute sum using Kahan summation for numerical accuracy."""
if len(data) == 0:
return 0
sum_val = float(data[0])
c = 0.0
for i in range(1, len(data)):
y = float(data[i]) - c
t = sum_val + y
c = (t - sum_val) - y
sum_val = t
return sum_val
def _validate_threads_per_block(threads_per_block):
if threads_per_block <= 0 or threads_per_block > 1024:
return "threads per block must be between 1 and 1024"
if (threads_per_block & (threads_per_block - 1)) != 0:
return (
"threads per block must be a power of 2 "
"(required by the shared-memory tree reduction kernel)"
)
return None
def run(
num_elements=1 << 24, threads_per_block=256, test_iterations=100, datatype="float"
):
"""Run two-stage reduction benchmark."""
print("\n" + "=" * 70)
print("Fast Array Sum using Shared Memory - Two-Stage Reduction")
print("=" * 70)
print("\nDemonstrates: Efficient parallel reduction using shared memory")
# Map datatype
dtype_map = {"int": np.int32, "float": np.float32, "double": np.float64}
if datatype not in dtype_map:
print(f"Unknown datatype '{datatype}', using 'float'")
datatype = "float"
dtype = dtype_map[datatype]
itemsize = np.dtype(dtype).itemsize
# Initialize device
device = Device()
device.set_current()
major, minor = device.compute_capability
print("\nDevice Information:")
print(f" Name: {device.name}")
print(f" Compute Capability: sm_{major}.{minor}")
# Configuration
print("\nConfiguration:")
print(f" Array size: {num_elements:,} elements")
print(f" Data type: {datatype}")
print(f" Memory: {num_elements * itemsize / (1024**2):.2f} MB")
print(f" Threads per block: {threads_per_block}")
# Calculate number of blocks
# Each block processes threads_per_block * 2 elements
num_blocks = (num_elements + threads_per_block * 2 - 1) // (threads_per_block * 2)
print("\nTwo-Stage Reduction Strategy:")
print(" Stage 1: GPU block reduction")
print(f" - Number of blocks: {num_blocks}")
print(f" - Elements per block: {threads_per_block * 2}")
print(f" - Output: {num_blocks} partial sums")
print(" Stage 2: CPU final reduction")
print(f" - Combine {num_blocks} partial sums → 1 final result")
# Compile kernel
print("\nCompiling CUDA kernel...")
program_options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
prog = Program(REDUCTION_KERNEL, code_type="c++", options=program_options)
mod = prog.compile("cubin")
kernel_name = f"blockReduceKernel_{datatype}"
kernel = mod.get_kernel(kernel_name)
print(f" Kernel '{kernel_name}' compiled successfully")
# Generate input data
print("\n> Generating random input data...")
rng = np.random.default_rng(42)
if datatype == "int":
h_input = rng.integers(0, 256, size=num_elements, dtype=dtype)
else:
h_input = (rng.random(num_elements) * 256).astype(dtype)
# cuda.core stream for launch/events; CuPy copies use the same stream via
# Stream.from_external.
stream = device.create_stream()
cp_stream = cp.cuda.Stream.from_external(stream)
try:
d_blockSums = cp.empty(num_blocks, dtype=dtype)
with cp_stream:
d_input = cp.asarray(h_input, dtype=dtype)
stream.sync()
# Compute CPU reference
print("> Computing reference result on CPU...")
cpu_start = time.perf_counter()
cpu_result = reduce_cpu(h_input)
cpu_time = time.perf_counter() - cpu_start
print(f" CPU time: {cpu_time:.6f} seconds")
# Configure launch
shared_mem_bytes = threads_per_block * itemsize
config = LaunchConfig(
grid=num_blocks, block=threads_per_block, shmem_size=shared_mem_bytes
)
# Warm-up
print("\n> Warming up GPU...")
launch(
stream,
config,
kernel,
d_input.data.ptr,
d_blockSums.data.ptr,
np.uint32(num_elements),
)
stream.sync()
print(" Warm-up completed")
# Benchmark Stage 1 (GPU)
print("\n> Benchmarking Stage 1 (GPU block reduction)...")
print(f" Running {test_iterations} iterations...")
# cuda.core event elapsed time (end - start) is in milliseconds (CUDA API).
stage1_times_ms = []
event_options = EventOptions(enable_timing=True)
start_event = stream.device.create_event(options=event_options)
end_event = stream.device.create_event(options=event_options)
for _ in range(test_iterations):
stream.record(start_event)
launch(
stream,
config,
kernel,
d_input.data.ptr,
d_blockSums.data.ptr,
np.uint32(num_elements),
)
stream.record(end_event)
end_event.sync()
stage1_times_ms.append(float(end_event - start_event))
avg_stage1_ms = np.mean(stage1_times_ms)
avg_stage1_s = avg_stage1_ms / 1000.0
# Stage 2 (CPU)
print("\n> Running Stage 2 (CPU final reduction)...")
# Device → Host: after stream sync, partial sums are visible on host.
stream.sync()
with cp_stream:
h_blockSums = cp.asnumpy(d_blockSums)
stage2_start = time.perf_counter()
gpu_result = float(np.sum(h_blockSums))
stage2_time = time.perf_counter() - stage2_start
total_time = avg_stage1_s + stage2_time
# Performance metrics (use seconds for throughput; CPU times are in seconds)
bytes_processed = num_elements * itemsize
throughput = bytes_processed / avg_stage1_s / 1e9
print("\n" + "=" * 70)
print("Performance Results")
print("=" * 70)
print("\nStage 1 (GPU block reduction):")
print(f" Average time: {avg_stage1_ms:.6f} ms")
print(f" Throughput: {throughput:.2f} GB/s")
print("\nStage 2 (CPU final reduction):")
print(f" Time: {stage2_time * 1000:.6f} ms")
print(f" ({num_blocks} partial sums)")
print(f"\nTotal time: {total_time * 1000:.6f} ms")
print(f"Speedup vs CPU: {cpu_time / total_time:.2f}x")
# Validation
print("\n> Validating results...")
if datatype == "int":
print(f" GPU result: {int(gpu_result):,}")
print(f" CPU result: {int(cpu_result):,}")
rtol, atol = 0.0, 0.0
else:
precision = 8 if datatype == "float" else 12
print(f" GPU result: {gpu_result:.{precision}f}")
print(f" CPU result: {cpu_result:.{precision}f}")
if datatype == "float":
rtol, atol = 1e-5, 1e-8 * num_elements
else: # double
rtol, atol = 1e-8, 1e-12 * num_elements
success = verify_array_result(
np.array([gpu_result]),
np.array([cpu_result]),
rtol=rtol,
atol=atol,
verbose=True,
)
# Summary
print("\n" + "=" * 70)
print("Summary")
print("=" * 70)
print("Key optimizations:")
half_reads = num_elements // 2
print(
" - Load 2 elements per thread: "
f"{half_reads:,} global reads (50% savings)"
)
print(" - Shared memory for reduction: ~10-20x faster than global memory")
print(f" - Parallel block outputs: {num_blocks} independent writes")
print(f"Result: {throughput:.2f} GB/s throughput")
print("=" * 70)
if success:
print("Two-Stage Reduction completed successfully!")
else:
print("Two-Stage Reduction FAILED!")
print("=" * 70 + "\n")
return 0 if success else 1
finally:
stream.close()
def main():
"""Main entry point with argument parsing."""
parser = argparse.ArgumentParser(
description="Two-Stage Reduction with Shared Memory",
epilog="See README.md for usage examples and detailed documentation.",
)
parser.add_argument(
"--n",
type=int,
default=1 << 24,
help="Number of elements to reduce (default: 16777216 = 2^24)",
)
parser.add_argument(
"--threads",
type=int,
default=256,
help="Threads per block, power of 2 in [1, 1024] (default: 256)",
)
parser.add_argument(
"--type",
type=str,
default="float",
choices=["int", "float", "double"],
help="Data type for reduction (default: float)",
)
parser.add_argument(
"--iterations",
type=int,
default=100,
help="Number of benchmark iterations (default: 100)",
)
args = parser.parse_args()
# Validate arguments
if args.n <= 0:
print("Error: n must be positive")
return 1
err = _validate_threads_per_block(args.threads)
if err:
print(f"Error: {err}")
return 1
try:
exit_code = run(
num_elements=args.n,
threads_per_block=args.threads,
test_iterations=args.iterations,
datatype=args.type,
)
sys.exit(exit_code)
except Exception as e:
print(f"\nError: {e}")
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == "__main__":
main()

8
python/2_CoreConcepts/reduction/requirements.txt Normal file
View File

@@ -0,0 +1,8 @@
# Python CUDA Reduction Sample Requirements
# Install with: pip install -r requirements.txt
numpy>=2.3.2
cuda-core>=0.6.0
cuda-python>=13.0.0
# Use cupy-cuda13x>=14.0.0 for cp.cuda.Stream.from_external(stream)
cupy-cuda13x>=14.0.0

129
python/2_CoreConcepts/reductionMultiBlockCG/README.md Normal file
View File

@@ -0,0 +1,129 @@
# Sample: Single-Pass Multi-Block Reduction with Cooperative Groups (Python)
## Description
Single-kernel, two-stage reduction using **Cooperative Groups** and `grid.sync()` so all blocks synchronize inside one launch—no second kernel or CPU stage for the reduction tree.
**Stack:** `cuda-core` (device, compile, cooperative `launch()`, stream, **CUDA events** for GPU timing). **CuPy** for H↔D copies on the same stream (`Stream.from_external(cuda.core_stream)`, `ndarray.data.ptr` to `launch()`). **`try`/`finally`** closes the stream if cooperative launch fails. Requires **compute capability > 6.0** (Pascal+).
## What you will learn
- `cooperative_groups::grid_group` and `grid.sync()` across the grid
- Cooperative `LaunchConfig(..., cooperative_launch=True)` and sizing blocks for residency
- Timing the GPU path with `EventOptions` / `stream.record()` / event elapsed time
## Key libraries
| Library | Role |
|---------|------|
| `cuda-core` | Device, stream, events, `Program` / `ProgramOptions`, cooperative `launch()` |
| `cupy` | `cp.empty`, `cp.asarray`, `cp.asnumpy`, `Stream.from_external` |
| `numpy` | Host data, reference sum, `default_rng` |
## Requirements
- NVIDIA GPU, **Pascal or newer**; **CUDA Toolkit 13+**; **Python 3.10+**
- NVRTC must see **`cooperative_groups.h`** and **CCCL** headers (`cuda/std/*`)
```bash
pip install -r requirements.txt
```
Pick a CuPy wheel that matches your CUDA major version (e.g. `cupy-cuda13x` in `requirements.txt`).
## How to run
**`--cuda-include-dir` is required** (colon-separated list). Typical desktop layout:
```bash
python reductionMultiBlockCG.py \
--cuda-include-dir /usr/local/cuda/include/cccl:/usr/local/cuda/include
```
**Jetson / split include trees:** pass every directory NVRTC needs in one `--cuda-include-dir` argument, e.g.
`/usr/local/cuda/include/cccl:/usr/local/cuda/targets/sbsa-linux/include` (adjust paths to your image). If headers are scattered, you can instead merge them into one tree with symlinks and point `--cuda-include-dir` at that folder.
Defaults: **2²⁵** elements, threads = device max (capped at 1024), auto `--maxblocks`, **100** iterations. Other flags: `--n`, `--threads`, `--maxblocks`, `--iterations`. See **`python reductionMultiBlockCG.py --help`**.
## Output
```
======================================================================
Single-Pass Multi-Block Reduction with Cooperative Groups
======================================================================
Demonstrates: Multi-stage reduction in a single kernel using grid.sync()
Device Information:
Name: NVIDIA Thor
Compute Capability: sm_11.0
Reduction Configuration:
Number of elements: 33,554,432
Data size: 128.00 MB
Compiling CUDA kernel...
Kernel compiled successfully
Launch Configuration:
Threads per block: 1024
Number of blocks: 20
Total threads: 20,480
Shared memory per block: 4096 bytes
Launch mode: Cooperative (grid-wide sync enabled)
> Generating random input data...
> Computing reference result on CPU...
CPU time: 0.008903 seconds
> Warming up GPU...
Warm-up successful
> Running benchmark (100 iterations)...
> Performance Results:
Average GPU time: 0.977166 ms
Throughput: 137.35 GB/s
Speedup vs CPU: 9.11x
> Validating results...
Test PASSED
======================================================================
Summary
======================================================================
Single-kernel two-stage reduction:
Stage 1: 20 blocks → 20 partial sums
grid.sync() ← All blocks synchronize (KEY innovation)
Stage 2: Block 0 → 1 final result
Total: 1 kernel launch, 137.35 GB/s
Comparison:
• Traditional: 2 kernel launches or kernel + CPU
• This sample: 1 kernel with grid.sync() between stages
• Benefit: Eliminates ~5-20μs launch overhead per stage
======================================================================
Single-Pass Multi-Block Reduction completed successfully!
======================================================================
```
## Troubleshooting (short)
- **Cooperative launch not supported / fails:** need sm_60+; reduce `--maxblocks` or `--threads` so all blocks can be resident.
- **Compile errors missing headers:** extend `--cuda-include-dir` with the path that contains CCCL / cooperative groups (see Jetson note above).
- **Low throughput:** often block count vs occupancy; try defaults first, then tune `--threads` / `--maxblocks`.
## Related samples
**blockArraySum** (atomics + grid-stride) → **reduction** (two-stage shared memory) → **this sample** (single kernel + `grid.sync()`).
## Further reading
- [CUDA Cooperative Groups](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cooperative-groups)
- [Reduction whitepaper (PDF)](https://developer.download.nvidia.com/assets/cuda/files/reduction.pdf)
## Files
`reductionMultiBlockCG.py` · `requirements.txt` · `README.md`

472
python/2_CoreConcepts/reductionMultiBlockCG/reductionMultiBlockCG.py Normal file
View File

@@ -0,0 +1,472 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# distribution and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Single-Pass Multi-Block Reduction with Cooperative Groups
Demonstrates single-kernel multi-stage reduction using grid-wide
synchronization. Traditional reduction needs multiple kernel launches,
but with grid.sync() from Cooperative Groups, we can complete all
stages in ONE kernel.
Key Features:
- Grid-wide synchronization with grid.sync()
- Two-stage reduction in a single kernel (no atomic operations)
- Requires compute capability 6.0+ and cooperative launch
- Achieves 400-700 GB/s on modern GPUs
How it differs from other samples:
- blockArraySum.py: Basic thread/block indexing + atomicAdd
- reduction.py: High-performance shared memory, two-kernel approach
- This sample: Single-kernel multi-stage with grid.sync()
Transfers use CuPy on the same CUDA stream as ``launch()`` (``Stream.from_external``),
not ``cuda.bindings.driver`` memcpy. GPU timing uses CUDA events.
"""
import argparse
import os
import sys
import time
try:
import cupy as cp
import numpy as np
from cuda.core import (
Device,
EventOptions,
LaunchConfig,
Program,
ProgramOptions,
launch,
)
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install dependencies:")
print(" pip install -r requirements.txt")
sys.exit(1)
# Import utilities
utilities_path = os.path.join(os.path.dirname(__file__), "..", "..", "Utilities")
sys.path.insert(0, utilities_path)
from cuda_samples_utils import verify_array_result # noqa: E402
def _validate_threads_arg(threads):
if threads is None:
return None
if threads <= 0 or threads > 1024:
return "threads must be between 1 and 1024"
if (threads & (threads - 1)) != 0:
return (
"threads must be a power of 2 "
"(required by the shared-memory tree reduction kernel)"
)
return None
# Single-pass multi-block reduction kernel with grid-wide sync
REDUCTION_KERNEL = """
/*
* Single-Kernel Multi-Stage Reduction using grid.sync()
*
* Strategy:
* Stage 1: Each block reduces its portion → partial sum
* grid.sync() ← KEY: All blocks synchronize
* Stage 2: Block 0 reduces all partial sums → final result
*
* Key feature: grid.sync() enables multi-stage within ONE kernel
*/
#include <cooperative_groups.h>
namespace cg = cooperative_groups;
extern "C" __global__ void reduceSinglePassMultiBlockCG(
const float *__restrict__ g_idata,
float *__restrict__ g_odata,
unsigned int n)
{
cg::thread_block cta = cg::this_thread_block();
cg::grid_group grid = cg::this_grid();
extern __shared__ float sdata[];
unsigned int tid = threadIdx.x;
unsigned int blockSize = blockDim.x;
// Stage 1: Grid-stride loop + block reduction
float sum = 0.0f;
for (unsigned int i = grid.thread_rank(); i < n; i += grid.size()) {
sum += g_idata[i];
}
sdata[tid] = sum;
cg::sync(cta);
// Block reduction (sequential addressing)
for (unsigned int s = blockSize / 2; s > 0; s >>= 1) {
if (tid < s) {
sdata[tid] += sdata[tid + s];
}
cg::sync(cta);
}
if (tid == 0) {
g_odata[blockIdx.x] = sdata[0];
}
// KEY: Grid-wide synchronization (all blocks wait here)
grid.sync();
// Stage 2: Block 0 reduces all partial sums → final result
// Use a stride loop so all gridDim.x partial sums are covered even
// when gridDim.x > blockDim.x.
if (blockIdx.x == 0) {
// mySum stays 0.0f when tid >= gridDim.x (loop never executes),
// implicitly zero-filling sdata for threads beyond the partial-sum count.
float mySum = 0.0f;
for (unsigned int i = tid; i < gridDim.x; i += blockSize) {
mySum += g_odata[i];
}
sdata[tid] = mySum;
cg::sync(cta);
for (unsigned int s = blockSize / 2; s > 0; s >>= 1) {
if (tid < s) {
sdata[tid] += sdata[tid + s];
}
cg::sync(cta);
}
if (tid == 0) {
g_odata[0] = sdata[0];
}
}
}
"""
def get_max_cooperative_blocks(device, kernel, threads_per_block, shared_mem_bytes):
"""
Calculate max blocks for cooperative launch (all must be resident).
This is a conservative estimate that ignores shared memory limits;
for precise tuning, use cudaOccupancyMaxActiveBlocksPerMultiprocessor.
"""
# Get device properties
prop = device.properties
# Calculate maximum blocks per SM
# Note: We use cudaOccupancyMaxActiveBlocksPerMultiprocessor functionality
# For simplicity in Python, we'll use a conservative estimate
num_sms = prop.multiprocessor_count
max_threads_per_sm = prop.max_threads_per_multiprocessor
max_blocks_per_sm = max_threads_per_sm // threads_per_block
# Total blocks = blocks per SM × number of SMs
max_blocks = max_blocks_per_sm * num_sms
# Also respect max_grid_dim_x
max_blocks = min(max_blocks, prop.max_grid_dim_x)
return max_blocks
def run(
num_elements=1 << 25,
max_threads=None,
max_blocks=None,
test_iterations=100,
cuda_include_dir=None,
):
"""Run single-pass multi-block reduction benchmark."""
if cuda_include_dir is None:
raise ValueError("cuda_include_dir is required")
print("\n" + "=" * 70)
print("Single-Pass Multi-Block Reduction with Cooperative Groups")
print("=" * 70)
msg = "Multi-stage reduction in a single kernel using grid.sync()"
print(f"\nDemonstrates: {msg}")
# Initialize device
device = Device()
device.set_current()
major, minor = device.compute_capability
print("\nDevice Information:")
print(f" Name: {device.name}")
print(f" Compute Capability: sm_{major}.{minor}")
# Get device properties for configuration
prop = device.properties
# Determine threads per block
if max_threads is None:
max_threads = prop.max_threads_per_block
threads_per_block = min(max_threads, 1024)
# Define data type and itemsize
itemsize = np.dtype(np.float32).itemsize
print("\nReduction Configuration:")
print(f" Number of elements: {num_elements:,}")
print(f" Data size: {num_elements * itemsize / (1024**2):.2f} MB")
# Compile kernel
print("\nCompiling CUDA kernel...")
# Support colon-separated multiple include paths
include_paths = cuda_include_dir.split(":")
program_options = ProgramOptions(
std="c++17", arch=f"sm_{device.arch}", include_path=include_paths
)
prog = Program(REDUCTION_KERNEL, code_type="c++", options=program_options)
mod = prog.compile("cubin")
kernel = mod.get_kernel("reduceSinglePassMultiBlockCG")
print(" Kernel compiled successfully")
# Calculate blocks for cooperative launch
shared_mem_bytes = threads_per_block * itemsize
if max_blocks is None:
max_blocks = get_max_cooperative_blocks(
device, kernel, threads_per_block, shared_mem_bytes
)
# Calculate optimal blocks (all must be resident)
num_blocks = min(
max_blocks, (num_elements + threads_per_block - 1) // threads_per_block
)
print("\nLaunch Configuration:")
print(f" Threads per block: {threads_per_block}")
print(f" Number of blocks: {num_blocks}")
print(f" Total threads: {num_blocks * threads_per_block:,}")
print(f" Shared memory per block: {shared_mem_bytes} bytes")
print(" Launch mode: Cooperative (grid-wide sync enabled)")
# Generate random input data
print("\n> Generating random input data...")
rng = np.random.default_rng(42)
h_idata = (rng.random(num_elements) * 256).astype(np.float32)
stream = device.create_stream()
cp_stream = cp.cuda.Stream.from_external(stream)
try:
d_odata = cp.empty(num_blocks, dtype=np.float32)
with cp_stream:
d_idata = cp.asarray(h_idata, dtype=np.float32)
stream.sync()
# Compute CPU reference
print("> Computing reference result on CPU...")
cpu_start = time.perf_counter()
cpu_result = float(np.sum(h_idata))
cpu_time = time.perf_counter() - cpu_start
print(f" CPU time: {cpu_time:.6f} seconds")
# Warm-up
print("\n> Warming up GPU...")
launch_config = LaunchConfig(
grid=(num_blocks, 1, 1),
block=(threads_per_block, 1, 1),
shmem_size=shared_mem_bytes,
cooperative_launch=True,
)
n_u32 = np.uint32(num_elements)
ptr_in = d_idata.data.ptr
ptr_out = d_odata.data.ptr
try:
launch(stream, launch_config, kernel, ptr_in, ptr_out, n_u32)
except Exception as e:
print(f" Cooperative launch failed: {e}")
return 1
stream.sync()
print(" Warm-up successful")
# Benchmark (CUDA events — not host wall clock around the whole loop)
print(f"\n> Running benchmark ({test_iterations} iterations)...")
event_options = EventOptions(enable_timing=True)
start_event = stream.device.create_event(options=event_options)
end_event = stream.device.create_event(options=event_options)
# cuda.core event elapsed time (end - start) is in milliseconds (CUDA API).
gpu_times_ms = []
for _ in range(test_iterations):
try:
stream.record(start_event)
launch(stream, launch_config, kernel, ptr_in, ptr_out, n_u32)
stream.record(end_event)
end_event.sync()
gpu_times_ms.append(float(end_event - start_event))
except Exception as e:
print(f"Benchmark iteration failed: {e}")
return 1
avg_gpu_ms = float(np.mean(gpu_times_ms))
avg_gpu_s = avg_gpu_ms / 1000.0
stream.sync()
with cp_stream:
h_result = cp.asnumpy(d_odata[:1])
gpu_result = float(h_result[0])
# Performance metrics use seconds for throughput and speedup.
# CPU time is already in seconds.
bytes_processed = num_elements * 4
throughput_gb_s = bytes_processed / avg_gpu_s / 1e9
print("\n> Performance Results:")
print(f" Average GPU time: {avg_gpu_ms:.6f} ms")
print(f" Throughput: {throughput_gb_s:.2f} GB/s")
print(f" Speedup vs CPU: {cpu_time / avg_gpu_s:.2f}x")
# Validate results
print("\n> Validating results...")
success = verify_array_result(
np.array([gpu_result]),
np.array([cpu_result]),
rtol=1e-5,
atol=1e-5,
)
# Summary
print("\n" + "=" * 70)
print("Summary")
print("=" * 70)
print(f"""
Single-kernel two-stage reduction:
Stage 1: {num_blocks} blocks → {num_blocks} partial sums
grid.sync() ← All blocks synchronize (KEY innovation)
Stage 2: Block 0 → 1 final result
Total: 1 kernel launch, {throughput_gb_s:.2f} GB/s
Comparison:
• Traditional: 2 kernel launches or kernel + CPU
• This sample: 1 kernel with grid.sync() between stages
• Benefit: Eliminates ~5-20μs launch overhead per stage
""")
print("=" * 70)
if success:
print("Single-Pass Multi-Block Reduction completed successfully!")
else:
print("Single-Pass Multi-Block Reduction FAILED!")
print("=" * 70 + "\n")
return 0 if success else 1
finally:
stream.close()
def main():
"""Main entry point with argument parsing."""
parser = argparse.ArgumentParser(
description="Single-Pass Multi-Block Reduction with Cooperative Groups"
)
parser.add_argument(
"--n",
type=int,
default=1 << 25,
help="Number of elements to reduce (default: 33554432 = 2^25)",
)
parser.add_argument(
"--threads",
type=int,
default=None,
help=(
"Threads per block, power of 2 in [1, 1024]; "
"default: device maximum (typically 1024)"
),
)
parser.add_argument(
"--maxblocks",
type=int,
default=None,
help=(
"Maximum number of blocks "
"(default: auto-calculated for cooperative launch)"
),
)
parser.add_argument(
"--iterations",
type=int,
default=100,
help="Number of benchmark iterations (default: 100)",
)
parser.add_argument(
"--cuda-include-dir",
type=str,
required=True,
help=(
"CUDA include directory for NVRTC "
"(can use colon-separated paths, e.g., /path1:/path2)"
),
)
args = parser.parse_args()
# Validate arguments
if args.n <= 0:
print("Error: n must be positive")
return 1
err_threads = _validate_threads_arg(args.threads)
if err_threads:
print(f"Error: {err_threads}")
return 1
if args.maxblocks is not None and args.maxblocks <= 0:
print("Error: maxblocks must be positive")
return 1
try:
exit_code = run(
num_elements=args.n,
max_threads=args.threads,
max_blocks=args.maxblocks,
test_iterations=args.iterations,
cuda_include_dir=args.cuda_include_dir,
)
sys.exit(exit_code)
except Exception as e:
print(f"\nError: {e}")
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == "__main__":
main()

10
python/2_CoreConcepts/reductionMultiBlockCG/requirements.txt Normal file
View File

@@ -0,0 +1,10 @@
# Python Multi-Block Cooperative Reduction Sample Requirements
# Install with: pip install -r requirements.txt
numpy>=2.3.2
cuda-core>=0.6.0
cuda-python>=13.0.0
# Headers for NVRTC: cooperative_groups.h includes cuda/std/* (CCCL)
cuda-cccl>=1.0.0
# Use cupy-cuda13x>=14.0.0 for cp.cuda.Stream.from_external(stream)
cupy-cuda13x>=14.0.0

141
python/2_CoreConcepts/simpleZeroCopy/README.md Normal file
View File

@@ -0,0 +1,141 @@
# Sample: simpleZeroCopy (Python)
## Description
This sample demonstrates zero-copy access using **`cuda.core`** to compile and launch a kernel, and **`cuda.bindings.runtime`** for mapped pinned host memory (`cudaHostAlloc` with `cudaHostAllocMapped`, `cudaHostGetDevicePointer`, and `cudaFreeHost`). The GPU loads and stores through **device** addresses that refer to that host memory—no `cudaMemcpy` in or out. The example is vector add with inputs and output as NumPy views of the host side of those buffers.
## What you will learn
- How to allocate **mapped** pinned host memory with `cudaHostAlloc` (via `cuda.bindings.runtime`) so the GPU can use `cudaHostGetDevicePointer` addresses in a kernel
- How `cuda.core.PinnedMemoryResource` differs (staging/copies; not guaranteed to be `cudaHostAllocMapped` for direct kernel access)
- How to build NumPy views of host addresses with `ctypes` and `numpy.frombuffer`
- How to launch CUDA kernels with `cuda.core`’s `Program` and `launch`, passing **device** pointers for mapped buffers
- When zero-copy is beneficial vs. device memory with explicit transfers
- How to validate results on the host without a D2H memcpy
## Key libraries
- `numpy` – CPU arrays and reference computation
- `cuda-core` – `Device`, stream, `Program`, `LaunchConfig`, `launch`
- `cuda-python` (`cuda.bindings.runtime`) – `cudaHostAlloc` / `cudaHostGetDevicePointer` / `cudaFreeHost` for mapped host memory
## Key APIs
**From cuda.core:** `Device`, `device.create_stream()`, `Program`, `ProgramOptions`, `LaunchConfig`, `launch`
**From cuda.bindings.runtime:** `cudaHostAlloc` (with `cudaHostAllocMapped` | `cudaHostAllocPortable`), `cudaHostGetDevicePointer`, `cudaFreeHost`
**From the standard library:** `ctypes` – wrap host pointers for `numpy.frombuffer` float32 views
**Memory management:** Free host memory with `cudaFreeHost` in a `finally` block; call `stream.close()` when done.
## Zero-Copy Memory: When to Use
### Benefits
- **No explicit transfers**: Simplifies code by eliminating `cudaMemcpy` calls
- **Automatic synchronization**: Host can access results immediately after kernel completes
- **Good for small data**: Overhead of explicit transfers can exceed benefits for small arrays
- **Excellent for integrated GPUs**: On systems like Jetson (Tegra), CPU and GPU share physical memory
### Limitations
- **Slower access**: Limited by PCIe bandwidth vs. device memory bandwidth
- **Not for compute-intensive**: Device memory is much faster for frequently accessed data
- **Discrete GPU overhead**: Each access crosses PCIe bus
### Best Use Cases
1. Small data sets where transfer overhead dominates
2. Data accessed infrequently by GPU
3. Integrated GPU platforms (shared memory)
4. Streaming data from host to device
5. Prototyping and debugging (simplifies memory management)
## Requirements
1. **NVIDIA GPU** and a **driver** compatible with your installed `cuda-python` / `cuda-core` wheels.
2. **Python 3.10 or newer**
3. Install **`pip install -r requirements.txt`** (NumPy, `cuda-python`, `cuda-core`). A **system** CUDA Toolkit is not strictly required if the process can load the driver/runtime; use `LD_LIBRARY_PATH` in *How to run* if you hit missing-library errors.
**Install packages:**
```bash
pip install -r requirements.txt
```
Or manually:
```bash
pip install numpy>=2.3.2 cuda-core>=0.6.0 cuda-python>=13.0.0
```
## How to run
Basic usage:
```bash
# Pre-steps: Set library path
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
# Run with default parameters (1M elements)
python simpleZeroCopy.py
```
With custom parameters:
```bash
# Use 2M elements
python simpleZeroCopy.py --num_elements 2097152
# Show help
python simpleZeroCopy.py --help
```
### Command line arguments
- `--num_elements`: Number of elements in vectors (default: 1048576)
- Each vector uses `num_elements * 4 bytes` (float32)
- Default: ~4 MB per vector, ~12 MB total
## Expected Output
Device name and compute capability **depend on your system**; the rest of the log should match this shape when validation passes.
```
======================================================================
simpleZeroCopy - CUDA Python Sample
======================================================================
Device Information:
Name: <your GPU>
Compute Capability: <major>.<minor>
> Memory: mapped pinned host (cudaHostAlloc + cudaHostGetDevicePointer)
Compiling CUDA kernel...
Kernel compiled successfully
Allocating memory:
Vector size: 1,048,576 elements
Memory per vector: 4.00 MB
Total memory: 12.00 MB
> Allocating mapped pinned host memory...
Mapped host memory allocated successfully
> Initializing vectors on host...
> Computing reference result on CPU...
> Launching vectorAddGPU kernel...
Note: GPU accesses host memory directly (zero-copy)
Kernel execution complete
> Checking results from vectorAddGPU()...
Comparing 1,048,576 elements...
Relative error: 0.000000e+00
Validation PASSED
======================================================================
simpleZeroCopy completed successfully!
======================================================================
```
## Files
- `simpleZeroCopy.py` – Main Python implementation
- `README.md` – This file
- `requirements.txt` – Python package dependencies

6
python/2_CoreConcepts/simpleZeroCopy/requirements.txt Normal file
View File

@@ -0,0 +1,6 @@
# simpleZeroCopy - Requirements
# Install with: pip install -r requirements.txt
numpy>=2.3.2
cuda-python>=13.0.0
cuda-core>=0.6.0

275
python/2_CoreConcepts/simpleZeroCopy/simpleZeroCopy.py Normal file
View File

@@ -0,0 +1,275 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# distribution and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
import argparse
import ctypes
import sys
from pathlib import Path
try:
import numpy as np
from cuda.bindings import runtime as cuda_rt
from cuda.core import (
Device,
LaunchConfig,
Program,
ProgramOptions,
launch,
)
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install from requirements.txt:")
print(" pip install -r requirements.txt")
sys.exit(1)
# Add parent directory to path to import utilities
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
def _mapped_host_alloc(num_floats, stream):
"""
Allocate page-locked host memory mapped for device access; return
(host_ptr, device_ptr) for CPU views and for ``launch()``.
"""
nbytes = int(num_floats) * np.dtype(np.float32).itemsize
if nbytes <= 0:
return 0, 0
err, h_ptr = cuda_rt.cudaHostAlloc(
nbytes, cuda_rt.cudaHostAllocMapped | cuda_rt.cudaHostAllocPortable
)
if err != cuda_rt.cudaError_t.cudaSuccess:
raise RuntimeError(f"cudaHostAlloc failed: {err}")
err, d_ptr = cuda_rt.cudaHostGetDevicePointer(h_ptr, 0)
if err != cuda_rt.cudaError_t.cudaSuccess:
cuda_rt.cudaFreeHost(h_ptr)
raise RuntimeError(f"cudaHostGetDevicePointer failed: {err}")
# Ensure prior work on this stream is visible before host fills buffers.
if stream is not None:
stream.sync()
return h_ptr, d_ptr
def _float_view(host_ptr, num_floats):
return np.frombuffer(
(ctypes.c_float * num_floats).from_address(host_ptr),
dtype=np.float32,
count=num_floats,
)
# CUDA C++: vector add with grid-stride loop
VECTOR_ADD_KERNEL = """
extern "C" __global__
void vectorAddGPU(float* c, const float* a, const float* b, int N) {
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
size_t stride = gridDim.x * blockDim.x;
for (size_t i = tid; i < N; i += stride) {
c[i] = a[i] + b[i];
}
}
"""
def run(num_elements=1048576):
"""
Zero-copy vector add: map host memory, launch kernel with device
pointers, validate on CPU.
This function shows how to:
1. Allocate pinned (page-locked) host memory
2. Map host memory into GPU address space (zero-copy)
3. Access host memory directly from GPU kernel
4. Validate results
Parameters
----------
num_elements : int
Number of elements in vectors (default: 1048576)
"""
print("\n" + "=" * 70)
print("simpleZeroCopy - CUDA Python Sample")
print("=" * 70)
# Initialize device
device = Device()
device.set_current()
major, minor = device.compute_capability
print("\nDevice Information:")
print(f" Name: {device.name}")
print(f" Compute Capability: {major}.{minor}")
# Create stream
stream = device.create_stream()
mapped_host_ptrs = []
try:
print(
"\n> Memory: mapped pinned host "
"(cudaHostAlloc + cudaHostGetDevicePointer)"
)
print("\nCompiling CUDA kernel...")
program_options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
prog = Program(VECTOR_ADD_KERNEL, code_type="c++", options=program_options)
mod = prog.compile("cubin")
kernel = mod.get_kernel("vectorAddGPU")
print(" Kernel compiled successfully")
bytes_total = num_elements * np.dtype(np.float32).itemsize
print("\nAllocating memory:")
print(f" Vector size: {num_elements:,} elements")
print(f" Memory per vector: {bytes_total / (1024**2):.2f} MB")
print(f" Total memory: {3 * bytes_total / (1024**2):.2f} MB")
print("\n> Allocating mapped pinned host memory...")
h_a, d_a = _mapped_host_alloc(num_elements, stream)
mapped_host_ptrs.append(h_a)
h_b, d_b = _mapped_host_alloc(num_elements, stream)
mapped_host_ptrs.append(h_b)
h_c, d_c = _mapped_host_alloc(num_elements, stream)
mapped_host_ptrs.append(h_c)
a = _float_view(h_a, num_elements)
b = _float_view(h_b, num_elements)
c = _float_view(h_c, num_elements)
print(" Mapped host memory allocated successfully")
print("\n> Initializing vectors on host...")
rng = np.random.default_rng(42)
a[:] = rng.random(num_elements).astype(np.float32)
b[:] = rng.random(num_elements).astype(np.float32)
c[:] = 0
print("> Computing reference result on CPU...")
reference = a + b
print("\n> Launching vectorAddGPU kernel...")
print(" Note: GPU accesses host memory directly (zero-copy)")
block_size = 256
grid_size = (num_elements + block_size - 1) // block_size
config = LaunchConfig(grid=grid_size, block=block_size)
# Pass device pointers from cudaHostGetDevicePointer, not raw host VAs.
launch(
stream,
config,
kernel,
int(d_c),
int(d_a),
int(d_b),
np.int32(num_elements),
)
stream.sync()
print(" Kernel execution complete")
print("\n> Checking results from vectorAddGPU()...")
print(f" Comparing {num_elements:,} elements...")
# ``c`` is a host view of the same buffer; no cudaMemcpy D2H needed.
if np.allclose(c, reference, rtol=1e-5, atol=1e-6):
error_norm = np.linalg.norm(c - reference)
ref_norm = np.linalg.norm(reference)
relative_error = error_norm / ref_norm
print(f" Relative error: {relative_error:.6e}")
print(" Validation PASSED")
success = True
else:
max_error = np.max(np.abs(c - reference))
print(f" Max error: {max_error}")
print(" Validation FAILED")
success = False
print("\n" + "=" * 70)
if success:
print("simpleZeroCopy completed successfully!")
else:
print("simpleZeroCopy FAILED!")
print("=" * 70 + "\n")
return 0 if success else 1
finally:
for h in reversed(mapped_host_ptrs):
if h:
cuda_rt.cudaFreeHost(h)
stream.close()
def main():
"""Parse CLI, call ``run()``, and exit with validation status."""
parser = argparse.ArgumentParser(
description="Demonstrate zero-copy memory access with CUDA",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
python simpleZeroCopy.py
python simpleZeroCopy.py --num_elements 2097152
What is Zero-Copy Memory?
Zero-copy allows the GPU to directly access host (CPU) memory without
explicit memory transfers. This is useful for:
- Small data that doesn't benefit from explicit transfers
- Data that is accessed infrequently
- Integrated GPUs that share memory with CPU
Trade-offs:
- Slower than device memory (PCIe bandwidth limited)
- No explicit transfers needed (simpler code)
- Good for discrete GPUs with small data
- Excellent for integrated GPUs (e.g., Tegra)
""",
)
parser.add_argument(
"--num_elements",
type=int,
default=1048576,
help="Number of elements in vectors (default: 1048576)",
)
args = parser.parse_args()
if args.num_elements <= 0:
print("Error: num_elements must be positive")
sys.exit(1)
try:
exit_code = run(num_elements=args.num_elements)
except Exception as e:
print(f"\nError: {e}")
import traceback
traceback.print_exc()
exit_code = 1
sys.exit(exit_code)
if __name__ == "__main__":
main()

98
python/2_CoreConcepts/streamingCopyComputeOverlap/README.md Normal file
View File

@@ -0,0 +1,98 @@
# Sample: Streaming Copy + Compute Overlap (Python)
## Description
Demonstrate how to overlap memory transfers (H2D/D2H) with kernel computation using CUDA streams. This technique hides transfer latency and improves GPU utilization.
## What You'll Learn
- Using `PinnedMemoryResource` for async-capable host memory
- Using `DeviceMemoryResource` for GPU memory allocation
- Creating multiple streams with `Device.create_stream()`
- Async memory copies with `Buffer.copy_to()`
- Overlapping H2D transfers, kernel execution, and D2H transfers
## Key Concept
**Without overlap (sequential):**
```
[====H2D====][====Compute====][====D2H====]
```
**With overlap (multiple streams):**
```
Stream 0: [H2D][Compute][D2H]
Stream 1: [H2D][Compute][D2H]
Stream 2: [H2D][Compute][D2H]
```
## Key APIs (all from `cuda.core`)
- `Device` - Device management
- `Device.create_stream()` - Create CUDA streams
- `Stream.sync()` - Synchronize stream
- `PinnedMemoryResource` - Pinned host memory (required for async transfers)
- `DeviceMemoryResource` - GPU device memory
- `Buffer.copy_to(dst, stream=stream)` - Async memory copy
- `Program`, `LaunchConfig`, `launch` - Kernel compilation and execution
### From `numpy`:
- `np.from_dlpack()` - Zero-copy view of pinned memory buffers
## Requirements
- CUDA Toolkit 13.0+
- Python 3.10+
- `cuda-python`, `cuda-core`, `numpy`
## Installation
```bash
pip install -r requirements.txt
```
## How to Run
```bash
python streamingCopyComputeOverlap.py
```
## Expected Output
```
============================================================
Streaming Copy + Compute Overlap
Using pure cuda.core APIs
============================================================
Device: NVIDIA GeForce RTX XXXX
Kernel compiled ✓
Problem size: 16,000,000 elements (61 MB)
--- Sequential (no overlap) ---
Timeline: [H2D][Compute][D2H]
Time: X.XX ms (±X.XX)
--- Streamed (with overlap) ---
Stream 0: [H2D][Compute][D2H]
Stream 1: [H2D][Compute][D2H]
Stream 2: [H2D][Compute][D2H]
...
2 streams: X.XX ms (±X.XX) - speedup: X.XXx
4 streams: X.XX ms (±X.XX) - speedup: X.XXx
8 streams: X.XX ms (±X.XX) - speedup: X.XXx
============================================================
Key: Pinned memory + multiple streams = overlap transfers with compute
Note: Speedup depends on hardware characteristics. This technique
benefits most when transfer time is significant relative to compute.
============================================================
```
## See Also
- [cuda.core Documentation](https://nvidia.github.io/cuda-python/cuda-core/latest/)
- [CUDA Streams Best Practices](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#overlapping-data-transfers)

6
python/2_CoreConcepts/streamingCopyComputeOverlap/requirements.txt Normal file
View File

@@ -0,0 +1,6 @@
# Streaming Copy Compute Overlap Sample Requirements
# Requires Python 3.10 or newer
cuda-python>=13.0.0
cuda-core>=0.6.0
numpy>=2.3.2

312
python/2_CoreConcepts/streamingCopyComputeOverlap/streamingCopyComputeOverlap.py Normal file
View File

@@ -0,0 +1,312 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Streaming Copy + Compute Overlap
Demonstrates how to overlap memory transfers with kernel computation using
CUDA streams to maximize GPU utilization.
Uses pure cuda.core APIs:
- Device, Stream for device and stream management
- PinnedMemoryResource, DeviceMemoryResource for memory allocation
- Buffer.copy_to() for async memory copies
- Program, LaunchConfig, launch for kernel compilation and execution
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
try:
import numpy as np
from cuda.core import (
Device,
DeviceMemoryResource,
EventOptions,
LaunchConfig,
PinnedMemoryResource,
Program,
ProgramOptions,
launch,
)
from cuda_samples_utils import print_gpu_info
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Install with: pip install -r requirements.txt")
sys.exit(1)
# CUDA Kernel - compute-intensive vector operation (grid-stride loop)
VECTOR_SCALE_KERNEL = r"""
extern "C" __global__
void vector_scale(const float* input, float* output, float scale, size_t N) {
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
size_t stride = (size_t)gridDim.x * blockDim.x;
for (size_t i = tid; i < N; i += stride) {
float val = input[i] * scale;
// Add compute work to make kernel non-trivial
for (int j = 0; j < 50; j++) {
val = sqrtf(val * val + 1.0f);
}
output[i] = val;
}
}
"""
def buffer_to_numpy(buffer, n_elements):
"""Create numpy view of cuda.core Buffer via DLPack."""
return np.from_dlpack(buffer).view(np.float32).reshape(n_elements)
def main():
print("=" * 60)
print("Streaming Copy + Compute Overlap")
print("Using pure cuda.core APIs")
print("=" * 60)
# Initialize device
device = Device(0)
device.set_current()
print()
print_gpu_info(device)
# Compile kernel
arch = f"sm_{device.arch}"
program = Program(
VECTOR_SCALE_KERNEL, code_type="c++", options=ProgramOptions(arch=arch)
)
kernel = program.compile(target_type="cubin").get_kernel("vector_scale")
print("Kernel compiled ✓")
# Parameters
N = 16_000_000 # 16M elements
n_bytes = N * 4
scale = 2.5
n_runs = 10
print(f"\nProblem size: {N:,} elements ({n_bytes / 1024 / 1024:.0f} MB)")
# Create memory resources
pinned_mr = PinnedMemoryResource()
device_mr = DeviceMemoryResource(device.device_id)
default_stream = device.create_stream()
# =========================================================================
# Sequential Execution
# =========================================================================
print("\n--- Sequential (no overlap) ---")
print("Timeline: [H2D][Compute][D2H]")
h_in = h_out = d_in = d_out = None
try:
# Pre-allocate buffers
h_in = pinned_mr.allocate(n_bytes, default_stream)
h_out = pinned_mr.allocate(n_bytes, default_stream)
d_in = device_mr.allocate(n_bytes, default_stream)
d_out = device_mr.allocate(n_bytes, default_stream)
# Sync before numpy access (numpy operations aren't stream ordered)
default_stream.sync()
# Initialize input
np_in = buffer_to_numpy(h_in, N)
np_in[:] = np.random.rand(N).astype(np.float32) * 100
config = LaunchConfig(grid=((N + 255) // 256,), block=(256,))
event_opts = EventOptions(enable_timing=True)
# Warm up
h_in.copy_to(d_in, stream=default_stream)
launch(
default_stream,
config,
kernel,
d_in,
d_out,
np.float32(scale),
np.uint64(N),
)
d_out.copy_to(h_out, stream=default_stream)
default_stream.sync()
# Benchmark with CUDA events
times = []
for _ in range(n_runs):
start_ev = device.create_event(options=event_opts)
end_ev = device.create_event(options=event_opts)
default_stream.record(start_ev)
h_in.copy_to(d_in, stream=default_stream) # Async H2D
launch(
default_stream,
config,
kernel,
d_in,
d_out,
np.float32(scale),
np.uint64(N),
)
d_out.copy_to(h_out, stream=default_stream) # Async D2H
default_stream.record(end_ev)
default_stream.sync()
times.append(end_ev - start_ev)
seq_time = np.mean(times)
print(f"Time: {seq_time:.2f} ms (±{np.std(times):.2f})")
# Verification: compute expected on CPU and compare
default_stream.sync()
np_out = buffer_to_numpy(h_out, N)
expected = np_in.astype(np.float32) * scale
for _ in range(50):
expected = np.sqrt(expected * expected + 1.0).astype(np.float32)
if np.allclose(np_out, expected, rtol=1e-4, atol=1e-4):
print("Verification: PASSED")
else:
print("Verification: FAILED")
finally:
for buf in (h_in, h_out, d_in, d_out):
if buf is not None:
buf.close()
default_stream.close()
# =========================================================================
# Streamed Execution
# =========================================================================
print("\n--- Streamed (with overlap) ---")
print("Stream 0: [H2D][Compute][D2H]")
print("Stream 1: [H2D][Compute][D2H]")
print("Stream 2: [H2D][Compute][D2H]")
print("...")
for n_streams in [2, 4, 8]:
chunk_size = N // n_streams
chunk_bytes = chunk_size * 4
# Create streams
streams = [device.create_stream() for _ in range(n_streams)]
# Pre-allocate per-stream buffers
h_ins, h_outs, d_ins, d_outs = [], [], [], []
try:
for i in range(n_streams):
h_ins.append(pinned_mr.allocate(chunk_bytes, streams[i]))
h_outs.append(pinned_mr.allocate(chunk_bytes, streams[i]))
d_ins.append(device_mr.allocate(chunk_bytes, streams[i]))
d_outs.append(device_mr.allocate(chunk_bytes, streams[i]))
# Initialize input data
for i in range(n_streams):
streams[i].sync()
np_view = buffer_to_numpy(h_ins[i], chunk_size)
np_view[:] = np.random.rand(chunk_size).astype(np.float32) * 100
chunk_config = LaunchConfig(grid=((chunk_size + 255) // 256,), block=(256,))
# Warm up
for i in range(n_streams):
h_ins[i].copy_to(d_ins[i], stream=streams[i])
launch(
streams[i],
chunk_config,
kernel,
d_ins[i],
d_outs[i],
np.float32(scale),
np.uint64(chunk_size),
)
d_outs[i].copy_to(h_outs[i], stream=streams[i])
for stream in streams:
stream.sync()
# Benchmark with CUDA events (use stream 0 for timing)
times = []
event_opts = EventOptions(enable_timing=True)
for _ in range(n_runs):
start_ev = device.create_event(options=event_opts)
end_ev = device.create_event(options=event_opts)
streams[0].record(start_ev)
# Issue all operations - they overlap across streams
for i in range(n_streams):
h_ins[i].copy_to(d_ins[i], stream=streams[i]) # Async H2D
launch(
streams[i],
chunk_config,
kernel,
d_ins[i],
d_outs[i],
np.float32(scale),
np.uint64(chunk_size),
)
d_outs[i].copy_to(h_outs[i], stream=streams[i]) # Async D2H
# Wait for all streams, record end on stream 0
for stream in streams:
stream.sync()
streams[0].record(end_ev)
streams[0].sync()
times.append(end_ev - start_ev)
avg = np.mean(times)
speedup = seq_time / avg
print(
f"{n_streams} streams: {avg:.2f} ms (±{np.std(times):.2f}) "
f"- speedup: {speedup:.2f}x"
)
# Verification (streamed): concatenate chunks and compare to expected
for s in streams:
s.sync()
out_chunks = [
buffer_to_numpy(h_outs[i], chunk_size) for i in range(n_streams)
]
in_chunks = [
buffer_to_numpy(h_ins[i], chunk_size) for i in range(n_streams)
]
np_out = np.concatenate(out_chunks)
np_in = np.concatenate(in_chunks)
expected = np_in.astype(np.float32) * scale
for _ in range(50):
expected = np.sqrt(expected * expected + 1.0).astype(np.float32)
if not np.allclose(np_out, expected, rtol=1e-4, atol=1e-4):
print(f" Verification: FAILED for {n_streams} streams")
finally:
for buf in h_ins + h_outs + d_ins + d_outs:
buf.close()
for s in streams:
s.close()
print("\n" + "=" * 60)
print("Key: Pinned memory + multiple streams = overlap transfers with compute")
print("\nNote: Speedup depends on hardware characteristics. This technique")
print("benefits most when transfer time is significant relative to compute.")
print("=" * 60)
if __name__ == "__main__":
main()

138
python/2_CoreConcepts/tmaTensorMap/README.md Normal file
View File

@@ -0,0 +1,138 @@
# tmaTensorMap (Python)
## Description
This sample demonstrates how to use Tensor Memory Accelerator (TMA)
descriptors with `cuda.core` on Hopper and later GPUs (compute
capability >= 9.0). TMA enables efficient bulk data movement between
global and shared memory using hardware-managed tensor map
descriptors, which are a key building block for modern GEMM kernels
and large shared-memory tile loads.
The sample:
1. Creates a TMA tiled descriptor from a CuPy device array via
`StridedMemoryView.from_any_interface(...).as_tensor_map(...)`.
2. Passes the descriptor by value (as `__grid_constant__`) to a
kernel that uses libcudacxx TMA/barrier wrappers to bulk-load a
tile into shared memory, then copies it out to verify correctness.
3. Reuses the same descriptor against a new source tensor with
`replace_address()` to avoid rebuilding it.
## What You'll Learn
- Creating a TMA descriptor from a strided device tensor via
`StridedMemoryView.as_tensor_map(box_dim=...)`
- Passing a tensor map to a kernel by value using
`__grid_constant__`
- Using libcudacxx (`cuda/barrier`) to coordinate TMA loads with a
block-scoped barrier
- Reusing a descriptor against a new source buffer via
`tensor_map.replace_address(new_tensor)`
- Compiling a kernel to CUBIN for a specific target arch so Hopper
features are available
- Using `cuda.pathfinder` to locate the CUDA toolkit include directory
CCCL headers and libcudacxx
## Key Libraries
- [`cuda.core`](https://nvidia.github.io/cuda-python/cuda-core/latest/) - compilation, launching, and tensor-map helpers
- `cuda.pathfinder` - locate the CUDA toolkit include directory
- `cupy` - allocate and fill device tensors
- `numpy` - scalar kernel arguments
## Key APIs
### From `cuda.core`
- `StridedMemoryView.from_any_interface(tensor, stream_ptr=-1)` - build a typed view from any DLPack/CUDA-array-interface tensor
- `StridedMemoryView.as_tensor_map(box_dim=(...))` - produce a TMA descriptor for the given tile shape
- `tensor_map.replace_address(new_tensor)` - retarget an existing descriptor at a new tensor
- `Program(code, code_type="c++", options=ProgramOptions(std="c++17", arch="sm_90", include_path=[...]))` - compile a C++ kernel against libcudacxx
- `program.compile("cubin")` - produce a CUBIN so `__grid_constant__` and TMA intrinsics are fully supported
- `launch(stream, config, kernel, tensor_map, ...)` - pass the TMA descriptor as a kernel argument
### From `cuda.pathfinder`
- `get_cuda_path_or_home()` - return the detected CUDA toolkit root for locating `include/cccl`
### From `cuda_samples_utils`
- `print_gpu_info()` - print device name and compute capability
## Requirements
### Hardware
- NVIDIA Hopper or newer GPU with Compute Capability 9.0 or higher (H100, H200, B200, ...)
- On GPUs older than Hopper the sample exits cleanly without running the kernel
- Minimum GPU memory: 512 MB
### Software
- CUDA Toolkit 13.0 or newer with libcudacxx (cccl) headers
- Python 3.10 or newer
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=0.6.0)
- `cupy-cuda13x` (>=13.0.0)
## Installation
Install the required packages from `requirements.txt`:
```bash
cd /path/to/cuda-samples/python/2_CoreConcepts/tmaTensorMap
pip install -r requirements.txt
```
The `requirements.txt` installs:
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=0.6.0)
- `cupy-cuda13x` (>=13.0.0)
## How to Run
### Basic usage
```bash
cd cuda-samples/python/2_CoreConcepts/tmaTensorMap
python tmaTensorMap.py
```
### With custom parameters
```bash
# Larger tensor (must be a multiple of the 128-element tile)
python tmaTensorMap.py --elements 8192
# Use a specific GPU
python tmaTensorMap.py --device 1
```
## Expected Output
On a Hopper (sm_90) GPU:
```
Device: NVIDIA H100 PCIe
Compute Capability: 9.0
TMA copy verified: 1024 elements across 8 tiles
replace_address verified: descriptor reused with new source tensor
```
**Note:** Device name and compute capability will vary based on your GPU.
## Files
- `tmaTensorMap.py` - Python implementation using `cuda.core` TMA APIs
- `README.md` - This file
- `requirements.txt` - Sample dependencies
- `../../Utilities/cuda_samples_utils.py` - Common utilities (imported by this sample)
## See Also
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [TMA in the CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#tensor-memory-accelerator)
- [`cuda::barrier` reference](https://nvidia.github.io/cccl/libcudacxx/extended_api/synchronization_primitives/barrier.html)

4
python/2_CoreConcepts/tmaTensorMap/requirements.txt Normal file
View File

@@ -0,0 +1,4 @@
cuda-python>=13.0.0
cuda-core>=0.6.0
cupy-cuda13x>=13.0.0
numpy>=1.24.0

281
python/2_CoreConcepts/tmaTensorMap/tmaTensorMap.py Normal file
View File

@@ -0,0 +1,281 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
TMA Tensor Map with cuda.core
This sample demonstrates how to use Tensor Memory Accelerator (TMA)
descriptors with cuda.core on Hopper and later GPUs (compute
capability >= 9.0). TMA enables efficient bulk data movement between
global and shared memory using hardware-managed tensor map
descriptors.
The sample:
1. Creates a TMA tiled descriptor from a CuPy device array via
``StridedMemoryView.from_any_interface(...).as_tensor_map(...)``.
2. Passes the descriptor by value (as ``__grid_constant__``) to a
kernel that uses libcudacxx TMA/barrier wrappers to bulk-load a
tile into shared memory.
3. Reuses the same descriptor against a new source tensor with
``replace_address()`` to avoid rebuilding it.
On GPUs older than Hopper (sm < 90), the sample prints a diagnostic
and exits cleanly.
Ported from ``cuda_core/examples/tma_tensor_map.py`` in the
`cuda-python` repository.
"""
import os
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
try:
import cupy as cp
import numpy as np
from cuda.core import (
Device,
LaunchConfig,
Program,
ProgramOptions,
StridedMemoryView,
launch,
)
from cuda.pathfinder import find_nvidia_header_directory, get_cuda_path_or_home
from cuda_samples_utils import print_gpu_info
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install from requirements.txt:")
print(" pip install -r requirements.txt")
sys.exit(1)
TILE_SIZE = 128 # elements per tile, must match the kernel constant
KERNEL_SRC = r"""
#include <cuda/barrier>
// Minimal definition of the 128-byte opaque tensor-map struct.
struct __align__(64) TensorMap { unsigned long long opaque[16]; };
static constexpr int TILE_SIZE = 128;
using TmaBarrier = cuda::barrier<cuda::thread_scope_block>;
extern "C"
__global__ void tma_copy(
const __grid_constant__ TensorMap tensor_map,
float* output,
int N)
{
__shared__ __align__(128) float smem[TILE_SIZE];
__shared__ TmaBarrier bar;
const int tid = threadIdx.x;
const int tile_start = blockIdx.x * TILE_SIZE;
if (tid == 0)
{
init(&bar, 1);
}
__syncthreads();
if (tid == 0)
{
cuda::device::experimental::cp_async_bulk_tensor_1d_global_to_shared(
smem,
reinterpret_cast<const CUtensorMap*>(&tensor_map),
tile_start,
bar);
bar.wait(cuda::device::barrier_arrive_tx(bar, 1, TILE_SIZE * sizeof(float)));
}
__syncthreads();
if (tid < TILE_SIZE)
{
const int idx = tile_start + tid;
if (idx < N)
output[idx] = smem[tid];
}
}
"""
def _get_cccl_include_paths() -> list:
"""Locate the CUDA toolkit and libcudacxx (cccl) include directories.
``cuda.pathfinder.find_nvidia_header_directory`` searches pip-installed
CUDA packages, conda environments, and the standard system install
locations, so this works without requiring ``CUDA_PATH`` or
``CUDA_HOME``. ``get_cuda_path_or_home`` is used as a final fallback.
"""
include_path: list = []
# libcudacxx (cccl) - preferred, provides <cuda/barrier> used below.
try:
cccl_dir = find_nvidia_header_directory("cccl")
if cccl_dir and os.path.isdir(cccl_dir):
include_path.append(cccl_dir)
except Exception: # noqa: S110 - fallback probes continue below
pass
# CUDA runtime headers - needed for the CUtensorMap driver type.
try:
cudart_dir = find_nvidia_header_directory("cudart")
if (
cudart_dir
and os.path.isdir(cudart_dir)
and cudart_dir not in include_path
):
include_path.append(cudart_dir)
except Exception: # noqa: S110 - fallback probes continue below
pass
# Fallback: use CUDA_PATH / CUDA_HOME when pathfinder comes up empty.
if not include_path:
cuda_path = get_cuda_path_or_home()
if cuda_path is not None:
cuda_include = os.path.join(cuda_path, "include")
if os.path.isdir(cuda_include):
include_path.append(cuda_include)
cccl_include = os.path.join(cuda_include, "cccl")
if os.path.isdir(cccl_include):
include_path.insert(0, cccl_include)
if not include_path:
print(
"Could not locate CUDA toolkit headers.\n"
"Tried cuda.pathfinder (pip/conda/system installs) and "
"CUDA_PATH / CUDA_HOME; none succeeded.\n"
"Set CUDA_HOME to your toolkit root (containing include/cccl "
"and include/cuda_runtime.h) and retry.",
file=sys.stderr,
)
sys.exit(1)
return include_path
def main() -> int:
import argparse
parser = argparse.ArgumentParser(
description="Use a TMA tensor map to bulk-copy data on Hopper+ GPUs"
)
parser.add_argument(
"--elements",
type=int,
default=1024,
help="Total number of float32 elements (must be a multiple of 128)",
)
parser.add_argument("--device", type=int, default=0, help="CUDA device id")
args = parser.parse_args()
if args.elements % TILE_SIZE != 0:
print(f"--elements must be a multiple of TILE_SIZE={TILE_SIZE}")
return 1
dev = Device(args.device)
print_gpu_info(dev)
arch = dev.compute_capability
if arch < (9, 0):
print(
f"\nTMA requires compute capability >= 9.0 (Hopper or later); "
f"this device is {arch.major}.{arch.minor}. Exiting cleanly."
)
return 0
dev.set_current()
include_path = _get_cccl_include_paths()
# Compile with the CUBIN code type to target the exact device arch.
prog = Program(
KERNEL_SRC,
code_type="c++",
options=ProgramOptions(
std="c++17",
arch=f"sm_{dev.arch}",
include_path=include_path,
),
)
mod = prog.compile("cubin")
kernel = mod.get_kernel("tma_copy")
# (1) Prepare input data and verify the initial TMA copy.
n = args.elements
src = cp.arange(n, dtype=cp.float32)
output = cp.zeros(n, dtype=cp.float32)
dev.sync() # CuPy uses its own stream
tensor_map = StridedMemoryView.from_any_interface(
src, stream_ptr=-1
).as_tensor_map(box_dim=(TILE_SIZE,))
n_tiles = n // TILE_SIZE
config = LaunchConfig(grid=n_tiles, block=TILE_SIZE)
launch(
dev.default_stream,
config,
kernel,
tensor_map,
output.data.ptr,
np.int32(n),
)
dev.sync()
if not cp.array_equal(output, src):
print("TMA copy produced incorrect results")
return 1
print(f"TMA copy verified: {n} elements across {n_tiles} tiles")
# (2) Demonstrate replace_address() without rebuilding the descriptor.
replacement = cp.full(n, fill_value=42.0, dtype=cp.float32)
dev.sync()
tensor_map.replace_address(replacement)
output2 = cp.zeros(n, dtype=cp.float32)
launch(
dev.default_stream,
config,
kernel,
tensor_map,
output2.data.ptr,
np.int32(n),
)
dev.sync()
if not cp.array_equal(output2, replacement):
print("replace_address produced incorrect results")
return 1
print("replace_address verified: descriptor reused with new source tensor")
return 0
if __name__ == "__main__":
sys.exit(main())