NASP/cuda-samples
84
0
Fork 0
mirror of https://github.com/NVIDIA/cuda-samples.git synced 2026-10-13 00:08:26 +08:00
Code Issues Packages Projects Releases Wiki Activity

CUDA 13.2 samples update (#432)

Browse Source 
- Added Python samples for CUDA Python 1.0 release
- Renamed top-level `Samples` directory to `cpp` to accommodate Python samples.
This commit is contained in:
Dheemanth
2026-05-13 15:13:18 -07:00
committed by GitHub
parent a4526d5229
commit aeab82ff30
2072 changed files with 15077 additions and 1970 deletions
Download Patch File Download Diff File Expand all files Collapse all files

140
python/2_CoreConcepts/cudaGraphs/README.md Normal file
View File

@@ -0,0 +1,140 @@
# cudaGraphs (Python)
## Description
This sample demonstrates how to capture a multi-stage kernel pipeline as a
CUDA graph with `cuda.core` and replay it with a single driver call.
The sample runs a three-stage elementwise pipeline
`r3 = (a + b) * c - a` in two modes:
1. **Individual launches** - one `launch(stream, ...)` per stage, repeated
for every iteration of the pipeline.
2. **CUDA graph replay** - the same three launches are recorded into a
`Graph` once and replayed with `graph.launch(stream)` on each
iteration.
Both paths are timed over N iterations and their results are verified
against a reference computation. The sample also re-launches the graph
after mutating the input buffers to show that the graph captures
pointers (not data), so the same graph can process new inputs without
rebuilding.
## What You'll Learn
- Creating a `GraphBuilder` from a stream with `stream.create_graph_builder()`
- Capturing launches with `begin_building()` and `end_building()`
- Completing a graph with `builder.complete()` and uploading it to a stream
- Replaying the graph with `graph.launch(stream)`
- Measuring the launch-overhead savings for small kernels
- Re-running the same graph against updated input data
## Key Libraries
- [`cuda.core`](https://nvidia.github.io/cuda-python/cuda-core/latest/) - Pythonic access to CUDA runtime, programs, and graphs
- `cupy` - input buffers and result verification
- `numpy` - scalar kernel arguments
## Key APIs
### From `cuda.core`
- `Stream.create_graph_builder()` - obtain a `GraphBuilder`
- `GraphBuilder.begin_building()` / `end_building()` - begin and finish recording launches issued against the builder
- `GraphBuilder.complete()` - produce an executable `Graph`
- `Graph.upload(stream)` - upload the graph structure to the device
- `Graph.launch(stream)` - replay the entire graph
- `launch(graph_builder, config, kernel, ...)` - record a kernel launch into the graph being built
### From `cuda_samples_utils`
- `print_gpu_info()` - print device name and compute capability
## Requirements
### Hardware
- NVIDIA GPU with Compute Capability 7.0 or higher
- Minimum GPU memory: 512 MB
### Software
- CUDA Toolkit 13.0 or newer (matches `cuda-python` 13.x)
- Python 3.10 or newer
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=0.6.0)
- `cupy-cuda13x` (>=13.0.0)
## Installation
Install the required packages from `requirements.txt`:
```bash
cd /path/to/cuda-samples/python/2_CoreConcepts/cudaGraphs
pip install -r requirements.txt
```
The `requirements.txt` installs:
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=0.6.0)
- `cupy-cuda13x` (>=13.0.0)
## How to Run
### Basic usage
```bash
cd cuda-samples/python/2_CoreConcepts/cudaGraphs
python cudaGraphs.py
```
### With custom parameters
```bash
# Larger vectors and more iterations
python cudaGraphs.py --elements 4096 --iters 2000
# Use a specific GPU
python cudaGraphs.py --device 1
```
Short vectors exaggerate the launch-overhead savings; larger vectors
will show the two approaches converging because per-launch overhead
becomes negligible next to kernel runtime.
## Expected Output
Speedup numbers vary with GPU and host CPU.
```
Device: <Your GPU Name>
Compute Capability: <X.Y>
Individual launches: 1000 iters in 0.0085s (8.49 us/iter)
Building CUDA graph...
Graph replay: 1000 iters in 0.0034s (3.41 us/iter)
Graph speedup: 2.49x
Graph replay on updated data verified (same graph, new buffer contents)
Done
```
**Note:** Device name, compute capability, and speedup will vary based on
your GPU and host CPU.
## Files
- `cudaGraphs.py` - Python implementation using `cuda.core` CUDA graphs
- `README.md` - This file
- `requirements.txt` - Sample dependencies
- `../../Utilities/cuda_samples_utils.py` - Common utilities (imported by this sample)
## See Also
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [`cuda.core` graphs API](https://nvidia.github.io/cuda-python/cuda-core/latest/api.html#cuda-graphs)
- Upstream `cuda.core` example: [`cuda_graphs.py`](https://github.com/NVIDIA/cuda-python/blob/main/cuda_core/examples/cuda_graphs.py)
- [CUDA Graphs programming guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs)

266
python/2_CoreConcepts/cudaGraphs/cudaGraphs.py Normal file
View File

@@ -0,0 +1,266 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
CUDA Graphs with cuda.core
CUDA graphs let you record a DAG of operations once, then replay the entire
graph with a single driver call. For workflows that issue many small kernels
this can significantly reduce CPU-side launch overhead.
This sample runs a three-stage elementwise pipeline (add -> multiply ->
subtract) in two modes:
1. Individually launched kernels on a stream.
2. A single CUDA graph that captures the same three launches and is
replayed with ``graph.launch(stream)``.
We then measure the wall-clock time of each mode across many iterations to
illustrate the graph replay advantage for short kernels, and demonstrate that
a graph can be relaunched against new data (the pointers are baked in, but
the contents of those buffers are not).
"""
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent.parent / "Utilities"))
try:
import cupy as cp
import numpy as np
from cuda.core import Device, LaunchConfig, Program, ProgramOptions, launch
from cuda_samples_utils import print_gpu_info # noqa: E402
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install from requirements.txt:")
print(" pip install -r requirements.txt")
sys.exit(1)
PIPELINE_KERNELS = r"""
extern "C" __global__
void vec_add(const float* A, const float* B, float* C, size_t N) {
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
size_t stride = (size_t)gridDim.x * blockDim.x;
for (size_t i = tid; i < N; i += stride) C[i] = A[i] + B[i];
}
extern "C" __global__
void vec_mul(const float* A, const float* B, float* C, size_t N) {
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
size_t stride = (size_t)gridDim.x * blockDim.x;
for (size_t i = tid; i < N; i += stride) C[i] = A[i] * B[i];
}
extern "C" __global__
void vec_sub(const float* A, const float* B, float* C, size_t N) {
size_t tid = blockIdx.x * blockDim.x + threadIdx.x;
size_t stride = (size_t)gridDim.x * blockDim.x;
for (size_t i = tid; i < N; i += stride) C[i] = A[i] - B[i];
}
"""
def run_pipeline_individual(stream, kernels, config, buffers, size, n_iters):
"""Run the 3-stage pipeline `n_iters` times with one launch per stage."""
add_k, mul_k, sub_k = kernels
a, b, c, r1, r2, r3 = buffers
stream.sync()
t0 = time.perf_counter()
for _ in range(n_iters):
launch(
stream, config, add_k, a.data.ptr, b.data.ptr, r1.data.ptr, np.uint64(size)
)
launch(
stream, config, mul_k, r1.data.ptr, c.data.ptr, r2.data.ptr, np.uint64(size)
)
launch(
stream, config, sub_k, r2.data.ptr, a.data.ptr, r3.data.ptr, np.uint64(size)
)
stream.sync()
return time.perf_counter() - t0
def build_graph(stream, kernels, config, buffers, size):
"""Capture the 3-stage pipeline into a CUDA graph and return it."""
add_k, mul_k, sub_k = kernels
a, b, c, r1, r2, r3 = buffers
graph_builder = stream.create_graph_builder()
graph_builder.begin_building()
launch(
graph_builder,
config,
add_k,
a.data.ptr,
b.data.ptr,
r1.data.ptr,
np.uint64(size),
)
launch(
graph_builder,
config,
mul_k,
r1.data.ptr,
c.data.ptr,
r2.data.ptr,
np.uint64(size),
)
launch(
graph_builder,
config,
sub_k,
r2.data.ptr,
a.data.ptr,
r3.data.ptr,
np.uint64(size),
)
graph_builder.end_building()
graph = graph_builder.complete()
graph.upload(stream)
return graph_builder, graph
def run_pipeline_graph(stream, graph, n_iters):
"""Launch the compiled graph `n_iters` times."""
stream.sync()
t0 = time.perf_counter()
for _ in range(n_iters):
graph.launch(stream)
stream.sync()
return time.perf_counter() - t0
def main() -> int:
import argparse
parser = argparse.ArgumentParser(description="CUDA Graphs demo with cuda.core")
parser.add_argument(
"--elements",
type=int,
default=1 << 12,
help="Elements per vector (default: 4096 - small to emphasize launch overhead)",
)
parser.add_argument(
"--iters",
type=int,
default=1000,
help="Number of pipeline iterations to time (default: 1000)",
)
parser.add_argument("--device", type=int, default=0, help="CUDA device id")
args = parser.parse_args()
device = Device(args.device)
device.set_current()
print_gpu_info(device)
stream = device.create_stream()
# Tell CuPy to order its allocations on our stream so buffer initialization
# below is serialized with the kernels we launch.
cp.cuda.ExternalStream(int(stream.handle)).use()
graph_builder = graph = None
try:
program_options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
program = Program(PIPELINE_KERNELS, code_type="c++", options=program_options)
module = program.compile("cubin")
add_k = module.get_kernel("vec_add")
mul_k = module.get_kernel("vec_mul")
sub_k = module.get_kernel("vec_sub")
kernels = (add_k, mul_k, sub_k)
N = args.elements
rng = cp.random.default_rng(seed=0)
a = rng.random(N, dtype=cp.float32)
b = rng.random(N, dtype=cp.float32)
c = rng.random(N, dtype=cp.float32)
r1 = cp.empty_like(a)
r2 = cp.empty_like(a)
r3 = cp.empty_like(a)
buffers = (a, b, c, r1, r2, r3)
expected = (a + b) * c - a
config = LaunchConfig(grid=(N + 255) // 256, block=256)
device.sync()
# Warm up compilation/caches, then measure individual launches.
run_pipeline_individual(stream, kernels, config, buffers, N, n_iters=5)
t_individual = run_pipeline_individual(
stream, kernels, config, buffers, N, n_iters=args.iters
)
assert cp.allclose(r3, expected, rtol=1e-5, atol=1e-5), (
"Individual pipeline produced incorrect results"
)
print(
f"\nIndividual launches: {args.iters} iters in {t_individual:.4f}s"
f" ({t_individual * 1e6 / args.iters:.2f} us/iter)"
)
# Capture the same pipeline as a graph and measure the replay.
print("\nBuilding CUDA graph...")
graph_builder, graph = build_graph(stream, kernels, config, buffers, N)
run_pipeline_graph(stream, graph, n_iters=5) # warm up
t_graph = run_pipeline_graph(stream, graph, n_iters=args.iters)
assert cp.allclose(r3, expected, rtol=1e-5, atol=1e-5), (
"Graph pipeline produced incorrect results"
)
print(
f"Graph replay: {args.iters} iters in {t_graph:.4f}s"
f" ({t_graph * 1e6 / args.iters:.2f} us/iter)"
)
if t_graph > 0:
print(f"Graph speedup: {t_individual / t_graph:.2f}x")
# Demonstrate that the graph replays against current buffer contents.
a[:] = cp.ones(N, dtype=cp.float32)
b[:] = cp.full(N, 2.0, dtype=cp.float32)
c[:] = cp.full(N, 3.0, dtype=cp.float32)
device.sync()
# r3 = (a + b) * c - a = (1 + 2) * 3 - 1 = 8
graph.launch(stream)
stream.sync()
assert cp.allclose(r3, 8.0), "Graph replay with new data produced wrong result"
print(
"\nGraph replay on updated data verified (same graph, new buffer contents)"
)
print("\nDone")
return 0
finally:
if graph is not None:
graph.close()
if graph_builder is not None:
graph_builder.close()
stream.close()
cp.cuda.Stream.null.use()
if __name__ == "__main__":
sys.exit(main())

5
python/2_CoreConcepts/cudaGraphs/requirements.txt Normal file
View File

@@ -0,0 +1,5 @@
# CUDA Graphs Sample Requirements
cuda-python>=13.0.0
cuda-core>=0.6.0
cupy-cuda13x>=13.0.0