NASP/cuda-samples
84
0
Fork 0
mirror of https://github.com/NVIDIA/cuda-samples.git synced 2026-10-13 00:08:26 +08:00
Code Issues Packages Projects Releases Wiki Activity

CUDA 13.4 samples update - v13.4-public

Browse Source 
Release 13.4 of the CUDA Samples supported by CUDA Toolkit 13.4.
See Changelog for more information.
This commit is contained in:
Dheemanth
2026-09-09 15:07:08 -07:00
committed by GitHub
parent b7c5481c55
commit 5443602d89
871 changed files with 6843 additions and 321849 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
python/2_CoreConcepts/blockwiseSum/README.md
View File

@@ -98,5 +98,5 @@ Done
## See Also
- [cuda.core Documentation](https://nvidia.github.io/cuda-python/cuda-core/latest/)
- [CUDA Shared Memory](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#shared-memory)
- [CUDA Shared Memory](https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/writing-cuda-kernels.html#shared-memory)
- [CuPy Documentation](https://docs.cupy.dev/)

7
python/2_CoreConcepts/cudaComputeLambdas/cudaComputeLambdas.py
View File

@@ -160,6 +160,13 @@ def main():
print_gpu_info(device)
print()
# cuda.compute allocates temporary storage from the device's default memory
# pool, which requires CUDA memory-pool support. This is not available on
# every platform (for example, Windows in TCC mode).
if not device.properties.memory_pools_supported:
print("CUDA memory pools are not supported on this platform.")
return 2
ok = True
ok &= demo_reduce_lambda()
print()

2
python/2_CoreConcepts/cudaGraphs/README.md
View File

@@ -137,4 +137,4 @@ your GPU and host CPU.
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [`cuda.core` graphs API](https://nvidia.github.io/cuda-python/cuda-core/latest/api.html#cuda-graphs)
- Upstream `cuda.core` example: [`cuda_graphs.py`](https://github.com/NVIDIA/cuda-python/blob/main/cuda_core/examples/cuda_graphs.py)
- [CUDA Graphs programming guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs)
- [CUDA Graphs programming guide](https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html#cuda-graphs)

2
python/2_CoreConcepts/greenContext/README.md
View File

@@ -246,5 +246,5 @@ other concurrent GPU work.
## See Also
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [Green Contexts in the CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#green-contexts)
- [Green Contexts in the CUDA Programming Guide](https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/green-contexts.html#green-contexts)
- [`cuda.core` green-context test suite](https://github.com/NVIDIA/cuda-python/blob/main/cuda_core/tests/test_green_context.py) - the authoritative API reference

8
python/2_CoreConcepts/launchConfigTuning/README.md
View File

@@ -94,10 +94,10 @@ elapsed_ms = (end_event - start_event) / n_iterations
The benchmark loops in this sample read kernel results back from
`ManagedMemoryResource` allocations between launches, which requires the
device property `concurrent_managed_access=True`. This is only supported on
Linux with HMM (Pascal and newer). On Windows (WDDM/MCDM/TCC) the property
is `False`, so the sample exits early with a waive message and exit code
`2`.
device property `concurrent_managed_access=True`. Devices that report `False`
— Windows (WDDM/MCDM/TCC), and older integrated GPUs such as Jetson Orin —
keep managed allocations GPU-exclusive, so the sample reports the missing
capability and exits with code `2`.
## Installation

15
python/2_CoreConcepts/launchConfigTuning/launchConfigTuning.py
View File

@@ -341,14 +341,6 @@ def main():
3. Benchmarking different thread block configurations
4. Finding optimal threads-per-block for various kernel types
"""
if sys.platform == "win32":
print(
"This sample relies on ManagedMemoryResource with concurrent host "
"access, which is not supported on Windows "
"(concurrent_managed_access=False). Waiving this sample."
)
sys.exit(2)
print("=" * 60)
print("Launch Configuration Tuning (cuda.core)")
print("Finding the Best Block Size for Your Kernel")
@@ -358,6 +350,13 @@ def main():
device = Device(0)
device.set_current()
# This sample builds host NumPy views over managed memory. Devices without
# concurrent managed access keep managed allocations GPU-exclusive while
# the GPU is active, so those host views fault.
if not device.properties.concurrent_managed_access:
print("Concurrent managed memory access is not supported on this platform.")
sys.exit(2)
# Print GPU information
print_gpu_info(device)

22
python/2_CoreConcepts/memoryResources/README.md
View File

@@ -72,15 +72,19 @@ verified on the host.
### Platform Support
The `ManagedMemoryResource` demo in this sample exercises **concurrent host
access** to managed allocations while the GPU is active, which requires the
device property `concurrent_managed_access=True`. This is only supported on
Linux with HMM (Pascal and newer). On Windows (WDDM/MCDM/TCC) the property
is `False`, so the sample exits early with a waive message and exit code
`2`. The `DeviceMemoryResource` + `PinnedMemoryResource` demos in this
sample would still work on Windows on their own, but to keep the sample
self-contained the entire script waives when concurrent managed access is
unavailable.
This sample needs two device capabilities, and exits with code `2` when
either is missing:
- `PinnedMemoryResource` is backed by a host memory pool, which requires
`host_memory_pools_supported=True`. Windows in TCC mode and older
integrated GPUs such as Jetson Orin report `False`.
- The `ManagedMemoryResource` demo builds host NumPy views over managed
allocations, which requires `concurrent_managed_access=True`. Windows
(WDDM/MCDM/TCC) and older integrated GPUs such as Jetson Orin report
`False`.
The demos are checked up front rather than individually to keep the sample
self-contained.
## Installation

21
python/2_CoreConcepts/memoryResources/memoryResources.py
View File

@@ -222,18 +222,23 @@ def main():
parser.add_argument("--device", type=int, default=0, help="CUDA device id")
args = parser.parse_args()
if sys.platform == "win32":
print(
"This sample relies on ManagedMemoryResource with concurrent host "
"access, which is not supported on Windows "
"(concurrent_managed_access=False). Waiving this sample."
)
sys.exit(2)
device = Device(args.device)
device.set_current()
print_gpu_info(device)
# PinnedMemoryResource is backed by a host memory pool, which is not
# available on every device.
if not device.properties.host_memory_pools_supported:
print("Host pinned memory pools are not supported on this platform.")
return 2
# This sample builds host NumPy views over managed memory. Devices without
# concurrent managed access keep managed allocations GPU-exclusive while
# the GPU is active, so those host views fault.
if not device.properties.concurrent_managed_access:
print("Concurrent managed memory access is not supported on this platform.")
return 2
stream = device.create_stream()
try:

4
python/2_CoreConcepts/parallelHistogram/README.md
View File

@@ -113,5 +113,5 @@ Test PASSED
## See Also
- [cuda.core Documentation](https://nvidia.github.io/cuda-python/cuda-core/latest/)
- [CUDA Atomic Functions](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#atomic-functions)
- [CUDA Shared Memory](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#shared-memory)
- [CUDA Atomic Functions](https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/writing-cuda-kernels.html#atomics)
- [CUDA Shared Memory](https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/writing-cuda-kernels.html#shared-memory)

16
python/2_CoreConcepts/parallelReduction/parallelReduction.py
View File

@@ -290,7 +290,7 @@ def benchmark_cuda_compute(
return result, float(np.mean(times))
def main() -> bool:
def main() -> int:
"""Main function demonstrating parallel reduction."""
print("=" * 70)
print("Parallel Reduction - Efficient GPU Array Summation")
@@ -298,6 +298,14 @@ def main() -> bool:
device = Device(0)
device.set_current()
# cuda.compute allocates temporary storage from the device's default memory
# pool, which requires CUDA memory-pool support. This is not available on
# every platform (for example, Windows in TCC mode).
if not device.properties.memory_pools_supported:
print("CUDA memory pools are not supported on this platform.")
return 2
stream = device.create_stream()
cp_stream = cp.cuda.Stream.from_external(stream)
@@ -363,13 +371,13 @@ def main() -> bool:
)
if custom_ok and compute_ok:
print("\nTest PASSED!")
return True
return 0
else:
print("\nTest FAILED - Error too large!")
return False
return 1
finally:
stream.close()
if __name__ == "__main__":
sys.exit(0 if main() else 1)
sys.exit(main())

175
python/2_CoreConcepts/persistentProgramCache/README.md Normal file
View File

@@ -0,0 +1,175 @@
# Persistent Program Cache (Python)
## Description
This sample demonstrates how to persist and reuse compiled CUDA artifacts
with `cuda.core`. It generates a specialized matmul + epilogue kernel,
compiles it to CUBIN on a cache miss, stores the artifact in
`FileStreamProgramCache`, and reloads the CUBIN on later runs.
## Core Cache Flow
The central cache logic is in `compile_or_load_kernel`:
```python
options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
program = Program(source, code_type="c++", options=options)
key = make_program_cache_key(...)
with FileStreamProgramCache(cache_dir) as cache:
cached = cache.get(key)
if cached is None:
module = program.compile("cubin")
cache[key] = module
status = "MISS"
else:
module = ObjectCode.from_cubin(cached, name="persistentProgramCache")
status = "HIT"
kernel = module.get_kernel(KERNEL_NAME)
```
## What You'll Learn
- Generating a CUDA source string from workload configuration
- Building a persistent cache key with `make_program_cache_key`
- Storing compiled CUBIN bytes with `FileStreamProgramCache`
- Reconstructing a loadable `ObjectCode` with `ObjectCode.from_cubin`
- Measuring cache lookup, compile, module load, and kernel execution time
- Validating that cached and freshly compiled artifacts produce the same result
## Key Libraries
- [`cuda.core`][cuda-core] - Pythonic access to CUDA programs, object code, launches, and events
- `cupy` - input and output buffers on the GPU
- `numpy` - deterministic input generation and host reference computation
## Key APIs
### From `cuda.core`
- `Program(...).compile("cubin")` - compile generated CUDA source to CUBIN
- `ObjectCode.from_cubin(...)` - reconstruct loadable object code from cached bytes
- `ObjectCode.get_kernel(name)` - fetch the kernel from the compiled artifact
- `LaunchConfig` and `launch(...)` - configure and launch the generated kernel
- `EventOptions(timing_enabled=True)` - time repeated kernel launches
### From `cuda.core.utils`
- `FileStreamProgramCache` - disk-backed, process-safe program cache
- `make_program_cache_key(...)` - derive a cache key from source, options, and target type
## Requirements
### Hardware
- NVIDIA GPU with Compute Capability 7.0 or higher
### Software
- CUDA Toolkit 13.0 or newer (matches `cuda-python` 13.x)
- Python 3.10 or newer
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=1.0.0)
- `cupy-cuda13x` (>=14.0.0)
- `numpy` (>=2.3.2)
## Installation
Install the required packages from `requirements.txt`:
```bash
cd /path/to/cuda-samples/python/2_CoreConcepts/persistentProgramCache
pip install -r requirements.txt
```
The `requirements.txt` installs:
- `cuda-python` (>=13.0.0)
- `cuda-core` (>=1.0.0)
- `cupy-cuda13x` (>=14.0.0)
- `numpy` (>=2.3.2)
## How to Run
### Basic usage
Run once with a cleared cache to force compilation:
```bash
cd cuda-samples/python/2_CoreConcepts/persistentProgramCache
python persistentProgramCache.py --clear-cache
```
Run again with the same configuration to reuse the cached artifact:
```bash
python persistentProgramCache.py
```
### With custom parameters
```bash
# Compile a different generated kernel variant
python persistentProgramCache.py --tile-size 32 --epilogue identity
# Use a custom matrix size
python persistentProgramCache.py --m 1024 --n 1024 --k 1024
# Use a specific GPU
python persistentProgramCache.py --device 1
```
Changing the tile size, epilogue, source code, compile options, or target
GPU architecture changes the cache key and produces a cache miss.
## Expected Output
The output includes the run configuration, cache status, timings, and
validation result:
```text
Persistent Program Cache
Device: <Your GPU Name>
Compute Capability: <X.Y>
Cache directory: <cache path>
Workload: C = relu(A @ B + bias)
Matrix sizes: M=512, N=512, K=512
Tile size: 16
Timed launches: warmup=5, iterations=20
Cache status: MISS | HIT
Cache key: <key prefix>...
Artifact size: <size> KiB
Cache lookup: <time> ms
Compile time: <time> ms | skipped
Cache store: <time> ms (MISS only)
Module load: <time> ms
Host prep time: <time> ms
Kernel time: <time> ms
Max error: <error>
Validation: PASSED
```
Run with `--clear-cache` first to force a miss, then run again with the same
configuration to get a hit. Compare `Host prep time`: the miss path compiles
and stores the CUBIN, while the hit path reloads the cached CUBIN.
**Note:** Device name, timing, artifact size, and cache key will vary based
on GPU, driver, CUDA Toolkit, and host system.
## Files
- `persistentProgramCache.py` - Python implementation using `cuda.core` program cache utilities
- `README.md` - This file
- `requirements.txt` - Sample dependencies
## See Also
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [`cuda.core` program caches API](
https://nvidia.github.io/cuda-python/cuda-core/latest/api.html#program-caches)
- [`cuda.core` examples](https://github.com/NVIDIA/cuda-python/tree/main/cuda_core/examples)
[cuda-core]: https://nvidia.github.io/cuda-python/cuda-core/latest/

419
python/2_CoreConcepts/persistentProgramCache/persistentProgramCache.py Normal file
View File

@@ -0,0 +1,419 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# * Neither the name of NVIDIA CORPORATION nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""
Persistent Program Cache with cuda.core
This sample demonstrates how to persist and reuse compiled CUDA artifacts
with cuda.core. A generated matmul + epilogue kernel is compiled to CUBIN
on a cache miss, stored in a disk-backed FileStreamProgramCache, and loaded
from that cache on later runs.
"""
from __future__ import annotations
import argparse
import sys
import time
from dataclasses import dataclass
from pathlib import Path
try:
import cupy as cp
import numpy as np
from cuda.core import (
Device,
EventOptions,
LaunchConfig,
ObjectCode,
Program,
ProgramOptions,
launch,
)
from cuda.core.utils import FileStreamProgramCache, make_program_cache_key
except ImportError as e:
print(f"Error: Required package not found: {e}")
print("Please install from requirements.txt:")
print(" pip install -r requirements.txt")
sys.exit(1)
KERNEL_NAME = "matmul_epilogue"
DEFAULT_CACHE_DIR = Path.home() / ".cache" / "cuda-samples" / "persistentProgramCache"
# Tiled matrix multiply followed by bias and an optional ReLU.
KERNEL_TEMPLATE = r"""
#define TILE_SIZE {tile_size}
extern "C" __global__
void {kernel_name}(const float* __restrict__ A,
const float* __restrict__ B,
const float* __restrict__ bias,
float* __restrict__ C,
int M, int N, int K)
{{
__shared__ float As[TILE_SIZE][TILE_SIZE];
__shared__ float Bs[TILE_SIZE][TILE_SIZE];
int tx = threadIdx.x;
int ty = threadIdx.y;
int row = blockIdx.y * TILE_SIZE + ty;
int col = blockIdx.x * TILE_SIZE + tx;
float acc = 0.0f;
int num_tiles = (K + TILE_SIZE - 1) / TILE_SIZE;
for (int tile = 0; tile < num_tiles; ++tile) {{
int a_col = tile * TILE_SIZE + tx;
int b_row = tile * TILE_SIZE + ty;
As[ty][tx] = (row < M && a_col < K) ? A[row * K + a_col] : 0.0f;
Bs[ty][tx] = (b_row < K && col < N) ? B[b_row * N + col] : 0.0f;
__syncthreads();
#pragma unroll
for (int i = 0; i < TILE_SIZE; ++i) {{
acc += As[ty][i] * Bs[i][tx];
}}
__syncthreads();
}}
if (row < M && col < N) {{
float value = acc + bias[col];
C[row * N + col] = {epilogue_expr};
}}
}}
"""
EPILOGUE_EXPRESSIONS = {
"identity": "value",
"relu": "value > 0.0f ? value : 0.0f",
}
# Return value from compile_or_load_kernel: the launchable kernel plus cache
# status, artifact size, and timings for comparing miss and hit paths.
@dataclass
class CacheResult:
kernel: object
status: str
key_hex: str
artifact_bytes: int
cache_lookup_ms: float
compile_ms: float | None
cache_store_ms: float | None
module_load_ms: float
@property
def host_preparation_ms(self) -> float:
"""Total cache lookup, compile/store, and module-load time."""
total = self.cache_lookup_ms + self.module_load_ms
if self.compile_ms is not None:
total += self.compile_ms
if self.cache_store_ms is not None:
total += self.cache_store_ms
return total
def elapsed_ms(start: float) -> float:
return (time.perf_counter() - start) * 1000.0
def build_kernel_source(tile_size: int, epilogue: str) -> str:
"""Generate a specialized matmul + epilogue CUDA kernel."""
return KERNEL_TEMPLATE.format(
tile_size=tile_size,
kernel_name=KERNEL_NAME,
epilogue_expr=EPILOGUE_EXPRESSIONS[epilogue],
)
def cache_clear(cache_dir: Path) -> None:
with FileStreamProgramCache(cache_dir) as cache:
cache.clear()
def compile_or_load_kernel(
source: str,
device: Device,
cache_dir: Path,
) -> CacheResult:
"""Compile the generated kernel or load its CUBIN bytes from cache."""
options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
program = Program(source, code_type="c++", options=options)
# For typical use, pass cache=cache to Program.compile(...). This sample
# uses explicit cache operations so the miss and hit paths are visible.
key = make_program_cache_key(
code=source,
code_type="c++",
options=options,
target_type="cubin",
)
key_hex = key.hex()
# This is the core persistent-cache flow: derive the same key for the same
# source/options/target, load cached CUBIN bytes on a hit, or compile and
# store the artifact on a miss.
with FileStreamProgramCache(cache_dir) as cache:
# These timings are included to compare cache misses and hits.
start = time.perf_counter()
cached = cache.get(key)
cache_lookup_ms = elapsed_ms(start)
if cached is not None:
start = time.perf_counter()
module = ObjectCode.from_cubin(cached, name="persistentProgramCache")
kernel = module.get_kernel(KERNEL_NAME)
module_load_ms = elapsed_ms(start)
return CacheResult(
kernel=kernel,
status="HIT",
key_hex=key_hex,
artifact_bytes=len(cached),
cache_lookup_ms=cache_lookup_ms,
compile_ms=None,
cache_store_ms=None,
module_load_ms=module_load_ms,
)
start = time.perf_counter()
module = program.compile("cubin")
compile_ms = elapsed_ms(start)
artifact_bytes = len(bytes(module.code))
start = time.perf_counter()
cache[key] = module
cache_store_ms = elapsed_ms(start)
start = time.perf_counter()
kernel = module.get_kernel(KERNEL_NAME)
module_load_ms = elapsed_ms(start)
return CacheResult(
kernel=kernel,
status="MISS",
key_hex=key_hex,
artifact_bytes=artifact_bytes,
cache_lookup_ms=cache_lookup_ms,
compile_ms=compile_ms,
cache_store_ms=cache_store_ms,
module_load_ms=module_load_ms,
)
def make_host_inputs(m: int, n: int, k: int) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
rng = np.random.default_rng(seed=123)
a = rng.standard_normal((m, k)).astype(np.float32)
b = rng.standard_normal((k, n)).astype(np.float32)
bias = rng.standard_normal(n).astype(np.float32)
return a, b, bias
def host_reference(
a: np.ndarray,
b: np.ndarray,
bias: np.ndarray,
epilogue: str,
) -> np.ndarray:
out = a @ b
out += bias.reshape(1, -1)
if epilogue == "relu":
out = np.maximum(out, 0.0)
return out.astype(np.float32, copy=False)
def print_cache_report(result: CacheResult) -> None:
print(f"Cache status: {result.status}")
print(f"Cache key: {result.key_hex[:24]}...")
print(f"Artifact size: {result.artifact_bytes / 1024:.1f} KiB")
print(f"Cache lookup: {result.cache_lookup_ms:.3f} ms")
if result.compile_ms is None:
print("Compile time: skipped")
else:
print(f"Compile time: {result.compile_ms:.3f} ms")
if result.cache_store_ms is not None:
print(f"Cache store: {result.cache_store_ms:.3f} ms")
print(f"Module load: {result.module_load_ms:.3f} ms")
print(f"Host prep time: {result.host_preparation_ms:.3f} ms")
def run_with_cache(
device_id: int,
m: int,
n: int,
k: int,
tile_size: int,
epilogue: str,
cache_dir: Path,
clear_cache: bool,
warmup: int,
iterations: int,
) -> bool:
if clear_cache:
cache_clear(cache_dir)
device = Device(device_id)
device.set_current()
cp.cuda.Device(device_id).use()
stream = device.create_stream()
cp.cuda.Stream.from_external(stream).use()
try:
print("\nPersistent Program Cache")
cc = device.compute_capability
print(f"Device: {device.name}")
print(f"Compute Capability: {cc.major}.{cc.minor}")
print(f"Cache directory: {cache_dir}")
print(f"Workload: C = {epilogue}(A @ B + bias)")
print(f"Matrix sizes: M={m}, N={n}, K={k}")
print(f"Tile size: {tile_size}")
print(f"Timed launches: warmup={warmup}, iterations={iterations}")
print()
source = build_kernel_source(tile_size=tile_size, epilogue=epilogue)
cache_result = compile_or_load_kernel(
source=source,
device=device,
cache_dir=cache_dir,
)
print_cache_report(cache_result)
host_a, host_b, host_bias = make_host_inputs(m, n, k)
expected = host_reference(host_a, host_b, host_bias, epilogue)
d_a = cp.asarray(host_a)
d_b = cp.asarray(host_b)
d_bias = cp.asarray(host_bias)
d_c = cp.empty((m, n), dtype=cp.float32)
stream.sync()
grid = ((n + tile_size - 1) // tile_size, (m + tile_size - 1) // tile_size)
block = (tile_size, tile_size)
config = LaunchConfig(grid=grid, block=block)
kernel_args = (
d_a.data.ptr,
d_b.data.ptr,
d_bias.data.ptr,
d_c.data.ptr,
np.int32(m),
np.int32(n),
np.int32(k),
)
for _ in range(warmup):
launch(stream, config, cache_result.kernel, *kernel_args)
stream.sync()
event_options = EventOptions(timing_enabled=True)
start_event = device.create_event(options=event_options)
end_event = device.create_event(options=event_options)
stream.record(start_event)
for _ in range(iterations):
launch(stream, config, cache_result.kernel, *kernel_args)
stream.record(end_event)
end_event.sync()
kernel_time_ms = (end_event - start_event) / iterations
stream.sync()
actual = cp.asnumpy(d_c)
max_error = float(np.max(np.abs(actual - expected)))
ok = bool(np.allclose(actual, expected, rtol=1e-3, atol=1e-3))
print(f"Kernel time: {kernel_time_ms:.3f} ms")
print(f"Max error: {max_error:.6f}")
print(f"Validation: {'PASSED' if ok else 'FAILED'}")
return ok
finally:
cp.cuda.Stream.null.use()
stream.close()
def positive_int(value: str) -> int:
parsed = int(value)
if parsed <= 0:
raise argparse.ArgumentTypeError("value must be positive")
return parsed
def cache_dir_arg(value: str) -> Path:
return Path(value).expanduser().resolve()
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Persistent cuda.core program cache for generated CUDA kernels"
)
parser.add_argument("--device", type=int, default=0, help="CUDA device id")
parser.add_argument("--m", type=positive_int, default=512, help="Rows of A and C")
parser.add_argument("--n", type=positive_int, default=512, help="Columns of B and C")
parser.add_argument("--k", type=positive_int, default=512, help="Columns of A / rows of B")
parser.add_argument(
"--tile-size",
type=int,
choices=(16, 32),
default=16,
help="Compile-time tile size specialization",
)
parser.add_argument(
"--epilogue",
choices=tuple(EPILOGUE_EXPRESSIONS),
default="relu",
help="Compile-time epilogue specialization",
)
parser.add_argument(
"--cache-dir",
type=cache_dir_arg,
default=DEFAULT_CACHE_DIR,
help=f"Program cache directory (default: {DEFAULT_CACHE_DIR})",
)
parser.add_argument("--clear-cache", action="store_true", help="Clear the cache before running")
parser.add_argument("--warmup", type=positive_int, default=5, help="Warmup launches")
parser.add_argument("--iterations", type=positive_int, default=20, help="Timed launches")
return parser.parse_args()
def main() -> int:
args = parse_args()
ok = run_with_cache(
device_id=args.device,
m=args.m,
n=args.n,
k=args.k,
tile_size=args.tile_size,
epilogue=args.epilogue,
cache_dir=args.cache_dir,
clear_cache=args.clear_cache,
warmup=args.warmup,
iterations=args.iterations,
)
return 0 if ok else 1
if __name__ == "__main__":
sys.exit(main())

6
python/2_CoreConcepts/persistentProgramCache/requirements.txt Normal file
View File

@@ -0,0 +1,6 @@
# Persistent Program Cache Sample Requirements
cuda-python>=13.0.0
cuda-core>=1.0.0
cupy-cuda13x>=14.0.0
numpy>=2.3.2

18
python/2_CoreConcepts/prefixSum/prefixSum.py
View File

@@ -53,14 +53,22 @@ except ImportError as e:
sys.exit(1)
def main() -> bool:
"""Run prefix sum sample. Returns True if all tests passed."""
def main() -> int:
"""Run prefix sum sample. Returns 0 if all tests passed."""
print("=" * 60)
print("Prefix Sum (Scan) - Using cuda.compute")
print("=" * 60)
device = Device(0)
device.set_current()
# cuda.compute allocates temporary storage from the device's default memory
# pool, which requires CUDA memory-pool support. This is not available on
# every platform (for example, Windows in TCC mode).
if not device.properties.memory_pools_supported:
print("CUDA memory pools are not supported on this platform.")
return 2
stream = device.create_stream()
cp_stream = cp.cuda.Stream.from_external(stream)
@@ -187,13 +195,11 @@ def main() -> bool:
print("• cuda.core Stream integrates with CuPy via Stream.from_external")
print("• Applications: stream compaction, radix sort, histograms")
print("=" * 60)
return ok
return 0 if ok else 1
finally:
cp.cuda.Stream.null.use()
stream.close()
if __name__ == "__main__":
success = main()
if not success:
sys.exit(1)
sys.exit(main())

14
python/2_CoreConcepts/processCheckpoint/README.md
View File

@@ -100,6 +100,20 @@ running ---------> locked ------------> checkpointed -----------> locked -------
- Python 3.10 or newer.
- `cuda-core >= 1.0.0`.
### Unsupported configurations
The sample returns 2 without running the lifecycle where the checkpoint
API is unavailable:
- Non-Linux platforms, since the checkpoint API is Linux-only.
- Integrated GPUs (Tegra / Jetson / Thor), where `Process.lock()` succeeds
and only `Process.checkpoint()` reports the missing support, leaving the
process locked and hanging on exit. The sample checks the device property
up front so that never happens.
- Any system where the driver answers the first `Process.lock()` with
`CUDA_ERROR_NOT_SUPPORTED`, which covers Confidential Computing mode and
vGPU guests. A failure later in the lifecycle is reported as an error.
## Installation
Install the required packages from `requirements.txt`:

42
python/2_CoreConcepts/processCheckpoint/processCheckpoint.py
View File

@@ -74,6 +74,18 @@ extern "C" __global__ void fill_pattern(float *out, unsigned long long n)
"""
# cuda.core raises a RuntimeError opening with this sentence whenever the
# driver has no usable checkpoint API. Match the whole sentence rather than
# a fragment, so an unrelated RuntimeError is never mistaken for it.
UNSUPPORTED_MESSAGE = (
"CUDA checkpointing is not supported by the installed NVIDIA driver."
)
class CheckpointUnsupportedError(Exception):
"""The driver rejects the checkpoint API on this system."""
@dataclass
class StepTiming:
label: str
@@ -123,7 +135,16 @@ def run_lifecycle(proc: checkpoint.Process, lock_timeout_ms: int) -> List[StepTi
"""
timings: List[StepTiming] = [StepTiming("initial", 0.0, proc.state)]
ms = _time_call(proc.lock, timeout_ms=lock_timeout_ms)
try:
ms = _time_call(proc.lock, timeout_ms=lock_timeout_ms)
except RuntimeError as exc:
# The driver returns CUDA_ERROR_NOT_SUPPORTED where checkpointing
# is unavailable, for example under Confidential Computing or in a
# vGPU guest. Only the first call is treated this way: a later
# failure means checkpointing started and then broke.
if not str(exc).startswith(UNSUPPORTED_MESSAGE):
raise
raise CheckpointUnsupportedError from exc
timings.append(StepTiming("lock", ms, proc.state))
ms = _time_call(proc.checkpoint)
@@ -178,8 +199,8 @@ def main():
args = parser.parse_args()
if sys.platform != "linux":
print("Error: CUDA process checkpointing is Linux-only.")
return 1
print("CUDA process checkpointing is Linux-only.")
return 2
if args.buffer_mib <= 0:
print("Error: --buffer-mib must be positive")
@@ -197,10 +218,11 @@ def main():
# CUDA process checkpointing relies on kernel-mode driver features
# that aren't shipped on integrated-GPU platforms (e.g. Tegra /
# Jetson / Thor). On those, Process.lock() can hang indefinitely
# instead of returning a clean "not supported" error. Skip cleanly
# rather than hanging. Remove this guard once integrated platforms
# gain checkpoint support.
# Jetson / Thor). There Process.lock() still succeeds and only
# Process.checkpoint() reports the missing support, which leaves the
# process locked and hanging on exit, so the check below cannot cover
# it. Remove this guard once integrated platforms gain checkpoint
# support.
if device.properties.integrated:
print(
f"CUDA process checkpointing is not supported on integrated "
@@ -233,7 +255,11 @@ def main():
print()
print("Running checkpoint lifecycle on self ...")
proc = checkpoint.Process(os.getpid())
timings = run_lifecycle(proc, args.lock_timeout_ms)
try:
timings = run_lifecycle(proc, args.lock_timeout_ms)
except CheckpointUnsupportedError:
print("CUDA process checkpointing is not supported on this system.")
return 2
print_timings(timings)
hash_after = hash_device_buffer(device_buffer, host)

3
python/2_CoreConcepts/processCheckpoint/requirements.txt
View File

@@ -1,4 +1,3 @@
# cuda-bindings 13.3.0 drops the CUcheckpointRestoreArgs alias that cuda-core requires
cuda-python>=13.0.0,<13.3.0
cuda-python>=13.3.1
cuda-core>=1.0.0
numpy>=2.3.2

2
python/2_CoreConcepts/reductionMultiBlockCG/README.md
View File

@@ -132,7 +132,7 @@ Single-Pass Multi-Block Reduction completed successfully!
## Further reading
- [CUDA Cooperative Groups](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cooperative-groups)
- [CUDA Cooperative Groups](https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cooperative-groups.html#cooperative-groups)
- [Reduction whitepaper (PDF)](https://developer.download.nvidia.com/assets/cuda/files/reduction.pdf)
## Files

10
python/2_CoreConcepts/streamingCopyComputeOverlap/streamingCopyComputeOverlap.py
View File

@@ -96,6 +96,13 @@ def main():
print()
print_gpu_info(device)
# This sample stages transfers through pinned memory, which is backed by a
# host memory pool and requires CUDA memory-pool support. This is not
# available on every platform (for example, Windows in TCC mode).
if not device.properties.host_memory_pools_supported:
print("Host pinned memory pools are not supported on this platform.")
return 2
# Compile kernel
arch = f"sm_{device.arch}"
program = Program(
@@ -306,7 +313,8 @@ def main():
print("\nNote: Speedup depends on hardware characteristics. This technique")
print("benefits most when transfer time is significant relative to compute.")
print("=" * 60)
return 0
if __name__ == "__main__":
main()
sys.exit(main())

2
python/2_CoreConcepts/tmaTensorMap/README.md
View File

@@ -134,5 +134,5 @@ replace_address verified: descriptor reused with new source tensor
## See Also
- [CUDA Python Documentation](https://nvidia.github.io/cuda-python/)
- [TMA in the CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#tensor-memory-accelerator)
- [TMA in the CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/async-copies.html#using-the-tensor-memory-accelerator-tma)
- [`cuda::barrier` reference](https://nvidia.github.io/cccl/libcudacxx/extended_api/synchronization_primitives/barrier.html)