mirror of https://github.com/NVIDIA/cuda-samples.git synced 2026-09-12 01:06:53 +08:00

Release 13.4 of the CUDA Samples supported by CUDA Toolkit 13.4.
See Changelog for more information.

2026-09-09 17:07:08 -05:00

5.1 KiB

Raw Blame History

Persistent Program Cache (Python)

Description

This sample demonstrates how to persist and reuse compiled CUDA artifacts with cuda.core. It generates a specialized matmul + epilogue kernel, compiles it to CUBIN on a cache miss, stores the artifact in FileStreamProgramCache, and reloads the CUBIN on later runs.

Core Cache Flow

The central cache logic is in compile_or_load_kernel:

options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
program = Program(source, code_type="c++", options=options)
key = make_program_cache_key(...)

with FileStreamProgramCache(cache_dir) as cache:
    cached = cache.get(key)

    if cached is None:
        module = program.compile("cubin")
        cache[key] = module
        status = "MISS"
    else:
        module = ObjectCode.from_cubin(cached, name="persistentProgramCache")
        status = "HIT"

kernel = module.get_kernel(KERNEL_NAME)

What You'll Learn

Generating a CUDA source string from workload configuration
Building a persistent cache key with make_program_cache_key
Storing compiled CUBIN bytes with FileStreamProgramCache
Reconstructing a loadable ObjectCode with ObjectCode.from_cubin
Measuring cache lookup, compile, module load, and kernel execution time
Validating that cached and freshly compiled artifacts produce the same result

Key Libraries

cuda.core - Pythonic access to CUDA programs, object code, launches, and events
cupy - input and output buffers on the GPU
numpy - deterministic input generation and host reference computation

Key APIs

From `cuda.core`

Program(...).compile("cubin") - compile generated CUDA source to CUBIN
ObjectCode.from_cubin(...) - reconstruct loadable object code from cached bytes
ObjectCode.get_kernel(name) - fetch the kernel from the compiled artifact
LaunchConfig and launch(...) - configure and launch the generated kernel
EventOptions(timing_enabled=True) - time repeated kernel launches

From `cuda.core.utils`

FileStreamProgramCache - disk-backed, process-safe program cache
make_program_cache_key(...) - derive a cache key from source, options, and target type

Requirements

Hardware

NVIDIA GPU with Compute Capability 7.0 or higher

Software

CUDA Toolkit 13.0 or newer (matches cuda-python 13.x)
Python 3.10 or newer
cuda-python (>=13.0.0)
cuda-core (>=1.0.0)
cupy-cuda13x (>=14.0.0)
numpy (>=2.3.2)

Installation

Install the required packages from requirements.txt:

cd /path/to/cuda-samples/python/2_CoreConcepts/persistentProgramCache
pip install -r requirements.txt

The requirements.txt installs:

cuda-python (>=13.0.0)
cuda-core (>=1.0.0)
cupy-cuda13x (>=14.0.0)
numpy (>=2.3.2)

How to Run

Basic usage

Run once with a cleared cache to force compilation:

cd cuda-samples/python/2_CoreConcepts/persistentProgramCache
python persistentProgramCache.py --clear-cache

Run again with the same configuration to reuse the cached artifact:

python persistentProgramCache.py

With custom parameters

# Compile a different generated kernel variant
python persistentProgramCache.py --tile-size 32 --epilogue identity

# Use a custom matrix size
python persistentProgramCache.py --m 1024 --n 1024 --k 1024

# Use a specific GPU
python persistentProgramCache.py --device 1

Changing the tile size, epilogue, source code, compile options, or target GPU architecture changes the cache key and produces a cache miss.

Expected Output

The output includes the run configuration, cache status, timings, and validation result:

Persistent Program Cache
Device:             <Your GPU Name>
Compute Capability: <X.Y>
Cache directory:    <cache path>
Workload:           C = relu(A @ B + bias)
Matrix sizes:       M=512, N=512, K=512
Tile size:          16
Timed launches:     warmup=5, iterations=20

Cache status:       MISS | HIT
Cache key:          <key prefix>...
Artifact size:      <size> KiB
Cache lookup:       <time> ms
Compile time:       <time> ms | skipped
Cache store:        <time> ms          (MISS only)
Module load:        <time> ms
Host prep time:     <time> ms
Kernel time:        <time> ms
Max error:          <error>
Validation:         PASSED

Run with --clear-cache first to force a miss, then run again with the same configuration to get a hit. Compare Host prep time: the miss path compiles and stores the CUBIN, while the hit path reloads the cached CUBIN.

Note: Device name, timing, artifact size, and cache key will vary based on GPU, driver, CUDA Toolkit, and host system.

Files

persistentProgramCache.py - Python implementation using cuda.core program cache utilities
README.md - This file
requirements.txt - Sample dependencies

5.1 KiB Raw Blame History