cuda-samples/python/2_CoreConcepts/persistentProgramCache/README.md

Persistent Program Cache (Python)

Description

This sample demonstrates how to persist and reuse compiled CUDA artifacts with cuda.core. It generates a specialized matmul + epilogue kernel, compiles it to CUBIN on a cache miss, stores the artifact in FileStreamProgramCache, and reloads the CUBIN on later runs.

Core Cache Flow

The central cache logic is in compile_or_load_kernel:

options = ProgramOptions(std="c++17", arch=f"sm_{device.arch}")
program = Program(source, code_type="c++", options=options)
key = make_program_cache_key(...)

with FileStreamProgramCache(cache_dir) as cache:
    cached = cache.get(key)

    if cached is None:
        module = program.compile("cubin")
        cache[key] = module
        status = "MISS"
    else:
        module = ObjectCode.from_cubin(cached, name="persistentProgramCache")
        status = "HIT"

kernel = module.get_kernel(KERNEL_NAME)

What You'll Learn

• Generating a CUDA source string from workload configuration
• Building a persistent cache key with make_program_cache_key
• Storing compiled CUBIN bytes with FileStreamProgramCache
• Reconstructing a loadable ObjectCode with ObjectCode.from_cubin
• Measuring cache lookup, compile, module load, and kernel execution time
• Validating that cached and freshly compiled artifacts produce the same result

Key Libraries

• cuda.core - Pythonic access to CUDA programs, object code, launches, and events
• cupy - input and output buffers on the GPU
• numpy - deterministic input generation and host reference computation

Key APIs

From cuda.core

• Program(...).compile("cubin") - compile generated CUDA source to CUBIN
• ObjectCode.from_cubin(...) - reconstruct loadable object code from cached bytes
• ObjectCode.get_kernel(name) - fetch the kernel from the compiled artifact
• LaunchConfig and launch(...) - configure and launch the generated kernel
• EventOptions(timing_enabled=True) - time repeated kernel launches

From cuda.core.utils

• FileStreamProgramCache - disk-backed, process-safe program cache
• make_program_cache_key(...) - derive a cache key from source, options, and target type

Requirements

Hardware

• NVIDIA GPU with Compute Capability 7.0 or higher

Software

• CUDA Toolkit 13.0 or newer (matches cuda-python 13.x)
• Python 3.10 or newer
• cuda-python (>=13.0.0)
• cuda-core (>=1.0.0)
• cupy-cuda13x (>=14.0.0)
• numpy (>=2.3.2)

Installation

Install the required packages from requirements.txt:

cd /path/to/cuda-samples/python/2_CoreConcepts/persistentProgramCache
pip install -r requirements.txt

The requirements.txt installs:

• cuda-python (>=13.0.0)
• cuda-core (>=1.0.0)
• cupy-cuda13x (>=14.0.0)
• numpy (>=2.3.2)

How to Run

Basic usage

Run once with a cleared cache to force compilation:

cd cuda-samples/python/2_CoreConcepts/persistentProgramCache
python persistentProgramCache.py --clear-cache

Run again with the same configuration to reuse the cached artifact:

python persistentProgramCache.py

With custom parameters

# Compile a different generated kernel variant
python persistentProgramCache.py --tile-size 32 --epilogue identity

# Use a custom matrix size
python persistentProgramCache.py --m 1024 --n 1024 --k 1024

# Use a specific GPU
python persistentProgramCache.py --device 1

Changing the tile size, epilogue, source code, compile options, or target GPU architecture changes the cache key and produces a cache miss.

Expected Output

The output includes the run configuration, cache status, timings, and validation result:

Persistent Program Cache
Device:             <Your GPU Name>
Compute Capability: <X.Y>
Cache directory:    <cache path>
Workload:           C = relu(A @ B + bias)
Matrix sizes:       M=512, N=512, K=512
Tile size:          16
Timed launches:     warmup=5, iterations=20

Cache status:       MISS | HIT
Cache key:          <key prefix>...
Artifact size:      <size> KiB
Cache lookup:       <time> ms
Compile time:       <time> ms | skipped
Cache store:        <time> ms          (MISS only)
Module load:        <time> ms
Host prep time:     <time> ms
Kernel time:        <time> ms
Max error:          <error>
Validation:         PASSED

Run with --clear-cache first to force a miss, then run again with the same configuration to get a hit. Compare Host prep time: the miss path compiles and stores the CUBIN, while the hit path reloads the cached CUBIN.

Note: Device name, timing, artifact size, and cache key will vary based on GPU, driver, CUDA Toolkit, and host system.

Files

• persistentProgramCache.py - Python implementation using cuda.core program cache utilities
• README.md - This file
• requirements.txt - Sample dependencies

See Also

• CUDA Python Documentation
• cuda.core program caches API
• cuda.core examples