9.8 KiB
API参考资料 https://docs.ray.io/en/latest/serve/api/doc/ray.serve.llm.LLMConfig.html
ray.serve.llm.LLMConfig pydantic model ray.serve.llm.LLMConfig[source] The configuration for starting an LLM deployment.
PublicAPI (alpha): This API is in alpha and may change before becoming stable.
field accelerator_type: str | None = None The type of accelerator runs the model on. Only the following values are supported: [‘V100’, ‘P100’, ‘T4’, ‘P4’, ‘K80’, ‘A10G’, ‘L4’, ‘L40S’, ‘A100’, ‘H100’, ‘H200’, ‘H20’, ‘B200’, ‘Intel-GPU-Max-1550’, ‘Intel-GPU-Max-1100’, ‘Intel-GAUDI’, ‘AMD-Instinct-MI100’, ‘AMD-Instinct-MI250X’, ‘AMD-Instinct-MI250X-MI250’, ‘AMD-Instinct-MI210’, ‘AMD-Instinct-MI300A’, ‘AMD-Instinct-MI300X-OAM’, ‘AMD-Instinct-MI300X-HF’, ‘AMD-Instinct-MI308X’, ‘AMD-Instinct-MI325X-OAM’, ‘AMD-Instinct-MI350X-OAM’, ‘AMD-Instinct-MI355X-OAM’, ‘AMD-Radeon-R9-200-HD-7900’, ‘AMD-Radeon-HD-7900’, ‘aws-neuron-core’, ‘TPU-V2’, ‘TPU-V3’, ‘TPU-V4’, ‘TPU-V5P’, ‘TPU-V5LITEPOD’, ‘TPU-V6E’, ‘Ascend910B’, ‘Ascend910B4’, ‘MXC500’, ‘MXC550’, ‘A100-40G’, ‘A100-80G’]
field callback_config: CallbackConfig [Optional] Callback configuration to use for model initialization. Can be a string path to a class or a Callback subclass.
field deployment_config: Dict[str, Any] [Optional] The Ray @server.deployment options. Supported fields are: name, num_replicas, ray_actor_options, max_ongoing_requests, autoscaling_config, max_queued_requests, user_config, health_check_period_s, health_check_timeout_s, graceful_shutdown_wait_loop_s, graceful_shutdown_timeout_s, logging_config, request_router_config. For more details, see the Ray Serve Documentation.
field engine_kwargs: Dict[str, Any] = {} Additional keyword arguments for the engine. In case of vLLM, this will include all the configuration knobs they provide out of the box, except for tensor-parallelism which is set automatically from Ray Serve configs.
field experimental_configs: Dict[str, Any] [Optional] Experimental configurations for Ray Serve LLM. This is a dictionary of key-value pairs. Current supported keys are: - stream_batching_interval_ms: Ray Serve LLM batches streaming requests together. This config decides how long to wait for the batch before processing the requests. Defaults to 50.0. - num_ingress_replicas: The number of replicas for the router. Ray Serve will take the max amount all the replicas. Default would be 2 router replicas per model replica.
field llm_engine: str = 'vLLM' The LLMEngine that should be used to run the model. Only the following values are supported: [‘vLLM’]
field log_engine_metrics: bool | None = True Enable additional engine metrics via Ray Prometheus port.
field lora_config: Dict[str, Any] | LoraConfig | None = None Settings for LoRA adapter. Validated against LoraConfig.
field model_loading_config: Dict[str, Any] | ModelLoadingConfig [Required] The settings for how to download and expose the model. Validated against ModelLoadingConfig.
field placement_group_config: Dict[str, Any] | None = None Ray placement group configuration for scheduling vLLM engine workers. Defines resource bundles and placement strategy for multi-node deployments. Should contain ‘bundles’ (list of resource dicts) and optionally ‘strategy’ (defaults to ‘PACK’). Example: {‘bundles’: [{‘GPU’: 1, ‘CPU’: 2}], ‘strategy’: ‘PACK’}
field runtime_env: Dict[str, Any] | None = None The runtime_env to use for the model deployment replica and the engine workers.
apply_checkpoint_info(model_id_or_path: str, trust_remote_code: bool = False) → None[source] Apply the checkpoint info to the model config.
classmethod from_file(path: str, **kwargs) → ModelT Load a model from a YAML file path.
get_engine_config() → None | VLLMEngineConfig[source] Returns the engine config for the given LLM config.
LLMConfig not only has engine config but also deployment config, etc.
get_or_create_callback() → CallbackBase | None[source] Get or create the callback instance for this process.
This ensures one callback instance per process (singleton pattern). The instance is cached so the same object is used across all hooks.
Returns : Instance of class that implements Callback
multiplex_config() → ServeMultiplexConfig[source] classmethod parse_yaml(file, **kwargs) → ModelT setup_engine_backend()[source] update_engine_kwargs(**kwargs: Any) → None[source] Update the engine_kwargs and the engine_config engine_kwargs.
This is typically called during engine starts, when certain engine_kwargs (e.g., data_parallel_rank) become available.
validator validate_accelerator_type » accelerator_type[source] validator validate_deployment_config » deployment_config[source] Validates the deployment config dictionary.
validator validate_experimental_configs » experimental_configs[source] Validates the experimental configs dictionary.
validator validate_llm_engine » llm_engine[source] Validates the llm_engine string value.
validator validate_lora_config » lora_config[source] Validates the lora config dictionary.
validator validate_model_loading_config » model_loading_config[source] Validates the model loading config dictionary.
property input_modality: str Returns the input modality of the model. There could be more types in the future. Right now assumes if the model doesn’t support version, it’ll be text.
property max_request_context_length: int | None property model_architecture: str property model_id: str property supports_vision: bool
Python API
ray serve api https://docs.ray.io/en/latest/serve/api/index.html#serve-api
Python API Writing Applications serve.Deployment
Class (or function) decorated with the @serve.deployment decorator.
serve.Application
One or more deployments bound with arguments that can be deployed together.
Deployment Decorators serve.deployment
Decorator that converts a Python class to a Deployment.
serve.ingress
Wrap a deployment class with an ASGI application for HTTP request parsing.
serve.batch
Converts a function to asynchronously handle batches.
serve.multiplexed
Wrap a callable or method used to load multiplexed models in a replica.
Deployment Handles Note
The deprecated RayServeHandle and RayServeSyncHandle APIs have been fully removed as of Ray 2.10. See the model composition guide for how to update code to use the DeploymentHandle API instead.
serve.handle.DeploymentHandle
A handle used to make requests to a deployment at runtime.
serve.handle.DeploymentResponse
A future-like object wrapping the result of a unary deployment handle call.
serve.handle.DeploymentResponseGenerator
A future-like object wrapping the result of a streaming deployment handle call.
Running Applications serve.start
Start Serve on the cluster.
serve.run
Run an application and return a handle to its ingress deployment.
serve.delete
Delete an application by its name.
serve.status
Get the status of Serve on the cluster.
serve.shutdown
Completely shut down Serve on the cluster.
serve.shutdown_async
Completely shut down Serve on the cluster asynchronously.
Configurations serve.config.ProxyLocation
Config for where to run proxies to receive ingress traffic to the cluster.
serve.config.gRPCOptions
gRPC options for the proxies.
serve.config.HTTPOptions
HTTP options for the proxies.
serve.config.AutoscalingConfig
Config for the Serve Autoscaler.
serve.config.AutoscalingPolicy
PublicAPI (alpha): This API is in alpha and may change before becoming stable.
serve.config.AutoscalingContext
Rich context provided to custom autoscaling policies.
serve.config.AggregationFunction
An enumeration.
serve.config.RequestRouterConfig
Config for the Serve request router.
Schemas serve.schema.ServeActorDetails
Detailed info about a Ray Serve actor.
serve.schema.ProxyDetails
Detailed info about a Ray Serve ProxyActor.
serve.schema.ApplicationStatusOverview
Describes the status of an application and all its deployments.
serve.schema.ServeStatus
Describes the status of Serve.
serve.schema.DeploymentStatusOverview
Describes the status of a deployment.
serve.schema.EncodingType
Encoding type for the serve logs.
serve.schema.AutoscalingMetricsHealth
An enumeration.
serve.schema.AutoscalingStatus
An enumeration.
serve.schema.ScalingDecision
One autoscaling decision with minimal provenance.
serve.schema.DeploymentAutoscalingDetail
Deployment-level autoscaler observability.
serve.schema.ReplicaRank
Replica rank model.
Request Router serve.request_router.ReplicaID
A unique identifier for a replica.
serve.request_router.PendingRequest
A request that is pending execution by a replica.
serve.request_router.RunningReplica
Contains info on a running replica.
serve.request_router.FIFOMixin
Mixin for FIFO routing.
serve.request_router.LocalityMixin
Mixin for locality routing.
serve.request_router.MultiplexMixin
Mixin for multiplex routing.
serve.request_router.RequestRouter
Abstract interface for a request router (how the router calls it).
Advanced APIs serve.get_replica_context
Returns the deployment and replica tag from within a replica at runtime.
serve.context.ReplicaContext
Stores runtime context info for replicas.
serve.get_multiplexed_model_id
Get the multiplexed model ID for the current request.
serve.get_app_handle
Get a handle to the application's ingress deployment by name.
serve.get_deployment_handle
Get a handle to a deployment by name.
serve.grpc_util.RayServegRPCContext
Context manager to set and get gRPC context.
serve.exceptions.BackPressureError
Raised when max_queued_requests is exceeded on a DeploymentHandle.
serve.exceptions.RayServeException
serve.exceptions.RequestCancelledError
Raise when a Serve request is cancelled.
serve.exceptions.DeploymentUnavailableError
Raised when a Serve deployment is unavailable to receive requests.