Organizations running multiple custom AI models often face the expensive problem of idle GPU capacity, where hardware sits underutilized because individual models don't have enough traffic to justify dedicated compute resources. To solve this, AWS has partnered with the vLLM community to refine Multi-Low-Rank Adaptation (Multi-LoRA) for Mixture of Experts (MoE) models like GPT-OSS and Qwen. This approach keeps the heavy base model frozen while swapping tiny, specialized "adapters" in and out of the GPU memory on the fly to meet specific user requests.

The technical breakthrough involves a new "fused_moe_lora" kernel that manages two layers of sparsity simultaneously: expert routing, which directs data to specialized parts of the model, and adapter selection, which chooses the right customization for the specific task. By optimizing how these "skinny" matrices are processed on the GPU hardware, the team eliminated significant performance bottlenecks. They also introduced "Programmatic Dependent Launch," a feature that allows a second computational task to begin preparing while the first is still finishing, effectively removing the idle "bubbles" that typically slow down AI response times.

The results are tangible for developers using Amazon SageMaker AI or Amazon Bedrock. Benchmarking on the GPT-OSS 20B model showed a 19% increase in output speed and an 8% reduction in the time it takes for the model to start generating text (Time to First Token). By allowing five or more customers to share the same GPU without performance degradation, this update transforms underutilized hardware into a streamlined, cost-effective engine for personalized, large-scale AI applications.

Organizations running multiple custom AI models often face the expensive problem of idle GPU capacity, where hardware sits underutilized because individual models don't have enough traffic to justify dedicated compute resources. To solve this, AWS has partnered with the vLLM community to refine Multi-Low-Rank Adaptation (Multi-LoRA) for Mixture of Experts (MoE) models like GPT-OSS and Qwen. This approach keeps the heavy base model frozen while swapping tiny, specialized "adapters" in and out of the GPU memory on the fly to meet specific user requests.

The technical breakthrough involves a new "fused_moe_lora" kernel that manages two layers of sparsity simultaneously: expert routing, which directs data to specialized parts of the model, and adapter selection, which chooses the right customization for the specific task. By optimizing how these "skinny" matrices are processed on the GPU hardware, the team eliminated significant performance bottlenecks. They also introduced "Programmatic Dependent Launch," a feature that allows a second computational task to begin preparing while the first is still finishing, effectively removing the idle "bubbles" that typically slow down AI response times.

The results are tangible for developers using Amazon SageMaker AI or Amazon Bedrock. Benchmarking on the GPT-OSS 20B model showed a 19% increase in output speed and an 8% reduction in the time it takes for the model to start generating text (Time to First Token). By allowing five or more customers to share the same GPU without performance degradation, this update transforms underutilized hardware into a streamlined, cost-effective engine for personalized, large-scale AI applications.