A new round of benchmarking shows that Alibaba's Qwen 3.8 27B open-weight AI model, despite its promising intelligence scores and reasonable VRAM footprint of roughly 17GB in four-bit quantized form, fails to deliver usable performance on Nvidia's flagship RTX 5090 GPU. The culprit is not memory capacity but severe bottlenecks in the software and inference engine stack. With time-to-first-token stretching to roughly 30 minutes on the RTX 5090 using llama.cpp, the test results underscore a growing disconnect between raw hardware specs and real-world local AI performance.

What You Need to Know

Alibaba's Qwen 3.8 27B is a dense 27-billion-parameter model with built-in multimodal capabilities. Its four-bit quantized version fits within the VRAM of high-end desktop GPUs, including the RTX 5090, RTX 4090, and RTX 3090. However, inference engines designed for older dense models struggle with Qwen's architecture, leading to extremely slow processing. The model's multi-token prediction feature, or MTP, is also not universally supported across all runtimes and hardware configurations.

The Software Bottleneck on RTX 5090

Tom's Hardware testbed, built around a Ryzen 7 9800X3D and 64GB of DDR5 memory, attempted to run Qwen 3.8 27B on an RTX 5090 using both llama.cpp and vLLM. The llama.cpp results were dire: time-to-first-token exceeded 30 minutes, and tokens-per-second throughput remained far below expectations for a card with 32GB of GDDR7 and 1.8 TB/s of memory bandwidth. This suggests a fundamental software incompatibility rather than a hardware limitation. Although llama.cpp can allocate the full 262K context window on the RTX 5090, processing speeds at long contexts are unusable for practical workloads.

  • llama.cpp: Extremely slow time-to-first-token (over 30 minutes) and poor throughput on RTX 5090; not suitable for Qwen 3.8 27B.
  • vLLM: Production-grade engine but requires substantial host memory; even with 64GB of system RAM, performance may still be limited.
  • MTP support: Multi-token prediction could not be enabled across all platforms due to VRAM constraints and engine compatibility.

Comparing Hardware Platforms: Discrete vs Unified Memory

The test suite extended beyond discrete GPUs to systems with unified memory architectures, including the DGX Spark, Mac Studio, and Ryzen AI Halo. These platforms typically handle dense models more gracefully because they avoid the CPU-GPU data transfer bottleneck. However, even on these systems, the inference engine remains the decisive factor. The DGX Spark, for example, relies on a tailored Linux environment and specific driver stacks that may not yet be fully optimized for Qwen 3.8 27B. Similarly, the Mac Studio's unified memory can offer ample capacity, but its bandwidth and software ecosystem (often using Apple's MLX or Core ML) impose different trade-offs.

Why This Matters

The findings directly affect anyone investing in high-end consumer GPUs for local AI inference. The assumption that more VRAM alone guarantees strong performance is now clearly false. Dense models like Qwen 3.8 27B demand inference engines that exploit GPU compute capabilities effectively, and current open-source solutions lag behind. For users, this means that upgrading to an RTX 5090 may not deliver better local AI performance until software catches up. Meanwhile, unified memory platforms such as the Mac Studio and DGX Spark can sometimes offer a smoother experience, though they too are limited by inference engine maturity. The broader implication is that the local AI ecosystem needs to prioritize software optimization as much as hardware capacity, especially as models grow more complex and users expect production-level throughput.

What You Need to Know (Continued)

For now, anyone looking to run Qwen 3.8 27B locally should test multiple inference engines and be prepared for significant performance variability. The model's MTP feature, when supported, can improve throughput, but it requires careful VRAM management. Tom's Hardware plans to continue evaluating software stacks as they evolve, but the initial benchmark data is a clear warning: VRAM is not the answer if the software cannot use it.