SlotStream

Slotstream: SSD Expert Streaming Could Break the LLM Memory Wall

Slotstream, a new technique from singularitymoments.com, proposes using SSDs as an expert streaming layer to overcome the LLM memory wall. By offloading model experts to SSD and streaming them on demand, it aims to reduce the memory footprint of large models like Mixture-of-Experts (MoE) architectures. This approach could enable running models with hundreds of billions of parameters on a single consumer GPU, bypassing the need for expensive multi-GPU setups. The technique reportedly achieves near-HBM performance by leveraging SSD bandwidth and smart prefetching, potentially making large-scale inference more accessible.