SambaNova’s SN50 RDU for AI at Hot Chips 2026
SambaNova SN50 Hot Chips 2026 SN50 Dataflow
For as new as the dedicated AI accelerator field is, SambaNova is one of the older and more established hardware vendors. The company is now in the fifth generation of their reconfigurable dataflow unit (RDU) technology, with the SN50 that was launched earlier this year. As with the other major AI vendors at this year’s Hot Chips conference, the company has come to present new technical details on SN50, and outline what makes it competitive in the burgeoning field of dedicated AI accelerators.
SambaNova’s hardware has taken on an increased prominence in the industry thanks to the company’s connection to Intel. While Intel itself is still trying to catch up on AI accelerators, the company has become increasingly attached to the hip to SambaNova, whose RDUs provide the dedicated, high-efficiency and low-latency AI accelerators that round out Intel’s hardware stack. Thus the company’s progress with the SN50 (and future RDUs) is material not just for SambaNova, but for Intel as well.
SambaNova SN50 Hot Chips 2026 Inference Decode
Setting the stage, agentic inference is all the rage right now. Where does all the execution time go? SambaNova has a breakdown of it. Most time is spent in decode, especially on DeepSeek V3 where it’s 97% of the time, versus 3% for prefill.
SambaNova SN50 Hot Chips 2026 Decode Bandwidth Bound
And decode, in turn, is bandwidth-bound. The FLOPS-per-byte ratio is quite low, even for large batch sizes.
SambaNova SN50 Hot Chips 2026 HBM Bandwidth Utilization
Bandwidth utilization is often misunderstood. SambaNova is laying out what they mean for this talk. In short, they aren’t talking about just how much HBM bandwidth is being used, but rather the Model Bandwidth Utilization (MBU) model. And specifically, what fraction of that is being used to cache data and otherwise handle data usage.
SambaNova SN50 Hot Chips 2026 Model Bandwidth Utilization
Looking at the current state of tech, GPUs offer low bandwidth usage, even with highly optimized GPU-friendly benchmarks.
SambaNova SN50 Hot Chips 2026 GPU Scaling
Things get worse for GPUs when you scale up the number of them; performance does go up, but MBU drops significantly.
SambaNova SN50 Hot Chips 2026 Frontier Models
Meanwhile frontier models require being able to scale up.
SambaNova SN50 Hot Chips 2026 Power Capacity
Again with a GPU example, a GPU can get to around 30TB/second of model bandwidth. But they can’t get past that.
SambaNova SN50 Hot Chips 2026 SN50 Dataflow
Enter SambaNova’s SN50 dataflow RBU. They have doubled-down on what worked well from SN40, such as the large on-chip SRAM. 5x as many FLOPS as SN40, and it is designed to scale-up to a much larger domain of 256+ chips. And there is a separate scale-out network using 400Gb networking.
Notably, there are no I/O dies or similar here. Instead it is just two max reticle dies for the logic, and then HBM stacks for the memory.
Though it is interesting that SambaNova’s choice of HBM here is quite dated; SN50 still uses HBM2e here (which is going to be a problem in the future as production of the memory is already ramping down).
SambaNova SN50 Hot Chips 2026 SN50 Rack
Moving up to the SN50 rack architecture, there are 16 RDUs in a single air-cooled rack, split over two nodes.
SambaNova SN50 Hot Chips 2026 Chip Architecture
Diving a bit deeper into SN50 and the dataflow architecture. The core element of the SN50 is the sea of compute cores (PCUs) and memory cores (PMUs). There is no hardware memory management; this is all software managed. Every unit operates when it has input and sends it to the outputs.
SambaNova SN50 Hot Chips 2026 Transformer Structure
To better illustrate how the dataflow architecture works, here is an example of how it maps to a transformer.
SambaNova SN50 Hot Chips 2026 GPU Transformers
Here is what a GPU looks like.
SambaNova SN50 Hot Chips 2026 SN50 Transformers
And how it look on the SN50.
SambaNova SN50 Hot Chips 2026 Compute Comms Overlap
For compute, data from the HBM is fed into the AGCU portal that control off-chip access, and from there into the PCUs and PMUs.
SambaNova SN50 Hot Chips 2026 Double-Buffered PMUs
The SRAM amount used is not a function of the size of the model.
SambaNova SN50 Hot Chips 2026 No Global Synchronization
SambaNova SN50 Hot Chips 2026 SN50 Scaling
That was one RDU. How do things scale up for multiple RDUs? SambaNova employs both scale-up and scale-out networking. The Scale-up network is based on 800GbE, while scale-out is 400GbE. And then there is a front-end network.
SambaNova SN50 Hot Chips 2026 Scale-up and Scale-Out
SambaNova uses an all-to-all topology for an 8 socket configuration.
SambaNova SN50 Hot Chips 2026 Scaling Beyond One Node
To go above 8 sockets, then the scale-up network is employed using Ethernet switches. Links are ganged, and every node is connected to each of two switches in this 64 chip socket configuration.
SambaNova SN50 Hot Chips 2026 512 Socket Example
Then things can be scaled out further, in this case employing both scale-up and out for a 512 socket configuration.
SambaNova SN50 Hot Chips 2026 Model Parallelism
The key to performance on SN50 is overlap. SN50 supports all forms of model parallelism, and the collective communication forms that these models are built on.
SambaNova SN50 Hot Chips 2026 GEMM Benchmarking
Here is a brief look at performance with GEMM benchmarking. The utilization is consistently 70% or higher even at 32 sockets.
SambaNova SN50 Hot Chips 2026 Importance of Overlap
If you are able to overlap, you can do the compute and communications in parallel. That kind of overlap is not something GPUs can do.
SambaNova SN50 Hot Chips 2026 HBM and Compute Overlap
The building block for SambaNova is collective communication, which is the purple boxes in these diagrams. And the SRAMs can stream from one to another without having to go through a higher layer (e.g. HBM).
SambaNova SN50 Hot Chips 2026 MoEs with TP
Here’s a look at parallelism with tensor parallel.
SambaNova SN50 Hot Chips 2026 DeepSeek TP-32
Here is a look at the bandwidth utilization that SN has achieved with DeepSeek.
SambaNova SN50 Hot Chips 2026 MoEs with TP and EP
Meanwhile they can also use expert parallel (EP) as an additional form of parallelism. This relies on broadcast-dispatch as well as all-to-all dispatch-combine.
SambaNova SN50 Hot Chips 2026 Token Dispatch
With all-to-all, one way is to dynamically send everything to the target RDUs.
SambaNova SN50 Hot Chips 2026 Dispatch Broadcast
Alternatively, you can just blast everything to all of the RDUs and then filter out things afterwards.
SambaNova SN50 Hot Chips 2026 Dispatch All-to-All
The all-to-all method requires a group-by operation at the end of the router to collect (group) the tokens before transmitting them SRAM-to-SRAM. All-to-all also means allowing dynamic traffic.
SambaNova SN50 Hot Chips 2026 Dispatch Broadcast and Filter
Now here’s the other method of broadcast + filter. That is still an SRAM-to-SRAM operation, but with a filter operation on the PCUs of the receiving RDU. This keeps the network traffic parallel; though it does increase it a bit. And by not depending on the router, the transfer can be started early.
SambaNova SN50 Hot Chips 2026 EP on 64-Socket SN50
Here is another DeepSeek example, with SambaNova getting close to 80% bandwidth utilization for loading the experts in MoE.
SambaNova SN50 Hot Chips 2026 High MBU at Scale
As a result of this, SN50 achieves a high MBU value even at scale, with MBU holding at 45% even with 256 SN50s. And this makes it possible to keep adding RDUs to scale up things even further. This, in turn, means that models don’t have to give up bandwidth.
SambaNova SN50 Hot Chips 2026 SN50 Power Scaling
Going back to SambaNova’s original chart about power scaling, here is what SN50 clusters of different sizes look like. A 512 RDU configuration is able to scale up to an aggregate model bandwidth capacity of over 350 TB/second. The systems can strongly scale, with MBUs still in the 40% range at 512 sockets.
SambaNova SN50 Hot Chips 2026 SN50 Heterogeneous Disaggregation
Ultimately SambaNova is promoting a very similar picture as other dedicated inference chip firms, using one type of chips for prefill (and midfill), while using separate accelerators (i.e. SN50) for decode. Specifically, they’ve been using NVIDIA H200 + SN50, with RoCE for transferring between them.
SambaNova SN50 Hot Chips 2026 SN50 Heterogeneous Disaggregation In Action
Finally, taking a look at that performance in action, based on an Artificial Analysis benchmark of SN50. The hardware achieves over 750 tokens-per-second in MiniMax M2.7.