LLM 推理计算器 | LLM Inference Calculator
**Deployable**Llama 3.1 8B（Dense · 8B） on H100 SXM5 × 1
28.5 ms TTFT (first token)
8.79 ms TPOT (per token)
9.03 s End-to-end Latency
113 tok/s System Throughput
Parallel Layout
Candidate layouts constrained by "TP ≤ GPUs per node (1)", sorted by TP ⇒ EP ⇒ PP
VRAM Usage (per GPU)
17.3 GB / 72.0 GB (24%)Max Batch size: 198
Weights 16.0 GB KV cache 201.3 MB Activation 8.4 MB Overhead 1.1 GB
Capacity accounts for headroom. Max batch assumes requests evenly distributed across generation (avg inputLen + outputLen/2).
Phase Details
| Prefill (compute-bound) | | --- | | Total FLOPs | 16.9 TFLOP | | Compute Time | 28.5 ms | | Communication Time (total) | 0 µs | | TTFT | **28.5 ms** | | Throughput (input token/s) | **35.9k** | | Compute Utilization | **60%** 593.4 / 989.0 TFLOPS |
| Decode (bandwidth-bound) | | --- | | Weight Read / GPU·step | 16.0 GB | | KV Read / GPU·step | 201.4 MB | | Bandwidth Time | 8.79 ms | | Communication Time (total) | 0 µs | | TPOT | **8.79 ms** | | Throughput (output token/s) | **114** | | Bandwidth Utilization | **55%** 1843 / 3350 GB/s |
Note: "Communication Time (total)" is raw communication; what enters latency is the exposed portion = total × (1 − overlap coefficient). When overlap = 1 (ideal), communication is fully hidden by compute, TTFT / TPOT exclude communication; lower TP / EP / PP communication overlap in Calibration panel to see exposed cost.
Loading chart…