GPT-OSS on Day 0
Day-0 support for GPT‑OSS on H200 by adapting ROSE with FP8, sink attention, and MoE optimizations
TL;DR
- Perplexity achieved Day-0 support for OpenAI's `gpt-oss-20b` and `gpt-oss-120b` models on NVIDIA H200 GPUs.
- The inference stack, ROSE, was adapted to use FP8 precision instead of FP4 due to H200's hardware limitations.
- Key model architecture adaptations included support for attention sinks and sparse Mixture-of-Expert (MoE) layers.
- ROSE is a flexible Python-based inference engine with performance-critical components being migrated to Rust.
- Performance benchmarks and cost analysis were conducted for various Expert Parallelism (EP), Data Parallelism (DP), and Tensor Parallelism (TP) configurations.
- The Harmony tokenizer and chat format were integrated by adding a pluggable formatter/parser to the JSON API frontend service.
- For prefill, `batch_size=1` and `DP=1` yielded the best performance, with varying TP configurations offering trade-offs between latency and cost.
- EP4 DP1 TP4 was found to be the optimal configuration for decode in most cases.