Skip to content
←Back to Blog
Integrating SGLang into IBM Spyre in Under a Week with AI
LLM InferenceSGLangIBM Spyre

Integrating SGLang into IBM Spyre in Under a Week with AI

Two engineers used AI to add SGLang support to IBM Spyre in under a week with a roughly 1,500-line plugin, reaching 3.6x the torch-spyre reference throughput and vLLM parity on two models.

Gal Bloch, Itay Etelis, Raghu Ganti, Mudhakar Srivatsa, Alex NailsNovember 12, 20267 min read19 views
ZH

TL;DR. IBM engineers Gal Bloch and Itay Etelis, working with Raghu Ganti, Mudhakar Srivatsa, and RadixArk's Alex Nails, used AI coding agents to bring SGLang to IBM Spyre in less than a week. The resulting plugin is roughly 1,500 lines of code. Existing Spyre support in PyTorch and the extension points exposed by SGLang made it possible to reuse the hardware backend from the existing vLLM integration instead of rebuilding Spyre support from scratch. SGLang reached about 3.6x the throughput of the reference PyTorch inference implementation built on torch-spyre, and matched vLLM on the workloads measured for Granite 3.3 8B and Gemma 4 26B A4B. SGLang's RadixCache prefix sharing also worked without any Spyre-specific changes.

The project started with a practical goal: make SGLang's serving stack, including its built-in prefix sharing, available to users of IBM's Spyre accelerator. The team already had a working PyTorch path for Spyre. What they needed was a serving layer that could turn that device support into a usable inference engine.

Two engineers completed the initial integration in under a week with the help of AI coding agents. The agents did not invent a new accelerator backend. They worked against concrete reference code from the existing PyTorch implementation of Spyre operations and adapted that code to SGLang's backend interfaces. The task was therefore mostly about connecting two well-defined layers: the existing Spyre support and the serving-framework extension points.

Spyre and the starting point

Spyre is IBM's accelerator for enterprise AI inference. It connects to PyTorch through torch-spyre, which registers Spyre as a PyTorch device and integrates it with torch.compile. That design exposes Spyre through standard PyTorch APIs instead of requiring every serving engine to implement device discovery, compilation, and a separate operator layer.

This is the important architectural precondition. When the hardware already speaks through a mature compiler and operator API, a new inference server does not need to own every low-level implementation. It can focus on scheduling, memory management, and the interfaces between the serving engine and the device backend.

The result: a small serving plugin, not a new hardware stack

The initial SGLang plugin was about 1,500 lines. Four small patches extended SGLang's backend support. Two removed CUDA-specific assumptions from existing execution paths so those paths could run on another device. The other two registered Spyre as a supported device and as a non-Triton attention backend. The compatibility patches can be useful to other accelerators; the device-registration changes are specific to Spyre.

SGLang on Spyre reached approximately 3.6x the throughput of the reference PyTorch inference implementation using torch-spyre. The team also compared SGLang with vLLM on Spyre using Granite 3.3 8B and Gemma 4 26B A4B. Across the measured workloads, the two serving systems performed at essentially the same level.

Relative latency of SGLang and vLLM/spyre-inference on Granite 3.3 8B and Gemma 4 26B-A4B, normalized to vLLM at 1.0
Relative performance on IBM Spyre, normalized to vLLM / spyre-inference at 1.0. Lower is better. The chart uses TP / DP = 4 / 1 and batch size 4. Granite 3.3 8B measures 0.993 for SGLang, or about 0.7% faster than vLLM; Gemma 4 26B-A4B measures 1.008, or about 0.8% slower than vLLM. Image: PyTorch Foundation.

Why the integration was fast

With a working compiler and an existing inference server, adding SGLang became a focused integration problem. PyTorch and torch.compile supplied a shared implementation layer. Spyre's behavior was already expressed through APIs that more than one serving framework could call.

SGLang supplied clear places to attach that implementation. Platform registration identifies the device. The attention backend supplies the model-specific attention path. The KV pool owns cache buffers and allocation. Operator dispatch connects framework operations to device implementations. These boundaries let the team preserve SGLang's scheduler, request handling, and serving loop while changing only the hardware-facing pieces.

flowchart LR
  A["SGLang request scheduler
RadixCache prefix reuse"] --> B["Platform registration
device = spyre"] B --> C["Spyre attention backend
spyre_paged"] C --> D["Device-agnostic KV pool
Spyre buffer override"] D --> E["Operator dispatch
attention, RMSNorm, SiLU, GeLU, RoPE, linear"] E --> F["torch-spyre + torch.compile"] F --> G["IBM Spyre accelerator"]

The reference implementation came from the existing vLLM integration for Spyre, which contained PyTorch implementations of accelerator operations. That gave the AI coding agents concrete code describing Spyre's hardware behavior. SGLang's extension points defined the interfaces the adapted code had to satisfy. The agents therefore had less to infer both about the hardware and about where the new code belonged.

The size difference is informative. The existing vLLM Spyre backend was about 3,500 lines, compared with roughly 1,500 for the SGLang plugin. In the initial SGLang integration, the attention base class supplied the surrounding logic, leaving the Spyre backend to override the inner scaled dot-product attention operation. Reusing that base class accounted for roughly two-thirds of the difference. SGLang's device-agnostic KV pool also reduced buffer creation to a single method override, although allocation still needed PyTorch-based handling where the existing path depended on Triton.

Reusing the vLLM Spyre implementation

The team reused the attention code from the vLLM Spyre integration, including its input bucketing and tensor-shape handling. They also reused its PyTorch-based KV-cache allocation strategy. For RMSNorm, SiLU, GeLU, rotary embeddings, and linear layers, the existing Spyre implementations were connected to SGLang's operator dispatch.

Some features required no Spyre-specific work at all. RadixCache is the clearest example. Prefix sharing is resolved entirely inside SGLang's scheduler before execution reaches the accelerator, so the existing scheduling logic ran unchanged. This is a useful separation of concerns: cache lookup and request grouping remain portable framework logic, while the device backend only sees the work that must execute on Spyre.

The plugin is available as open source alongside upstream SGLang.

Running SGLang with Spyre

The example assumes that the Spyre runtime, torch-spyre, the SGLang Spyre plugin, and its compatibility patches are installed and configured. The engine explicitly selects device="spyre", the spyre_paged attention backend, a page size of 64, one running request, and disables CUDA graphs.

import sglang as sgl

models = [
    "ibm-ai-platform/micro-g3.3-8b-instruct-1b",
    "mistralai/Ministral-8B-Instruct-2410",
]

for model in models:
    engine = sgl.Engine(
        model_path=model,
        device="spyre",
        attention_backend="spyre_paged",
        dtype="float16",
        page_size=64,
        max_running_requests=1,
        disable_cuda_graph=True,
    )

    try:
        output = engine.generate(
            "Why is the sky blue?",
            {"temperature": 0, "max_new_tokens": 64},
        )
        print(model, output["text"])
    finally:
        engine.shutdown()

How to read the benchmark

The 3.6x figure compares SGLang with the reference PyTorch inference implementation based on torch-spyre; it is not a 3.6x claim against vLLM. The vLLM comparison is separate. On the two measured models, the chart reports SGLang at 0.993 and 1.008 relative to vLLM, which supports the article's description of near parity rather than a clear advantage for either serving system.

The scope is also deliberately narrow. The result covers two models and the workloads selected by the authors on IBM Spyre, with a 4 / 1 TP / DP layout and batch size 4. It demonstrates that SGLang can use Spyre without giving up its scheduler or RadixCache behavior; it does not establish performance across every model, sequence length, concurrency level, or deployment topology.

The development-time result should be read the same way. The project shows what became possible when an existing compiler backend, a reusable hardware implementation, and stable framework interfaces were already present. It is not a controlled study of AI coding-agent productivity, nor evidence that integrating an arbitrary accelerator takes only a week.

What comes next

The Spyre software stack is still moving quickly. Over the coming months, the team expects to add dynamic-shape support, which should remove much of the shape-management work currently required. Later, they expect Triton support and RadixAttention on Spyre, opening additional opportunities for kernel optimization.

Keeping hardware support in PyTorch and integration code at SGLang's extension points gives future compiler and runtime improvements a direct path into the serving stack. The next integration can start from the same foundation, with concrete implementations and interfaces that both developers and AI coding tools can inspect and reuse.

Source

This article is an adapted bilingual edition of Integrating SGLang into Spyre in Under a Week with AI, published by the PyTorch Foundation on November 12, 2026. Original authors: Gal Bloch, Itay Etelis, Raghu Ganti, Mudhakar Srivatsa, and Alex Nails.

Related Articles

As an Amazon Associate, we earn from qualifying purchases.