Resolve review comments

Author: Bihan Rana

docs/blog/posts/pd-disaggregation.md

@@ -26,9 +26,12 @@ For inference, `dstack` provides a [services](../../docs/concepts/services.md) a
 
 > If you’re new to Prefill–Decode disaggregation, see the official [SGLang docs](https://docs.sglang.io/advanced_features/pd_disaggregation.html).
 
+!!! note "Deprecation notice"
+    Configuring the SGLang router in a gateway is deprecated and will be disallowed in a future release. To run router and workers as separate replica groups, see [SGLang PD disaggregation (router as replica group)](https://dstack.ai/examples/inference/sglang/#pd-disaggregation).
+
 ## Services
 
-With `dstack` `0.20.17`, you can define a service with separate replica groups for Router, Prefill and Decode workers and run PD disaggregated Inference.
+With `dstack` `0.20.10`, you can define a service with separate replica groups for Prefill and Decode workers and enable PD disaggregation directly in the `router` configuration.
 
 <div editor-title="glm45air.dstack.yml">
 
@@ -43,21 +46,6 @@ env:
 image: lmsysorg/sglang:latest
 
 replicas:
-  - count: 1
-    # For now replica group with router must have count: 1
-    commands:
-      - pip install sglang_router
-      - |
-          python -m sglang_router.launch_router \
-            --host 0.0.0.0 \
-            --port 8000 \
-            --pd-disaggregation \
-            --prefill-policy cache_aware
-    router:
-      type: sglang
-    resources:
-      cpu: 4
-
   - count: 1..4
     scaling:
       metric: rps
@@ -67,7 +55,7 @@ replicas:
           python -m sglang.launch_server \
             --model-path $MODEL_ID \
             --disaggregation-mode prefill \
-            --disaggregation-transfer-backend nixl \
+            --disaggregation-transfer-backend mooncake \
             --host 0.0.0.0 \
             --port 8000 \
             --disaggregation-bootstrap-port 8998
@@ -83,7 +71,7 @@ replicas:
           python -m sglang.launch_server \
             --model-path $MODEL_ID \
             --disaggregation-mode decode \
-            --disaggregation-transfer-backend nixl \
+            --disaggregation-transfer-backend mooncake \
             --host 0.0.0.0 \
             --port 8000
     resources:
@@ -94,8 +82,12 @@ model: zai-org/GLM-4.5-Air-FP8
 
 probes:
   - type: http
-    url: /health
+    url: /health_generate
     interval: 15s
+
+router:
+  type: sglang
+  pd_disaggregation: true
 ```
 
 </div>
@@ -111,33 +103,32 @@ $ dstack apply -f glm45air.dstack.yml
 
 </div>
 
-### SSH fleet
+### Gateway
 
-Create an [SSH fleet](https://dstack.ai/docs/concepts/fleets/#apply-a-configuration) that includes one CPU host for the router and one or more GPU hosts for the workers. Make sure the CPU and GPU hosts are in the same network.
+Just like `dstack` relies on the SGLang router for cache-aware routing, Prefill–Decode disaggregation also requires a [gateway](../../docs/concepts/gateways.md#sglang) configured with the SGLang router.
 
-<div editor-title="pd-fleet.dstack.yml">
+<div editor-title="gateway-sglang.dstack.yml">
 
 ```yaml
-type: fleet
-name: pd-disagg
-
-placement: cluster
-
-ssh_config:
-  user: ubuntu
-  identity_file: ~/.ssh/id_rsa
-  hosts:
-    - 89.169.108.16   # CPU Host (router)
-    - 89.169.123.100  # GPU Host (prefill/decode workers)
-    - 89.169.110.65   # GPU Host (prefill/decode workers)
+type: gateway
+name: inference-gateway
+
+backends: [kubernetes]
+region: any
+
+domain: example.com
+
+router:
+  type: sglang
+  policy: cache_aware
 ```
 
 </div>
 
 ## Limitations
-* The router replica group is currently limited to `count: 1` (no HA yet). Support for multiple router replicas for HA is planned.
-* Prefill–Decode disaggregation is currently available with the SGLang backend (Nvidia-dynamo and vLLM support is coming).
+* Because the SGLang router requires all workers to be on the same network, and `dstack` currently runs the router inside the gateway, the gateway and the service must be running in the same cluster.
 * Autoscaling supports RPS as the metric for now; TTFT and ITL metrics are planned next.
+* Prefill–Decode disaggregation is currently available with the SGLang backend (vLLM support is coming).
 
 With native support for inference and now Prefill–Decode disaggregation, `dstack` makes it easier to run high-throughput, low-latency model serving across GPU clouds, and Kubernetes or bare-metal clusters.
 

docs/docs/concepts/gateways.md

@@ -99,7 +99,7 @@ If you configure the `sglang` router, [services](../concepts/services.md) can ru
     To run services with PD disaggregation see [SGLang PD disaggregation](https://dstack.ai/examples/inference/sglang/#pd-disaggregation).
 
 !!! note "Deprecation"
-    Configuring the SGLang router in a gateway will be deprecated in a future release.
+    Configuring the SGLang router in a gateway is deprecated and will be disallowed in a future release.
 
 ??? info "Policy"
     The `policy` property allows you to configure the routing policy:

examples/inference/sglang/README.md

@@ -179,8 +179,6 @@ replicas:
 
 port: 8000
 model: zai-org/GLM-4.5-Air-FP8
-# SSH fleet containing both router (CPU) and workers (GPU).
-fleets: [pd-disagg]
 
 # Custom probe is required for PD disaggregation.
 probes:
@@ -193,31 +191,15 @@ probes:
 
 Currently, auto-scaling only supports `rps` as the metric. TTFT and ITL metrics are coming soon.
 
-#### SSH fleet
+#### Fleet
 
-Create an [SSH fleet](https://dstack.ai/docs/concepts/fleets/#apply-a-configuration) that includes one CPU host for the router and one or more GPU hosts for the workers. Make sure the CPU and GPU hosts are in the same network.
+Create a [fleet](https://dstack.ai/docs/concepts/fleets/) that can provision both a CPU node (for the router replica group) and GPU nodes (for the prefill/decode replica groups).
+You can create an SSH fleet, elastic Cloud fleet (nodes: 0..) or kubernetes cluster. Just don't specify any resource constraints in the fleet, and dstack will automatically provision the correct instances (both CPU and GPU, in the same fleet) based on the resources specified in replicas in the run configuration.
 
-<div editor-title="pd-fleet.dstack.yml">
-
-```yaml
-type: fleet
-name: pd-disagg
-
-placement: cluster
-
-ssh_config:
-  user: ubuntu
-  identity_file: ~/.ssh/id_rsa
-  hosts:
-    - 89.169.108.16   # CPU Host (router)
-    - 89.169.123.100  # GPU Host (prefill/decode workers)
-    - 89.169.110.65   # GPU Host (prefill/decode workers)
-```
-
-</div>
+The only requirement is that the router and worker replicas run in the same network. In practice, this typically means using a single fleet where the backend and region are the same or using `placement: cluster` if the backend supports it.
 
 !!! note "Gateway-based routing (deprecated)"
-    If you create a gateway with the [`sglang` router](https://dstack.ai/docs/concepts/gateways/#sglang), you can also run SGLang with PD disaggregation. This method will be deprecated in the future in favor of running the router as a replica.
+    If you create a gateway with the [`sglang` router](https://dstack.ai/docs/concepts/gateways/#sglang), you can also run SGLang with PD disaggregation. This method is deprecated and will be disallowed in a future release in favor of running the router as a replica.
 
 ## Source code
 

examples/inference/sglang/pd.dstack.yml

@@ -56,10 +56,6 @@ replicas:
 port: 8000
 model: zai-org/GLM-4.5-Air-FP8
 
-# SSH fleet containing both router (CPU) and workers (GPU).
-# Router and workers must be in the same fleet (same network).
-fleets: [pd-disagg]
-
 probes:
   - type: http
     url: /health

src/dstack/_internal/proxy/gateway/resources/nginx/service.jinja2

@@ -47,8 +47,8 @@ server {
     }
     {% endfor %}
 
-    {# For SGLang router: block all requests except whitelisted locations added dynamically above #}
-    {% if router is not none and router.type == "sglang" %}
+    {# For router services: block all requests except whitelisted locations added dynamically above #}
+    {% if has_router_replica or (router is not none and router.type == "sglang") %}
     location / {
         return 403;
     }

src/dstack/_internal/proxy/gateway/services/registry.py

@@ -342,7 +342,9 @@ async def get_nginx_service_config(
 ) -> ServiceConfig:
     limit_req_zones: list[LimitReqZoneConfig] = []
     locations: list[LocationConfig] = []
-    is_sglang = service.router and service.router.type == RouterType.SGLANG
+    is_sglang = (
+        service.router is not None and service.router.type == RouterType.SGLANG
+    ) or service.has_router_replica
     sglang_limits: dict[str, LimitReqConfig] = {}
     sglang_prefix_lengths: dict[str, int] = {}  # Track prefix lengths for most-specific selection
 

src/dstack/_internal/server/services/proxy/repo.py

@@ -75,6 +75,7 @@ async def get_service(self, project_name: str, run_name: str) -> Optional[Servic
             (g for g in run_spec.configuration.replica_groups if g.router is not None),
             None,
         )
+        has_router_replica = router_group is not None
         router = run_spec.configuration.router
         replicas = []
         for job in jobs:
@@ -139,6 +140,7 @@ async def get_service(self, project_name: str, run_name: str) -> Optional[Servic
             client_max_body_size=DEFAULT_SERVICE_CLIENT_MAX_BODY_SIZE,
             strip_prefix=run_spec.configuration.strip_prefix,
             replicas=tuple(replicas),
+            has_router_replica=has_router_replica,
             router=router,
         )
 

src/dstack/_internal/server/services/proxy/services/service_proxy.py

@@ -41,7 +41,9 @@ async def proxy(
     if not service.strip_prefix:
         path = concat_url_path(request.scope.get("root_path", "/"), request.url.path)
 
-    if service.router is not None and service.router.type == RouterType.SGLANG:
+    if (
+        service.router is not None and service.router.type == RouterType.SGLANG
+    ) or service.has_router_replica:
         path_for_match = path if path.startswith("/") else f"/{path}"
         if not _is_whitelisted_path(path_for_match, SGLANG_WHITELISTED_PATHS):
             raise ProxyError("Path is not allowed for this service", status.HTTP_403_FORBIDDEN)

src/dstack/_internal/server/services/runs/spec.py

@@ -199,6 +199,21 @@ def _check_can_update_configuration(
         raise ServerClientError(
             f"Configuration type changed from {current.type} to {new.type}, cannot update"
         )
+
+    if isinstance(current, ServiceConfiguration) and isinstance(new, ServiceConfiguration):
+        current_router_group = next(
+            (g for g in current.replica_groups if g.router is not None), None
+        )
+        new_router_group = next((g for g in new.replica_groups if g.router is not None), None)
+        current_router_group_name = (
+            current_router_group.name if current_router_group is not None else None
+        )
+        new_router_group_name = new_router_group.name if new_router_group is not None else None
+        if current_router_group_name != new_router_group_name:
+            raise ServerClientError(
+                "Cannot update router replica groups in-place (adding/removing `router` or changing "
+                "which replica group is the router is not supported). Stop the run and apply again."
+            )
     updatable_fields = _CONF_UPDATABLE_FIELDS + _TYPE_SPECIFIC_CONF_UPDATABLE_FIELDS.get(
         new.type, []
     )

src/dstack/_internal/server/services/services/__init__.py

@@ -275,24 +275,12 @@ def _build_service_router_config(
     service_configuration: ServiceConfiguration,
 ) -> Optional[AnyServiceRouterConfig]:
     """
-    Router metadata to store on the gateway proxy for this service (`service.router`).
-
-    A replica-group `router:` does **not** depend on the gateway having its own global SGLang
-    router block—the router runs on service replicas. When the gateway has no global SGLang
-    config but a replica group does declare `router:`, we still return a default
-    `SGLangServiceRouterConfig` so nginx/proxy code can treat the service as SGLang and apply path
-    rules. When the gateway *does* have global SGLang, we merge gateway policy with service-level
-    `configuration.router` overrides as before.
+    Build router config from gateway (type, policy) + service (pd_disaggregation, policy override).
+    Service's policy overrides gateway's if present. Keeps backward compat: SGLang enabled
+    automatically when gateway has it configured.
     """
-    has_replica_group_router = any(
-        g.router is not None for g in service_configuration.replica_groups
-    )
     if not _gateway_has_sglang_router(gateway_configuration):
-        if not has_replica_group_router:
-            return None
-        # In later releases we will deprecate service-level and gateway-level router
-        # configuration and return `ReplicaGroupRouterConfig` here instead.
-        return SGLangServiceRouterConfig()
+        return None
 
     gateway_router = gateway_configuration.router
     assert gateway_router is not None  # ensured by _gateway_has_sglang_router