Fix issues found in independent review

An independent review of the custom-security-group feature (previous two
commits) found several real gaps. Fixed all of them:

- security_group was dropped when a run provisioned a new instance into an
  existing fleet (only the fleet-apply path honored it). Fixed by threading
  security_group through the Requirements pipeline exactly like reservation
  already is (Requirements.security_group, combine_fleet_and_run_profiles/
  combine_fleet_and_run_requirements, and sourcing run_job's
  InstanceConfiguration from job.job_spec.requirements.security_group instead
  of the run's raw profile).

- security_group was silently ignored when an offer resolved to a backend
  that doesn't support it (e.g. GCP). offers.py now narrows backend_types to
  BACKENDS_WITH_SECURITY_GROUP_SUPPORT when security_group is set, mirroring
  the existing reservation filtering.

- AWS: the configurator forbade combining security_group_name with
  security_group_ids, but compute.py implements (and docs/tests described) a
  fallback from ids to name - the combination is now allowed. Also added
  validation catching region-key typos in security_group_ids, and a clearer
  ComputeError instead of a confusing NoCapacityError retry loop when a
  configured security group doesn't exist in the target VPC.

- Azure: renamed network_security_group_ids to network_security_group_names
  since the values are NSG names (not IDs) scoped to the backend's
  resource_group, and added region-key typo validation.

- GCP: create_firewall_rules no longer disables the gateway firewall rule,
  matching AWS/Azure/OCI where gateway security resources are always
  auto-managed regardless of the custom-security-group settings.

- OCI: the shared subnet has no security_list_ids, so it inherits the VCN's
  permissive default security list (SSH open to 0.0.0.0/0, allow-all egress).
  Since OCI evaluates security lists and NSGs as a union of allows, a custom
  NSG could not actually restrict anything. Fixed by routing custom-NSG
  instances into a separate, dedicated VCN/subnet with no security list, so
  the NSG becomes the sole security boundary. The default VCN/subnet used by
  auto-managed-NSG instances is completely untouched. Also added region-key
  typo validation and corrected the docs to accurately attribute default SSH
  exposure to the security list, not the NSG.

Author: james-boydell

mkdocs/docs/concepts/backends.md

@@ -489,7 +489,9 @@ There are two ways to configure Azure: using a client secret or using the defaul
     By default, `dstack` creates and manages its own network security group (opening SSH to the internet
     and allowing all traffic within the group so multi-node clusters work out of the box).
     Azure NSG names must be unique within a resource group regardless of region, so a custom NSG is
-    configured per location via `network_security_group_ids`:
+    configured per location via `network_security_group_names`. The values are plain NSG names
+    within the configured `resource_group` (not full Azure resource IDs), so NSGs in a different
+    resource group cannot be referenced:
 
     ```yaml
     projects:
@@ -499,12 +501,12 @@ There are two ways to configure Azure: using a client secret or using the defaul
             creds:
               type: default
             regions: [westeurope, eastus]
-            network_security_group_ids:
+            network_security_group_names:
               westeurope: my-network-security-group-we
               eastus: my-network-security-group-eus
     ```
 
-    Locations not covered by `network_security_group_ids` fall back to dstack's auto-created network
+    Locations not covered by `network_security_group_names` fall back to dstack's auto-created network
     security group. Either way, `dstack` attaches the network security group to instances as-is and
     never adds, removes, or modifies its rules. You're responsible for SSH reachability and, for
     multi-node clusters, for allowing traffic between instances in the group.
@@ -730,11 +732,11 @@ gcloud projects list --format="json(projectId)"
     Additionally, [Cloud NAT](https://cloud.google.com/nat/docs/overview) must be configured to provide access to external resources for provisioned instances.
 
 ??? info "Custom firewall rules"
-    By default, `dstack` creates VPC firewall rules allowing inbound SSH (and, for gateways, HTTP/HTTPS) from
-    the internet, scoped to the `dstack-runner-instance` and `dstack-gateway-instance` target tags.
+    By default, `dstack` creates a VPC firewall rule allowing inbound SSH from the internet to instances,
+    scoped to the `dstack-runner-instance` target tag.
     Unlike AWS/Azure/OCI, GCP firewall rules apply to the whole VPC rather than to a single attachable resource,
     so there's no per-fleet override — if you manage your own firewall rules and don't want `dstack` creating
-    rules that open ports to `0.0.0.0/0`, disable this at the project level with `create_firewall_rules: false`:
+    a rule that opens port 22 to `0.0.0.0/0`, disable this at the project level with `create_firewall_rules: false`:
 
     ```yaml
     projects:
@@ -749,7 +751,9 @@ gcloud projects list --format="json(projectId)"
     ```
 
     You're then responsible for ensuring your VPC's own firewall rules allow whatever SSH and cluster traffic
-    `dstack` needs.
+    `dstack` needs. This setting only affects the instance SSH rule — the separate firewall rule `dstack`
+    creates for gateways (allowing HTTP/HTTPS from the internet, scoped to the `dstack-gateway-instance`
+    target tag) is always auto-managed, since gateways are meant to be internet-reachable.
 
 ### Lambda
 
@@ -1158,8 +1162,13 @@ There are two ways to configure OCI: using client credentials or using the defau
     ```
 
 ??? info "Custom network security group"
-    By default, `dstack` creates and manages its own network security group per project (opening SSH to
-    `0.0.0.0/0` and allowing all traffic within the VCN so multi-node clusters work out of the box).
+    By default, `dstack` places instances in a shared subnet whose OCI security list opens SSH
+    (TCP port 22) to `0.0.0.0/0` and permits all outbound traffic. On top of that, `dstack` creates
+    and manages its own network security group (NSG) per project, which only adds a rule allowing all
+    traffic within the group so multi-node clusters work out of the box. In other words, the
+    permissive SSH ingress and the outbound access come from the subnet's security list, not from the
+    auto-managed NSG.
+
     OCI network security groups are region-scoped, so a custom NSG is configured per region via
     `network_security_group_ids`:
 
@@ -1176,9 +1185,20 @@ There are two ways to configure OCI: using client credentials or using the defau
     ```
 
     Regions not covered by `network_security_group_ids` fall back to dstack's auto-created network
-    security group. Either way, `dstack` attaches the network security group to instances as-is and
-    never adds, removes, or modifies its rules. You're responsible for SSH reachability and, for
-    multi-node clusters, for allowing traffic between instances in the group.
+    security group and shared subnet.
+
+    When a custom NSG is used, `dstack` never adds, removes, or modifies its rules, and it places the
+    affected instances in a separate VCN and subnet that has **no** OCI security list (a second subnet
+    in the same VCN isn't possible here, since the default subnet already occupies the whole VCN's
+    address space). This makes the NSG the sole security boundary — there is no longer any implicit
+    SSH-from-anywhere or implicit outbound-all coming from a security list. As a result, your custom
+    NSG is fully responsible for:
+
+    - **Ingress**, including SSH (TCP port 22) from wherever you connect.
+    - **Egress**, including outbound internet access. Without an egress rule (e.g. allow all to
+      `0.0.0.0/0`), instances will have no outbound connectivity and will fail to pull Docker images
+      and start runs.
+    - For multi-node clusters, **traffic between instances** in the group.
 
     You can also override this per fleet or run using the `security_group` profile property.
 

src/dstack/_internal/core/backends/aws/compute.py

@@ -380,9 +380,18 @@ def create_instance(
                 )
             except botocore.exceptions.ClientError as e:
                 logger.warning("Got botocore.exceptions.ClientError: %s", e)
-                if e.response["Error"]["Code"] == "InvalidParameterValue":
+                error_code = e.response["Error"]["Code"]
+                if error_code == "InvalidParameterValue":
                     msg = e.response["Error"].get("Message", "")
                     raise ComputeError(f"Invalid AWS request: {msg}")
+                if error_code == "InvalidGroup.NotFound":
+                    # A misconfigured security group (e.g. wrong VPC/region) is not a
+                    # capacity issue, so surface it clearly instead of retrying other AZs.
+                    msg = e.response["Error"].get("Message", "")
+                    raise ComputeError(
+                        f"Security group not found for instance in region"
+                        f" {instance_offer.region}: {msg}"
+                    )
                 continue
             instance = response[0]
             # wait_until_running() is only needed so that instance is immediately ready for volume attach.

src/dstack/_internal/core/backends/aws/configurator.py

@@ -148,9 +148,16 @@ def _check_config_iam_instance_profile(
             )
 
     def _check_config_security_group(self, config: AWSBackendConfigWithCreds):
-        if config.security_group_name is not None and config.security_group_ids is not None:
+        if config.security_group_ids is None:
+            return
+        regions = config.regions if config.regions is not None else DEFAULT_REGIONS
+        unknown_regions = [r for r in config.security_group_ids if r not in regions]
+        if unknown_regions:
             raise ServerClientError(
-                msg="Only one of `security_group_name` and `security_group_ids` can be specified"
+                msg=(
+                    f"`security_group_ids` specifies regions not in `regions`: {unknown_regions}."
+                    " This is likely a typo — remove the extra keys or add them to `regions`"
+                )
             )
 
     def _check_config_vpc(self, session: Session, config: AWSBackendConfigWithCreds):

src/dstack/_internal/core/backends/azure/compute.py

@@ -149,8 +149,8 @@ def create_instance(
             allocate_public_ip=allocate_public_ip,
         )
         network_security_group = instance_config.security_group
-        if network_security_group is None and self.config.network_security_group_ids is not None:
-            network_security_group = self.config.network_security_group_ids.get(location)
+        if network_security_group is None and self.config.network_security_group_names is not None:
+            network_security_group = self.config.network_security_group_names.get(location)
         if network_security_group is None:
             network_security_group = azure_utils.get_default_network_security_group_name(
                 resource_group=self.config.resource_group,

src/dstack/_internal/core/backends/azure/configurator.py

@@ -104,6 +104,7 @@ def validate_config(self, config: AzureBackendConfigWithCreds, default_creds_ena
         self._check_config_resource_group(config=config, credential=credential)
         self._check_config_vm_managed_identity(config=config, credential=credential)
         self._check_config_vpc(config=config, credential=credential)
+        self._check_config_network_security_groups(config)
 
     def create_backend(
         self, project_name: str, config: AzureBackendConfigWithCreds
@@ -126,7 +127,7 @@ def create_backend(
             resource_group=config.resource_group,
             locations=config.regions,
             create_default_network=config.vpc_ids is None and config.subnet_ids is None,
-            network_security_group_ids=config.network_security_group_ids,
+            network_security_group_names=config.network_security_group_names,
         )
         return BackendRecord(
             config=AzureStoredConfig(
@@ -283,6 +284,27 @@ def _check_config_vpc(
                     except BackendError as e:
                         raise ServerClientError(e.args[0])
 
+    def _check_config_network_security_groups(self, config: AzureBackendConfigWithCreds):
+        if not config.network_security_group_names:
+            return
+        # When `regions` is None, all regions are used and there is no feasible way to validate
+        # location keys against the full list of Azure regions here, so the check is skipped.
+        if config.regions is None:
+            return
+        configured_regions = set(config.regions)
+        unknown_locations = sorted(
+            location
+            for location in config.network_security_group_names
+            if location not in configured_regions
+        )
+        if unknown_locations:
+            raise ServerClientError(
+                f"Locations {unknown_locations} in `network_security_group_names` are not in"
+                " `regions`. Otherwise, these network security groups would be silently ignored"
+                " and instances in those locations would fall back to dstack's auto-created"
+                " network security group."
+            )
+
     def _check_config_vm_managed_identity(
         self, config: AzureBackendConfigWithCreds, credential: auth.AzureCredential
     ):
@@ -341,7 +363,7 @@ def _create_network_resources(
         resource_group: str,
         locations: List[str],
         create_default_network: bool,
-        network_security_group_ids: Optional[Dict[str, str]] = None,
+        network_security_group_names: Optional[Dict[str, str]] = None,
     ):
         def func(location: str):
             network_manager = NetworkManager(
@@ -354,9 +376,9 @@ def func(location: str):
                     name=azure_utils.get_default_network_name(resource_group, location),
                     subnet_name=azure_utils.get_default_subnet_name(resource_group, location),
                 )
-            if location not in (network_security_group_ids or {}):
+            if location not in (network_security_group_names or {}):
                 # Skipped when the user supplies their own network security group for this
-                # location via `network_security_group_ids` - dstack does not create or manage
+                # location via `network_security_group_names` - dstack does not create or manage
                 # it in that case.
                 network_manager.create_network_security_group(
                     resource_group=resource_group,

src/dstack/_internal/core/backends/azure/models.py

@@ -81,13 +81,15 @@ class AzureBackendConfig(CoreModel):
             )
         ),
     ] = None
-    network_security_group_ids: Annotated[
+    network_security_group_names: Annotated[
         Optional[Dict[str, str]],
         Field(
             description=(
                 "The mapping from Azure locations to the names of existing network security groups"
-                " (in the configured resource group) to use for instances instead of the one `dstack`"
-                " creates and manages automatically."
+                " to use for instances instead of the one `dstack` creates and manages automatically."
+                " The values are plain NSG names (not full Azure resource IDs) and must refer to"
+                " network security groups within the configured `resource_group` — NSGs in a different"
+                " resource group cannot be referenced."
                 " Locations not present in this mapping fall back to dstack's auto-created"
                 " network security group."
                 " When set, `dstack` does not add, remove, or modify any rules on these network"

src/dstack/_internal/core/backends/base/compute.py

@@ -321,7 +321,7 @@ def run_job(
             ssh_keys=[SSHKey(public=project_ssh_public_key.strip())],
             volumes=volumes,
             reservation=job.job_spec.requirements.reservation,
-            security_group=run.run_spec.merged_profile.security_group,
+            security_group=job.job_spec.requirements.security_group,
             tags=run.run_spec.merged_profile.tags,
         )
         instance_offer = instance_offer.copy()

src/dstack/_internal/core/backends/gcp/compute.py

@@ -561,7 +561,10 @@ def create_gateway(
         self,
         configuration: GatewayComputeConfiguration,
     ) -> GatewayProvisioningData:
-        if self.config.vpc_project_id is None and self.config.create_firewall_rules is not False:
+        if self.config.vpc_project_id is None:
+            # Gateway firewall rules are intentionally not gated by `create_firewall_rules`:
+            # gateways are meant to be internet-reachable, and this keeps behavior consistent
+            # with AWS/Azure/OCI, where gateway security groups/NSGs are always auto-managed.
             gcp_resources.create_gateway_firewall_rules(
                 firewalls_client=self.firewalls_client,
                 project_id=self.config.project_id,

src/dstack/_internal/core/backends/gcp/models.py

@@ -84,10 +84,12 @@ class GCPBackendConfig(CoreModel):
         Optional[bool],
         Field(
             description=(
-                "A flag to enable/disable `dstack` creating VPC firewall rules that allow SSH"
-                " (and, for gateways, HTTP/HTTPS) traffic from the internet."
+                "A flag to enable/disable `dstack` creating a VPC firewall rule that allows SSH"
+                " traffic from the internet to instances."
                 " Set to `false` if you manage your own firewall rules and don't want `dstack`"
-                " creating rules that open ports to `0.0.0.0/0`. Defaults to `true`"
+                " creating a rule that opens port 22 to `0.0.0.0/0`. Defaults to `true`."
+                " This does not affect the separate firewall rule `dstack` creates for gateways,"
+                " which are meant to be internet-reachable and are always auto-managed"
             )
         ),
     ] = None

src/dstack/_internal/core/backends/oci/compute.py

@@ -143,7 +143,8 @@ def create_instance(
             security_group_id = self.config.network_security_group_ids.get(
                 instance_offer.region
             )
-        if security_group_id is None:
+        using_custom_security_group = security_group_id is not None
+        if not using_custom_security_group:
             security_group = resources.get_or_create_security_group(
                 f"dstack-{instance_config.project_name}-default-security-group",
                 subnet.vcn_id,
@@ -154,10 +155,25 @@ def create_instance(
                 security_group.id, region.virtual_network_client
             )
             security_group_id = security_group.id
+            firewall_allow_from_subnet = resources.VCN_CIDR
+        else:
+            # A user-managed (custom) NSG is in use. Place the instance in a
+            # dedicated VCN/subnet that has no OCI security list, so the NSG is
+            # the sole security boundary. This is a *separate* VCN, not a second
+            # subnet in the default one: the default subnet already occupies the
+            # default VCN's entire CIDR block, so a second subnet there would
+            # conflict. The default VCN/subnet is left completely untouched,
+            # so instances using dstack's auto-managed NSG are unaffected.
+            subnet = resources.set_up_restricted_network_resources_in_region(
+                compartment_id=self.config.compartment_id,
+                project_name=instance_config.project_name,
+                client=region.virtual_network_client,
+            )
+            firewall_allow_from_subnet = resources.RESTRICTED_VCN_CIDR
 
         cloud_init_user_data = get_user_data(
             authorized_keys=instance_config.get_public_keys(),
-            firewall_allow_from_subnets=[resources.VCN_CIDR],
+            firewall_allow_from_subnets=[firewall_allow_from_subnet],
         )
 
         display_name = generate_unique_instance_name(instance_config)