Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 110 additions & 2 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,13 +59,24 @@ jobs:

with open("pyproject.toml", "rb") as file:
pyproject = tomllib.load(file)
with open("Cargo.toml", "rb") as file:
cargo = tomllib.load(file)

package_version = pyproject["project"]["version"]
workspace_version = cargo["workspace"]["package"]["version"]
if workspace_version != package_version:
raise SystemExit(
"Python package version "
f"{package_version!r} does not match Rust workspace version "
f"{workspace_version!r}"
)

expected = f"v{package_version}"
if tag != expected:
raise SystemExit(
f"release tag {tag!r} does not match pyproject version {package_version!r}"
)
print(f"release tag {tag} matches pyproject version {package_version}")
print(f"release tag {tag} matches Python and Rust version {package_version}")
PY

python-release-checks:
Expand Down Expand Up @@ -417,7 +428,7 @@ jobs:
publish:
name: Publish distributions
if: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') }}
needs: [source-dist, wheels]
needs: [source-dist, wheels, publish-rust-server]
runs-on: ubuntu-latest
environment: pypi
permissions:
Expand All @@ -438,3 +449,100 @@ jobs:

- name: Create GitHub Release
uses: softprops/action-gh-release@v2

publish-rust-protocol:
name: Publish switchyard-protocol
if: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') }}
needs: [source-dist, wheels]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Publish crate
env:
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
run: cargo publish --locked --package switchyard-protocol

publish-rust-foundations:
name: Publish ${{ matrix.crate }}
if: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') }}
needs: [publish-rust-protocol]
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
crate: [switchyard-libsy, switchyard-translation]
steps:
- uses: actions/checkout@v4
- name: Wait for registry dependencies
shell: bash
run: |
set -euo pipefail
version="${GITHUB_REF_NAME#v}"
for attempt in {1..30}; do
cargo info --registry crates-io "switchyard-protocol@${version}" >/dev/null 2>&1 \
&& exit 0
echo "Waiting for switchyard-protocol ${version} (${attempt}/30)"
sleep 10
done
echo "switchyard-protocol ${version} did not reach the crates.io index" >&2
exit 1
- name: Publish crate
env:
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
run: cargo publish --locked --package "${{ matrix.crate }}"

publish-rust-client:
name: Publish switchyard-llm-client
if: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') }}
needs: [publish-rust-foundations]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Wait for registry dependencies
shell: bash
run: |
set -euo pipefail
version="${GITHUB_REF_NAME#v}"
for attempt in {1..30}; do
ready=true
for crate in switchyard-libsy switchyard-translation; do
cargo info --registry crates-io "${crate}@${version}" >/dev/null 2>&1 \
|| ready=false
done
if [[ "${ready}" == true ]]; then
exit 0
fi
echo "Waiting for Rust foundation crates ${version} (${attempt}/30)"
sleep 10
done
echo "Rust foundation crates ${version} did not reach the crates.io index" >&2
exit 1
- name: Publish crate
env:
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
run: cargo publish --locked --package switchyard-llm-client

publish-rust-server:
name: Publish switchyard-server
if: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') }}
needs: [publish-rust-client]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Wait for registry dependencies
shell: bash
run: |
set -euo pipefail
version="${GITHUB_REF_NAME#v}"
for attempt in {1..30}; do
cargo info --registry crates-io "switchyard-llm-client@${version}" >/dev/null 2>&1 \
&& exit 0
echo "Waiting for switchyard-llm-client ${version} (${attempt}/30)"
sleep 10
done
echo "switchyard-llm-client ${version} did not reach the crates.io index" >&2
exit 1
- name: Publish crate
env:
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
run: cargo publish --locked --package switchyard-server
19 changes: 8 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,28 +59,25 @@ switchyard launch claude --model my-route --config routes.toml

### Server Path

Use this path to build and run the standalone Rust proxy. Install
[Rust with Cargo](https://rust-lang.org/tools/install/), then build the release
binary from source:
Use this path to install and run the standalone Rust proxy. Install
[Rust with Cargo](https://rust-lang.org/tools/install/), then install the
published binary:

```bash
git clone https://github.com/NVIDIA-NeMo/Switchyard.git
cd Switchyard
cargo build --locked --release -p switchyard-server
./target/release/switchyard-server --help
cargo install --locked switchyard-server
switchyard-server --help
```

Prebuilt binaries are not published yet. `rustup` installs the pinned toolchain
automatically when you run Cargo from the repository.
Cargo builds the release binary and installs it into `~/.cargo/bin` by default.

Create `routes.toml` using the
[Getting Started guide](docs/getting_started.md#server-path), then validate it
and start the server:

```bash
export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
./target/release/switchyard-server --config routes.toml --dry-run
./target/release/switchyard-server --config routes.toml --host 127.0.0.1 --port 4000
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000
```

Verify the proxy in another terminal:
Expand Down
5 changes: 5 additions & 0 deletions crates/libsy-llm-client/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,14 @@ version.workspace = true
description = "HTTP LLM client speaking Switchyard's neutral IR directly"
authors.workspace = true
edition.workspace = true
homepage = "https://github.com/NVIDIA-NeMo/Switchyard"
license.workspace = true
readme = "README.md"
repository.workspace = true
rust-version.workspace = true
documentation = "https://docs.rs/switchyard-llm-client"
keywords = ["llm", "client", "openai", "anthropic"]
publish = ["crates-io"]

[dependencies]
switchyard-libsy.workspace = true
Expand Down
7 changes: 6 additions & 1 deletion crates/switchyard-server/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,14 @@ version.workspace = true
description = "Rust HTTP server surface for libsy algorithms"
authors.workspace = true
edition.workspace = true
homepage = "https://github.com/NVIDIA-NeMo/Switchyard"
license.workspace = true
readme = "README.md"
repository.workspace = true
rust-version.workspace = true
documentation = "https://docs.rs/switchyard-server"
keywords = ["llm", "proxy", "routing", "openai", "anthropic"]
publish = ["crates-io"]

[dependencies]
async-stream.workspace = true
Expand All @@ -20,7 +25,7 @@ rustls = { version = "0.23", default-features = false, features = ["aws-lc-rs",
clap = { version = "4", features = ["derive", "env"] }
futures-util.workspace = true
http.workspace = true
libsy = { package = "switchyard-libsy", path = "../libsy" }
libsy = { package = "switchyard-libsy", path = "../libsy", version = "0.2.0" }
Comment thread
grahamking marked this conversation as resolved.
opentelemetry = { version = "0.32", default-features = false, features = ["metrics", "trace"] }
opentelemetry-otlp = { version = "0.32", default-features = false, features = ["http-proto", "metrics", "reqwest-blocking-client", "reqwest-rustls", "trace"] }
opentelemetry-prometheus = "0.32"
Expand Down
3 changes: 2 additions & 1 deletion crates/switchyard-server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ confidence_threshold = 0.5

```bash
export API_KEY="..."
cargo run -p switchyard-server -- --config routes.toml
cargo install --locked switchyard-server
switchyard-server --config routes.toml
```

Ctrl+C and Unix `SIGTERM` stop new connections and allow active requests to drain for up to
Expand Down
14 changes: 5 additions & 9 deletions docs/cli_reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,19 +50,15 @@ This command does not install or run the standalone `switchyard-server` binary.

## Server Path: `switchyard-server`

Build the standalone binary with
`cargo build --locked --release -p switchyard-server`. It reads the same native
TOML deployment schema accepted by the launcher.
Install the standalone binary with `cargo install --locked switchyard-server`.
It reads the same native TOML deployment schema accepted by the launcher.

### Usage

```bash
switchyard-server --config <deployment.toml> [options]
```

When built from the repository, invoke it as
`./target/release/switchyard-server`.

| Option | Default | Purpose |
|---|---|---|
| `--config PATH` | Required | TOML file defining LLM clients, targets, and algorithm routes. |
Expand All @@ -80,8 +76,8 @@ When built from the repository, invoke it as
Validate a deployment, then start the proxy:

```bash
./target/release/switchyard-server --config routes.toml --dry-run
./target/release/switchyard-server --config routes.toml \
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml \
--host 127.0.0.1 --port 4000
```

Expand All @@ -102,7 +98,7 @@ switchyard launch claude --model my-route --config routes.toml
```

The CLI does not save provider credentials or deployment paths.
Use `./target/release/switchyard-server --config routes.toml --dry-run` to
Use `switchyard-server --config routes.toml --dry-run` to
validate a native deployment before starting the standalone server.

## Related Documentation
Expand Down
20 changes: 8 additions & 12 deletions docs/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,20 +107,16 @@ cargo --version
uv --version
```

### Build the server
### Install the server

Build the Rust server from source:
Install the Rust server from crates.io:

```bash
git clone https://github.com/NVIDIA-NeMo/Switchyard.git
cd Switchyard
cargo build --locked --release -p switchyard-server
./target/release/switchyard-server --help
cargo install --locked switchyard-server
switchyard-server --help
```

The repository pins Rust `1.96.1` in `rust-toolchain.toml`; `rustup` selects and
installs it automatically when Cargo runs from the repository. Prebuilt Rust
binaries are not published yet.
Cargo builds the release binary and installs it into `~/.cargo/bin` by default.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

### Configure

Expand Down Expand Up @@ -166,8 +162,8 @@ socket, then start the release binary:

```bash
export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
./target/release/switchyard-server --config routes.toml --dry-run
./target/release/switchyard-server --config routes.toml \
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml \
--host 127.0.0.1 --port 4000
```

Expand Down Expand Up @@ -211,7 +207,7 @@ the complete TOML schema, route options, TLS, and metrics.

```bash
test -n "$OPENROUTER_API_KEY" && echo "key is set" || echo "key is missing"
./target/release/switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml --dry-run
```

Confirm that `api_key_env` in `routes.toml` names the environment variable you
Expand Down
19 changes: 17 additions & 2 deletions docs/internal/release_workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ Switchyard currently follows the OSS-style NeMo path for GitHub builds:
- manual dev builds create one Linux x86_64 wheel as a one-day GitHub Actions artifact;
- manual dev matrix builds create the full sdist and wheel set as GitHub Actions artifacts;
- root `vMAJOR.MINOR.PATCH` tags run the complete release validation and wheel matrix;
- public PyPI/GitHub publishing happens only from approved `vMAJOR.MINOR.PATCH` tag releases.
- public PyPI, crates.io, and GitHub publishing happens only from approved
`vMAJOR.MINOR.PATCH` tag releases.

Wheel metadata uses the public distribution name `nemo-switchyard`, while the Python import and CLI
stay `switchyard`.
Expand Down Expand Up @@ -69,7 +70,8 @@ Create a root `vMAJOR.MINOR.PATCH` tag only when a real release has been approve
- native wheel smoke installs where the runner can execute the artifact.

The workflow rejects release tags that do not exactly match `pyproject.toml`'s package version. For
example, package version `0.0.1` must be released with the `v0.0.1` tag.
example, package version `0.2.0` must be released with the `v0.2.0` tag. The Rust workspace and
Python package versions must also match.

The official `publish` job uses `uv publish --trusted-publishing always`, so PyPI project creation
and uploads require a matching pending trusted publisher:
Expand All @@ -85,6 +87,19 @@ and uploads require a matching pending trusted publisher:
Do not create a root release tag until the PyPI pending publisher and GitHub `pypi` environment are
ready.

The same tag publishes these crates to crates.io in dependency order:

1. `switchyard-protocol`
2. `switchyard-libsy`
3. `switchyard-translation`
4. `switchyard-llm-client`
5. `switchyard-server`

Add a repository Actions secret named `CARGO_REGISTRY_TOKEN` containing a crates.io API token that
can publish all five crates and create new crates. The job waits for each version to reach the
crates.io index before publishing its dependents. If publication stops partway through, use
GitHub's **Re-run failed jobs** action so successful crate jobs are not repeated.

## Local Metadata Helper

To preview the metadata stamp locally:
Expand Down
8 changes: 4 additions & 4 deletions docs/routing_algorithms/escalation_router_routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,14 +117,14 @@ resets the streak to zero.

## Run the route

After building the Rust server, as described in
[Getting Started](../getting_started.md#build-the-server), export the provider
After installing the Rust server, as described in
[Getting Started](../getting_started.md#install-the-server), export the provider
credential, validate the configuration, and start the binary:

```bash
export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
./target/release/switchyard-server --config routes.toml --dry-run
./target/release/switchyard-server --config routes.toml \
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml \
--host 127.0.0.1 --port 4000
```

Expand Down
6 changes: 3 additions & 3 deletions docs/routing_algorithms/llm_classifier_routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,14 +209,14 @@ user-message text.

## Run the route

After [building the Rust server](../getting_started.md#build-the-server), export
After [installing the Rust server](../getting_started.md#install-the-server), export
the provider credential, validate the configuration, and start the release
binary:

```bash
export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
./target/release/switchyard-server --config routes.toml --dry-run
./target/release/switchyard-server --config routes.toml \
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml \
--host 127.0.0.1 --port 4000
```

Expand Down
Loading
Loading