Source code of SCOPE: Cost-Efficient Model Selection for Compound AI Systems under Quality Constraints

Reproduction Environment

• OS: Linux 4.15.0-197-generic (x86_64)
• Python: 3.10.18
• Runtime entrypoint: run.py (project root)
• Core dependencies: see requirements.txt

Minimal Installation Steps

From a directory that already contains this source code:

conda create -n venv-scope python=3.10 -y
conda activate venv-scope
pip install -r requirements.txt

Optional: set model API keys, where API KEYs from 4 platforms are required.

export OPENAI_API_KEY=...
export GOOGLE_API_KEY=...
export DEEPINFRA_API_KEY=...
export ANTHROPIC_API_KEY=...

If API keys are not set, it will still run through cached evaluations and raise errors for unseen evaluations.

Reproducing the results

The experiments in our paper are reproducible via one-shot runs. Taking SCOPE's best feasible cost in RQ1 as an example, it can be output by the command below:

python run.py --workflow text-to-sql --optimizer SCOPEOptimizer

Note that throughout the process, around 5000 USD are spent in calling the LLMs. To maximize reproducibility during the review process, we upload the data used and cached throughout the experiments, including the datasets and the LLM output cache, to an anonymous platform.

From the project root, one can download and extract the data with:

wget -O data_workspace.tar.gz "https://osf.io/download/698caf1549e3eb9b9cc728e6/?view_only=3aa1f1f1509e499abee999fb210566a4"
tar -xzf data_workspace.tar.gz

After extraction, two folders, data/ and workspace/, will appear at the project root directory.

This way, if one's environment setting is similar to ours (therefore the random seed behavior is identical), then most runs can directly use the cache and reproduce the same results.

However, in our observations, due to nondeterministic parallel execution and floating-point numerical differences, an optimizer's search trajectory can vary slightly even on the same machine (after several CUP days).