A guide for setting up the LLaMA 2 LLM on a local machine.
- LLaMA-2 Setup Guide
You'll need a capable machine that:
- Has enough RAM to fit any models you want to use in memory
- GPU memory (VRAM) is highly preferred (see CPU vs GPU Memory)
- Has sufficient local storage to store models
- Has Python 3.8 or higher installed
- Preferably runs Linux (Ubuntu is recommended)
Note that MacOS does not support CUDA (required for GPU acceleration), but you can still run models on the CPU.
LLaMA 2 is available in three sizes: 7B, 13B, and 70B. These numbers refer to the number of parameters in the model. Generally, smaller models are much faster but less accurate, while larger models are slower but more accurate.
In my experience, the smaller models are sufficient for question answering and text generation, but struggle with more complex tasks like logic puzzles and decision making based on evidence. I recommend trying out various models to see what works best for your use case.
You might come across a few different formats of LLaMA2 models online, including transformers, GPTQ, GGML, GGUF, and others. Each of these has a different setup process and requirements (which I won't include here). I reccomend using GGUF models via llama.cpp because:
- They support splitting models between CPU and GPU memory (see CPU vs GPU Memory)
- They support quantization, which greatly reduces the memory footprint with minimal accuracy loss
- Easy setup and transportability (since they are just a single file)
When running any LLM, the entire model must be loaded into the memory of the machine. Every model has different memory requirements, typically listed somewhere in the model's documentation. The larger the model, the more memory it requires.
There are two types of memory available on most machines: CPU (RAM) and GPU (VRAM). CPU memory is typically much larger than GPU memory, but is also significantly slower. I highly recommend running your model on a GPU if possible.
One benefit of using a GGUF model is that you can split the model between CPU and GPU memory. For instance, if you are running a 20GB model on a system with only 12GB of VRAM, you can split the model into roughly 12GB on the GPU and 8GB on the CPU. It will be slower than running the entire model on the GPU, but much faster than running the entire model on the CPU.
HuggingFace is a great resource for finding and downloading models. You can find the list of all Llama 2 models here. Remember that there are multiple model formats, so look for GGUF if you want to use llama.cpp.
GGUF models often have multiple downloads for different quantization levels (Q4_K_M, Q2_K, etc). This affects the accuracy and memory footprint of the model. I recommend starting with Q4_K_M for balanced size and quality.
- orca_mini_v3_7B is a great starting point and one of the best 7B models I've found. It also comes in 13B and 70B sizes if you need better accuracy.
- Upstage-Llama-2-70B-instruct-v2-GGUF is one of the top ranked 70B models on huggingface and has been effective for decision making and other complex tasks.
Many models on HuggingFace are only available in the transformers (hf) format. They can be converted to the GGUF format by following this guide.
This is optional if you only plan to run models on the CPU (not recommended). Make sure your GPU is CUDA compatible before following these steps. You can check your GPU's compatibility here.
[!] Important: The steps here are for intended for Linux systems. If you are using Windows, see CUDA Setup For Windows. MacOS does not support CUDA.
First, check if you have CUDA installed already:
nvcc --versionIf you get a "command not found" error, you'll need to install the CUDA toolkit from here. Once installed, nvcc --version should return a version number.
If you still get an error running nvcc --version after installation, you may need to manually add CUDA to your path. Check out this post for instructions.
[!] I highly recommend using a virtual environment for this installation (either conda or venv). Documentation on this can be found here.
We will use the llama-cpp-python package to run our model:
- Install the
llama-cpp-pythonpackage in your virtual environment. The command will differ depending on platform:
# CPU Only
pip install llama-cpp-python
# With CUDA Support (Linux)
CMAKE_ARGS=-DLLAMA_CUBLAS=on FORCE_CMAKE=1 pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir
# With CUDA Support (Windows)
# Note: These are two separate commands, run them separately
set CMAKE_ARGS=-DLLAMA_CUBLAS=on FORCE_CMAKE=1
pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir- Create a python script to load and query your model. See this example (also in
sample_query.py):
from llama_cpp import Llama
llm = Llama(
model_path="orca_mini_v3_7b.Q4_K_M.gguf", # Path to model file
n_ctx=4096, # Context window size
n_gpu_layers=-1, # -1 to use all GPU layers, 0 to use only CPU
verbose=False # Whether to print debug info
)
output = llm(
"Q: Name the planets in the solar system? A: ", # Prompt
max_tokens=50, # Generate up to 32 tokens
stop=["Q:", "\n"], # Keywords to stop generation at
echo=True # Echo the prompt back in the output
)
print(output['choices'][0]['text']) # Print the generated outputRemember to replace PATH_TO_MODEL with the path to your model file (from this script).
- Run the script!
python sample_query.pyYou should see an output similar to this:
Q: Name the planets in the solar system? A: 8 Planets in our Solar System are Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus and Neptune
Note that this may take a while to run since the model must be loaded into memory. Once the model is loaded (by calling Llama()), subsequent calls to llm() will be much faster.
To ensure you are using GPU memory, you can run with verbose=True to verify that layers have been loaded onto the GPU's VRAM via CUDA. You can also run nvidia-smi -l 1 in a separate terminal to monitor GPU memory usage (VRAM usage should go up when you run it).
You may see CUDA error: out of memory if the model is too large to fit in GPU memory. There are a few ways to work around this:
- Split layers between the CPU and GPU by modifying the
n_gpu_layersparameter in the code. Theverbose=Trueoutput should show the total number of layers the model has. Try settingn_gpu_layersto a number less than this. - Use a model with a smaller quantization level (e.g.
Q2_Kinstead ofQ4_K_M) - Use a smaller model (e.g. 7B instead of 13B)
While you could run and query your model from directly within your application, it is often more convenient to run the model on a separate server and query it via an API (similar to how OpenAI's API works). There are many possible ways to do this, but a simple solution is a basic Flask API. A sample of this can be found in sample_api.py.
To use this, simply pip install flask, run the script, and send a POST request to http://127.0.0.1:5000 with the following JSON body:
{
"prompt": "Tell me a fun fact about dogs",
"max_tokens": 50,
}By default, your Flask API will only be accessible from your local machine. You can use a service like ngrok to expose your API to external requests. See the docs for more information.
Guidance is a tool that serves as a language for controlling LLMs, offering features such as constrained generation, templates, custom grammars, and more. Check out the Guidance repo here. A brief example has been provided in sample_guidance.py for reference.
Setting up CUDA on Windows involves a few extra steps, detailed below. These steps are kindly provided by Qingwen Zeng.
Download GCC for Windows here: https://sourceforge.net/projects/mingw/
Once installed, add the bin folder to your $PATH variable:
C:\MinGW\binDownload Visual Studio here: https://visualstudio.microsoft.com/downloads/
The installer will also install CMake, which you will also need to add to your $PATH variable:
C:\Program Files\CMake\binCopy the 4 VS integration files:
# copy the 4 files from:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\extras\visual_studio_integration\MSBuildExtensions
# to both of these locations:
C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\BuildCustomizations
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\BuildCustomizationsIf you encounter a No CUDA toolset found error, see here
Download the CUDA Toolkit from here. Once installed, add the bin folder to your $PATH variable:
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\binTo verify that CUDA is installed, run the following command:
nvcc --version # should return a version numberIf this works, CUDA is installed! You may proceed with running your model.