Custom LLM?

[BruthaVoodoo]

Hello, is there a way to use a LLM other than OpenAI, Gemini or Ollama? My organization has given me access to a LLM via URL and API key. If not, are there plans to add this capability?

Thanks

[coleam00]

Coming soon! Stay tuned!

[BruthaVoodoo]

Hello @coleam00.

Hope you're doing well. I just did a pull to get the latest. Was there any movement on this? I see some extra LLM stuff but nothing that I could identify as a "custom" LLM?

thanks
Jason

[Chillbruhhh]

yes that should be something rather easy to integrate since amazon works with openai compatible systems. definitely something that shouldn't be hard to integrate, just an easy way to swap base urls and model names for custom providers that support openai's structure.

[Chillbruhhh]

@coleam00 if this is something you would like to add i dont mind building the setup, whether it just be a setting where you drop openais base url or offering a custom icon to click in the llm providers tab to configure how you would like. maybe a custom "+" provider icon that allows you to add custom llm provider models, starting with openai compatible providers, and then a custom sliding scroll bar so it doesn't clog the UI and flows smoothly within the settings container

[Chillbruhhh]

i dont want to bloat my current pr since its streamed at the focus of the llm provider redesign, trying to keep it granular. I plan on upgrading the local support (ollama) to work with LM Studio and docker models in a different PR anyways by swapping the url from there. i could add it in something like that or its own separate pr if you'd like

[py88ai]

@coleam00 Am a fan of you and the other contributor's fantastic work on this project. Building on this request, it would be great if we can also use to LM Studio, or OpenRouter. Thanks and keep it going !

[Chillbruhhh]

@coleam00 Am a fan of you and the other contributor's fantastic work on this project. Building on this request, it would be great if we can also use to LM Studio, or OpenRouter. Thanks and keep it going !

lol OpenRouter has been added, update your version to the most recent!