Skip to content

API_DOCUMENTATION.md

Latest commit

 

History

History
768 lines (614 loc) · 12.8 KB

API_DOCUMENTATION.md

File metadata and controls

768 lines (614 loc) · 12.8 KB
Raw

StudyBot + ovserve API Documentation

Overview

The system provides two API layers:

  1. Client API (Flask) - Port 5000 - User-facing chat interface
  2. Server API (FastAPI/ovserve) - Port 11435 - Model inference engine (Ollama-compatible)

Client API (Flask - Port 5000)

1. Main Chat Endpoint

Request:

POST /chat
Content-Type: application/json

{
  "message": "Explain photosynthesis in simple terms",
  "session_id": "user_session_123",
  "model": "OpenVINO/gemma-3-4b-it-int4-ov",
  "image": "data:image/png;base64,iVBORw0KGgoAAAANS...",  // Optional
  "study_mode": false                                          // Optional
}

Response: Server-Sent Events (SSE) Stream

data: {"token": "Photosynthesis"}
data: {"token": " is"}
data: {"token": " the"}
...
data: {"done": true}

Status Codes:

  • 200 - Stream started successfully
  • 400 - Empty message (no message and no image)
  • 500 - Server error during generation

Notes:

  • Response streams in real-time via SSE
  • System prompt is injected server-side
  • Study Mode adds follow-up questions to responses
  • Image support for vision models (Gemma VLM)
  • Session automatically created if doesn't exist

2. Abort Generation

Request:

POST /abort
Content-Type: application/json

{
  "session_id": "user_session_123"
}

Response:

{
  "status": "ok",
  "session_id": "user_session_123"
}

Status Codes:

  • 200 - Abort relayed to server
  • 400 - Missing session_id
  • 500 - Server communication error

Notes:

  • Stops token generation for the specified session
  • Relays request to /api/abort on ovserve

3. Clear Session

Request:

POST /clear
Content-Type: application/json

{
  "session_id": "user_session_123"
}

Response:

{
  "status": "cleared"
}

Notes:

  • Deletes all messages from session
  • Session entry remains (can be recreated)

4. List All Sessions

Request:

GET /api/history

Response:

[
  {
    "session_id": "session_123",
    "title": "Explain photosynthesis...",
    "updated_at": "2026-04-25T12:34:56.789Z"
  },
  {
    "session_id": "session_456",
    "title": "How do neural networks...",
    "updated_at": "2026-04-25T11:20:15.123Z"
  }
]

Status Codes:

  • 200 - Success

5. Load Specific Session

Request:

GET /api/history/<session_id>

Response:

{
  "session_id": "session_123",
  "title": "Explain photosynthesis in simple terms",
  "messages": [
    {
      "role": "user",
      "content": "Explain photosynthesis in simple terms"
    },
    {
      "role": "assistant",
      "content": "Photosynthesis is the process plants use to convert sunlight into chemical energy..."
    }
  ]
}

Status Codes:

  • 200 - Session found
  • 404 - Session not found (implicit - returns empty messages)

6. Delete Session

Request:

DELETE /api/delete/<session_id>

Response:

{
  "status": "deleted"
}

Status Codes:

  • 200 - Session deleted

7. Server Health Check

Request:

GET /health

Response (Server Running):

{
  "ollama": "running",
  "models": [
    "OpenVINO/gemma-3-4b-it-int4-ov",
    "OpenVINO/qwen2.5-7b-instruct-int4-ov"
  ]
}

Response (Server Down):

{
  "ollama": "not running"
}

Status Codes:

  • 200 - Server is healthy
  • 503 - Server is not running

Server API (FastAPI/ovserve - Port 11435)

1. Multi-turn Chat (Ollama Compatible)

Request:

POST /api/chat
Content-Type: application/json

{
  "model": "OpenVINO/gemma-3-4b-it-int4-ov",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "What is machine learning?"
    }
  ],
  "stream": true,                          // Optional, default: true
  "session_id": "user_session_123",        // Optional
  "options": {
    "num_predict": 2048,                  // Max tokens
    "device": "AUTO"                      // Device: AUTO, CPU, GPU, NPU
  }
}

Response (Streaming): NDJSON format

{"model": "...", "created_at": "2026-04-25T12:34:56.789Z", "message": {"role": "assistant", "content": "Machine"}, "done": false}
{"model": "...", "created_at": "2026-04-25T12:34:56.790Z", "message": {"role": "assistant", "content": " learning"}, "done": false}
...
{"model": "...", "created_at": "2026-04-25T12:34:57.123Z", "message": {"role": "assistant", "content": ""}, "done": true, "done_reason": "stop"}

Status Codes:

  • 200 - Stream started

2. Single Prompt Generate (Ollama Compatible)

Request:

POST /api/generate
Content-Type: application/json

{
  "model": "OpenVINO/gemma-3-4b-it-int4-ov",
  "prompt": "What is photosynthesis?",
  "stream": true,
  "image": "data:image/png;base64,...",   // Optional (for VLMs)
  "options": {
    "num_predict": 512,
    "device": "GPU"
  }
}

Response (Streaming):

{"model": "...", "created_at": "...", "response": "Photosynthesis", "done": false}
{"model": "...", "created_at": "...", "response": " is", "done": false}
...

3. OpenAI-Compatible Chat Completions

Request:

POST /v1/chat/completions
Content-Type: application/json

{
  "model": "OpenVINO/gemma-3-4b-it-int4-ov",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "What is the capital of France?"
    }
  ],
  "max_tokens": 512,
  "stream": false
}

Response (Non-Streaming):

{
  "id": "chatcmpl-123abc",
  "object": "chat.completion",
  "created": 1703070000,
  "model": "OpenVINO/gemma-3-4b-it-int4-ov",
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 5,
    "total_tokens": 30
  },
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ]
}

4. List Available Models

Request:

GET /api/tags

Response:

{
  "models": [
    {
      "name": "OpenVINO/gemma-3-4b-it-int4-ov",
      "modified_at": "2026-04-20T10:15:00Z",
      "size": 2.4
    },
    {
      "name": "OpenVINO/qwen2.5-7b-instruct-int4-ov",
      "modified_at": "2026-04-18T08:30:00Z",
      "size": 4.2
    }
  ]
}

5. Pull Model from HuggingFace

Request:

POST /api/pull
Content-Type: application/json

{
  "model": "microsoft/Phi-3-mini-4k-instruct",
  "stream": true
}

Response (Streaming): NDJSON format

{"status": "pulling", "digest": "sha256:abc123", "total": 5000000000}
{"status": "downloading", "digest": "sha256:abc123", "completed": 1000000000, "total": 5000000000}
...
{"status": "done", "digest": "sha256:abc123"}

Status Codes:

  • 200 - Pull started

6. Delete Model

Request:

POST /api/delete
Content-Type: application/json

{
  "model": "microsoft/Phi-3-mini-4k-instruct"
}

Response:

{
  "status": "deleted",
  "model": "microsoft/Phi-3-mini-4k-instruct"
}

7. Load Model onto Device

Request:

POST /api/load
Content-Type: application/json

{
  "model": "OpenVINO/gemma-3-4b-it-int4-ov",
  "device": "GPU"   // AUTO, CPU, GPU, NPU
}

Response:

{
  "status": "loaded",
  "model": "OpenVINO/gemma-3-4b-it-int4-ov",
  "device": "GPU"
}

8. List Running Models

Request:

GET /api/ps

Response:

{
  "models": [
    "OpenVINO/gemma-3-4b-it-int4-ov"
  ]
}

9. Get Device Information

Request:

GET /api/devices

Response:

{
  "devices": {
    "CPU": "Intel(R) Core(TM) i9-13900KS",
    "GPU": "Intel(R) Arc(TM) A770M Graphics",
    "NPU": "Intel(R) AI Boost NPU"
  }
}

10. Get Benchmark Results

Request:

GET /api/benchmarks

Response:

{
  "benchmarks": {
    "OpenVINO/gemma-3-4b-it-int4-ov": {
      "NPU": {
        "device": "NPU",
        "worked": true,
        "model_load_s": 1.234,
        "tokens_per_sec": 120.5,
        "stateful": false
      },
      "GPU": {
        "device": "GPU",
        "worked": true,
        "model_load_s": 2.156,
        "tokens_per_sec": 85.3,
        "stateful": false
      },
      "CPU": {
        "device": "CPU",
        "worked": true,
        "model_load_s": 0.567,
        "tokens_per_sec": 15.2,
        "stateful": false
      }
    }
  }
}

11. Abort Generation

Request:

POST /api/abort
Content-Type: application/json

{
  "session_id": "user_session_123"
}

Response:

{
  "status": "abort requested",
  "session_id": "user_session_123"
}

12. Server Version

Request:

GET /api/version

Response:

{
  "version": "0.1.0-ovserve"
}

Error Handling

Common Error Responses

Model Not Found:

{
  "error": "Model 'invalid-model' not found in registry"
}

Device Not Available:

{
  "error": "Device 'NPU' not available on this machine"
}

Generation Timeout:

{
  "error": "Model loading timed out after 300 seconds"
}

Invalid Input:

{
  "error": "Image exceeds maximum size of 10MB"
}

Streaming Format Details

Server-Sent Events (SSE) - Client API

event: (empty)
data: {"token": "The"}
retry: (optional)

event: (empty)
data: {"token": " quick"}

...

event: (empty)
data: {"done": true}

NDJSON - Server API

{"model": "...", "created_at": "...", "message": {"role": "assistant", "content": "The"}, "done": false}
{"model": "...", "created_at": "...", "message": {"role": "assistant", "content": " quick"}, "done": false}
...
{"model": "...", "created_at": "...", "message": {"role": "assistant", "content": ""}, "done": true, "done_reason": "stop"}

Rate Limiting (TODO)

Currently no rate limiting is implemented. Recommended:

Per Client IP:
  - 60 requests per minute for chat endpoints
  - 10 requests per minute for model operations

Per Session:
  - 1 concurrent generation (queue others)
  - Maximum 100 messages in history

Authentication (TODO)

Recommended implementation:

Authorization: Bearer <jwt_token>

JWT payload:

{
  "sub": "user_id",
  "exp": 1703156400,
  "iat": 1703070000,
  "scopes": ["chat", "models"]
}

Webhook Events (Future)

POST /webhooks/callback
{
  "event": "generation.complete",
  "session_id": "...",
  "model": "...",
  "tokens_generated": 42,
  "generation_time_ms": 5234,
  "timestamp": "2026-04-25T12:34:56.789Z"
}

SDK Examples

Python

import requests

# Single chat message
response = requests.post(
    "http://localhost:11435/api/chat",
    json={
        "model": "OpenVINO/gemma-3-4b-it-int4-ov",
        "messages": [{"role": "user", "content": "Hello!"}],
        "stream": True
    },
    stream=True
)

for line in response.iter_lines():
    if line:
        chunk = line.json()
        print(chunk["message"]["content"], end="", flush=True)

JavaScript/Node.js

const response = await fetch("http://localhost:5000/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    message: "Explain recursion",
    session_id: "user_123",
    model: "OpenVINO/gemma-3-4b-it-int4-ov"
  })
});

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  
  const chunk = decoder.decode(value);
  const data = JSON.parse(chunk.substring(6)); // Remove "data: "
  console.log(data.token);
}

Testing Endpoints with cURL

# List models
curl http://localhost:11435/api/tags

# Chat
curl -X POST http://localhost:11435/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "OpenVINO/gemma-3-4b-it-int4-ov",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'

# Health check
curl http://localhost:5000/health

# List sessions
curl http://localhost:5000/api/history

Performance Metrics

Typical Response Times (on Intel Arc GPU)

Operation Duration
Model load (first time) 2-5 seconds
Model load (cached) <100ms
First token generation 0.5-1 second
Subsequent tokens 50-200ms each
Full 256-token response 8-12 seconds

Resource Usage

Model Memory VRAM Min CPU
Gemma 3 4B (int4) 2.4GB 2GB (GPU) 10%
Qwen 2.5 7B (int4) 4.2GB 4GB (GPU) 15%

Backward Compatibility

  • API version: 0.1.0-ovserve
  • Ollama API: Fully compatible with v1 endpoints
  • OpenAI API: Partial support (/v1/chat/completions)
  • Breaking changes will be announced with version bump