VisionX Smart Camera System - Abstract

1. Introduction

VisionX is an intelligent vision-based camera system that combines hardware integration with AI-powered analysis to provide real-time scene understanding and object recognition. The system leverages the ESP32 AI Thinker Cam Module for video capture, a Flask-based backend for image processing, and Ollama's vision models for scene analysis.

Market Context: The growing demand for smart home assistants and AI-powered surveillance solutions has created a need for low-cost, privacy-focused vision systems that operate locally without cloud dependencies.

Key Features:

Live video streaming from ESP32 camera module
Touch sensor-triggered AI detection via TTP223B
Real-time scene description using Ollama vision models (gemma3:27b-cloud)
128x32 I2C OLED display for short labels
Text-to-Speech (TTS) output for accessibility
Video recording capability
Mobile-responsive web interface

2. Problem Statement

2.1 Accessibility Barriers

Existing AI vision systems require expensive cloud subscriptions and compromise user privacy by sending video data to external servers. VisionX addresses this by processing all analysis locally using Ollama.

2.2 Education Gaps

Users need an intuitive way to understand what their camera sees without technical knowledge. The system provides natural language descriptions and optional TTS output.

2.3 Technical Limitations

Traditional smart cameras lack local processing capabilities. VisionX integrates ESP32 hardware with Python AI backend to create a self-contained vision system.

2.4 Project Objectives

Build a functional prototype with <$30 hardware cost
Achieve <5 second detection latency
Provide both detailed descriptions and OLED-compatible short labels
Enable touch-triggered capture for hands-free operation

3. System Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        SYSTEM ARCHITECTURE                      │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────────┐    │
│  │   ESP32     │────▶│   Web UI    │────▶│   Flask API     │    │
│  │  Camera &   │     │  (VX3.html) │     │  (app.py)       │    │
│  │  Touch      │     │             │     │                 │    │
│  └─────────────┘     └─────────────┘     └────────┬────────┘    │
│       │                    │                      │             │
│       │                    │                      ▼             │
│  ┌────▼────┐          ┌────▼────┐           ┌──────────────┐    │
│  │ Video   │          │ Stream  │           │   Ollama     │    │
│  │ Stream  │          │ Display │           │ gemma3:27b   │    │
│  │ :9087   │          │ :80     │           │   Vision     │    │
│  └─────────┘          └─────────┘           └──────────────┘    │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Communication Protocol

ESP32 → Web UI: MJPEG stream on port 9087, REST endpoints on port 80
Web UI → Flask API: HTTP POST to /api/detect
Flask → Ollama: HTTP POST to http://localhost:11434/api/chat

Request/Response JSON Formats

Detection Request (multipart/form-data):

{
  "image": "<binary JPEG data>"
}

Detection Response:

{
  "success": true,
  "result": {
    "filename": "frame.jpg",
    "mime_type": "image/jpeg",
    "size_bytes": 24567,
    "description": "Alice and Bob are having a conversation in the living room. Both are facing the camera.",
    "short-description": "Alice & Bob"
  }
}

4. Hardware Design

Component Table with Costs

Component	Model	Cost (USD)
Main Controller	ESP32 AI Thinker Cam	$8.50
Touch Sensor	TTP223B	$1.20
Display	128x32 I2C OLED	$3.50
Battery Management	TP4056	$1.00
Battery	300mAh LiPo	$2.50
Total		$16.70

Pin Configuration

GPIO	Function	Connection
GPIO2	Camera D0	Camera module
GPIO4	Camera D1	Camera module
GPIO16	Camera D2	Camera module
GPIO17	Camera D3	Camera module
GPIO5	Camera CLK	Camera module
GPIO21	I2C SDA	OLED SDA
GPIO22	I2C SCL	OLED SCL
GPIO13	Touch Input	TTP223B OUT

Technical Specifications

Parameter	Value
ESP32 Flash	4MB
Camera Resolution	640x480
OLED Resolution	128x32 pixels
Battery Voltage	3.7V
Operating Current	~250mA

5. Software Architecture

Firmware Modules (ESP32)

Camera initialization and streaming
Touch sensor polling (500ms interval)
REST API handlers (/touch, /display)
WiFi STA mode

Web Interface Features (VX3.html)

Live video stream display
Start/Stop streaming controls
Video recording with WebM export
Touch sensor status display
AI detection results panel
Text-to-Speech (auto/manual)
Mobile-responsive accordion layout

Grip Presets Table

N/A - This is a camera system, not a gripper.

API Endpoints Table

Endpoint	Method	Function
`/`	GET	Serve web UI
`/api/detect`	POST	AI image analysis
`/touch`	GET	Touch sensor status
`/display`	GET	Send text to OLED

Example curl Commands

# Touch sensor status
curl http://192.168.1.XXX:80/touch

# Send to OLED display
curl "http://192.168.1.XXX:80/display?msg=Alice%20%26%20Bob"

# AI detection (via Flask backend)
curl -X POST -F "image=@frame.jpg" http://192.168.1.XXX:80/api/detect

6. Mechanical Design

Finger Actuation Mechanism

N/A - This is a camera system, not a robotic gripper.

Assembly Sequence

Connect ESP32 AI Thinker to camera module
Wire TTP223B touch sensor to GPIO13
Connect OLED display via I2C (GPIO21/22)
Integrate TP4056 for battery management
Flash firmware and configure WiFi

Design Modifications

Initial design used larger 128x64 OLED, downgraded to 128x32 for compact form factor
Added TP4056 for portable operation with 300mAh battery

7. Implementation Details

Development Environment

Python 3.x with Flask and flask-cors
ESP32 Arduino framework
Ollama local AI server (gemma3:27b-cloud model)
Web browser with Web Speech API support

Source Code Structure

PythonImageTesting/
├── app.py              # Flask API server
└── templates/
    └── VX3.html        # Web interface

Key Functions

Flask API (app.py):

detect(): Handles image upload, calls Ollama, returns JSON with description and short label
load_images(): Loads known face embeddings (placeholder for face recognition)

Web UI (VX3.html):

toggleStream(): Start/stop ESP32 video stream
pollTouch(): Query touch sensor every 500ms
triggerDetect(): Capture frame, send to API, display results
speakText(): Web Speech API TTS output

8. Testing & Calibration

Hardware Verification Steps

Verify ESP32 WiFi connection
Confirm camera stream accessible at :9087
Test touch sensor response via /touch endpoint
Validate OLED display text rendering

Calibration Procedures

Adjust touch sensor sensitivity via onboard capacitor
Position camera for optimal field of view
Configure Ollama model timeout (60s for analysis, 30s for short label)

9. Key Features

Local AI Processing: All vision analysis runs on-device via Ollama, ensuring privacy and eliminating cloud costs.
Touch-Triggered Detection: TTP223B touch sensor initiates AI analysis without voice commands or app interaction.
Dual Output: Provides full scene descriptions for web UI and compact labels for 128x32 OLED display.
Accessibility Support: Text-to-Speech converts detection results to audio for visually impaired users.

10. System Workflows

Startup Sequence

┌──────────────────┐
│   Power On       │
└────────┬─────────┘
         ▼
┌──────────────────┐
│ ESP32 Init WiFi  │
└────────┬─────────┘
         ▼
┌──────────────────┐
│ Camera Stream    │
│ Available :9087  │
└────────┬─────────┘
         ▼
┌──────────────────┐
│ Web UI Ready     │
│ Port 80          │
└──────────────────┘

Command Processing

User Presses Touch Sensor
         │
         ▼
ESP32 /touch Returns HIGH
         │
         ▼
Web UI Triggers captureFrame()
         │
         ▼
POST to /api/detect
         │
         ▼
Flask Calls Ollama Vision
         │
         ▼
JSON Response Parsed
         │
         ▼
Display + TTS Output

11. Technology Summary

Hardware Table

Component	Purpose
ESP32 AI Thinker	Main controller, WiFi, camera interface
TTP223B Touch Sensor	User input trigger
128x32 I2C OLED	Compact status display
TP4056 + 300mAh	Battery management

Software Table

Software	Role
Flask + CORS	REST API server
VX3.html	Web interface
Ollama gemma3:27b	Vision AI model
Web Speech API	Text-to-Speech

Dependencies

flask
flask-cors
requests
opencv-python (placeholder)
Ollama running locally

12. Advantages & Benefits

Privacy-Focused: All image processing happens locally; no video data leaves the device.
Low Cost: Total hardware cost under $20 USD, compared to $100+ commercial alternatives.
Open Source: Built on Flask, standard web technologies, and Ollama - no proprietary components.
Extensible: Modular architecture allows adding face recognition, object detection, or gesture control.

13. Challenges & Limitations

Processing Speed: Ollama vision inference takes 3-5 seconds; not suitable for real-time applications.
WiFi Range: ESP32 operates in STA mode; requires consistent network coverage.
Power Constraints: 300mAh battery provides ~1 hour of operation; needs external power for extended use.

14. Future Enhancements

Face Recognition: Integrate face_recognition library to identify known individuals.
Object Detection: Add YOLO or MobileNet for multi-class object detection.
Edge AI: Migrate to TensorFlow Lite on ESP32 for offline inference.
Mobile App: Develop native iOS/Android companion app for alerts and configuration.

15. Conclusion

VisionX demonstrates a practical implementation of a privacy-focused smart camera system using locally-hosted AI. The prototype successfully streams video, processes images through Ollama vision models, and provides both visual and audio feedback to users.

Learning Outcomes:

ESP32 camera integration and REST API development
Flask web service architecture
Local LLM integration for vision tasks
Hardware-software co-design for embedded systems

Project Directory Structure:

/run/media/techtiger/Devs/MajorHackathonProjects/VisionX/
├── PythonImageTesting/
│   ├── app.py              # Flask backend API
│   ├── templates/
│   │   └── VX3.html        # Web interface
│   └── .env                # Configuration (not committed)
└── Abstract.md             # This document

FilesExpand file tree

Abstract.md

Latest commit

History