Observability Platform for DigitalOcean

A complete, production-ready observability platform for monitoring DigitalOcean droplets with multi-tenant support. Built with OpenTelemetry, ClickHouse, Prometheus, and Grafana.

🏗️ Architecture Overview

✨ Features

• Multi-Tenant Support: Each monitored droplet belongs to a separate tenant with isolated telemetry
• Complete Observability Stack:
  • 📊 Metrics: Prometheus for metrics storage and querying
  • 📝 Logs: ClickHouse columnar storage with full-text search
  • 🔍 Traces: Distributed tracing with ClickHouse storage
• Persistent Storage: DigitalOcean volumes for data durability
• Production-Ready: VPC networking, firewall rules, and security best practices
• Auto-instrumentation: OpenTelemetry agents automatically collect host metrics
• Visualization: Pre-configured Grafana dashboards for all telemetry types
• Scalable: Designed for horizontal scaling

📦 Components

Component	Size	Purpose	Volume
Monitored Droplets (×5)	1GB, 1 vCPU	Applications being monitored	-
OTel Gateway	2GB, 2 vCPU	Central telemetry routing	-
ClickHouse	4GB, 2 vCPU	Traces & logs storage	50GB
Prometheus	2GB, 2 vCPU	Metrics storage	30GB
Grafana	2GB, 2 vCPU	Visualization & dashboards	-

Estimated Cost: ~$100-120/month in Sydney region

🚀 Quick Start

Prerequisites

1. DigitalOcean Account with API access
2. Terraform (>= 1.0) - Install Guide
3. doctl (optional but recommended) - Install Guide
4. SSH Key at ~/.ssh/id_rsa.pub (or specify custom path)

Installation Steps

1. Clone/Navigate to the repository:

   cd /Users/achandna/dev/hack-observability

2. Set up DigitalOcean authentication:

   # Option 1: Using doctl (recommended)
   doctl auth init
   
   # Option 2: Export token directly
   export DIGITALOCEAN_TOKEN="your_token_here"

3. Configure your deployment:

   cd terraform
   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars with your preferences
   nano terraform.tfvars

4. Deploy the platform:

   cd ..
   ./deploy.sh

5. Access Grafana (after 5-10 minutes for cloud-init to complete):

   # Get the Grafana URL from outputs
   cd terraform
   terraform output grafana_url

🔧 Configuration

Key Variables

Edit terraform/terraform.tfvars to customize:

region                   = "syd1"              # Sydney region
project_name             = "observability"      # Resource prefix
monitored_droplet_count  = 5                    # Number of monitored droplets
grafana_admin_password   = "YourPassword123!"   # Change this!

# Optional customizations
tenant_names = [
  "acme-corp",
  "widgets-inc",
  "tech-startup",
  "finance-co",
  "retail-shop"
]

# Droplet sizes
clickhouse_size          = "s-2vcpu-4gb"
prometheus_size          = "s-2vcpu-2gb"
grafana_size             = "s-2vcpu-2gb"
monitored_droplet_size   = "s-1vcpu-1gb"

# Storage volumes
clickhouse_volume_size   = 50  # GB
prometheus_volume_size   = 30  # GB

📊 Using the Platform

Accessing Services

After deployment:

1. Grafana Dashboard: http://<grafana-ip>:3000

   • Username: admin
   • Password: (from your terraform.tfvars)

2. Prometheus: http://<prometheus-ip>:9090

   • View metrics and targets
   • Query using PromQL

3. ClickHouse: <clickhouse-ip>:8123 (HTTP) or :9000 (Native)

   • Direct SQL queries for traces/logs

Pre-configured Dashboards

Grafana comes with several dashboards:

1. Host Metrics - Multi-Tenant: CPU, memory, disk, network by tenant
2. Traces Overview: Distributed tracing analysis
3. Logs Overview: Log aggregation and search

Querying Data

ClickHouse Traces:

-- Get traces for a specific tenant
SELECT * FROM otel.otel_traces 
WHERE TenantId = 'tenant-alpha' 
  AND Timestamp > now() - INTERVAL 1 HOUR
LIMIT 100;

-- Trace latency percentiles by service
SELECT 
  ServiceName,
  quantile(0.5)(Duration) as p50,
  quantile(0.95)(Duration) as p95,
  quantile(0.99)(Duration) as p99
FROM otel.otel_traces
WHERE TenantId = 'tenant-alpha'
GROUP BY ServiceName;

ClickHouse Logs:

-- Search logs by tenant
SELECT * FROM otel.otel_logs 
WHERE TenantId = 'tenant-beta' 
  AND SeverityText = 'ERROR'
  AND Timestamp > now() - INTERVAL 1 HOUR
ORDER BY Timestamp DESC;

Prometheus Metrics:

# CPU usage by tenant
100 - (avg by (tenant, instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage by tenant
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

🔒 Security Considerations

Current Setup

• ✅ VPC with private networking for internal communication
• ✅ Firewall rules restricting access
• ✅ SSH key authentication
• ✅ Grafana password protection

Production Hardening

Before using in production, consider:

1. Enable HTTPS: Add TLS certificates (Let's Encrypt)
2. Restrict IPs: Limit Grafana/Prometheus access to your IPs
3. Enable Authentication:
   • ClickHouse: Add user authentication
   • Prometheus: Enable basic auth
4. Monitoring: Set up alerts for security events
5. Backups: Schedule volume snapshots
6. Secrets Management: Use DigitalOcean Secrets or Vault
7. Update Strategy: Plan for security updates

🔍 Monitoring and Troubleshooting

Check Cloud-Init Progress

# SSH to any droplet
ssh root@<droplet-ip>

# Watch cloud-init logs
tail -f /var/log/cloud-init-output.log

# Check service status
systemctl status otel-agent        # On monitored droplets
systemctl status otel-collector     # On gateway
systemctl status clickhouse-server  # On ClickHouse
systemctl status prometheus         # On Prometheus
systemctl status grafana-server     # On Grafana

Common Issues

Services not starting:

# Check logs
journalctl -u otel-agent -f
journalctl -u clickhouse-server -f

Volume not mounted:

# Check mounts
df -h
lsblk

# Remount if needed
mount /mnt/clickhouse_data

Can't connect to Grafana:

• Wait 10 minutes for cloud-init to complete
• Check firewall rules: terraform state show digitalocean_firewall.grafana
• Verify service: systemctl status grafana-server

📈 Scaling

Horizontal Scaling

Add more monitored droplets:

# In terraform.tfvars
monitored_droplet_count = 10

# Add tenant names
tenant_names = ["tenant-1", "tenant-2", ..., "tenant-10"]

Scale storage backend:

• Resize volumes: doctl compute volume resize <volume-id> --size 100
• Add more ClickHouse nodes (requires manual ClickHouse cluster setup)
• Add Prometheus federation or remote storage (Thanos/Cortex)

Vertical Scaling

Change droplet sizes in terraform.tfvars:

clickhouse_size = "s-4vcpu-8gb"   # Upgrade to 8GB
prometheus_size = "s-4vcpu-8gb"   # More RAM = longer retention

Then:

terraform apply

🗑️ Teardown

To destroy all resources:

./destroy.sh

Warning: This permanently deletes all data!

🎯 Gaps and Future Improvements

Current Gaps

1. No High Availability: Single points of failure

   • Mitigation: Add ClickHouse replication, Prometheus HA pairs

2. Limited Retention: 30-day TTL

   • Mitigation: Increase TTL or add cold storage (S3/Spaces)

3. Basic Authentication: No SSO/SAML

   • Mitigation: Configure Grafana OAuth with GitHub/Google

4. No Alerting: Alerts configured but not routed

   • Mitigation: Add Alertmanager for Prometheus alerts

5. Manual Backups: No automated backup strategy

   • Mitigation: Schedule volume snapshots via API

6. No Load Balancing: Single gateway could bottleneck

   • Mitigation: Add multiple OTel gateways behind load balancer

Future Enhancements

• Add Tempo for native trace storage (alternative to ClickHouse)
• Implement Loki for log aggregation (alternative to ClickHouse)
• Add Alertmanager for alert routing
• Configure backup automation
• Add Kubernetes support for better orchestration
• Implement log sampling for cost optimization
• Add APM dashboards for application performance
• Create Terraform module for reusability
• Add CI/CD pipeline for infrastructure updates

📚 Architecture Deep Dive

Data Flow

1. Metrics:

   Monitored Droplet → OTel Agent (hostmetrics) → OTel Gateway → Prometheus → Grafana
   

2. Traces:

   Application → OTel SDK → OTel Agent → OTel Gateway → ClickHouse → Grafana
   

3. Logs:

   Application → OTel SDK → OTel Agent → OTel Gateway → ClickHouse → Grafana
   

Multi-Tenancy Implementation

Tenant isolation is achieved through:

• Resource attributes: tenant.id and service.namespace
• ClickHouse table partitioning by TenantId
• Prometheus labels: tenant="tenant-name"
• Grafana variable filtering by tenant

Storage Strategy

ClickHouse Tables:

• Partitioned by date for efficient TTL
• Compressed with ZSTD codec (~5-10x compression)
• Indexed on tenant, service, and trace IDs
• Materialized views for common queries

Prometheus:

• 30-day local retention
• Can enable remote write for long-term storage
• Supports federation for multi-cluster

🤝 Contributing

Ideas for improvement:

1. Add more pre-built dashboards
2. Create alerting rule templates
3. Add service discovery for dynamic droplets
4. Implement automatic scaling policies

📄 License

This project is provided as-is for educational and production use.

🆘 Support

For issues:

1. Check the troubleshooting section
2. Review Terraform state: terraform show
3. Check DigitalOcean console for resource status
4. Review cloud-init logs on droplets

---

Ready to deploy?

./deploy.sh

🎉 Happy monitoring!