Initial version

Author: omersiar

.env.example

@@ -0,0 +1,42 @@
+# Kafka Configuration
+RIPT_KAFKA_BROKERS=localhost:9092
+
+# Kafka Authentication - only SSL and PLAINTEXT is tested, but other options should work as well
+# OAUTHBEARER, AWS MSK AUTH testers are welcome
+
+# RIPT_KAFKA_SECURITY_PROTOCOL=PLAINTEXT         # PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL
+# RIPT_KAFKA_SASL_MECHANISM=                     # PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER
+# RIPT_KAFKA_SASL_USERNAME=
+# RIPT_KAFKA_SASL_PASSWORD=
+# RIPT_KAFKA_SASL_OAUTHBEARER_TOKEN_ENDPOINT_URL=
+# RIPT_KAFKA_SASL_OAUTHBEARER_CLIENT_ID=
+# RIPT_KAFKA_SASL_OAUTHBEARER_CLIENT_SECRET=
+# RIPT_KAFKA_SASL_OAUTHBEARER_SCOPE=
+
+# TLS / mTLS
+# RIPT_KAFKA_TLS_CA_CERT_FILE=                   # Path to CA certificate (PEM)
+# RIPT_KAFKA_TLS_CLIENT_CERT_FILE=               # Path to client certificate (PEM) for mTLS
+# RIPT_KAFKA_TLS_CLIENT_KEY_FILE=                # Path to client private key (PEM) for mTLS
+# RIPT_KAFKA_TLS_INSECURE_SKIP_VERIFY=false
+
+# Scan Configuration
+RIPT_SCAN_INTERVAL_MINUTES=5
+
+# State / Tracker Configuration
+RIPT_STATE_TOPIC=ript-state
+RIPT_KAFKA_CONSUMER_GROUP_ID=ript-scan
+RIPT_STATE_TOPIC_PARTITIONS=6
+RIPT_STATE_TOPIC_REPLICATION_FACTOR=2
+RIPT_STATE_LOAD_TIMEOUT_SECONDS=30
+RIPT_INSTANCE_ID=
+
+# Staleness Thresholds - used for cosmetics and can be set to different values on frontend
+RIPT_STALE_PARTITION_DAYS=7
+RIPT_UNUSED_TOPIC_DAYS=30
+
+# HTTP Server Configuration
+RIPT_HTTP_PORT=8080
+RIPT_HTTP_HOST=0.0.0.0
+
+# Logging Configuration
+RIPT_LOG_LEVEL=info

.github/copilot-instructions.md

@@ -0,0 +1,75 @@
+# Copilot Instructions
+
+## Project Overview
+
+RIPT (Reclaimer of Inactive Partitions and Topics) is a stateless Go daemon that monitors Apache Kafka topics to identify unused ones. It persists state in its own internal Kafka topic (`ript-state`) and exposes a REST API and web dashboard.
+
+## Build, Test, and Run
+
+```bash
+make build        # Build binary to bin/ript
+make run          # Build and run locally (requires .env or env vars)
+make test         # go test -v ./...
+make clean        # Remove build artifacts
+make deps         # go mod download && go mod tidy
+
+# Docker
+make docker-up    # Start Zookeeper + Kafka + RIPT
+make docker-down  # Tear down stack
+make docker-logs  # Tail RIPT logs
+```
+
+**Single test:**
+```bash
+go test -v -run TestFunctionName ./internal/package_name
+```
+
+**Integration tests** (requires Docker):
+```bash
+./test-integration.sh
+```
+
+## Architecture
+
+```
+cmd/ript/           Entry point — wires up all components, handles OS signals, graceful shutdown
+internal/config/    Loads and validates env vars with defaults (via godotenv)
+internal/models/    Domain types: TopicStatus, PartitionInfo, ClusterSnapshot, Duration
+internal/kafka/     Kafka client (topic metadata + offset fetching) and StateManager (snapshot persistence)
+internal/tracker/   TopicTracker — core scan loop, staleness logic, in-memory snapshot
+internal/api/       Gin HTTP server — routes and handlers read from TopicTracker snapshot
+internal/logging/   Custom leveled logger (DEBUG/INFO/WARN/ERROR/FATAL)
+web/templates/      dashboard.html — Bootstrap 5 frontend served by the API server
+```
+
+**Data flow:** `main` → initialize config/logging → connect Kafka (with retry) → `StateManager.Load()` → start `TopicTracker` scan loop → start `api.Server` → serve traffic from in-memory snapshot.
+
+Each scan: fetch Kafka metadata + offsets → compute ages → update snapshot → persist to `ript-state` (log-compacted, one message per topic key).
+
+**Staleness thresholds (configurable):**
+- Partition "stale": no offset update in `RIPT_STALE_PARTITION_DAYS` days (default 7)
+- Topic "unused": all partitions stale for `RIPT_UNUSED_TOPIC_DAYS` days (default 30)
+
+## Key Conventions
+
+**Concurrency:** Snapshot access uses `sync.RWMutex` (read-heavy). Goroutines are coordinated with context cancellation and `sync.WaitGroup` for shutdown.
+
+**HTTP handlers (Gin):** All handlers are pointer receiver methods on `api.Server`. Return `503 ServiceUnavailable` if the tracker snapshot is not yet initialized. Use `gin.H` maps for all JSON responses.
+
+**Error handling:** Wrap errors with `fmt.Errorf("context: %w", err)`. Log warnings and degrade gracefully for non-fatal errors; use `logger.Fatal()` only for initialization failures.
+
+**State persistence:** `StateManager` writes one compacted Kafka message per topic (topic name as key). On restart, it reads back to EOF to restore the last snapshot. The app is stateless from infrastructure's perspective — Kafka is the durable store.
+
+**Configuration:** All config via environment variables with `RIPT_` prefix. `.env` file auto-loaded if present. See `.env.example` for all supported variables. Key vars: `RIPT_KAFKA_BROKERS`, `RIPT_SCAN_INTERVAL_MINUTES`, `RIPT_HTTP_PORT`, `RIPT_LOG_LEVEL`.
+
+**Models:** `Duration` is a custom type with `Days/Hours/Minutes/Seconds` fields and a `String()` method for human-readable output. Use `models.CalculateDuration(t time.Time)` to convert timestamps.
+
+## Local Development Setup
+
+```bash
+cp .env.example .env   # Configure Kafka brokers and other settings
+make docker-up         # Start Kafka stack (listens on localhost:9092)
+make run               # Run RIPT against local Kafka
+```
+
+The RIPT dashboard is available at `http://localhost:8080` once running.

.github/workflows/docker-publish.yml

@@ -0,0 +1,95 @@
+name: Docker
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+on:
+  push:
+    branches: [ "main" ]
+    # Publish semver tags as releases.
+    tags: [ 'v*.*.*' ]
+  pull_request:
+    branches: [ "main" ]
+
+env:
+  # Use docker.io for Docker Hub if empty
+  REGISTRY: ghcr.io
+  # github.repository as <account>/<repo>
+  IMAGE_NAME: ${{ github.repository }}
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+      # This is used to complete the identity challenge
+      # with sigstore/fulcio when running outside of PRs.
+      id-token: write
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Install cosign
+        if: github.event_name != 'pull_request'
+        uses: sigstore/cosign-installer@59acb6260d9c0ba8f4a2f9d9b48431a222b68e20
+        with:
+          cosign-release: "v2.2.4"
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@f95db51fddba0c2d1ec667646a06c2ce06100226
+
+      - name: Log into registry ${{ env.REGISTRY }}
+        if: github.event_name != 'pull_request'
+        uses: docker/login-action@343f7c4344506bcbf9b4de18042ae17996df046d
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Compute build metadata
+        id: vars
+        run: |
+          echo "build_time=$(date -u +%Y%m%dT%H%M%SZ)" >> "$GITHUB_OUTPUT"
+          echo "git_commit=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
+          if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+            echo "version=${GITHUB_REF_NAME}" >> "$GITHUB_OUTPUT"
+          else
+            echo "version=dev" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Extract Docker metadata
+        id: meta
+        uses: docker/metadata-action@96383f45573cb7f253c731d3b3ab81c87ef81934
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          tags: |
+            type=ref,event=branch
+            type=ref,event=tag
+            type=sha
+
+      - name: Build and push Docker image
+        id: build-and-push
+        uses: docker/build-push-action@0565240e2d4ab88bba5387d719585280857ece09
+        with:
+          context: .
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          build-args: |
+            VERSION=${{ steps.vars.outputs.version }}
+            GIT_COMMIT=${{ steps.vars.outputs.git_commit }}
+            BUILD_TIME=${{ steps.vars.outputs.build_time }}
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Sign the published Docker image
+        if: ${{ github.event_name != 'pull_request' }}
+        env:
+          TAGS: ${{ steps.meta.outputs.tags }}
+          DIGEST: ${{ steps.build-and-push.outputs.digest }}
+        run: echo "${TAGS}" | xargs -I {} cosign sign --yes {}@${DIGEST}

.github/workflows/release-artifacts.yml

@@ -0,0 +1,56 @@
+name: Release Artifacts
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: write
+
+jobs:
+  build-and-upload:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: "1.23"
+
+      - name: Build binaries
+        run: |
+          mkdir -p dist
+          VERSION="${{ github.event.release.tag_name }}"
+          GIT_COMMIT="${GITHUB_SHA::7}"
+          BUILD_TIME="$(date -u +%Y%m%dT%H%M%SZ)"
+          LDFLAGS="-s -w \
+            -X github.com/omersiar/ript/internal/version.Version=${VERSION} \
+            -X github.com/omersiar/ript/internal/version.GitCommit=${GIT_COMMIT} \
+            -X github.com/omersiar/ript/internal/version.BuildTime=${BUILD_TIME}"
+
+          GOOS=linux   GOARCH=amd64 go build -ldflags "$LDFLAGS" -o dist/ript-linux-amd64          ./cmd/ript
+          GOOS=linux   GOARCH=arm64 go build -ldflags "$LDFLAGS" -o dist/ript-linux-arm64          ./cmd/ript
+          GOOS=darwin  GOARCH=amd64 go build -ldflags "$LDFLAGS" -o dist/ript-darwin-amd64         ./cmd/ript
+          GOOS=darwin  GOARCH=arm64 go build -ldflags "$LDFLAGS" -o dist/ript-darwin-arm64         ./cmd/ript
+          GOOS=windows GOARCH=amd64 go build -ldflags "$LDFLAGS" -o dist/ript-windows-amd64.exe    ./cmd/ript
+
+      - name: Generate checksums
+        run: |
+          cd dist
+          sha256sum * > checksums.txt
+
+      - name: Upload assets to release
+        uses: softprops/action-gh-release@v2
+        with:
+          files: |
+            dist/ript-linux-amd64
+            dist/ript-linux-arm64
+            dist/ript-darwin-amd64
+            dist/ript-darwin-arm64
+            dist/ript-windows-amd64.exe
+            dist/checksums.txt
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -0,0 +1,46 @@
+# Binaries
+bin/
+*.exe
+*.exe~
+*.dll
+*.so
+*.so.*
+*.dylib
+
+# Test binaries
+*.test
+*.out
+
+# Build cache
+*.o
+*.a
+
+# Go modules
+vendor/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+.DS_Store
+
+# Environment
+.env
+.env.local
+.env.*.local
+
+# Docker
+.dockerignore
+
+# Logs
+*.log
+logs/
+
+# Temporary files
+tmp/
+temp/
+*.tmp
+
+kafkactl

ACLs.md

@@ -0,0 +1,46 @@
+### Kafka ACLs (secured clusters)
+
+If your Kafka cluster enforce ACLs, grant the RIPT principal permissions for:
+
+- Cluster metadata discovery
+- Reading monitored topics (offset and metadata inspection)
+- Read/write access to the internal state topic
+- Consumer group access for scan workload balancing (`RIPT_KAFKA_CONSUMER_GROUP_ID`)
+
+Example (`User:ript`, Kafka ACL authorizer):
+
+Adjust principal format as needed for your environment (for example, mTLS DNs or SASL usernames).
+
+```bash
+# 1) Cluster metadata
+kafka-acls.sh --bootstrap-server localhost:9092 \
+   --add --allow-principal User:ript \
+   --operation Describe --cluster
+
+# 2) Read monitored topics (use --topic '*' for all topics, or scope to prefixes)
+kafka-acls.sh --bootstrap-server localhost:9092 \
+   --add --allow-principal User:ript \
+   --operation Describe --operation Read \
+   --topic '*'
+
+# 3) Internal state topic access
+kafka-acls.sh --bootstrap-server localhost:9092 \
+   --add --allow-principal User:ript \
+   --operation Describe --operation Read --operation Write \
+   --topic ript-state
+
+# 4) Consumer group access used for balancing scans
+kafka-acls.sh --bootstrap-server localhost:9092 \
+   --add --allow-principal User:ript \
+   --operation Describe --operation Read \
+   --group ript-scan
+```
+
+Optional (only if RIPT should create `ript-state` itself):
+
+```bash
+kafka-acls.sh --bootstrap-server localhost:9092 \
+   --add --allow-principal User:ript \
+   --operation Create \
+   --topic ript-state
+```

CHANGELOG.md

@@ -0,0 +1,11 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## v0.0.1 - 2026-04-02
+### Added
+- Initial version