Crawly - Bitcoin Network Crawler

A comprehensive, high-performance Bitcoin P2P network crawler and analyzer designed to discover, monitor, and visualize Bitcoin nodes across multiple networks (IPv4, IPv6, Tor, I2P, CJDNS, Yggdrasil).

🚀 Features

• Advanced P2P Crawling: Implements AddrV2 (BIP 155) for modern node discovery.
• Multi-Network Support: Native support for Clearnet, Tor (v2/v3), I2P, CJDNS, and Yggdrasil.
• Real-time Analytics: Live monitoring of node availability, versions, and network health.
• Historical Data: Automated hourly snapshots for trend analysis and long-term statistics.
• Geo-Location: Integrated MaxMind GeoIP support for physical node mapping.
• Cross-Platform: Rust-based high-performance backend and React Native mobile application.

� Prerequisites

Before building the project, ensure you have the following software installed:

Core Dependencies

• Rust: Latest stable version (via rustup).
• Node.js & npm: Required for the mobile application (LTS version recommended).
• PostgreSQL: Database for storing node data and statistics.
• Java (JDK 17+): Required for building the Android app.

Network Proxies

To crawl privacy networks, you must have the respective proxies running:

• Tor: Required for .onion addresses.
  • Default configuration (SOCKS5 on port 9050).
• i2pd: Required for .i2p addresses.
  • IMPORTANT: This project is configured to use port 4446 for the I2P SOCKS proxy.
  • Note: The default i2pd port is usually 4447. You must configure your i2pd.conf or tunnel settings to listen on port 4446, or update the backend code.

GeoIP Data

For geolocation enrichment to work, the application expects the MaxMind databases to be present:

1. Create a directory named database in the same location as the executable (or project root).
2. Download GeoLite2-City.mmdb and GeoLite2-ASN.mmdb.
3. Place both files inside the database directory.

🛠️ Installation & Build

1. Backend (Rust)

# Clone the repository
git clone https://github.com/jochemin/crawly.git
cd crawly/backend

# Configure Environment
cp .env.example .env
# Edit .env with your database credentials and API keys

# Install SQLx CLI for migrations
cargo install sqlx-cli

# Run Database Migrations
sqlx migrate run

# Build and Run
cargo build --release
cargo run --release

2. Mobile App (React Native)

cd crawly/mobile

# Install Dependencies
npm install

# Start Metro Bundler
npm start

# Run on Android
npm run android

⚙️ Configuration

The backend is configured via the .env file. Key variables include:

• DATABASE_URL: PostgreSQL connection string.
• GEOIP_DB_PATH: Path to MaxMind GeoIP databases (optional).
• I2P_PROXY_ADDRESS: (Optional) If you modified the code to support env vars, otherwise defaults to 127.0.0.1:4446.

📊 Architecture

The project consists of two main components:

1. Crawler Backend (Rust): Handles P2P handshakes, database operations, and exposes a REST API.
2. Mobile Client (React Native): Consumes the API to provide a user-friendly dashboard.

🤝 Contributing

Contributions are what make the open-source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

🛠️ Our CI/CD Pipeline

This project uses a fully automated pipeline powered by GitHub Actions to ensure stability across environments:

• Test Environment: Pushes to the test branch trigger an automated deployment to a Raspberry Pi (Backend) and generate a fresh Test APK (Mobile).
• Production Environment: Pushes to main via Version Tags (v*) trigger the generation of signed Production App Bundles (AAB) ready for Google Play.

🚀 How to Contribute

1. Fork the Project.
2. Create your Feature Branch (git checkout -b feature/AmazingFeature).
3. Backend (Rust): Ensure you have a local .env file. Do not commit this file; the CI/CD handles secrets securely on the server.
4. Mobile (Expo): Always use npm install to keep the package-lock.json in sync.
5. Commit your changes (git commit -m 'Add some AmazingFeature').
6. Push to the Branch (git push origin feature/AmazingFeature).
7. Open a Pull Request against the test branch for verification.

⚠️ Important Technical Notes

• Dynamic Configuration: Our CI/CD script automatically patches the BASE_URL in mobile/src/services/api.js during the build process to point to the correct environment. You don't need to change this manually for deployment.
• ARM64 Compatibility: The backend is compiled natively for ARM64 (Raspberry Pi). If you add dependencies, ensure they are compatible with Linux ARM64 architectures.

📄 License

Distributed under the MIT License. See LICENSE for more information.

Attribution

Any use of this software must include a clear acknowledgement to "Crawly" and display the official image of this software.