IPCrawler

Workflow-driven reconnaissance framework for penetration testing and CTF/HTB challenges. Execute multi-stage scans with automatic data passing between tools - all configured via simple YAML files.

---

Quick Start

# Build and install
git clone https://github.com/neur0map/ipcrawler.git
cd ipcrawler
make build

# Run a scan
ipcrawler scan -t 10.10.10.1

# Full HTB/CTF methodology
ipcrawler scan -t 10.10.10.1 -w htb_full

# Run a single tool
ipcrawler scan -t 10.10.10.1 --tool nmap_fast

# List available workflows
ipcrawler list

# Check tool availability
ipcrawler tools

Core Features

Workflow-Driven Scanning

• Multi-stage pipelines defined in YAML
• Automatic data passing between stages (ports -> URLs -> vulnerabilities)
• Conditional stage execution (has_http_ports, has_port(445))
• Parallel tool execution within stages

Automatic /etc/hosts Management

• Discovered domains (like box.htb) are automatically added to /etc/hosts
• Essential for CTF/HTB where domains must resolve for web tools to work
• Manual management: ipcrawler hosts --add 10.10.10.1 box.htb
• Cleanup after scan: ipcrawler hosts --cleanup

YAML-Driven Tools

• Add new tools by creating a YAML file - no code changes needed
• Automatic target format conversion (host, host:port, URL)
• Dynamic privilege handling (elevated vs normal commands)
• Built-in installer definitions for multiple package managers

Output

• Structured JSON findings
• Markdown reports grouped by severity
• Shared data files (ports.txt, urls.txt, services.json)
• Complete raw logs preserved

Installation

Requirements:

• Rust 1.70+ (rustup.rs)
• Security tools: nmap, httpx, dig, whois (install with make install-deps)

git clone https://github.com/neur0map/ipcrawler.git
cd ipcrawler
make install-deps  # Install security tools
make build         # Build and create symlink
make doctor        # Verify installation

Available Workflows

Workflow	Description
quick_scan	Fast initial reconnaissance (default)
htb_full	Full HTB/CTF methodology with vuln scanning
web_recon	Web-focused enumeration
network_recon	Network and DNS reconnaissance
smb_enum	SMB/Windows enumeration

Included Tools

Category	Tools
Port Scanning	nmap_fast, nmap_full, rustscan
Web Analysis	httpx_enumeration, whatweb, nikto
Directory Fuzzing	ffuf, gobuster
Vulnerability Scanning	nuclei
DNS/Network	dig, whois, traceroute
SMB Enumeration	enum4linux, smbclient
Exploit Search	searchsploit

Adding Custom Tools

Create tools/mytool.yaml:

name: "mytool"
description: "My custom tool"
command: "mytool {{target}}"
input:
  target_format: url  # or: host, host_port, https_url
output:
  type: "json"  # or: xml, regex, raw

Adding Custom Workflows

Create workflows/myworkflow.yaml:

name: myworkflow
description: "My custom workflow"
version: "1.0"
stages:
  - name: discovery
    tools: [nmap_fast]
  - name: web_enum
    tools: [httpx_enumeration]
    depends_on: [discovery]
    condition: "has_http_ports"

Legal Notice

For authorized security testing only:

• Penetration testing with written permission
• CTF competitions and security research
• Defensive security operations

Do not use for unauthorized scanning.

License

Apache License 2.0 - See LICENSE file.

---

IPCrawler is part of the prowl.sh security tools ecosystem

Maintained by neur0map