HubFly Architecture: A Deep Dive

HT
Hubfly Team
••Announcement

A deep technical breakdown of how HubFly works, the architecture behind it, and what makes it fast, reliable, and unique.

Introduction

HubFly was built with a singular objective: to deliver a modern, reliable, and developer centric hosting platform capable of handling substantial workloads with speed and precision. Achieving this required designing a cracked internal architecture powered by a suite of custom tools, open-source components, and tightly optimized workflows.

This post provides a transparent look at the engineering that powers HubFly. While some proprietary internals are omitted, the following is a comprehensive breakdown of our core systems.

High Level Architecture

HubFly operates as a distributed system of specialized services:

  • HubFly API: The central orchestrator for all platform operations.
  • Docker Engine: Executes project containers within isolated networks.
  • HubLock + HubFly-Storage: Powers our virtual, encrypted storage engine.
  • HubFly-Builder: Manages image construction using BuildKit.
  • HubFly Reverse Proxy: Handles routing, SSL termination, and stream forwarding.
  • Harbor Registry: Stores and scans for vulnerabilities in built images.
  • Grafana/Prometheus: Provides deep system monitoring and observability.
  • HubTunnel: Enables secure internal access to your containers without exposing public ports.

HubLock: The Virtual Storage Layer

HubLock serves as the foundational storage backbone for HubFly-storage.

What HubLock Provides

  • Virtual File System: Optimized for speed, reliability, or a balanced configuration depending on workload needs.
  • Encryption: Data is encrypted both at rest and in transit.
  • Flexibility: Attachable to any container and sharable across a project network.

HubLock integrates tightly into hubfly-storage, which exposes storage management capabilities to the wider platform.

HubFly-Storage: Dynamic Volume Manager

HubFly-Storage is an open-source component built in Go, designed to handle dynamic Docker volume creation and management.

Key Features

  • Dynamic Sizing: Create volumes with precise size constraints.
  • Maintenance: Automatically cleans up lost+found artifacts.
  • Observability: Provides detailed storage usage statistics.
  • Access: Generates temporary web access URLs via a modified FileBrowser instance.
  • Backend Management: Orchestrates NFS and HubLock-backed storage drivers.

This component acts as the persistence layer for all user projects/containers.

HubFly-Builder: Smart & Reliable Image Builder

HubFly-Builder ingests build jobs and processes them using BuildKit, ensuring fast and cache efficient builds.

capabilities

  • Concurrency: Handles multiple builds simultaneously with configurable resource limits.
  • Resilience: Features a SQLite based persistent job queue that is fully recoverable after crashes.
  • Automation: Performs runtime detection to automatically generate Dockerfiles when none are present.
  • Safety: Includes automatic workspace cleanup, log rotation, and a strict command allowlist.
  • Integration: Deeply integrated with Git for context aware builds.

This system ensures HubFly can reliably build and deploy hundreds of images without using any external services.

HubFly Reverse Proxy: Smart Routing & SSL Automation

Our routing layer is a lightweight reverse proxy tool built on NGINX and Certbot, wrapped with a custom Go REST API.

Features

  • SSL Automation: Zero config SSL certificate provisioning.
  • Analytics: Real time tracking of IPs, requests, bandwidth, and error rates.
  • Logging: JSON structured logging for downstream analytics.
  • Protocol Support:
    • HTTP(S) proxying
    • TCP/UDP stream proxying (essential for SSH, Databases, etc.)

This infrastructure allows HubFly to securely route traffic to any service exposed by your project.

HubTunnel: Secure Access Without Public Ports

HubTunnel enables access to internal containers without exposing them to the public internet. It functions by creating a temporary, isolated, and authenticated tunnel via the HubFly CLI.

For a detailed breakdown of this feature, read our HubTunnel deep dive.

Observability: Prometheus + Grafana

HubFly leverages Prometheus for metric collection and Grafana for visualization, providing a comprehensive view of platform health.

Roadmap Features:

  • Per user Grafana dashboards.
  • Granular container performance analytics.
  • Advanced health-based scaling recommendations.

Harbor: Secure Image Registry

Harbor serves as the authoritative store for all images built by HubFly.

Security Features:

  • Automated vulnerability scanning.
  • Image signing and trust verification.
  • Granular role based access control.

This ensures that every image is verified and secure before it reaches the deployment stage.

System Integration

Each project is assigned a dedicated isolated Private network:

  • Containers within the network can communicate freely.
  • The Proxy Manager controls all ingress traffic.
  • Supported protocols include TCP, UDP, and HTTP/S.

This segmentation guarantees security and prevents cross project interference, mimicking a virtual private cloud environment.

Conclusion

HubFly’s architecture represents hundreds of engineering decisions focused on reliability, performance, and developer experience. By combining custom built systems like HubLock, HubFly-Storage, and HubFly-Builder with industry proven tools like Docker, BuildKit, and NGINX, the platform achieves a balance of power and simplicity.

As HubFly evolves, we plan to open-source more components and introduce more developer friendly tools to help them achieve enterprise grade security,deployment and performance.

If you are using HubFly today, you are building on a foundation engineered to scale.

References

Related