HanyanOS 部署手记

发表于 分类于

1. The Problem

HanyanOS runs seven distinct services across two physical locations: a Singapore Lightsail VPS (front-end) and a Brisbane N100 home server (back-end). Each service needs its own subdomain with HTTPS termination — but AWS Lightsail instance pricing limits us to a single public IPv4 address. Running seven separate ports on the public internet would be unmanageable, insecure, and unfriendly to clients behind corporate firewalls.

The constraint was absolute: one public port (443) must intelligently route all seven domains to their respective backends, supporting both TLS-terminated web services and a non-TLS VPN protocol on the same entry point.

2. Architecture Overview

The solution uses a two-layer Nginx architecture:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Internet :443
    │
    ▼
nginx stream (ssl_preread on)    ← Layer 1: SNI router
    │                               No TLS termination
    ├── blog/www/ai/api/mail.chenyun.org  → :8443 (web_backend)
    └── vpn.chenyun.org                   → :8444 (xray_backend)

:8443 — nginx http (ssl on)       ← Layer 2: TLS termination + reverse proxy
    ├── blog.chenyun.org   → proxy_pass :7444 → FRP → N100:8081
    ├── www.chenyun.org    → root /var/www/hexo (static files)
    ├── chenyun.org        → root /var/www/hexo
    ├── ai.chenyun.org     → proxy_pass :7445 → FRP → N100:8090
    ├── api.chenyun.org    → proxy_pass :7445 → FRP → N100:8090
    └── mail.chenyun.org   → proxy_pass :8091 → SSH → N100:8091

Layer 1 — SNI Routing: Nginx’s stream module with ssl_preread on port 443 inspects the TLS ClientHello SNI field without terminating the TLS connection. It then proxies the raw TCP stream to the appropriate upstream based on the domain name.

Layer 2 — TLS Termination: All web services land on port 8443 where a second Nginx http block terminates TLS, applies headers, and reverse-proxies to local ports (which connect through FRP or SSH tunnels to the N100 back-end).

3. Configuration Breakdown

3.1 Stream SNI Router (stream.conf)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
stream {
    map $ssl_preread_server_name $backend {
        blog.chenyun.org      web_backend;
        www.chenyun.org       web_backend;
        chenyun.org           web_backend;
        api.chenyun.org       web_backend;
        ai.chenyun.org        web_backend;
        mail.chenyun.org      web_backend;
        vpn.chenyun.org       xray_backend;
        default               xray_backend;
    }

    upstream web_backend {
        server 127.0.0.1:8443;
    }
    upstream xray_backend {
        server 127.0.0.1:8444;
    }

    server {
        listen 443;
        ssl_preread on;
        proxy_pass $backend;
    }
}

Key design decisions:

  • ssl_preread on reads only the SNI field, not the full handshake — minimal overhead.
  • The map block evaluates the SNI and selects the upstream; the default catch-all routes unknown domains to Xray for cloaking.
  • Six web domains share one upstream (web_backend → :8443); the VPN domain has a separate path.

3.2 HTTPS Virtual Hosts (blog-ssl.conf)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# WordPress via FRP tunnel
server {
    listen 8443 ssl;
    server_name blog.chenyun.org;

    ssl_certificate /etc/ssl/chenyun/fullchain.cer;
    ssl_certificate_key /etc/ssl/chenyun/chenyun.org.key;
    ssl_protocols TLSv1.2 TLSv1.3;

    location / {
        proxy_pass http://127.0.0.1:7444;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_buffering off;
    }
    client_max_body_size 64M;
}

# Static Hexo site (local VPS)
server {
    listen 8443 ssl;
    server_name www.chenyun.org chenyun.org;

    ssl_certificate /etc/ssl/chenyun/fullchain.cer;
    ssl_certificate_key /etc/ssl/chenyun/chenyun.org.key;

    root /var/www/hexo;
    index index.html;

    location / {
        try_files $uri $uri/ /index.html;
    }
}

Each virtual server on port 8443 uses the same wildcard SSL certificate (/etc/ssl/chenyun/fullchain.cer) issued by Let’s Encrypt for *.chenyun.org. The static sites (www, root domain) serve files directly from the VPS filesystem, while dynamic services proxy through FRP tunnels to the N100.

3.3 AI/API Gateway (ai.conf, api.conf)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
server {
    listen 8443 ssl;
    server_name ai.chenyun.org;

    ssl_certificate /etc/ssl/chenyun/fullchain.cer;
    ssl_certificate_key /etc/ssl/chenyun/chenyun.org.key;

    location / {
        proxy_pass http://127.0.0.1:7445;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Both ai.chenyun.org and api.chenyun.org proxy through port 7445 — an FRP tunnel to the N100’s AI/API service on port 8090. Separate server_name directives allow future routing differentiation.

4. Tunnel Integration

The FRP tunnel architecture connects the VPS front-end to the N100 back-end:

VPS Port FRP Tunnel N100 Port Service
7444 FRP 8081 WordPress
7445 FRP 8090 AI/API Services
8091 SSH -R 8091 SnappyMail

The FRP migration from SSH tunnels (completed 2026-05-12) significantly improved reliability — FRP provides automatic reconnection, health checks, and load balancing, whereas SSH tunnels would silently die under network instability.

5. Challenges & Lessons

Challenge 1: Stream SNI Breaks on Non-TLS Traffic

The ssl_preread module expects TLS on port 443. HTTP requests (port 80) are handled separately with a simple redirect. We considered enforcing HTTPS-only but kept port 80 for backward compatibility with legacy clients.

Challenge 2: SSL Certificate Management

A single wildcard certificate simplifies management but creates a single point of failure. If the Let’s Encrypt renewal fails, all domains go down. The fix was a cron-driven acme.sh renewal with Route53 DNS challenge, verified daily.

Challenge 3: Debugging SNI Routing

Troubleshooting the stream layer is tricky — Nginx doesn’t log SNI routing decisions unless explicitly configured. We added:

1
2
log_format stream_log '$remote_addr [$time_local] $protocol $ssl_preread_server_name $upstream_addr';
access_log /var/log/nginx/stream-access.log stream_log;

This was invaluable during the initial deployment when we discovered that some mail clients sent SNI with trailing dots (FQDN format), causing map mismatches.

Challenge 4: Xray REALITY and Non-TLS Protocols

Xray’s REALITY protocol uses TLS-like handshakes but isn’t standard HTTPS. The ssl_preread module correctly reads the SNI, but the proxied connection must preserve the raw bytes. Using the stream module (not http) for the Xray backend was critical — any attempt to terminate TLS would break the Xray protocol.

6. Security Considerations

  • UFW lockdown: Only ports 22, 80, 443, 25, 465, 587, 993, 8443, 7443 are open on the public interface.
  • TLS version restriction: TLSv1.2 TLSv1.3 only — no legacy protocol support.
  • Cipher restriction: HIGH:!aNULL:!MD5 — modern ciphers only, no anonymous or MD5-based suites.
  • Default catch-all: Unknown SNI domains route to Xray, providing cloaking and preventing information leakage about the domain list.
  • Rate limiting: Configured via conf.d/rate-limit.conf to prevent brute-force and DDoS.

7. Performance

Under typical load, the stream SNI router adds approximately 0.3–0.8ms of latency per connection (the time to read the 5-byte TLS record header and SNI extension). The VPS (Singapore Lightsail, 2 vCPU, 2GB RAM) handles all seven domains without measurable CPU impact. During a stress test with 500 concurrent connections across all domains, CPU stayed below 15%.

8. Summary

This two-layer Nginx architecture — stream SNI routing on port 443, then TLS termination on port 8443 — solves the one-public-IP problem elegantly. The design is extensible: adding a new subdomain requires one line in the stream map block and one server block on port 8443. Future plans include migrating mail.chenyun.org from SSH tunnel to FRP, adding HSTS preload headers, and evaluating HTTP/3 (QUIC) support via a separate UDP listener.

TL;DR: One IP, one port (443), seven domains, two layers of Nginx, three types of tunnels — and everything just works.