Twilight Drive Phase 2 — Infrastructure Hardening

📤 内部参考

本 spec 引用了 P1 阶段的对外定价（¥198/月、WeChat Pay）作为上下文。对外内容由独立项目承接，不展示于 dev.fsagent.cc 的对外路径；本文呈现仅限技术决策（Token Gateway / 数据仓库 / Profile Manager）讨论。

Date: 2026-05-07 Status: Draft (pending user review) Source: /Users/syone/prd/MASTER_PLAN.md §3 (Initiatives A, B, C)

Background

Phase 1 ships a paid A-share research bot on WeChat (¥198/mo). After P1.0–P1.3, we have:

FastAPI on Vultr with DuckDB cache serving price/fundamentals/search
WeChat Pay → provisioning flow → per-user Hermes gateway → WeChat bot
Multi-source data (Tushare + pytdx3 + akshare + Yahoo) behind the same API
Web search via DashScope → Google fallback

Phase 2 addresses three infrastructure gaps that block scale and reliability:

Initiative A: Centralized data warehouse — eliminate duplicate API calls, add data quality checks, 4-layer data model
Initiative B: Profile replication + ownership — git-backed profiles, claim/push/pull/sync CLI, template system
Initiative C: Token gateway — SiliconFlow key pooling with New-API, SearXNG web search, guardrails

Goal

Ship three infrastructure foundations that let Twilight Drive scale from "a few alpha users" to "dozens of paying users with reliable, auditable data":

A: twilight-data hub — scheduled batch ingestion → data quality checks → 4-layer DuckDB warehouse → redistribution API
B: hermes-profile-manager — git-backed profile registry with ownership, templates, multi-host sync
C: New-API + SearXNG — LLM key pooling with auto-fallback + self-hosted web search (no per-engine API keys)

Non-Goals

Real-time data streaming (P3)
Multi-tenant Kubernetes orchestration (overkill for ≤ 100 users)
Stripe integration (Phase 2 payment scope, not infra)
Research report PDF ingestion (Phase 2 content scope, not infra)

Initiative A: Centralized Data Warehouse

A.1 — Problem

Currently, each Hermes profile and the FastAPI backend independently fetch data from upstream sources (Tushare, akshare, Alpha Vantage, etc.):

Duplicate API calls: Profile #1 and Profile #2 both query 600519 daily — wasting Tushare points
No data quality control: No cross-source verification, no anomaly detection
No historical persistence: Every query is fresh from upstream; no backfill
No unified access layer: Downstream consumers (Hermes profiles, external APIs) each wire their own adapters

A.2 — Architecture

┌─────────────────────────────────────────────────────────────┐
│                    twilight-data Hub                        │
│                    (Vultr + Mac Mini)                       │
│                                                             │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────────┐ │
│  │ Tushare  │  │  akshare │  │  pytdx3  │  │  Yahoo/AV  │ │
│  │  (A股)   │  │  (宏观)  │  │  (历史)  │  │  (全球)    │ │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └─────┬──────┘ │
│       └──────────────┴──────────────┴──────────────┘        │
│                          ▼                                  │
│              ┌───────────────────────┐                      │
│              │   Data Quality Layer  │                      │
│              │  - Cross-source check │                      │
│              │  - Anomaly detection  │                      │
│              │  - Schema validation  │                      │
│              │  - Freshness monitor  │                      │
│              └───────────┬───────────┘                      │
│                          ▼                                  │
│              ┌───────────────────────┐                      │
│              │    DuckDB Warehouse   │                      │
│              │                       │                      │
│              │  Raw ──► Normalized   │                      │
│              │   ──► Curated ──►     │                      │
│              │   Feature             │                      │
│              └───────────┬───────────┘                      │
│                          ▼                                  │
│              ┌───────────────────────┐                      │
│              │   Redistribution API  │                      │
│              │   FastAPI (unified)   │                      │
│              │   extends P1.0 API    │                      │
│              └───────────────────────┘                      │
└─────────────────────────────────────────────────────────────┘
                          │
                          ▼
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
    ┌──────────┐  ┌──────────────┐  ┌──────────┐
    │ Hermes #1│  │ Hermes #2..N │  │  External│
    │ (crypto) │  │  (stocks)    │  │  Users   │
    └──────────┘  └──────────────┘  └──────────┘

A.3 — 4-Layer Data Model

Layer	Purpose	Mutability	Retention
Raw	Exact upstream response, immutable	Append-only	Forever
Normalized	Cleaned, typed, deduplicated, unified schema	Rewrite on re-ingest	Forever
Curated	Business logic applied (splits adjusted, corporate actions)	Rewrite on logic change	Forever
Feature	Derived metrics (PE bands, momentum, factor scores)	Rewrite on recomputation	90 days (regenerate on demand)

A.4 — DuckDB Schemas

sql

-- Layer 0: Raw (append-only, JSON blobs)
CREATE TABLE raw_tushare_daily (
  id           BIGINT AUTO_INCREMENT PRIMARY KEY,
  code         VARCHAR,
  fetched_at   TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  response_json  JSON,              -- exact Tushare response
  source       VARCHAR DEFAULT 'tushare'
);

-- Layer 1: Normalized (unified schema)
CREATE TABLE normalized_daily (
  code         VARCHAR NOT NULL,
  trade_date   DATE NOT NULL,
  open         DOUBLE,
  high         DOUBLE,
  low          DOUBLE,
  close        DOUBLE,
  volume       BIGINT,
  amount       DOUBLE,
  source       VARCHAR,              -- 'tushare', 'pytdx3', 'yahoo'
  ingested_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (code, trade_date, source)
);

-- Layer 2: Curated (business logic)
CREATE TABLE curated_daily (
  code         VARCHAR NOT NULL,
  trade_date   DATE NOT NULL,
  open         DOUBLE,
  high         DOUBLE,
  low          DOUBLE,
  close        DOUBLE,              -- split-adjusted
  volume       BIGINT,
  amount       DOUBLE,
  adj_factor   DOUBLE,              -- adjustment factor for backtesting
  PRIMARY KEY (code, trade_date)
);

-- Layer 3: Feature (derived metrics)
CREATE TABLE feature_daily (
  code         VARCHAR NOT NULL,
  trade_date   DATE NOT NULL,
  ma_5         DOUBLE,
  ma_10        DOUBLE,
  ma_20        DOUBLE,
  ma_60        DOUBLE,
  pe_ttm       DOUBLE,
  pb           DOUBLE,
  momentum_5d  DOUBLE,
  momentum_20d DOUBLE,
  volatility_20d DOUBLE,
  PRIMARY KEY (code, trade_date)
);

-- Data quality log
CREATE TABLE quality_log (
  id           BIGINT AUTO_INCREMENT PRIMARY KEY,
  check_name   VARCHAR,              -- 'cross_source', 'anomaly', 'freshness'
  code         VARCHAR,
  trade_date   DATE,
  severity     VARCHAR,              -- 'info', 'warning', 'error'
  message      VARCHAR,
  created_at   TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Source freshness tracker
CREATE TABLE source_freshness (
  source       VARCHAR PRIMARY KEY,
  last_success TIMESTAMP,
  last_failure TIMESTAMP,
  error_message VARCHAR,
  row_count_today INT DEFAULT 0
);

A.5 — Data Quality Rules

Rule	Check	Action
Cross-source agreement	Tushare close vs pytdx3 close for same (code, date)	Log warning if diff > 0.01%
Freshness	Last successful fetch per source	Alert if > 24h stale
Schema validation	All rows match expected types, no nulls in key fields	Reject + quarantine bad rows
Anomaly detection	Price change > 20% single day	Flag for manual review
Completeness	Expected columns present	Reject incomplete rows

A.6 — Ingestion Scheduler

APScheduler in-process, daily batch:

python

# src/data/ingest/scheduler.py
from apscheduler.schedulers.background import BackgroundScheduler

scheduler = BackgroundScheduler()

# Daily: pull all stocks from Tushare (1 call → all 5000 stocks)
scheduler.add_job(
    ingest_tushare_daily,
    'cron',
    hour=18, minute=30,  # after market close
    id='tushare_daily',
    replace_existing=True,
)

# Historical backfill (run once, then disable)
# scheduler.add_job(
#     backfill_pytdx3,
#     'cron',
#     hour=2, minute=0,
#     id='pytdx3_backfill',
# )

# Akshare macro data (weekly)
scheduler.add_job(
    ingest_akshare_macro,
    'cron',
    hour=3, minute=0,
    day_of_week='mon',
    id='akshare_macro',
)

# Quality checks (after each ingest)
scheduler.add_job(
    run_quality_checks,
    'cron',
    hour=19, minute=0,
    id='quality_checks',
)

# Freshness check (every 6 hours)
scheduler.add_job(
    check_source_freshness,
    'interval',
    hours=6,
    id='freshness_check',
)

A.7 — Redistribution API

Extends the P1.0 FastAPI. New endpoints:

# Warehouse reads (authoritative, not cached upstream calls)
GET  /warehouse/daily?code=600519.SH&start=2026-01-01&end=2026-04-30
                                       → {rows: [...], source: "curated"}
GET  /warehouse/fundamentals?code=600519.SH&period=2025Q4
                                       → {code, period, claims: [...], source: "curated"}
GET  /warehouse/features?code=600519.SH&date=2026-04-30
                                       → {code, date, features: {...}, source: "feature"}

# Data quality
GET  /quality/status                     → {sources: {...}, last_checks: [...]}
GET  /quality/logs?severity=warning      → [{check_name, code, message, ...}]

# Admin (internal)
POST /admin/ingest/run?source=tushare    → trigger manual ingest
POST /admin/ingest/backfill              → trigger pytdx3 backfill

All warehouse reads go through the Curated layer by default. Callers can request ?layer=raw or ?layer=normalized for debugging.

A.8 — Mac Mini ↔ Vultr Split

Host	Role
Mac Mini (home, 48GB RAM)	Warehouse PRIMARY — writes + nightly batch (pytdx3 backfill, daily fills). Heavy batch ops, no user-facing latency requirement.
Vultr (cloud, 4GB RAM)	Warehouse REPLICA (rsync'd from Mac Mini nightly) + FastAPI serving + all Hermes gateways. Read-only DuckDB replica for low-latency serving.

Mac Mini (PRIMARY - writes)
  └── DuckDB warehouse (full)
         │
         └── nightly rsync ──► Vultr (REPLICA - reads)
                                   └── FastAPI reads from replica

A.9 — Files Created / Modified (Initiative A)

File	Action
`twilight-drive/src/data/ingest/scheduler.py`	NEW
`twilight-drive/src/data/ingest/tushare.py`	NEW
`twilight-drive/src/data/ingest/pytdx3.py`	NEW
`twilight-drive/src/data/ingest/akshare.py`	NEW
`twilight-drive/src/data/ingest/yahoo.py`	NEW
`twilight-drive/src/data/quality/checks.py`	NEW
`twilight-drive/src/data/quality/rules.py`	NEW
`twilight-drive/src/data/warehouse/schemas.py`	NEW
`twilight-drive/src/data/warehouse/loader.py`	NEW
`twilight-drive/src/service/routes/warehouse.py`	NEW
`twilight-drive/src/service/routes/quality.py`	NEW
`twilight-drive/scripts/backfill_pytdx3.py`	NEW
`twilight-drive/scripts/admin.py`	MODIFY (add warehouse admin commands)
`twilight-drive/tests/test_ingest_tushare.py`	NEW
`twilight-drive/tests/test_quality_checks.py`	NEW
`twilight-drive/tests/test_warehouse_routes.py`	NEW

A.10 — Success Criteria (Initiative A)

[ ] Daily Tushare ingest runs at 18:30 → 5,000 stocks in DuckDB within 5 min
[ ] GET /warehouse/daily?code=600519.SH returns curated data in < 10ms
[ ] Cross-source check: Tushare vs pytdx3 for 600519 close → diff < 0.01% → no warning
[ ] Anomaly: inject a row with close = 0 → quality check flags it as error
[ ] Freshness: disable Tushare ingest for 24h → freshness check reports stale
[ ] Mac Mini → Vultr rsync completes nightly, replica readable by FastAPI

Initiative B: Profile Replication & Ownership

B.1 — Problem

Hermes profiles live in ~/.hermes/profiles/<name>/. Current issues:

No version control: Profile configs change, no history of who changed what
No push/pull sync: Mac Mini ↔ VPS profiles diverge; manual copy required
No ownership model: Anyone with filesystem access can modify any profile
No template system: Provisioner clones from a hardcoded directory, no formal templates

B.2 — Architecture

┌──────────────────────────────────────────────────┐
│              Profile Registry                     │
│              (bare git repo)                      │
│              /var/lib/hermes-profiles/            │
│                                                   │
│  origin/                                          │
│  ├── profiles/                                    │
│  │   ├── template-stock-research-pro/             │
│  │   ├── template-stock-research-lite/            │
│  │   ├── template-crypto/                         │
│  │   ├── user-abc123/                             │
│  │   └── user-def456/                             │
│  ├── OWNERS                                       │
│  └── README.md                                    │
│                                                   │
│  Every change = git commit (versioned, auditable) │
└──────────────────────────────────────────────────┘
         │                      │
    git pull              git push
         │                      │
    ┌────┴────┐          ┌─────┴──────┐
    │ Mac Mini │          │   Vultr    │
    │ (local)  │          │  (local)   │
    └─────────┘          └────────────┘

B.3 — Git-Backed Storage

Bare repo setup:

bash

# On Vultr (central registry)
mkdir -p /var/lib/hermes-profiles
cd /var/lib/hermes-profiles
git init --bare

# On Mac Mini (working copy)
git clone vultr:/var/lib/hermes-profiles ~/hermes-profiles-registry
cd ~/hermes-profiles-registry

# Migrate existing profiles
cp -r ~/.hermes/profiles/stock-research-agent profiles/stock-research-agent
cp -r ~/.hermes/profiles/main profiles/main
git add -A
git commit -m "migrate existing profiles to git-backed storage"
git push origin main

B.4 — Ownership Model

OWNERS file at repo root:

# Profile ownership registry
# Format: profile_name | owner_id | verified_by | claimed_at
# ---
stock-research-agent | liang | telegram:5838121981 | 2026-04-30
main | liang | telegram:5838121981 | 2026-04-10
user-abc123 | user-abc123 | payment:pay_xxxxx | 2026-05-15

Claim flow:

hermes profile claim stock-research-agent
  → verifies identity (Telegram ID or payment proof)
  → writes to OWNERS file
  → git commit + push
  → "Profile stock-research-agent claimed by liang"

B.5 — CLI: `hermes-profile-manager`

hermes-profile-manager
  ├── claim    <profile-name>     Claim ownership of a profile
  ├── push                        Push local profile changes to registry
  ├── pull                        Pull latest profile configs from registry
  ├── sync                        Pull + resolve conflicts + push
  ├── list                        List all profiles + ownership status
  ├── show     <profile-name>     Show profile config diff vs last commit
  ├── template
  │   ├── list                    List available templates
  │   ├── create <name>           Create template from current profile
  │   └── apply  <name> <target>  Clone template into new profile
  └── ownership
      ├── list                    Show ownership registry
      └── transfer <from> <to>    Transfer ownership (requires auth from both)

B.6 — Template System

Template structure:

profiles/template-stock-research-pro/
├── config.yaml.template           # config with {{PLACEHOLDER}} variables
├── SOUL.md                        # identity (shared across all stock profiles)
├── AGENTS.md                      # research methodology (shared)
├── CLAUDE.md                      # project instructions (shared)
├── secrets.schema.json            # required secrets schema
│   {
│     "TWILIGHT_API_TOKEN": {"type": "string", "generated": true},
│     "TUSHARE_TOKEN": {"type": "string", "from_pool": "tushare"},
│     "WEIXIN_HOME_CHANNEL": {"type": "string", "generated": true},
│   }
└── crontab.json                   # default cron jobs

Provisioner clone flow:

python

async def clone_profile(template: str, profile_name: str, user_id: UUID) -> str:
    """Clone a template into a new user profile."""
    
    # 1. Copy template directory
    template_dir = REGISTRY / "profiles" / template
    target_dir = REGISTRY / "profiles" / profile_name
    shutil.copytree(template_dir, target_dir)
    
    # 2. Fill placeholders in config.yaml.template
    config = load_template(target_dir / "config.yaml.template")
    config["model"]["api_key"] = await get_key_from_pool("siliconflow")
    config["platforms"]["weixin"]["home_channel"] = generate_ilink_channel()
    save_config(target_dir / "config.yaml", config)
    
    # 3. Generate secrets
    secrets = load_secrets_schema(target_dir / "secrets.schema.json")
    for key, schema in secrets.items():
        if schema.get("generated"):
            plant_secret(target_dir, key, secrets.token_urlsafe(32))
        elif schema.get("from_pool"):
            plant_secret(target_dir, key, await get_key_from_pool(schema["from_pool"]))
    
    # 4. Git commit
    subprocess.run(["git", "add", profile_name])
    subprocess.run(["git", "commit", "-m", f"provision: {profile_name} from {template}"])
    subprocess.run(["git", "push"])
    
    # 5. Register ownership
    append_owner(profile_name, user_id)
    
    return profile_name

B.7 — Multi-Host Sync

┌─────────────────────────────────────────────────────────────┐
│                     Sync Flow                               │
│                                                             │
│  Mac Mini (working copy)          Vultr (bare repo)        │
│                                                             │
│  hermes-profile-manager push ──────────────► git receive   │
│                                                             │
│  hermes-profile-manager pull ◄────────────── git fetch     │
│                                                             │
│  hermes-profile-manager sync (pull → merge → push)         │
│                                                             │
│  On conflict:                                               │
│    - 3-way merge for YAML files (automerge safe)           │
│    - SOUL.md / AGENTS.md: ours (local) wins, theirs logged  │
│    - config.yaml: manual resolution required, abort push    │
└─────────────────────────────────────────────────────────────┘

B.8 — Files Created / Modified (Initiative B)

File	Action
`hermes-profile-manager/cli.py`	NEW
`hermes-profile-manager/claim.py`	NEW
`heriles-profile-manager/push_pull.py`	NEW
`hermes-profile-manager/ownership.py`	NEW
`hermes-profile-manager/templates.py`	NEW
`hermes-profile-manager/secrets.py`	NEW
`hermes-profile-manager/sync.py`	NEW
`hermes-profile-manager/pyproject.toml`	NEW
`profiles/template-stock-research-pro/`	NEW dir
`profiles/template-stock-research-pro/config.yaml.template`	NEW
`profiles/template-stock-research-pro/secrets.schema.json`	NEW
`profiles/template-stock-research-pro/crontab.json`	NEW
`profiles/template-crypto/`	NEW dir
`profiles/OWNERS`	NEW
`/var/lib/hermes-profiles/` (bare repo)	NEW
`twilight-drive/src/provisioner/hermes.py`	MODIFY (use profile-manager clone)

B.9 — Success Criteria (Initiative B)

[ ] hermes-profile-manager list shows all profiles + ownership
[ ] hermes-profile-manager claim stock-research-agent → ownership recorded in OWNERS
[ ] hermes-profile-manager push → git commit + push to bare repo
[ ] hermes-profile-manager pull → git fetch + merge, local profiles updated
[ ] Provisioner clones from template → new profile with filled placeholders → pushed to registry
[ ] Mac Mini ↔ Vultr sync resolves YAML merges automatically, flags config.yaml conflicts
[ ] Second webhook for same user → idempotent (no duplicate profile)

Initiative C: Token Gateway — New-API + SearXNG

C.1 — Problem

Both Hermes profiles currently share API keys, causing:

HTTP 429 rate limits (profiles competing for same SiliconFlow key)
No usage tracking or guardrails
No automatic fallback when a key is exhausted
API keys exposed in config files (not managed centrally)
Web search depends on DashScope (1000/day cap) + Google CSE ($5/1000 queries)

C.2 — LLM Gateway: New-API

Recommended: New-API — active fork of One-API, web UI, channel-based key pooling.

┌──────────────────────────────────────────────────────────┐
│                    New-API Gateway                       │
│                    (Docker on Vultr)                     │
│                                                          │
│  Channels:                                               │
│  ┌────────────────────────────────────────────────────┐  │
│  │ #1 SiliconFlow key-abc  [active]   45% used  ¥225  │  │
│  │ #2 SiliconFlow key-def  [active]   12% used  ¥60   │  │
│  │ #3 SiliconFlow key-ghi  [active]   78% used  ¥390  │  │
│  │ #4 SiliconFlow key-jkl  [disabled] 100% used ¥500  │  │
│  │ #5 DashScope key-mno    [active]   30% used  ¥150  │  │
│  └────────────────────────────────────────────────────┘  │
│                                                          │
│  Routing: Round-robin among healthy channels             │
│  Fallback: Auto-switch on 429/5xx to next healthy        │
│  Guardrails: RPM, TPM, daily budget per channel          │
│                                                          │
│  Output: OpenAI-compatible endpoint                      │
│  http://localhost:3000/v1                                │
└──────────────────────────────────────────────────────────┘
                          │
                          ▼
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
    ┌──────────┐  ┌──────────────┐  ┌──────────┐
    │ Hermes #1│  │ Hermes #2    │  │ Twilight │
    │ (crypto) │  │ (stocks)     │  │ Backend  │
    └──────────┘  └──────────────┘  └──────────┘

C.3 — Web Search: SearXNG

Recommended: SearXNG — privacy-respecting metasearch engine, 70+ engines, self-hosted.

┌──────────────────────────────────────────────────────────┐
│                   SearXNG (Web Search)                   │
│                    (Docker on Vultr)                     │
│                                                          │
│  Enabled engines:                                        │
│  ┌────────────────────────────────────────────────────┐  │
│  │ Chinese:  Baidu  │  Sogou  │  360 Search  │  Bing  │  │
│  │ Global:   Google │  DuckDuckGo  │  Brave  │  Yep   │  │
│  └────────────────────────────────────────────────────┘  │
│                                                          │
│  Features:                                               │
│  - No per-engine API keys needed                         │
│  - Rate limiting + Redis cache built-in                  │
│  - JSON API: /search?q=...&format=json&language=zh-CN   │
│  - Privacy: no query logging, no user tracking           │
│                                                          │
│  Output: http://localhost:8080/search                    │
└──────────────────────────────────────────────────────────┘
                          │
                          ▼
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
    ┌──────────┐  ┌──────────────┐  ┌──────────┐
    │ Hermes #1│  │ Hermes #2    │  │ Twilight │
    │ (crypto) │  │ (stocks)     │  │ Backend  │
    └──────────┘  └──────────────┘  └──────────┘

C.4 — Why SearXNG over DashScope/Google

Aspect	DashScope + Google CSE	SearXNG
API keys needed	2 (DashScope + Google)	0 (self-hosted)
Chinese search quality	Good (DashScope) + limited (Google)	Excellent (Baidu + Sogou + 360)
Cost	$5/1000 queries above free tier	Free (server cost only)
Rate limits	DashScope 1000/day, Google 100/day free	Self-managed, no hard cap
Privacy	Queries logged by providers	No logging, privacy-first
Maintenance	Depends on upstream availability	Docker compose, self-managed

C.5 — Docker Compose

yaml

# docker-compose.gateway.yml
version: "3.8"

services:
  new-api:
    image: calciumion/new-api:latest
    container_name: new-api
    restart: always
    ports:
      - "3000:3000"
    environment:
      - SQL_DSN=sqlite:///data/new-api.db
      - REDIS_CONN_STRING=redis://redis:6379
      - SYNC_FREQUENCY=60
    volumes:
      - /var/lib/twilight/new-api:/data
    depends_on:
      - redis

  searxng:
    image: searxng/searxng:latest
    container_name: searxng
    restart: always
    ports:
      - "8080:8080"
    environment:
      - SEARXNG_BASE_URL=http://localhost:8080/
      - SEARXNG_SECRET=$(openssl rand -hex 32)
      - UWSGI_WORKERS=4
      - UWSGI_THREADS=4
    volumes:
      - /var/lib/twilight/searxng:/etc/searxng:rw
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID

  redis:
    image: redis:7-alpine
    container_name: searxng-redis
    restart: always
    command: redis-server --save 30 5 --maxmemory 256mb --maxmemory-policy allkeys-lru
    volumes:
      - /var/lib/twilight/redis:/data

C.6 — Guardrails Configuration

Guardrail	Setting	Action
RPM limit (per channel)	100 requests/min	Throttle → queue → fallback
TPM limit (per channel)	500K tokens/min	Throttle → queue → fallback
Daily budget (per channel)	¥500/day	Auto-disable channel when exhausted
Health check	Every 5 min	Test with simple request, mark unhealthy on failure
Auto-recovery	Every 30 min	Re-enable disabled channels, test if recovered
SearXNG rate limit	10 requests/min per IP	Return 429, Twilight backend retries with backoff
SearXNG cache (Redis)	256 MB, LRU eviction	Repeated queries return cached results

C.7 — Hermes Profile Migration

Each Hermes profile's config.yaml updated:

yaml

# Before:
model:
  provider: siliconflow
  api_key: sk-xxxxx
  model: Qwen/Qwen3.5-27B

# After:
model:
  provider: openai_compatible
  base_url: http://localhost:3000/v1
  api_key: new-api-proxy-key    # single key for all profiles
  model: Qwen/Qwen3.5-27B       # New-API routes to SiliconFlow

SearXNG wired into Twilight backend:

python

# Before (P1.3):
async def search(query: str):
    return await dashscope_search(query)

# After (Phase 2 C):
async def search(query: str):
    return await searxng_search(
        url="http://localhost:8080/search",
        params={"q": query, "format": "json", "language": "zh-CN"}
    )

C.8 — Files Created / Modified (Initiative C)

File	Action
`twilight-drive/docker-compose.gateway.yml`	NEW
`twilight-drive/searxng/settings.yml`	NEW
`twilight-drive/searxng/limiter.toml`	NEW
`twilight-drive/scripts/deploy-gateway.sh`	NEW
`twilight-drive/scripts/configure-searxng.py`	NEW
`twilight-drive/scripts/configure-new-api.py`	NEW
`twilight-drive/src/service/routes/search.py`	MODIFY (use SearXNG)
`~/.hermes/profiles/stock-research-agent/config.yaml`	MODIFY (point to New-API)
`~/.hermes/config.yaml`	MODIFY (point to New-API)

C.9 — Success Criteria (Initiative C)

[ ] docker compose -f docker-compose.gateway.yml up -d → New-API + SearXNG running
[ ] New-API web UI at http://vultr-ip:3000 → 5 channels visible, health status green
[ ] curl http://localhost:3000/v1/chat/completions → response via SiliconFlow key #1
[ ] Kill key #1 → next request automatically uses key #2 (no error to caller)
[ ] Set key #1 daily budget to ¥0 → channel disabled → traffic routes to #2, #3
[ ] SearXNG at http://localhost:8080/search?q=600519+业绩&format=json&language=zh-CN → results from Baidu + Sogou
[ ] Hermes profiles point to New-API → no more 429 errors from SiliconFlow
[ ] Twilight /search uses SearXNG → no more DashScope quota consumption

Phase 2 — Combined Execution Plan

Week	Initiative	Tasks
W6	A	DuckDB schemas, Tushare ingest scheduler, daily batch
W7	A	pytdx3 backfill, akshare/Yahoo wiring, quality checks, warehouse API
W8	B	Bare git repo, migrate profiles, CLI (claim/push/pull/list), templates
W9	C	New-API Docker deploy, add channels, guardrails; SearXNG Docker deploy, configure engines
W9	C	Migrate Hermes profiles to New-API, Twilight backend to SearXNG
W10	All	Integration testing, load testing, docs updated, monitoring setup

Test Plan (Initiative A)

Test	What it verifies
`test_tushare_daily_ingest`	5000 stocks ingested into Raw + Normalized within 5 min
`test_normalized_schema`	All rows have required columns, correct types
`test_curated_adjustment`	Split-adjusted close differs from raw close by adj_factor
`test_feature_computation`	MA5/MA10/MA20/PE/momentum computed correctly
`test_cross_source_agreement`	Tushare vs pytdx3 close < 0.01% diff → no warning
`test_cross_source_disagreement`	Inject 1% diff → warning logged
`test_anomaly_detection`	Price change > 20% → error flagged
`test_freshness_check`	Source stale > 24h → freshness check reports stale
`test_warehouse_route_returns_curated`	`GET /warehouse/daily` → curated layer data
`test_warehouse_route_respects_layer_param`	`?layer=raw` → raw layer data

Test Plan (Initiative B)

Test	What it verifies
`test_claim_writes_owner`	`claim` command writes to OWNERS file + git commit
`test_claim_idempotent`	Second claim by same owner → no duplicate entry
`test_claim_rejected_if_owned`	Claim by different user → rejected with error
`test_push_commits_and_pushes`	`push` creates commit + pushes to bare repo
`test_pull_fetches_and_merges`	`pull` fetches + merges without conflicts
`test_sync_resolves_yaml_automerge`	`sync` auto-merges YAML files
`test_sync_flags_config_conflict`	`sync` on conflicting config.yaml → aborts, asks for resolution
`test_template_clone_fills_placeholders`	Clone from template → placeholders filled, secrets planted
`test_template_clone_idempotent`	Clone same template twice → different profiles, no collision

Test Plan (Initiative C)

Test	What it verifies
`test_new_api_channel_routing`	Request → routed to healthy channel, response returned
`test_new_api_fallback_on_429`	Channel returns 429 → next channel tried automatically
`test_new_api_budget_exhaustion`	Channel budget exhausted → disabled, traffic rerouted
`test_new_api_health_check`	Health check fails → channel marked unhealthy
`test_searxng_search_returns_results`	Search query → results from multiple engines
`test_searxng_chinese_quality`	Chinese query → results from Baidu + Sogou + 360
`test_searxng_cache_hit`	Repeat query → cached result, no upstream call
`test_searxng_rate_limit`	> 10 req/min from same IP → 429

Risks & Mitigations

Risk	Impact	Mitigation
New-API upstream breaking changes	High	Pin Docker image tag, test before upgrade
SearXNG engine changes (upstream API breaks)	Medium	Monitor SearXNG engine test results page, swap broken engines
DuckDB concurrent read/write locks	Medium	Mac Mini is sole writer; Vultr reads from rsync replica
Git merge conflicts on profiles	Low	YAML files auto-merge; config.yaml conflicts flagged for manual resolution
SiliconFlow key pool exhaustion	Medium	Add fallback channel (DashScope via New-API model routing)
SearXNG IP ban by search engines	Medium	Rotate user-agent, add delays, use proxy if needed

Out of Scope (Explicit)

Real-time data streaming (P3)
Multi-tenant Kubernetes orchestration
Per-user dashboards (P2 content scope)
Research report PDF ingestion (P2 content scope)
Stripe integration (P2 payment scope, separate spec)
Usage caps enforcement (soft logging only in Phase 2)

Twilight Drive Phase 2 — Infrastructure Hardening ​

Background ​

Goal ​

Non-Goals ​

Initiative A: Centralized Data Warehouse ​

A.1 — Problem ​

A.2 — Architecture ​

A.3 — 4-Layer Data Model ​

A.4 — DuckDB Schemas ​

A.5 — Data Quality Rules ​

A.6 — Ingestion Scheduler ​

A.7 — Redistribution API ​

A.8 — Mac Mini ↔ Vultr Split ​

A.9 — Files Created / Modified (Initiative A) ​

A.10 — Success Criteria (Initiative A) ​

Initiative B: Profile Replication & Ownership ​

B.1 — Problem ​

B.2 — Architecture ​

B.3 — Git-Backed Storage ​

B.4 — Ownership Model ​

B.5 — CLI: hermes-profile-manager ​

B.6 — Template System ​

B.7 — Multi-Host Sync ​

B.8 — Files Created / Modified (Initiative B) ​

B.9 — Success Criteria (Initiative B) ​

Initiative C: Token Gateway — New-API + SearXNG ​

C.1 — Problem ​

C.2 — LLM Gateway: New-API ​

C.3 — Web Search: SearXNG ​

C.4 — Why SearXNG over DashScope/Google ​

C.5 — Docker Compose ​

C.6 — Guardrails Configuration ​

C.7 — Hermes Profile Migration ​

C.8 — Files Created / Modified (Initiative C) ​

C.9 — Success Criteria (Initiative C) ​

Phase 2 — Combined Execution Plan ​

Test Plan (Initiative A) ​

Test Plan (Initiative B) ​

Test Plan (Initiative C) ​

Risks & Mitigations ​

Out of Scope (Explicit) ​

Twilight Drive Phase 2 — Infrastructure Hardening

Background

Goal

Non-Goals

Initiative A: Centralized Data Warehouse

A.1 — Problem

A.2 — Architecture

A.3 — 4-Layer Data Model

A.4 — DuckDB Schemas

A.5 — Data Quality Rules

A.6 — Ingestion Scheduler

A.7 — Redistribution API

A.8 — Mac Mini ↔ Vultr Split

A.9 — Files Created / Modified (Initiative A)

A.10 — Success Criteria (Initiative A)

Initiative B: Profile Replication & Ownership

B.1 — Problem

B.2 — Architecture

B.3 — Git-Backed Storage

B.4 — Ownership Model

B.5 — CLI: `hermes-profile-manager`

B.6 — Template System

B.7 — Multi-Host Sync

B.8 — Files Created / Modified (Initiative B)

B.9 — Success Criteria (Initiative B)

Initiative C: Token Gateway — New-API + SearXNG

C.1 — Problem

C.2 — LLM Gateway: New-API

C.3 — Web Search: SearXNG

C.4 — Why SearXNG over DashScope/Google

C.5 — Docker Compose

C.6 — Guardrails Configuration

C.7 — Hermes Profile Migration

C.8 — Files Created / Modified (Initiative C)

C.9 — Success Criteria (Initiative C)

Phase 2 — Combined Execution Plan

Test Plan (Initiative A)

Test Plan (Initiative B)

Test Plan (Initiative C)

Risks & Mitigations

Out of Scope (Explicit)