# Watchtower — Market Watcher Service

**Spec v0.2 — 2026-09-28 — Vesper 🌟**

A standalone service that watches prices (stocks, futures, metals, crypto) against declarative alert rules, and notifies Ben via Telegram when — and only when — something actionable happens. Built to replace ad-hoc prompt-based cron watchers with a deterministic, auditable engine.

---

## 1. Problem

Today, every "watch this level" request becomes a bespoke OpenClaw cron job with the rules written in prose inside an LLM prompt. The gold re-entry watcher exposed the weaknesses in production:

| Gap | What happened |
|---|---|
| **Cadence** | Daily 09:00 SGT check; gold plunged to the 4,170 trigger at ~14:30 SGT — 19h before the next check |
| **No state/latching** | Once a condition stays true, the job either spams daily or must be told to stay quiet — and then can't re-arm for deeper levels |
| **Prose rules** | Levels buried in a prompt paragraph; hard to audit, edit, or diff |
| **Not scalable** | Every new symbol (LULU, NIO, XAU…) = another hand-written cron job |
| **Nondeterministic** | An LLM re-interprets the rules every run; costs tokens, can misread |
| **No history** | No log of what was checked, when, at what price |

**Watchtower inverts this:** deterministic Python evaluates the rules; the LLM (me) only writes the human-facing alert copy *after* a trigger fires — or a plain templated message goes out with zero LLM involvement.

## 2. Goals

1. **One config file per watch** — declarative YAML; add/remove watches without touching code
2. **Intraday cadence** — poll every 1–5 min for 24h assets, market-hours-aware for equities
3. **Stateful alert lifecycle** — fire once, latch, re-arm on rules; no spam, no missed re-tests
4. **Multi-source** — Yahoo Finance (stocks/futures/FX), gold-api (spot metals), ccxt/Binance (crypto)
5. **Reliable delivery** — Telegram alerts with price, trigger, and pre-agreed action plan
6. **Auditable** — every evaluation and alert logged; `watchtower status` shows all armed watches
7. **Cheap** — near-zero token cost; LLM optional and post-trigger only

### Non-goals (v1)

- No order execution — alerts only (Vigil owns execution)
- No strategy backtesting — this is a watcher, not a research tool
- No news/fundamental triggers (v2 candidate via Ray)

## 3. Watch Definition (YAML)

One file per watch in `watchtower/watches/*.yaml`:

```yaml
# watches/gold-reentry.yaml
id: gold-reentry
symbol: XAU            # logical symbol
sources:
  spot:    { provider: gold_api, symbol: XAU }
  futures: { provider: yahoo, symbol: "GC=F" }
primary: spot           # series used unless a rule says otherwise
schedule:
  interval: 5m          # 24h asset → poll around the clock
  quiet_hours: "01:00-07:00 Asia/Singapore"   # optional: batch non-urgent alerts
context: |
  Daily H&S top, head 4755 (Aug 25), neckline 4330-4360 broken Sep 14-16.
  Plan: tranche adds — small on reclaim 4445-4460, bigger at 4150 and 4000-4050.
rules:
  - id: zone_entry
    when: "price <= 4170"
    priority: high
    once: true                    # latch after first fire
    message: "Gold entered the 4000-4170 accumulation zone. Prepare tranche adds."
  - id: tranche_4150
    when: "price <= 4150"
    priority: high
    once: true
    message: "Tranche-2 add level hit."
  - id: breakdown_close
    when: "daily_close(futures) < 4325"     # ~spot 4290
    priority: medium
    once: true
    message: "H&S breakdown confirmed on daily close. Expect continuation toward 4000."
  - id: reclaim_4460
    when: "price >= 4460"
    priority: high
    once: true
    rearm: "price < 4400"         # re-arms if it dips back under, catches second reclaim
    message: "Pattern failing — reclaim entry plan active."
notify:
  channel: telegram
  to: "131270671"
  compose: llm          # llm = Vesper writes the alert; template = raw message field
expires: 2026-12-31     # auto-disable + notify when stale
```

### Condition language (v1 operators)

| Expression | Meaning |
|---|---|
| `price <= X`, `price >= X` | spot/last vs level |
| `daily_close(series) < X` | last **completed** daily close |
| `cross_above(sma(50))`, `cross_below(X)` | crossing events (needs prev sample) |
| `pct_change(1d) < -5` | move filters (gap/crash detection) |
| `rsi(14) < 30`, `rsi(14) > 70` | indicator triggers |
| `volume_ratio(20d) > 3` | volume spike vs average |
| `high(5d) > X and price < Y` | boolean combinators `and`/`or`/`not` |

Implemented as a small safe expression evaluator (whitelist of functions over pandas series — **no** `eval` on raw input).

## 4. Alert Lifecycle

```
ARMED ──condition true──▶ FIRED ──once:true──▶ LATCHED
  ▲                                               │
  └────────────── rearm condition true ◀──────────┘
```

- **ARMED**: evaluated every poll
- **FIRED**: alert sent exactly once; timestamp + price recorded
- **LATCHED**: silent; still evaluated so `rearm` can restore ARMED
- **Cooldown**: global per-watch min gap (default 30 min) so multiple rules firing together are batched into one message
- **Expiry**: watches with `expires` in the past are disabled with a final "watch expired, still relevant?" ping

## 5. Dashboard (visual monitoring + self-help management)

Web UI served on the existing nginx stack (`watch.invesper.ai`), mobile-first — alerts arrive on Telegram, so the dashboard must read well on a phone screen one tap later.

### 5.1 Views

**Watchlist (home)**
- Grid/cards: one card per watch — symbol, last price, 24h/1d % change, mini sparkline (last ~50 samples)
- Per-rule state chips: 🟢 ARMED / 🔴 FIRED / ⚪ LATCHED / ⏸ PAUSED
- **Distance-to-trigger**: progress bar + % for each armed rule (e.g. `tranche_4150 — 0.3% away`), sorted so the closest trigger across all watches floats to the top
- Staleness indicator: last poll time per watch; red if data older than 2× interval

**Watch detail**
- Candlestick chart (TradingView `lightweight-charts`, OSS) with **rule levels drawn as horizontal lines**, colored by state; fired rules show a marker at the fire candle
- Timeframe toggle: 1h / 4h / 1D
- Rule table: condition, state, fired-at price/time, re-arm condition, cooldown remaining
- Fire history for this watch (price, time, message sent)
- The `context` block from the YAML rendered as a note (the thesis, so future-me remembers why)

**History feed**
- Chronological list of every alert fired across all watches, with prices and delivery status — the audit trail for weekly reviews

**System health**
- Provider status (last success, error counts, backoff state), poll-loop latency, state-db size, uptime

### 5.2 Self-help management (no SSH needed)

| Action | UI | Effect |
|---|---|---|
| Pause / resume watch | toggle on card | sets `paused` flag, survives restart |
| Mute until… | time picker | suppress alerts, keep evaluating + logging |
| Re-arm a latched rule | button on rule row | back to ARMED (e.g. after taking the tranche, re-watch same level) |
| Edit levels | inline form on rule | validates → rewrites YAML → hot reload; change logged with old→new values |
| New watch from template | "Add watch" wizard | symbol + template (level watch / breakout / RSI) → generates YAML |
| Test fire | button per rule | dry-run against live data, shows would-fire result — nothing sent |
| Delete watch | with confirm | archives YAML to `watches/archive/`, never hard-deletes |

Every mutation goes through the same validated path as the CLI (single source of truth in the engine — the UI is a client, not a second brain).

### 5.3 UI requirements (non-functional)

- **Mobile-first**, dark theme default (matches the md-viewer aesthetic already in use)
- **Live updates**: SSE push on each poll cycle — no manual refresh; sub-second render budget on a 20-watch list
- **Auth**: HTTP basic auth (or a signed token link) at nginx level; no public read access
- **Deep links**: every Telegram alert includes `https://watch.invesper.ai/w/<id>` straight to the watch detail
- **Zero build ceremony**: server-rendered (FastAPI + Jinja + htmx) + `lightweight-charts` from CDN — no SPA toolchain to maintain
- **Graceful degradation**: if the engine is down, nginx serves a static "engine offline since <t>" page (same pattern as vigil-fallback)

## 6. Architecture

```
watchtower/
├── watches/*.yaml          # declarative watch definitions
├── state/state.sqlite      # rule states, fire history, last prices
├── logs/watchtower.jsonl   # every evaluation cycle (compact)
└── src/
    ├── main.py             # scheduler loop (asyncio)
    ├── providers/          # yahoo.py, gold_api.py, binance_ccxt.py
    ├── engine.py           # condition parser + evaluator
    ├── lifecycle.py        # ARMED/FIRED/LATCHED state machine
    ├── notify.py           # Telegram via OpenClaw gateway hook or bot API
    ├── web/                # FastAPI app: dashboard views, SSE, management API
    └── cli.py              # watchtower add|list|status|pause|test|history
```

- **Runtime**: Python 3.10+, single asyncio process, systemd service (`watchtower.service`) — same ops pattern as vigil-scanner
- **Market hours**: `exchange_calendars` for equities; 24h for metals/crypto/futures (with maintenance-break awareness)
- **Data**: poll → normalize to OHLCV frame per source → cache last N bars in memory, persist last sample to state
- **Failure handling**: provider retry w/ backoff; if a source is down >30 min, one ops alert (not per-cycle spam); stale-data guard (never evaluate on data older than 2× interval)
- **Notify paths**:
  - `compose: template` → direct Telegram send (deterministic, instant)
  - `compose: llm` → wake Vesper via OpenClaw hook with trigger context; I write the alert in-voice with current structure/plan
- **Vesper integration**: I write the YAML from our conversations ("watch LULU 104 breakout" → file + `watchtower reload`), and read `status`/`history` during reviews

### CLI sketch

```
watchtower list                     # all watches + rule states
watchtower status gold-reentry     # detail: prices, armed/latched, history
watchtower test gold-reentry       # dry-run rules against live data now
watchtower pause gold-reentry --until 2026-10-05
watchtower history --days 7        # what fired, when, at what price
```

## 7. Milestones

| | Deliverable | Est. |
|---|---|---|
| **M1** | Engine + yahoo/gold_api providers, fixed-interval polling, template alerts, state machine, `list/status/test` CLI. Gold + LULU watches live | 1–2 days |
| **M2** | Market-hours calendars, indicator conditions (sma/rsi/volume), cooldown batching, jsonl audit log, systemd hardening | 1–2 days |
| **M3** | LLM compose path (OpenClaw wake), `rearm` logic, expiry pings, ccxt/Binance provider | 1 day |
| **M4** | Dashboard §5: watchlist + detail views, SSE live updates, pause/re-arm/edit/test actions, nginx + auth at watch.invesper.ai | 2–3 days |
| **v2** | News triggers via Ray, multi-timeframe conditions, alert analytics (which rule types earn) | later |

## 8. Risks & mitigations

- **Yahoo rate limits / API drift** → 5m interval is gentle (~300 req/day/symbol); provider abstraction isolates breakage; add stooq/polygon fallback if needed
- **Duplicate alerting during restart** → state in sqlite, not memory; idempotent fire-check on (watch, rule, fired_at)
- **Condition bugs = missed money** → `watchtower test` dry-run required before arming; unit tests on the evaluator; the old cron stays enabled in parallel for 1 week as shadow
- **Scope creep toward Vigil** → hard non-goal: no execution, ever, in Watchtower
- **Dashboard mutations corrupting watches** → UI writes go through the same validator as CLI; YAML changes are diffed, logged, and archived — one-click revert

## 9. First two watches at launch

1. **gold-reentry** — port of the current cron (zone 4170, tranches 4150/4050/4000, breakdown close, reclaim 4460/4560)
2. **lulu-base** — `daily_close >= 104` breakout (target 110/115) or `price <= 97.5` range-low retest, per 2026-09-28 analysis

---

*Questions for Ben:* interval preference (1m vs 5m)? Quiet hours at night, or wake for high-priority only? Green-light M1?