# Data Sync

Cara kerja sinkronisasi data dari provider eksternal ke database.

## Arsitektur Sync

```
Provider (Yahoo/CoinGecko/Indodax)
        │
        ▼
  DataRouter (router.py)
        │
        ├── YahooProvider     → stocks + stock_us
        ├── CoinGeckoProvider → crypto profiles
        └── IndodaxProvider   → crypto OHLCV
        │
        ▼
  Database (MySQL)
        │
        ├── coins              → Master data
        ├── coin_profiles      → Profile + metadata
        ├── ohlcv_data         → Candlestick data
        ├── stock_extended_data → Yahoo extended (financials, estimates, dll)
        └── market_cache       → Cache market-level data
```

## Provider & Data Type

### CoinGecko (Crypto)
- **Profile:** Nama, market cap, volume, deskripsi, icon
- **OHLCV:** Harga historis per timeframe (via Indodax)

### Yahoo Finance (Stocks)
- **Profile:** Company info, officers, sector, industry
- **OHLCV:** Harga historis (1m, 15m, 30m, 1h, 4h, 1D, 1W)
- **Extended Data:** (disimpan di `stock_extended_data`)

| Data Type | Deskripsi |
|-----------|-----------|
| `financials` | Income statement (annual) |
| `quarterly_financials` | Income statement (quarterly) |
| `balance_sheet` | Neraca (annual) |
| `quarterly_balance_sheet` | Neraca (quarterly) |
| `cash_flow` | Arus kas (annual) |
| `quarterly_cash_flow` | Arus kas (quarterly) |
| `ttm_cash_flow` | Trailing twelve months cash flow |
| `recommendations` | Analyst recommendations |
| `recommendations_summary` | Summary recommendations |
| `analyst_price_targets` | Target harga analyst |
| `earnings_dates` | Tanggal earnings |
| `dividends` | Riwayat dividen |
| `insider_purchases` | Insider transactions |
| `institutional_holders` | Institusi pemegang saham |
| `major_holders` | Pemegang saham utama |
| `growth_estimates` | Estimasi pertumbuhan |
| `revenue_estimate` | Estimasi revenue |
| `earnings_estimate` | Estimasi earnings |
| `eps_trend` | Trend EPS per periode |
| `eps_revisions` | Revisi EPS |
| `calendar` | Kalender earnings/dividen |
| `sec_filings` | SEC filings (US only) |
| `esg_scores` | ESG sustainability scores |

### Market-Level Data (via `MarketCache`)
- **Market Status:** Buka/tutup bursa US (`us_market`)
- **Market Calendars:** Earnings, IPO, stock splits, economic events

## Alur Sync CLI

```bash
python3 scripts/sync_data.py --mode stock_us --verbose
```

1. **Load coins** dari database berdasarkan `asset_type`
2. **Filter** berdasarkan `--scope`, `--coin`, `--limit`
3. **Per coin:**
   a. Fetch profile dari provider
   b. Download icon (jika belum ada)
   c. Fetch OHLCV per timeframe
   d. Fetch extended data (Yahoo only)
   e. Upsert ke database
4. **Summary:** jumlah success/fail, durasi

## Alur Sync via GUI

Admin bisa trigger sync dari `/admin/sync`:
1. Pilih mode (crypto/stock/stock_us)
2. Pilih scope (all/never synced/stale)
3. Klik "Start Sync"
4. Progress ditampilkan via SSE (Server-Sent Events)

## Remote Sync (Lokal ↔ Production)

```
Local DB ←──push/pull──→ Production DB
              via REST API
```

### Push (Lokal → Production)
Data yang di-push: OHLCV, scores, signals, profiles, ML predictions.

### Pull (Production → Lokal)
Data yang di-pull: Users, watchlists, portfolio, trade history.

### Cara Kerja
1. Cek timestamp terakhir sync per tabel
2. Query records yang berubah sejak timestamp
3. Kirim/terima via REST API (`/api/v1/sync/*`)
4. Upsert ke database tujuan
5. Update timestamp sync

## Caching

| Data | TTL | Storage |
|------|-----|---------|
| Market Status | 30 menit | `market_cache` table |
| Market Calendars | 1 jam | `market_cache` table |
| Extended Data | Per fetch | `stock_extended_data` table |
| OHLCV | Per fetch | `ohlcv_data` table |