{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "oow6ugb647",
   "source": "# Track Management & Data Persistence\n\nThis notebook provides a hands-on tutorial for managing track lifecycles and persisting tracking data using PyTCL's SQL and HDF5 storage backends.\n\n## What You'll Learn\n\n1. **Track Lifecycle Theory** - Detection, initiation, confirmation, coasting, and deletion\n2. **SQL Backend (TrackDatabaseManager)** - Real-time track storage, queries, and lifecycle operations\n3. **HDF5 Backend (TrackHDF5Storage)** - Scenario archival, time-series retrieval, and compression\n4. **Workflow Integration** - Real-time SQL pipeline feeding into archival HDF5 storage\n5. **Performance Analysis** - Query latency and storage efficiency across backends\n\n## Key Concepts\n\n- **TrackDatabaseManager**: SQLite-backed real-time track lifecycle management\n- **TrackHDF5Storage**: HDF5-backed archival storage for large-scale tracking datasets\n- **Lifecycle states**: TENTATIVE → CONFIRMED → COASTING → DEAD\n- **Detection association**: Linking raw sensor measurements to maintained tracks\n- **Scenario archival**: Exporting completed missions from SQL to compressed HDF5\n\n## Prerequisites\n\n```bash\npip install nrl-tracker matplotlib numpy h5py\n```\n\n## Estimated Time: 30-40 minutes",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "46prtdt8vpb",
   "source": "import os\nimport tempfile\nimport time\n\nimport numpy as np\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\nfrom pytcl.dynamic_estimation.kalman import kf_predict, kf_update\nfrom pytcl.io import (\n    TrackDatabaseManager,\n    TrackDatabaseStatus,\n    TrackHDF5Storage,\n)\n\nnp.random.seed(42)\n\n# Plotly dark theme template\ndark_template = go.layout.Template()\ndark_template.layout = go.Layout(\n    paper_bgcolor='#0d1117',\n    plot_bgcolor='#0d1117',\n    font=dict(color='#e6edf3'),\n    xaxis=dict(gridcolor='#30363d', zerolinecolor='#30363d'),\n    yaxis=dict(gridcolor='#30363d', zerolinecolor='#30363d'),\n)\n\n# Temporary directory for database files\nTMPDIR = tempfile.mkdtemp(prefix=\"pytcl_nb_\")\n\nprint(\"PyTCL track management modules imported successfully\")\nprint(f\"Working directory: {TMPDIR}\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "9n0r5o9y1cm",
   "source": "## 1. Track Lifecycle Theory\n\n### The Detection-to-Track Pipeline\n\nIn a tracking system, raw sensor measurements (detections) must be processed through a pipeline that manages the full lifecycle of each track:\n\n```\nSensor Measurements\n        │\n        ▼\n  ┌─────────────┐\n  │  Detection   │  Store raw measurements with sensor ID, timestamp\n  │  Storage     │\n  └──────┬──────┘\n         │\n         ▼\n  ┌─────────────┐\n  │    Data      │  Associate detections to existing tracks\n  │ Association  │  (GNN, JPDA, MHT)\n  └──────┬──────┘\n         │\n    ┌────┴────┐\n    │         │\n    ▼         ▼\nMatched    Unmatched\n    │         │\n    ▼         ▼\n  ┌──────┐  ┌──────────┐\n  │Update│  │ Initiate │  Create new tentative track\n  │Track │  │ New Track│\n  └──────┘  └──────────┘\n```\n\n### Track Lifecycle States\n\nEach track transitions through a state machine:\n\n| State | Meaning | Transition Rule |\n|-------|---------|----------------|\n| **TENTATIVE** | Newly created, awaiting confirmation | Created from unassociated detections |\n| **CONFIRMED** | Reliably tracked target | After N consecutive hits (e.g., 3) |\n| **COASTING** | Temporarily lost, predicting forward | After M consecutive misses without deletion |\n| **DEAD** | Terminated, awaiting cleanup | After K total misses or manual deletion |\n\n### Track Quality Metrics\n\n- **Hits**: Number of successful measurement-to-track associations\n- **Misses**: Consecutive scans without association (resets on hit)\n- **Total misses**: Cumulative missed associations over track lifetime\n- **Confidence score**: Derived metric combining hits, misses, and association quality\n\n### Database Design\n\nPyTCL provides two complementary storage backends:\n\n| Feature | SQL (TrackDatabaseManager) | HDF5 (TrackHDF5Storage) |\n|---------|---------------------------|------------------------|\n| **Use case** | Real-time operations | Post-mission archival |\n| **Query speed** | Fast random access by ID/time | Fast sequential/bulk reads |\n| **Schema** | Relational (detections, tracks, states) | Hierarchical (groups, datasets) |\n| **Compression** | None (small records) | gzip chunked (large arrays) |\n| **Concurrency** | SQLite WAL mode | Single-writer |\n| **Best for** | Live tracking loop | Analysis & replay |",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "id": "det3qqdkpld",
   "source": "## 2. SQL Tutorial: TrackDatabaseManager\n\nThe `TrackDatabaseManager` provides real-time track lifecycle management backed by SQLite. It stores:\n\n- **Detections** — raw sensor measurements with metadata\n- **Tracks** — track state, status, and lifecycle counters\n- **Track states** — full state/covariance history over time\n- **Associations** — detection-to-track linkages with confidence\n\n### 2.1 Creating the Database and Storing Detections",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "yexbox6ygrf",
   "source": "# Create a new tracking database\ndb_path = os.path.join(TMPDIR, \"tracking_tutorial.db\")\ndb = TrackDatabaseManager(db_path)\ndb.open(mode=\"w\")\n\n# Simulate a simple radar scenario: 3 targets, 30 time steps\ndt = 1.0\nn_steps = 30\nn_targets = 3\nR = np.eye(2) * 4.0  # Measurement noise covariance (2m std)\n\n# True target trajectories: [x, vx, y, vy]\ntargets = [\n    {\"id\": \"tgt_A\", \"x0\": np.array([0.0, 2.0, 0.0, 1.5])},\n    {\"id\": \"tgt_B\", \"x0\": np.array([80.0, -1.0, 10.0, 2.0])},\n    {\"id\": \"tgt_C\", \"x0\": np.array([40.0, 0.5, 60.0, -1.0])},\n]\n\n# Constant velocity transition matrix\nF = np.array([\n    [1, dt, 0, 0],\n    [0, 1, 0, 0],\n    [0, 0, 1, dt],\n    [0, 0, 0, 1],\n])\n\nH = np.array([[1, 0, 0, 0], [0, 0, 1, 0]])  # Observe position only\n\n# Generate and store detections\nrng = np.random.default_rng(42)\ntrue_states = {t[\"id\"]: [] for t in targets}\ndetection_count = 0\n\nfor k in range(n_steps):\n    timestamp = k * dt\n    for tgt in targets:\n        # Propagate true state\n        state = tgt[\"x0\"].copy()\n        state[0] += state[1] * timestamp\n        state[2] += state[3] * timestamp\n        true_states[tgt[\"id\"]].append(state.copy())\n\n        # Generate detection (90% probability of detection)\n        if rng.random() < 0.9:\n            pos = H @ state\n            meas = pos + rng.multivariate_normal([0, 0], R)\n\n            det_id = f\"det_{k:03d}_{tgt['id']}\"\n            db.store_detection(\n                detection_id=det_id,\n                measurement=meas,\n                sensor_id=\"radar_01\",\n                timestamp=timestamp,\n                covariance=R,\n                metadata={\"snr\": float(rng.uniform(10, 30))},\n            )\n            detection_count += 1\n\n# Convert to arrays for later use\nfor tid in true_states:\n    true_states[tid] = np.array(true_states[tid])\n\nprint(f\"Database created: {db_path}\")\nprint(f\"Stored {detection_count} detections over {n_steps} time steps\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "id": "j4b86gmki1c",
   "source": "# Query detections by time range and sensor\nearly_dets = db.retrieve_detections(start_time=0.0, end_time=5.0, sensor_id=\"radar_01\")\nprint(f\"Detections in [0, 5] seconds: {len(early_dets)}\")\n\n# Inspect a single detection\ndet = db.retrieve_detection(\"det_000_tgt_A\")\nprint(f\"\\nDetection 'det_000_tgt_A':\")\nprint(f\"  Measurement: [{det['measurement'][0]:.2f}, {det['measurement'][1]:.2f}]\")\nprint(f\"  Timestamp:   {det['timestamp']}\")\nprint(f\"  Sensor:      {det['sensor_id']}\")\nprint(f\"  Status:      {det['association_status']}\")\nprint(f\"  Metadata:    SNR = {det['metadata'].get('snr', 'N/A'):.1f} dB\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "worp5jecwu",
   "source": "### 2.2 Track Initiation and State Updates\n\nNow we'll run a Kalman filter over the detections and store each track's state history in the database. This demonstrates the core real-time workflow:\n\n1. **Initiate** tracks from early detections\n2. **Predict → Update** with the Kalman filter\n3. **Store** each state update in the database\n4. **Associate** detections to tracks",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "jsoo1m80m8",
   "source": "# Process noise covariance\nq = 0.1\nQ = q * np.array([\n    [dt**3/3, dt**2/2, 0, 0],\n    [dt**2/2, dt, 0, 0],\n    [0, 0, dt**3/3, dt**2/2],\n    [0, 0, dt**2/2, dt],\n])\n\n# Initial covariance (large uncertainty)\nP0 = np.diag([R[0, 0], 1.0, R[1, 1], 1.0])\n\n# Track state: {track_id: (x, P)}\ntrack_filters = {}\n\nfor k in range(n_steps):\n    timestamp = k * dt\n\n    # Get detections at this time step\n    dets = db.retrieve_detections(start_time=timestamp, end_time=timestamp)\n\n    # Predict existing tracks\n    for tid, (x, P) in track_filters.items():\n        pred = kf_predict(x, P, F, Q)\n        track_filters[tid] = (pred.x, pred.P)\n\n    # Simple nearest-neighbor association for each target\n    for tgt in targets:\n        tid = f\"trk_{tgt['id']}\"\n        det_id = f\"det_{k:03d}_{tgt['id']}\"\n\n        # Find the matching detection (if it exists)\n        matching = [d for d in dets if d[\"detection_id\"] == det_id]\n        if not matching:\n            # Missed detection: store prediction as coasting update\n            if tid in track_filters:\n                x, P = track_filters[tid]\n                db.update_track_state(tid, x, P, timestamp, update_type=\"prediction\")\n            continue\n\n        z = matching[0][\"measurement\"]\n\n        if tid not in track_filters:\n            # Initiate new track\n            x0 = np.array([z[0], 0.0, z[1], 0.0])\n            db.initiate_track(tid, x0, P0, timestamp)\n            track_filters[tid] = (x0, P0.copy())\n\n        # Kalman update\n        x, P = track_filters[tid]\n        upd = kf_update(x, P, z, H, R)\n        track_filters[tid] = (upd.x, upd.P)\n\n        # Store updated state and associate detection\n        db.update_track_state(tid, upd.x, upd.P, timestamp, update_type=\"update\")\n        db.associate_detection(det_id, tid, confidence=0.95)\n\n# Confirm all tracks (they've been running for 30 steps)\nfor tgt in targets:\n    tid = f\"trk_{tgt['id']}\"\n    db.confirm_track(tid)\n\nprint(f\"Processed {n_steps} time steps\")\nprint(f\"Active tracks: {len(track_filters)}\")\nfor tid in sorted(track_filters):\n    info = db.get_track(tid)\n    print(f\"  {tid}: status={info['status']}, hits={info['hits']}, misses={info['misses']}\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "s6w95ps84sq",
   "source": "### 2.3 Querying Track History and Visualization\n\nThe database stores the full state/covariance timeline for each track. Let's retrieve and visualize it.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "de35jj7s87g",
   "source": "# Retrieve and plot track histories vs ground truth\ncolors = ['#00d4ff', '#ff4757', '#00ff88']\nfig = make_subplots(\n    rows=1, cols=2,\n    subplot_titles=('2D Track Trajectories', 'Position Error Over Time'),\n    horizontal_spacing=0.12,\n)\n\nfor i, tgt in enumerate(targets):\n    tid = f\"trk_{tgt['id']}\"\n    history = db.get_track_history(tid)\n    est_states = history[\"states\"]\n    est_times = history[\"timestamps\"]\n    truth = true_states[tgt[\"id\"]]\n\n    # Left: 2D trajectories\n    fig.add_trace(\n        go.Scatter(x=truth[:, 0], y=truth[:, 2], mode='lines',\n                   name=f'True {tgt[\"id\"]}', line=dict(color=colors[i], width=2),\n                   opacity=0.4, legendgroup=tgt[\"id\"]),\n        row=1, col=1,\n    )\n    fig.add_trace(\n        go.Scatter(x=est_states[:, 0], y=est_states[:, 2], mode='lines',\n                   name=f'Est {tgt[\"id\"]}',\n                   line=dict(color=colors[i], width=2, dash='dash'),\n                   legendgroup=tgt[\"id\"]),\n        row=1, col=1,\n    )\n\n    # Right: Position error\n    min_len = min(len(est_states), len(truth))\n    pos_err = np.sqrt(\n        (est_states[:min_len, 0] - truth[:min_len, 0]) ** 2\n        + (est_states[:min_len, 2] - truth[:min_len, 2]) ** 2\n    )\n    fig.add_trace(\n        go.Scatter(x=est_times[:min_len], y=pos_err, mode='lines',\n                   name=f'Error {tgt[\"id\"]}', line=dict(color=colors[i], width=2),\n                   legendgroup=tgt[\"id\"], showlegend=False),\n        row=1, col=2,\n    )\n\nfig.update_layout(\n    template=dark_template, height=450,\n    legend=dict(x=0.5, y=-0.15, xanchor='center', orientation='h'),\n)\nfig.update_xaxes(title_text='X (m)', row=1, col=1)\nfig.update_yaxes(title_text='Y (m)', row=1, col=1)\nfig.update_xaxes(title_text='Time (s)', row=1, col=2)\nfig.update_yaxes(title_text='Position Error (m)', row=1, col=2)\nfig.show()",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "wm4h18haa1q",
   "source": "### 2.4 Track Lifecycle Management\n\nThe database supports full lifecycle operations: marking tracks as coasting/dead, pruning old data, and merging duplicate tracks.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "r4xh5jrn3ur",
   "source": "# Demonstrate lifecycle transitions\nprint(\"=== Track Lifecycle Demo ===\\n\")\n\n# 1. Check current status\nall_tracks = db.retrieve_all_tracks()\nprint(\"Current track statuses:\")\nfor t in all_tracks:\n    print(f\"  {t['track_id']}: {t['status']} (hits={t['hits']}, misses={t['misses']})\")\n\n# 2. Mark one track as coasting (simulating lost contact)\ndb.mark_track_coasting(\"trk_tgt_C\")\ninfo = db.get_track(\"trk_tgt_C\")\nprint(f\"\\nAfter marking trk_tgt_C as coasting: status={info['status']}\")\n\n# 3. Mark it dead (target left surveillance region)\ndb.mark_track_dead(\"trk_tgt_C\")\ninfo = db.get_track(\"trk_tgt_C\")\nprint(f\"After marking trk_tgt_C as dead: status={info['status']}\")\n\n# 4. Query by status\nconfirmed = db.retrieve_all_tracks(status=TrackDatabaseStatus.CONFIRMED)\ndead = db.retrieve_all_tracks(status=TrackDatabaseStatus.DEAD)\nprint(f\"\\nConfirmed tracks: {len(confirmed)}\")\nprint(f\"Dead tracks: {len(dead)}\")\n\n# 5. Prune old unassociated detections (older than 10s from newest)\npruned = db.prune_old_detections(age_threshold=10.0)\nprint(f\"\\nPruned {pruned} old unassociated detections\")\n\n# 6. Restore trk_tgt_C for subsequent sections\ndb.mark_track_confirmed(\"trk_tgt_C\")\nprint(\"\\nRestored trk_tgt_C to confirmed for next sections\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "9cg32suxmkw",
   "source": "### 2.5 Bulk Operations and pytcl Integration\n\nThe `TrackDatabaseManager` can convert between database records and pytcl's native `Track` / `TrackList` containers, enabling seamless integration with existing filter pipelines.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "ciebkgkiahq",
   "source": "# Convert database tracks to pytcl Track objects\ntrack_obj = db.track_to_pytcl(\"trk_tgt_A\")\nprint(f\"pytcl Track object for trk_tgt_A:\")\nprint(f\"  id:     {track_obj.id}\")\nprint(f\"  state:  [{', '.join(f'{v:.2f}' for v in track_obj.state)}]\")\nprint(f\"  status: {track_obj.status}\")\nprint(f\"  hits:   {track_obj.hits}\")\nprint(f\"  time:   {track_obj.time}\")\n\n# Convert all confirmed tracks to a TrackList\ntrack_list = db.tracks_to_tracklist(status=TrackDatabaseStatus.CONFIRMED)\nprint(f\"\\nTrackList with {len(track_list)} confirmed tracks\")\n\n# Store a batch of state history at once\nbatch_states = np.random.randn(5, 4)  # 5 timesteps, 4-dim state\nbatch_covs = np.array([np.eye(4) for _ in range(5)])\nbatch_times = np.arange(100.0, 105.0)\n\ndb.initiate_track(\"trk_batch_demo\", batch_states[0], batch_covs[0], 100.0)\ndb.store_track_history(\"trk_batch_demo\", batch_states, batch_covs, batch_times)\nhistory = db.get_track_history(\"trk_batch_demo\")\nprint(f\"\\nBatch-stored track 'trk_batch_demo': {len(history['timestamps'])} state entries\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "scxzicvkq0c",
   "source": "## 3. HDF5 Tutorial: TrackHDF5Storage\n\nThe `TrackHDF5Storage` backend is optimized for archiving completed tracking scenarios. It uses:\n\n- **Chunked datasets** for efficient sequential I/O\n- **gzip compression** for reduced file size (typically 5-10x)\n- **Hierarchical groups** for organizing tracks and scenarios\n- **Resizable datasets** for appending new states incrementally\n\n### 3.1 Storing Tracks in HDF5",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "atfta3hffa",
   "source": "# Create HDF5 storage and archive track data from the SQL database\nh5_path = os.path.join(TMPDIR, \"scenario_archive.h5\")\nstore = TrackHDF5Storage(h5_path, compression=\"gzip\", compression_level=4)\nstore.open(mode=\"w\")\n\n# Store each track's full trajectory from the SQL database\nfor tgt in targets:\n    tid = f\"trk_{tgt['id']}\"\n    history = db.get_track_history(tid)\n    track_info = db.get_track(tid)\n\n    store.store_track(\n        track_id=tid,\n        states=history[\"states\"],\n        covariances=history[\"covariances\"],\n        timestamps=history[\"timestamps\"],\n        metadata={\n            \"status\": track_info[\"status\"],\n            \"hits\": track_info[\"hits\"],\n            \"birth_time\": track_info[\"birth_time\"],\n        },\n    )\n\n# List stored tracks\nstored_tracks = store.list_tracks()\nprint(f\"HDF5 file: {h5_path}\")\nprint(f\"Stored tracks: {stored_tracks}\")\n\n# Retrieve and verify a track\nretrieved = store.retrieve_track(\"trk_tgt_A\")\nprint(f\"\\nRetrieved trk_tgt_A:\")\nprint(f\"  States shape:      {retrieved['states'].shape}\")\nprint(f\"  Covariances shape: {retrieved['covariances'].shape}\")\nprint(f\"  Time range:        [{retrieved['timestamps'][0]:.1f}, {retrieved['timestamps'][-1]:.1f}] s\")\nprint(f\"  Metadata:          {retrieved['metadata']}\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "8618mwbdeet",
   "source": "### 3.2 Time-Series Queries\n\nHDF5 supports efficient time-sliced retrieval and interpolation — useful for post-mission analysis.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "qxo6co8lxfg",
   "source": "# Time-sliced trajectory extraction\ntraj = store.get_track_trajectory(\"trk_tgt_B\", start_time=5.0, end_time=20.0)\nprint(f\"Trajectory slice [5s, 20s]:\")\nprint(f\"  States shape: {traj['states'].shape}\")\nprint(f\"  Time range:   [{traj['timestamps'][0]:.1f}, {traj['timestamps'][-1]:.1f}] s\")\n\n# State interpolation at arbitrary time\nstate_nearest = store.get_state_at_time(\"trk_tgt_B\", time=7.5, interpolate=False)\nstate_interp = store.get_state_at_time(\"trk_tgt_B\", time=7.5, interpolate=True)\n\nprint(f\"\\nState at t=7.5s (nearest):      [{', '.join(f'{v:.2f}' for v in state_nearest['state'])}]\")\nprint(f\"State at t=7.5s (interpolated): [{', '.join(f'{v:.2f}' for v in state_interp['state'])}]\")\nprint(f\"Nearest timestamp: {state_nearest['timestamp']:.1f}s\")\nprint(f\"Interpolated timestamp: {state_interp['timestamp']:.1f}s\")\n\n# Spatial query: find tracks within a bounding box\ntracks_in_region = store.get_tracks_in_region(\n    bbox=[20, 20, 80, 80],  # [x_min, y_min, x_max, y_max]\n    time_range=[10.0, 25.0],\n    state_indices=(0, 2),  # x is index 0, y is index 2\n)\nprint(f\"\\nTracks passing through [20,20]-[80,80] between t=10-25s: {tracks_in_region}\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "0a1ux65b5cje",
   "source": "### 3.3 Scenario Archival and Comparison\n\nHDF5 supports storing complete tracking scenarios (tracks + detections) under named groups, enabling side-by-side comparison of different algorithm configurations.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "fe8pucvj7l9",
   "source": "# Store two scenarios with different noise levels\nfor scenario_name, noise_mult in [(\"low_noise\", 0.5), (\"high_noise\", 2.0)]:\n    tracks_data = {}\n    rng_sc = np.random.default_rng(123)\n\n    for tgt in targets:\n        tid = tgt[\"id\"]\n        n = n_steps\n        states_sc = np.zeros((n, 4))\n        covs_sc = np.array([np.eye(4) * noise_mult for _ in range(n)])\n        times_sc = np.arange(n, dtype=np.float64)\n\n        for k in range(n):\n            state = tgt[\"x0\"].copy()\n            state[0] += state[1] * k * dt\n            state[2] += state[3] * k * dt\n            # Add noise proportional to scenario\n            state[:2] += rng_sc.normal(0, noise_mult, 2)\n            state[2:] += rng_sc.normal(0, noise_mult, 2)\n            states_sc[k] = state\n\n        tracks_data[tid] = {\n            \"states\": states_sc,\n            \"covariances\": covs_sc,\n            \"timestamps\": times_sc,\n        }\n\n    store.store_tracking_scenario(\n        scenario_id=scenario_name,\n        tracks=tracks_data,\n        metadata={\"noise_multiplier\": noise_mult, \"n_targets\": n_targets},\n    )\n\n# List scenarios and compare\nscenarios = store.list_scenarios()\nprint(f\"Stored scenarios: {scenarios}\")\n\ncomparison = store.compare_scenarios(\"low_noise\", \"high_noise\")\nprint(f\"\\nScenario comparison (low_noise vs high_noise):\")\nprint(f\"  Common tracks:  {comparison['common_tracks']}\")\nprint(f\"  State RMSE per track:\")\nfor tid, rmse in comparison[\"state_differences\"].items():\n    print(f\"    {tid}: {rmse:.3f}\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "id": "e6lgrr9qpjv",
   "source": "# Visualize scenario comparison\nfig = make_subplots(\n    rows=1, cols=2,\n    subplot_titles=('Low Noise Scenario', 'High Noise Scenario'),\n    horizontal_spacing=0.1,\n)\n\nfor col, scenario_id in enumerate([\"low_noise\", \"high_noise\"], 1):\n    scenario = store.retrieve_tracking_scenario(scenario_id)\n\n    for i, (tid, tdata) in enumerate(scenario[\"tracks\"].items()):\n        fig.add_trace(\n            go.Scatter(\n                x=tdata[\"states\"][:, 0], y=tdata[\"states\"][:, 2],\n                mode='lines', name=tid if col == 1 else None,\n                line=dict(color=colors[i], width=2),\n                legendgroup=tid, showlegend=(col == 1),\n            ),\n            row=1, col=col,\n        )\n\nfig.update_layout(\n    template=dark_template, height=400,\n    legend=dict(x=0.5, y=-0.15, xanchor='center', orientation='h'),\n)\nfig.update_xaxes(title_text='X (m)', row=1, col=1)\nfig.update_xaxes(title_text='X (m)', row=1, col=2)\nfig.update_yaxes(title_text='Y (m)', row=1, col=1)\nfig.update_yaxes(title_text='Y (m)', row=1, col=2)\nfig.show()",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "bjx1bwfhsow",
   "source": "### 3.4 Compression Analysis\n\nHDF5 compression reduces storage requirements significantly for tracking data, which contains highly correlated floating-point arrays.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "ry6n4xpxwpp",
   "source": "# Compare file sizes across compression levels\ncompression_results = []\nn_tracks_large = 20\nn_steps_large = 100\nstate_dim = 6  # [x, vx, y, vy, z, vz]\n\n# Generate a larger dataset\nrng_comp = np.random.default_rng(99)\nlarge_tracks = {}\nfor i in range(n_tracks_large):\n    states_l = np.cumsum(rng_comp.normal(0, 1, (n_steps_large, state_dim)), axis=0)\n    covs_l = np.array([np.eye(state_dim) * (1 + 0.01 * k) for k in range(n_steps_large)])\n    times_l = np.arange(n_steps_large, dtype=np.float64)\n    large_tracks[f\"trk_{i:03d}\"] = {\n        \"states\": states_l,\n        \"covariances\": covs_l,\n        \"timestamps\": times_l,\n    }\n\n# Raw data size\nraw_bytes = n_tracks_large * n_steps_large * (\n    state_dim * 8 + state_dim * state_dim * 8 + 8  # states + covs + timestamps\n)\n\nfor level in [0, 1, 4, 9]:\n    path = os.path.join(TMPDIR, f\"compress_test_{level}.h5\")\n    s = TrackHDF5Storage(path, compression=\"gzip\", compression_level=max(level, 1))\n    s.open(mode=\"w\")\n    s.store_tracking_scenario(\"test\", large_tracks)\n    s.close()\n    file_size = os.path.getsize(path)\n    ratio = raw_bytes / file_size if file_size > 0 else 0\n    compression_results.append({\n        \"level\": level,\n        \"file_kb\": file_size / 1024,\n        \"ratio\": ratio,\n    })\n\n# Display results\nprint(f\"Raw data size: {raw_bytes / 1024:.1f} KB ({n_tracks_large} tracks x {n_steps_large} steps x {state_dim}D state)\")\nprint(f\"\\n{'Level':>6} {'File Size':>12} {'Compression Ratio':>18}\")\nprint(\"-\" * 40)\nfor r in compression_results:\n    print(f\"{r['level']:>6} {r['file_kb']:>10.1f} KB {r['ratio']:>15.1f}x\")\n\n# Visualize\nfig = go.Figure()\nfig.add_trace(go.Bar(\n    x=[str(r[\"level\"]) for r in compression_results],\n    y=[r[\"file_kb\"] for r in compression_results],\n    text=[f\"{r['ratio']:.1f}x\" for r in compression_results],\n    textposition='outside',\n    marker_color=['#ff4757', '#ffb800', '#00d4ff', '#00ff88'],\n))\nfig.add_hline(y=raw_bytes / 1024, line_dash=\"dash\", line_color=\"#e6edf3\",\n              annotation_text=f\"Raw: {raw_bytes/1024:.0f} KB\", annotation_position=\"top left\")\nfig.update_layout(\n    template=dark_template, height=400,\n    title='HDF5 Compression: File Size by gzip Level',\n    xaxis_title='gzip Compression Level',\n    yaxis_title='File Size (KB)',\n)\nfig.show()",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "x2j9ehujle",
   "source": "## 4. Workflow Demo: Real-Time SQL to Archival HDF5\n\nA common operational pattern is:\n\n1. **Real-time phase**: Use SQL for fast detection storage, track updates, and lifecycle management during a live mission\n2. **Archive phase**: Export completed mission data from SQL to compressed HDF5 for long-term storage and offline analysis\n\nThis demonstrates the `import_from_sql` / `export_to_sql` interoperability.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "l60d8sxk6fg",
   "source": "# === PHASE 1: Real-time tracking with SQL ===\nrt_db_path = os.path.join(TMPDIR, \"realtime_mission.db\")\nrt_db = TrackDatabaseManager(rt_db_path)\nrt_db.open(mode=\"w\")\n\n# Simulate a 50-step mission with 5 targets\nrng_rt = np.random.default_rng(77)\nn_rt_targets = 5\nn_rt_steps = 50\n\n# Generate diverse target initial conditions\nrt_targets = []\nfor i in range(n_rt_targets):\n    angle = 2 * np.pi * i / n_rt_targets\n    x0 = np.array([\n        50 * np.cos(angle), 1.5 * np.cos(angle + 0.3),\n        50 * np.sin(angle), 1.5 * np.sin(angle + 0.3),\n    ])\n    rt_targets.append(x0)\n\n# Run real-time tracking loop\nrt_filters = {}\nfor k in range(n_rt_steps):\n    t = k * dt\n    for i, x0 in enumerate(rt_targets):\n        tid = f\"rt_trk_{i:02d}\"\n        # True position (constant velocity)\n        true_pos = np.array([x0[0] + x0[1] * t, x0[2] + x0[3] * t])\n\n        # Detection (95% PD)\n        if rng_rt.random() > 0.95:\n            if tid in rt_filters:\n                x, P = rt_filters[tid]\n                pred = kf_predict(x, P, F, Q)\n                rt_filters[tid] = (pred.x, pred.P)\n                rt_db.update_track_state(tid, pred.x, pred.P, t, update_type=\"prediction\")\n            continue\n\n        z = true_pos + rng_rt.multivariate_normal([0, 0], R)\n        det_id = f\"rt_det_{k:03d}_{i:02d}\"\n        rt_db.store_detection(det_id, z, f\"sensor_{i % 2}\", t)\n\n        if tid not in rt_filters:\n            x_init = np.array([z[0], 0.0, z[1], 0.0])\n            rt_db.initiate_track(tid, x_init, P0, t)\n            rt_filters[tid] = (x_init, P0.copy())\n            if k > 3:\n                rt_db.confirm_track(tid)\n\n        x, P = rt_filters[tid]\n        pred = kf_predict(x, P, F, Q)\n        upd = kf_update(pred.x, pred.P, z, H, R)\n        rt_filters[tid] = (upd.x, upd.P)\n        rt_db.update_track_state(tid, upd.x, upd.P, t, update_type=\"update\")\n        rt_db.associate_detection(det_id, tid)\n\n# Confirm all tracks\nfor i in range(n_rt_targets):\n    tid = f\"rt_trk_{i:02d}\"\n    rt_db.confirm_track(tid)\n\nrt_tracks = rt_db.retrieve_all_tracks()\nprint(f\"=== Real-Time Phase Complete ===\")\nprint(f\"SQL database: {rt_db_path}\")\nprint(f\"Tracks: {len(rt_tracks)}\")\nprint(f\"Database size: {os.path.getsize(rt_db_path) / 1024:.1f} KB\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "id": "vhvj7n40iyd",
   "source": "# === PHASE 2: Archive to HDF5 ===\narchive_path = os.path.join(TMPDIR, \"mission_archive.h5\")\narchive = TrackHDF5Storage(archive_path, compression=\"gzip\", compression_level=4)\narchive.open(mode=\"w\")\n\n# Import from SQL into HDF5 as a named scenario\nt_start = time.perf_counter()\narchive.import_from_sql(rt_db, scenario_id=\"mission_001\")\nt_archive = time.perf_counter() - t_start\n\narchive.close()\nrt_db.close()\n\narchive_size = os.path.getsize(archive_path)\nsql_size = os.path.getsize(rt_db_path)\n\nprint(f\"=== Archive Phase Complete ===\")\nprint(f\"HDF5 archive: {archive_path}\")\nprint(f\"Archive time: {t_archive * 1000:.1f} ms\")\nprint(f\"\\nStorage comparison:\")\nprint(f\"  SQL database: {sql_size / 1024:.1f} KB\")\nprint(f\"  HDF5 archive: {archive_size / 1024:.1f} KB\")\nprint(f\"  Ratio: {sql_size / archive_size:.1f}x\")\n\n# Verify round-trip: read back from HDF5\narchive.open(mode=\"r\")\nscenario = archive.retrieve_tracking_scenario(\"mission_001\")\nprint(f\"\\nRound-trip verification:\")\nprint(f\"  Tracks recovered: {len(scenario['tracks'])}\")\nprint(f\"  Detections recovered: {len(scenario['detections'])}\")\n\n# Spot-check a track\ntid_check = \"rt_trk_00\"\nif tid_check in scenario[\"tracks\"]:\n    t_data = scenario[\"tracks\"][tid_check]\n    print(f\"  {tid_check} states shape: {t_data['states'].shape}\")\n\narchive.close()",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "10pkcom9xeai",
   "source": "## 5. Interactive Exploration: Query Performance\n\nUnderstanding the performance characteristics of each backend helps you choose the right tool for your use case. Let's measure query latency across different operations.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "q1ujo9a4lop",
   "source": "# Benchmark SQL operations\nperf_db_path = os.path.join(TMPDIR, \"perf_bench.db\")\nperf_db = TrackDatabaseManager(perf_db_path)\nperf_db.open(mode=\"w\")\n\nn_bench_tracks = 50\nn_bench_steps = 100\nrng_bench = np.random.default_rng(55)\n\n# Populate benchmark database\nfor i in range(n_bench_tracks):\n    tid = f\"bench_trk_{i:03d}\"\n    x0 = rng_bench.normal(0, 10, 4)\n    perf_db.initiate_track(tid, x0, np.eye(4), 0.0)\n    for k in range(1, n_bench_steps):\n        x_k = x0 + rng_bench.normal(0, 0.1, 4) * k\n        perf_db.update_track_state(tid, x_k, np.eye(4), float(k))\n\n    det_id = f\"bench_det_{i:03d}\"\n    perf_db.store_detection(det_id, rng_bench.normal(0, 5, 2), \"radar\", float(i))\n\n# Benchmark: single track state retrieval\nn_trials = 50\nsql_timings = {}\n\nt0 = time.perf_counter()\nfor _ in range(n_trials):\n    perf_db.get_track_state(f\"bench_trk_{rng_bench.integers(0, n_bench_tracks):03d}\")\nsql_timings[\"get_track_state\"] = (time.perf_counter() - t0) / n_trials * 1000\n\n# Benchmark: track history retrieval\nt0 = time.perf_counter()\nfor _ in range(n_trials):\n    perf_db.get_track_history(f\"bench_trk_{rng_bench.integers(0, n_bench_tracks):03d}\")\nsql_timings[\"get_track_history\"] = (time.perf_counter() - t0) / n_trials * 1000\n\n# Benchmark: detection query by time range\nt0 = time.perf_counter()\nfor _ in range(n_trials):\n    perf_db.retrieve_detections(start_time=0.0, end_time=25.0)\nsql_timings[\"query_detections\"] = (time.perf_counter() - t0) / n_trials * 1000\n\n# Benchmark: list all tracks\nt0 = time.perf_counter()\nfor _ in range(n_trials):\n    perf_db.retrieve_all_tracks()\nsql_timings[\"list_all_tracks\"] = (time.perf_counter() - t0) / n_trials * 1000\n\nperf_db.close()\n\n# Benchmark HDF5 operations\nperf_h5_path = os.path.join(TMPDIR, \"perf_bench.h5\")\nperf_store = TrackHDF5Storage(perf_h5_path)\nperf_store.open(mode=\"w\")\n\n# Populate HDF5\nh5_tracks = {}\nfor i in range(n_bench_tracks):\n    tid = f\"bench_trk_{i:03d}\"\n    states_b = np.cumsum(rng_bench.normal(0, 1, (n_bench_steps, 4)), axis=0)\n    covs_b = np.array([np.eye(4) for _ in range(n_bench_steps)])\n    times_b = np.arange(n_bench_steps, dtype=np.float64)\n    perf_store.store_track(tid, states_b, covs_b, times_b)\n\nh5_timings = {}\n\n# Benchmark: full track retrieval\nt0 = time.perf_counter()\nfor _ in range(n_trials):\n    perf_store.retrieve_track(f\"bench_trk_{rng_bench.integers(0, n_bench_tracks):03d}\")\nh5_timings[\"retrieve_track\"] = (time.perf_counter() - t0) / n_trials * 1000\n\n# Benchmark: trajectory slice\nt0 = time.perf_counter()\nfor _ in range(n_trials):\n    perf_store.get_track_trajectory(\n        f\"bench_trk_{rng_bench.integers(0, n_bench_tracks):03d}\",\n        start_time=20.0, end_time=60.0,\n    )\nh5_timings[\"trajectory_slice\"] = (time.perf_counter() - t0) / n_trials * 1000\n\n# Benchmark: state interpolation\nt0 = time.perf_counter()\nfor _ in range(n_trials):\n    perf_store.get_state_at_time(\n        f\"bench_trk_{rng_bench.integers(0, n_bench_tracks):03d}\",\n        time=45.5, interpolate=True,\n    )\nh5_timings[\"interpolate_state\"] = (time.perf_counter() - t0) / n_trials * 1000\n\n# Benchmark: spatial query\nt0 = time.perf_counter()\nfor _ in range(min(n_trials, 10)):\n    perf_store.get_tracks_in_region([-10, -10, 10, 10])\nh5_timings[\"spatial_query\"] = (time.perf_counter() - t0) / min(n_trials, 10) * 1000\n\nperf_store.close()\n\n# Display results\nprint(f\"=== Query Performance ({n_bench_tracks} tracks, {n_bench_steps} steps each) ===\\n\")\nprint(f\"{'SQL Operation':<25} {'Latency':>10}\")\nprint(\"-\" * 37)\nfor op, ms in sql_timings.items():\n    print(f\"  {op:<23} {ms:>8.2f} ms\")\n\nprint(f\"\\n{'HDF5 Operation':<25} {'Latency':>10}\")\nprint(\"-\" * 37)\nfor op, ms in h5_timings.items():\n    print(f\"  {op:<23} {ms:>8.2f} ms\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "id": "mahyh4qzptb",
   "source": "# Visualize performance comparison\nall_ops = list(sql_timings.keys()) + list(h5_timings.keys())\nall_vals = list(sql_timings.values()) + list(h5_timings.values())\nall_backends = [\"SQL\"] * len(sql_timings) + [\"HDF5\"] * len(h5_timings)\nbar_colors = [\"#00d4ff\"] * len(sql_timings) + [\"#00ff88\"] * len(h5_timings)\n\nfig = go.Figure()\nfig.add_trace(go.Bar(\n    x=list(sql_timings.keys()),\n    y=list(sql_timings.values()),\n    name=\"SQL\",\n    marker_color=\"#00d4ff\",\n    text=[f\"{v:.2f}\" for v in sql_timings.values()],\n    textposition=\"outside\",\n))\nfig.add_trace(go.Bar(\n    x=list(h5_timings.keys()),\n    y=list(h5_timings.values()),\n    name=\"HDF5\",\n    marker_color=\"#00ff88\",\n    text=[f\"{v:.2f}\" for v in h5_timings.values()],\n    textposition=\"outside\",\n))\nfig.update_layout(\n    template=dark_template, height=450,\n    title=\"Query Latency by Backend and Operation\",\n    xaxis_title=\"Operation\",\n    yaxis_title=\"Latency (ms)\",\n    barmode=\"group\",\n)\nfig.show()",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "y41nj4pb66n",
   "source": "## Exercises\n\n### Exercise 1: Track Confirmation Logic\n\n**Objective**: Implement an M-of-N confirmation rule\n\n**Task**:\n1. Create a new `TrackDatabaseManager` database\n2. Initiate 5 tentative tracks\n3. Simulate detections where each track receives hits with 80% probability per scan\n4. Implement a 3-of-5 confirmation rule: confirm a track if it gets at least 3 hits in the first 5 scans\n5. Print which tracks were confirmed and which were deleted\n\n**Hint**: Use `db.get_track(tid)` to check `hits` and `misses` counts, then call `db.confirm_track()` or `db.mark_track_dead()`.\n\n---\n\n### Exercise 2: HDF5 Scenario Replay\n\n**Objective**: Load an archived scenario and replay it through a different filter\n\n**Task**:\n1. Open the `scenario_archive.h5` file created earlier (read-only)\n2. Retrieve the `low_noise` scenario\n3. For each track, treat the stored states as \"measurements\" and run a Kalman smoother (or simply a KF with different Q)\n4. Store the re-filtered results as a new scenario called `refiltered`\n5. Use `compare_scenarios` to quantify the difference between original and refiltered tracks\n\n**Hint**: Use `store.retrieve_tracking_scenario(\"low_noise\")` then `store.store_tracking_scenario(\"refiltered\", ...)`.\n\n---\n\n### Exercise 3: Multi-Sensor Detection Fusion\n\n**Objective**: Manage detections from multiple sensors with different noise characteristics\n\n**Task**:\n1. Create a database and store detections from two sensors:\n   - `radar_01`: 5m noise std, 1Hz update rate, 85% PD\n   - `lidar_01`: 1m noise std, 10Hz update rate, 95% PD\n2. Initiate tracks and run a Kalman filter that uses detections from both sensors\n3. Store separate covariance matrices for each sensor's detections\n4. Query detections by `sensor_id` to analyze each sensor's contribution\n5. Visualize the track uncertainty over time, highlighting which sensor provided each update\n\n**Hint**: Use `db.store_detection(..., sensor_id=\"radar_01\")` and query with `db.retrieve_detections(sensor_id=\"lidar_01\")`.\n\n---\n\n### Exercise 4: Track Merge Detection\n\n**Objective**: Detect and merge duplicate tracks from the same target\n\n**Task**:\n1. Create a scenario where two tracks are initiated on the same target (simulating a brief detection gap)\n2. Store both tracks' state histories in the database\n3. Implement a merge criterion: if two tracks' latest states are within 5m Mahalanobis distance, merge them\n4. Use `db.merge_tracks(keep_id, merge_id)` to combine the histories\n5. Verify that the merged track has the combined hit count and the merged track is marked DEAD\n\n**Hint**: Compare `db.get_track_state(tid1)` and `db.get_track_state(tid2)` states using Mahalanobis distance.",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "id": "x63ep91y2v",
   "source": "## Key Takeaways\n\n- **TrackDatabaseManager** (SQL) is ideal for real-time tracking loops: fast random access, lifecycle management, detection-track association\n- **TrackHDF5Storage** (HDF5) excels at post-mission archival: compressed storage, time-series queries, scenario comparison\n- **Lifecycle management** (TENTATIVE → CONFIRMED → COASTING → DEAD) provides structured track quality control\n- **SQL → HDF5 pipeline** enables a natural workflow: live tracking followed by compressed archival\n- **Both backends interoperate** via `import_from_sql` / `export_to_sql` for seamless data exchange\n\n## Next Steps\n\n- Explore **Notebook 03 (Multi-Target Tracking)** for data association algorithms (GNN, JPDA)\n- See `examples/track_management_workflows.py` for complete end-to-end pipeline examples\n- Review `pytcl.io` API documentation for the full method reference\n\n## References\n\n### Core Textbooks\n1. **Bar-Shalom, Y., Li, X. R., & Kirubarajan, T.** (2001). *Estimation with Applications to Tracking and Navigation*. Wiley. — Chapters 6-8 on track management and data association\n2. **Blackman, S. S., & Popoli, R.** (1999). *Design and Analysis of Modern Tracking Systems*. Artech House. — Comprehensive track lifecycle management\n\n### Data Storage\n3. **The HDF Group.** *HDF5 User's Guide*. — Chunking, compression, and hierarchical data organization\n4. **SQLite Documentation.** *Write-Ahead Logging*. — WAL mode for concurrent read/write access\n\n### PyTCL API\n- `pytcl.io.TrackDatabaseManager` — SQL track lifecycle management\n- `pytcl.io.TrackHDF5Storage` — HDF5 archival storage\n- `pytcl.io.TrackDatabaseStatus` — Lifecycle state enumeration\n- `examples/track_management_workflows.py` — End-to-end workflow examples",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "id": "xxadkd7b5",
   "source": "# Cleanup: close open connections and remove temp files\ndb.close()\nstore.close()\n\nimport shutil\nshutil.rmtree(TMPDIR, ignore_errors=True)\nprint(\"Temporary files cleaned up.\")",
   "metadata": {},
   "execution_count": null,
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.13.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}