synor/docs/PLAN/PHASE10-Database-Advanced.md

# Phase 10 Advanced Database Features

> Implementation plan for SQL, Graph, and Replication features in Synor Database L2

## Overview

These advanced features extend the Synor Database to support:
1. **Relational (SQL)** - SQLite-compatible query subset for structured data
2. **Graph Store** - Relationship queries for connected data
3. **Replication** - Raft consensus for high availability

## Feature 1: Relational (SQL) Store

### Purpose
Provide a familiar SQL interface for developers who need structured relational queries, joins, and ACID transactions.

### Architecture

```text
┌─────────────────────────────────────────────────────────────┐
│                     SQL QUERY LAYER                          │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │  SQL Parser  │  │   Planner    │  │    Executor      │   │
│  │  (sqlparser) │  │  (logical)   │  │  (physical)      │   │
│  └──────────────┘  └──────────────┘  └──────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              Table Storage Engine                     │   │
│  │  - Row-oriented storage                               │   │
│  │  - B-tree indexes                                     │   │
│  │  - Transaction log (WAL)                              │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
└─────────────────────────────────────────────────────────────┘
```

### Supported SQL Subset

| Category | Statements |
|----------|------------|
| DDL | CREATE TABLE, DROP TABLE, ALTER TABLE |
| DML | SELECT, INSERT, UPDATE, DELETE |
| Clauses | WHERE, ORDER BY, LIMIT, OFFSET, GROUP BY, HAVING |
| Joins | INNER JOIN, LEFT JOIN, RIGHT JOIN |
| Functions | COUNT, SUM, AVG, MIN, MAX, COALESCE |
| Operators | =, !=, <, >, <=, >=, AND, OR, NOT, IN, LIKE |

### Data Types

| SQL Type | Rust Type | Storage |
|----------|-----------|---------|
| INTEGER | i64 | 8 bytes |
| REAL | f64 | 8 bytes |
| TEXT | String | Variable |
| BLOB | Vec<u8> | Variable |
| BOOLEAN | bool | 1 byte |
| TIMESTAMP | u64 | 8 bytes (Unix ms) |

### Implementation Components

```
crates/synor-database/src/sql/
├── mod.rs           # Module exports
├── parser.rs        # SQL parsing (sqlparser-rs)
├── planner.rs       # Query planning & optimization
├── executor.rs      # Query execution engine
├── table.rs         # Table definition & storage
├── row.rs           # Row representation
├── types.rs         # SQL type system
├── transaction.rs   # ACID transactions
└── index.rs         # SQL-specific indexing
```

### API Design

```rust
// Table definition
pub struct TableDef {
    pub name: String,
    pub columns: Vec<ColumnDef>,
    pub primary_key: Option<String>,
    pub indexes: Vec<IndexDef>,
}

pub struct ColumnDef {
    pub name: String,
    pub data_type: SqlType,
    pub nullable: bool,
    pub default: Option<SqlValue>,
}

// SQL execution
pub struct SqlEngine {
    tables: HashMap<String, Table>,
    transaction_log: TransactionLog,
}

impl SqlEngine {
    pub fn execute(&mut self, sql: &str) -> Result<SqlResult, SqlError>;
    pub fn begin_transaction(&mut self) -> TransactionId;
    pub fn commit(&mut self, txn: TransactionId) -> Result<(), SqlError>;
    pub fn rollback(&mut self, txn: TransactionId);
}
```

### Gateway Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/db/:db/sql` | POST | Execute SQL query |
| `/db/:db/sql/tables` | GET | List tables |
| `/db/:db/sql/tables/:table` | GET | Get table schema |
| `/db/:db/sql/tables/:table` | DELETE | Drop table |

---

## Feature 2: Graph Store

### Purpose
Enable relationship-based queries for social networks, knowledge graphs, recommendation engines, and any connected data.

### Architecture

```text
┌─────────────────────────────────────────────────────────────┐
│                     GRAPH QUERY LAYER                        │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │  Graph Query │  │   Traversal  │  │   Path Finding   │   │
│  │   Parser     │  │    Engine    │  │   (Dijkstra)     │   │
│  └──────────────┘  └──────────────┘  └──────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │                Graph Storage Engine                   │   │
│  │  - Adjacency list storage                             │   │
│  │  - Edge index (source, target, type)                  │   │
│  │  - Property storage                                   │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
└─────────────────────────────────────────────────────────────┘
```

### Data Model

```text
Node (Vertex):
  - id: NodeId (32 bytes)
  - labels: Vec<String>
  - properties: JsonValue

Edge (Relationship):
  - id: EdgeId (32 bytes)
  - source: NodeId
  - target: NodeId
  - edge_type: String
  - properties: JsonValue
  - directed: bool
```

### Query Language (Simplified Cypher-like)

```
// Find all friends of user Alice
MATCH (a:User {name: "Alice"})-[:FRIEND]->(friend)
RETURN friend

// Find shortest path between two nodes
MATCH path = shortestPath((a:User {id: "123"})-[*]-(b:User {id: "456"}))
RETURN path

// Find mutual friends
MATCH (a:User {name: "Alice"})-[:FRIEND]->(mutual)<-[:FRIEND]-(b:User {name: "Bob"})
RETURN mutual
```

### Implementation Components

```
crates/synor-database/src/graph/
├── mod.rs           # Module exports
├── node.rs          # Node definition & storage
├── edge.rs          # Edge definition & storage
├── store.rs         # Graph storage engine
├── query.rs         # Query language parser
├── traversal.rs     # Graph traversal algorithms
├── path.rs          # Path finding (BFS, DFS, Dijkstra)
└── index.rs         # Graph-specific indexes
```

### API Design

```rust
pub struct Node {
    pub id: NodeId,
    pub labels: Vec<String>,
    pub properties: JsonValue,
}

pub struct Edge {
    pub id: EdgeId,
    pub source: NodeId,
    pub target: NodeId,
    pub edge_type: String,
    pub properties: JsonValue,
}

pub struct GraphStore {
    nodes: HashMap<NodeId, Node>,
    edges: HashMap<EdgeId, Edge>,
    adjacency: HashMap<NodeId, Vec<EdgeId>>,  // outgoing
    reverse_adj: HashMap<NodeId, Vec<EdgeId>>, // incoming
}

impl GraphStore {
    // Node operations
    pub fn create_node(&mut self, labels: Vec<String>, props: JsonValue) -> NodeId;
    pub fn get_node(&self, id: &NodeId) -> Option<&Node>;
    pub fn update_node(&mut self, id: &NodeId, props: JsonValue) -> Result<(), GraphError>;
    pub fn delete_node(&mut self, id: &NodeId) -> Result<(), GraphError>;

    // Edge operations
    pub fn create_edge(&mut self, source: NodeId, target: NodeId, edge_type: &str, props: JsonValue) -> EdgeId;
    pub fn get_edge(&self, id: &EdgeId) -> Option<&Edge>;
    pub fn delete_edge(&mut self, id: &EdgeId) -> Result<(), GraphError>;

    // Traversal
    pub fn neighbors(&self, id: &NodeId, direction: Direction) -> Vec<&Node>;
    pub fn edges_of(&self, id: &NodeId, direction: Direction) -> Vec<&Edge>;
    pub fn shortest_path(&self, from: &NodeId, to: &NodeId) -> Option<Vec<NodeId>>;
    pub fn traverse(&self, start: &NodeId, query: &TraversalQuery) -> Vec<TraversalResult>;
}
```

### Gateway Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/db/:db/graph/nodes` | POST | Create node |
| `/db/:db/graph/nodes/:id` | GET | Get node |
| `/db/:db/graph/nodes/:id` | PUT | Update node |
| `/db/:db/graph/nodes/:id` | DELETE | Delete node |
| `/db/:db/graph/edges` | POST | Create edge |
| `/db/:db/graph/edges/:id` | GET | Get edge |
| `/db/:db/graph/edges/:id` | DELETE | Delete edge |
| `/db/:db/graph/query` | POST | Execute graph query |
| `/db/:db/graph/path` | POST | Find shortest path |
| `/db/:db/graph/traverse` | POST | Traverse from node |

---

## Feature 3: Replication (Raft Consensus)

### Purpose
Provide high availability and fault tolerance through distributed consensus, ensuring data consistency across multiple nodes.

### Architecture

```text
┌─────────────────────────────────────────────────────────────┐
│                    RAFT CONSENSUS LAYER                      │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │   Leader     │  │  Follower    │  │   Candidate      │   │
│  │   Election   │  │  Replication │  │   (Election)     │   │
│  └──────────────┘  └──────────────┘  └──────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │                    Log Replication                    │   │
│  │  - Append entries                                     │   │
│  │  - Commit index                                       │   │
│  │  - Log compaction (snapshots)                         │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │                  State Machine                        │   │
│  │  - Apply committed entries                            │   │
│  │  - Database operations                                │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
└─────────────────────────────────────────────────────────────┘
```

### Raft Protocol Overview

```text
Leader Election:
1. Followers timeout → become Candidate
2. Candidate requests votes from peers
3. Majority votes → become Leader
4. Leader sends heartbeats to maintain authority

Log Replication:
1. Client sends write to Leader
2. Leader appends to local log
3. Leader replicates to Followers
4. Majority acknowledge → entry committed
5. Leader applies to state machine
6. Leader responds to client
```

### Implementation Components

```
crates/synor-database/src/replication/
├── mod.rs           # Module exports
├── raft.rs          # Core Raft implementation
├── state.rs         # Node state (Leader/Follower/Candidate)
├── log.rs           # Replicated log
├── rpc.rs           # RPC messages (AppendEntries, RequestVote)
├── election.rs      # Leader election logic
├── snapshot.rs      # Log compaction & snapshots
├── cluster.rs       # Cluster membership
└── client.rs        # Client for forwarding to leader
```

### API Design

```rust
#[derive(Clone, Copy, PartialEq)]
pub enum NodeRole {
    Leader,
    Follower,
    Candidate,
}

pub struct RaftConfig {
    pub node_id: u64,
    pub peers: Vec<PeerAddress>,
    pub election_timeout_ms: (u64, u64),  // min, max
    pub heartbeat_interval_ms: u64,
    pub snapshot_threshold: u64,
}

pub struct LogEntry {
    pub term: u64,
    pub index: u64,
    pub command: Command,
}

pub enum Command {
    // Database operations
    KvSet { key: String, value: Vec<u8> },
    KvDelete { key: String },
    DocInsert { collection: String, doc: JsonValue },
    DocUpdate { collection: String, id: DocumentId, update: JsonValue },
    DocDelete { collection: String, id: DocumentId },
    // ... other operations
}

pub struct RaftNode {
    config: RaftConfig,
    state: RaftState,
    log: ReplicatedLog,
    state_machine: Arc<Database>,
}

impl RaftNode {
    pub async fn start(&mut self) -> Result<(), RaftError>;
    pub async fn propose(&self, command: Command) -> Result<(), RaftError>;
    pub fn is_leader(&self) -> bool;
    pub fn leader_id(&self) -> Option<u64>;
    pub fn status(&self) -> ClusterStatus;
}

// RPC Messages
pub struct AppendEntries {
    pub term: u64,
    pub leader_id: u64,
    pub prev_log_index: u64,
    pub prev_log_term: u64,
    pub entries: Vec<LogEntry>,
    pub leader_commit: u64,
}

pub struct RequestVote {
    pub term: u64,
    pub candidate_id: u64,
    pub last_log_index: u64,
    pub last_log_term: u64,
}
```

### Cluster Configuration

```yaml
# docker-compose.raft.yml
services:
  db-node-1:
    image: synor/database:latest
    environment:
      RAFT_NODE_ID: 1
      RAFT_PEERS: "db-node-2:5000,db-node-3:5000"
      RAFT_ELECTION_TIMEOUT: "150-300"
      RAFT_HEARTBEAT_MS: 50
    ports:
      - "8484:8484"  # HTTP API
      - "5000:5000"  # Raft RPC

  db-node-2:
    image: synor/database:latest
    environment:
      RAFT_NODE_ID: 2
      RAFT_PEERS: "db-node-1:5000,db-node-3:5000"
    ports:
      - "8485:8484"
      - "5001:5000"

  db-node-3:
    image: synor/database:latest
    environment:
      RAFT_NODE_ID: 3
      RAFT_PEERS: "db-node-1:5000,db-node-2:5000"
    ports:
      - "8486:8484"
      - "5002:5000"
```

### Gateway Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/cluster/status` | GET | Get cluster status |
| `/cluster/leader` | GET | Get current leader |
| `/cluster/nodes` | GET | List all nodes |
| `/cluster/nodes/:id` | DELETE | Remove node from cluster |
| `/cluster/nodes` | POST | Add node to cluster |

---

## Implementation Order

### Step 1: SQL Store
1. Add `sqlparser` dependency
2. Implement type system (`types.rs`)
3. Implement row storage (`row.rs`, `table.rs`)
4. Implement SQL parser wrapper (`parser.rs`)
5. Implement query planner (`planner.rs`)
6. Implement query executor (`executor.rs`)
7. Add transaction support (`transaction.rs`)
8. Add gateway endpoints
9. Write tests

### Step 2: Graph Store
1. Implement node/edge types (`node.rs`, `edge.rs`)
2. Implement graph storage (`store.rs`)
3. Implement query parser (`query.rs`)
4. Implement traversal algorithms (`traversal.rs`, `path.rs`)
5. Add graph indexes (`index.rs`)
6. Add gateway endpoints
7. Write tests

### Step 3: Replication
1. Implement Raft state machine (`state.rs`, `raft.rs`)
2. Implement replicated log (`log.rs`)
3. Implement RPC layer (`rpc.rs`)
4. Implement leader election (`election.rs`)
5. Implement log compaction (`snapshot.rs`)
6. Implement cluster management (`cluster.rs`)
7. Integrate with database operations
8. Add gateway endpoints
9. Write tests
10. Create Docker Compose for cluster

---

## Pricing Impact

| Feature | Operation | Cost (SYNOR) |
|---------|-----------|--------------|
| SQL | Query/million | 0.02 |
| SQL | Write/million | 0.05 |
| Graph | Traversal/million | 0.03 |
| Graph | Path query/million | 0.05 |
| Replication | Included | Base storage cost |

---

## Success Criteria

### SQL Store
- [ ] Parse and execute basic SELECT, INSERT, UPDATE, DELETE
- [ ] Support WHERE clauses with operators
- [ ] Support ORDER BY, LIMIT, OFFSET
- [ ] Support simple JOINs
- [ ] Support aggregate functions
- [ ] ACID transactions

### Graph Store
- [ ] Create/read/update/delete nodes and edges
- [ ] Traverse neighbors (in/out/both)
- [ ] Find shortest path between nodes
- [ ] Execute pattern matching queries
- [ ] Support property filters

### Replication
- [ ] Leader election works correctly
- [ ] Log replication achieves consensus
- [ ] Reads from any node (eventual consistency)
- [ ] Writes only through leader
- [ ] Node failure handled gracefully
- [ ] Log compaction reduces storage