synor/docs/PLAN/PHASE10-Database-Advanced.md

Phase 10 Advanced Database Features

Implementation plan for SQL, Graph, and Replication features in Synor Database L2

Overview

These advanced features extend the Synor Database to support:

1. Relational (SQL) - SQLite-compatible query subset for structured data
2. Graph Store - Relationship queries for connected data
3. Replication - Raft consensus for high availability

Feature 1: Relational (SQL) Store

Purpose

Provide a familiar SQL interface for developers who need structured relational queries, joins, and ACID transactions.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     SQL QUERY LAYER                          │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │  SQL Parser  │  │   Planner    │  │    Executor      │   │
│  │  (sqlparser) │  │  (logical)   │  │  (physical)      │   │
│  └──────────────┘  └──────────────┘  └──────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              Table Storage Engine                     │   │
│  │  - Row-oriented storage                               │   │
│  │  - B-tree indexes                                     │   │
│  │  - Transaction log (WAL)                              │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Supported SQL Subset

Category	Statements
DDL	CREATE TABLE, DROP TABLE, ALTER TABLE
DML	SELECT, INSERT, UPDATE, DELETE
Clauses	WHERE, ORDER BY, LIMIT, OFFSET, GROUP BY, HAVING
Joins	INNER JOIN, LEFT JOIN, RIGHT JOIN
Functions	COUNT, SUM, AVG, MIN, MAX, COALESCE
Operators	=, !=, <, >, <=, >=, AND, OR, NOT, IN, LIKE

Data Types

SQL Type	Rust Type	Storage
INTEGER	i64	8 bytes
REAL	f64	8 bytes
TEXT	String	Variable
BLOB	Vec	Variable
BOOLEAN	bool	1 byte
TIMESTAMP	u64	8 bytes (Unix ms)

Implementation Components

crates/synor-database/src/sql/
├── mod.rs           # Module exports
├── parser.rs        # SQL parsing (sqlparser-rs)
├── planner.rs       # Query planning & optimization
├── executor.rs      # Query execution engine
├── table.rs         # Table definition & storage
├── row.rs           # Row representation
├── types.rs         # SQL type system
├── transaction.rs   # ACID transactions
└── index.rs         # SQL-specific indexing

API Design

// Table definition
pub struct TableDef {
    pub name: String,
    pub columns: Vec<ColumnDef>,
    pub primary_key: Option<String>,
    pub indexes: Vec<IndexDef>,
}

pub struct ColumnDef {
    pub name: String,
    pub data_type: SqlType,
    pub nullable: bool,
    pub default: Option<SqlValue>,
}

// SQL execution
pub struct SqlEngine {
    tables: HashMap<String, Table>,
    transaction_log: TransactionLog,
}

impl SqlEngine {
    pub fn execute(&mut self, sql: &str) -> Result<SqlResult, SqlError>;
    pub fn begin_transaction(&mut self) -> TransactionId;
    pub fn commit(&mut self, txn: TransactionId) -> Result<(), SqlError>;
    pub fn rollback(&mut self, txn: TransactionId);
}

Gateway Endpoints

Endpoint	Method	Description
/db/:db/sql	POST	Execute SQL query
/db/:db/sql/tables	GET	List tables
/db/:db/sql/tables/:table	GET	Get table schema
/db/:db/sql/tables/:table	DELETE	Drop table

---

Feature 2: Graph Store

Purpose

Enable relationship-based queries for social networks, knowledge graphs, recommendation engines, and any connected data.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     GRAPH QUERY LAYER                        │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │  Graph Query │  │   Traversal  │  │   Path Finding   │   │
│  │   Parser     │  │    Engine    │  │   (Dijkstra)     │   │
│  └──────────────┘  └──────────────┘  └──────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │                Graph Storage Engine                   │   │
│  │  - Adjacency list storage                             │   │
│  │  - Edge index (source, target, type)                  │   │
│  │  - Property storage                                   │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Data Model

Node (Vertex):
  - id: NodeId (32 bytes)
  - labels: Vec<String>
  - properties: JsonValue

Edge (Relationship):
  - id: EdgeId (32 bytes)
  - source: NodeId
  - target: NodeId
  - edge_type: String
  - properties: JsonValue
  - directed: bool

Query Language (Simplified Cypher-like)

// Find all friends of user Alice
MATCH (a:User {name: "Alice"})-[:FRIEND]->(friend)
RETURN friend

// Find shortest path between two nodes
MATCH path = shortestPath((a:User {id: "123"})-[*]-(b:User {id: "456"}))
RETURN path

// Find mutual friends
MATCH (a:User {name: "Alice"})-[:FRIEND]->(mutual)<-[:FRIEND]-(b:User {name: "Bob"})
RETURN mutual

Implementation Components

crates/synor-database/src/graph/
├── mod.rs           # Module exports
├── node.rs          # Node definition & storage
├── edge.rs          # Edge definition & storage
├── store.rs         # Graph storage engine
├── query.rs         # Query language parser
├── traversal.rs     # Graph traversal algorithms
├── path.rs          # Path finding (BFS, DFS, Dijkstra)
└── index.rs         # Graph-specific indexes

API Design

pub struct Node {
    pub id: NodeId,
    pub labels: Vec<String>,
    pub properties: JsonValue,
}

pub struct Edge {
    pub id: EdgeId,
    pub source: NodeId,
    pub target: NodeId,
    pub edge_type: String,
    pub properties: JsonValue,
}

pub struct GraphStore {
    nodes: HashMap<NodeId, Node>,
    edges: HashMap<EdgeId, Edge>,
    adjacency: HashMap<NodeId, Vec<EdgeId>>,  // outgoing
    reverse_adj: HashMap<NodeId, Vec<EdgeId>>, // incoming
}

impl GraphStore {
    // Node operations
    pub fn create_node(&mut self, labels: Vec<String>, props: JsonValue) -> NodeId;
    pub fn get_node(&self, id: &NodeId) -> Option<&Node>;
    pub fn update_node(&mut self, id: &NodeId, props: JsonValue) -> Result<(), GraphError>;
    pub fn delete_node(&mut self, id: &NodeId) -> Result<(), GraphError>;

    // Edge operations
    pub fn create_edge(&mut self, source: NodeId, target: NodeId, edge_type: &str, props: JsonValue) -> EdgeId;
    pub fn get_edge(&self, id: &EdgeId) -> Option<&Edge>;
    pub fn delete_edge(&mut self, id: &EdgeId) -> Result<(), GraphError>;

    // Traversal
    pub fn neighbors(&self, id: &NodeId, direction: Direction) -> Vec<&Node>;
    pub fn edges_of(&self, id: &NodeId, direction: Direction) -> Vec<&Edge>;
    pub fn shortest_path(&self, from: &NodeId, to: &NodeId) -> Option<Vec<NodeId>>;
    pub fn traverse(&self, start: &NodeId, query: &TraversalQuery) -> Vec<TraversalResult>;
}

Gateway Endpoints

Endpoint	Method	Description
/db/:db/graph/nodes	POST	Create node
/db/:db/graph/nodes/:id	GET	Get node
/db/:db/graph/nodes/:id	PUT	Update node
/db/:db/graph/nodes/:id	DELETE	Delete node
/db/:db/graph/edges	POST	Create edge
/db/:db/graph/edges/:id	GET	Get edge
/db/:db/graph/edges/:id	DELETE	Delete edge
/db/:db/graph/query	POST	Execute graph query
/db/:db/graph/path	POST	Find shortest path
/db/:db/graph/traverse	POST	Traverse from node

---

Feature 3: Replication (Raft Consensus)

Purpose

Provide high availability and fault tolerance through distributed consensus, ensuring data consistency across multiple nodes.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    RAFT CONSENSUS LAYER                      │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │   Leader     │  │  Follower    │  │   Candidate      │   │
│  │   Election   │  │  Replication │  │   (Election)     │   │
│  └──────────────┘  └──────────────┘  └──────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │                    Log Replication                    │   │
│  │  - Append entries                                     │   │
│  │  - Commit index                                       │   │
│  │  - Log compaction (snapshots)                         │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │                  State Machine                        │   │
│  │  - Apply committed entries                            │   │
│  │  - Database operations                                │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Raft Protocol Overview

Leader Election:
1. Followers timeout → become Candidate
2. Candidate requests votes from peers
3. Majority votes → become Leader
4. Leader sends heartbeats to maintain authority

Log Replication:
1. Client sends write to Leader
2. Leader appends to local log
3. Leader replicates to Followers
4. Majority acknowledge → entry committed
5. Leader applies to state machine
6. Leader responds to client

Implementation Components

crates/synor-database/src/replication/
├── mod.rs           # Module exports
├── raft.rs          # Core Raft implementation
├── state.rs         # Node state (Leader/Follower/Candidate)
├── log.rs           # Replicated log
├── rpc.rs           # RPC messages (AppendEntries, RequestVote)
├── election.rs      # Leader election logic
├── snapshot.rs      # Log compaction & snapshots
├── cluster.rs       # Cluster membership
└── client.rs        # Client for forwarding to leader

API Design

#[derive(Clone, Copy, PartialEq)]
pub enum NodeRole {
    Leader,
    Follower,
    Candidate,
}

pub struct RaftConfig {
    pub node_id: u64,
    pub peers: Vec<PeerAddress>,
    pub election_timeout_ms: (u64, u64),  // min, max
    pub heartbeat_interval_ms: u64,
    pub snapshot_threshold: u64,
}

pub struct LogEntry {
    pub term: u64,
    pub index: u64,
    pub command: Command,
}

pub enum Command {
    // Database operations
    KvSet { key: String, value: Vec<u8> },
    KvDelete { key: String },
    DocInsert { collection: String, doc: JsonValue },
    DocUpdate { collection: String, id: DocumentId, update: JsonValue },
    DocDelete { collection: String, id: DocumentId },
    // ... other operations
}

pub struct RaftNode {
    config: RaftConfig,
    state: RaftState,
    log: ReplicatedLog,
    state_machine: Arc<Database>,
}

impl RaftNode {
    pub async fn start(&mut self) -> Result<(), RaftError>;
    pub async fn propose(&self, command: Command) -> Result<(), RaftError>;
    pub fn is_leader(&self) -> bool;
    pub fn leader_id(&self) -> Option<u64>;
    pub fn status(&self) -> ClusterStatus;
}

// RPC Messages
pub struct AppendEntries {
    pub term: u64,
    pub leader_id: u64,
    pub prev_log_index: u64,
    pub prev_log_term: u64,
    pub entries: Vec<LogEntry>,
    pub leader_commit: u64,
}

pub struct RequestVote {
    pub term: u64,
    pub candidate_id: u64,
    pub last_log_index: u64,
    pub last_log_term: u64,
}

Cluster Configuration

# docker-compose.raft.yml
services:
  db-node-1:
    image: synor/database:latest
    environment:
      RAFT_NODE_ID: 1
      RAFT_PEERS: "db-node-2:5000,db-node-3:5000"
      RAFT_ELECTION_TIMEOUT: "150-300"
      RAFT_HEARTBEAT_MS: 50
    ports:
      - "8484:8484"  # HTTP API
      - "5000:5000"  # Raft RPC

  db-node-2:
    image: synor/database:latest
    environment:
      RAFT_NODE_ID: 2
      RAFT_PEERS: "db-node-1:5000,db-node-3:5000"
    ports:
      - "8485:8484"
      - "5001:5000"

  db-node-3:
    image: synor/database:latest
    environment:
      RAFT_NODE_ID: 3
      RAFT_PEERS: "db-node-1:5000,db-node-2:5000"
    ports:
      - "8486:8484"
      - "5002:5000"

Gateway Endpoints

Endpoint	Method	Description
/cluster/status	GET	Get cluster status
/cluster/leader	GET	Get current leader
/cluster/nodes	GET	List all nodes
/cluster/nodes/:id	DELETE	Remove node from cluster
/cluster/nodes	POST	Add node to cluster

---

Implementation Order

Step 1: SQL Store

1. Add sqlparser dependency
2. Implement type system (types.rs)
3. Implement row storage (row.rs, table.rs)
4. Implement SQL parser wrapper (parser.rs)
5. Implement query planner (planner.rs)
6. Implement query executor (executor.rs)
7. Add transaction support (transaction.rs)
8. Add gateway endpoints
9. Write tests

Step 2: Graph Store

1. Implement node/edge types (node.rs, edge.rs)
2. Implement graph storage (store.rs)
3. Implement query parser (query.rs)
4. Implement traversal algorithms (traversal.rs, path.rs)
5. Add graph indexes (index.rs)
6. Add gateway endpoints
7. Write tests

Step 3: Replication

1. Implement Raft state machine (state.rs, raft.rs)
2. Implement replicated log (log.rs)
3. Implement RPC layer (rpc.rs)
4. Implement leader election (election.rs)
5. Implement log compaction (snapshot.rs)
6. Implement cluster management (cluster.rs)
7. Integrate with database operations
8. Add gateway endpoints
9. Write tests
10. Create Docker Compose for cluster

---

Pricing Impact

Feature	Operation	Cost (SYNOR)
SQL	Query/million	0.02
SQL	Write/million	0.05
Graph	Traversal/million	0.03
Graph	Path query/million	0.05
Replication	Included	Base storage cost

---

Success Criteria

SQL Store

• Parse and execute basic SELECT, INSERT, UPDATE, DELETE
• Support WHERE clauses with operators
• Support ORDER BY, LIMIT, OFFSET
• Support simple JOINs
• Support aggregate functions
• ACID transactions

Graph Store

• Create/read/update/delete nodes and edges
• Traverse neighbors (in/out/both)
• Find shortest path between nodes
• Execute pattern matching queries
• Support property filters

Replication

• Leader election works correctly
• Log replication achieves consensus
• Reads from any node (eventual consistency)
• Writes only through leader
• Node failure handled gracefully
• Log compaction reduces storage