CAPYSQUASH

---

CAPYSQUASH documentation page.

---

Title: 'PostgreSQL Internals: The Science Behind Migration Optimization' Description: 'Deep dive into pg_query_go parser, dependency resolution algorithms, and the computer science that powers pgsquash'

PostgreSQL Internals: The Science Behind Migration Optimization

Explore the computer science and PostgreSQL internals that make pgsquash the world's most accurate migration optimization engine.

Unlike regex-based tools that guess at SQL structure, pgsquash leverages PostgreSQL's actual parser (pg_query_go) and advanced computer science algorithms to achieve mathematical precision in migration consolidation.

THE PARSING REVOLUTION

WHY POSTGRESQL'S PARSER MATTERS

TRADITIONAL APPROACH

Custom regex parsers

Edge case failures

Manual syntax updates

80-90% accuracy

Breaks on new features

  • PGSQUASH APPROACH

  • PostgreSQL's actual C parser

  • Battle-tested for decades

  • Automatic syntax support

  • 100% accuracy

  • Future-proof by design

PG_QUERY_GO: THE MAGIC BULLET

What is pg_query_go?

Pg_query_go is a Go wrapper around PostgreSQL's libpg_query - the exact same parser that PostgreSQL uses internally.

TECHNICAL ARCHITECTURE

PostgreSQL C Parser

Decades of production use, handles every edge case

  • libpg_query Library

  • Exposes parser as standalone C library

  • Go CGO Bindings

  • Safe Go wrapper around C parser

  • capysquash Integration

  • Seamless integration with consolidation engine

Parser Accuracy Demonstration

REAL-WORLD ACCURACY EXAMPLES

COMPLEX TYPE DEFINITIONS

-- Custom composite type
CREATE TYPE address AS (
    street TEXT,
    city TEXT,
    zip_code TEXT,
    country TEXT DEFAULT 'US'
);

-- Table using the type
CREATE TABLE users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
address address,
created_at TIMESTAMPTZ DEFAULT NOW()
);
  • ☑ pg_query_go parses both the type definition and usage perfectly

  • ADVANCED FUNCTION SIGNATURES

-- Complex function with polymorphic types
CREATE OR REPLACE FUNCTION process_data(
    input_data ANYARRAY,
    config JSONB DEFAULT '{}',
    debug_mode BOOLEAN DEFAULT FALSE
) RETURNS TABLE (
    processed_items BIGINT[],
    metadata JSONB,
    processing_time INTERVAL
) AS $$
BEGIN
    -- Function implementation
END;
$$ LANGUAGE plpgsql SECURITY DEFINER;
  • ☑ Handles polymorphic types, defaults, and complex return types

  • PARTIAL INDEXES WITH EXPRESSIONS

-- Complex partial index
CREATE INDEX idx_active_users_email
ON users (LOWER(email))
WHERE status = 'active'
AND last_login > NOW() - INTERVAL '30 days'
AND deleted_at IS NULL;
  • ☑ Correctly parses expressions, WHERE clauses, and function calls

ABSTRACT SYNTAX TREES (AST)

Understanding Parse Trees

Every SQL statement becomes a structured tree that capysquash can analyze programmatically.

SIMPLE EXAMPLE: CREATE TABLE

SQL INPUT

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT NOW()
);
  • AST STRUCTURE

  • CreateStmt

  • ├── relation: "users"

  • ├── columns:

  • │ ├── ColumnDef

  • │ │ ├── colname: "id"

  • │ │ ├── typeName: SERIAL

  • │ │ └── constraints:

  • │ │ └── Constraint (PRIMARY KEY)

  • │ ├── ColumnDef

  • │ │ ├── colname: "email"

  • │ │ ├── typeName: VARCHAR(255)

  • │ │ └── constraints:

  • │ │ ├── Constraint (UNIQUE)

  • │ │ └── Constraint (NOT NULL)

  • │ └── ColumnDef

  • │ ├── colname: "created_at"

  • │ ├── typeName: TIMESTAMP

  • │ └── constraints:

  • │ └── Constraint (DEFAULT NOW())

  • └── options: []

AST Analysis Capabilities

OBJECT EXTRACTION Identify all tables, columns, indexes, functions

DEPENDENCY MAPPING Track relationships between objects

  • CHANGE DETECTION
  • Compare ASTs to identify modifications

PARSER FEATURES

Supported PostgreSQL Features

COMPREHENSIVE SQL COVERAGE

DDL OPERATIONS

► CREATE/ALTER/DROP TABLE

► CREATE/ALTER/DROP INDEX

► CREATE/ALTER/DROP VIEW

► CREATE/ALTER/DROP FUNCTION

► CREATE/ALTER/DROP TRIGGER

► CREATE/ALTER/DROP SEQUENCE

► CREATE/ALTER/DROP TYPE

► CREATE/ALTER/DROP SCHEMA

► CREATE EXTENSION

► COMMENT ON OBJECTS

  • ADVANCED FEATURES

  • ► Row Level Security (RLS) Policies

  • ► Partial and Expression Indexes

  • ► Generated Columns

  • ► Table Partitioning

  • ► Inheritance

  • ► Arrays and JSONB

  • ► Window Functions

  • ► CTEs and Recursive Queries

  • ► Stored Procedures

  • ► Foreign Data Wrappers

Version Compatibility

POSTGRESQL VERSION SUPPORT

  • PostgreSQL 12+ - Full support

  • PostgreSQL 14+ - Enhanced features

  • PostgreSQL 16+ - Latest syntax

  • PostgreSQL 17 - In development

ERROR HANDLING & RECOVERY

Parse Error Intelligence

SMART ERROR ANALYSIS

PRECISE ERROR LOCATION

-- Error example
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) UNIQUE,
    created_at TIMESTAMP DEFAULT NOW()
    -- Missing comma here
    name TEXT NOT NULL
);
  • Parser Output:

  • Error at line 5, column 5: syntax error at or near "name"

  • CONTEXTUAL SUGGESTIONS

  • Common Fixes:

  • ► Add missing comma

  • ► Check bracket matching

  • ► Verify keyword spelling

  • ► Complete statement syntax

  • capysquash Actions:

  • ► Skip problematic statements

  • ► Continue with valid SQL

  • ► Report detailed errors

  • ► Suggest corrections

PERFORMANCE OPTIMIZATION

Parser Performance

LIGHTNING FAST PARSING

Benchmarks (10MB SQL file):

► Parse time: ~200ms

► Memory usage: ~50MB

► Statements processed: 1,247

► Accuracy: 100%

  • Optimization Techniques:

  • ► Streaming parser for large files

  • ► AST node caching

  • ► Parallel statement processing

  • ► Memory-efficient data structures

Memory Management

// Efficient parsing with memory management
func ParseMigrations(files []string) ([]*ParseTree, error) {
    parser := pg_query.NewParser()
    defer parser.Cleanup() // Free C memory

    var trees []*ParseTree

    for _, file := range files {
        content, err := os.ReadFile(file)
        if err != nil {
            return nil, err
        }

        // Parse with automatic memory management
        tree, err := parser.Parse(string(content))
        if err != nil {
            return nil, fmt.Errorf("parse error in %s: %w", file, err)
        }

        trees = append(trees, tree)
    }

    return trees, nil

}

    </div>

    ## INTEGRATION EXAMPLES

    ### Real-World Parsing Scenarios

    <div className="space-y-6 my-8">
    <div className="brutalist-card p-6">
    <h4 className="font-black uppercase mb-3">SUPABASE MIGRATIONS</h4>
    <div className="brutalist-code text-sm">
    ```sql

\ - Supabase auth schema.

CREATE TABLE IF NOT EXISTS auth.users (
id UUID NOT NULL PRIMARY KEY,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
email TEXT,
phone TEXT,
raw_user_meta_data JSONB DEFAULT '',
email_confirmed_at TIMESTAMPTZ,
phone_confirmed_at TIMESTAMPTZ
);

\ - Row Level Security.

ALTER TABLE auth.users ENABLE ROW LEVEL SECURITY;

\ - RLS Policy.

CREATE POLICY "Users can view their own profile"

ON auth.users FOR SELECT

USING (auth.uid() = id);



- ☑ Correctly parses auth schema, RLS, and policy functions





- COMPLEX DATABASE MIGRATION



```sql
-- Multi-step schema evolution
CREATE TYPE user_role AS ENUM ('admin', 'user', 'moderator');

CREATE TABLE users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
email TEXT UNIQUE NOT NULL,
role user_role DEFAULT 'user',
settings JSONB DEFAULT '{}',
created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Generated column
ALTER TABLE users
ADD COLUMN email_domain TEXT
GENERATED ALWAYS AS (split_part(email, '@', 2)) STORED;

-- Partial index
CREATE INDEX idx_users_admin
ON users (email)
WHERE role = 'admin';

-- Trigger function
CREATE FUNCTION update_updated_at()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = NOW();
RETURN NEW;
END;

$$
LANGUAGE plpgsql;

-- Trigger
CREATE TRIGGER users_updated_at
BEFORE UPDATE ON users
FOR EACH ROW EXECUTE FUNCTION update_updated_at();
  • ☑ Handles enums, generated columns, partial indexes, and triggers

ADVANCED TECHNIQUES

Custom Analysis Rules

BUILDING CUSTOM ANALYSIS

The AST enables sophisticated analysis beyond simple parsing:

Dependency Extraction

Automatically build dependency graphs from AST relationships

  • Pattern Recognition

  • Identify common patterns like audit columns, soft deletes

  • Semantic Analysis

  • Understand the meaning and intent of SQL operations

Integration with pgsquash-engine

// How pgsquash uses the parser
type MigrationAnalyzer struct {
    parser *pg_query.Parser
    tracker *ObjectTracker
}

func (ma \*MigrationAnalyzer) AnalyzeMigration(sql string) (\*AnalysisResult, error) {
// Parse SQL into AST
tree, err := ma.parser.Parse(sql)
if err != nil {
return nil, err
}

    // Extract objects and relationships
    objects := ma.extractObjects(tree)
    dependencies := ma.buildDependencyGraph(objects)

    // Track object lifecycle
    ma.tracker.UpdateObjects(objects)

    // Generate consolidation opportunities
    opportunities := ma.findConsolidationOpportunities(dependencies)

    return &AnalysisResult{
        Objects: objects,
        Dependencies: dependencies,
        Opportunities: opportunities,
    }, nil

}

    </div>

    ## TROUBLESHOOTING

    ### Common Parser Issues

    <div className="brutalist-card-lg section-accent-warning p-8 my-8">
    <h3 className="text-2xl font-black uppercase mb-6">PARSER TROUBLESHOOTING</h3>

    <div className="space-y-6">
    <div>
    <h4 className="font-black uppercase mb-3">ENCODING ISSUES</h4>
    <div className="brutalist-card-sm p-4">
    <p className="font-bold text-sm mb-2">Problem:</p>
    <p className="text-xs font-bold">Non-UTF8 characters causing parse failures</p>
    <p className="font-bold text-sm mb-2 mt-2">Solution:</p>
    <p className="text-xs font-bold">Ensure all migration files are UTF-8 encoded</p>
    </div>
    - </div>

    - <div>
    - <h4 className="font-black uppercase mb-3">VERSION INCOMPATIBILITY</h4>
    - <div className="brutalist-card-sm p-4">
    - <p className="font-bold text-sm mb-2">Problem:</p>
    - <p className="text-xs font-bold">Using PostgreSQL 17 features with older parser</p>
    - <p className="font-bold text-sm mb-2 mt-2">Solution:</p>
    - <p className="text-xs font-bold">Update capysquash to latest version for newest syntax support</p>
    - </div>
    - </div>

    - <div>
    - <h4 className="font-black uppercase mb-3">MEMORY LIMITS</h4>
    - <div className="brutalist-card-sm p-4">
    - <p className="font-bold text-sm mb-2">Problem:</p>
    - <p className="text-xs font-bold">Large migration files causing memory issues</p>
    - <p className="font-bold text-sm mb-2 mt-2">Solution:</p>
    - <p className="text-xs font-bold">Use streaming mode for files > 10MB</p>
    - </div>
    - </div>
    - </div>
    </div>

    ## NEXT STEPS

    - [Dependency Resolution Algorithms](/docs/advanced/dependency-resolution-algorithms) - How we build dependency graphs
    - [Migration Optimization Strategies](/docs/advanced/migration-optimization-strategies) - Advanced consolidation techniques
    - [SQL Parsing Internals](/docs/advanced/sql-parsing) - Deep dive into parsing implementation
    - [Architecture Deep Dive](/docs/advanced/architecture-deep-dive) - Complete system overview

    ---

    <div className="brutalist-card-lg category-announcement p-8 my-8">
    <h3 className="text-2xl font-black uppercase mb-4">KEY TAKEAWAY</h3>
    <p className="font-bold text-lg">
    By using PostgreSQL's actual parser, pgsquash achieves <strong>100% accuracy</strong> and <strong>future-proof compatibility</strong> - no custom parser can match this level of reliability.
    </p>
    </div>
    $$.

How is this guide?

On this page