Neuro-Symbolic Optimization of SQL Execution Plans in Distributed Data Fabrics

Abstract

This proliferation of distributed data fabrics (DDFs) has revealed severe deficiencies in traditional rule-based SQL execution plan optimizers with correlated multi-join workloads, which have cardinality estimation errors in excess of 104x under these conditions. This paper proposes a novel near-optimal SQL execution plan generation method called neuro-symbolic optimization (NSO) that combines deep reinforcement learning (DRL), graph neural networks (GNNs), and symbolic constraint reasoning, to generate near-optimal plans across heterogeneous, distributed storage nodes. The NSO framework performs 67% better on the Join Order Benchmark (JOB) than against four other base systems: PostgreSQL, Neo, ReJOIN and SkinnerDB, with a cardinality estimation accuracy of 95% and a plan quality score of 92 out of 100. Convergence occurs in 400 episodes of training, which is 200 episodes fewer than competing learned approaches. The experiments of scalability (2 to 64 nodes) show speedup ranging from 1.2x to 11.2x, and demonstrate the framework's suitability for large-scale deployment of DDF. Knowledge graph embeddings are used to represent the relationship between different schemas, and Tree-LSTM structures are used to represent the hierarchical plan representation. The findings highlight the promising ability of a neuro-symbolic architecture to overcome the well-known accuracy–generalization dichotomy that has limited the capabilities of purely neural query optimizers.