The Self-Load Manager is an autonomous background subsystem of ShannonBase’s Rapid Engine. Its primary responsibility is to continuously and automatically decide which InnoDB tables should be loaded into the secondary execution engine (Rapid / IMCS) and which should be evicted.

These decisions are driven by a combination of runtime signals, including:

• Observed query access patterns
• Actual execution location (Primary MySQL vs. Secondary Rapid)
• Table size and global memory constraints
• Temporal decay of historical importance
• System quietness and change-propagation safety

The overall goal is to maximize query acceleration while strictly respecting memory limits, without requiring explicit user configuration. Conceptually, the Self-Load Manager behaves like an adaptive cache combined with a cost-based admission controller for IMCS tables.

Core Components

SelfLoadManager

The SelfLoadManager is implemented as a singleton and serves as the global coordinator for all autonomous load and unload activities. Its responsibilities include:

• Discovering eligible InnoDB tables from MySQL system catalogs
• Maintaining per-table metadata and runtime statistics
• Running a background coordination and evaluation thread
• Issuing load and unload decisions to the Rapid Engine

Key global state maintained by the manager includes:

• m_rpd_mirror_tables: mirror metadata for all eligible tables
• m_schema_tables: mapping from schema ID to schema name
• m_table_stats: estimated memory footprint for each table
• m_worker_state: lifecycle and control state of the background worker

TableInfo and TableAccessStats

Every table tracked by the Autopilot system is represented by a TableInfo structure. This structure captures both static properties and dynamic behavior.

• Schema name and table name
• Estimated in-memory footprint
• Partitioning and layout status
• Eligibility for secondary engine execution
• Associated runtime statistics (TableAccessStats)

Tracked runtime statistics include:

• Access counters (Primary MySQL vs. Rapid)
• Timestamp of the most recent queries
• An atomic importance score with time-based decay
• Current load state (LOADED, NOT_LOADED, INSUFFICIENT_MEMORY)
• Load origin (explicit USER request vs. autonomous SELF decision)

Summary

The Autopilot Self-Load system transforms the Rapid Engine into a truly self-optimizing secondary execution layer. By continuously learning from real query behavior, it dynamically reshapes in-memory data placement to match the active workload.

This architecture delivers three key benefits:

• Hands-free performance acceleration
• Predictable and bounded memory usage
• Graceful adaptation to workload shifts over time

In essence, the Self-Load Manager brings database-native intelligence to table placement decisions, eliminating the need for DBAs to manually micromanage which tables should reside in memory.