Four-Layer Architecture
- Layer 1: Ingestion
- Layer 2: Processing
- Layer 3: Intelligence
- Layer 4: Application
Loads data from any source into the pipeline as a unified
SourceDocument.Data Flow
Every pipeline follows the same linear path from raw source to delivered output:Module Map
Extension Points
Every layer exposes a registry-based extension point. Register custom implementations and they participate in the full pipeline with zero changes to core code.Design Decisions
Modularity: use only what you need
Modularity: use only what you need
Every component works standalone.
NERExtractor runs without a graph store. VectorStore runs without decision tracking. The framework never forces a full stack instantiation: you pay only for what you import.Pluggability: extend without modifying core
Pluggability: extend without modifying core
Custom ingestors, extractors, validators, and exporters follow the same base class pattern. Register them via
PluginRegistry and they participate in the full pipeline: provenance tracking, retry policies, and parallel execution included: with no changes to core code.Provenance by default
Provenance by default
Lineage tracking is built into graph construction at the lowest level. Every node and edge carries a
source_id pointing back to the originating document, extraction method, and timestamp. There’s no opt-in required: provenance is always on.Configuration over convention
Configuration over convention
Centralized
ConfigManager with environment variable overrides. No magic defaults: all behavior is explicit and overridable. Suitable for multi-environment deployments where dev, staging, and production need different backends.Performance Characteristics
- Modules — Full module documentation with code examples.
- Learning More — Configuration reference, performance guide, and troubleshooting.
- Pipeline Reference — Pipeline orchestration, workers, and retry policies.
- Core Reference — Framework lifecycle, plugin registry, and configuration.
