Four-Layer Architecture
- Layer 1: Ingestion
- Layer 2: Processing
- Layer 3: Intelligence
- Layer 4: Application
Loads data from any source into the pipeline as a unified
SourceDocument.Data Flow
Every pipeline follows the same linear path from raw source to delivered output:Module Map
Extension Points
Every layer exposes a registry-based extension point. Register custom implementations and they participate in the full pipeline with zero changes to core code.Design Decisions
Modularity: use only what you need
Modularity: use only what you need
Every component works standalone.
NERExtractor runs without a graph store. VectorStore runs without decision tracking. The framework never forces a full stack instantiation; you pay only for what you import.Pluggability: extend without modifying core
Pluggability: extend without modifying core
Custom ingestors, extractors, validators, and exporters follow the same base class pattern. Register them via
PluginRegistry and they participate in the full pipeline (provenance tracking, retry policies, and parallel execution included) with no changes to core code.Provenance by default
Provenance by default
Lineage tracking is built into graph construction at the lowest level. Every node and edge carries a
source_id pointing back to the originating document, extraction method, and timestamp. There is no opt-in required; provenance is always on.Configuration over convention
Configuration over convention
Centralized
ConfigManager with environment variable overrides. No magic defaults; all behavior is explicit and overridable. Suitable for multi-environment deployments where dev, staging, and production need different backends.Performance Characteristics
- Modules: full module documentation with code examples.
- Learning More: configuration reference, performance guide, and troubleshooting.
- Pipeline Reference: pipeline orchestration, workers, and retry policies.
- Core Reference: framework lifecycle, plugin registry, and configuration.
