Split Module

• Six chunking strategies: recursive, semantic, entity-aware, relation-aware, sliding window, structural
• SemanticChunker uses embedding-based topic-shift detection to split only when content changes
• EntityAwareChunker keeps entity mentions intact across chunk boundaries
• RelationAwareChunker keeps subject-predicate-object triplets within a single chunk
• Chunking quality directly determines downstream embedding accuracy and entity extraction quality

​

Why Chunking Matters

• An entity mention like “Apple Inc.” split across two chunks loses its context in both
• A relation triplet like “Steve Jobs founded Apple” split at “Steve Jobs” leaves a dangling subject
• Embedding a chunk that mixes two unrelated topics produces a centroid vector that matches neither

​

Exported Classes

​

What You Get

• TextSplitter — Unified interface for 11 chunking strategies: swap methods without changing downstream code.
• Semantic Chunking — Embedding-based topic shift detection: splits only when the topic actually changes.
• Entity-Aware Chunking — Entity spans never cross chunk boundaries: guaranteed by boundary adjustment.
• Relation-Aware Chunking — Subject–predicate–object triplets kept within a single chunk for KG pipelines.
• Chunk Object — Output dataclass with text, character offsets, optional id, and method-specific metadata.

​

Quick Start

from semantica.split import TextSplitter

splitter = TextSplitter(
    method="recursive",   # see Splitting Methods table
    chunk_size=1000,
    chunk_overlap=200,
)

chunks = splitter.split(text)

for chunk in chunks:
    print(f"  Start: {chunk.start_index}, End: {chunk.end_index}")
    print(f"  Method: {chunk.metadata.get('method')}")
    print(f"  Preview: {chunk.text[:80]}...")

# split_documents() accepts any object with a .text attribute,
# or a plain string: no specific document class required.
class Doc:
    def __init__(self, text, metadata=None):
        self.text = text
        self.metadata = metadata or {}

doc = Doc(text="Annual report content...", metadata={"source": "annual_report.pdf"})

splitter = TextSplitter(method="structural")
chunks   = splitter.split_documents([doc])

for chunk in chunks:
    print(f"  {chunk.text[:80]}...")

# split_documents() returns a flat List[Chunk] across all inputs
all_chunks = splitter.split_documents(docs)

for chunk in all_chunks:
    print(chunk.text[:80])

​

Splitting Methods

​

Choosing a Strategy

• Building a KG? → relation_aware (keeps triplets intact), then entity_aware for pure NER
• RAG system where retrieval quality matters most? → semantic_transformer
• Dense overlap for bi-encoder retrieval (ColBERT, DPR)? → sliding_window
• Preparing prompts for a fixed-window LLM? → token
• Structured text with headings? → structural
• Paragraph-level coherence? → paragraph or sentence
• Fast splitting with no NLP overhead? → recursive or character

​

TextSplitter Constructor

from semantica.split import TextSplitter

splitter = TextSplitter(
    method="semantic_transformer",   # chunking strategy: see Splitting Methods table
    chunk_size=1000,                 # target size in characters
    chunk_overlap=200,               # character overlap between adjacent chunks
    similarity_threshold=0.7,        # cosine similarity cutoff (semantic_transformer only)
    model="all-MiniLM-L6-v2",        # sentence-transformers model name (semantic_transformer only)
    ner_method="ml",                 # NER method (entity_aware only)
    relation_method="ml",            # relation extraction method (relation_aware only)
)

​

Splitting Method Details

splitter = TextSplitter(method="recursive", chunk_size=1000, chunk_overlap=200)
chunks   = splitter.split(text)

from semantica.split import TextSplitter

splitter = TextSplitter(
    method="semantic_transformer",
    model="all-MiniLM-L6-v2",      # any sentence-transformers model name
    similarity_threshold=0.7,       # 0.6 = more splits, 0.8 = fewer splits
    chunk_size=800,
    chunk_overlap=0,                # not needed: chunks are already coherent
)
chunks = splitter.split(text)

from semantica.split import TextSplitter
import os

# ner_method is passed through to the internal NERExtractor.
# Use "llm" for highest accuracy, "ml" (default) for speed.
splitter = TextSplitter(
    method="entity_aware",
    chunk_size=512,
    chunk_overlap=50,
    ner_method="ml",   # "pattern" | "regex" | "ml" | "huggingface" | "llm"
)
chunks = splitter.split(text)

for chunk in chunks:
    print(f"  entities in chunk: {chunk.metadata.get('entity_count', 0)}")
    print(f"  preview: {chunk.text[:80]}...")

from semantica.split import TextSplitter

# relation_method is passed through to the internal RelationExtractor.
# Use "llm" for highest accuracy, "ml" (default) for speed.
splitter = TextSplitter(
    method="relation_aware",
    chunk_size=512,
    relation_method="ml",   # "ml" | "llm" | "huggingface"
)
chunks = splitter.split(text)

for chunk in chunks:
    print(f"  relations in chunk: {chunk.metadata.get('relation_count', 0)}")
    for rel in chunk.metadata.get("relationships", []):
        print(f"  {rel}")

from semantica.split import TextSplitter

splitter = TextSplitter(method="structural")
chunks   = splitter.split(text)

for chunk in chunks:
    print(f"  {chunk.text[:80]}...")
    print(f"  start: {chunk.start_index}, end: {chunk.end_index}")

​

Chunk Schema

@dataclass
class Chunk:
    text:        str                    # the chunk's text content
    start_index: int                    # character offset of start in source text
    end_index:   int                    # character offset of end in source text
    metadata:    Dict[str, Any]         # method-specific fields: see table below
    id:          Optional[str] = None   # optional chunk identifier

​

Tokenizer Options

​

Pipeline Integration

from semantica.parse import DocumentParser
from semantica.split import TextSplitter
from semantica.semantic_extract import NERExtractor

# Parse
parser = DocumentParser()
parsed = parser.parse("data/report.pdf")   # returns a dict with "full_text" key

# Split
splitter = TextSplitter(method="semantic_transformer", chunk_size=512)
chunks   = splitter.split(parsed["full_text"])

# Extract from each chunk
ner = NERExtractor(method="ml")

for chunk in chunks:
    entities = ner.extract(chunk.text)
    print(f"  {len(entities)} entities in chunk starting at {chunk.start_index}")

• Parse — Parse documents before chunking: produces sections and metadata.
• Embeddings — Embed chunks for vector search and semantic chunking.
• Semantic Extract — Extract entities and relations from individual chunks.
• Pipeline — Integrate splitting as a named pipeline step.