Normalize entity names before deduplication. Canonical forms such as "Apple Inc." vs "apple inc" may score below threshold due to case alone. Run EntityNormalizer or TextNormalizer first for reliable matching.
from semantica.deduplication import DuplicateDetectordetector = DuplicateDetector( similarity_threshold=0.7, # default 0.7: minimum score to include a candidate confidence_threshold=0.6, # default 0.6: minimum confidence to include a candidate max_results=100, # optional: hard cap on total candidates returned top_k_per_entity=3, # optional: max candidates per entity min_similarity=0.75, # optional: additional floor applied after sorting sort_by="confidence", # "confidence" (default) | "similarity_score")# Returns List[DuplicateCandidate]candidates = detector.detect_duplicates(entities)for c in candidates: print(c.entity1.get("name"), "vs", c.entity2.get("name")) print(" similarity_score:", c.similarity_score) print(" confidence: ", c.confidence) print(" reasons: ", c.reasons)# Detect duplicate groups with union-find (returns List[DuplicateGroup])groups = detector.detect_duplicate_groups(entities)for g in groups: print("Group of {}: confidence: {:.2f}: representative: {}".format( len(g.entities), g.confidence, g.representative and g.representative.get("name") ))# Incremental: compare new entities against existing (returns List[DuplicateCandidate])new_entities = [{"id": "4", "name": "Apple Corp.", "type": "Company"}]candidates = detector.incremental_detect(new_entities, entities)
Tune similarity_threshold before confidence_threshold. The similarity threshold gates which entity pairs are even considered. The confidence threshold further filters those pairs based on multi-factor scoring. Start with similarity_threshold=0.7 and raise it to reduce false positives.
Use detect_duplicate_groups() when you need to merge. The "group" detection strategy uses union-find to form transitive clusters: if A≈B and B≈C, all three land in the same group. Plain detect_duplicates() returns individual pairs without transitivity.
The method= parameter of the detect_duplicates() convenience function controls how
the comparison is performed. These are independent of the SimilarityCalculator string
method used internally:
DuplicateCandidate fields are entity1, entity2, similarity_score: not entity_a, entity_b, similarity. Accessing the wrong field names raises AttributeError.
Merges detected duplicate groups into canonical entities:
from semantica.deduplication import EntityMerger, MergeStrategy# Basic usagemerger = EntityMerger(preserve_provenance=True)operations = merger.merge_duplicates(entities, strategy="keep_most_complete")# operations is List[MergeOperation]for op in operations: merged = op.merged_entity # the resulting merged entity dict sources = op.source_entities # list of original entities that were merged conflicts = op.merge_result.conflicts # unresolved conflicts, if any# Merge a known group directly (no duplicate detection)pair = [ {"id": "1", "name": "Apple Inc.", "type": "Company"}, {"id": "2", "name": "Apple", "type": "Company"},]op = merger.merge_entity_group(pair, strategy="keep_most_complete")print(op.merged_entity)# Retrieve full merge historyhistory = merger.get_merge_history()print("Total merges performed:", len(history))
merge_entities() and EntityMerger.merge_duplicates() return List[MergeOperation], not a list of entity dicts. Access .merged_entity on each operation to get the merged dict.
PropertyMergeRule is a dataclass, not an Enum. The merge strategy Enum is MergeStrategy (KEEP_FIRST, KEEP_LAST, KEEP_MOST_COMPLETE, KEEP_HIGHEST_CONFIDENCE, MERGE_ALL). Per-property rules are added via merger.merge_strategy_manager.add_property_rule(name, strategy).
