Evals Module

semantica.evals measures the quality of decision intelligence outputs. It takes the decisions, audit trails, and reasoning text your pipeline produces and scores them against expectations you define, returning a structured summary you can log, assert on in tests, or track across runs.

A registry of named evaluators, from exact string matching to ROUGE overlap and LLM-as-judge
decision_scores, a composite evaluator for Decision objects that checks outcome, confidence bounds, required fields, provenance, and (optionally) policy compliance
A evaluate() runner that applies several evaluators to a list of cases and aggregates pass / fail / error counts
Per-evaluator objectives that let you override an evaluator’s built-in verdict at the run level

The module is versioned separately from the package: semantica.evals.__version__ is "0.1.0". The public surface described here is stable, but expect additive changes (new evaluators, new objective options) before it reaches 1.0.

Public API

Name	Kind	Role
`evaluate(cases, evaluators, config=None, target_fn=None)`	function	Run named evaluators over each case, return an `EvalSummary`
`evaluate_repeated(cases, evaluators, config=None, target_fn=None, runs=10)`	function	Rerun `target_fn` per case and aggregate stats, return a `RepeatedSummary`
`list_evaluators()`	function	Sorted names of every registered evaluator
`get_evaluator(name)`	function	Look up a single evaluator function by name
`EvalMetric`	dataclass (frozen)	One evaluator’s result: `score`, `passed`, `meta`
`CaseResult`	namedtuple	One case’s result: `case_id`, `status`, `metrics`, `details`
`EvalSummary`	dataclass	Aggregate across cases: `total`, `passed`, `failed`, `errors`, `pass_rate`, `cases`
`SampleStats`	dataclass (frozen)	Per-evaluator statistics over repeated runs: `n`, `passes`, `errors`, `pass_rate`, `mean_score`, `stddev`, `any_passed`, `all_passed`, `objective_passed`, `samples`
`RepeatedCaseResult`	dataclass (frozen)	Repeated outcome for one case: `case_id`, `verdict`, `stats`
`RepeatedSummary`	dataclass	Aggregate across repeated-sampled cases: `runs`, `stable_pass`, `flaky`, `stable_fail`, `errors`, `cases`

import semantica.evals as evals
from semantica.evals import evaluate, evaluate_repeated, list_evaluators, get_evaluator

Built-in evaluators

Every evaluator is a plain function fn(actual, expected, config=None) -> EvalMetric registered under a stable name. list_evaluators() returns the current set:

>>> list_evaluators()
['decision_scores', 'exact_match', 'keyword_check', 'length_range',
 'levenshtein', 'llm_as_judge', 'numeric_range', 'regex_match', 'rouge',
 'temporal_range']

Name	Passes when	Relevant `config` keys
`exact_match`	`actual == expected`	none
`regex_match`	`re.search(expected, actual)` matches	none
`keyword_check`	every required term appears in `actual` (word-boundary)	`required` (falls back to `expected`)
`numeric_range`	`min <= actual <= max`	`min`, `max` (both required)
`temporal_range`	ISO datetime `actual` falls in `[min, max]`	`min`, `max` as ISO strings (both required)
`length_range`	`min <= len(actual) <= max`	`min` (default 0), `max` (required)
`levenshtein`	normalized similarity `>= threshold`	`threshold` (default 0.8)
`rouge`	ROUGE-1 F1 `> 0` and `>= threshold`	`threshold` (default 0.0)
`llm_as_judge`	caller-supplied `judge_fn(actual, expected)` returns truthy	`judge_fn` (required callable)
`decision_scores`	all configured sub-checks on a `Decision` pass	see below

An evaluator that cannot run (bad regex, unparseable datetime, no judge_fn) returns an EvalMetric with an "error" key in meta rather than raising. Evaluators that require numeric bounds (numeric_range, length_range) instead return a failing metric with a "reason" key when the bound is missing — they do not raise and do not set "error".

`decision_scores`

decision_scores accepts a Decision (from semantica.context.decision_models) or its dict form and runs a set of field-level and governance checks. The score is the fraction of checks that passed; passed is True only when all of them did.

Sub-check	Controlled by
Outcome matches	`expected_outcome` in config, or the case’s `expected`; skipped when neither is set
Confidence in range	`min_confidence` (default 0.0), `max_confidence` (default 1.0); always run
`decision_maker`, `reasoning`, `scenario` non-empty	always run
Provenance present in metadata	`provenance_key` (default `"provenance"`); always run
Policy compliance	`policy_engine` and `policy_id` both set; skipped otherwise

Passing causal_chain_exists in config raises NotImplementedError. That key is a reserved slot for a future release.

Running an evaluation

evaluate() takes a list of cases and a list of evaluator names. A case is either a (expected, actual) tuple or a dict:

{
    "id": "loan-001",          # optional, generated if absent
    "expected": ...,           # optional; some evaluators read it, some don't
    "actual": ...,             # the value under test
    "config": {...},           # optional, per-evaluator settings for this case
    "target_fn": callable,     # optional, called with the case to produce `actual`
}

If actual is missing, the runner calls the case’s target_fn (or the target_fn passed to evaluate()) to produce it. Per-case config is deep-merged over the top-level config, so a case can override one evaluator’s settings without discarding the rest.

from datetime import datetime

from semantica.context.decision_models import Decision
from semantica.evals import evaluate

decision = Decision(
    decision_id="d-1",
    category="loan",
    scenario="loan-request",
    reasoning="vetted against lending policy v3",
    outcome="approve",
    confidence=0.87,
    timestamp=datetime.now(),
    decision_maker="approver-a",
    metadata={"provenance": "workflow:loan/v3"},
)

cases = [
    {
        "id": "loan-001",
        "actual": decision,
        "config": {
            "decision_scores": {
                "expected_outcome": "approve",
                "min_confidence": 0.7,
            }
        },
    },
]

summary = evaluate(cases, ["decision_scores"])
print(summary.pass_rate)   # 1.0

Evaluators run independently per case. If one raises, that case’s status becomes "error" and the exception text is captured in the metric’s meta; the rest of the run continues.

Objectives

By default each evaluator decides its own pass / fail. An objective overrides that verdict at the run level, keyed by evaluator name under config:

# Raise levenshtein's bar from its default 0.8 to 0.9
evaluate(
    [("apple", "aple")],
    evaluators=["levenshtein"],
    config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.9}}},
)

# Lower is better
evaluate(
    [("night", "nacht")],
    evaluators=["levenshtein"],
    config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.5}}},
)

# Expect the metric NOT to match
evaluate(
    [("ok", "ok")],
    evaluators=["exact_match"],
    config={"exact_match": {"objective": {"expect": False}}},
)

Rules:

maximize with threshold: pass iff score >= threshold. maximize with no threshold is a no-op and the evaluator’s own verdict stands.
minimize with threshold: pass iff score <= threshold. minimize requires a threshold; omitting it raises ValueError.
expect (True / False): pass iff bool(score) equals it. Cannot be combined with direction or threshold, and must be a real boolean.
A metric that already carries an "error" in its meta is unaffected by any objective.
Invalid objective config is validated for every case before any evaluator runs, so a bad objective fails the whole run up front rather than partway through.

Reading the summary

summary = evaluate(cases, ["decision_scores"])

summary.total, summary.passed, summary.failed, summary.errors
summary.pass_rate    # passed / total, or 1.0 for an empty case list

for case in summary.cases:
    print(case.case_id, case.status)          # status: "pass" | "fail" | "error"
    for name, metric in case.metrics.items():
        print(name, metric.score, metric.passed)
        print(metric.meta.get("reasons", {}))  # per-sub-check failure reasons

EvalMetric, SampleStats, and RepeatedCaseResult are frozen dataclasses (score: float, passed: bool, meta: dict, …). CaseResult is a namedtuple and the *Summary classes are plain dataclasses, so all are straightforward to serialize for logging or regression tracking.

Repeated sampling

For nondeterministic targets (LLM-backed extraction, agent pipelines, sampling-based judges) a single verdict says little. evaluate_repeated reruns target_fn per case n times and aggregates per-evaluator statistics:

from semantica.evals import evaluate_repeated

summary = evaluate_repeated(
    cases,                       # must be dict cases; see below
    evaluators=["exact_match"],
    target_fn=pipeline_run,      # called once per run to produce a fresh `actual`
    runs=10,
)

summary.runs            # 10
summary.cases[0].verdict  # "stable_pass" | "flaky" | "stable_fail" | "error"
summary.cases[0].stats["exact_match"].pass_rate   # e.g. 0.8
summary.cases[0].stats["exact_match"].stddev      # e.g. 0.13

Per-evaluator SampleStats carries n, passes, errors, pass_rate (passes / n), mean_score, stddev, and the derived booleans any_passed (observed pass@n) and all_passed (observed pass^n). Verdicts pool all samples: stable_pass when every run passed every evaluator, stable_fail when none passed, flaky when the pool mixes passes and fails, and error when a run errored. Requirements and edge cases:

Each case can be a dict or a (expected, actual) tuple, matching evaluate(). A case that carries a non-null static actual has nothing to sample; using it with runs > 1 raises ValueError. An actual of None is treated as absent, so it falls back to the resolver exactly as in evaluate().
target_fn exceptions and evaluator failures become per-run error samples and mark the case error; they never crash the run.
The objective layer applies per run as in evaluate(), and the same objective gates the aggregate pass_rate through SampleStats.objective_passed (e.g. {direction: maximize, threshold: 0.8} requires an 80% pass rate across runs).
At least one evaluator is required, and evaluator names must be unique.
runs must be >= 1.

Notes

llm_as_judge needs config["judge_fn"], a callable judge_fn(actual, expected) -> bool you supply. No LLM backend is imported unless you pass one in.
decision_scores governance checks are opt-in: policy compliance is only evaluated when both policy_engine and policy_id are present.

Context & Intelligence

Storage

Data Pipeline

Quality & Provenance

Output

Utilities

Public API

Built-in evaluators

`decision_scores`

Running an evaluation

Objectives

Reading the summary

Repeated sampling

Notes

See also

​Public API

​Built-in evaluators

​decision_scores

​Running an evaluation

​Objectives

​Reading the summary

​Repeated sampling

​Notes

​See also

Public API

Built-in evaluators

`decision_scores`

Running an evaluation

Objectives

Reading the summary

Repeated sampling

Notes

See also