LLMs Module

semantica.llms provides a consistent generation API across major LLM providers and a separate typed-decision API for TypeSafe Jev:

Generative providers are interchangeable wherever a provider instance is taken: the llm_provider= argument of AgentContext.query_with_reasoning(), ContextRetriever.query_with_reasoning() and TemporalQueryRewriter
Extractors take no instance: they build the provider themselves from provider=, llm_model= and api_key=
LiteLLM routes to 100+ providers with a single class and model-string prefixes
HuggingFaceLLM runs fully on-premise: no API key, no network calls
Structured output via generate_typed() for schema-validated extraction
Jev and AsyncJev return Choice, Noul, or Score decisions without pretending to generate text
Streaming, tool use, and generate_batch() for bulk inference

Exported Classes

from semantica.llms import (
    AsyncJev,
    Groq,
    HuggingFaceLLM,
    Jev,
    JevDecisionResult,
    LiteLLM,
    OpenAI,
)

Class	Provider	API Key Required
`Groq`	Groq Cloud	`GROQ_API_KEY`
`OpenAI`	OpenAI / any OpenAI-compatible gateway	`OPENAI_API_KEY`
`LiteLLM`	100+ providers via LiteLLM routing	Depends on model
`HuggingFaceLLM`	Local HuggingFace Transformers	None (local)
`Jev`	TypeSafe System One, synchronous	`TYPESAFE_API_KEY`
`AsyncJev`	TypeSafe System One, asynchronous	`TYPESAFE_API_KEY`

Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, Cohere, and 90+ others are all available via LiteLLM using their model-string prefix. See the LiteLLM section below.

What You Get

Unified LLMProvider interface: swap providers with a one-line change, no application code changes
LiteLLM: single class for 100+ providers using model-string routing
Local models: HuggingFaceLLM runs fully on-premise, no API key
Typed decisions: Jev supports bounded Choice, Noul, and Score decisions with probability metadata
Streaming: token-by-token output for low-latency UX
Custom gateways: point OpenAI at any OpenAI-compatible endpoint via base_url

Choosing a Provider

Free tier, fastest inference, zero setup friction. Best for development and high-throughput extraction pipelines.


Speed	Very fast: 100+ tok/s
Cost	Free tier available
Context	128k
Best for	Development, high-throughput extraction

import os
from semantica.llms import Groq

llm = Groq(
    model="llama-3.1-8b-instant",
    api_key=os.getenv("GROQ_API_KEY"),
    temperature=0.0,
)

Get your free key at console.groq.com.

Highest accuracy, best JSON mode and function calling. Use for production pipelines where extraction quality matters.


Speed	Fast
Cost	Medium
Context	128k
Best for	Production quality, JSON extraction, function calling

import os
from semantica.llms import OpenAI

llm = OpenAI(
    model="gpt-4o",
    api_key=os.getenv("OPENAI_API_KEY"),
    temperature=0.0,
    max_tokens=4096,
)

Fully on-premise: no API key, no data leaves your infrastructure. Required for air-gapped deployments.


Speed	Medium (hardware-dependent)
Cost	Free (local compute only)
Context	Varies by model
Best for	Privacy, air-gapped, custom fine-tunes

# Install Ollama and pull a model first
ollama pull llama3.2:3b

from semantica.llms import LiteLLM

llm = LiteLLM(
    model="ollama/llama3.2:3b",
    api_base="http://localhost:11434",  # Ollama default port
)

No API key required. Ensure the Ollama server is running (ollama serve) before creating the LiteLLM instance.

Largest context window, best multi-hop reasoning, highest safety bar. Use for complex analysis and long-document extraction.


Speed	Fast
Cost	Medium
Context	200k
Best for	Complex reasoning, long documents, safety-critical outputs

import os
from semantica.llms import LiteLLM

llm = LiteLLM(
    model="anthropic/claude-sonnet-5",
    api_key=os.getenv("ANTHROPIC_API_KEY"),
    temperature=0.0,
)

Lowest cost per token for high-volume workloads. Strong on coding and structured data extraction.


Speed	Fast
Cost	Very low
Context	64k
Best for	High-volume pipelines, coding tasks, budget-sensitive workloads

import os
from semantica.llms import LiteLLM

llm = LiteLLM(
    model="deepseek/deepseek-chat",
    api_key=os.getenv("DEEPSEEK_API_KEY"),
    temperature=0.0,
)

Decision-only inference for bounded routing, classification, binary checks, and scoring. Use a generative provider when you need prose or multi-step reasoning.


Speed	Very fast
Output	Choice, Noul, or Score
Interface	`decide()`; not `generate()`
Best for	Confidence-gated application decisions

from semantica.llms import Jev

decider = Jev()  # reads TYPESAFE_API_KEY
result = decider.decide(
    state={"risk_score": 0.62},
    question="How should this case be routed?",
    kind="choice",
    choices=["approve", "escalate"],
)

API Key Setup

Environment Variables (Recommended)

# Add to your shell profile (.bashrc, .zshrc, etc.)
export GROQ_API_KEY="your_groq_api_key_here"
export OPENAI_API_KEY="your_openai_api_key_here"
export ANTHROPIC_API_KEY="your_anthropic_api_key_here"
export TYPESAFE_API_KEY="your_typesafe_api_key_here"

# Reload your shell
source ~/.bashrc

Configuration File Method

# config.yaml
llm_provider:
  name: groq
  model: llama-3.1-8b-instant
  temperature: 0.0
# Set GROQ_API_KEY environment variable and pass to constructor

Programmatic Setup

import os
from semantica.llms import Groq, LiteLLM

# Method 1: Direct API key
llm = Groq(api_key="your-api-key-here", model="llama-3.1-8b-instant")

# Method 2: Environment variable (preferred)
llm = Groq(api_key=os.getenv("GROQ_API_KEY"), model="llama-3.1-8b-instant")

# Method 3: Multiple providers via LiteLLM
providers = {
    "fast": LiteLLM(model="groq/llama-3.1-8b-instant", api_key=os.getenv("GROQ_API_KEY")),
    "smart": LiteLLM(model="anthropic/claude-sonnet-5", api_key=os.getenv("ANTHROPIC_API_KEY"))
}

Security Best Practices

Never commit API keys to version control. Use environment variables or secure secret management.

# ❌ Bad - API key in code
llm = Groq(api_key="gsk_abc123...", model="llama-3.1-8b-instant")

# ✅ Good - Environment variable
llm = Groq(api_key=os.getenv("GROQ_API_KEY"), model="llama-3.1-8b-instant")

TypeSafe Jev

Jev and AsyncJev wrap the official typesafe-sdk System One client. They are decision-only providers: neither class implements generate(), generate_structured(), or generate_typed(), and neither can stand in for a generative provider: not as the llm_provider= argument of a reasoning query, and not as provider= in an extractor.

Jev is experimental and early-access. The llm-typesafe extra currently supports the reviewed TypeSafe SDK 0.7 API. Like Semantica itself, it requires Python 3.11 or newer; the SDK is optional and guarded, so the base installation does not need it.

TypeSafe SDK 0.7 emits full request and response bodies through its typesafe_sdk logger at DEBUG; authorization headers are redacted, but application state is not. Keep that logger at INFO or higher when state may be sensitive, even if the root logger uses DEBUG.

import logging

logging.getLogger("typesafe_sdk").setLevel(logging.INFO)

pip install "semantica[llm-typesafe]"

Constructors

Jev(
    model="jev-latest",
    api_key=None,
    client=None,
    **client_options,
)

AsyncJev(
    model="jev-latest",
    api_key=None,
    client=None,
    **client_options,
)

Argument	Type	Description
`model`	`str`	TypeSafe model name or alias. Defaults to `jev-latest`. The result preserves the concrete model returned by the service.
`api_key`	`Optional[str]`	API key. When omitted, the SDK reads `TYPESAFE_API_KEY`.
`client`	compatible client or `None`	Optional injected `TypeSafeClient`/`AsyncTypeSafeClient`, useful for custom transports and tests. Injected clients remain caller-owned.
`**client_options`	`Any`	Passed to the SDK client constructor, including `retry`, `timeout`, `headers`, `transport`, or `base_url`.

Methods

Method	Returns	Description
`decide(state, question, kind, , choices=None, criteria=None, *request_options)`	`JevDecisionResult`	Make one Choice, Noul, or Score decision. `AsyncJev.decide()` is awaited.
`is_available()`	`bool`	Check locally for an injected client or both the SDK and a configured API key. It does not validate the key or make a network request.
`close()`	`None`	Close a synchronous client created by `Jev`; injected clients are not closed.
`aclose()`	`None`	Asynchronous equivalent for `AsyncJev`.

state accepts a string, mapping, or list. question is the instruction evaluated against that state. Inputs by kind:

kind="choice": pass choices as a sequence of labels or a label-to-description mapping. criteria= is accepted as an equivalent explicit name. Do not pass both.
kind="noul": optionally pass criteria={"true": ..., "false": ...}.
kind="score": pass criteria as an ordered sequence of rubric levels, beginning at score zero.

Additional request options are forwarded to system_one(), such as per-call model, retry, timeout, extra_headers, or extra_body.

from semantica.llms import Jev

jev = Jev()  # TYPESAFE_API_KEY
result = jev.decide(
    state={"ticket": "Charged twice; please refund this today."},
    question="Which queue should handle this ticket?",
    kind="choice",
    choices={
        "billing": "Charges, invoices, or refunds",
        "technical": "Product or integration failures",
        "other": None,
    },
)

`JevDecisionResult`

Field	Type	Description
`kind`	`Literal["choice", "noul", "score"]`	Primitive used for the decision.
`value`	`Union[str, bool, float]`	Selected label, Noul boolean at the 0.5 boundary, or expected score.
`probability`	`Optional[float]`	Selected-label probability for Choice, raw probability of yes for Noul, and `None` for Score.
`confidence`	`float`	SDK confidence for Choice/Score; `abs(2p - 1)` routing certainty for Noul.
`probabilities`	`dict`	Full Choice or Score probability distribution. Empty for Noul.
`model`	`str`	Concrete model reported by TypeSafe.
`request_id`	`Optional[str]`	Request ID when returned by the API.
`usage`	`dict`	Reported input/output token usage.
`legend`	`dict[int, Any]`	Score-level rubric; empty for Choice and Noul.

result.to_dict() returns a serializable copy suitable for Semantica decision metadata or cross_system_context. It deliberately retains the raw Noul probability separately from the derived confidence.

from semantica.llms import AsyncJev

async with AsyncJev() as jev:
    result = await jev.decide(
        state="The message asks for an immediate duplicate-charge refund.",
        question="Does the message explicitly communicate urgency?",
        kind="noul",
    )

Providers

import os
from semantica.llms import Groq

llm = Groq(
    model="llama-3.3-70b-versatile",   # recommended; implementation default: llama-3.1-8b-instant
    api_key=os.getenv("GROQ_API_KEY"),
    max_tokens=64000,
    temperature=0.0,
)
# **Best for:** high-throughput extraction, fast inference at low cost

import os
from semantica.llms import OpenAI

llm = OpenAI(
    model="gpt-4o",                     # recommended; implementation default: gpt-3.5-turbo
    api_key=os.getenv("OPENAI_API_KEY"),
    temperature=0.0,
)
# **Best for:** general purpose, function calling, JSON mode

import os
from semantica.llms import LiteLLM

# pip install "semantica[llm-litellm]"

# Anthropic Claude
llm = LiteLLM(model="anthropic/claude-opus-4-7",         api_key=os.getenv("ANTHROPIC_API_KEY"))

# Google Gemini
llm = LiteLLM(model="gemini/gemini-1.5-pro",             api_key=os.getenv("GOOGLE_API_KEY"))

# Ollama (local: no API key)
llm = LiteLLM(model="ollama/llama3.2:3b",                api_base="http://localhost:11434")

# DeepSeek
llm = LiteLLM(model="deepseek/deepseek-chat",            api_key=os.getenv("DEEPSEEK_API_KEY"))

# Azure OpenAI
llm = LiteLLM(model="azure/gpt-4o",                      api_key=os.getenv("AZURE_API_KEY"))

# AWS Bedrock
llm = LiteLLM(model="bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0")

# Novita AI
llm = LiteLLM(model="novita/deepseek/deepseek-v3.2",     api_key=os.getenv("NOVITA_API_KEY"))

from semantica.llms import HuggingFaceLLM

llm = HuggingFaceLLM(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    device="cuda",           # "cpu" | "cuda" | "mps"
    max_new_tokens=512,
    temperature=0.1,
)
# Bring your own model: full local control, no API key

from semantica.llms import Jev

decider = Jev(model="jev-latest")  # reads TYPESAFE_API_KEY
result = decider.decide(
    state={"risk_score": 0.62},
    question="How should this case be routed?",
    kind="choice",
    choices=["approve", "escalate"],
)
# Best for typed, confidence-gated application decisions; no text generation

LiteLLM: 100+ Providers

LiteLLM is the recommended way to access any provider not directly exported by semantica.llms. Use the provider/model string format:

import os
from semantica.llms import LiteLLM

# Pattern: LiteLLM(model="<provider>/<model-name>")
providers = {
    "Anthropic":  LiteLLM(model="anthropic/claude-opus-4-7",       api_key=os.getenv("ANTHROPIC_API_KEY")),
    "Gemini":     LiteLLM(model="gemini/gemini-1.5-pro",            api_key=os.getenv("GOOGLE_API_KEY")),
    "Ollama":     LiteLLM(model="ollama/llama3.2:3b",               api_base="http://localhost:11434"),
    "DeepSeek":   LiteLLM(model="deepseek/deepseek-chat",           api_key=os.getenv("DEEPSEEK_API_KEY")),
    "Azure":      LiteLLM(model="azure/gpt-4o",                     api_key=os.getenv("AZURE_API_KEY")),
    "Bedrock":    LiteLLM(model="bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0"),
    "Cohere":     LiteLLM(model="cohere/command-r-plus",            api_key=os.getenv("COHERE_API_KEY")),
    "Novita AI":  LiteLLM(model="novita/deepseek/deepseek-v3.2",    api_key=os.getenv("NOVITA_API_KEY")),
}

# Every LiteLLM instance implements the same .generate() interface
response = providers["Anthropic"].generate("Explain GraphRAG in one paragraph.")

The full list of supported LiteLLM model strings is at docs.litellm.ai/docs/providers. Use the provider/model format shown above.

Custom / Enterprise Gateways

Any OpenAI-compatible endpoint: internal routing layers, Qwen proxies, or private LLaMA deployments:

import os
from semantica.llms import OpenAI

llm = OpenAI(
    model="qwen2.5-72b",
    api_key=os.getenv("GATEWAY_API_KEY"),
    base_url="https://my-internal-gateway.company.com/v1",
)

base_url is validated at construction time. Non-HTTP(S) schemes raise ValueError to prevent SSRF attacks (fixed in v0.5.0).

Using Generative Providers in Extractors

Extractors do not take a provider instance. They build one themselves from keyword configuration: provider names the backend (default "openai"), llm_model picks the model, and api_key is read from the matching <PROVIDER>_API_KEY variable when it is omitted. llm_provider= is not read here: it travels through **config into the provider factory and is kept in the constructed provider’s config mapping, which nothing consults — the instance never serves as the extractor’s provider, so the extractor runs against the backend named by provider= (the "openai" default when it is omitted). Jev is not usable here either, because it returns decisions rather than generated text:

import os
from semantica.semantic_extract import (
    NERExtractor,
    RelationExtractor,
    TripletExtractor,
)

llm_config = {
    "method": "llm",
    "provider": "groq",
    "api_key": os.environ["GROQ_API_KEY"],
    "llm_model": "llama-3.3-70b-versatile",
}

ner  = NERExtractor(**llm_config, max_retries=3)
rel  = RelationExtractor(**llm_config)
trip = TripletExtractor(**llm_config)

Use llm_model, not model. Both names configure an extractor’s own LLM call (llm_model wins), but TripletExtractor strips model out of the configuration it shares with the entity and relation extractors it builds internally, because there that keyword is the spaCy/HuggingFace model name. So model alone reaches the triplet stage only, and the two internal stages are built with no LLM model at all: they are constructed with an explicit model=None, which overrides the provider’s own default. llm_model configures all three.

Provider Comparison

Provider	Import	Speed	Cost	Local	Context	Best For
Groq	`Groq`	Very fast	Low	No	128k	High-throughput extraction
OpenAI	`OpenAI`	Fast	Medium	No	128k	General purpose, function calling
Anthropic	`LiteLLM(model="anthropic/...")`	Fast	Medium	No	200k	Complex reasoning, safety
Gemini	`LiteLLM(model="gemini/...")`	Fast	Low	No	1M	Long context, multimodal
Ollama	`LiteLLM(model="ollama/...")`	Medium	Free	Yes	Varies	Privacy, air-gapped
DeepSeek	`LiteLLM(model="deepseek/...")`	Fast	Very low	No	64k	Coding, analysis
Azure OpenAI	`LiteLLM(model="azure/...")`	Fast	Medium	No	128k	Enterprise, compliance
AWS Bedrock	`LiteLLM(model="bedrock/...")`	Fast	Varies	No	Varies	AWS-native workloads
HuggingFace	`HuggingFaceLLM`	Slow	Free	Yes	Varies	Custom models, BYOM
TypeSafe Jev	`Jev` / `AsyncJev`	Very fast	Provider pricing	No	Application state	Typed routing, classification, and scoring

For production extraction pipelines, Groq delivers the best throughput-to-cost ratio. For complex multi-hop reasoning, Claude Opus or GPT-4o provide the highest accuracy.

Defaults and Reproducibility

Documentation examples may showcase stronger models for better developer experience, while implementation defaults prioritize reliability and cost efficiency. Understanding actual defaults helps with reproducible results and consistent benchmarking. Verified Implementation Defaults:

Provider	Default Model	Notes
`Groq`	`llama-3.1-8b-instant`	Implementation default; examples use `llama-3.3-70b-versatile` for showcase
`OpenAI`	`gpt-3.5-turbo`	Implementation default; examples use `gpt-4o` for showcase
`HuggingFaceLLM`	`gpt2`	Lightweight, widely compatible
`Jev` / `AsyncJev`	`jev-latest`	Early-access alias; pass a concrete model in production when reproducibility requires it

These are the models used when you construct a provider without specifying model=. Examples throughout this documentation use stronger showcase models. Always pass model= explicitly in production for reproducible results. Why This Matters:

Reproducible extraction results across environments
Consistent baseline performance for benchmarking
Predictable costs when scaling production workloads

Performance and Reliability Tips

Extraction with Retries

import os
from semantica.semantic_extract import NERExtractor

ner = NERExtractor(
    method="llm",
    provider="groq",
    api_key=os.environ["GROQ_API_KEY"],
    llm_model="llama-3.3-70b-versatile",
    max_retries=3,
)

# Process multiple texts with automatic retries
texts = ["Document 1 text...", "Document 2 text...", "Document 3 text..."]
all_entities = []

for text in texts:
    entities = ner.extract(text)
    all_entities.extend(entities)

# Rate limiting handled automatically by provider

Model Selection by Use Case

Use Case	Recommended Provider/Model	Reasoning
Entity Extraction	`Groq("llama-3.3-70b-versatile")`	Fast, good accuracy for structured tasks
Relation Extraction	`OpenAI("gpt-4o")`	Best at complex relationship reasoning
Complex Analysis	`LiteLLM("anthropic/claude-sonnet-5")`	Highest reasoning capability
High Volume/Cost	`LiteLLM("deepseek/deepseek-chat")`	Lowest cost per token

Error Handling

import os
from semantica.semantic_extract import NERExtractor

# Automatic retries for rate limits and transient errors
extractor = NERExtractor(
    method="llm",
    provider="groq",
    api_key=os.environ["GROQ_API_KEY"],
    llm_model="llama-3.3-70b-versatile",
    max_retries=3,     # Retry failed requests automatically
)

Semantic Extract — Use LLMs for NER and relation extraction.
Agno Integration — LLM providers in Agno multi-agent teams.
Reasoning — LLM-backed deductive and abductive reasoning.
Context — GraphRAG uses LLMs for reasoning over knowledge graphs.

Context & Intelligence

Storage

Data Pipeline

Quality & Provenance

Output

Utilities

Exported Classes

What You Get

Choosing a Provider

API Key Setup

Environment Variables (Recommended)

Configuration File Method

Programmatic Setup

Security Best Practices

TypeSafe Jev

Constructors

Methods

`JevDecisionResult`

Providers

LiteLLM: 100+ Providers

Custom / Enterprise Gateways

Using Generative Providers in Extractors

Provider Comparison

Defaults and Reproducibility

Performance and Reliability Tips

Extraction with Retries

Model Selection by Use Case

Error Handling

​Exported Classes

​What You Get

​Choosing a Provider

​API Key Setup

​Environment Variables (Recommended)

​Configuration File Method

​Programmatic Setup

​Security Best Practices

​TypeSafe Jev

​Constructors

​Methods

​JevDecisionResult

​Providers

​LiteLLM: 100+ Providers

​Custom / Enterprise Gateways

​Using Generative Providers in Extractors

​Provider Comparison

​Defaults and Reproducibility

​Performance and Reliability Tips

​Extraction with Retries

​Model Selection by Use Case

​Error Handling

Exported Classes

What You Get

Choosing a Provider

API Key Setup

Environment Variables (Recommended)

Configuration File Method

Programmatic Setup

Security Best Practices

TypeSafe Jev

Constructors

Methods

`JevDecisionResult`

Providers

LiteLLM: 100+ Providers

Custom / Enterprise Gateways

Using Generative Providers in Extractors

Provider Comparison

Defaults and Reproducibility

Performance and Reliability Tips

Extraction with Retries

Model Selection by Use Case

Error Handling