LLM and embedding providers¶
Every provider declares sends_data_offsite; a pipeline policy that forbids
off-site providers refuses to construct one that sets it. In CI the cloud SDKs
and models are mocked; calls against live services run only when you supply keys
or models.
LLM providers¶
openbtk.llms.openai.OpenAIProvider ¶
OpenAIProvider(*, model: str = _DEFAULT_MODEL, api_key: str | None = None, base_url: str | None = None, max_retries: int = 5)
Bases: BaseLLMProvider
Generate text via OpenAI's Chat Completions API.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
The OpenAI model id, e.g. |
_DEFAULT_MODEL
|
api_key
|
str | None
|
Forwarded to the SDK client. |
None
|
base_url
|
str | None
|
Forwarded to the SDK client. |
None
|
max_retries
|
int
|
Forwarded to :func: |
5
|
No I/O happens in __init__ (docs/09_CODING_STANDARDS.md rule 11):
the SDK client is constructed lazily, on first real call.
Source code in src/openbtk/llms/openai.py
65 66 67 68 69 70 71 72 73 74 75 76 77 | |
generate ¶
generate(prompt: str, **kwargs: Any) -> LLMResponse
Generate a completion for a single prompt.
Implemented via :meth:chat with a single user message -- Chat
Completions is OpenAI's one real generation endpoint; there is no
separate legacy-completions call to prefer.
Source code in src/openbtk/llms/openai.py
120 121 122 123 124 125 126 127 | |
openbtk.llms.anthropic.AnthropicProvider ¶
AnthropicProvider(*, model: str = _DEFAULT_MODEL, max_tokens: int = _DEFAULT_MAX_TOKENS, api_key: str | None = None, base_url: str | None = None, max_retries: int = 5)
Bases: BaseLLMProvider
Generate text via Anthropic's Messages API.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
The Claude model id, e.g. |
_DEFAULT_MODEL
|
max_tokens
|
int
|
The Messages API's required output-token ceiling. Applied whenever a call does not supply its own. |
_DEFAULT_MAX_TOKENS
|
api_key
|
str | None
|
Forwarded to the SDK client. |
None
|
base_url
|
str | None
|
Forwarded to the SDK client. |
None
|
max_retries
|
int
|
Forwarded to :func: |
5
|
No I/O happens in __init__ (docs/09_CODING_STANDARDS.md rule 11):
the SDK client is constructed lazily, on first real call.
Source code in src/openbtk/llms/anthropic.py
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | |
generate ¶
generate(prompt: str, **kwargs: Any) -> LLMResponse
Generate a completion for a single prompt.
Implemented via :meth:chat with a single user message.
Source code in src/openbtk/llms/anthropic.py
139 140 141 142 143 144 | |
stream ¶
stream(prompt: str, **kwargs: Any) -> Iterator[str]
Stream a completion, token-by-token.
Only opening the stream (__enter__, where the SDK's
httpx-based client actually issues the request) goes through
:meth:_call_with_retry -- once tokens have started arriving,
retrying would mean silently re-running the prompt and duplicating
already-yielded output, which is worse than surfacing the error.
The context manager is still always closed via finally, retry
or not, so a failure mid-stream never leaks the connection.
Source code in src/openbtk/llms/anthropic.py
164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 | |
openbtk.llms.huggingface.HuggingFaceLocalProvider ¶
HuggingFaceLocalProvider(*, model: str, revision: str, device: str = 'cpu', max_new_tokens: int = _DEFAULT_MAX_NEW_TOKENS, do_sample: bool = False)
Bases: BaseLLMProvider
Generate text from a local transformers causal LM.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
A HuggingFace Hub model id or local path. |
required |
revision
|
str
|
A pinned commit SHA (never a floating branch name -- see this module's own docstring for why this has no default). |
required |
device
|
str
|
Passed to the loaded model's |
'cpu'
|
max_new_tokens
|
int
|
Default generation length cap, overridable per call. |
_DEFAULT_MAX_NEW_TOKENS
|
do_sample
|
bool
|
Default sampling mode, overridable per call. |
False
|
No I/O happens in __init__ (docs/09_CODING_STANDARDS.md rule 11):
the tokenizer and model are downloaded/loaded lazily, on first real
call -- genuinely necessary here, not just a style rule, since a model
download can be gigabytes.
Source code in src/openbtk/llms/huggingface.py
63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | |
stream ¶
stream(prompt: str, **kwargs: Any) -> Iterator[str]
Stream generated text via a background-thread TextIteratorStreamer
-- the standard transformers pattern: model.generate() is
synchronous and blocking, so it runs on its own thread while this
generator yields tokens off the streamer as they arrive.
streamer.end() is called unconditionally in the background
thread's finally, not only on success: TextIteratorStreamer
only signals completion (unblocking this generator's yield from)
when generate() finishes normally through its own hook. If
generate() raises before producing anything, nothing would ever
end the streamer's queue and the consumer would block forever --
a real hang, not a hypothetical one.
Source code in src/openbtk/llms/huggingface.py
177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 | |
openbtk.llms.openai_compatible.OpenAICompatibleProvider ¶
OpenAICompatibleProvider(*, base_url: str, model: str, api_key: str | None = None, max_retries: int = 5, timeout: float = _DEFAULT_TIMEOUT)
Bases: BaseLLMProvider
Generate text from any OpenAI-Chat-Completions-shaped HTTP endpoint.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base_url
|
str
|
The endpoint's base URL, e.g. |
required |
model
|
str
|
The model identifier the endpoint expects, e.g. a local vLLM deployment's served model name. |
required |
api_key
|
str | None
|
Sent as |
None
|
max_retries
|
int
|
Forwarded to :func: |
5
|
timeout
|
float
|
Per-request timeout in seconds, forwarded to |
_DEFAULT_TIMEOUT
|
No I/O happens in __init__ (docs/09_CODING_STANDARDS.md rule 11):
the httpx.Client is constructed lazily, on first real call.
Source code in src/openbtk/llms/openai_compatible.py
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | |
generate ¶
generate(prompt: str, **kwargs: Any) -> LLMResponse
Generate a completion for a single prompt.
Implemented via :meth:chat with a single user message.
Source code in src/openbtk/llms/openai_compatible.py
136 137 138 139 140 141 | |
stream ¶
stream(prompt: str, **kwargs: Any) -> Iterator[str]
Stream a completion via server-sent events.
Only opening the request (until the response headers arrive, where an error status would surface) goes through retry -- see AnthropicProvider.stream's docstring for why a partially-consumed stream is never retried.
Source code in src/openbtk/llms/openai_compatible.py
157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 | |
Cloud providers¶
Azure OpenAI, AWS Bedrock and Google Vertex AI. Each is tested against a mocked SDK
and against the real SDK libraries offline (request shapes validated by the
libraries themselves); none has been run against a live account. They need the
llms (Azure), bedrock or vertex extra, and read credentials from the SDK's own
environment lookup, never from Openbtk.
openbtk.llms.azure_openai.AzureOpenAIProvider ¶
AzureOpenAIProvider(*, model: str, azure_endpoint: str | None = None, api_key: str | None = None, api_version: str | None = None, max_retries: int = 5)
Bases: OpenAIProvider
Generate text through an Azure OpenAI deployment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
The deployment name (what you named the deployment in Azure), which the API takes where OpenAI takes a model id. |
required |
azure_endpoint
|
str | None
|
|
None
|
api_key
|
str | None
|
Forwarded to the SDK. |
None
|
api_version
|
str | None
|
The Azure API version. |
None
|
max_retries
|
int
|
Rate-limit retry attempts (see the OpenAI provider). |
5
|
Example
provider = AzureOpenAIProvider(model="my-deployment") # nothing is called provider.sends_data_offsite True
Source code in src/openbtk/llms/azure_openai.py
49 50 51 52 53 54 55 56 57 58 59 60 | |
openbtk.llms.bedrock.BedrockProvider ¶
BedrockProvider(*, model: str, region_name: str | None = None, max_retries: int = 5)
Bases: BaseLLMProvider
Generate text with a model hosted on AWS Bedrock.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
The Bedrock model id or inference profile id, e.g. a vendor's id with its version suffix. Pin a versioned id: an unversioned alias is the vendor's moving target. |
required |
region_name
|
str | None
|
The AWS region. |
None
|
max_retries
|
int
|
Throttling retry attempts. |
5
|
Example
provider = BedrockProvider(model="vendor.model-v1:0") # nothing is called provider.sends_data_offsite True
Source code in src/openbtk/llms/bedrock.py
81 82 83 84 85 86 87 | |
openbtk.llms.vertex.VertexAIProvider ¶
VertexAIProvider(*, model: str, project: str | None = None, location: str | None = None, max_retries: int = 5)
Bases: BaseLLMProvider
Generate text with a model on Google Vertex AI.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
The Vertex model id, with its version where the model has one. Pin a versioned id: an alias is the vendor's moving target. |
required |
project
|
str | None
|
The Google Cloud project. |
None
|
location
|
str | None
|
The region, e.g. |
None
|
max_retries
|
int
|
Rate-limit retry attempts. |
5
|
Example
provider = VertexAIProvider(model="model-001") # nothing is called provider.sends_data_offsite True
Source code in src/openbtk/llms/vertex.py
71 72 73 74 75 76 77 78 79 80 81 82 83 | |
Biomedical presets¶
Presets are configuration over the classes above, not new classes.
openbtk.llms.presets.list_llm_presets ¶
list_llm_presets() -> list[str]
Return every preset name, sorted.
Source code in src/openbtk/llms/presets.py
76 77 78 | |
openbtk.llms.presets.create_llm_preset ¶
create_llm_preset(name: str, /, **overrides: Any) -> BaseLLMProvider
Construct the provider a biomedical LLM preset names.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
One of :func: |
required |
**overrides
|
Any
|
Merged over the preset's own params before
construction -- e.g. |
{}
|
Returns:
| Type | Description |
|---|---|
BaseLLMProvider
|
The constructed, registered provider (currently always a |
BaseLLMProvider
|
class: |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
Example
sorted(list_llm_presets()) ['medgemma-27b-text', 'meditron-7b', 'openbiollm-8b']
Source code in src/openbtk/llms/presets.py
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | |
Embedding providers¶
openbtk.embeddings.huggingface.HuggingFaceEmbeddingProvider ¶
HuggingFaceEmbeddingProvider(*, model: str, revision: str, dimension: int, pooling: Pooling = 'mean', device: str = 'cpu', max_length: int = 512)
Bases: BaseEmbeddingProvider
Embed text with a local HuggingFace transformers encoder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
A HuggingFace Hub model id or local path. |
required |
revision
|
str
|
A pinned commit SHA (never a floating branch name --
same FR-P-05 reasoning as |
required |
dimension
|
int
|
The model's real output width -- see this module's own docstring for why this is required rather than derived. |
required |
pooling
|
Pooling
|
How to reduce token-level hidden states to one vector.
|
'mean'
|
device
|
str
|
Passed to the loaded model's |
'cpu'
|
max_length
|
int
|
Tokenizer truncation length. |
512
|
No I/O happens in __init__ (docs/09_CODING_STANDARDS.md rule 11):
the tokenizer and model are downloaded/loaded lazily, on first real
call.
Source code in src/openbtk/embeddings/huggingface.py
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 | |
openbtk.embeddings.openai.OpenAIEmbeddingProvider ¶
OpenAIEmbeddingProvider(*, model: str = _DEFAULT_MODEL, dimension: int | None = None, api_key: str | None = None, base_url: str | None = None, max_retries: int = 5)
Bases: BaseEmbeddingProvider
Embed text via OpenAI's Embeddings API.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
The OpenAI embedding model id. |
_DEFAULT_MODEL
|
dimension
|
int | None
|
The model's output width. Defaults to the known value
for |
None
|
api_key
|
str | None
|
Forwarded to the SDK client. |
None
|
base_url
|
str | None
|
Forwarded to the SDK client. |
None
|
max_retries
|
int
|
Forwarded to :func: |
5
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
No I/O happens in __init__: the SDK client is constructed lazily,
on first real call.
Source code in src/openbtk/embeddings/openai.py
75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 | |
Biomedical presets¶
openbtk.embeddings.presets.list_embedding_presets ¶
list_embedding_presets() -> list[str]
Return every preset name, sorted.
Source code in src/openbtk/embeddings/presets.py
122 123 124 | |
openbtk.embeddings.presets.create_embedding_preset ¶
create_embedding_preset(name: str, /, **overrides: Any) -> BaseEmbeddingProvider
Construct the provider a biomedical embedding preset names.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
One of :func: |
required |
**overrides
|
Any
|
Merged over the preset's own params before
construction -- e.g. |
{}
|
Returns:
| Type | Description |
|---|---|
BaseEmbeddingProvider
|
The constructed, registered provider (currently always a |
BaseEmbeddingProvider
|
class: |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
Example
sorted(list_embedding_presets()) # doctest: +NORMALIZE_WHITESPACE ['biobert', 'clinicalbert', 'medcpt-article', 'medcpt-query', 'pubmedbert', 'sapbert']
Source code in src/openbtk/embeddings/presets.py
127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 | |