Pool

infra.llm.pool

Summary

LLM pool adaptors with retry, rotation, and HTTP error handling.

---

Rotate through configured model ports, retrying failed chat completions and translating provider errors into RuntimeError failures.

Classes

AsyncPoolAdaptor

AsyncPoolAdaptor(**kwargs)

Asynchronous LLM pool with model rotation.

Attributes:

Name	Type	Description
httpx_client	AsyncClientPort	Client used for requests.
models	list	Model ports available for rotation.

Initialize the pool from keyword configuration.

Parameters:

Name	Type	Description	Default
**kwargs		Configuration; httpx_client and models are consumed, remaining keys become instance attributes.	{}

Functions

aclose async

aclose()

Close the underlying asynchronous HTTP client.

generate async

generate(
    prompt: str,
    system_prompt: Optional[str] = None,
    max_tokens: Optional[int] = None,
    models: Optional[list] = None,
) -> str

Generate text for a prompt asynchronously.

Parameters:

Name	Type	Description	Default
prompt	str	User prompt text.	required
system_prompt	Optional[str]	Optional system instruction.	None
max_tokens	Optional[int]	Token cap overriding model defaults.	None
models	Optional[list]	Model ports overriding the pool defaults.	None

Returns:

Name	Type	Description
str	str	Generated content.

PoolAdaptor

PoolAdaptor(**kwargs)

Synchronous LLM pool with model rotation.

Attributes:

Name	Type	Description
httpx_client	SyncClientPort	Client used for requests.
models	list	Model ports available for rotation.

Initialize the pool from keyword configuration.

Parameters:

Name	Type	Description	Default
**kwargs		Configuration; httpx_client and models are consumed, remaining keys become instance attributes.	{}

Functions

close

close()

Close the underlying HTTP client.

generate

generate(
    prompt: str,
    system_prompt: Optional[str] = None,
    max_tokens: Optional[int] = None,
    models: Optional[list] = None,
) -> str

Generate text for a prompt.

Parameters:

Name	Type	Description	Default
prompt	str	User prompt text.	required
system_prompt	Optional[str]	Optional system instruction.	None
max_tokens	Optional[int]	Token cap overriding model defaults.	None
models	Optional[list]	Model ports overriding the pool defaults.	None

Returns:

Name	Type	Description
str	str	Generated content.