> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lyceum.technology/llms.txt
> Use this file to discover all available pages before exploring further.
# Serverless Inference
> Pay-per-request, OpenAI-compatible inference endpoints
Serverless inference offers pay-per-request pricing with no idle compute cost, the right choice for low-volume or spiky traffic where reserving GPU replicas would be wasteful.
The endpoints are **OpenAI-compatible**. Point any OpenAI client at the base URL below, swap in a Lyceum API key and a model name, and everything else, streaming, tool calling, usage accounting, works the same.
## Endpoint and authentication
The base URL for serverless inference is:
```
https://api.lyceum.technology/api/v2/external/serverless
```
Authenticate with your Lyceum API key (`lk_...`) as a Bearer token. Most SDKs append the path for you, so you pass only the base URL above.
| Method | Path | Purpose |
| ------ | ------------------- | ------------------------------------------------------------------------------------------- |
| `POST` | `/chat/completions` | Chat completions (streaming and non-streaming) |
| `POST` | `/embeddings` | Text embeddings |
| `POST` | `/route` | Score a prompt and get the recommended model (see [Smart Routing](/docs/inference/routing)) |
| `GET` | `/models` | List the models currently available |
There is no legacy text `/completions` endpoint. All text generation goes through `/chat/completions` using the `messages` format.
## Quickstart
```python theme={null}
from openai import OpenAI
client = OpenAI(
api_key="lk_...",
base_url="https://api.lyceum.technology/api/v2/external/serverless",
)
resp = client.chat.completions.create(
model="z-ai/glm-5.2",
messages=[{"role": "user", "content": "Explain prompt caching in one sentence."}],
)
print(resp.choices[0].message.content)
```
```bash theme={null}
curl https://api.lyceum.technology/api/v2/external/serverless/chat/completions \
-H "Authorization: Bearer lk_..." \
-H "Content-Type: application/json" \
-d '{
"model": "z-ai/glm-5.2",
"messages": [{"role": "user", "content": "Explain prompt caching in one sentence."}]
}'
```
Browse the available model IDs on the [Active Models](/docs/inference/active-models) page, or let [Smart Routing](/docs/inference/routing) pick one for you.
## Streaming
Set `stream: true` to receive tokens as they are generated. This is the recommended mode for anything user-facing or long-form, it lowers time-to-first-token and keeps the connection active for the full generation.
```python theme={null}
stream = client.chat.completions.create(
model="z-ai/glm-5.2",
messages=[{"role": "user", "content": "Write a short story."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
```
## Timeouts and long-running requests
The platform allows a single request up to **300 seconds** of generation time. Large context windows or long outputs on a non-streaming call can genuinely take minutes, so the most common source of "cut off" errors is a **client-side timeout that fires before the response finishes**, not the server.
For long outputs, **stream the response**. Streaming keeps data flowing over the connection, which avoids idle/read timeouts and gives you tokens as they arrive instead of waiting for the whole completion.
If you must run non-streaming for long generations, raise your client's read timeout:
```python theme={null}
client = OpenAI(
api_key="lk_...",
base_url="https://api.lyceum.technology/api/v2/external/serverless",
timeout=600.0, # seconds; default in most SDKs is 600, but some frameworks set it lower
)
```
Some frameworks (for example litellm) apply a separate **stream inactivity timeout** that aborts a stream if no new token arrives within a fixed window. Under heavy load, time-to-first-token can spike, so if you stream through such a framework, raise its inactivity timeout as well as the overall request timeout.
## Function calling and tools
All chat models support **function / tool calling** using the standard OpenAI `tools` and `tool_choice` parameters.
```python theme={null}
resp = client.chat.completions.create(
model="z-ai/glm-5.2",
messages=[{"role": "user", "content": "What is the weather in Berlin?"}],
tools=[{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}],
tool_choice="auto",
)
print(resp.choices[0].message.tool_calls)
```
A small number of models prefer to answer in text rather than call a tool when `tool_choice` is left on `"auto"`. If you find a model returning prose instead of a tool call, set `tool_choice="required"` to force it.
## Prompt caching
Prompt caching is **automatic**, there is nothing to enable. When consecutive requests share an identical leading prefix (for example a fixed system prompt or a large shared context), the cached portion is reused and billed at a reduced input rate.
To benefit, keep the stable part of your prompt at the **front** and vary only the tail (the user's latest message). Caching is best-effort, a request may or may not hit a warm cache depending on recent traffic.
Cache hits are reported in the usage object:
```python theme={null}
resp = client.chat.completions.create(model="z-ai/glm-5.2", messages=[...])
print(resp.usage.prompt_tokens) # total input tokens
print(resp.usage.prompt_tokens_details.cached_tokens) # portion served from cache
```
## Reasoning models
Some models expose an explicit reasoning mode. For hybrid models that support it, toggle it with `enable_thinking` in the request body:
```python theme={null}
resp = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V4-Pro",
messages=[{"role": "user", "content": "Solve step by step: ..."}],
extra_body={"enable_thinking": True},
)
```
Always-on reasoning models generate their reasoning by default and do not accept the flag. When measuring latency or token usage on a reasoning model, remember that reasoning tokens count toward output.
## Embeddings
Text embeddings use the same base URL and key:
```python theme={null}
emb = client.embeddings.create(
model="Qwen/Qwen3-Embedding-8B",
input="text to embed",
)
print(len(emb.data[0].embedding))
```
## Error handling
Errors are returned in the OpenAI-compatible shape, an `error` object with `message`, `type`, and `code`:
```json theme={null}
{
"error": {
"message": "...",
"type": "invalid_request_error",
"code": 400
}
}
```
This holds for both pre-stream errors and errors that occur mid-stream, so a client that classifies errors by `error.type` works the same way here as against the OpenAI API. Retry on `429` (rate limit) and `5xx` / timeout responses with backoff.
## Serverless vs dedicated
| Use case | Pick |
| ---------------------------------------------------------------- | ------------------------------------------------ |
| Low-volume or spiky traffic, pay-per-request, no infra to manage | Serverless (this page) |
| Steady production traffic, predictable latency, your own model | [Dedicated Inference](/docs/inference/dedicated) |
| Let the platform choose a model per request by difficulty | [Smart Routing](/docs/inference/routing) |
Browse the model IDs and context windows currently available.
Automatically route each request to the right model.