Serverless inference offers pay-per-request pricing with no idle compute cost, the right choice for low-volume or spiky traffic where reserving GPU replicas would be wasteful.The endpoints are OpenAI-compatible. Point any OpenAI client at the base URL below, swap in a Lyceum API key and a model name, and everything else, streaming, tool calling, usage accounting, works the same.
Set stream: true to receive tokens as they are generated. This is the recommended mode for anything user-facing or long-form, it lowers time-to-first-token and keeps the connection active for the full generation.
stream = client.chat.completions.create( model="z-ai/glm-5.2", messages=[{"role": "user", "content": "Write a short story."}], stream=True,)for chunk in stream: delta = chunk.choices[0].delta.content if delta: print(delta, end="", flush=True)
The platform allows a single request up to 300 seconds of generation time. Large context windows or long outputs on a non-streaming call can genuinely take minutes, so the most common source of “cut off” errors is a client-side timeout that fires before the response finishes, not the server.
For long outputs, stream the response. Streaming keeps data flowing over the connection, which avoids idle/read timeouts and gives you tokens as they arrive instead of waiting for the whole completion.
If you must run non-streaming for long generations, raise your client’s read timeout:
client = OpenAI( api_key="lk_...", base_url="https://api.lyceum.technology/api/v2/external/serverless", timeout=600.0, # seconds; default in most SDKs is 600, but some frameworks set it lower)
Some frameworks (for example litellm) apply a separate stream inactivity timeout that aborts a stream if no new token arrives within a fixed window. Under heavy load, time-to-first-token can spike, so if you stream through such a framework, raise its inactivity timeout as well as the overall request timeout.
All chat models support function / tool calling using the standard OpenAI tools and tool_choice parameters.
resp = client.chat.completions.create( model="z-ai/glm-5.2", messages=[{"role": "user", "content": "What is the weather in Berlin?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }], tool_choice="auto",)print(resp.choices[0].message.tool_calls)
A small number of models prefer to answer in text rather than call a tool when tool_choice is left on "auto". If you find a model returning prose instead of a tool call, set tool_choice="required" to force it.
Prompt caching is automatic, there is nothing to enable. When consecutive requests share an identical leading prefix (for example a fixed system prompt or a large shared context), the cached portion is reused and billed at a reduced input rate.To benefit, keep the stable part of your prompt at the front and vary only the tail (the user’s latest message). Caching is best-effort, a request may or may not hit a warm cache depending on recent traffic.Cache hits are reported in the usage object:
resp = client.chat.completions.create(model="z-ai/glm-5.2", messages=[...])print(resp.usage.prompt_tokens) # total input tokensprint(resp.usage.prompt_tokens_details.cached_tokens) # portion served from cache
Always-on reasoning models generate their reasoning by default and do not accept the flag. When measuring latency or token usage on a reasoning model, remember that reasoning tokens count toward output.
This holds for both pre-stream errors and errors that occur mid-stream, so a client that classifies errors by error.type works the same way here as against the OpenAI API. Retry on 429 (rate limit) and 5xx / timeout responses with backoff.
