Kimi K3 API State: Tools, Caching, and Failure Modes

Your first Kimi K3 call works. The model reasons, calls a tool, and returns the right answer.

Then you add the second turn.

Part two explained state inside the model. This post follows the state your application must carry between API calls.

Your conversation store keeps role and content, just like the chat schema you have used for years. The next request still looks valid. It is also missing the reasoning state and tool-call object that K3 told you to replay.

Reasoning state is the returned context K3 expects to see again on later turns. It belongs to the protocol even when your product never displays it.

Use a case-file mental model. Each API request arrives with no conversation file attached unless your client sends it. The visible answer is one page; returned reasoning, tool requests, and tool results are the other pages.

This is the production boundary most quickstarts understate:

The Kimi API does not keep your conversation. Your client creates continuity by replaying the complete assistant message, including reasoning_content, content, and tool_calls.

The request boundary is conversation-stateless: the client must replay the conversation. A provider-managed prefix cache can still reuse computation for an identical beginning of the prompt. “Stateless” therefore describes the client contract, not every internal provider optimization.

Evidence labels in this post are narrow:

• Official fact: Moonshot's documented request or response contract.
• Derived calculation: a direct implication of documented numbers.
• Operational estimate: a workload-dependent capacity or cost projection.
• Hypothesis: a predicted undocumented symptom. Hypotheses stay labelled; the simulator does not invent an error string.

Open the simulator in “content only” mode, then compare it with complete replay. The useful observation is not a fabricated error string; it is which fields the client must persist before the next request can be valid.

After the simulator, apply one rule: if the API returned a replay-required field, preserve it unchanged until the next request is serialized. The widget shows field custody, not a guaranteed server response to broken custody.

The full assistant message is the unit of conversational state

The Kimi K3 quickstart states that multi-turn conversations and tool calls must append the complete assistant message returned by the API. Keeping only content is not enough.

A conceptual assistant response has three separate payloads:

{
  "role": "assistant",
  "reasoning_content": "internal reasoning returned by the API",
  "content": null,
  "tool_calls": [
    {
      "id": "call_123",
      "type": "function",
      "function": {
        "name": "get_weather",
        "arguments": "{\"city\":\"San Francisco\"}"
      }
    }
  ]
}

The fields serve different purposes:

• reasoning_content carries the model's returned reasoning history.
• content carries the visible final answer when one exists.
• tool_calls carries structured requests that the application must execute.

The reasoning-effort documentation and multi-turn guide repeat the preservation requirement. Omitting thinking history violates the documented Preserved Thinking contract; Moonshot does not promise one deterministic error response for that client bug.

The exact HTTP or model symptom after a client drops reasoning_content is not documented as one deterministic error. Do not invent one. The contract itself is explicit enough to shape storage:

Persist the assistant object you received.
Do not reconstruct a smaller assistant object from selected fields.

This changes event schemas. A chat table with only role and text cannot round-trip the protocol.

I prefer an application protocol schema only when captured-response tests prove round-trip fidelity. Raw storage is safer during protocol churn, but it needs a deliberate retention and privacy policy.

A tool call spans several messages

Tool calling is a protocol where the model asks the application to perform a named operation with structured arguments. It is not one request with a callback hidden inside the SDK.

The documented message order, plus one application safeguard, is a custody chain:

The tool-calling guide requires the application to append the assistant message before tool results and to return one result for each call. The idempotency key in step three is an application safeguard for mutating tools, not a Moonshot message field.

The example is pinned to openai==2.52.0, the version rechecked on August 3, 2026:

python3 -m pip install 'openai==2.52.0'

Optional implementation depth: the timeline above is the protocol. The loop below adds storage, validation, retry, and terminal-state guards around that protocol.

Here is a bounded Python loop:

import json
import os
 
from openai import OpenAI
 
 
def parse_weather_arguments(raw_arguments: str) -> str:
    arguments = json.loads(raw_arguments)
    if not isinstance(arguments, dict) or set(arguments) != {"city"}:
        raise ValueError("Expected exactly one city argument")
    city = arguments["city"]
    if not isinstance(city, str) or not city.strip():
        raise ValueError("city must be a non-empty string")
    return city
 
 
client = OpenAI(
    api_key=os.environ["MOONSHOT_API_KEY"],
    base_url="https://api.moonshot.ai/v1",                 # ①
)
 
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
            "additionalProperties": False,
        },
    },
}]
 
messages = [
    {"role": "user", "content": "What is the weather in San Francisco?"}
]
reasoning_effort = "high"                               # ②
seen_call_ids: set[str] = set()
 
for step in range(8):
    response = client.chat.completions.create(
        model="kimi-k3",
        messages=messages,
        tools=tools,
        tool_choice="required" if step == 0 else "auto",
        reasoning_effort=reasoning_effort,
        max_completion_tokens=16_384,
    )
 
    choice = response.choices[0]
    assistant = choice.message
    assistant_message = assistant.model_dump(          # ③
        mode="json",
        exclude_none=True,
    )
    messages.append(assistant_message)
 
    if choice.finish_reason == "length":
        raise RuntimeError("Kimi K3 completion truncated")
 
    if assistant.tool_calls:
        if choice.finish_reason != "tool_calls":
            raise RuntimeError(
                f"Tool calls ended with: {choice.finish_reason}"
            )
    else:
        if choice.finish_reason != "stop":
            raise RuntimeError(
                f"Unexpected terminal reason: {choice.finish_reason}"
            )
        if not assistant.content:
            raise RuntimeError("Kimi K3 returned no final content")
        print(assistant.content)
        break
 
    for call in assistant.tool_calls:
        if call.function.name != "get_weather":
            raise RuntimeError(f"Unexpected tool: {call.function.name}")
        if call.id in seen_call_ids:
            raise RuntimeError(f"Duplicate tool call ID: {call.id}")
        seen_call_ids.add(call.id)
 
        city = parse_weather_arguments(call.function.arguments)
        result = {
            "city": city,
            "weather": "demo fixture",
            "temperature_c": 24,
        }
 
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,
            "name": call.function.name,
            "content": json.dumps(result),
        })
else:
    raise RuntimeError("Tool loop exceeded step limit")

① Without Moonshot's base_url, the OpenAI SDK sends the request to the wrong service. Moonshot lists this as a common model_not_found cause in its troubleshooting guide.

② Keep reasoning effort stable inside a cache-sensitive session.

③ Serialize the complete SDK message at the pinned boundary, then append that JSON-compatible object before inspecting or executing calls. Captured response tests should prove the serialization round trip when the SDK changes.

The weather value is a deterministic local fixture, not a live observation. Replace that block with an authorized provider in a real application.

The loop also adds controls that the quickstart does not owe you:

• a maximum number of agent steps;
• an allowlist of expected tool names;
• JSON argument parsing;
• explicit handling for truncated output;
• rejection of unexpected terminal states;
• one result per tool-call ID.

Production code should validate arguments against the same schema supplied to the model. Model-generated JSON is input to your application, not trusted configuration.

required can force tool use; a named function cannot

K3 supports these tool_choice values:

The last option conflicts with thinking mode. Moonshot's tool-choice documentation documents this error:

400: tool_choice 'specified' is incompatible with thinking enabled

K3 always reasons, so use required when the first turn must retrieve or act. Then return to auto after the mandatory step.

This limitation matters for routers. If your application needs exactly one specific internal function, enforce that in your own controller:

1. declare only the allowed function for that step;
2. set tool_choice="required";
3. reject any unexpected name;
4. execute only after argument validation.

The API decides generation. Your application still owns authorization.

Dynamic tools preserve context by appending definitions

Large agents can waste context by sending hundreds of tool schemas on every request. Dynamic tool loading appends a complete tool definition when the conversation needs it instead of placing every schema at the beginning. K3 uses a system message for that declaration.

{
  "role": "system",
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "create_github_pr",
        "description": "Create a pull request",
        "parameters": {
          "type": "object",
          "properties": {},
          "additionalProperties": false
        }
      }
    }
  ]
}

The dynamic-tool documentation sets four constraints:

• the message contains the complete tool definition;
• it omits content;
• the tool becomes available from that position onward;
• the server does not retain the declaration, so later requests must replay it.

Appending definitions helps prefix caching because earlier history stays unchanged. Inserting a tool definition into old history invalidates cache reuse after that insertion point.

Moonshot documents two concrete failures:

• adding content to the dynamic-tool message returns HTTP 400;
• sending the K3 mechanism to K2.6 can produce tokenization failed.

Those are API outcomes. The broader recommendation is an engineering judgment: retrieve tool definitions just before they are needed, but keep authorization and selection logic outside the model.

Reasoning effort is session configuration

Reasoning effort is the per-session setting that asks K3 to spend less or more generation budget on reasoning. K3's top-level reasoning_effort field accepts:

low
high
max

The default is max. Thinking cannot be disabled.

Moonshot's launch blog said low and high effort would arrive later. The current API reference documents all three levels as available on August 3, 2026. The API reference is the operative contract; the launch post is historical context.

Moonshot does not publish a fixed token budget, latency multiplier—the change in elapsed request time—or accuracy delta for each value. Do not encode “high means 2× tokens” into a cost model without your own measurements.

The hosted K3 API also fixes several sampling parameters. The model-parameter reference lists:

Different values produce request errors. Omit fixed fields instead of copying sampling settings from another provider.

The reasoning choice affects more than output. Moonshot documents that changing reasoning_effort invalidates prefix-cache reuse. Choose it before a conversation begins when stable caching matters.

Completion limits constrain reasoning and visible output together

max_completion_tokens is the ceiling shared by returned reasoning and visible output. It is not a promise that the model will emit that many tokens.

The K3 quickstart documents:

• default max_completion_tokens: 131,072;
• maximum configurable completion: 1,048,576;
• effective maximum: context window minus prompt tokens.

Reasoning and visible final content share that completion budget. If the budget ends during reasoning, finish_reason—the API's terminal-status field—can be length while visible content is still empty.

That changes error handling:

choice = response.choices[0]
 
if choice.finish_reason != "stop":
    raise RuntimeError(
        f"Incomplete Kimi K3 response: {choice.finish_reason}"
    )

For a tool turn, tool_calls is an expected terminal reason for that request. For a final structured response, length is failure even when the partial text looks parseable.

The limit also affects rate admission. Moonshot's platform introduction says token rate checks reserve prompt tokens plus the requested, or default, completion limit. Billing uses actual generated tokens. A request that emits 500 tokens can still reserve a much larger token-per-minute allowance if you leave the 131,072 default untouched.

Set an explicit limit from the task's failure envelope, not the model maximum.

Prefix caching rewards immutable leading history

K3 prefix caching is automatic. The service can reuse work for an identical leading sequence of tokens; there is no cache ID, client-managed lifetime, or create/delete operation.

The context-caching guide documents these rules:

• the previous request's prompt must exceed 256 tokens;
• matching uses identical leading context;
• appending new messages preserves the existing prefix;
• changing, deleting, or inserting earlier messages invalidates reuse after that point;
• tool_choice and response_format changes do not invalidate the prefix;
• changing reasoning_effort does;
• usage.cached_tokens reports reused input.

Derived calculation. The wording says “exceed 256,” so behavior at exactly 256 tokens is not explicit. Treat 257 as the first documented eligible length.

A cache-friendly message layout looks like:

[long stable system/document prefix]
[stable dynamic-tool definitions already introduced]
[chronological complete user/assistant/tool history]
[new user message or newly appended tool definition]

Do not claim a fixed cache lifetime or guaranteed hit. Record usage.cached_tokens and calculate actual savings.

Small prompt edits can have large economic effects. Moving a timestamp near the front of the system message changes the prefix early. Appending the timestamp near the end preserves more reusable tokens.

Structured output constrains final content, not reasoning

Structured output constrains the shape of the final generated content. K3 supports JSON mode and strict schema output.

JSON mode:

{"type": "json_object"}

guarantees a JSON object, not its field names or types.

Strict output:

response_format={
    "type": "json_schema",
    "json_schema": {
        "name": "person",
        "strict": True,
        "schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "age": {"type": "integer"},
            },
            "required": ["name", "age"],
            "additionalProperties": False,
        },
    },
}

uses constrained decoding against Moonshot Flavored JSON Schema (MFJS), Moonshot's supported JSON Schema dialect. The structured-output guide recommends strict: true, explicit required fields, and additionalProperties: false.

Parse message.content, not reasoning_content and not the entire response. Then validate again in your application. Constrained generation protects shape; it does not grant authorization or semantic correctness.

If finish_reason="length", the object can be incomplete. Reject it before parsing as a successful business result.

A stream is complete only after [DONE]

Streaming sends a response in incremental chunks instead of one completed object. K3 streams reasoning and final content through separate deltas. Tool arguments may also arrive in fragments.

The K3 quickstart's streaming example requires the client to continue until the terminal server-sent events (SSE) [DONE] marker. Seeing finish_reason="stop" in a chunk is not enough to declare the transport complete.

Your stream state machine therefore needs:

reasoning delta
content delta
tool-call delta
finish reason
[DONE]

If the connection closes before [DONE], mark the response incomplete. Do not silently save the visible prefix as a successful answer.

Retries need idempotency at the application layer. A transport retry can cause the model to issue a tool call again. Tools that create tickets, send messages, or mutate data need request IDs or deduplication keys.

Hosted and self-hosted K3 expose different products

Hosted means Moonshot runs the model and API. Self-hosted means you run the released weights through your chosen inference engine, the runtime that schedules model execution.

“OpenAI-compatible” describes an API shape. It does not promise identical features.

The vision guide documents image and video input through base64 data URLs or Moonshot file IDs. It does not accept arbitrary public image URLs. SVG is rejected as image input; send SVG XML as text.

The SGLang K3 cookbook documented image-only preprocessing at launch. The vLLM recipe also warned that the model could emit a tool-call form its parser did not expect.

Pin the hosted model ID or the self-hosted model, engine, image, parser, and chat-format revisions separately. “Kimi K3” is not one operational surface.

Rate limits can fail before cost becomes the problem

Rate limits cap admitted work over time. The table uses requests per minute (RPM), tokens per minute (TPM), and tokens per day (TPD).

Kimi's rate-limit page listed these tiers for the Global Kimi Open Platform (platform.kimi.ai, API endpoint api.moonshot.ai) on August 3, 2026:

The limits apply at user level across keys and models. Moonshot may also adjust temporary limits under cluster pressure.

Derived calculation. One 1M-token request cannot pass a 500K TPM tier. Large completion reservations can trigger a 429 before actual generation reaches that size.

Operational estimate. Sustainable request rate still depends on observed service time, retries, prompt size, and completion reservation. The tier table alone cannot predict it.

Classify 429s. The error documentation distinguishes capacity overload from account limits. Increasing a recharge tier does not fix an overloaded model cluster.

Automatic SDK retries can multiply one logical tool step into several requests. Record request IDs, retry counts, and error classes instead of reporting only final latency.

Production failures cluster at protocol boundaries

Two important contract violations do not have one documented deterministic symptom:

• replaying only assistant content;
• omitting one of several required tool results.

Treat both as client bugs. Do not wait for a particular error string.

Ship the integration with these invariants

1. Persist complete assistant messages. Round-trip reasoning, content, and tool calls.
2. Correlate every tool result. One result per exact tool_call_id.
3. Bound agent steps. Infinite tool loops are an application failure.
4. Fix reasoning effort per session. Change it intentionally and expect a cache reset.
5. Set a task-sized completion limit. The default can consume rate admission far beyond expected output.
6. Treat length as failure. This applies to visible text and strict JSON.
7. Wait for [DONE]. A stream is not complete before its terminal marker.
8. Measure cached tokens. Do not use Moonshot's reported coding-workload cache rate as your own.
9. Validate model output twice. Constrained decoding plus application validation.
10. Pin every compatibility layer. Model, SDK, engine image, parser, and chat format.

K3's API is not difficult because it has many fields. It is difficult because those fields cross storage, tools, caches, rate limits, and retries.

If your client cannot replay the exact assistant message, it does not yet support a Kimi K3 conversation.

Part four opens the checkpoint itself: packed FP4 values, group scales, BF16 modules, and metadata produce about 1.56 TB from 2.8T logical parameters.

Sources and References

Conversation and reasoning

• K3 request and response examples: Kimi K3 quickstart
• Multi-turn message replay: Multi-turn conversations
• Reasoning effort contract: Reasoning effort
• Fixed model parameters: Model parameter reference

Tools, caching, and output

• Tool-call sequence: Complete tool calls
• Tool choice constraints: Tool choice
• Dynamic tool loading: Dynamic tools
• Prefix caching: Context caching
• Strict schema output: Structured output
• Streaming completion: Kimi K3 quickstart

Operations

• Rate limits: Recharge and rate limits
• Errors and overload: API errors
• Common integration mistakes: Troubleshooting
• Hosted media input: Vision input
• Self-hosted K3 behavior: SGLang cookbook
• vLLM parser warning: vLLM K3 recipe