❯ ls -la ~/blog
179 posts · 18 series
- datetitle
- 2026-09-21
FastLLM plus a DFlash2 draft runs Qwen3.8-27B FP8 at 153.8 tok/s on code across two modded 2080 Tis. Prose drops to 53.1. Full recipe and four traps.
- 2026-09-21
Ternary Bonsai 2 27B scored 130/140 on one modded 2080 Ti vs 131 for Qwen3.8-27B Q8_0 on two cards. Build steps, the flag that matters, why smaller is slower.
- 2026-09-20
A clickable INT4 / FP8 / FP4 lab: one outlier drops two of four weights to zero, and the scale itself is quantized, so the group maximum is off by 4.77e-7.
- 2026-09-16
A cube you pull apart into layers. What model.ai-muninn.com shows for Qwen3-30B-A3B, Qwen3-8B, Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash, and how to read it.
- 2026-09-14
A DGX Spark cut power twice under sustained video generation. I reproduced it on purpose, logged the curve off-box, and stopped it with a 2200 MHz clock cap.
- 2026-09-14
Drafting with MiniMax-H3 and refining with LTX-2.5 takes 118.8 s on a DGX Spark against 237.7 s native, and 73.3 s warm. Full ComfyUI setup, and the version set it needs.
- 2026-09-14
A modded RTX 2080 Ti 22G renders 1344x768 with audio in 165 s using MiniMax-H3 to draft and LTX-2.5 to refine. Six departures from the Blackwell workflow.
- 2026-09-13
NVIDIA's Sol-H3 two-stage recipe on an RTX 5090: MiniMax-H3 drafts at 672×384 in 25.5 s, LTX-2.5 refines only the take you keep. Downloadable ComfyUI workflow.
- 2026-09-12
A second DGX Spark takes Qwen3.8-Flash-Next NVFP4 from 41.7 to 51.9 tok/s and the KV pool to 2.07x, plus the ConnectX-7 cabling steps the playbook leaves out.
- 2026-09-11
Preset TTS voices don't sound like you. Zero-shot cloning with Qwen3-TTS and mlx-audio needs 24 seconds of reference audio and ten lines of Python, all local.
- 2026-09-11
LTX-2.5 and MiniMax-H3 on one RTX 5090, same Simplified Chinese line: 28.67s vs 81.22s end to end, both at CER 0%. Files, params, VRAM ceiling.
- 2026-09-06
NVIDIA's NVFP4 checkpoint at 41.7 tok/s on one DGX Spark via nine bind-mounted vLLM files, plus six figures on why the 47.68 GiB n-gram table lives on NVMe.
- 2026-09-06
Ref2VA is the only MiniMax-H3 mode that takes several reference images. Two characters, 243 frames, 437 s on one RTX 5090, and CER 6.8% on the dialogue.
- 2026-09-03
MiniMax-H3 at native 1344x768 on a single RTX 5090: 15 seconds of video in 518 s. What KJNodes chunking, torch cu130 and the NVFP4 kernels are each worth.
- 2026-08-31
A COLM 2026 paper strips every style signal from 61,608 stories and still tells human from AI at 93.2% macro-F1. The tell is structure, not em-dashes.
- 2026-08-31
Ten community effect embeddings for MiniMax-H3, tested at native 1344x768 on one RTX 5090. What each one actually does, the prompt shape that lets them work, and the placement rule that decides whether they fire at all.
- 2026-08-30
A 320B-A18B MoE at UD-Q2_K_XL fits in 121 GB. 15.40 tok/s on English prose, 16.23 on Chinese, 23.64 on Three.js code. Vision works only if you drop --spec-type.
- 2026-08-28
Qwen3.8-Flash-Next UD-Q4_K_XL is 104 GiB. GB10 holds all of it plus a 262,144 context in one 121 GB pool. 76.65 tok/s on a file edit, 20.51 on prose.
- 2026-08-28
Qwen3.8-Flash-Next (176.94B, arch qwen4exp) runs on 3× modded 2080 Ti 22G at 23.14 tok/s at 128K context — and 3.51× that on a file edit, with no draft model.
- 2026-08-23
GB10 gives you 273 GB/s and the model reads 18.77 GB per token, so the ceiling is 14.54 tok/s. Measured 12.3 — 85% of it. Kernel swaps can't recover the rest. Speculative decoding gets 5.28x, and the win came from one flag.
- 2026-08-23
NCCL died on two modded 2080 Tis with 'CUDA driver is a stub library'. Building v2.31.2 from source fixed it: PCIe traffic down 99%, code generation unchanged.
- 2026-08-23
Measured with nvidia-smi dmon on two modded 2080 Tis: AllReduce costs 67 MB/s at decode and 973 MB/s at prefill, and the bus peaks at 8.2% of Gen3 x16.
- 2026-08-23
Three mistakes shooting a wuxia scene in MiniMax-H3: a 502 that wasn't a rejection, blur upscaling can't fix, and a prompt that described a face instead of naming one.
- 2026-08-22
Two modded 22GB RTX 2080 Tis hit 59.6 tok/s on Qwen3.8-27B using llama.cpp's -sm tensor split mode plus MTP, after -sm row was deleted upstream. Config, gotchas and the failed routes included.
- 2026-08-21
Dissecting 41 Codex sessions turned up a wall-clock formula (tool calls x 14.3s) and cut a stuck ticket from 41 minutes down to 73 seconds.
- 2026-08-21
A frozen GGUF chat template was killing my KV cache; a community fix plus MTP draft depth 4 took Qwen3.8-27B from 41.6 to 47.6 tok/s on a 2080 Ti 22GB.
- 2026-08-20
Quantized weights are half the file size on a 2080 Ti but the speed doesn't move. Dumping the CUDA backend's .so with nm -D shows why, and which quant format is actually accelerated.
- 2026-08-18
A modded 22GB RTX 2080 Ti runs Qwen3.8-27B at 30-31 tok/s with 128K context. The full llama.cpp command, the MTP flags, and the thinking-budget trap.
- 2026-08-15
The open models are landing thick and fast right now, and my gallbladder picked exactly this moment to give out. Nothing new here for a couple of weeks.
- 2026-08-07
Fifteen seconds of 1080p video with audio in 741s on a GB10 DGX Spark. Swapping Real-ESRGAN for SPAN saved 282s, and nvidia-vfx ships x86_64 wheels only — nothing for ARM.
- 2026-08-06
The four files are 38 GiB on disk; the card has 22. A modded 2080 Ti 22G still renders 15s of 1080p with audio in 23 minutes. Full config, measured speed and quality, then how it got there.
- 2026-08-06
Three stacked changes took a 15-second 1080p MiniMax-H3 render from 625s to 314s on one RTX 5090: 14 steps, SageAttention 2.2.0, and RTX VSR replacing Real-ESRGAN.
- 2026-08-04
Beginner walkthrough for MiniMax-H3, the 33B model that generates video and stereo audio in one pass. Full precision is 115 GB; quantized it fits one RTX 5090. What to download, where it goes, how to prompt it.
- 2026-08-02
DeepSeek-V4-Flash-0731 on a DGX Spark: 17.5-17.9 tok/s at 256K context. Unified memory removes the offload decision entirely — and it is only 7% faster than a used 2080 Ti.
- 2026-08-02
How to run DeepSeek-V4-Flash-0731 (91GB, 284B MoE) on a single modded 22GB 2080 Ti at 16.5 tok/s and 1M context — including a formula for picking -ncmoe.
- 2026-07-29
A draft cleared fact-check and two rounds of native-speaker review, then one reader caught it in a sentence. The check that should have caught it was labelled manual, so it had never run. Here's the script that replaced it, and how I calibrated the thresholds.
- 2026-07-29
DeepSeek-V4-Flash is 284B. It decodes at 17.4 tok/s on a single modded 22GB 2080 Ti — faster than the DGX Spark I serve it on. Card count barely matters, and MTP speculative decoding dies on unimplemented runtime, not a missing GPU.
- 2026-07-27
My AI agent's durable memory auto-saved a wrong answer, then cited it back as fact — outranking the corrected truth. The three-part pathology, and why the fix is ranking memory, not adding more of it.
- 2026-07-25
Part 9's Entrpi engine hit 20 tok/s. Days later it OOM-crashed under real agent traffic. Root cause: two copies of the weights in memory. I rolled back.
- 2026-07-23
A GGUF isn't uniform precision. Inspect per-tensor types, then quantize only the tensors you choose with llama-quantize — pin every other family back to its own type so it copies through untouched, and never stack requant error.
- 2026-07-22
Got one 24GB consumer GPU (or a modded 2080 Ti 22G)? My current top free open-source pick is Huihui-ThinkingCap-Qwen3.6-27B-abliterated: Q4_K_S ~16GB, half the thinking tokens, ~38 tok/s with MTP, almost never refuses, all Apache-2.0.
- 2026-07-22
Poolside Laguna S 2.1, a 118B-A8B coding MoE, on one 22GB 2080 Ti via CPU/GPU hybrid offload plus a companion DFlash speculative-decoding draft at ~29 tok/s; attention-Q8 saves ~2.45 GiB, +7% decode.
- 2026-07-21
Part 11 said a haywire assistant is usually a broken car (tools, config, memory), not a dumb engine (the model). This is that checklist: a context_length set at the wrong level silently compresses early, Qwen thinking left on runs 10x slower, an MCP tool that connects but every call fails, and a sib running a different model than you think. Five real config gotchas, each with a check you can hand to your agent to run on itself, plus the fix.
- 2026-07-21
Asked Codex if my ds4 repo had a single-DGX-Spark optimization; found Entrpi. Swapped the engine, not the model — decode 14-16 to about 20 tok/s, prefill about 2×.
- 2026-07-21
A US$1.6k junk EPYC + 3× 2080 Ti 22G box (66G VRAM) runs a quantized 119B MoE at 74 tok/s. Expert offload to RAM costs about 2.4× decode — plus a multi-card OOM gotcha.
- 2026-07-18
The same modded 2080 Ti 22G ran Z-Image 3.4× slower on one machine than another. Not the hardware, not the OS, not a missing package — a Turing bf16→fp32 fallback hiding in one log line. One flag fixed it.
- 2026-07-17
One of my AI agents was auto-loading 193 skills into a 2% context budget, silently truncating every description. The fix was visibility governance, not deletion — an allowlist, thin-shell skills, and three layers that stop it re-bloating.
- 2026-07-16
Six posts in, my AI setup is really two axes of one system: durable knowledge and live task state, both in plain files. Here's how they converge so different AIs hand off work without re-explaining it or losing it.
- 2026-07-15
Search finds an AI agent's notes but hands back raw material to re-derive each session. I distill ~600 files into canonical claims — the goal is ending re-explanation, not enforcing agreement.
- 2026-07-14
My AI's long-term memory is ~600 markdown files in three layers: the files are the source of truth, a search engine makes them findable, and a knowledge graph links them by concept. Here's the design, why each layer exists, and the wrong turns I took building it.
- 2026-07-12
I set out to retire a 'dead' shared-memory folder my knowledge base had replaced. It was writing to itself at 2:30 AM. Three traps in doctrine-vs-runtime drift, and why you map a memory system's topology from the hub, not the spoke.
- 2026-07-12
How I hand engineering tasks to Codex: self-contained ticket files with machine-checkable acceptance, background execution, three flags I paid for in dead time, and a report-is-not-a-result gate.
- 2026-07-11
When an AI coding CLI runs out of quota mid-task, what dies isn't your knowledge base — it's the live state trapped in the session. A file-based handoff protocol lets Claude Code and Codex resume each other's half-finished work.
- 2026-07-10
A local brain on a headless Windows box kept dying under real load: the client caught a brief 503, the service quietly restarted itself, and the application log was blank. The worst part wasn't the crash — it was that my own log truncated the one line that explained it on restart. This is the hunt to dig that reason back out from under its own log: why a self-restarting service is the best at burying the reason for its own crash, why you go to the OS-layer Event Log first, and what 0xc0000409 actually means. Full disclosure: I haven't 100% pinned the root cause — this is an open investigation, not a closed case.
- 2026-07-10
Most hyped LLM optimizations don't survive contact with your own model and hardware. Three cheap checks — read the source, find the true ceiling, run one discriminating experiment — that tell you which are real, anchored in the FlashMemory investigation on DeepSeek-V4-Flash.
- 2026-07-10
Hermes agent, self-hosted from a desktop install to a local-model fleet: ChatGPT OAuth (no API key), Telegram/LINE control, local models, on Mac and Windows.
- 2026-07-09
V4-Flash already ships a native lightning indexer tracking true attention at 93–96%. FlashMemory pre-filters candidate chunks, but it makes near-random selections on my Q2 and reaches only 89–92% when retrained — still a NO-GO on GB10.
- 2026-07-07
A current 2026 guide to running local AI on DGX Spark: vLLM, official Gemma 4 NVFP4 weights, MTP, long-context multimodal options, and the traps still worth avoiding.
- 2026-07-07
DeepSeek-V4-Flash is a 284B MoE that stays fast at long context on a 128GB GB10 because its hybrid CSA/HCA attention and lightning indexer shrink the KV cache to ~871MiB at 64K and read only a few hundred compressed rows per step. What I found reading ds4's code and DeepSeek's V4 paper.
- 2026-07-07
A map for running a self-hosted AI agent: desktop body, ChatGPT OAuth brain, Telegram or LINE channels, tools, and local LLMs when you want autonomy.
- 2026-07-06
Depth-1 MTP speculative decode on DeepSeek-V4-Flash is lossless but workload-dependent on a GB10: +9.4% on agent turns and +6.5% on code, −3.6% on prose and −3.7% on Chinese chat. The sign follows the acceptance rate because decode here is verify-bound (108ms verify vs 4ms draft). It's not a global faster switch — route it by workload.
- 2026-07-05
A month after wiring DeepSeek-V4-Flash into a daily agent, it felt slow again. Two log lines explained it: the disk-KV cache was evicting live prefixes (hits=0), and tool-call turns never saved a checkpoint — so common=268 out of 14209 and every turn re-paid full prefill. The fix: 256G KV budget + PR #489.
- 2026-07-03
I swapped my home agent's brain for one that decodes 30-40 tok/s instead of 14, and it felt slower. The number I'd stared at for a year — tok/s — only measures how fast tokens come out, not how long before they start. On a hybrid model, a single cache miss re-prefills the entire prompt: same box, same brain, 2.6s warm vs 216s cold. Here's the live log.
- 2026-07-02
To ease the wait on a pokey local agent, I turned on Telegram streaming — which, the way this bot did it, means rewriting the same message every fraction of a second. On a 14 tok/s brain, a single 175-second reply works out to an estimated couple hundred edit requests, which slammed into Telegram's flood control and got the whole bot benched for four minutes — final answer included. The short, ugly lesson: slow models should not fake streaming with edits. Send the finished answer once. Live logs inside.
- 2026-07-01
Hermes has a /learn command that turns 'something you just did' into a reusable skill — a SKILL.md. I wired it into my own fleet: one Kanban card, a local 27B running on a modded 2080 Ti, and about 3 minutes later it handed back a clean, spec-compliant skill — plus two implementation details the docs don't spell out (slash command vs. dispatch, and where skills actually live). A plain-language walkthrough of what /learn does, how to use it, and where its limits are.
- 2026-06-29
After raising a fleet of assistants, I gave them the same one-line 'make a Tetris game' spec — no details at all — one card each, and let them each write a web Tetris in a single shot. I touched zero lines of game code; I only published the result. The surprise: from that one line, the Hermes harness plus a local model I tuned myself (on a modded 2080 Ti) filled in things I never asked for — a ghost piece and wall-kick — in one shot. You can play all three.
- 2026-06-29
Training with ai-toolkit on Windows + RTX 5090 hit three walls before it even started: Python 3.13 dependency hell, a HuggingFace download frozen at 0 bytes, and ssh killing the process. Each one's error pointed the wrong way — diagnosis and fix for all three.
- 2026-06-28
Once your character LoRA is trained, how do you control it? Why lightning flattens style, when to spend full steps, how to stack a style LoRA, and why the trigger word alone won't hold the look.
- 2026-06-27
I added up what my home agent pays before it reads a single word from me: ~23K tokens of overhead, and 17K of that is just the instruction manuals for its tools. Worse, it runs a hybrid model — on a cache miss it re-processes all 17K from scratch, and a single user turn can do that a dozen-plus times. This is context economics, badly underestimated. The fix isn't cutting tools; it's loading them on demand, the way skills already do.
- 2026-06-26
On a long conversation, every message makes the model re-read the whole thing (re-prefill) before it answers — worst right after a restart or a cache eviction. Stock llama.cpp can save the KV cache to disk (--slot-save-path) but won't do it on its own — the auto-persist feature request is closed as not planned. A tiny stdlib reverse-proxy restores instead of re-prefilling: 9.9s → 1.4s on a 5K chat (7×). Mechanism, proxy design, and why I haven't shipped it yet.
- 2026-06-25
Quantizing the main KV cache to q4 to save memory is fine. So I quantized the MTP draft cache too — it's just a little draft, surely a free win. It wasn't: q4 draft cache ran 29.6 tok/s, the un-quantized f16 ran 39.7, and f16 used less VRAM on top of that. The draft cache is one of the few places where quantizing is a net loss — here's the triple penalty.
- 2026-06-24
Hermes has a built-in Kanban, but on your phone all you get is Telegram's plain text. Muninn now pulls that board onto the phone: Running / Blocked / Done columns — who's working on what, which card got blocked — at a glance. Zero backend, pure P2P.
- 2026-06-24
The model card says n_ctx_train=262144. The card has 22GB. The 27B's Q4 weights are only 15.7GB. The math looks obvious: max it to 256K, plenty to spare. -c 262144, launch — loads fine, no error. A few turns of real conversation later: 503, the service restarts itself. No tidy out-of-memory in the log, just a lone 0xc0000409. nvidia-smi: free VRAM down to ~170 MiB. Where did the gigabytes go? This is the hunt: I first blamed context checkpoints, but the llama.cpp source says they live in host RAM — the real VRAM eater is the KV cache; free-VRAM-vs-context is nonlinear, and the one stable sweet spot isn't 256K — it's 128K.
- 2026-06-23
Hermes runs at home, but you lose it the second you walk out. Bridging through Telegram works but it's fiddly and routes every message through someone else's server. Muninn is an iOS app built for Hermes: give your agent one command, scan a QR, and your phone connects straight home over an encrypted iroh tunnel — no cloud in the path.
- 2026-06-23
Picking a local model, I looked at tok/s first too. Gemma 12B does 90-100 and it's great — until you put it on a kanban board, where it finishes the work and just walks away, never marking the card done. A Qwen 27B that's three times slower actually closes the loop. Why throughput is the wrong number for an agent — plus how grep almost lied to me about it.
- 2026-06-22
When an AI assistant loops, wanders, freezes, or answers the wrong question, your first instinct is 'this model is dumb.' But from my own debugging, eight times out of ten it's not the model — it's the ring around it (tools, config, memory). The model is the engine; that ring is the car. A car that won't move usually doesn't have a broken engine — it has a flat tire or a clogged fuel line.
- 2026-06-22
I dug up a 22GB-modded RTX 2080 Ti for ~$340 all-in (¥2079 sticker + shipping) — just enough to keep a resident 27B agent brain running on the same cheap old desktop. What the mod changes, and the gotchas.
- 2026-06-21
ds4 ships directional steering — a runtime activation edit that nudges the model along a chosen direction, and the math is literally abliteration with a continuous, signed scale. I got it running on GB10/CUDA (the tooling looks Metal-only, but the activation dump fires on CUDA too) and pulled a verbosity vector from our abliterated Q2 model. The dial works, but it ignores the textbook: the sweep is non-monotonic and positive scales collapse the output to a four-word fragment. Two cuts from the same scalpel, fighting each other.
- 2026-06-20
Your assistant is installed, but right now it only talks — it's all mouth. This post gives it hands: connect tools so it actually checks your folders, runs your commands, calls services you wrote yourself. The key idea is MCP, the 'universal outlet' standard for tools — plug one in and the assistant can use it. All running on your side, connected to your own stuff.
- 2026-06-19
We used ChatGPT as the assistant's brain. This post does something bolder — swaps that brain from the cloud to a local model running on your own machine (e.g. ds4). The payoff is an autonomous brain: no cloud model provider, your conversations stay on your machine, no usage caps. The honest cost: local brains are usually slower (~10 tok/s on my ds4) and need a capable machine. Swap the brain, keep the body — Hermes doesn't change at all.
- 2026-06-18
Train a Wan 2.2 character LoRA on your own RTX 5090 from a single reference image. Then generate the same person from text — new outfits, scenes, art styles, even video. No cloud, no bill.
- 2026-06-17
Comfortable with one assistant and want a second and third? Hermes gives each one its own home (config, memory, personality), each able to run a different model and handle different tasks. Plain-language: why split them, how, and the three I actually run. Honest: most people only need one — this is for when you want to tinker.
- 2026-06-16
Your AI assistant only reads text? Give it eyes and ears — send a photo it understands, send a voice clip it understands. Not by swapping in a pricier model, but by bolting on a small vision model as a perception side-car. Hermes's built-in auxiliary.vision + faster-whisper, measured end to end.
- 2026-06-16
The last and most satisfying step: set up a task that runs itself. Tell it in plain words, and every day it researches what you care about, sums it up, and messages your Telegram. Set it once, close the laptop, and it pings you the next morning.
- 2026-06-16
Order your assistant around from your phone. Chat with one official Telegram bot, get a key (token), hand it to Hermes — done. No public URL, no webhook, no tunnel, because Hermes fetches messages from Telegram itself.
- 2026-06-16
Install the Hermes Agent desktop app — no terminal. Download it, let it auto-install dependencies, sign in with your ChatGPT account, and your first AI assistant is running in about 15 minutes.
- 2026-06-16
An AI assistant = a brain + a body. Use your ChatGPT account as the brain and Hermes as the body — one fixed combo, nothing to choose. Here's why it's set up this way, and what to have ready before you install.
- 2026-06-16
You don't need to write code to have your own AI assistant. An agent framework already packages the hard parts so you just install and go. Here's why you shouldn't wire it yourself — and why this series uses Hermes.
- 2026-06-16
You mostly use ChatGPT one question at a time. A self-hosted AI assistant (agent) finishes the job with your own tools, runs on your side, and plugs into the apps you use daily. Lesson one of building your own assistant from zero.
- 2026-06-14
A retrieval-augmented support bot for my blog, running on a 2014 GTX 970 and a ~600MB embedding model. llama.cpp embeddings on CPU, numpy brute-force cosine over 3,475 chunks, an embedding-score guardrail, and Cloudflare Tunnel.
- 2026-06-14
On a tensor-core-less Maxwell GTX 970 running Gemma 4 E2B, Flash Attention nearly doubles long-context decode (24.3 → 42.5 tok/s) and saves ~430MB VRAM — while q8 KV cache barely saves memory and slows decode. The usual KV-cache advice flips.
- 2026-06-13
DiffusionGemma 26B-A4B runs on vLLM on a 128GB DGX Spark via an official prebuilt image — no PR-waiting, no cherry-picking. NVFP4 hits 158 tok/s single-stream and 257 aggregate. But a single tok/s number lies: diffusion speed is decided by whether the 256-token canvas fills.
- 2026-06-12
DeepSeek-V4-Flash (284B) only fits a 128GB box at asymmetric Q2 (~80GB). Sounds like suicide quantization — but it's surgical: only the layers that barely affect quality get cut. As a daily agent it ran 280 turns with zero degradation. Big enough weights survive 2-bit.
- 2026-06-12
A 284B model at 15 tok/s, wired into a daily agent. Two sets of settings make it comfortable — server-side and agent-framework-side. --no-mmap cuts cold start to 57s, the KV disk cache halves prefill, and one missing context_length will crash the whole session.
- 2026-06-12
DeepSeek-V4-Flash is 284B. I got it onto a single 128GB GB10 with antirez's ds4 engine and an asymmetric Q2 GGUF at 15.6 tok/s. The fun part: the broken tool calls weren't the 2-bit quant's fault. The runtime just couldn't parse DSML.
- 2026-06-11
Qwen3.5-122B-A10B tops out at 17 tok/s on a 128GB DGX Spark — the GDN wall in vLLM won't budge, not even with a merged perf PR. I swapped vLLM for the Atlas engine on the same abliterated NVFP4 weights and the throughput doubled to 33.9 tok/s (36.5 with MTP, ~2×), uncensored behavior intact. The real lever was outside the quant toolbox.
- 2026-06-09
A 2014 GTX 970 running Gemma 4 E2B (vision + audio) plus Piper TTS — a full offline voice assistant that sees, listens, talks back, and writes code. ~2.8s end-to-end, ~$15 of hardware.
- 2026-06-09
Four Gemma 4 E2B quants on a 2014 GTX 970. The bigger 3.2GB QAT Q4_0 beats the 2.9GB Q2_K — 47.6 vs 32.8 tok/s — because a tensor-core-less Maxwell card is dequant-bound, not bandwidth-bound.
- 2026-06-05
I benchmarked BF16 vs FP8 vs NVFP4 weight-only on gemma-4-12B across English (MMLU) and Traditional Chinese (TMMLU+) on a DGX Spark. FP8 is near-lossless on both; NVFP4 drops Chinese ~6pp but English only ~3pp.
- 2026-06-04
I quantized Google's new omni Gemma 4 12B on a DGX Spark GB10. Weight-only NVFP4 hits 24.9 tok/s in 7.7 GB and keeps image/audio/video working — full W4A4 is slower AND breaks multimodal.
- 2026-06-02
My local 35B agent went haywire generating images until I read its tool-call logs: 0% malformed calls. The model was fine — a broken ComfyUI tool was making it improvise. The fix was a clean ACI skill, not fine-tuning.
- 2026-06-01
NVFP4 took a distilled Sulphur 2 (LTX-2.3) video model from 29 to 19.5 GB on a GB10 DGX Spark with no quality loss and — since video is compute-bound — no speed gain (if anything a hair slower).
- 2026-06-01
On a GB10 DGX Spark, NVFP4 W4A4 went from 23 to 67 tok/s the moment I dropped --enforce-eager — beating FP8 by 29% and saving 16GB. The catch from Part 32 was real, just dense-only.
- 2026-05-30
On a GB10 DGX Spark, NVFP4 beats FP8 by ~1.5× for single-stream decode on a dense model. But the win is bandwidth (smaller weights), not the FP4 tensor cores — the fastest path never touches them.
- 2026-05-23
AI delivers wrong answers in the same confident tone as right ones. Three red flags to catch it early — impossible numbers, suspiciously specific details, answers that shift on a re-ask — plus a case where ChatGPT gave me a +205% P&L that can't exist.
- 2026-05-21
After Part 30's endpoint correction showed Round 1 didn't actually 2x chat throughput, Round 2 added 30k regenerated Chinese instruction samples and trained for 41 hours. Result: Round 2 B drafter delivers chat EN 45 tok/s / ZH 29 tok/s — essentially the same as v1 (EN 46 / ZH 27), and well below vanilla MTP n=4's EN 53 / ZH 45. The EAGLE-3 small head hits an architectural ceiling against the abliterated body; more data doesn't fix it. Plus we found a scheduler deadlock in the vLLM Gemma 4 preview image (`gemma4-0505-arm64-cu130`, internal build `0.20.2rc1.dev49+g9b4e83934`) under long-running extract_hidden_states use (hit three times, mitigated with a watchdog).
- 2026-05-19
I had a rule called 'fact-check before publishing.' I still shipped three fabrications. The problem wasn't the rule — it was where I put it. This is how I promoted it from skill to hook: a small script guarding the moment I press 'send,' so I can't even try without verification.
- 2026-05-16
RedHatAI's EAGLE-3 drafter fine-tuned to realign with huihui Gemma 4 26B-A4B abliterated FP8 on a single DGX Spark GB10 — 1 epoch / 50k Magpie samples / 11h. Inference bench on raw `/v1/completions`: pos 3 acceptance climbs from vanilla's 20.5% to 72.7%; n=4 throughput goes from ~50 to 100.36 tok/s aggregate. **A later paired bench revealed the throughput comparison used different endpoints for baseline (chat) and retrain (raw) — on production chat workloads the real uplift is far smaller than 2×; see the endpoint correction at the top of the post**. Part 28's mechanism observation (deep speculation acceptance scatters on abliterated distributions) still holds. Includes a Speculators upstream create_empty_sample dtype bug + patch and a Phase 0 catalog of 6 community prior-art repos.
- 2026-05-14
Part 28 explained why deep speculation breaks on an abliterated body; this post is the recipe for the part that already works. huihui Gemma 4 26B-A4B FP8 + Google's vanilla MTP draft at num_speculative_tokens=1 takes baseline 39.3 tok/s to 52.6 tok/s (+34%) on GB10, no retraining required. ~30 lines of docker plus a bind-mount of PR #41745's gemma4_mtp.py. Includes a 3-step sanity check and a clear list of when n=1 stops being enough.
- 2026-05-09
I self-quantized huihui's abliterated Gemma 4 26B-A4B to FP8-Dynamic and shipped it to HF. After sweeping num_speculative_tokens 1→4, the abliterated body is exactly as fast as vanilla on the same stack (39.4 vs 39.3 tok/s baseline) and the MTP boost at n=1 is equivalent — but per-position acceptance decays so steeply that deeper speculation is wasted. Three drafts of this article each smuggled in a different fabrication that Codex caught; this is the corrected version.
- 2026-05-06
Google announced Multi-Token Prediction drafters for Gemma 4 on 2026-05-05. The vLLM PR was opened and approved the same day; a preview Docker image shipped hours later. I tested it on DGX Spark: Gemma 4 26B-A4B-it FP8 + MTP γ=4 hits 108.78 tok/s single-stream (2.66× baseline), 674.28 tok/s aggregate at concurrency=8. One undocumented trap: the drafter pairs with -it, not base.
- 2026-05-05
I ran sysprog21/zhtw-mcp across 72 of my Traditional Chinese articles. Three sweeps, 128 cross-strait substitutions across 42 files. The real takeaway wasn't the count — it was discovering my blindspot isn't 'I don't know the right Taiwanese term,' it's 'when a Mainland term shows up I don't auto-doubt it.'
- 2026-05-04
Does Z-Image Turbo quantization break image quality? Two-axis benchmark — LPIPS (perceptual distance vs BF16) + CLIPScore (image-text alignment) — across 6 prompts × 4 configs × 3 seeds = 72 samples. Result: NVFP4 produces images that look different from BF16, but no measured regression in this sample — all 4 configs land within ±0.04 std on CLIPScore, smaller than the noise floor. Production users should re-verify with their own prompt set.
- 2026-05-04
I ran six Z-Image Turbo quantization configs on DGX Spark GB10 — BF16 baseline, FP8 cast standard, FP8 cast fast, FP8 scaled (Kijai), NVFP4, NVFP4+FP8 encoder. With N=10 isolated GPU, NVFP4 transformer hits 5.50s warm versus BF16 7.55s (1.37× faster). All three FP8 paths are slower than BF16. Model working set drops from 20.6 GB (BF16) to 11.5 GB (NVFP4+FP8 encoder) — 44% smaller.
- 2026-05-01
Same DGX Spark, different goal: watch a 3-minute Andrej Karpathy talk and output the spoken content + visual scene. 89 seconds wall, 53,842 prompt tokens, factually correct. The use_audio_in_video flag, the upstream-image gotcha, and the long-video knob math.
- 2026-05-01
Ten days ago I called NVFP4 a trap on DGX Spark GB10. Today the same hardware hits 74.75 tok/s on Nemotron 3 Nano W4A16, beating my own FP8 ceiling and the public 67 tok/s forum number. The 4-layer patch stack, the quant variant choice, and the bandwidth math behind it.
- 2026-04-30
ai-muninn.com burned through Vercel Hobby's 1M Edge Requests quota this month. It wasn't traffic, wasn't bots, wasn't large images. Next.js defaults /public/* to must-revalidate, which makes every conditional GET (even 304s) count as an edge request. Three lines of next.config.ts to fix. Three rounds of fact-check rewrites to publish.
- 2026-04-28
Qwen 3.6 35B-A3B FP8 hits 48.33% (145/300) on SWE-bench Lite with the same scaffold that gets Gemma 4 26B to 38.67%. The 9.66-point gap deserves an explanation. This is a deep dive on Qwen 3.6's 155 failures: 76% are wrong-logic patches, 14% are incomplete fixes, 10% never submit. The categorization is asymmetric — Gemma 4's failures haven't been classified the same way yet — so the cross-model comparison is part hypothesis, part data.
- 2026-04-28
Quantizing huihui-ai's Qwen3.6-35B-A3B abliterated to FP8 for vLLM on a 128 GB UMA box. Seven attempts, two distinct OOM modes, a model class that silently breaks vLLM's loader, and why streaming save_pretrained returns BF16 not FP8. Final result: 51.72 tok/s, 1.68× BF16.
- 2026-04-26
Ran huihui-ai's abliterated Qwen 3.6 35B through the same TMMLU+ harness as Part 21. Aggregate dropped 75.07% → 73.22%. The cost isn't uniform: regulatory subjects (信託 −7.7, 行政法 −7.1) lose the most, while pure logic and math actually improve. Hokkien also got worse — abliteration doesn't fix data scarcity.
- 2026-04-25
Two MoE models on the same DGX Spark, same harness, same 22,690 questions. Qwen 3.6 35B-A3B scored 75.07%, Gemma 4 26B-A4B scored 46.30%. Qwen won every single one of the 51 subjects — including Taiwan-specific topics where I expected Gemma to win.
- 2026-04-22
Part 19 proved NVFP4 is a trap on DGX Spark. This time we fight back: a Triton kernel that dequants NVFP4 to FP8 and feeds the FP8 tensor cores. 40.8 → 47.6 tok/s, with full code.
- 2026-04-21
NVFP4 should be faster than FP8 — fewer bits, less bandwidth. On DGX Spark's GB10 (SM121), it's 32% slower. Root cause: missing hardware instruction. Dual-engine proof with vLLM and SGLang.
- 2026-04-20
One scaffold (backticks + edit-tool + budget prompt), three models (Gemma 4 E4B, Gemma 4 26B, Qwen 3.6 35B), zero code changes between runs. Qwen 3.6 hit 48.33% — beating SWE-agent + Claude 3.7 Sonnet. The scaffold is the fixed cost; the model is the variable.
- 2026-04-17
Gemma 4 26B-A4B FP8 scored 116/300 on SWE-bench Lite, ranking #16 globally. Zero API cost on a DGX Spark. The scaffold — not the model — was the differentiator.
- 2026-04-17
Local AI isn't a budget ChatGPT. It's a knowledge extractor, private code assistant, and offline tool. Monthly power cost ~$1.20 vs ChatGPT Plus $20. This guide has a decision table for when to use which.
- 2026-04-16
AI is strong, but four things still trip it up in 2026: hallucinations, stale knowledge, short memory, and privacy defaults. Even Anthropic's own lawyers got caught by the first one.
- 2026-04-15
Two days running mini-swe-agent + vLLM on a GB10. From wrong doc conclusions to Gemma 4 self-submitting a clean patch in 38 steps — what actually unlocked it.
- 2026-04-15
How does Q4_K_M fit a 14B model into 4 bits without ruining it? Not by 'cutting off 75%' — but through three layers: K-quant super-blocks, TurboQuant random rotation, and a 1-bit JL sign sketch. A mechanism walkthrough without the equations.
- 2026-04-14
The first answer AI gives you is a rough draft, not the final answer. Learn 5 follow-up techniques — adding constraints, asking for comparisons, and letting AI ask YOU questions — to get dramatically better results.
- 2026-04-14
AI forgets what you said 20 messages ago. It's not broken — its desk is full. This guide explains context windows, why conversations go stale, and how to work around the limit.
- 2026-04-13
A feasibility test: can open-source models run SWE-Bench locally for free? Gemma 4 26B failed on OpenHands (40+ errors) but fixed a test bug in 9 steps on SWE-agent. Same model — the action format was the difference.
- 2026-04-13
Gemma 4 E2B / E4B / 26B MoE / 31B Dense benchmarked on DGX Spark, RTX 5090, and MacBook Pro. One table with speed, memory, quantization format. Selection guide included.
- 2026-04-13
You just started using Claude Code and the context window keeps filling up. Here's where the tokens actually go, what you can do about it, and how to make Claude remember things without re-reading everything.
- 2026-04-13
Your first question to AI shouldn't be 'help me do X.' It should be 'is there something that already does X?' This article teaches you how to use AI as a research assistant — finding tools, comparing alternatives, and verifying they're still alive.
- 2026-04-13
Your CLAUDE.md and MEMORY.md grow silently until they eat 10K+ tokens per turn. I built a /slim skill that lets Claude diagnose and fix its own bloat — here's how.
- 2026-04-13
Everything you need to go from a sealed DGX Spark box to serving your first local LLM. Hardware check, Ollama quickstart, vLLM production setup, model selection, and the 5 gotchas that cost hours.
- 2026-04-11
Same AI, same question, different results. The people who find ChatGPT life-changing and the people who think it's useless are doing completely different things — and the difference is a single mindset shift.
- 2026-04-10
Q4_K_M, Q8_0, FP16 — the same model comes in a dozen versions and the names look like hieroglyphs. This guide explains what quantization actually does, why it doesn't ruin the model, and which level to pick.
- 2026-04-10
Most people don't struggle with using AI — they struggle with knowing what to use it for. This article teaches you a simple method to let AI identify the repetitive parts of your workday you've stopped noticing.
- 2026-04-10
Which local AI model should you download? Gemma, Llama, Qwen, Mistral compared by size, speed, and quality. Simple formula: parameters × 0.6 = GB needed. Beginner-friendly guide.
- 2026-04-09
AI isn't Google — you're not searching, you're having a conversation. This article teaches you what to say when you first open ChatGPT, five things you can try right now, and how to adjust when the answer isn't quite right.
- 2026-04-09
ChatGPT, Claude, and Gemini — the three AI assistants you can start using right now. A no-jargon guide to what each one does best, how much they cost, and how to get started.
- 2026-04-08
Gemma 4 31B runs at 1.5 tok/s on MBP M1 Max with Ollama due to swap. The fix: reduce context window (9 tok/s) or switch to oMLX (12.8 tok/s). The real culprit is KV cache allocation, not model size.
- 2026-04-08
Gemma 4 E2B through 31B benchmarked on RTX 5090, M1 Max, DGX Spark, and M4 with Ollama. E2B hits 310 tok/s on 5090. 31B hits 1.5 tok/s on MBP — swap kills faster hardware. Memory capacity > bandwidth.
- 2026-04-08
Dense is everyone working. MoE is expert rotation. PLE is a dictionary on every floor. SSM is a speed reader. A zero-jargon guide to the four main AI model architectures and how to pick between them.
- 2026-04-07
Gemma 4 E2B is 44-82% faster than E4B across M1 Max, GB10, and M4. We benchmarked both on Ollama with 3 runs per scenario, unique prompts, and proper warm-up. Memory bandwidth predicts generation speed better than anything else.
- 2026-04-07
Gemma 4 E4B NVFP4A16 hits 49.9 tok/s on DGX Spark — 2.6x faster than BF16. First NVFP4 checkpoint on HuggingFace. PLE architecture, FP8 vs NVFP4, and the llm-compressor version hell that almost stopped us.
- 2026-04-07
Ollama is a microwave — one command and you're chatting with AI. vLLM is a professional oven — 30% faster, handles multiple users, but takes real setup. A zero-jargon guide to choosing between them.
- 2026-04-05
Gemma 4 31B-IT NVFP4 on GB10 maxes out at 7.0 tok/s — bandwidth-bound at 273 GB/s. The math predicted 4.4 tok/s theoretical; NVFP4 compression buys 60% but can't escape the wall. Choose MoE.
- 2026-04-05
Same Gemma 4 26B-A4B, same GPU, 30% speed gap. vLLM NVFP4 hits 52 tok/s while Ollama Q4_K_M tops at 40. Root cause: Marlin kernels, CUDA graphs, and an Ollama CPU/GPU split trap.
- 2026-04-05
Gemma 4 26B-A4B + NVFP4 hits 52 tok/s on DGX Spark (GB10) — 7.5× faster than the 31B dense, in 16.5 GB with 82 GB free for KV cache. Plus the vLLM 0.19 marlin/patch gotchas that make it work.
- 2026-04-02
DGX Spark power and thermal issues blew up after Carmack's criticism. This guide covers three distinct symptoms: 30W PD controller defect (needs RMA), 100W thermal throttling, and 5W driver bug (fixable). One command, 30 seconds to diagnose.
- 2026-03-30
Real benchmark numbers for Google's TurboQuant on a GB10/SM121 (DGX Spark) — actual compression ratios, Qwen2.5-3B accuracy validation, and why Qwen3.5-35B's hybrid attention architecture makes things complicated.
- 2026-03-24
Your ChatGPT Plus subscription already includes GPT-5.4 with 1M context. openclaw's OAuth flow lets you use it for AI agents — zero API credits, one command. Full setup guide.
- 2026-03-24
How to point NemoClaw's inference backend to a local Ollama or vLLM endpoint. Config location, model swap, and what OpenShell still enforces when the cloud is gone.
- 2026-03-23
NemoClaw's official installer fails on DGX Spark out of the box. This guide covers the 4 fixes — Node upgrade, npm link, OpenShell tar.gz, cgroupns — to get your first AI agent running in 30 minutes.
- 2026-03-23
NemoClaw bundles OpenClaw + OpenShell + NVIDIA Agent Toolkit into one installer for DGX Spark. Architecture breakdown, what it does, and whether it's worth your time.
- 2026-03-21
Replacing choppy editMessageText polling with Telegram's sendMessageDraft for live animated output. The patch, the think-block filter, and the optional chaining trap in DM chats.
- 2026-03-21
Connecting openclaw to a 131K context model and hitting 400 max_tokens must be at least 1, got -1292. The context budget math, the config key trap, and the fix.
- 2026-03-21
Adding --kv-cache-dtype fp8 to a vLLM serve script on GB10 causes outputs to degrade into repetition after ~500 tokens. Root cause: missing calibration data, q_scale defaults to 1.0.
- 2026-03-21
Building a multi-agent orchestrator with `claude -p` subprocess reveals a silent data loss problem. The SDK fix, session resume, parallel execution, and why setting_sources matters.
- 2026-03-19
The bot process is running. The token is valid. Messages are being consumed. Nobody is home. A systematic takedown of every wrong hypothesis — and the hidden causal chain that connects Tailscale routing tables to silent sendMessage failures in Node.js.
- 2026-03-19
How to get gpt-oss-120B running on a DGX Spark (GB10, SM121) with vLLM. The goal: a 120B model serving a local AI agent at zero API cost. The path: six bugs, one silent env var, and a startup log that tells you everything.
- 2026-03-19
After fixing the four SM121 NVFP4 bugs, Qwen3.5-122B boots cleanly and generates correct output. Then you check the speed. 14 tok/s. No flags to fix it. Here's why — and what to wait for.
- 2026-03-18
How to wire a callhelp tool into a local agent loop so it can spawn Codex CLI mid-reasoning. One permission flag you must set, and why Claude's quota stays mine.
- 2026-03-17
CUTLASS FP4 kernels target SM120 (GB200). On SM121 (GB10, DGX Spark) they run silently and produce garbage. Here's the full diagnostic story — 4 bugs, the row-identical failure signature, and the working fix.
- 2026-03-16
Why we stopped having the OpenClaw agent orchestrate multi-step tasks directly, and started spawning Codex subprocesses instead. The pattern that keeps agent context minimal and tasks reliable.
- 2026-03-13
Getting NVIDIA's Nemotron-3-Super-120B-NVFP4 running on an ASUS GX10 (SM121, 128GB). Four SM121-specific pitfalls, the env-var-that-does-nothing, and a working docker command.
- 2026-03-07
vLLM OOMed on restart despite 128GB unified memory. Cause: Ollama's KEEP_ALIVE=2h was holding 19-51GB in GPU. Diagnosis command, manual unload fix, and why to set KEEP_ALIVE=0 once vLLM is your primary stack.
- 2026-03-06
Adding --enable-chunked-prefill to a Qwen3.5-35B (SSM+MoE hybrid) dropped throughput from 47 tok/s to 5.7 tok/s. Why SSM recurrence and chunked prefill are fundamentally incompatible.
- 2026-03-05
Step-by-step guide: Ollama to vLLM on DGX Spark GB10. Qwen3.5-35B hits 47 tok/s with TTFT dropping from 3s to 0.12s. Covers 6 real gotchas including SSM + chunked prefill trap and GPU memory conflicts.
- 2026-03-05
Full stack local AI agent: Mac Mini M4 as the always-on gateway, GX10 for inference, Telegram as the UI. No subscriptions, no cloud APIs. Six deployment lessons from the trenches.
- 2026-03-01
GLM-4.7-Flash hits 57.8 tok/s on short context but drops to 42 tok/s at 8K. Qwen3.5-35B SSM hybrid: 56 tok/s at short, 56 tok/s at 8K. Why agents with long system prompts should care about this difference.
- 2026-02-26
A custom /debate command that pits Codex CLI against Gemini CLI on architecture, code, and decisions. Different training data, different blind spots — and the disagreements between them are usually the most useful output.
- 2026-02-26
How I replaced screenshot-heavy iOS test runs with ui_describe_all-first testing in Claude Code, cutting context usage by 81% for BPS Tracker. Plus Fastlane integration for App Store automation.
- 2026-02-25
Spent weeks restarting the OpenClaw gateway for every config change. Then discovered the file watcher. What hot-reloads instantly, what still needs a restart, and how to tell auth failures from transient network errors.
- 2026-02-19
A Claude Code config rule marked MANDATORY was skipped twice in one session. Here's the root cause — three architectural reasons why emphasis doesn't work — and three system-level solutions that do.
- 2026-02-19
Benchmarking 8 local LLMs on NVIDIA GB10 (128GB unified memory) across 7 task categories. Quantization surprises, a 120B model that fails at JSON, and thinking models that spend their entire budget thinking.
showing 179 posts