<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: InferenceKey</title>
    <description>The latest articles on DEV Community by InferenceKey (@inferencekeydev).</description>
    <link>https://dev.to/inferencekeydev</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4185415%2F46ebdd39-7d2b-4576-bf84-3bab513a7f4b.png</url>
      <title>DEV Community: InferenceKey</title>
      <link>https://dev.to/inferencekeydev</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/inferencekeydev"/>
    <language>en</language>
    <item>
      <title>Building on LFM3-Omni-2B: a silent-video trap, a text-only fast server and what to measure</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Mon, 16 Nov 2026 06:37:33 +0000</pubDate>
      <link>https://dev.to/inferencekey/building-on-lfm3-omni-2b-a-silent-video-trap-a-text-only-fast-server-and-what-to-measure-9an</link>
      <guid>https://dev.to/inferencekey/building-on-lfm3-omni-2b-a-silent-video-trap-a-text-only-fast-server-and-what-to-measure-9an</guid>
      <description>&lt;p&gt;The usual voice assistant is four models in a trench coat: speech recognition, a vision model, a language model and a speech synthesizer, so four servers to babysit. Liquid AI's &lt;a href="https://huggingface.co/LiquidAI/LFM3-Omni-2B" rel="noopener noreferrer"&gt;LFM3-Omni-2B&lt;/a&gt; is one set of weights, about 4.5 GB, that takes text, images, video with sound and recorded speech, and answers in text, speech or both at once.&lt;/p&gt;

&lt;p&gt;We deployed it on two small cards on our cloud, an RTX A5000 with 24 GB and an RTX 2000 Ada with 16 GB, and wrote up every measurement in the &lt;a href="https://inferencekey.com/research/lfm3-omni-voice-vision-one-model" rel="noopener noreferrer"&gt;LFM3-Omni-2B voice and vision test on inferencekey.com&lt;/a&gt;. Here is where it bites.&lt;/p&gt;

&lt;h2&gt;
  
  
  The video trap: forget one flag and the model says the clip is silent
&lt;/h2&gt;

&lt;p&gt;We sent a 9.7 second clip of a toddler reading, plus a typed question. With the soundtrack, the model reported that the child is speaking. With frames only, it claimed there was no audible speech or music in the clip. Confidently wrong.&lt;/p&gt;

&lt;p&gt;In the checkpoint's Transformers code (the reference Python implementation the model ships with), video with sound needs two settings. Leave out either one and the audio never reaches the model:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Video with its soundtrack, Transformers code path: both, or frames only&lt;/span&gt;
&lt;span class="n"&gt;load_audio_from_video&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;
&lt;span class="n"&gt;use_audio_in_video&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the hand-built SGLang branch, the flag only worked as a top-level &lt;code&gt;use_audio_in_video: true&lt;/code&gt;. Inside &lt;code&gt;mm_processor_kwargs&lt;/code&gt; it was silently dropped.&lt;/p&gt;

&lt;p&gt;Caveat: the run with sound also had a one-line system prompt asking the model to use what it sees and hears, so two things changed. Even so, the model only noticed sound when it got the sound. If your app takes video, write a test that proves the audio arrives.&lt;/p&gt;

&lt;h2&gt;
  
  
  The serving split: the fast server is about four times quicker and writes only text
&lt;/h2&gt;

&lt;p&gt;SGLang is an open-source inference server: it keeps a model loaded on the GPU and answers requests. The current release, 0.5.23, does not support this model and crashed at start-up on our card, despite the one-line launch in Liquid AI's docs. Support lives in an open pull request, &lt;a href="https://github.com/sgl-project/sglang/pull/46858" rel="noopener noreferrer"&gt;sgl-project/sglang#46858&lt;/a&gt; by tugot17. We built it by hand. With greedy decoding (always picking the most likely next token, so runs are repeatable) it gave the same answers as the Transformers path, at 158 tokens per second against 38 to 40. (A token is a word piece.) That build has no voice.&lt;/p&gt;

&lt;p&gt;So anything spoken goes through the model's own code, loaded by the Hugging Face Transformers library with &lt;code&gt;trust_remote_code&lt;/code&gt;. For a single caller the slow route was good enough; we did not test concurrent ones. For heavy text traffic, hold off until SGLang ships a release with support.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F18mxeab3s80nagy4k58y.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F18mxeab3s80nagy4k58y.jpg" alt="Three tiles summarising the LFM3-Omni-2B test: about 5 GB of GPU memory, a 10-second spoken answer in about 7 seconds of GPU time, and a 16 GB card as fast as a 24 GB one" width="800" height="320"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What to measure: first speech frame, round trip, and two kinds of memory
&lt;/h2&gt;

&lt;p&gt;The GPU finished a 10-second spoken answer in roughly 6 to 7.5 seconds; the caller got it after 8 to 10, because our backend sends one WAV at the end. Inside the server, the first slice of audio appeared within 0.2 to 0.5 seconds. Streaming means sending audio as it is produced, and we neither built nor tested it. Log both numbers anyway: the gap is what streaming would buy you.&lt;/p&gt;

&lt;p&gt;Memory splits too. PyTorch, the library holding the weights on the GPU, reported a peak of 4,285 to 4,434 MiB per request. &lt;code&gt;nvidia-smi&lt;/code&gt;, NVIDIA's own monitor, put the whole process at 4,772 to 4,832 MiB. PyTorch never counts the extra 400 MiB or so. To find the floor, we limited PyTorch on the A5000 with &lt;code&gt;torch.cuda.set_per_process_memory_fraction&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Cap&lt;/th&gt;
&lt;th&gt;Voice, photo, transcription, speech, 10 s video&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;RTX 2000 Ada 16 GB, real card&lt;/td&gt;
&lt;td&gt;all pass&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4.45 GiB (simulated)&lt;/td&gt;
&lt;td&gt;all pass&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4.35 GiB (simulated)&lt;/td&gt;
&lt;td&gt;the 10 s video runs out of memory&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Those rungs are a software limit on a large card, not a real small one. A one-minute video wants about 6 GB. Each rung is in the &lt;a href="https://inferencekey.com/research/lfm3-omni-voice-vision-one-model" rel="noopener noreferrer"&gt;GPU memory ladder for LFM3-Omni-2B&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Numbers read aloud need a check
&lt;/h2&gt;

&lt;p&gt;We asked "how tall is Kilimanjaro" out loud ten times. Nine answers had the meters right, and most got the feet wrong. Speech is generated with sampling, meaning a controlled dose of randomness (&lt;code&gt;audio_temperature&lt;/code&gt; 0.4 in our runs), and the written words follow the voice, so two runs rarely match. Give a number-reading assistant a tool or a check for conversions. Tool calling behaved: a spoken "set the volume to 30" became &lt;code&gt;set_volume({"volume": 30})&lt;/code&gt; in about half a second.&lt;/p&gt;

&lt;p&gt;Distrust quotes from muddy audio too: the model gave the toddler a sentence that matched neither Whisper's transcript nor its own.&lt;/p&gt;

&lt;h2&gt;
  
  
  The license is a gate, not a footnote
&lt;/h2&gt;

&lt;p&gt;The weights ship under the &lt;a href="https://huggingface.co/LiquidAI/LFM3-Omni-2B/blob/main/LICENSE" rel="noopener noreferrer"&gt;LFM Open License v1.0&lt;/a&gt;. Your organization may use it commercially only below 10 million US dollars of yearly revenue. Above that, agree terms with Liquid AI before production.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we did not prove
&lt;/h2&gt;

&lt;p&gt;One request at a time, our own short inputs, no human rating of the voice (Whisper large-v3 via faster-whisper read the audio back), no GGUF builds (the compressed files llama.cpp runs), no llama.cpp realtime server. No four-model pipeline on the same card either, so this cannot say one model beats it.&lt;/p&gt;

&lt;p&gt;Credit where it belongs: Liquid AI built the model, its audio codec, the benchmarks and the sample clips; tugot17 wrote the SGLang port; the Transformers maintainers made the code path we ran; the llama.cpp maintainers added GGUF and realtime support; OpenAI's Whisper gave the independent read-back; LibriSpeech supplied the transcription clip; Kyutai's Pocket TTS voiced our test questions; AI Search's video is where we spotted it. Our part: the custom backend, the two-card deployment and the measurements.&lt;/p&gt;

&lt;p&gt;The backend, request options, timings and SDK example are in the &lt;a href="https://inferencekey.com/research/lfm3-omni-voice-vision-one-model" rel="noopener noreferrer"&gt;full LFM3-Omni-2B deployment write-up&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>python</category>
      <category>gpu</category>
    </item>
    <item>
      <title>Running a dense 33B agent model on one 24 GB GPU: the offload trap and the empty reply</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Sun, 15 Nov 2026 06:38:34 +0000</pubDate>
      <link>https://dev.to/inferencekey/running-a-dense-33b-agent-model-on-one-24-gb-gpu-the-offload-trap-and-the-empty-reply-4dda</link>
      <guid>https://dev.to/inferencekey/running-a-dense-33b-agent-model-on-one-24-gb-gpu-the-offload-trap-and-the-empty-reply-4dda</guid>
      <description>&lt;p&gt;To self-host Nanbeige4.3-33B, the Apache-2.0 agent coding model the Nanbeige team (BOSS Zhipin) released on 24 October, two decisions matter most. First, which compressed copy of the weights you load for your card. Second, how you handle thinking mode, where the model reasons before replying: with the defaults, your agent can get an empty answer back.&lt;/p&gt;

&lt;p&gt;We ran it on one RTX 4090 on our cloud with llama.cpp, the open-source server for single-file GGUF models, using bartowski's files. The 4-bit file gave 42 to 44 tokens per second for one request (a token is a word or part of one). Every tool call, where the model asks your code to run a function, came back valid in the OpenAI format. The &lt;a href="https://inferencekey.com/research/nanbeige-33b-dense-agent-24gb" rel="noopener noreferrer"&gt;full Nanbeige4.3-33B test on 24 GB and 16 GB cards&lt;/a&gt; has every command and table. Here are the parts that bite.&lt;/p&gt;

&lt;h2&gt;
  
  
  Dense means every parameter is read for every token
&lt;/h2&gt;

&lt;p&gt;Many agent models that fit one card are mixtures of experts: each token goes through a small slice of the network, so 35 billion parameters may mean 3 billion working per token. &lt;a href="https://inferencekey.com/research/groot-19b-agent-moe-one-card" rel="noopener noreferrer"&gt;Groot-19B on a single card&lt;/a&gt; was one. Nanbeige4.3-33B is dense: all 33 billion take part in every token.&lt;/p&gt;

&lt;p&gt;So the GPU streams the entire 20.8 GB of the 4-bit file out of VRAM, its onboard memory, for every token. Read speed of that memory is your limit: smaller file, quicker model. Any weight parked in slower system RAM drags every token down.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trap: partial offload on a 16 GB card
&lt;/h2&gt;

&lt;p&gt;The reflex on a smaller card is offload: keep the 4-bit file and run some of the model's layers on the CPU. With a dense model, that is the wrong move. On the 4090 we stayed under 16 GB by moving 16 of the 52 layers to the CPU. Answers stayed correct, at 1.6 tokens per second.&lt;/p&gt;

&lt;p&gt;The fix is a smaller file that fits entirely on the GPU. bartowski's 3-bit file (IQ3_XXS) peaked at 14,730 MiB, and on the 4090 it ran at 56.5 tokens per second. On a real 16 GB card, an RTX 2000 Ada at 70 watts, the same file ran at 13.6 to 14.4 tokens per second and passed the same checks.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnsx4e2v9qislnq2a6m5o.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnsx4e2v9qislnq2a6m5o.jpg" alt="Three tiles: 42 tokens per second on one RTX 4090, 64K context in 24 GB, a real 16 GB card with the 3-bit file at 14 tokens per second" width="800" height="320"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two bits is too far. That file stayed under 12 GB with right answers, yet claimed the 8 in 9.8 is "in the hundredths place". For code, one wrong explanation in four checks rules it out.&lt;/p&gt;

&lt;h2&gt;
  
  
  Context costs you speed, not just memory
&lt;/h2&gt;

&lt;p&gt;Context is how much text the model takes in at once. On the 4090 a 64K-token prompt worked, and the model located a secret word planted at token 61,428. That meant squeezing the KV cache, the model's working notes on the prompt. At 16-bit it fits up to 16K, at 8-bit up to 32K and at 4-bit up to 64K, with under 1 GB left on the card.&lt;/p&gt;

&lt;p&gt;Benchmark generation speed with long prompts. At 64K it dropped to 16 tokens per second, since each new token looks back over roughly 61,000 tokens of notes. The secret word turned up at all three sizes, the only quality check we ran on the 4-bit cache.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug you will hit: the answer lands in the thinking field
&lt;/h2&gt;

&lt;p&gt;With thinking on, the default, some replies come back with &lt;code&gt;content&lt;/code&gt; empty and the whole answer in &lt;code&gt;reasoning_content&lt;/code&gt;. It hit all three 400-word essays we requested, and on the agent's final answer after the tool results in two of three runs, once on each card. Brief answers and the tool calls were fine every time.&lt;/p&gt;

&lt;p&gt;The cause is in the chat template, the wrapper that turns your messages into the model's prompt. It ends the prompt by opening a thinking block:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;&amp;lt;|im_start|&amp;gt;assistant
&amp;lt;think&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sometimes the model answers without closing the block, so llama.cpp's &lt;code&gt;--reasoning-format deepseek&lt;/code&gt; files everything as reasoning. With &lt;code&gt;enable_thinking=false&lt;/code&gt; sent straight to llama.cpp, the essay arrived in &lt;code&gt;content&lt;/code&gt;, all 3,151 characters. Server-wide, &lt;code&gt;--chat-template-kwargs '{"enable_thinking":false}'&lt;/code&gt; is the expected fix, but we never launched that flag, so its effect on the agent's answer is untested. The trade-off: the published scores were taken at maximum thinking, and the authors advise leaving it on for agents.&lt;/p&gt;

&lt;p&gt;Two gaps on our side are open: our endpoint drops the per-request &lt;code&gt;chat_template_kwargs&lt;/code&gt;, and the worker builds llama.cpp from source on first start (about five minutes on the 4090 host, nearly 20 on a throttled one). Meanwhile, check for an empty &lt;code&gt;content&lt;/code&gt; in your client and fall back to the reasoning field, as the SDK example does.&lt;/p&gt;

&lt;h2&gt;
  
  
  Other things worth checking before you trust it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Send your own system prompt, or the template adds a Chinese one naming the model and BOSS Zhipin.&lt;/li&gt;
&lt;li&gt;We parsed tool calls with llama.cpp's &lt;code&gt;--jinja&lt;/code&gt; support, not the &lt;code&gt;qwen3_coder&lt;/code&gt; parser the model page suggests for vLLM and SGLang.&lt;/li&gt;
&lt;li&gt;Our scope was narrow: one request at a time, single tool calls and one agent turn, never a real repository.&lt;/li&gt;
&lt;li&gt;The authors' own figures, not rerun by us: ahead of Qwen3.8-27B on Terminal-Bench 2.1 (80.9 vs 73.0) and DeepSWE (65.5 vs 42.5), behind on SWE-Bench Pro (60.3 vs 61.7).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Launch commands for 8K, 32K and 64K and timings for both cards are in the &lt;a href="https://inferencekey.com/research/nanbeige-33b-dense-agent-24gb" rel="noopener noreferrer"&gt;Nanbeige4.3-33B deployment write-up with the full VRAM ladder&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Credit to the Nanbeige team for the model, the card and the benchmarks, bartowski for the GGUF quants, axteam for an NVFP4A16 build for vLLM (untested by us), the llama.cpp maintainers for the server and its template support, and AI Search, whose news video put the model on our radar. Our part: the deployment, tests on two real cards, tracing the empty reply and the SDK example. To run the same checks on your own 24 GB NVIDIA card, start from the &lt;a href="https://inferencekey.com/research/nanbeige-33b-dense-agent-24gb" rel="noopener noreferrer"&gt;Nanbeige4.3-33B article on inferencekey.com&lt;/a&gt; and its SDK example.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>gpu</category>
      <category>selfhosted</category>
    </item>
    <item>
      <title>Clover-35B-A3B on llama.cpp: what "AR mode only" means, and a GGUF that answers in Gs</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Sat, 14 Nov 2026 06:38:19 +0000</pubDate>
      <link>https://dev.to/inferencekey/clover-35b-a3b-on-llamacpp-what-ar-mode-only-means-and-a-gguf-that-answers-in-gs-3lia</link>
      <guid>https://dev.to/inferencekey/clover-35b-a3b-on-llamacpp-what-ar-mode-only-means-and-a-gguf-that-answers-in-gs-3lia</guid>
      <description>&lt;p&gt;Four words in a pull request decide how fast DiffuSpace's Clover-35B-A3B-Preview runs on your card: "in AR mode only." It added Clover to llama.cpp, the free program many people use to run models locally. AR (autoregressive) means the model emits one token, a word piece, then starts the next. Clover was built to work differently.&lt;/p&gt;

&lt;p&gt;We loaded the official Q4_0 GGUF on one RTX 4090 on our cloud, with stock llama.cpp v0.9.0. GGUF is the single-file format llama.cpp reads; Q4_0 is a compressed build. After a small file fix, one request got 202 to 211 tokens per second, with the first token back in 0.13 to 0.31 seconds. The full 128K context window (prompt plus answer in memory) peaked at 21,924 MiB. Every timing is in the &lt;a href="https://inferencekey.com/research/clover-diffusion-llm-one-gpu" rel="noopener noreferrer"&gt;full Clover-35B-A3B test on one RTX 4090&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  "Supported" can mean the slow path
&lt;/h2&gt;

&lt;p&gt;Clover is a diffusion language model. A normal model is a typist: token, token, token. Clover drafts a block of 32 slots, commits in one pass every slot where its confidence reaches 95%, and revisits the others. That parallel filling is the speed story, and the engine has to implement it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/ggml-org/llama.cpp/pull/31306" rel="noopener noreferrer"&gt;PR #31306&lt;/a&gt; by Maphist0 landed on 2 November, and the v0.9.0 release carried it on 4 November. It runs Clover one token per step. A faster self-speculative mode (draft a block in parallel, then verify it the normal way) was left for a later pull request, and we found none open. So 208 tokens per second measures the typist, not the diffusion trick.&lt;/p&gt;

&lt;p&gt;DiffuSpace quotes 515 tokens per second on one H200 and 680 on a GB300, uncompressed, naming neither engine nor mode. The parallel modes live elsewhere. DiffuSpace's own Python code wants about 69 GB for the weights alone. &lt;a href="https://github.com/sgl-project/sglang/pull/46927" rel="noopener noreferrer"&gt;SGLang PR #46927&lt;/a&gt; from Zirui Wu brings draft-and-verify to SGLang, another open serving engine. It awaits review, and its own test logged 453.59 tokens per second on a single H200, one request at a time.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trap: a file that loads is not a file that works
&lt;/h2&gt;

&lt;p&gt;Out of the box, v0.9.0 rejects the official file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;key not found in model: clover_moe.expert_shared_count
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The tempting fix is to add that key. We did. It loaded, we asked for Australia's capital in one word, and got the letter G 600 times.&lt;/p&gt;

&lt;p&gt;A load error stops you. Garbage from a model that loads can slip into a pipeline. The root cause: DiffuSpace exported its files from its own llama.cpp build, which drifted from upstream in three places. One key is absent. There are 120 extra tensors (the named arrays of numbers that make up a model) only the diffusion mode reads. And 30 layers each keep a short vector of values written in another mathematical form.&lt;/p&gt;

&lt;p&gt;Our &lt;code&gt;patch_gguf.py&lt;/code&gt; handles all three and leaves the other weights untouched, so nothing gets compressed twice. On the 19.6 GB file it took 17 seconds, and Clover then answered Canberra. The &lt;a href="https://inferencekey.com/research/clover-diffusion-llm-one-gpu" rel="noopener noreferrer"&gt;notes on what patch_gguf.py changes&lt;/a&gt; name each group of tensors it touches. Status: fixed in our SDK example, open upstream in both DiffuSpace's files and llama.cpp. We checked the patched file only on v0.9.0.&lt;/p&gt;

&lt;p&gt;The habit: before benchmarking a new architecture, ask it one question whose answer you know. A model that replies in Gs still produces a tokens-per-second figure.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to measure on your own card
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqmkw172z97f2lgxph877.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqmkw172z97f2lgxph877.jpg" alt="Result tiles: 208 tokens per second on one RTX 4090, all 128K of context inside 24 GB, and a 16 GB memory budget at about 102 tokens per second" width="800" height="320"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Clover has 35 billion parameters, yet each token uses about 3 billion. It is a mixture of experts: 256 specialists, and each token consults 8 plus one generalist. All sit on the GPU unless llama.cpp's &lt;code&gt;--n-cpu-moe&lt;/code&gt; moves some layers of experts to system RAM, trading speed for card memory. With the 4090's memory capped:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Memory budget&lt;/th&gt;
&lt;th&gt;Experts in RAM&lt;/th&gt;
&lt;th&gt;Peak on the card&lt;/th&gt;
&lt;th&gt;Speed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;16 GB&lt;/td&gt;
&lt;td&gt;10 layers&lt;/td&gt;
&lt;td&gt;15,456 MiB&lt;/td&gt;
&lt;td&gt;about 102 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12 GB&lt;/td&gt;
&lt;td&gt;22 layers&lt;/td&gt;
&lt;td&gt;10,270 MiB&lt;/td&gt;
&lt;td&gt;about 65 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;td&gt;all&lt;/td&gt;
&lt;td&gt;2,494 MiB&lt;/td&gt;
&lt;td&gt;about 43 tokens/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All three budgets gave correct answers. Read the table for memory, not speed: a real 16 GB desktop runs at whatever its processor and RAM allow. Our real 16 GB attempt hit a broken host, so that number is missing.&lt;/p&gt;

&lt;p&gt;Three more checks:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A long prompt with a needle.&lt;/strong&gt; We buried a code in an 89,367-token prompt with the 128K window. Clover returned it in 13.45 seconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Thinking tokens.&lt;/strong&gt; Clover reasons before every answer. A 400-word essay cost 4,665 tokens and 22.7 seconds, mostly reasoning. Our gateway drops &lt;code&gt;chat_template_kwargs&lt;/code&gt;, the field that carries the off switch. That gap is ours, still open.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The base model.&lt;/strong&gt; Clover comes from Qwen3.6-35B-A3B by Alibaba's Qwen team. That base never ran on our 4090, so there is no head-to-head. Run both on your own workload.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Pin the version, not the latest
&lt;/h2&gt;

&lt;p&gt;On InferenceKey, the worker is the agent process on your GPU machine; it takes instructions from the platform and builds llama.cpp itself. Its default tag, &lt;code&gt;b9748&lt;/code&gt;, predates Clover. Pinning this one workload (a model deployment) to v0.9.0 solved it. The first compile for the GPU took 4 min 50 s. From then on, the endpoint (the URL your code calls) was ready in 11.7 seconds. The worker also needed three CUDA build variables it does not set yet; the example lists them.&lt;/p&gt;

&lt;p&gt;Credit: DiffuSpace built Clover and published the files, Alibaba's Qwen team made the base, Maphist0 wrote the llama.cpp support the maintainers released in v0.9.0, and Zirui Wu is building the SGLang path. We first saw the model on the AI Search channel. Our part was the patch and the measurements.&lt;/p&gt;

&lt;p&gt;The patch, the exact &lt;code&gt;llama-server&lt;/code&gt; command, the timings and the full memory ladder are in the &lt;a href="https://inferencekey.com/research/clover-diffusion-llm-one-gpu" rel="noopener noreferrer"&gt;Clover-35B-A3B write-up on inferencekey.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>gpu</category>
      <category>selfhosted</category>
    </item>
    <item>
      <title>Serving Hy-SimulMT live on one 48 GB GPU: the startup spike, the FP8 trap, what to measure</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Fri, 13 Nov 2026 06:36:34 +0000</pubDate>
      <link>https://dev.to/inferencekey/serving-hy-simulmt-live-on-one-48-gb-gpu-the-startup-spike-the-fp8-trap-what-to-measure-3j64</link>
      <guid>https://dev.to/inferencekey/serving-hy-simulmt-live-on-one-48-gb-gpu-the-startup-spike-the-fp8-trap-what-to-measure-3j64</guid>
      <description>&lt;p&gt;Hy-SimulMT is Tencent Hunyuan's open model for live Chinese and English interpreting, licensed Apache-2.0. One model both hears and translates, so output can begin while the speaker is mid-sentence. SimulInfer, a small service from the authors, cuts incoming audio into 1.6-second blocks and assembles the partial outputs. Behind it, vLLM (the open-source server that loads the weights onto the GPU and answers requests) runs the model through the authors' &lt;code&gt;simulst&lt;/code&gt; plugin.&lt;/p&gt;

&lt;p&gt;We ran the FP8 Interleave build on one RTX 6000 Ada 48 GB on our cloud. FP8 means the weights are stored as 8-bit numbers, which brings the files down to 32 GB. Interleave means the output also carries a transcript of the original speech. Four read clips went in at speaking speed, and first words came back 2.3 to 5.5 seconds in. The &lt;a href="https://inferencekey.com/research/hy-simulmt-live-speech-translation" rel="noopener noreferrer"&gt;full Hy-SimulMT test on one 48 GB card&lt;/a&gt; holds every table and command. Below is what you hit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Startup needs the whole card, whatever cap you set
&lt;/h2&gt;

&lt;p&gt;Once serving, the translator sat at 38 GB of the 48, hinting that a 32 GB card might do. Startup says otherwise.&lt;/p&gt;

&lt;p&gt;Before it accepts requests, vLLM pushes dummy audio through the audio encoder (the part of the model that turns sound into something the language part can read) to see how much room is left. That check filled the card, 48,510 MiB, in every run, before usage dropped back. With the authors' 16,384 batched tokens (the input vLLM may process in one step), it profiled 3,276 dummy audio items and printed thousands of out-of-memory warnings before recovering. Dropping &lt;code&gt;--max-num-batched-tokens&lt;/code&gt; to 4,096 did not remove the peak.&lt;/p&gt;

&lt;p&gt;Next we capped the server at 32 GB with &lt;code&gt;--gpu-memory-utilization 0.66&lt;/code&gt;, a shorter context (how much text the model keeps in view) and a single session. Translation kept its live pace, but startup still hit 47.4 GiB. The cap does not cover that peak, so a real 32 GB card such as an RTX 5090 could fail before serving anything. Size for 48 GB. Adding the voice clone, Qwen3-TTS-0.6B from Alibaba's Qwen team, puts usage at 48 GB instead of 38.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnwo9wfg1yx7gye45tlzf.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnwo9wfg1yx7gye45tlzf.jpg" alt="Three tiles: sentences done 0 to 2 s after the speaker in the first Chinese clip, 37.4 GiB on one 48 GB RTX 6000 Ada (47.2 GiB with the voice clone), 20.5 GiB for the 4-bit build, not servable yet" width="800" height="280"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  SimulInfer rejects the FP8 build until it finds a manifest
&lt;/h2&gt;

&lt;p&gt;Point SimulInfer at the FP8 repo and the first error is in Chinese: it cannot identify the model's output format.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ValueError: 无法识别所选模型的输出格式...
# raised in model_discovery.py: no export_manifest.json,
# and the repo name is not in PROVIDED_CHECKPOINTS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;SimulInfer only knows the full builds by repo name. Our custom backend (the code that installs and starts the services on the GPU) writes the missing &lt;code&gt;export_manifest.json&lt;/code&gt;, and then it starts. Upstream, this was still open at commit &lt;code&gt;7efcb18c&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The voice needs its own pin. vLLM-Omni 0.32.0 and 0.32.0rc1 both stop on an import error for &lt;code&gt;Qwen3TTSSpeakerEncoder&lt;/code&gt;. 0.31.0rc1 on vLLM 0.31.0 works, so the voice runs in its own environment beside the translator on vLLM 0.32.0.&lt;/p&gt;

&lt;p&gt;Check the language pair first. The model page tags 40 languages, yet SimulInfer throws a &lt;code&gt;ValueError&lt;/code&gt; for any direction except &lt;code&gt;zh-en&lt;/code&gt; and &lt;code&gt;en-zh&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Log every arrival, and read the complete flag
&lt;/h2&gt;

&lt;p&gt;"Latency" blurs two numbers. Time to first words depends on when the model judges it has heard enough. Block delay is how long the output for each 1.6-second block takes to arrive after that block's audio ends. Median block delay ran between 0.4 and 0.8 seconds across the four clips. At the 90th percentile it passed 1.5 seconds, and the worst block took 2.9. Timestamp each arrival, or the final text hides this.&lt;/p&gt;

&lt;p&gt;Quality depends on direction. We compared the output wording with human reference translations using chrF, a 0 to 100 score that counts shared character sequences. Into English, the two Chinese clips got 60.6 and 51.7. Into Chinese, the two English clips got 42.2 and 23.4.&lt;/p&gt;

&lt;p&gt;The clip at 23.4 is the warning. Two of its six sentences went missing, and a third reversed its meaning: "couldn't have included" came out as "could include". SimulInfer flagged that run with &lt;code&gt;complete=false&lt;/code&gt;, so have your client check that field. Missing output is easy to spot. A reversed negation reads as normal Chinese. Test English to Chinese on your own recordings first. The &lt;a href="https://inferencekey.com/research/hy-simulmt-live-speech-translation" rel="noopener noreferrer"&gt;per-clip timings and chrF scores for Hy-SimulMT&lt;/a&gt; also list the block delay percentiles and the whole-file runs.&lt;/p&gt;

&lt;p&gt;The authors report about 3 seconds of lag on long talks, using their StreamLAAL measure. Our lag figure is a rough approximation, not comparable. We did not reproduce their benchmarks or their comparison with human interpreters, and our sample was four read FLEURS clips, one session at a time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Send whole files unless someone is listening live
&lt;/h2&gt;

&lt;p&gt;Sent as one file, a clip of 50 to 56 seconds came back fully translated in about 26 seconds. For live use, stream PCM audio (raw, uncompressed samples) in short packets at real-time pace to SimulInfer's WebSocket.&lt;/p&gt;

&lt;p&gt;Our own API has a gap here. It takes a recorded file of up to about a minute, replays it at speaking speed and returns the arrival timeline at the end. A live mic has to talk to SimulInfer on the GPU directly for now. If you only need recorded speech turned into text, see our &lt;a href="https://inferencekey.com/research/nemotron-transcribe-instructable-asr-one-gpu" rel="noopener noreferrer"&gt;Nemotron 3.5 Transcribe test&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Tencent Hunyuan built the model, the &lt;code&gt;simulst&lt;/code&gt; plugin, SimulInfer, the TTS integration and the llama.cpp support for the 4-bit build. The Qwen team made Qwen3-TTS, served here through vLLM-Omni. Thanks also to the vLLM and llama.cpp projects, the SGLang-Omni team for their cookbook, Google for the FLEURS clips, and AI Search, whose news video pointed us to the model. Our part was the custom backend, the deployment and the measurements. Setup steps, the full memory ladder and all eight problems with their status are in the &lt;a href="https://inferencekey.com/research/hy-simulmt-live-speech-translation" rel="noopener noreferrer"&gt;Hy-SimulMT live speech translation write-up on inferencekey.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>gpu</category>
      <category>selfhosted</category>
    </item>
    <item>
      <title>Serving Groot-19B-A3B at 256K on one RTX 5090: the memory budget and the thinking trap</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Thu, 12 Nov 2026 06:37:15 +0000</pubDate>
      <link>https://dev.to/inferencekey/serving-groot-19b-a3b-at-256k-on-one-rtx-5090-the-memory-budget-and-the-thinking-trap-4b2</link>
      <guid>https://dev.to/inferencekey/serving-groot-19b-a3b-at-256k-on-one-rtx-5090-the-memory-budget-and-the-thinking-trap-4b2</guid>
      <description>&lt;p&gt;GrootTeam says its Groot-19B-A3B holds a 256K-token context on one RTX 5090, and labels that figure an estimate from memory sizes. A token is a word piece, and the context is how much text the model can see at once. We wanted a measurement, so we deployed the 20.9 GB FP8 build (weights stored in 8 bits to halve their size) with vLLM 0.32.0, the open-source server that runs the model, on one RTX 5090 on our cloud, using the authors' own settings.&lt;/p&gt;

&lt;p&gt;The estimate held: all 262,144 tokens of context started, and a 205,525-token prompt came back right. The lesson is how thin the margin is. The &lt;a href="https://inferencekey.com/research/groot-19b-agent-moe-one-card" rel="noopener noreferrer"&gt;full Groot-19B-A3B test on one RTX 5090&lt;/a&gt; has every number; below is what matters when you pick the card.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two lines in the budget: weights, then the notepad
&lt;/h2&gt;

&lt;p&gt;GPU memory pays for two things. The weights are a fixed cost. The KV cache is the model's notepad for the conversation so far, and it grows with every token of context you allow.&lt;/p&gt;

&lt;p&gt;Groot is a mixture of experts (MoE) cut down from the Qwen team's Qwen3.6-35B-A3B: about 3 billion of its 19 billion parameters work on each token, so it writes fast, yet the card must still hold all 19 billion. It inherits Qwen3.6's hybrid layout, where only 10 of its 40 layers write full notes for every token and the other 30 keep a short running summary. So the room for 256K comes from the weights dropping from 35B to 19B.&lt;/p&gt;

&lt;p&gt;On the 5090 the weights took 19.7 GiB. With &lt;code&gt;--gpu-memory-utilization 0.92&lt;/code&gt;, vLLM had 6.8 GiB left for the cache: room for 307,024 tokens, the full window plus about 45,000 spare. vLLM reports that as a maximum concurrency of 1.17x, meaning 1.17 full-length requests at once.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0gaczmenoseolx4synxq.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0gaczmenoseolx4synxq.jpg" alt="Same Groot-19B-A3B FP8 weights and vLLM command on one RTX 5090: at 32 GB the full 262,144-token context starts and a 205,525-token prompt is answered right; capped at 24 GB, vLLM refuses to start with 0.48 GiB free for context and a maximum of 9,648 tokens" width="800" height="480"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Trust the line vLLM prints, not the spec sheet
&lt;/h2&gt;

&lt;p&gt;No smaller cards were available, so we capped how much of the 5090's memory vLLM could use. That emulates a smaller budget. It is not a real smaller card.&lt;/p&gt;

&lt;p&gt;With a 24 GB budget and 65,536 tokens requested, the weights took 86% of the budget and vLLM refused to start:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ValueError: ... (1.66 GiB KV cache is needed, which is larger than the
available KV cache memory (0.48 GiB). Based on the available memory,
the estimated maximum model length is 9648.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That error answers "will it fit" with one figure: 9,648 tokens, against the 128K minimum the authors recommend. For this build, budget 32 GB. On a 24 GB card the authors suggest their 11.7 GB GGUF build (a format for llama.cpp), which we did not test.&lt;/p&gt;

&lt;p&gt;The 13.5 GB NVFP4 build (mostly 4-bit weights) at a 16 GB budget failed too, for another reason: a GPU kernel failed to compile at startup, a software fault and not a memory limit. Whether that build fits remains an open question.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trap at runtime: it thinks before every answer
&lt;/h2&gt;

&lt;p&gt;Groot reasons before every reply, and the reasoning counts as output tokens. "The capital of Australia, in one word" cost 111 to 131 tokens before "Canberra" appeared. A 300-word essay produced 2,383 to 7,156 tokens over four runs; in the two runs we counted, 2,043 and 6,785 of them were thinking. If your &lt;code&gt;max_tokens&lt;/code&gt; is tight, replies get cut off before the answer starts.&lt;/p&gt;

&lt;p&gt;Per request, the authors switch thinking off with &lt;code&gt;chat_template_kwargs&lt;/code&gt;. Our API drops that field today, so thinking stays on. That gap is on our side and still open. Until it closes, start the server with thinking off by default; the article and the SDK example's README give the one-line flag.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to measure on your own card
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cache tokens available&lt;/strong&gt; once weights load (vLLM's startup log).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Peak memory on your longest prompt.&lt;/strong&gt; Ours reached 32,016 of 32,607 MiB on the 205K prompt. Run nothing else on that GPU.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time to first token on a long prompt.&lt;/strong&gt; 34.6 seconds for 205,525 tokens, complete answer at 40.3.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Writing speed with and without the speed-up layer.&lt;/strong&gt; The authors ship a small extra layer that drafts two tokens ahead for the main model to check. With it: 262 to 304 tokens per second. Without: 170 to 211. The two measurements took different request paths, so read the roughly 25% gain as approximate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tool call shape.&lt;/strong&gt; Asked whether Lisbon or Oslo was warmer, it issued two parallel &lt;code&gt;get_weather&lt;/code&gt; calls in a single turn, then answered, 1.81 seconds end to end.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A cold start took 611.5 seconds, mostly vLLM warming up the engine; a restart with that work cached on disk warmed up in 87 seconds. Our API showed no measurable overhead in the earlier &lt;a href="https://inferencekey.com/research/sage-a1-5-agent-one-gpu" rel="noopener noreferrer"&gt;Sage-A1.5 single-GPU test&lt;/a&gt;; it was not re-measured here.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does not prove
&lt;/h2&gt;

&lt;p&gt;One GPU, one request at a time, our own prompts, a few runs each. A single correct answer from a hidden sentence shows the window works, not that the model reasons well over long documents. The agent scores are the authors' (52.1 on Terminal-Bench 2.1 against 43.8 for Qwen3.6-35B-A3B); apart from the two-call tool test, we ran no multi-step agent session. The &lt;a href="https://inferencekey.com/research/groot-19b-agent-moe-one-card" rel="noopener noreferrer"&gt;Groot-19B-A3B memory ladder and serve command&lt;/a&gt; in the appendix lists every rung we skipped.&lt;/p&gt;

&lt;p&gt;Credit to GrootTeam for the model, its four builds, the speed-up layer and the vLLM recipe, to the Qwen team for the base model, to the vLLM and FlashInfer projects for the server, and to AI Search, whose news video is where we found the model. We did the deployment, the tests and the measurements. The commands, tables and an SDK example that runs it on your own GPU are in the &lt;a href="https://inferencekey.com/research/groot-19b-agent-moe-one-card" rel="noopener noreferrer"&gt;full write-up of Groot-19B-A3B on one RTX 5090&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>gpu</category>
      <category>selfhosted</category>
    </item>
    <item>
      <title>Parsing a whole PDF in one request with Unlimited OCR 2: the crop_mode setting and a silent cut</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Wed, 11 Nov 2026 06:37:31 +0000</pubDate>
      <link>https://dev.to/inferencekey/parsing-a-whole-pdf-in-one-request-with-unlimited-ocr-2-the-cropmode-setting-and-a-silent-cut-3la1</link>
      <guid>https://dev.to/inferencekey/parsing-a-whole-pdf-in-one-request-with-unlimited-ocr-2-the-cropmode-setting-and-a-silent-cut-3la1</guid>
      <description>&lt;p&gt;The usual PDF pipeline is a loop: split the file, run OCR (optical character recognition, turning page images into text) on each page, concatenate the strings. When a table starts on one page and ends on another, every later page returns a headless fragment, and your code cannot tell they belong together.&lt;/p&gt;

&lt;p&gt;Unlimited OCR 2, a document parser Baidu's team released under the MIT license, drops the loop: you send the whole PDF in one request. We served it on one RTX 4090 with the 59-page IRS employer's tax guide, where one table spans pages 46 to 52. It worked once we flipped one setting. One failed run raised no error. The &lt;a href="https://inferencekey.com/research/unlimited-ocr-2-whole-book-parsing" rel="noopener noreferrer"&gt;full test of Unlimited OCR 2 on the 59-page tax guide&lt;/a&gt; has every table, flag and timing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setting that decides the result: crop_mode
&lt;/h2&gt;

&lt;p&gt;By default the model sees each page twice over: one overview image plus a grid of zoomed-in tiles, meant to help with small print. Set &lt;code&gt;crop_mode=false&lt;/code&gt; and it gets the overview alone.&lt;/p&gt;

&lt;p&gt;For this guide, the overview alone won on speed and accuracy:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;59 pages, one request, RTX 4090&lt;/th&gt;
&lt;th&gt;Tiles (default)&lt;/th&gt;
&lt;th&gt;crop_mode=false&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Whole document&lt;/td&gt;
&lt;td&gt;6 min 45 s&lt;/td&gt;
&lt;td&gt;3 min 44 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Distance to the PDF's own text&lt;/td&gt;
&lt;td&gt;38.1%&lt;/td&gt;
&lt;td&gt;4.2%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The errors are not spread evenly. Drop pages 46 to 54 and both modes sit near 2.4% on ordinary prose. The table did the damage. With tiles on, page 46 came back with eight rows crammed into one cell, followed by "Taxable if wages subject to social security tax and Medicare tax" written 1,508 times. Only the per-page cap of 20,000 tokens stopped it (a token is a word piece, the unit a model reads and writes). Pages 47, 50 and 52 were blank. The authors' anti-repetition guard was on, and resending pages 46 to 54 alone looped again.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsisaqawvy54nd6ghg8gf.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsisaqawvy54nd6ghg8gf.jpg" alt="Page 46 of IRS Publication 15 from Unlimited OCR 2 on one RTX 4090. Left, tile mode: 156 s, one sentence repeated 1,508 times. Right, crop_mode=false: 25 s, rows with four columns as printed, within 0.7% of the PDF text" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;With long tables, start with tiles off and spot-check the figures on the table pages.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trap: a context limit that truncates instead of failing
&lt;/h2&gt;

&lt;p&gt;Baidu ships the model with its own patched build of SGLang, an open source inference server (the program that loads the weights and answers HTTP requests). By default it reads in windows of 10 pages, freeing memory for pages that leave the window. A switch turns windowing off so the model sees all 59 pages together. The context, meaning everything the model can hold at once, is 131,072 tokens, and the page images alone used about 89,000 of it. Output ended halfway through page 27, and the server only wrote this to its log:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Requested token count (89331 input + 472000 new) exceeds the model's
context length (131072 tokens). Truncating max_new_tokens.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The cause is the &lt;code&gt;--allow-auto-truncate&lt;/code&gt; server flag. From the client side the stream simply finishes. Your only defence is to count pages in the output, which is cheap because every page opens with a &lt;code&gt;&amp;lt;PAGE&amp;gt;&lt;/code&gt; tag. Assert on it in every mode.&lt;/p&gt;

&lt;p&gt;One window with &lt;code&gt;crop_mode=false&lt;/code&gt; might fit (an 18,277-token prompt); untested.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to measure, and how to get a free reference
&lt;/h2&gt;

&lt;p&gt;A born-digital PDF grades its own OCR: the embedded text layer, the selectable text you can copy out, is an exact answer key. Our score was the share of characters you would need to edit to turn the model's text into that layer, so 0% is a perfect match. Two caveats. It punishes reading order, so a sidebar placed before its paragraph loses points with every word correct. And it needs a text layer, which scans lack; we tested no scans.&lt;/p&gt;

&lt;p&gt;Why run a model when the text is already in the file? The layer has no reading order, no block positions, no HTML tables and no markers across page breaks. Those come from the model, and they need their own checks. Each run flagged 31 to 34 paragraphs as continuing on the next page. The big table still arrived as seven per-page pieces, but five of its six joins carried an &lt;code&gt;append_table&lt;/code&gt; tag; the page 49 join was untagged. We never merged the pieces, so stitching is your code and your test.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sizing the GPU: read the share, not the peak
&lt;/h2&gt;

&lt;p&gt;SGLang reserves a fixed fraction of whatever GPU it lands on (70% unless told otherwise) and fills it with working memory. So the 19.4 GB peak on the 4090 measures that reservation, not the model's needs. A real 16 GB RTX 2000 Ada, a 70-watt workstation card, parsed the full guide with tiles off at a 13.8 GB peak and 5.6 pages per minute, 4.23% off the text layer, the same quality as the bigger card.&lt;/p&gt;

&lt;p&gt;Scope: one 59-page English document, one request at a time. Baidu aims at hundreds of pages; its 100 to 300-page benchmark is its claim, not our measurement.&lt;/p&gt;

&lt;h2&gt;
  
  
  Credit
&lt;/h2&gt;

&lt;p&gt;The model and paper are the work of Baidu's ERNIE team. Shu Wei, Jingjing Wu and Qunyi Xie are the core contributors, with Lingshu Zhang, Xiangcheng Du, Yannian Fu, Mengjie Wang, Zuyi Wang, Zhichao Liu, Yangliu Xu, Youyang Yin, Huanhuan Liu, YY and Yi Yang as contributors. It builds on their first Unlimited OCR and on DeepSeek-OCR, so the DeepSeek-OCR team shares the credit. The server is the &lt;a href="https://github.com/sgl-project/sglang" rel="noopener noreferrer"&gt;SGLang&lt;/a&gt; project's, patched by Baidu and shipped in the &lt;a href="https://huggingface.co/baidu/Unlimited-OCR-2" rel="noopener noreferrer"&gt;Unlimited OCR 2 model card repository&lt;/a&gt;. We spotted it in the AI Search channel's 1 November news video, then wrapped that server in an API and checked what it returned on a real document.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://inferencekey.com/research/unlimited-ocr-2-whole-book-parsing" rel="noopener noreferrer"&gt;write-up with the VRAM ladder, server flags and the two open rough edges&lt;/a&gt; links an SDK example for your own NVIDIA card.&lt;/p&gt;

&lt;p&gt;To compare your numbers with ours, start from the &lt;a href="https://inferencekey.com/research/unlimited-ocr-2-whole-book-parsing" rel="noopener noreferrer"&gt;Unlimited OCR 2 whole-document parsing results on inferencekey.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>python</category>
      <category>gpu</category>
    </item>
    <item>
      <title>Nemotron 3.5 Transcribe on one 32 GB Blackwell card: the prompt is your API, so test it</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Tue, 10 Nov 2026 11:28:24 +0000</pubDate>
      <link>https://dev.to/inferencekey/nemotron-35-transcribe-on-one-32-gb-blackwell-card-the-prompt-is-your-api-so-test-it-14pk</link>
      <guid>https://dev.to/inferencekey/nemotron-35-transcribe-on-one-32-gb-blackwell-card-the-prompt-is-your-api-so-test-it-14pk</guid>
      <description>&lt;p&gt;Most speech-to-text stacks bolt features on after the model: diarization (working out who said what), then a cleanup pass for the "um"s. NVIDIA's Nemotron 3.5 Transcribe moves those choices into the request. You send the audio plus one sentence, and the sentence decides whether you get every filler, a clean read, or speaker turns with a time on every word. The bugs move with it.&lt;/p&gt;

&lt;p&gt;NVIDIA released it on 28 October under the &lt;a href="https://openmdw.ai/license/1-1/" rel="noopener noreferrer"&gt;OpenMDW-1.1 license&lt;/a&gt;, with an open serving recipe: vLLM, the open-source engine that loads a model on a GPU and answers requests, plus NVIDIA's &lt;a href="https://github.com/NVIDIA-NeMo/Speech" rel="noopener noreferrer"&gt;NeMo Speech&lt;/a&gt; plugin. We took the smallest of its three files, 21.6 GB in a format called NVFP4 that needs a Blackwell card, and ran it on one 32 GB RTX PRO 4500 Blackwell on our cloud. Every figure is from the &lt;a href="https://inferencekey.com/research/nemotron-transcribe-instructable-asr-one-gpu" rel="noopener noreferrer"&gt;full Nemotron 3.5 Transcribe test on one 32 GB card&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The request barely changes, one field does the work
&lt;/h2&gt;

&lt;p&gt;We wrapped it in a custom backend, a short Python file telling our platform how to install and run a model on a GPU machine. It answers on the standard OpenAI speech route, &lt;code&gt;/v1/audio/transcriptions&lt;/code&gt;, like our Qwen3-ASR deployment, so existing client code keeps its shape. Only the prompt is new:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;audio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;transcriptions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"default"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"meeting.ogg"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"rb"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"speakers"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timestamp_granularities&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"word"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;speakers&lt;/code&gt; is a style name the backend swaps for a full instruction; free text also works. Word timings arrive in a separate &lt;code&gt;ctc_timestamps&lt;/code&gt; object: start, end and speaker for each word.&lt;/p&gt;

&lt;p&gt;On the opening three minutes of a &lt;a href="https://github.com/babylonhealth/primock57" rel="noopener noreferrer"&gt;PriMock57&lt;/a&gt; doctor-patient consultation, verbatim and clean each took 4.1 s of GPU time, and speakers with word timings 4.4 s for 561 timed words.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trap: an instruction is a contract the model may not keep
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu25hs2d6gn99lbf0wtnn.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu25hs2d6gn99lbf0wtnn.jpg" alt="The same 3-minute recording under two prompts: speakers keeps tags and fillers; clean drops fillers but adds 50 unrequested speaker tags and keeps the false start &amp;quot;Hello, how.&amp;quot;" width="800" height="410"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The clean style asks the model to "omit fillers and accidental repetitions; keep the remaining spoken wording unchanged". It closes with "Output: transcript only" and never mentions tags. The fillers did disappear. Yet the output carried 50 &lt;code&gt;&amp;lt;spk:&lt;/code&gt; tags across three minutes, and the false start "Hello, how." survived. Code expecting plain prose must strip them, or use the speakers style on purpose.&lt;/p&gt;

&lt;p&gt;When a sentence picks the format, each style is a code path that needs tests. Cheap ones: no &lt;code&gt;&amp;lt;spk:&lt;/code&gt; in clean output, a filler count in verbatim. Verbatim held on to 33 of the 34 "um" and "uh" in the human reference. A compliance team should measure that on its own audio. The write-up prints the &lt;a href="https://inferencekey.com/research/nemotron-transcribe-instructable-asr-one-gpu" rel="noopener noreferrer"&gt;exact verbatim, clean and speaker instructions we sent&lt;/a&gt;, so you can rerun those checks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Score each set separately
&lt;/h2&gt;

&lt;p&gt;Accuracy here is word error rate, the share of words wrong against a human transcript. On 40 short English clips from the &lt;a href="https://huggingface.co/datasets/hf-audio/open-asr-leaderboard" rel="noopener noreferrer"&gt;Open ASR Leaderboard data&lt;/a&gt;, Nemotron scored 6.82%. Scored the same way on those clips, &lt;a href="https://inferencekey.com/research/underdog-bark-speech-to-text-test" rel="noopener noreferrer"&gt;Bark 3.0 Pro and Qwen3-ASR-1.7B in our earlier speech-to-text test&lt;/a&gt; landed at 6.28% and 7.03%. In between: you gain output flexibility, not a lower error rate.&lt;/p&gt;

&lt;p&gt;The average hides a spread. On audiobook speech it was the best of the three at 0.94%. On AMI, recorded office meetings full of crosstalk, it dropped to 14.62%: worse than Bark (11.54%), better than Qwen3-ASR (16.92%). Meeting products: test your own recordings first.&lt;/p&gt;

&lt;p&gt;The sample is modest: 924 words of clips, a three-minute and a five-minute recording, one request at a time. NVIDIA's 4.71% mean error across nine English sets is its run of the largest version, which we did not repeat. The 2,238 seconds of audio per second on its &lt;a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B" rel="noopener noreferrer"&gt;model page&lt;/a&gt; comes from a packaged engine that is not open source yet, and NVIDIA expects the open path to be roughly two times slower. We did not measure throughput or parallel requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 24 GB question: a software cap is not a smaller card
&lt;/h2&gt;

&lt;p&gt;vLLM's &lt;code&gt;--gpu-memory-utilization&lt;/code&gt; flag sets what share of the card's memory it may take. Lowering it on the 32 GB card looks like a quick 24 GB test. It is not. With a 23.5 GiB budget, the card still used 25,712 MiB. Only a real 24 GB slice is evidence.&lt;/p&gt;

&lt;p&gt;That slice, from an RTX PRO 6000 Blackwell, gets a quarter of its compute. Two speed tricks had to go: speculative decoding (drafting several words ahead and checking them in one pass) and CUDA graphs (GPU work recorded once and replayed). With a 16,384-token context and 2 requests at once, it matched the 6.82%, but the three-minute speaker run took 35.3 s instead of 4.4. With speculative decoding on, memory ran out. NVFP4 needs Blackwell, so an RTX 4090 or anything older is out.&lt;/p&gt;

&lt;h2&gt;
  
  
  What will bite you in the build
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;FlashInfer kernel build killed in a 62 GB container.&lt;/strong&gt; About 18 parallel nvcc compiles. Fixed with the prebuilt kernels and &lt;code&gt;MAX_JOBS=6&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A vLLM process left behind after a delete.&lt;/strong&gt; Our platform stops custom backends with SIGKILL, so &lt;code&gt;vllm serve&lt;/code&gt; kept 25 to 29 GB and the next start failed. Worked around with &lt;code&gt;PR_SET_PDEATHSIG&lt;/code&gt; and a stale pid file; the platform fix (SIGTERM with a grace period) is still to file.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A 2 MB upload cap on our transcription route.&lt;/strong&gt; Roughly eight minutes of compressed speech, against the hour the model accepts. Still open, so split long recordings; whether speaker labels carry across pieces is untested.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;NVIDIA built the model, the NeMo Speech plugin and the serving recipe. The vLLM project built the engine and the FlashInfer team the prebuilt kernels. SGLang-Omni provides an alternative serving path. Babylon Health published PriMock57, hf-audio the test clips, and we spotted the model in an AI Search news video. Our part was the backend, the deployment and the measurements.&lt;/p&gt;

&lt;p&gt;The flags, full tables, VRAM ladder and an SDK example that runs all three styles on your recording are in the &lt;a href="https://inferencekey.com/research/nemotron-transcribe-instructable-asr-one-gpu" rel="noopener noreferrer"&gt;Nemotron 3.5 Transcribe research write-up on inferencekey.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>gpu</category>
      <category>python</category>
    </item>
    <item>
      <title>Serving a stateful video world model on one RTX 4090: memory, flags and seeds in Hy-Nomas</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Mon, 09 Nov 2026 06:37:54 +0000</pubDate>
      <link>https://dev.to/inferencekey/serving-a-stateful-video-world-model-on-one-rtx-4090-memory-flags-and-seeds-in-hy-nomas-2k63</link>
      <guid>https://dev.to/inferencekey/serving-a-stateful-video-world-model-on-one-rtx-4090-memory-flags-and-seeds-in-hy-nomas-2k63</guid>
      <description>&lt;p&gt;Most open video models give you a clip and drop it. Hy-Nomas holds on to the world between calls, and that reshapes the server around it.&lt;/p&gt;

&lt;p&gt;The model comes from Tencent Hunyuan, with weights and code under Apache-2.0 on the &lt;a href="https://huggingface.co/tencent/Hy-Nomas" rel="noopener noreferrer"&gt;Hy-Nomas model card&lt;/a&gt; and in the &lt;a href="https://github.com/Tencent-Hunyuan/Hy-Nomas" rel="noopener noreferrer"&gt;Tencent-Hunyuan/Hy-Nomas repository&lt;/a&gt;. You steer it in 5-second turns, each built on the frames already made. The default launch shares one video between eight GPUs, but the authors also ship a low-memory preset for a lone RTX 4090. We tried it as published, on a 24 GB RTX 4090 on our cloud. Every figure below comes from the &lt;a href="https://inferencekey.com/research/hy-nomas-world-model-one-gpu" rel="noopener noreferrer"&gt;Hy-Nomas single-GPU research write-up&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Headline numbers: 88 to 94 seconds of GPU work per 5-second segment once warm (after the first call has loaded everything), an 18.5 GiB peak on the card, and a scene that stayed coherent for three turns.&lt;/p&gt;

&lt;h2&gt;
  
  
  The server is stateful, so plan in worlds
&lt;/h2&gt;

&lt;p&gt;As it draws, the model writes notes on the scene and reads them before the next turn. The preset keeps only the newest few seconds of notes on the GPU and parks the rest in main memory (host RAM). Our backend, a small program between the platform's API and the model, wraps the authors' own generation service and keeps a single world resident. A request carries one turn and returns a 5-second MP4; the next one picks up where it ended.&lt;/p&gt;

&lt;p&gt;So one GPU holds exactly one live world, for one person. Size the fleet by open worlds, not requests per second.&lt;/p&gt;

&lt;p&gt;Host RAM is the second budget. The parked notes grew by about 5.9 GiB per segment, linearly over the three segments we ran. Our SDK example requests 64 GB of RAM, a provisioning pick rather than a measured floor. A one-minute world is twelve segments, so check the host side first.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trap: two decode flags from the live-demo script
&lt;/h2&gt;

&lt;p&gt;Decoding turns the model's internal result into video frames. The authors' live-demo script starts it while the video model still sits on the GPU:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;inference.seg_ar_segment_vae_decode=true
model.vae.stable_prefix_streaming=true
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On 24 GB, the pair runs out of memory. We removed both, so decoding waits until drawing is done. Status: worked around in our example, not yet reported upstream.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hardware check that is easy to miss
&lt;/h2&gt;

&lt;p&gt;The preset compiles a GPU routine (a kernel) for its decoder on NVIDIA's CUTLASS, using 8-bit numbers (FP8). It targets one GPU generation only: Ada, the RTX 4090 family. Any other generation gets an error, so an RTX 5090 or an A40 should fail, though we ran neither. The spec is "24 GB and Ada", not just "24 GB".&lt;/p&gt;

&lt;p&gt;To find the floor, we capped memory on the same card:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Memory available&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;24 GB RTX 4090, real card&lt;/td&gt;
&lt;td&gt;works, 88 to 94 s per segment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;capped at 20 GiB&lt;/td&gt;
&lt;td&gt;works over two turns, no slowdown&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;capped at 16 GiB&lt;/td&gt;
&lt;td&gt;fails while moving the text reader to the GPU&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;At 16 GiB it crashes before drawing anything, so lower resolution would not help. We have not tried a real 20 GB card, and the authors' own one-card run hit 20.53 GiB with segments longer than ours.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to log on every turn
&lt;/h2&gt;

&lt;p&gt;Of a ninety-second turn, roughly 35 seconds is drawing, roughly 9 is writing the scene notes, and the remainder goes to decoding frames and encoding the MP4. A cold first call after a restart took as long as 138 seconds. The &lt;a href="https://inferencekey.com/research/hy-nomas-world-model-one-gpu" rel="noopener noreferrer"&gt;per-stage timings in the Hy-Nomas write-up&lt;/a&gt; show which stage a faster turn would have to shrink.&lt;/p&gt;

&lt;p&gt;Log per call:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;GPU seconds per segment, warm and cold&lt;/li&gt;
&lt;li&gt;peak GPU memory, which crept up each turn for reasons we did not find&lt;/li&gt;
&lt;li&gt;host RAM after each segment&lt;/li&gt;
&lt;li&gt;an md5 hash of each segment's video file&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A seed is the fixed random number that drives generation. The same turns and seed produced byte-identical files in four runs on one machine. A bug report can ship its exact inputs and reproduce the clip. The seed also belongs to the world: an early backend answered a seedless continue turn with an HTTP 400, as the seed no longer matched. Backend 0.1.2 now keeps the seed from the first turn.&lt;/p&gt;

&lt;p&gt;Two platform limits, both open, shaped the API: calls that wait for the answer stop at 300 seconds, so one segment per call, and messages cap at 8 MB, so the client stitches segments itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the model did and did not follow
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxmw3wioykl1ss0mekbos.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxmw3wioykl1ss0mekbos.jpg" alt="Turn 2 and turn 3 of one Hy-Nomas world on an RTX 4090: blue flowers appear beside the fox as asked, while the requested snow never comes and the fox walks on when asked to stop" width="800" height="385"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Continuity was the strength: fox, path and light survived from call to call. Steering was weak: one scene change in four landed. Flowers showed up; snow, a storm and requested actions did not. Our prompts were handwritten, without Nomas-PE, the authors' far larger prompt rewriter; whether it brings the weather is untested, like camera paths.&lt;/p&gt;

&lt;p&gt;At roughly 18 times slower than real time, this is for sketching scenes that stay put across turns. Think rehearsal room, not stage.&lt;/p&gt;

&lt;p&gt;Credits: the model, its single-GPU preset, its INT8 and FP8 weights and the generation service our backend wraps are the work of Tencent Hunyuan, described in their &lt;a href="https://arxiv.org/abs/2610.36729" rel="noopener noreferrer"&gt;paper on arXiv&lt;/a&gt;. Tencent fine-tuned the decoder from PixelUMM (Wei et al., 2026) and published a Diffusers conversion built on a fork by jiaxiangc. Qwen2.5-VL-7B, from the Qwen team, reads the instructions, and NVIDIA's CUTLASS sits under the FP8 kernel. We first saw the model in a news video from AI Search. Our share: the backend, measurements, memory caps and SDK example. The per-turn table, request format, setup steps and every problem with its status are in the &lt;a href="https://inferencekey.com/research/hy-nomas-world-model-one-gpu" rel="noopener noreferrer"&gt;full Hy-Nomas test on one RTX 4090&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>gpu</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Serving Sage-A1.5 on an RTX 4090: one broken flag, one queue, and a self-check mode to measure</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Sun, 08 Nov 2026 06:36:39 +0000</pubDate>
      <link>https://dev.to/inferencekey/serving-sage-a15-on-an-rtx-4090-one-broken-flag-one-queue-and-a-self-check-mode-to-measure-4cdj</link>
      <guid>https://dev.to/inferencekey/serving-sage-a15-on-an-rtx-4090-one-broken-flag-one-queue-and-a-self-check-mode-to-measure-4cdj</guid>
      <description>&lt;p&gt;Run Sage-A1.5 as a research agent on your own card and the surprises are not in the model. They hide in a reasoning flag, a request limit, a token budget and one system prompt string.&lt;/p&gt;

&lt;p&gt;Sage-A1.5 comes from Shanghai AI Laboratory (the InternScience organisation on Hugging Face), with a paper by Wenjie Lou and 98 co-authors. It is retrained from the Qwen team's open 35B model and the lab's earlier Agents-A1. The authors publish an official &lt;a href="https://huggingface.co/InternScience/Sage-A1.5-Q4_K_M-GGUF" rel="noopener noreferrer"&gt;Q4_K_M GGUF&lt;/a&gt;, a compressed single-file version of about 22 GB, under Apache-2.0. Stock llama.cpp, the open server that runs models on one machine, loads it without changes. We ran it on one RTX 4090 on our cloud; every measurement is in the &lt;a href="https://inferencekey.com/research/sage-a1-5-agent-one-gpu" rel="noopener noreferrer"&gt;Sage-A1.5 single-GPU research write-up&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why it fits on 24 GB, and what that means for sizing
&lt;/h2&gt;

&lt;p&gt;It is a mixture-of-experts model: the weights are split into 256 small specialist blocks, and each token (roughly a word piece) goes through 8 of them plus one shared block. So of 35 billion parameters on disk, about 3 billion do work per token. Speed follows the 3 billion. Memory follows the 35.&lt;/p&gt;

&lt;p&gt;Size the card for the whole file plus context, the conversation the model keeps in memory. With 32,768 tokens of context and two parallel slots (requests served at once), text only, it peaked at 21.3 of the card's 23.5 GiB. The card's 64K setup with vision peaked at 23.1 GiB, 128K text only at 23.4 GiB, with no loss of speed. At 32K it wrote about 136 tokens per second for one request, first token after about a quarter second.&lt;/p&gt;

&lt;p&gt;Below 24 GB, llama.cpp can park experts in system RAM (&lt;code&gt;--n-cpu-moe&lt;/code&gt;). We only simulated smaller cards: a simulated 16 GB card ran about 5.6 times slower and still got the research question right. We never tried real 16 and 12 GB cards, the RTX 3090 or the uncompressed model.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap one: the model card's reasoning flag
&lt;/h2&gt;

&lt;p&gt;The llama.cpp command in the model card will not start.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nt"&gt;--reasoning-format&lt;/span&gt; deepthink   &lt;span class="c"&gt;# llama-server exits: "Unknown reasoning format: deepthink"&lt;/span&gt;
&lt;span class="nt"&gt;--reasoning-format&lt;/span&gt; deepseek    &lt;span class="c"&gt;# what llama.cpp accepts&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The flag tells the server how to split thinking from answer. Even a much newer llama.cpp lacks &lt;code&gt;deepthink&lt;/code&gt;. Our example uses &lt;code&gt;deepseek&lt;/code&gt;; the authors' card still carries the old value.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap two: sub-agents that wait in line
&lt;/h2&gt;

&lt;p&gt;Agentic Orchestration gives the lead model one tool that hands out tasks to sub-agents, copies of the model that search on their own and report back. Asked for the years between the discoveries of hafnium, rhenium and technetium, the lead spawned three sub-agents at once, right in all three test runs. Every tool call arrived in the exact format our code needed.&lt;/p&gt;

&lt;p&gt;"At once" needs a server that takes several requests. Our platform's default after a deploy is one request in flight, so the sub-agents queued and the run took 72.0 seconds. With the limit at two, runs took 50 and 56 seconds. That default has not changed yet. Serving it yourself? Check your slot count (&lt;code&gt;-np&lt;/code&gt;) and any proxy in front.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap three: the thinking eats your token budget
&lt;/h2&gt;

&lt;p&gt;The model reasons through hidden tokens before it answers, and those count against &lt;code&gt;max_tokens&lt;/code&gt;. A two-sentence definition spent 1,905 tokens reasoning and took 14.4 seconds.&lt;/p&gt;

&lt;p&gt;With &lt;code&gt;max_tokens&lt;/code&gt; at 4,096, our first plain question returned 4,096 tokens of "Thinking Process:" and no answer. Our non-streaming endpoint also dropped the reasoning field once content was set. The example asks for 16,384 and streams the agent loops, getting tokens as they are written. Plan for research-job latency, not chat-box latency.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verification Mode: measure it, do not assume it
&lt;/h2&gt;

&lt;p&gt;The authors' Verification Mode tells the model to check its work before answering. On BrowseComp, a benchmark of hard-to-find web facts, they report it lifting the score from 72.5% to 82.9% over all 1,266 questions. We did not reproduce their benchmarks.&lt;/p&gt;

&lt;p&gt;On our easy, Wikipedia-backed questions it changed no answer and doubled the work:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Off&lt;/th&gt;
&lt;th&gt;On&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Answers right&lt;/td&gt;
&lt;td&gt;6 of 6&lt;/td&gt;
&lt;td&gt;6 of 6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Average time&lt;/td&gt;
&lt;td&gt;11.2 s&lt;/td&gt;
&lt;td&gt;25.0 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Average tokens written&lt;/td&gt;
&lt;td&gt;934&lt;/td&gt;
&lt;td&gt;1,890&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The added turns re-read facts the model already had. Runs varied: in the second hafnium pair, the checked run used one tool call less than the unchecked one.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F01me4i0gs5dd99bvjj0c.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F01me4i0gs5dd99bvjj0c.jpg" alt="The same hafnium question on one RTX 4090 with Verification Mode off (13.6 s, 5 tool calls, 1,159 tokens) and on (24.7 s, 9 tool calls, 2,320 tokens), both with the right answer, the on side re-checking facts already confirmed" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The switch lives in the chat template, the script that turns messages into the model's prompt. Directly against llama-server, the documented &lt;code&gt;chat_template_kwargs={"verification": true}&lt;/code&gt; works, with a non-empty system message. Our endpoint does not pass those kwargs yet. So the example appends the template's trigger line to the system prompt, with a lowercase "protocol" (the project page capitalises it, the template does not). We checked that the model sees the same prompt both ways.&lt;/p&gt;

&lt;p&gt;Log wall time, tool calls, completion tokens and correctness per question, off and on. Try it on your hard cases and keep it off for lookups.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does not prove
&lt;/h2&gt;

&lt;p&gt;A handful of easy questions, one agent at a time, on one card, exercises tool calls, task hand-off and answers. It says nothing about research quality at scale.&lt;/p&gt;

&lt;p&gt;The GPU memory ladder, speed table, deploy timings and every issue with its status are in the &lt;a href="https://inferencekey.com/research/sage-a1-5-agent-one-gpu" rel="noopener noreferrer"&gt;complete Sage-A1.5 on one RTX 4090 test&lt;/a&gt;. Runnable code: the &lt;a href="https://github.com/inferencekey/inferencekey-sdk/tree/main/examples/sage-a1-5-research-agent-llamacpp-private-nvidia" rel="noopener noreferrer"&gt;Sage-A1.5 SDK example&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Credit to the Sage-A1.5 team for the model, both modes and the GGUF, to the Qwen team for the base model, to the llama.cpp project for the server, to Wikipedia for its public API, and to AI Search, whose news video pointed us to the model. Before your first deploy, read the &lt;a href="https://inferencekey.com/research/sage-a1-5-agent-one-gpu" rel="noopener noreferrer"&gt;problems and fixes section of the Sage-A1.5 article&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>opensource</category>
      <category>gpu</category>
    </item>
    <item>
      <title>Ming-3.0-tts on a 16 GB card: the GPU sat mostly idle, and why a bigger one barely helps</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Sat, 07 Nov 2026 06:37:23 +0000</pubDate>
      <link>https://dev.to/inferencekey/ming-30-tts-on-a-16-gb-card-the-gpu-sat-mostly-idle-and-why-a-bigger-one-barely-helps-4bo0</link>
      <guid>https://dev.to/inferencekey/ming-30-tts-on-a-16-gb-card-the-gpu-sat-mostly-idle-and-why-a-bigger-one-barely-helps-4bo0</guid>
      <description>&lt;p&gt;When a speech model runs slower than real time, the reflex is to rent a bigger GPU. With Ming-3.0-tts-0.5B that buys very little. An RTX 2000 Ada sat idle more than two thirds of the time. An RTX 4090 worked at most one moment in five, and the speedup was about 17%.&lt;/p&gt;

&lt;p&gt;The model comes from the inclusionAI team at Ant Group. They released weights and code under MIT, with their own benchmarks and a frank list of limitations. It copies a voice from a few seconds of sample audio, builds one from a text prompt describing it, and turns inline tags like &lt;code&gt;[Laughter]&lt;/code&gt; or &lt;code&gt;[Sigh]&lt;/code&gt; into sounds. Their model page only mentions H800-80GB and H20-96G cards, with no memory figure. We ran their unmodified code on a small card to get one. Every number is in our &lt;a href="https://inferencekey.com/research/ming-tts-voice-design-small-gpu" rel="noopener noreferrer"&gt;Ming-3.0-tts memory and speed test on an RTX 2000 Ada&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Low GPU utilisation is the first thing to check
&lt;/h2&gt;

&lt;p&gt;"Real time" is a ratio: seconds of audio produced per second of compute. A 20.7-second English paragraph cost 31 to 32 seconds on the RTX 2000 Ada. On the 4090, a 20.9-second paragraph cost 27 seconds. Its faster host processor may explain part of that gain.&lt;/p&gt;

&lt;p&gt;The culprit is the decode loop, which builds the audio step by step, 80 milliseconds per step. Between steps, Python code on the CPU prepares and launches the next one. Keeping up with playback takes 12.5 steps per second. We measured about 8.3 to 8.9. The GPU is a fast bus stuck behind a slow ticket check at every stop.&lt;/p&gt;

&lt;p&gt;Before sizing hardware for a text-to-speech (TTS) model, run &lt;code&gt;nvidia-smi&lt;/code&gt; while it speaks and read utilisation, the share of time the GPU is working. If it stays low, a faster card will not buy you speed. One possible way out, untested by us: vLLM-Omni, an engine built for fast serving, which Ming's roadmap marks as supported.&lt;/p&gt;

&lt;h2&gt;
  
  
  Emulate the small card before you rent it
&lt;/h2&gt;

&lt;p&gt;No real card below 16 GB ran. Two 12 GB RTX 3080 Ti machines booted without the SSH access our installer needs, and no 8 GB card was available to rent. So we faked one. PyTorch, the library Ming runs on, has a call, &lt;code&gt;torch.cuda.set_per_process_memory_fraction&lt;/code&gt;, that limits how much GPU memory the process may take. We lowered it on the 16 GB card until loading broke.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwep6wetg2x3vf5dtjxql.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwep6wetg2x3vf5dtjxql.jpg" alt="Speed on real cards, RTX 4090 at 0.76x against RTX 2000 Ada at 0.65x real time, and the PyTorch memory cap ladder on the 16 GB card: 8, 4 and 3 GiB gave the same audio, 2.75 GiB failed to load" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;With caps of 8, 4 and 3 GiB, the output WAVs had the same md5 checksum as the uncapped run, and speed did not change. At 2.75 GiB, setup died with "CUDA out of memory". Add roughly 0.4 GB for the driver and the article's estimate lands at a 4 GB card. Nobody ran that on real hardware, and a smaller chip may also be slower.&lt;/p&gt;

&lt;p&gt;Copy the hashing habit: loading under a cap proves little, matching bytes prove the cap changed nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Log the timings with every request
&lt;/h2&gt;

&lt;p&gt;Our custom backend, a short Python file that wraps the authors' code behind an HTTP API, returns its own measurements next to the audio. This came back for a cloned line with three tags on the RTX 2000 Ada:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"timings"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"gpu_s"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;11.02&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"realtime_factor"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.63&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"peak_vram_mib"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2922&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That request spent 11.02 seconds of GPU time on 7.0 seconds of audio. Strip the tags and the same line took 8.05 seconds for 5.3 seconds of audio. Tags are extra sound, and sound costs time. Log these fields from day one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three traps you will hit
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Numbers.&lt;/strong&gt; Before speaking, English text goes through normalisation, a step that spells numbers and symbols out as words. It read 4521 as "four thousand five hundred twenty, one", pause included, and turned the phone number 555-0142 into a subtraction. Spanish text bypasses that step, and "1.249,99 euros, un 12,5 %" turned into nonsense. Spelling English numbers as words did not help. Check every price by ear.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tags by language.&lt;/strong&gt; The authors document tags for English and Chinese. We only tried them in English, so tags in Spanish text remain untested.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The OpenAI speech route.&lt;/strong&gt; OpenAI-style apps request speech from a dedicated endpoint, which on InferenceKey cannot yet carry a custom backend's audio (open, IK-531). The example goes through the chat route and decodes the WAV out of the message text.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What the test does and does not show
&lt;/h2&gt;

&lt;p&gt;We ran each clip through Whisper, OpenAI's speech-to-text model: 17 of 19 lines matched word for word across English, Spanish and Chinese, and both misses were numbers. A speaker-recognition model rated the clones 0.87 in English and 0.84 in Spanish, both cloned from an English reference. Keep in mind that Ming relies on that same model to capture a voice, which tilts the score towards Ming. No human listened, each line ran once, and the reference was a clean synthetic clip from Kyutai's Pocket TTS. The backend handles a single request at once, so concurrent load is untested.&lt;/p&gt;

&lt;p&gt;My read: good for recorded narration, wrong for a live voice agent. If you need Spanish speed more than cloning or tags, our &lt;a href="https://inferencekey.com/research/pocket-tts-spanish-cpu-voice" rel="noopener noreferrer"&gt;Spanish Pocket TTS voice on CPU&lt;/a&gt; reached 1.05x real time on a desktop CPU. Nobody has compared the two for quality.&lt;/p&gt;

&lt;p&gt;From empty caches, the backend was ready to serve after 60.3 seconds, about half of it installing Python libraries. The article's appendix has the &lt;a href="https://inferencekey.com/research/ming-tts-voice-design-small-gpu" rel="noopener noreferrer"&gt;setup, per-request tables and VRAM ladder for Ming-3.0-tts&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Credit for the model goes to inclusionAI at Ant Group, to OpenAI for Whisper, to Kyutai, who made Pocket TTS, and to the AI Search channel, whose news video pointed us to Ming. The SDK example, the audio samples and every measurement are in the &lt;a href="https://inferencekey.com/research/ming-tts-voice-design-small-gpu" rel="noopener noreferrer"&gt;Ming-3.0-tts research article on inferencekey.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>python</category>
      <category>gpu</category>
    </item>
    <item>
      <title>Deploying Underdog Bark 3.0 Pro: verbatim transcripts, custom backends and the chunk setting</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Fri, 06 Nov 2026 06:36:57 +0000</pubDate>
      <link>https://dev.to/inferencekey/deploying-underdog-bark-30-pro-verbatim-transcripts-custom-backends-and-the-chunk-setting-57dk</link>
      <guid>https://dev.to/inferencekey/deploying-underdog-bark-30-pro-verbatim-transcripts-custom-backends-and-the-chunk-setting-57dk</guid>
      <description>&lt;p&gt;When you pick a speech-to-text model for a product, the first question is not which one hears better. It is whether your transcripts should keep every "uh, you know" the speaker said. We ran into that while putting Underdog Bark 3.0 Pro next to Qwen3-ASR-1.7B on one RTX 3090 on our cloud.&lt;/p&gt;

&lt;p&gt;On 40 English clips drawn from the Open ASR Leaderboard test data, Bark got 6.28% of the words wrong and Qwen3-ASR-1.7B got 7.03%. Bark is 1,254 MB on disk, Qwen3-ASR-1.7B is 4,699 MB. Bark is the third link in a chain: NVIDIA's Parakeet TDT 0.6B v3 is the base, Moondream improved it, and Conway Research post-trained and compressed the result. The clips and the leaderboard itself come from the hf-audio team. We packaged it and did the counting. The per-set numbers and the method are in the &lt;a href="https://inferencekey.com/research/underdog-bark-speech-to-text-test" rel="noopener noreferrer"&gt;full Bark 3.0 Pro vs Qwen3-ASR test write-up&lt;/a&gt;. Here is what matters when you build on it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Split the error rate before you trust it
&lt;/h2&gt;

&lt;p&gt;Word error rate adds up words heard wrong, words left out and words added. The total hides which one you have:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Heard wrong&lt;/th&gt;
&lt;th&gt;Left out&lt;/th&gt;
&lt;th&gt;Added&lt;/th&gt;
&lt;th&gt;Total&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Bark 3.0 Pro&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;6.28%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-ASR-1.7B&lt;/td&gt;
&lt;td&gt;28&lt;/td&gt;
&lt;td&gt;29&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;7.03%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Qwen3-ASR actually heard slightly better. It lost on words left out, and most of those were fillers and repeats: "you know", "on, on", "how, how". The leaderboard's human answer keys keep those words. Bark is verbatim on purpose. Its authors write that it "does not clean speech up". Qwen3-ASR tends to tidy them away.&lt;/p&gt;

&lt;p&gt;Before comparing, we ran both texts through the English normalizer OpenAI wrote for Whisper, a cleanup pass that drops case, punctuation and "uh" or "um", then scored with jiwer. If your product wants clean text, build references in that style, or the benchmark rewards the wrong habit. Being literal cuts both ways: Bark writes stutters such as "h-how" as heard, and the scorer marks them wrong.&lt;/p&gt;

&lt;p&gt;Only 924 words were scored, so a couple of words can move a set. Call it a near tie that leans toward Bark.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why it needs its own backend
&lt;/h2&gt;

&lt;p&gt;The usual serving programs, vLLM, SGLang and llama.cpp, load a model and answer requests, but none runs Parakeet TDT models. So Bark gets a custom backend: one Python file, under 200 lines, built on Hugging Face's Transformers library, as the model page says. Any client of OpenAI's transcription API can call it. The weights load in bf16, the 16-bit number format they are published in. Each recording gets 0.25 s of silence in front, as the model card recommends. Decoding is greedy: the model takes its single most likely output at each step instead of weighing alternatives.&lt;/p&gt;

&lt;p&gt;The trap was dependencies. Bark needs Transformers 5.19.0, and the &lt;code&gt;qwen-asr&lt;/code&gt; package pins 4.57.6. With no shared environment, Qwen3-ASR ran on SGLang, so any speed comparison compares setups as much as models.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setting that decides your GPU is chunk length
&lt;/h2&gt;

&lt;p&gt;Peak memory allocated by PyTorch, the library underneath Transformers, was 1.243 GiB on the short clips. The five-minute file, sent as one request, went to 2.151 GiB.&lt;/p&gt;

&lt;p&gt;We capped the 3090 at about 2 GB to imitate a smaller card. The clips passed. The five-minute file failed with &lt;code&gt;CUDA out of memory. Tried to allocate 476.00 MiB ... 2.00 GiB allowed&lt;/code&gt;. Cutting long audio into 60-second pieces fixed it: peak 1.369 GiB, the same transcripts on all 40 clips, and no slowdown. Our example defaults to 600-second pieces, so lower it for small cards.&lt;/p&gt;

&lt;p&gt;The same knobs run it with no GPU at all:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;BARK_DEVICE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;cpu
&lt;span class="nv"&gt;BARK_CPU_THREADS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;4
&lt;span class="nv"&gt;BARK_CHUNK_SECONDS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;60
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With four CPU threads on a server, the five-minute file finished in 31 seconds; sixteen threads cut that to 14. Output was identical to the GPU run on every one of the 40 clips. Changing any of these variables needs a delete and a fresh deployment.&lt;/p&gt;

&lt;p&gt;A cap mimics how little memory a small card has, but not how slow it is, and none of the seven small card types we tried to rent was in stock. So we cannot name one as enough.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to measure on your own audio
&lt;/h2&gt;

&lt;p&gt;Measure these:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Errors split into heard wrong, left out and added, against references in your own style.&lt;/li&gt;
&lt;li&gt;GPU time and round trip (until the answer is back) separately. Bark needed about 0.12 seconds of GPU work per clip, while a typical clip took half a second round trip. The network was the slow part.&lt;/li&gt;
&lt;li&gt;Peak memory on your longest real file.&lt;/li&gt;
&lt;li&gt;Whether the whole transcript comes back. On the five-minute file, Qwen3-ASR on SGLang with default settings gave back 212 words and then quit. We have not dug into the cause.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On InferenceKey, transcription uploads stop at about 2 MB, roughly 65 seconds of uncompressed audio, so a 9.7 MB WAV was rejected. The same five minutes as compressed audio fit in 1.1 MB. That limit is open, and the &lt;a href="https://inferencekey.com/research/underdog-bark-speech-to-text-test" rel="noopener noreferrer"&gt;SDK example and its README in the Bark research post&lt;/a&gt; document the workaround.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does not tell you
&lt;/h2&gt;

&lt;p&gt;We did not reproduce the model card's leaderboard score of 4.29%, which is self-reported. We used the original, uncleaned test sets, so our absolute numbers do not compare with the card's table, only the two models with each other. One real GPU. Bark is English only, and its authors say it has no word timings or speaker names yet. If you need those, our test of &lt;a href="https://inferencekey.com/research/nemotron-transcribe-instructable-asr-one-gpu" rel="noopener noreferrer"&gt;Nemotron 3.5 Transcribe with timed, speaker-tagged output&lt;/a&gt; covers a model that does.&lt;/p&gt;

&lt;p&gt;The full measurements and the step-by-step setup are in &lt;a href="https://inferencekey.com/research/underdog-bark-speech-to-text-test" rel="noopener noreferrer"&gt;the Underdog Bark 3.0 Pro speech-to-text test on inferencekey.com&lt;/a&gt;. Thanks to Conway Research, the Underdog team, Moondream, NVIDIA, the Qwen team and the hf-audio team for the models and the data, and to the AI Search video where we spotted Bark.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>python</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Serving Claude Code from DeepInfer on 4x B200: the cache, the tail and the memory ceiling</title>
      <dc:creator>InferenceKey</dc:creator>
      <pubDate>Thu, 05 Nov 2026 06:44:51 +0000</pubDate>
      <link>https://dev.to/inferencekey/serving-claude-code-from-deepinfer-on-4x-b200-the-cache-the-tail-and-the-memory-ceiling-22o3</link>
      <guid>https://dev.to/inferencekey/serving-claude-code-from-deepinfer-on-4x-b200-the-cache-the-tail-and-the-memory-ceiling-22o3</guid>
      <description>&lt;p&gt;If you serve a coding agent from your own GPUs, the model and the engine are now the easy part. DeepSeek trained DeepSeek V4.1 Flash, wrote DeepInfer and released it under Apache 2.0. Anthropic's Claude Code needs only a few environment variables to talk to another endpoint. What decides whether the machine pays off sits between them: a cache, a queue and a memory budget.&lt;/p&gt;

&lt;p&gt;We ran this on one server with four NVIDIA B200 cards in the inferencekey cloud. We gave 100 synthetic agents about 450,000 tokens (the word pieces a model reads) of conversation each. The median wait for the first word of an answer was 2.55 seconds, with zero failed requests. Numbers, method and limits are in the &lt;a href="https://inferencekey.com/research/deepinfer-4xb200-one-machine-claude-code" rel="noopener noreferrer"&gt;full DeepInfer on 4x B200 research post&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cache is the product, not an optimisation
&lt;/h2&gt;

&lt;p&gt;A coding agent resends the whole conversation on every step, and the model must read all of it before replying. A prefix cache keeps the result of that reading for the part already seen, so only the new tail is read. In our runs 96% of the tokens the model read came from the cache.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/deepseek-ai/DeepInfer" rel="noopener noreferrer"&gt;DeepInfer&lt;/a&gt; leaves the cache for you to build, as its own page says. With no cache, our four cards got through about 200K tokens per second, starting each conversation from zero. The inferencekey cache lifted that to about 1.8 million on the same cards: nine times the read rate.&lt;/p&gt;

&lt;p&gt;It has two tiers: about 57 million tokens in GPU memory (roughly 120 of our conversations) and 617 million in host RAM for quiet sessions.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fosa509bqyoxtojdm7nj7.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fosa509bqyoxtojdm7nj7.webp" alt="Diagram: Claude Code sends requests via ANTHROPIC_BASE_URL to the inferencekey API and cache (57 M tokens in GPU memory, 617 M in RAM), then to DeepInfer running DeepSeek V4.1 Flash on 4x NVIDIA B200" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug that a load test would have hidden
&lt;/h2&gt;

&lt;p&gt;Our first adapter, running the engine as four ranks (one process per card), served zero cached tokens. It told the cache to drop each entry when the final token came out, yet the engine only closes a sequence on the step after that. Every request was read in full, silently.&lt;/p&gt;

&lt;p&gt;A small quality gate caught it first. Hide one fact in a long document, ask for it with an empty cache, then ask again with the document cached, and compare. Temperature 0 makes the model pick its most likely word every time, so the two answers should match. The fixed backend (0.3.2) served repeats 95 to 99% from the cache, and five documents of up to 200,000 tokens returned the same correct fact both ways. Run a check like this first and log the cache hit rate per request. A silent miss looks like a slow GPU.&lt;/p&gt;

&lt;h2&gt;
  
  
  Watch the tail, not the median
&lt;/h2&gt;

&lt;p&gt;Time to first token (TTFT) is how long an agent waits before the answer starts streaming. The 100-agent run had a 2.55 s median TTFT and still failed our bar: 95% of turns starting within 15 s. That 95th percentile, p95, sat at 37 seconds. Warm turns, whose conversation was already cached, queued behind seven cold ones: brand-new 450K conversations read from zero.&lt;/p&gt;

&lt;p&gt;Two untested fixes: let cached turns jump the queue, or limit how many fresh sessions start together. Chart p95 TTFT beside the median, split into warm and cold turns.&lt;/p&gt;

&lt;h2&gt;
  
  
  GPU memory is a hard ceiling at long context
&lt;/h2&gt;

&lt;p&gt;At 500K tokens per conversation, we had 200 agents ask at once. Their conversations overflowed GPU memory and the median TTFT went past two minutes. No request failed, yet nobody could have worked like that. Host RAM only rescues idle sessions. With agents this busy, the cards hold about 120 such conversations at once.&lt;/p&gt;

&lt;p&gt;Size the GPU tier for sessions active at once, not for the ones that exist. The &lt;a href="https://inferencekey.com/research/deepinfer-4xb200-one-machine-claude-code" rel="noopener noreferrer"&gt;per-step results for 128K, 500K and 1M conversations&lt;/a&gt; show what limits each size: compute for new prompts at 128K, GPU memory at 500K.&lt;/p&gt;

&lt;h2&gt;
  
  
  Point Claude Code at it, every model role included
&lt;/h2&gt;

&lt;p&gt;Claude Code asks for a different model name per role, so map every role to the one model you serve:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;ANTHROPIC_BASE_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;https://api.inferencekey.com/endpoint/&amp;lt;project&amp;gt;/&amp;lt;workload&amp;gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;ANTHROPIC_MODEL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;deepseek-v4.1-flash
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;ANTHROPIC_DEFAULT_HAIKU_MODEL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;deepseek-v4.1-flash
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The full block adds the key and the Sonnet and Opus roles, and the &lt;a href="https://docs.inferencekey.com/guides/deepinfer-4xb200/" rel="noopener noreferrer"&gt;DeepInfer on 4x B200 guide&lt;/a&gt; walks through it. One trap: different values in &lt;code&gt;Authorization: Bearer&lt;/code&gt; and &lt;code&gt;x-api-key&lt;/code&gt; return a 401. Pointed at this endpoint, Claude Code 2.1.310 implemented a function against a 15-test file in 39 seconds, with 14 tool calls and 15 of 15 passing. A smoke test, not a score.&lt;/p&gt;

&lt;h2&gt;
  
  
  Size by call rate, not by seats
&lt;/h2&gt;

&lt;p&gt;This part is modelled. A simulator calibrated on 7,879 real turns, load-tested with NVIDIA's aiperf, puts the machine at about 250 agents if 95% of answers must start within 15 s, and about 400 at 30 s. Either way it handles near 450 model calls a minute; more agents only wait longer.&lt;/p&gt;

&lt;p&gt;At two agents per power user, 400 agents is about 200 subscriptions. Counted by the peak call rate of our reference user, a very heavy user who also runs autonomous agents, it is about 80. The machine costs 27,900 USD a month, and it only beats Claude Max 20x above about 140 subscriptions.&lt;/p&gt;

&lt;p&gt;Before you buy, pull each person's peak calls per second from your team's Claude Code logs.&lt;/p&gt;

&lt;h2&gt;
  
  
  On quality, try it on your own code
&lt;/h2&gt;

&lt;p&gt;LiveBench's agentic coding board of 10 October puts DeepSeek V4.1 Flash at 77.3, top of 69 models, with Claude Opus 5.5 at 71.7. Artificial Analysis, a broader index, reverses the order: Opus 5.5 at 58, Flash at 39. We ran neither. Our needle check proves the cache keeps answers intact, not that the model codes well on your repository.&lt;/p&gt;

&lt;p&gt;Per-step tables, the cold stress test and the cost derivation are in the &lt;a href="https://inferencekey.com/research/deepinfer-4xb200-one-machine-claude-code" rel="noopener noreferrer"&gt;research post on one 4x B200 machine taking the Claude Code load of about 200 subscriptions&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>gpu</category>
      <category>devops</category>
    </item>
  </channel>
</rss>
