You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PDF text increases prompt usage but not count_tokens on gemini-4-hydrogen #3379
Adding embedded text to a 10-page PDF substantially increases gemini-4-hydrogen's reported input usage, but its count_tokens estimate is unchanged. The comparison below uses a blank PDF with the same page count and dimensions as a control.
Observed through google-genai; I have not isolated whether the cause is in the SDK or backend, or tested raw REST. Please redirect me if another tracker is more appropriate.
Environment details
Python 3.12.13
macOS 26.5.2 (arm64)
google-genai==3.3.0, Gemini Developer API
thinking_level=LOW; no system instruction, tools, or response schema
The script generates both synthetic PDFs and calls count_tokens and generate_content with identical contents for each model. No real documents or sensitive data are included.
Actual result
One run on 2026-11-16:
PDF
Model
count_tokens
prompt_token_count
TEXT
text.pdf
gemini-3.8-flash
5,609
5,329
9
text.pdf
gemini-4-hydrogen
5,638
14,642
9,322
blank.pdf
gemini-3.8-flash
5,609
5,329
9
blank.pdf
gemini-4-hydrogen
5,638
5,704
384
All four requests report 5,320 IMAGE tokens. On Hydrogen, adding PDF text increases reported usage by 8,938 TEXT tokens without changing the estimate.
Expected behavior
count_tokens should account for PDF text included in prompt usage, or document the difference and provide a supported way to estimate input usage before generation.
Hi @equialgo,
Thank you for the clear report and reproduction steps! We have confirmed that this discrepancy occurs on the backend (where generate_content includes extracted PDF TEXT tokens in prompt_token_count while count_tokens only accounts for page IMAGE tokens) rather than in the google-genai Python SDK, we will file a bug internally.
Activity
Venkaiahbabuneelam commentedon Nov 17, 2026
Hi @equialgo,
Thank you for the clear report and reproduction steps! We have confirmed that this discrepancy occurs on the backend (where
generate_contentincludes extracted PDFTEXTtokens inprompt_token_countwhilecount_tokensonly accounts for pageIMAGEtokens) rather than in thegoogle-genaiPython SDK, we will file a bug internally.Thanks