Repository navigation
fix(e6): flatten FROM_JSON schema whitespace so the type descriptor stays backslash-free - #320
Merged
Merged
Conversation
…tays backslash-free
FROM_JSON's 2nd arg is a type descriptor that the e6 engine re-parses via a JsonSchema()
sub-parser over the raw string body. That lexer skips whitespace but has no rule for a
backslash, so a view whose schema is written multi-line -- the real newlines are emitted
as "\n" (backslash-n) -- fails at parse time with "Lexical error ... Encountered \ (92)"
at the column right after "array<struct<".
Whitespace is insignificant in a type descriptor, so collapse it (" ".join(val.split()))
when rendering a FROM_JSON string arg, dropping the newlines that would otherwise be
backslash-escaped. Already single-line schemas are unchanged (idempotent).
Adds test_from_json_schema_newlines.
suyashkhare1403
approved these changes
Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Freshworks Power BI queries that
SELECT … FROM <view>where the view usesEXPLODE(FROM_JSON(col, 'array<struct<…>>'))fail at parse time with:(Confirmed in the planner logs — the query is transpiled
databricks → E6, thentryParseWithE6dies on the backslash.)Root cause
FROM_JSON's 2nd argument is a type descriptor, not a normal string value. The engine parses it at SQL-parse time viaDataTypeParser.parseJsonSchemaString(), which spins up aJsonSchema()sub-parser over the raw string body (so it resets to line 1). That lexer skips whitespace but has no rule for a backslash.The views write the schema multi-line for readability, so the source string literal contains real newlines. e6 escapes those newlines as
\n(backslash-n) on output, and the sub-parser dies on the\— at column 14, which is exactly the char afterarray<struct<(13 chars).Fix
Whitespace is insignificant inside a type descriptor, so when rendering a
FROM_JSONstring arg (anonymous_sql), collapse it with" ".join(val.split()). That drops the newlines that would otherwise be backslash-escaped, soarray<struct<\n id:string…becomesarray<struct< id:string…— backslash-free, and the sub-parser skips the spaces and derives the identical type. Already single-line schemas are unchanged (idempotent), so nothing else in the workload shifts.Tests
Adds
test_from_json_schema_newlines: a multi-line schema transpiles to a clean single-line, backslash-free descriptor; a single-line schema is left untouched. The existingtest_json_extract(value:→`value`rewrite) still passes.make checkis green — style (ruff, ruff-format, mypy),test(1010) andtest-rs(1010).