ArrowNotImplementedError: Function 'equal' has no kernel matching input types (string_view, string)

[danking]

What happens?

I can't seem to compare a string_view Arrow column (via the pyarrow Dataset API) to a literal. Is there anyway to force the literal to be a string_view instead of a string?

I originally reported this in duckdb/duckdb#19861 and was instructed to refile here.

To Reproduce

Install the latest versions of: duckdb, vortex-data, and typing-extensions (due to a missing dependency in vortex).

uv add duckdb vortex-data typing-extensions

Python code:

import vortex as vx
import duckdb
vx.io.write(vx.array([{"column": "a string"}]), 'foo.vortex')
x = vx.open('foo.vortex').to_dataset()
duckdb.execute("select * from x where \"column\" == 'hello world'").to_arrow_table()

Python output:

Traceback (most recent call last):
  File "<python-input-0>", line 5, in <module>
    duckdb.execute("select * from x where \"column\" == 'hello world'").to_arrow_table()
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_duckdb.Error: ArrowNotImplementedError: Function 'equal' has no kernel matching input types (string_view, string)

At:
  pyarrow/error.pxi(92): pyarrow.lib.check_status

---

I can't replicate with the DuckDB CLI. Whatever path that takes must avoid a string_view?

install vortex; load vortex; select * from read_vortex('foo.vortex') where "column" == 'hello world';
┌─────────┐
│ column  │
│ varchar │
├─────────┤
│ 0 rows  │
└─────────┘

---

The full transcript of what I did:

# mkdir bar
# cd bar
# uv init
Initialized project `bar`
# uv add duckdb vortex-data typing-extensions
Using CPython 3.14.0
Creating virtual environment at: .venv
Resolved 7 packages in 58ms
Installed 6 packages in 15ms
 + duckdb==1.4.2
 + protobuf==5.29.5
 + pyarrow==22.0.0
 + substrait==0.25.0
 + typing-extensions==4.15.0
 + vortex-data==0.56.0
# uv run python
Python 3.14.0 (main, Oct 31 2025, 23:20:55) [Clang 21.1.4 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import vortex as vx
... import duckdb
... vx.io.write(vx.array([{"column": "a string"}]), 'foo.vortex')
... x = vx.open('foo.vortex').to_dataset()
... duckdb.execute("select * from x where \"column\" == 'hello world'").to_arrow_table()
... 
Traceback (most recent call last):
  File "<python-input-0>", line 5, in <module>
    duckdb.execute("select * from x where \"column\" == 'hello world'").to_arrow_table()
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_duckdb.Error: ArrowNotImplementedError: Function 'equal' has no kernel matching input types (string_view, string)

At:
  pyarrow/error.pxi(92): pyarrow.lib.check_status

>>> 

OS:

macOS

DuckDB Package Version:

1.4.2

Python Version:

1.4.2

Full Name:

Daniel King

Affiliation:

SpiralDB, Inc.

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[paultiq]

Reproduced with just pyarrow.

import pyarrow as pa
import duckdb 

mytable = pa.table({'col1': pa.array(['foo', 'bar'] * 50, type=pa.string_view())})
duckdb.sql("select * from mytable where col1='a string' ")

Looks like root cause is here in the push down, not also checking for a ArrowVariableSizeType::VIEW:

duckdb-python/src/duckdb_py/arrow/pyarrow_filter_pushdown.cpp

Lines 122 to 130 in 4ebe06f

	case LogicalTypeId::VARCHAR:
	return dataset_scalar(constant.ToString());
	case LogicalTypeId::BLOB: {
	if (type.GetTypeInfo<ArrowStringInfo>().GetSizeType() == ArrowVariableSizeType::VIEW) {
	py::handle binary_view_type = import_cache.pyarrow.binary_view();
	return dataset_scalar(scalar(py::bytes(constant.GetValueUnsafe<string>()), binary_view_type()));
	}
	return dataset_scalar(py::bytes(constant.GetValueUnsafe<string>()));
	}

* trying to query a scanner here will cause a duckdb segfault: x = vx.open('foo.vortex').to_dataset().scanner()

[danking]

Yeah, I also noticed the segfault. I’m not sure what’s wrong with the Scanner. It could be our (Vortex’s) fault.

[paultiq]

The segfault seems to be inside pyarrow, not duckdb:

(I'll file in Vortex)

0x00007fff96ab2d3c in __pyx_getprop_7pyarrow_8_dataset_7Scanner_projected_schema(_object*, void*) [clone .lto_priv.0] () from >/home/extra/git/goodenv/.venv/lib/python3.13/site-pack
ages/pyarrow/_dataset.cpython-313-x86_64-linux-gnu.so

MRE of the segfault:

import vortex as vx

vx.io.write(vx.array([{"col1": "a string"}]), 'foo.vortex')
x = vx.open('foo.vortex').to_dataset().scanner().projected_schema()

[evertlammerts]

Hmm I'll have to look deeper. Comparing a string view column with a string view constant doesn't work either. This is more or less what duckdb does internally:

from pyarrow import dataset
import pyarrow as pa

# create the filter expression
field = dataset.field("col")
const = dataset.scalar(pa.scalar("val1", pa.string_view())) # string view instead of a string
filter_expr = field == const
# create the dataset
table = pa.table({"col": pa.array(["val1", "val2"], type=pa.string_view())})
dset = dataset.dataset(table)
# creating the scanner fails
dset.scanner(columns=["col"], filter=filter_expr)

Traceback (most recent call last):
File "", line 1, in
File "pyarrow/_dataset.pyx", line 415, in pyarrow._dataset.Dataset.scanner
File "pyarrow/_dataset.pyx", line 3674, in pyarrow._dataset.Scanner.from_dataset
File "pyarrow/_dataset.pyx", line 3587, in pyarrow._dataset.Scanner._make_scan_options
File "pyarrow/_dataset.pyx", line 3517, in pyarrow._dataset._populate_builder
File "pyarrow/_compute.pyx", line 2884, in pyarrow._compute._bind
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Function 'equal' has no kernel matching input types (string_view, string_view)

[evertlammerts]

string to string comparison does work:

from pyarrow import dataset
import pyarrow as pa

# create the filter expression
field = dataset.field("col")
const = dataset.scalar("val1")
filter_expr = field.cast(pa.string()) == dataset.scalar("val1")
# create the dataset
table = pa.table({"col": pa.array(["val1", "val2"], type=pa.string_view())})
dset = dataset.dataset(table)
# this works
dset.scanner(columns=["col"], filter=filter_expr)

This can be a workaround as long as string view comparisons aren't implemented in pyarrow for datasets...

edit: looks like the scanner can't be consumed:

dset.scanner(columns=["col"], filter=filter_expr).to_table()

Traceback (most recent call last):
File "", line 1, in
File "pyarrow/_dataset.pyx", line 3939, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Function 'array_filter' has no kernel matching input types (string_view, bool)

[evertlammerts]

@danking I'm not very much at home in pyarrow, so please bear with me, and suggestions are very welcome.

Afaict, string view comparison is not implemented in pyarrow yet. It doesn't seem to work on scanners nor on tables:

table = pa.table({'col': pa.array(['field1', 'field2'], type=pa.string_view())})
field = dataset.field('col')
expr = field == dataset.scalar(pa.scalar("field1", pa.string_view()))
table.filter(expr)

Traceback (most recent call last):
File "", line 1, in
File "pyarrow/table.pxi", line 2271, in pyarrow.lib._Tabular.filter
File "/Users/evert/projects/duckdb-python/.venv/lib/python3.10/site-packages/pyarrow/acero.py", line 380, in _filter_table
result = decl.to_table(use_threads=True)
File "pyarrow/_acero.pyx", line 591, in pyarrow._acero.Declaration.to_table
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Function 'equal' has no kernel matching input types (string_view, string_view)

The only way I've found so far is to project a string view column as a string column if it's being filtered on, but that's a pretty invasive workaround and feels more like hiding the issue. At that point it's more transparent to let users project the columns they want to filter on to string themselves.

Wdyt?

[paultiq]

While I like where you're going with this (not casting) -

• You'd then pass the filter through rather than raising an exception, right?
• DuckDB already casts binary views, so there's a inconsistency here: binary views being cast but string views not?

FYI, also reproducible with Polars (using CompatLevel.newest()):

import polars as pl
import duckdb

x = pl.DataFrame({"col1": ["a", "B"]}).to_arrow(compat_level=pl.CompatLevel.newest())
duckdb.execute("select * from x where col1 = 'hello world'")

And reproducible with DuckDB alone:

import duckdb

with duckdb.connect() as conn:
    conn.execute("set arrow_output_version='1.5';set produce_arrow_string_view=True;");
    x = conn.execute("select 'a' as col1 from range(2)").fetch_arrow_table()
    duckdb.execute("select * from x where col1 = 'hello world'")

[evertlammerts]

Binary views aren't cast yet either, that code is a placeholder: filters on binary view columns aren't pushed down yet.

The problem here is that (Py)arrow does not support binary and string view filters in scanners / tables / readers yet. Since we rely on pyarrow to apply filters that can be pushed down, we can't yet push down filters on binary and string view columns.

This PR in core makes sure that string view column filters won't be pushed down, but will be applied after scanning. With this change the following will work:

import duckdb
import pyarrow as pa
import pyarrow.dataset as pd

# Create a PyArrow Table with a string_view array
table = pa.table({"col": pa.array(['abc', 'def'], type=pa.string_view())})
dset = pd.dataset(table)

# Run the query. Comparison uses a standard string literal.
con = duckdb.connect()
query_result = con.sql("SELECT * FROM dset WHERE col = 'abc'")
print(query_result)

  ┌─────────┐
  │   col   │
  │ varchar │
  ├─────────┤
  │ abc     │
  └─────────┘