Invalid base class on Protocol (typing_extensions)

[autumnjolitz]

The following minimal program validates with pytype and mypy

from typing import List

def foo(fh) -> List[bytes]:
    items = []
    for line in iter(fh.readline, b''):
        items.append(line)
    return items


if __name__ == '__main__':
    import io
    test_buffer = io.BytesIO(b'foo\nbar')
    assert foo(test_buffer) == [b'foo\n', b'bar']

The following test program validates with MyPy but faults on pytype with the following messages:

Computing dependencies
Analyzing 1 sources with 0 local dependencies
ninja: Entering directory `/Users/ben/software/horcrux/.pytype'
[1/1] check sample
FAILED: /Users/ben/software/horcrux/.pytype/pyi/sample.pyi
pytype-single --imports_info /Users/ben/software/horcrux/.pytype/imports/sample.imports --module-name sample -V 3.7 -o /Users/ben/software/horcrux/.pytype/pyi/sample.pyi --analyze-annotated --nofail --quick /Users/ben/software/horcrux/sample.py
File "/Users/ben/software/horcrux/sample.py", line 5, in <module>: Invalid base class: <instance of typing_extensions._SpecialForm> [base-class-error]
File "/Users/ben/software/horcrux/sample.py", line 7, in readline: bad option in return type [bad-return-type]
  Expected: bytes
  Actually returned: None
File "/Users/ben/software/horcrux/sample.py", line 20, in <module>: Function foo was called with the wrong arguments [wrong-arg-types]
  Expected: (fh: IReadlineable)
  Actually passed: (fh: io.BytesIO)

For more details, see https://google.github.io/pytype/errors.html.
ninja: build stopped: subcommand failed.

from typing import List
from typing_extensions import Protocol


class IReadlineable(Protocol):
    def readline(self) -> bytes:
        ...


def foo(fh: IReadlineable) -> List[bytes]:
    items = []
    for line in iter(fh.readline, b''):
        items.append(line)
    return items


if __name__ == '__main__':
    import io
    test_buffer = io.BytesIO(b'foo\nbar')
    assert foo(test_buffer) == [b'foo\n', b'bar']

Expected result:

pytype correctly validates Protocol-derived structural subtyping.

[rchen152]

Thanks for the report! I think there are two things going on here:

• pytype defines Protocol in typing rather than typing_extensions. Fixing this should be easy enough.
• We don't yet support defining and using a protocol in the same module. I'll take a look and figure out what we'd need to do to support this.

[autumnjolitz]

I isolated my Protocol subclasses out to a project specific typing.py file. While calling python -m pytype.tools.analyze_project.main on files that from .typing import ... worked, calling it directly on this typing.py file would throw errors (zero uses of the defined classes).

This is quite surprising, because it almost works (project level analysis faults, but calling pytype on modules that use the Protocol definitions will work as long as you keep pytype away from the actual files that define the Protocols)

Example:

(python) Fateweaver:~/software$ mkdir module
(python) Fateweaver:~/software$ echo '' > module/__init__.py
(python) Fateweaver:~/software$ cat > module/typing.py
import typing
if typing.TYPE_CHECKING:
    from typing import Protocol
else:
    try:
        from typing import Protocol
    except ImportError:
        from typing_extensions import Protocol


class IReadlineable(Protocol):
    def readline(self) -> bytes:
        ...

(python) Fateweaver:~/software$ cat > module/test.py
from typing import List
from .typing import IReadlineable


def foo(fh: IReadlineable) -> List[bytes]:
    items = []
    for line in iter(fh.readline, b''):
        items.append(line)
    return items


if __name__ == '__main__':
    import io
    test_buffer = io.BytesIO(b'foo\nbar')
    assert foo(test_buffer) == [b'foo\n', b'bar']

(python) Fateweaver:~/software$ python -m module.test
(python) Fateweaver:~/software$ mypy module
(python) Fateweaver:~/software$ python -m pytype.tools.analyze_project.main module/test.py
Computing dependencies
Analyzing 1 sources with 1 local dependencies
ninja: Entering directory `/Users/ben/software/.pytype'
[2/2] check module.test
Success: no errors found
(python) Fateweaver:~/software$ python -m pytype.tools.analyze_project.main module
Computing dependencies
Analyzing 3 sources with 0 local dependencies
ninja: Entering directory `/Users/ben/software/.pytype'
[2/3] check module.typing
FAILED: /Users/ben/software/.pytype/pyi/module/typing.pyi
pytype-single --imports_info /Users/ben/software/.pytype/imports/module.typing.imports --module-name module.typing -V 3.7 -o /Users/ben/software/.pytype/pyi/module/typing.pyi --analyze-annotated --nofail --quick /Users/ben/software/module/typing.py
File "/Users/ben/software/module/typing.py", line 13, in readline: bad option in return type [bad-return-type]
  Expected: bytes
  Actually returned: None

For more details, see https://google.github.io/pytype/errors.html#bad-return-type.
ninja: build stopped: subcommand failed.
(python) Fateweaver:~/software$

[rchen152]

The bad-return-type error is because pytype is trying to check the empty method body against the declared return type :(

I've mailed out fixes for all of these issues; they'll be exported to GitHub sometime today and should be in this Friday's release assuming nothing breaks.