Do not handle `NotImplementedType` as `Any` anymore. #20114

tyralla · 2025-10-24T12:04:31Z

Fixes #18914
Completes #363

Handling NotImplementedType as Any decreases type safety and interferes with using it in type annotations (as discussed in python/typing#1548 and elsewhere). There was already an attempt to change this in typeshed (python/typeshed#13488).

This PR modifies Mypy's internal representation of NotImplementedType as if typeshed would not imply it inherits from Any. Additionally, it stops treating NotImplementedType as Any itself during return type analysis.

Most of the Mypy primer results suggest this change is directly helpful. However, some type annotations would need to be extended with NotImplementedType, which is only available since Python 3.10, and Mypy currently still supports Python 3.9.

A (hopefully complete enough) summary of the Mypy primer results:

Usages of NotImplemented to indicate that a method must be overridden (instead of relying on abc):

    def _enter(self) -> bool:
        return NotImplemented  # error: Incompatible return value type (got "_NotImplementedType", expected "bool")  [return-value]

Incompletely annotated helper functions:

    def __compare(self, other: object, func: Callable[[Any, Any], bool]) -> bool:
        if isinstance(other, type(self)):
            return func(set(self), set(other))
        return NotImplemented  # error

    def __lt__(self: TypeDevices, other: TypeDevices) -> bool:
        return self.__compare(other, operator.lt)

Usages of NotImplemented as a placeholder for features not available so far:

OPTIONS: dict[str, bool] = {
    ...,
    "option666": NotImplemented,  # error
}
for option in OPTIONS:
    if option is not NotImplemented:  # error
        ...

Usage of NotImplemented to mark class attributes that must be overridden:

class ReportBase:
    COLUMN_NAMES: list[str] = NotImplemented  # error

Accessing the (unmodified) results of the corresponding dunder methods:

def test_le_not_implemented() -> None:
    url = URL("http://example1.com/")

    assert url.__le__(123) is NotImplemented  # error

In my opinion, the last case is the only problematic one if the relevant dunder method's return type is defined in another library. I hope such cases are relatively rare, but maybe typeshed should be updated in sync??? (And then, if necessary, other type checkers??????) cc @davidhalter

for more information, see https://pre-commit.ci

…ny' into feature/notimplementedtype_not_any # Conflicts: # mypy/checker.py

sterliakov

The change looks reasonable to me, but I'm a bit concerned about overriding typeshed definition in code, essentially pretending that there's no Any inheritance

sterliakov · 2025-10-24T23:55:07Z

test-data/unit/check-overloading.test

 reveal_type(headers)  # N: Revealed type is "Union[__main__.Headers, typing.Iterable[tuple[builtins.bytes, builtins.bytes]]]"
 [builtins fixtures/isinstancelist.pyi]
+
+[case testReturnNotImplementedInBinaryMagicMethods]


Why did you move this to check-overloading?

These tests were originally in the "Returning Any" section of check-warnings.test, but that section no longer fits. Since returning NotImplemented is usually applied in the context of operator overloading, I moved the tests to check-overloading.test. Do you know a place that fits better?

I'd probably put them in check-classes.test, but that one is already too big, so I'm fine with your decision - just a bit weird to have a set of tests without a single @overload in an overloads test file:)

Yes, it is. But we may add many explicit overloads to them later...

sterliakov · 2025-10-24T23:58:18Z

mypy/mro.py

-    # The property of falling back to Any is inherited.
-    info.fallback_to_any = any(baseinfo.fallback_to_any for baseinfo in info.mro)
+    # The property of falling back to Any is (usually) inherited.
+    if info.fullname == "builtins._NotImplementedType":


This is a nice duct tape application, but maybe we should consider changing that in typeshed (or our own typeshed patch) directly?

Yes, my favourite would be to update typeshed. However, this was closed in typeshed/13488 due to something with __eq__ and the required adjustments for type checkers. I will ask around.

I have not modified Mypy's own typeshed patch so far. I will try it...

...it worked!

tyralla · 2025-10-25T08:21:41Z

Oh, with the current state of this PR:

class A:
    def __add__(self, x: A)-> int | NotImplementedType: ...
reveal_type(A() + A())  # int | NotImplementedType

Here, it is clear what to do (remove NotImplementedType from the returned union), and it is hopefully not complicated.

But what about the following (very special) case?

class A:
    def __add__(self, x: A) -> NotImplementedType: ...
reveal_type(A() + A())  # NotImplementedType

Ideally, it would mean A does not support the + operator itself, but is happy if another type steps in. But I think this is not where we are yet. So I tend to return Any in this specific case (which is consistent with the old behaviour).

A5rocks · 2025-10-25T13:44:13Z

Replace NotImplementedType with Never?

…inary methods ("+", "and", etc.).

for more information, see https://pre-commit.ci

tyralla · 2025-10-25T15:59:27Z

I added erase_notimplemented and applied it in TypeChecker.check_return_stmt? (similar to before) and in ExpressionChecker.visit_op_expr, ExpressionChecker.visit_comparison_expr, and ExpressionChecker.check_op. There are other places where applying it could be useful in special cases, but I tried to use it sparingly. The test case testNotImplementedReturnedFromBinaryMagicMethod` covers all relevant cases that came to my mind.

tyralla · 2025-10-25T16:19:35Z

Replace NotImplementedType with Never?

Wouldn't this cause more problems?

class A:
    def __add__(self, x: A) -> NotImplementedType: ...
reveal_type(A() + A())  # Never
a = A() + A()  # error: Need type annotation for "a"

(I hope this is only a temporary state and we soon have clear guidelines on how to use NotImplementedType correctly as a type hint - if at all).

sterliakov · 2025-10-25T16:26:18Z

mypy/typeshed/stdlib/builtins.pyi


 @final
 @type_check_only
-class _NotImplementedType(Any):


...wait, but you still need a patch, right? (misc/typeshed_patches directory seems to be where they live) Our typeshed updates routine (misc/sync-typeshed.py) is "clone the HEAD of current typeshed, then apply patches in order", so your change will be lost during the next sync

I had a short look at it, and you are likely right. I have to admit that I am absolutely unfamiliar with the workflow and not very interested in spending time learning it. Would you like to contribute this change here (or in a separate PR - I do not know...)? Otherwise, I would simply go back to the original solution.

I'm not deeply familiar with that either, but... Just git format-patch -1 -o misc/typeshed_patches/ 9b584b mypy/typeshed/ should do the trick (rename and edit the Subject line of the new file if you wish, commit&push) - I can open a PR with that file in your fork, but might be easier to just generate it on your end? (I didn't run the command, but the patches all look like format-patch output, and I'm moderately certain that I remember its arguments correctly)

Cool, thanks a lot for your help; it seems to have worked exactly as you suggested!

sterliakov

LG!

sterliakov · 2025-10-25T20:43:10Z

mypy/checker.py

        return typ

+    @staticmethod
+    def is_notimplemented(t: ProperType) -> bool:


Nit: are staticmethods as fast as bare functions with mypyc? (I really don't know the answer here, just wondering)

I have no idea. But yes, maybe a relevant point, as it could be called quite often for many code bases.

I had a look. Both @staticmethod and @classmethod are very rarely used in performance-critical code. (The only static method in semanal, get_deprecated, was introduced by me...) So performance might, in fact, be an issue. Or it's just a question of style. Whatever, I turned both is_notimplemented and erase_notimplemented into normal functions and moved them to typeops.

Cool! I still don't know whether it makes any difference, but using free functions avoids this question altogether:)

…unctions

github-actions · 2025-10-26T07:43:56Z

Diff from mypy_primer, showing the effect of this PR on open source code:

spack (https://github.com/spack/spack)
+ lib/spack/spack/llnl/util/lock.py:779: error: Incompatible return value type (got "_NotImplementedType", expected "bool")  [return-value]
+ lib/spack/spack/llnl/util/lock.py:782: error: Incompatible return value type (got "_NotImplementedType", expected "bool")  [return-value]

hydpy (https://github.com/hydpy-dev/hydpy)
+ hydpy/core/devicetools.py:1119: error: Incompatible return value type (got "_NotImplementedType", expected "bool")  [return-value]

spark (https://github.com/apache/spark)
+ python/pyspark/pandas/numpy_compat.py:202: error: Incompatible return value type (got "_NotImplementedType", expected "IndexOpsMixin")  [return-value]
+ python/pyspark/pandas/numpy_compat.py:229: error: Incompatible return value type (got "_NotImplementedType", expected "IndexOpsMixin")  [return-value]

Tanjun (https://github.com/FasterSpeeding/Tanjun)
+ tanjun/annotations.py:2324: error: Dict entry 0 has incompatible type "type[Attachment]": "_NotImplementedType"; expected "Any": "tuple[Callable[..., Any], ...]"  [dict-item]
+ tanjun/annotations.py:2326: error: Dict entry 2 has incompatible type "type[PartialChannel]": "_NotImplementedType"; expected "Any": "tuple[Callable[..., Any], ...]"  [dict-item]
+ tanjun/annotations.py:2327: error: Dict entry 3 has incompatible type "type[InteractionChannel]": "_NotImplementedType"; expected "Any": "tuple[Callable[..., Any], ...]"  [dict-item]
+ tanjun/annotations.py:2331: error: Dict entry 7 has incompatible type "type[InteractionMember]": "_NotImplementedType"; expected "Any": "tuple[Callable[..., Any], ...]"  [dict-item]
+ tanjun/annotations.py:2451: error: Non-overlapping identity check (left operand type: "tuple[Callable[..., Any], ...]", right operand type: "_NotImplementedType")  [comparison-overlap]

sympy (https://github.com/sympy/sympy)
+ sympy/core/relational.py:864: error: Incompatible return value type (got "_NotImplementedType", expected "Self | BooleanTrue | BooleanFalse")  [return-value]
+ sympy/polys/domains/gaussiandomains.py:163: error: Incompatible return value type (got "_NotImplementedType", expected "tuple[Self, Self]")  [return-value]
+ sympy/polys/rings.py:1113: error: Incompatible return value type (got "_NotImplementedType", expected "tuple[PolyElement[Er], PolyElement[Er]]")  [return-value]
+ sympy/matrices/matrixbase.py:3118: error: Incompatible return value type (got "_NotImplementedType", expected "MatrixBase | Expr")  [return-value]
+ sympy/matrices/matrixbase.py:3121: error: Incompatible return value type (got "_NotImplementedType", expected "MatrixBase | Expr")  [return-value]
+ sympy/matrices/matrixbase.py:3301: error: Incompatible return value type (got "_NotImplementedType", expected "MatrixBase")  [return-value]
+ sympy/matrices/matrixbase.py:3304: error: Incompatible return value type (got "_NotImplementedType", expected "MatrixBase")  [return-value]
+ sympy/algebras/quaternion.py:972: error: Incompatible return value type (got "_NotImplementedType", expected "Quaternion")  [return-value]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/arrays/masked.py:855: error: Incompatible return value type (got "_NotImplementedType", expected "BooleanArray")  [return-value]

discord.py (https://github.com/Rapptz/discord.py)
+ discord/ui/view.py:300: error: Incompatible return value type (got "_NotImplementedType", expected "list[dict[str, Any]]")  [return-value]

ibis (https://github.com/ibis-project/ibis)
+ ibis/expr/types/core.py:979: error: Incompatible return value type (got "_NotImplementedType", expected "Value")  [return-value]
+ ibis/expr/types/generic.py:1224: error: Incompatible return value type (got "_NotImplementedType", expected "BooleanValue")  [return-value]
+ ibis/expr/types/numeric.py:1624: error: Incompatible return value type (got "_NotImplementedType", expected "IntegerValue")  [return-value]

scipy (https://github.com/scipy/scipy)
+ scipy/optimize/_trustregion_constr/report.py:4: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "list[str]")  [assignment]
+ scipy/optimize/_trustregion_constr/report.py:5: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "list[int]")  [assignment]
+ scipy/optimize/_trustregion_constr/report.py:6: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "list[str]")  [assignment]
+ scipy/integrate/_ivp/rk.py:76: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "ndarray[tuple[Any, ...], dtype[Any]]")  [assignment]
+ scipy/integrate/_ivp/rk.py:77: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "ndarray[tuple[Any, ...], dtype[Any]]")  [assignment]
+ scipy/integrate/_ivp/rk.py:78: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "ndarray[tuple[Any, ...], dtype[Any]]")  [assignment]
+ scipy/integrate/_ivp/rk.py:79: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "ndarray[tuple[Any, ...], dtype[Any]]")  [assignment]
+ scipy/integrate/_ivp/rk.py:80: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "ndarray[tuple[Any, ...], dtype[Any]]")  [assignment]
+ scipy/integrate/_ivp/rk.py:81: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "int")  [assignment]
+ scipy/integrate/_ivp/rk.py:82: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "int")  [assignment]
+ scipy/integrate/_ivp/rk.py:83: error: Incompatible types in assignment (expression has type "_NotImplementedType", variable has type "int")  [assignment]

yarl (https://github.com/aio-libs/yarl)
+ tests/test_url_cmp_and_hash.py:42:12: error: Non-overlapping identity check (left operand type: "bool", right operand type: "_NotImplementedType")  [comparison-overlap]
+ tests/test_url_cmp_and_hash.py:42:12: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-comparison-overlap for more info
+ tests/test_url_cmp_and_hash.py:55:12: error: Non-overlapping identity check (left operand type: "bool", right operand type: "_NotImplementedType")  [comparison-overlap]
+ tests/test_url_cmp_and_hash.py:75:12: error: Non-overlapping identity check (left operand type: "bool", right operand type: "_NotImplementedType")  [comparison-overlap]
+ tests/test_url_cmp_and_hash.py:88:12: error: Non-overlapping identity check (left operand type: "bool", right operand type: "_NotImplementedType")  [comparison-overlap]

xarray (https://github.com/pydata/xarray)
+ xarray/core/dataset.py: note: In member "_binary_op" of class "Dataset":
+ xarray/core/dataset.py:7686: error: Incompatible return value type (got "_NotImplementedType", expected "Dataset")  [return-value]
+ xarray/core/dataarray.py: note: In member "_binary_op" of class "DataArray":
+ xarray/core/dataarray.py:4890: error: Incompatible return value type (got "_NotImplementedType", expected "Self")  [return-value]
+ xarray/core/datatree.py: note: In member "_binary_op" of class "DataTree":
+ xarray/core/datatree.py:1913: error: Incompatible return value type (got "_NotImplementedType", expected "DataTree")  [return-value]

tyralla and others added 2 commits October 24, 2025 14:01

Do not handle NotImplementedType as Any anymore.

4143846

[pre-commit.ci] auto fixes from pre-commit.com hooks

39d56bf

for more information, see https://pre-commit.ci