gh-140995: [asyncio] Add high-level API to run functions in subinterpreters

[deadlovelll]

This PR adds a high-level asynchronous API asyncio.run_in_subinterpreter for running Python functions
in subinterpreters. It uses InterpreterPoolExecutor internally and chooses the number of subinterpreters
based on the CPU count.

Motivation:

• Enables CPU-bound tasks to run in parallel without being limited by the GIL.
• Provides a lighter alternative to ProcessPoolExecutor.
• Offers asyncio-friendly syntax.

Example usage:

import asyncio
from asyncio import run_in_subinterpreter

def cpu_task(x):
    return x * x

async def main():
    result = await run_in_subinterpreter(cpu_task, 10)
    # or
    result = await asyncio.run_in_subinterpreter(cpu_task, 10)
    print(result)

asyncio.run(main())

Tests:

• Added unit tests for return values, argument passing, exception propagation, parallel calls,
  and complex objects.

• Issue: [asyncio] Add high-level API to run functions in subinterpreters #140995

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[ZeroIntensity]

This needs to be skipped when the implementation doesn't support subinterpreters. You probably just want something like:

try:
    from asyncio import run_in_subinterpreter
except ImportError:
    raise unittest.SkipTest("subinterpreters not supported")

[deadlovelll]

Thank you, fixed

[ZeroIntensity]

You also need to handle other implementations here. If this fails with an ImportError, it shouldn't prevent the whole module from being imported.

[deadlovelll]

Thank you, fixed

[ZeroIntensity]

Don't we only want one worker?

[ZeroIntensity]

As far as I can tell, this never yields to the event loop, so it doesn't let anything else run while the subinterpreter executes (defeating the point of this API entirely).

I think you want to use something like loop.run_in_executor or asyncio.to_thread. The event loop needs to yield while executing the synchronous code in another interpreter. Here's a simple POC:

import asyncio
from concurrent import interpreters


async def run_in_subinterpreter(func):
    interp = interpreters.create()
    try:
        await asyncio.to_thread(interp.call, func)
    finally:
        interp.close()

def compute_synchronous_stuff():
    for n in range(10000):
        n ** n

async def work():
    print("doing async I/O")
    await asyncio.sleep(1)
    print("done")

async def main():
    task = asyncio.create_task(work())
    print("doing sync I/O")
    # The task can execute while this is running
    await run_in_subinterpreter(compute_synchronous_stuff)
    print("done")
    await task


asyncio.run(main())

[deadlovelll]

Thank you, it was like a blueprint for the issue i created, also changed this to non-blocking implementation

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[picnixz]

Closing as it was deemed unnecessary