wiki:Notes/PythonMultiprocessing

Python Multiprocessing And AsyncIO

Catch-all for synchronous/asynchronous process frameworks and in-process concurrent programming - crunchers.

Python Multiprocessing Module

No just about breaking the Python GIL lock ... it's an entire process framework with high level abstractions.

And no external dependencies !

https://docs.python.org/2/library/multiprocessing.html

http://pymotw.com/2/multiprocessing/

http://www.laurentluce.com/posts/python-threads-synchronization-locks-rlocks-semaphores-conditions-events-and-queues/

http://www.bogotobogo.com/python/Multithread/python_multithreading_Synchronization_Condition_Objects_Producer_Consumer.php

Async IO

http://asyncio.org/

Python Async IO Resources - Curated content from around the net to get started with asyncio

https://bitbucket.org/mrdon/asyncio-mongo

An asynchronous Python driver for the Mongo database, based on Python's asyncio. This project is based on TxMongo? ... still in the very early alpha stage and shouldn't be used for production.

Python 3 Async IO

A reason to move up to Python 3.

https://docs.python.org/3/library/asyncio.html

New in version 3.4 ...

The asyncio package has been included in the standard library on a provisional basis. Backwards incompatible changes (up to and including removal of the module) may occur if deemed necessary by the core developers ...

This module provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives ...

Pluggable event loop with various system-specific implementations;
Transport and protocol abstractions (similar to those in Twisted);
Concrete support for TCP, UDP, SSL, subprocess pipes, delayed calls, and others (some may be system-dependent);
Future class that mimics the one in the concurrent.futures module, but adapted for use with the event loop;
Coroutines and tasks based on yield from (PEP 380), to help write concurrent code in a sequential fashion;
Cancellation support for Futures and coroutines;
Synchronization primitives for use between coroutines in a single thread, mimicking those in the threading module;
Interface for passing work off to a threadpool, for times when you absolutely, positively have to use a library that makes blocking I/O calls.

See: PythonTwisted, WebSocket

Search wiki for 'async'

Greenlets

https://pypi.python.org/pypi/greenlet

The greenlet package is a spin-off of Stackless, a version of CPython that supports micro-threads called “tasklets”. Tasklets run pseudo-concurrently (typically in a single or a few OS-level threads) and are synchronized with data exchanges on “channels”.

Problematic on AppleOSX, unfortunately ... see AppleOSX#lnstallingGreenlet

Gevent

http://learn-gevent-socketio.readthedocs.org/en/latest/index.html

Gevent is the use of simple, sequential programming in python to achieve scalability provided by asynchronous IO and lightweight multi-threading (as opposed to the callback-style of programming using Twisted’s Deferred).

It is built on top of libevent/libev (for asynchronous I/O) and greenlets (lightweight cooperative multi-threading).

http://learn-gevent-socketio.readthedocs.org/en/latest/greenlets.html

What are greenlets?

Greenlets are lightweight thread-like structures that are scheduled and managed inside the process. They are references to the part of the stack that is used by the thread. Compared to POSIX threads (pthreads), there is no stack allocated up front and there is only as much stack as is actually used by the greenlet

In python, we implement greenlets via the gevent package and we implement pthreads via python’s built-in threading module.

Both green threads (greenlets) and POSIX threads (pthreads) are mechanisms to support multithreaded execution of programs ...

http://learn-gevent-socketio.readthedocs.org/en/latest/gevent.html#gevent-with-other-python-extensions

gevent’s monkey patch

A monkey patch is a way to extend or modify the run-time code of dynamic languages without altering the original source code. Monkey patching as a programming technique is very powerful but can result in hard-to-debug code in the wrong hands.

Hey, I resent that insinuation !

... so in other words, monkey patching can be a problem, as the name implies.

Twisted

See PythonTwisted

http://pymotw.com/2/multiprocessing/basics.html

Last modified 16 months ago Last modified on 01/17/2016 07:36:12 PM