Details on the System’s Design

By stripping the hood of ChimeraPy and see its interworkings, it comes down to a combination of AIOHTTP, ZeroMQ, and parallel programming. The communication within ChimeraPy can be broken down into two categories: HTTP/WebSocket and PUB/SUB. The Manager and Worker send messages via HTTP and WebSockets, including the Server and Client. These messages are not the raw data streams, instead they are operational instructions and information that orchestrates the clusters execution. The data streams use a different communication protocol and channels. The intre-communication between Node<chimerapy.Node uses ZeroMQ’s PUB/SUB pattern, optimized for speed, robustness, and latency. In specific, the node-communication uses the Publisher and Subscriber implementations.

Multiprocessing is at the heart of ChimeraPy, as the base class Node is a subclass of Python’s build-in multiprocessing’s Process. Each Node executes its prep, step, and teardown within its own process, to reduce CPU bound limitations.

In the other side of the parallel programming spectrum, multithreading and AsyncIO are used for relieve the IO bound. More specifically, multithreading is used in active working while waiting, such as writing to video to memory, while AsyncIO is used for networking.

More details in how each component works can be found in the Developer’s Documentation.