6.2. boing.core — The pipeline infrastructure

The module boing.core contains all the classes that constitute the infrastructure of Boing pipelines.

The Producer and Consumer classes are build over the Observer design pattern of the module boing.core.observer: the Producer is an Observable object enabled to post products; a consumer is an Observer object that can subscribe itself to many producers. When a producer has a new product, it triggers the registered consumers; the triggered consumers will immediately or at regular time interval demand the producer the new products.

A Worker object is both a Producer and a Consumer and it is used as base class for defining processing nodes. By connecting producers, workers and consumers it is possible to create processing pipelines.

A WiseWorker is a Worker able to automatically detect whenever it should not propose its own offer or its own request; this is done in order to save computational time.

The Functor class is practical base class for inheriting custom processing Worker objects. Instead of implementing the classic Consumer._consume() method, the Functor proposes the more powerfull method Functor._process().

Multiple Producer, Consumer and Worker instances can be composed into Composite objects. There are three types of composites:

• a CompositeProducer works as it was a single producer;
• a CompositeConsumer works as it was a single consumer;
• a CompositeWorker works as it was a single worker;

See also

The pipeline architecture

6.2.1. Producers

class boing.core.Producer(offer, tags=None, store=None, retrieve=None, haspending=None, parent=None)

Producer instances are Observable objects able to post products to a set of subscribed Consumer instances. The argument offer must be an instance of Offer and it define the products this producer will supply, while tags must be a dict or None. The argument store can be a callable object to be used as a handler for storing posted products (see _store() for the handler arguments) or None, while retrieve can be a callable object to be used as a handler for retrieving stored products (see _retrieveAndDeliver() for the handler arguments) or None. parent defines the consumer’s parent.

When a producer is demanded to posts a product, for each registered consumer it tests the product with the consumer’s request and only if the match is valid it triggers the consumer.

Public signals:

demandChanged

Signal emitted when the aggregate demand changes.

offerChanged

Signal emitted when its own offer changes.

demandedOfferChanged

Signal emitted when its own demanded offer changes.

Available methods:

aggregateDemand()

Return the union of all the subscribed consumers’ requests.

demandedOffer()

Return the producer’s demanded offer.

meetsRequest()

Return whether the product’s offer meets request.

offer()

Return the producer’s offer.

postProduct(product)

Post product. In concrete terms, it triggers the registered consumers that require product, then it stores the product.

class boing.core.Offer(*args, iter=None)

An Offer defines the list of products that a producer advertises to be its deliverable objects.

Note

A producer’s offer only estimates the products that are normally produced. There is no guarantee that such products will ever be posted, neither that products that do not match the offer won’t be produced.

Offer.UNDEFINED can be used to define the producer’s offer, when the real offer cannot be defined a priori. This avoids to have empty offers, when they cannot be predeterminated.

6.2.2. Consumers

class boing.core.Consumer(request, consume=None, hz=None, parent=None)

Consumer objects are Observer objects that can be subscribed to several Producer instances for receiving their products. When a producer posts a product, it triggers the registered consumers; then the consumers will immediately or at regular time interval demand to the producer the new products.

Warning

Many consumers can be subscribed to a single producer. Each new product is actually shared within the different consumers, therefore a consumer MUST NOT modify any received product, unless it is supposed to be the only consumer.

Consumers have a Request. When a producer is demanded to posts a product, it tests the product with the consumer’s request and only if the match is valid it triggers the consumer.

request()

Return the consumer’s request.

_consume(products, producer)

Consume the products posted from producer.

class boing.core.Request

The class Request is an abstract class used by Consumer objects for specifing the set of products they are insterested to. The method test() is used to check whether a product matches the request.

Request.NONE and Request.ANY define respectively a “no product” and “any product” requests.

Request objects may also indicate the internal parts of a product to which a producer may be interested. The method items() returns the sequence of the product’s parts a producer is interested to.

The class Request implements the design pattern “Composite”: different requests can be combined into a single request by using the sum operation (e.g. comp = r1 + r2). A composite request matches the union of the products that are matched by the requests whom it is composed. Request.NONE is the identity element of the sum operation.

Request objects are immutable.

test(product)

Return whether the product matches the request.

items(product)

Return an iterator over the product‘s internal parts (i.e. (key, value) pairs) that match the request.

class boing.core.QRequest(string)

The QRequest is a Request defined by a QPath.

6.2.3. Workers

class boing.core.Worker

A Worker instance is both a Producer and a Consumer. The class Worker is by itself an abstract class. Consider using the concrete class BaseWorker instead.

class boing.core.BaseWorker(request, offer, tags=None, store=None, retrieve=None, haspending=None, consume=None, hz=None, parent=None)

A BaseWorker is the simplest concrete Worker. By default it does nothing with the products it receives, but it is able to propagate the offers of the producers it is subscribed to, and it is able to propagate the requests of the consumers that are subscribed to it.

class boing.core.NopWorker(store=None, retrieve=None, hz=None, parent=None)

NopWorker instances simply forwards all the products they receive.

class boing.core.WiseWorker(request, offer, **kwargs)

A WiseWorker instance is able to automatically detect whenever it should not propose its own offer or its own request; this is done in order to save computational time. The behaviour of WiseWorker objects works as follows:

1. Its own request is deactivated if there is no consumer insterested to its offer (i.e. no items in demandedoffer), which means: <<If nobody is interested to its products, why should it requires the products necessary to the processing?>>.
2. Its own offer is deactivated if there is no producer that meets its request, which means: <<If nobody can provide it the products it requires for processing, how can it propose its offer?>>.

A WiseWorker is active (see method isActive()) when its own request is not deactivated (i.e. case 1).

A WiseWorker can be forced to be activated or deactivated. Use the method setForcing() to impose a specific behaviour.

Use WiseWorker.TUNNELING as the worker Offer, when the worker is supposed to process and then forward a subset of the products it receives. This is necessary since sometimes it may be possible that the worker cannot have a predefined offer since it always depend from the offer of the observed producers.

isActive()

The WiseWorker is active only if its offer is requested from any subscribed consumer, unless it is forced (see forcing()).

isTunneling()

Return True if the worker is forwarding a subset of the products it receives.

forcing()

Return whether the WiseWorker is being forced. Return a value within (WiseWorker.ACTIVATED, WiseWorker.DEACTIVATED, WiseWorker.NONE).

setForcing(forcing)

Impose to the WiseWorker to be activated, deactivated or remove a previous forcing. forcing must be a value within (WiseWorker.ACTIVATED, WiseWorker.DEACTIVATED, WiseWorker.NONE).

ACTIVATED

Value used to force the WiseWorker to be activated.

DEACTIVATED

Value used to force the WiseWorker to be deactivated.

NONE

Value used to let the WiseWorker decide it it should be active.

TUNNELING

Value used to set the WiseWorker the tunneling mode.

class boing.core.Functor(args, offer, blender, process=None, **kwargs)

The Functor class is practical base class for inheriting custom processing Workers. Instead of implementing the classic Consumer._consume() method, the Functor proposes the more powerfull method _process(). This handler method receives as argument sequence, an iterator over the operands, which are iterators over the couples (key, value) obtained from applying the method Request.items() of the request of the functor on the list of received products. This enables to access directly to the name and values of the required data without the need of reimplementing the code to get them. The method _process() is a generator method and it it supposed to yield the the couples (key, value) representing the result of the node processing. The yielded results are automatically considered by the functor to create a new product that will be automatically posted.

The functor uses a Functor.Blender object to create the new product. A set of predefined blenders are used to set the functor behaviour:

• Functor.MERGE — Join the original product and results of the functor.
• Functor.MERGECOPY — Make a deep copy of the original product and then join the results of the functor.
• Functor.RESULTONLY — Post as product only the result of the functor.

A Functor instance propagates the requests only if the current blender is a MergeBlender and it propagates the offers only if the current blender is a MergeBlender or if it is tunneling (see WiseWorker.isTunneling()).

blender()

Return the current active Functor.Blender.

_process(sequence, producer)

This handler method receives as argument sequence, an iterator over the operands, which are iterators over the couples (key, value) obtained from applying the method Request.items() of the request of the functor on the list of received products. This enables to access directly to the name and values of the required data without the need of reimplementing the code to get them. This is a generator method and it it supposed to yield the the couples (key, value) representing the result of the node processing.

6.2.4. Composites

class boing.core.Composite(*internals, parent=None)

A Composite instance stores a set of internal objects using strong references. Use the method internals() to scroll the references objects.

internals()

Return an iterator over the set of internal nodes for which a strong reference is stored.

class boing.core.CompositeProducer(*producers, internals=set(), parent=None)

A CompositeProducer combines in parallel the list of producers so that they can be considered as a single Producer. The argument internals can be used to specify the object for which a strong reference should be kept. All producers are by default added as internals of the Composite.

producers()

Return an iterator over the first level producers.

class boing.core.CompositeConsumer(*consumers, internals=set(), parent=None)

A CompositeConsumer combines in parallel the list of consumers so that they can be considered as a single Consumer. The argument internals can be used to specify the object for which a strong reference should be kept. All consumers are by default added as internals of the Composite.

consumers()

Return an iterator over the first level consumers.

class boing.core.CompositeWorker(consumers, producers, internals=set(), parent=None)

A CompositeWorker object combines in parallel the list of producers and in parallel the list of consumers so that they can be considered as a single Worker. The argument internals can be used to specify the object for which a strong reference should be kept. All producers and consumers are by default added as internals of the Composite.