pypond.io package

Submodules

pypond.io.input module

Classes to handle pipeline input.

class pypond.io.input.Bounded

Bases: pypond.io.input.PipelineIn

For the pipeline - source of a fixed size - like a collection.

on_emit()
start()
stop()
class pypond.io.input.PipelineIn

Bases: pypond.bases.Observable

For the pipeline - raise exceptions if an attempt is made to add heterogenous types.

class pypond.io.input.Stream

Bases: pypond.io.input.PipelineIn

For the pipeline - a source that has no container of its own.

add_event(event)

Type check and event and emit it if we are running have have observers.

Parameters:event (Event) – Some Event class
events()

Raise an exception - can’t iterate an unbounded source.

start()
stop()

pypond.io.output module

Objects to handle Pipeline output and event collection.

class pypond.io.output.CollectionOut(pipeline, callback, options)

Bases: pypond.io.output.PipelineOut

Output object for when processor results are being returned as a collection.

Parameters:
  • pipeline (Pipeline) – A reference to the calling Pipeline instance.
  • callback (function or None) – Will either be a function that the collector callback will pass things to or None which will pass the results back to the calling Pipeline.
  • options (Options) – An Options object.
add_event(event)

Add an event to the collector.

Parameters:event (Event) – An event object
flush()

Flush the collector and mark the results_done = True in the pipeline if there is no longer an observer.

on_emit(callback)

Sets the internal callback.

Parameters:callback (function or None) – Value to set the intenal _callback to.
class pypond.io.output.Collector(options, on_trigger)

Bases: pypond.bases.PypondBase

A Collector is used to accumulate events into multiple collections, based on potentially many strategies. In this current implementation a collection is partitioned based on the window that it falls in and the group it is part of.

Collections are emitted from this class to the supplied onTrigger callback.

Parameters:
  • options (Options) – A pipeline options instance
  • on_trigger (function) – Callback to handle the emitted Collection
add_event(event)

Add and event to the _collections dict and act accordingly depending on how _emit_on is set.

Parameters:event (Event) – An event.
Raises:PipelineIOException – Raised on bad args.
emit_collections(collections)

Emit all of the collections to the trigger callback that was passed in by the Processor

Parameters:collections (dict) – A dict of string keys and Capsule objects containing the window_key, group_by_key and a Collection.
flush_collections()

Emit the remaining collections.

class pypond.io.output.EventOut(pipeline, callback=None, options=<pypond.util.Options object>)

Bases: pypond.io.output.PipelineOut

Output object for when processor results are being returned as events.

Parameters:
  • pipeline (Pipeline) – A reference to the calling Pipeline instance.
  • callback (function or None) – Will either be a function that the collector callback will pass things to or None which will pass the results back to the calling Pipeline.
  • options (Options) – An Options object.
add_event(event)

Add an event to the pipeline or callback.

Parameters:event (Event) – An event object
flush()

Mark the results_done = True in the pipeline if there is no longer an observer.

on_emit(callback)

Sets the internal callback.

Parameters:callback (function or None) – Value to set the intenal _callback to.
class pypond.io.output.PipelineOut(pipeline)

Bases: pypond.bases.PypondBase

Base class for pipeline output classes

Parameters:pipeline (Pipeline) – The Pipeline

Module contents