Once pipeline

class OncePipeline(pipeline=None, *namespaces)

Pipeline that runs only once before or after the main pipeline

add_namespace(*namespaces)

classmethod concat(pipe1, pipe2)

Concatenate two pipelines

copy()

Make a shallow copy of the dataset object

import_model(name, source)

Import a model

Parameters:

• name (str) – a name with which the model is stored in this pipeline

• source – a model or a pipeline to import from

Examples

Import a given model instance:

pipeline.before.import_model('my-model', custom_resnet_model)

Import my-model from the pipeline:

pipeline.before.import_model('my-model', train_pipeline)

Import resnet model from train_pipeline and give it a name my-model:

pipeline.before.import_model('my-model', train_pipeline.m('resnet'))

init_lock(name='lock', **kwargs)

Create a lock as a pipeline variable

Parameters:

name (string) – a lock name

Return type:

self - in order to use it in the pipeline chains

Examples

>>> pp = dataset.p.before
            .init_lock("model_update")

init_model(name, model_class=None, mode='dynamic', config=None, source=None)

Initialize a static or dynamic model by building or importing it

Parameters:

• name (str) – a name for the model (to refer to it later when training or infering).

• model_class (class or named expression) – a model class (might also be specified in the config).

• mode ({'static', 'dynamic'}) – model creation mode: - static - the model is created right now, during the pipeline definition - dynamic - the model is created at the first iteration when the pipeline is run (default)

• config (dict or Config) – (optional) model configurations parameters, where each key and value could be named expressions.

• source – a model or a pipeline to import from

Examples

Build a model:

pipeline.before.init_model('my-model', MyModel, 'static')

Import a model:

pipeline.before.init_model('my-model', source=train_pipeline)

Build a model with a config:

pipeline.before
  .init_variable('images_shape', [256, 256])
  .init_model('my_model', MyModel, 'static', config={'input_shape': V('images_shape')})

pipeline.before
  .init_variable('shape_name', 'images_shape')
  .init_model('my_model', C('model'), 'dynamic', config={V('shape_name)': B('images_shape')})

init_variable(name, default=None, lock=True, **kwargs)

Create a variable if not exists. If the variable exists, does nothing.

Parameters:

• name (string) – a name of the variable

• default – an initial value for the variable set when pipeline is created

• lock (bool) – whether to lock a variable before each update (default: True)

Return type:

self - in order to use it in the pipeline chains

Examples

>>> pp = dataset.p.before
            .init_variable("iterations", default=0)
            .init_variable("accuracy")
            .init_variable("loss_history", [])

load_model(name, model_class=None, mode='dynamic', *args, **kwargs)

Load a model

run()

Execute all actions

save_model(name, *args, **kwargs)

Save a model

save_to(dst, value=None)

Save a value of a given named expression lazily during pipeline execution

Parameters:

• dst (NamedExpression or any data container) – destination

• value – an updating value, could be a value of any type or a named expression

Return type:

self - in order to use it in the pipeline chains

Notes

This method does not change a value of the variable until the pipeline is run. So it should be used in pipeline definition chains only. save_data_to() is imperative and may be used to change variable value within actions.

update(expr, value=None)

Update a value of a given named expression lazily during pipeline execution

Parameters:

• expr (NamedExpression) – an expression

• value – an updating value, could be a value of any type or a named expression

Return type:

self - in order to use it in the pipeline chains

Notes

This method does not change a value of the variable until the pipeline is run. So it should be used in pipeline definition chains only. set_variable is imperative and may be used to change variable value within actions.