Named expressions

Contains named expression classes

eval_expr(expr, no_eval=None, **kwargs)

Evaluate a named expression recursively

Parameters:

• expr – an expression to evaluate

• no_eval (sequence of str) – a list of arguments not to evalute. Applicable only if expr is a dict which keys are checked against no_eval list.

swap(op)

Swap args

add_ops(cls)

Add arithmetic operations to a class. Allows to create and parse syntax trees using operations like ‘+’, ‘-’, ‘*’, ‘/’.

Parameters:

op_cls (class) – The class which represents an arithmetics expression.

class MetaNamedExpression

Meta class to allow for easy instantiation through attribution

Examples

B.images is equal to B(‘images’), but requires fewer letters to type

class NamedExpression(name=None, mode='w', **kwargs)

Base class for a named expression

name

a name

Type:

str

mode

a default assignment method: write, append, extend, update. Can be shrotened to jiust the first letter: w, a, e, u.

• ‘w’ - overwrite with a new value. This is a default mode.

• ‘a’ - append a new value

  (see list.append https://docs.python.org/3/tutorial/datastructures.html#more-on-lists)

• ‘e’ - extend with a new value

  (see list.extend https://docs.python.org/3/tutorial/datastructures.html#more-on-lists)

• ‘u’ - update with a new value

  (see dict.update https://docs.python.org/3/library/stdtypes.html#dict.update or set.update https://docs.python.org/3/library/stdtypes.html#frozenset.update)

Type:

str

str()

Convert a named expression value to a string

format(string)

Convert a value to a formatted representation, controlled by format spec.

Examples

Unlike Python built-in function, the usage is value.format(format_spec), for example:

V('variable').format('Value of the variable is {:7.7}')

set_params(**kwargs)

get(**kwargs)

Return a value of a named expression

Notes

This method should be overriden in child classes. In the first line it should usually call _get_params method:

name, kwargs = self._get_params(**kwargs)

set(value, mode=None, eval=True, **kwargs)

Set a value to a named expression

Parameters:

• mode (str) – an assignment method: write, append, extend, update. A default mode may be specified when instantiating an expression.

• eval (bool) – whether to evaluate value before assigning it to the expression (as value might contain other named expressions, so it should be processed recursively)

assign(value, **kwargs)

Assign a value to a named expression

append(value, *args, **kwargs)

Append a value to a named expression

if a named expression is a dict or set, update is called, or append otherwise.

See also

list.append, dict.update, set.update

extend(value, *args, **kwargs)

Extend a named expression with a new value (see list.extend https://docs.python.org/3/tutorial/datastructures.html#more-on-lists)

update(value, *args, **kwargs)

Update a named expression with a new value (see dict.update https://docs.python.org/3/library/stdtypes.html#dict.update or set.update https://docs.python.org/3/library/stdtypes.html#frozenset.update)

class AlgebraicNamedExpression(op=None, a=None, b=None, c=None)

Algebraic expression over named expressions

get(**kwargs)

Return a value of an algebraic expression

assign(value, **kwargs)

Assign a value to a named expression

class IF

Select either true or false, based on condition. Useful for simple variables that change along the run of a pipeline.

Examples

Select model mode based on the current pipeline iteration::

mode = IF(condition=I.current<450, true=’train’, false=’eval’)

Train the last 20% with larger batch size::

batch_size = IF(condition=I.ratio > 0.8, true=256, false=128)

Notes

An alternative to this named expression is to use F::

def select_batch_size(ratio):

return 256 if ratio > 0.8 else 128

batch_size = F(select_batch_size)(ratio=I.ratio)

Or, with a lambda::

batch_size = F(lambda ratio: 256 if ratio > 0.8 else 128)(ratio=I.ratio)

F is recommended where more flexibility is needed, and IF can be used for simple binary choices.

get(**kwargs)

Select based on condition.

assign(value, **kwargs)

Assign a value to a named expression, based on condition.

class B(name=None, mode='w', copy=False)

Batch component or attribute name

Notes

B() return the batch itself.

To avoid unexpected data changes the copy of the batch may be returned, if copy=True.

Examples

B('size')
B('images_shape')
B(copy=True)

get(**kwargs)

Return a value of a batch component

assign(value, **kwargs)

Assign a value to a batch component

class L(name=None, mode='w', **kwargs)

List of objects or a batch component with a list of objects.

Note

L('comp').attr is equivalent to the list comprehension [val.attr for val in batch.comp]. L('comp').func(*args, **kwargs) is equivalent to the list comprehension ``[val.func(*args, **kwargs)

for val in batch.comp]``.

L('comp')[item] is equivalent to the list comprehension [val[item] for val in batch.comp]. Any chains of consecutive calls of items or attribures like ``L(‘comp’).attr[item].attr2 … `` are also allowed.

get(**kwargs)

Returns an instance of the class that allows one to access attributes or items stored in the batch component or call a method from it.

assign(value, **kwargs)

Assign a value to batch component or item/attribute stored in the batch component

class PipelineNamedExpression(name=None, mode='w', **kwargs)

Base class for pipeline expressions

class C(name=None, mode='w', **kwargs)

A pipeline config option

Notes

C() return config itself.

Examples

Get a value from the current pipeline config:

C('model_class', default=ResNet)
C('GPU')

Get the whole config from the current pipeline:

C()

Get a value from another pipeline config:

C('model_class', pipeline=train_pipeline)

get(**kwargs)

Return a value of a pipeline config

assign(value, **kwargs)

Assign a value to a pipeline config

class V(name=None, mode='w', **kwargs)

Pipeline variable name

Examples

Get a variable value from the current pipeline:

V('model_name')

Get a variable value from another pipeline:

V('loss_history', pipeline=train_pipeline)

get(**kwargs)

Return a value of a pipeline variable

assign(value, **kwargs)

Assign a value to a pipeline variable

class M(name=None, mode='w', **kwargs)

Model name

Examples

Get a model from the current pipeline:

M('model_name')

Get a model from a given pipeline:

M('model_name', pipeline=train_pipeline)

Get a model from a pipeline specified in the current pipeline config:

M('model_name', pipeline=C('train_pipeline'))

Get a model from a pipeline specified in another pipeline config:

M('model_name', pipeline=C('train_pipeline', pipeline=test_template))

get(**kwargs)

Return a model from a pipeline

assign(value, batch=None, pipeline=None)

Assign a value to a model

class I(name='c', mode='w', **kwargs)

Iteration counter

Parameters:

name (str) –

Determines returned value. One of:

• ’current’ or its substring - current iteration number, default.

• ’maximum’ or ‘total’ or their substring - total number of iterations to be performed. If total number is not defined, raises an error.

• ’ratio’ or its substring - current iteration divided by a total number of iterations.

Raises:

• ValueError –

• If name is not valid. –

• If name is 'm' or 'r' and total number of iterations is not defined. –

Examples

I('current')
I('max')
I('max')
R('normal', loc=0, scale=I('ratio')*100)

get(**kwargs)

Return current or maximum iteration number or their ratio

assign(*args, **kwargs)

Assign a value by calling a callable

class R(name, *args, seed=None, size=None, **kwargs)

A random value

Parameters:

• name (str) – a distribution name

• seed (int, SeedSequence, Generator, BitGenerator, RandomState) – a random state (see make_rng())

• args – distribution parameters

• kwargs – distribution parameters

Notes

If size is needed, it should be specified as a named, not a positional argument.

Examples

R('normal', 0, 1)
R('poisson', lam=5.5, seed=42, size=3)
R(['metro', 'taxi', 'bike'], p=[.6, .1, .3], size=10)

get(size=None, **kwargs)

Return a value of a random variable

Parameters:

size (int, tuple of int) – Output shape. If the given shape is (m, n, k), then m * n * k samples are drawn and returned as m x n x k array. If size was also specified at instance creation, then output shape is extended from the beginning. So size is treated like a batch size, while size specified at instantiation is an item size.

Examples

ne = R('normal', 0, 1, size=(10, 20)))
value = ne.get(batch=batch)
# value.shape will be (10, 20)

value = ne.get(size=30, batch=batch)
# value.shape will be (30, 10, 20)
# so size is treated like a batch size

assign(*args, **kwargs)

Assign a value

class F(name=None, mode='w', **kwargs)

A function, method or any other callable that might take arguments

Examples

F(MyBatch.rotate)(B(), angle=30)
F(make_data)
F(prepare_data)(batch=B(), item=10)

get(_call=True, **kwargs)

Return a value from a callable

Parameters:

_call (bool) – Whether to call name-function while evaluating the expression. Sometimes we might not want calling the func, e.g. when evaluating an F-expr within a call-expression F(func)(1, arg2=10), since we want to evaluate the whole expression.

assign(*args, **kwargs)

Assign a value by calling a callable

class D(name=None, mode='w', **kwargs)

Dataset attribute or dataset itself

Examples

D()
D('classes')
D('organization')

get(**kwargs)

Return a value of a dataset attribute

assign(value, **kwargs)

Assign a value to a dataset attribute

class W(name=None, mode='w', **kwargs)

A wrapper which returns the wrapped named expression without evaluating it

Examples

W(V('variable'))
W(B(copy=True))
W(R('normal', 0, 1, size=B('size')))

get(**kwargs)

Return a wrapped named expression

assign(value, **kwargs)

Assign a value

class P(name=None, mode='w', **kwargs)

A wrapper for values passed to actions parallelized with @inbatch_parallel

Examples

Each image in the batch will be rotated at its own angle:

pipeline
    .rotate(angle=P(R('normal', 0, 1)))

Without P all images in the batch will be rotated at the same angle, as an angle is randomized across batches only:

pipeline
    .rotate(angle=R('normal', 0, 1))

To put it simply, R(...) is evaluated as R(..., size=batch.size).

Generate 3 categorical random samples for each batch item:

pipeline
    .calc_route(P(R(['metro', 'taxi', 'bike'], p=[.6, 0.1, 0.3], size=3))

Generate a random number of random samples for each batch item:

pipeline
    .some_action(P(R('normal', 0, 1, size=R('randint', 3, 8))))

P works with arbitrary iterables too:

pipeline
    .do_something(n=P([1, 2, 3, 4, 5]))

The first batch item will get n=1, the second n=2 and so on.

See also

inbatch_parallel()

get(*args, parallel=False, **kwargs)

Calculate and return a value of the expression

assign(value, **kwargs)

Assign a value

class PP(name=None, mode='w', **kwargs)

A wrapper for single-value expressions passed to actions parallelized with @inbatch_parallel PP(expr) is essentialy P([expr for _ in batch.indices])

Examples

Each image in the batch will be rotated at its own angle:

pipeline
    .rotate(angle=PP(F(get_single_angle)))

as get_single_angle will be called batch.size times.

R(...) will be evaluated only once within P(...), but many times within PP(...):

pipeline
    .rotate(angle=PP(R('normal', 0, 1)))

That is why P(R(...)) is much more efficient than PP(R(...)).

However, PP is indispensable for shape-specific operations like @ or broadcasting:

pipeline
    .rotate(angle=PP(R('normal', R('normal', 50, 15, size=3), 15)))

Internal R specifies a 3D angle mean and thus defines the shape. External R knows nothing about that shape and will throw an exception within P, but it’ll work fine within PP.

See also

inbatch_parallel(), P

get(*_, **kwargs)

Calculate and return a value of the expression