curiosidade.output_handlers.handlers

Handle probing results appropriately.

Module Contents

Classes

MetricPack	Store and aggregate metrics from probe model training.
ProbingResults	Class to store probing results per split.

class curiosidade.output_handlers.handlers.MetricPack

Store and aggregate metrics from probe model training.

__getitem__(self, key: tuple[Any, Ellipsis]) → float

__iter__(self) → Iterator[tuple[tuple[Any, Ellipsis], float]]

__repr__(self) → str

Return repr(self).

append(self, metrics: dict[Any, float], *args: Any, merge_keys: bool = True) → MetricPack

Append extra metrics in stored items.

Parameters

• metrics (dict[t.Any, float]) – Metrics to store.

• *args (tuple[t.Any]) – Extra arguments to build the keys.

• merge_keys (bool, default=True) – If True, colapse containers within existing keys in metrics to merge with provided *args (if any). If False, the new keys will be a 2-tuple in the format (previous_key, *args).

Return type

self

expand_key_dim(self, *args: Any) → MetricPack

Add a new value at the start of every stored key.

combine(self, metrics: MetricPack) → MetricPack

Combine stored items with items in metrics.

to_pandas(self, aggregate_by: Optional[Sequence[str]] = None, aggregate_fn: Callable[[Sequence[float]], float] = np.mean) → pandas.DataFrame

Build a pandas DataFrame with stored values.

Parameters

• aggregate_by (t.Sequence[str] or None, default=None) – If given, aggregate results by the provided keys. It must be a sequence of strings contained a subset of {‘epoch’, ‘metric_name’, ‘module’, ‘batch_index’}. The function used to aggregate values that fall in the same bucket is aggregate_fn.

• aggregate_fn (callable or sequence of callables, default=numpy.mean) – Function, or sequence of callables used to aggregate values in a same bucket. Also accept values supported by pandas Aggregate function. Must receive a sequence of values, and return a single number. Ignored if aggregate_by=None.

Returns

dataframe – Stored values in the form of a pandas dataframe.

Return type

pandas.DataFrame

Examples

>>> import curiosidade
...
>>> metrics = curiosidade.output_handlers.MetricPack()
>>> metrics.append({
...     (0, 'loss', 'relu1', 0): 1.50,
...     (0,  'acc', 'relu1', 0): 0.10,
...     (0, 'loss', 'relu1', 1): 1.40,
...     (0,  'acc', 'relu1', 1): 0.20,
...     (1, 'loss', 'relu1', 0): 1.15,
...     (1,  'acc', 'relu1', 0): 0.33,
...     (1, 'loss', 'relu1', 1): 0.85,
...     (1,  'acc', 'relu1', 1): 0.43,
... })
MetricPack with 8 values stored in:
  (0, 'loss', 'relu1', 0): 1.5
  (0, 'acc', 'relu1', 0): 0.1
  (0, 'loss', 'relu1', 1): 1.4
  ...
  (1, 'acc', 'relu1', 0): 0.33
  (1, 'loss', 'relu1', 1): 0.85
  (1, 'acc', 'relu1', 1): 0.43
>>> metrics.to_pandas(
...     aggregate_by=['batch_index'],
...     aggregate_fn=[np.max, np.min],
... ).values
array([[0, 'acc', 'relu1', 0.2, 0.1],
       [0, 'loss', 'relu1', 1.5, 1.4],
       [1, 'acc', 'relu1', 0.43, 0.33],
       [1, 'loss', 'relu1', 1.15, 0.85]], dtype=object)

class curiosidade.output_handlers.handlers.ProbingResults

Bases: NamedTuple

Class to store probing results per split.

train :Union[pandas.DataFrame, MetricPack]

eval :Optional[Union[pandas.DataFrame, MetricPack]]

test :Optional[Union[pandas.DataFrame, MetricPack]]

__repr__(self) → str

Return repr(self).

to_pandas(self, aggregate_by: Optional[Sequence[str]] = None, aggregate_fn: Callable[[Sequence[float]], float] = np.mean) → ProbingResults

Build a pandas DataFrame with stored values.

Parameters

• aggregate_by (t.Sequence[str] or None, default=None) – If given, aggregate results by the provided keys. It must be a sequence of strings contained a subset of {‘epoch’, ‘metric_name’, ‘module’, ‘batch_index’}. The function used to aggregate values that fall in the same bucket is aggregate_fn.

• aggregate_fn (callable or sequence of callables, default=numpy.mean) – Function, or sequence of callables used to aggregate values in a same bucket. Also accept values supported by pandas Aggregate function. Must receive a sequence of values, and return a single number. Ignored if aggregate_by=None.

Returns

dataframes – Results casted to pandas DataFrames.

Return type

ProbingResults