curiosidade.core
Main entry point for probing models.
Module Contents
Classes
Container of probing models. |
Functions
|
Attach probing models to specified base_model modules. |
|
Return a list of module names available for probing from |
- class curiosidade.core.ProbingModelContainer(device: Union[torch.device, str] = 'cpu', random_seed: Optional[int] = None)
Container of probing models.
- Parameters
device ({'cpu', 'cuda'}, default='cpu') – Device type to train probing models.
random_seed (int or None, default=None) – If specified, this random seed will be used while instantiating the probing models and also during their training, ensuring reprodutible results.
- __repr__(self) str
Return repr(self).
- property task_name(self) str
Return task name.
- property probed_modules(self) tuple[str, Ellipsis]
Return names of all probed modules.
- property pruned_modules(self) tuple[str, Ellipsis]
Return names of all pruned modules.
- __getitem__(self, key: str) curiosidade.probers.ProbingModelWrapper
- __iter__(self) Iterator[tuple[str, curiosidade.probers.ProbingModelWrapper]]
- __len__(self) int
- attach(self, base_model: torch.nn.Module, probing_model_factory: curiosidade.probers.ProbingModelFactory, modules_to_attach: Union[Pattern[str], str, Sequence[str]], match_modules_to_attach_as_regex: bool = True, modules_input_dim: curiosidade.input_handlers.ModuleInputDimType = None, prune_unrelated_modules: Optional[Union[Sequence[str], Literal[infer]]] = None, enable_cuda_in_inspection: bool = True) ProbingModelContainer
Attach probing models to specified base_model modules.
- Parameters
base_model (torch.nn.Module or transformers.PreTrainedModel) – Pretrained base model to attach probing models to.
probing_model_factory (probers.ProbingModelFactory) – Probing model factory object.
modules_to_attach (t.Pattern[str] or str or t.Sequence[str]) – A list or regular expression pattern specifying which model modules should be probed. Use base_model.named_modules() to check available model modules for probing.
match_modules_to_attach_as_regex (bool, default=True) – If True, interpret modules_to_attach value as a regular expression in the form ^s*(?:modules_to_attach)s*$. This argument only has effect when type(modules_to_attach)=str, otherwise it is ignored.
modules_input_dim (dict[str, int] or None, default=None) –
Input dimension of each probing model.
If mapping (dict), should map the module name to its corresponding input dimension. Input dimension of modules not present in this mapping will be inferred;
If None, the input dimensions will be inferred from the output dimensions sequences in base_model.named_modules().
prune_unrelated_modules (t.Sequence[str] or 'infer' or None, default=None) –
Whether or not to prune pretrained modules unrelated to any probing model. This avoids unnecessary computations, speeding up the training procedure substantially if no probing model depends on final pretrained modules. Attempting to forward through pruned modules will interrupt immediately the forward flow, therefore saving computations since no further activations are required to train the probing models.
If ‘infer’, attempt to find the first module such that no probing model depends on any further module outputs. This heuristics only works properly if the model forward flow is deterministic and ‘one-dimensional’ (no bifurcations). This strategy is expected to work most of the time for any regular pretrained model;
If list, must contain the module names to prune;
If None, no module will be pruned, and the pretrained forward flow will be computed on its entirety.
enable_cuda_in_inspection (bool, default=True) – Whether cuda can be enabled during probing input dimension inference, or while in search for unrelated modules in pretrained model. Argument used only if container device is set to ‘cuda’, otherwise it is ignored.
- Return type
self
- detach(self) ProbingModelContainer
Detach and remove all attached probers, if any.
- train(self, num_epochs: int = 1, gradient_accumulation_steps: int = 1, show_progress_bar: Literal[ProbingModelContainer.train.epoch, True, None] = None) curiosidade.output_handlers.ProbingResults
Train probing models.
- Parameters
num_epochs (int, default=1) – Number of training epochs.
gradient_accumulation_steps (int, default=1) – Number of batches before one weight update.
show_progress_bar ({'epoch', True, None}, default=None) –
If ‘epoch’, display a progress bar for each epoch.
If True, display a single progress bar for the entire training procedure.
If None, progress bar is omitted.
- Returns
results – Probing results, separated per split (train, eval, and test).
- Return type
output_handlers.ProbingResults
- curiosidade.core.attach_probers(base_model: torch.nn.Module, probing_model_factory: curiosidade.probers.ProbingModelFactory, modules_to_attach: Union[Pattern[str], str, Sequence[str]], match_modules_to_attach_as_regex: bool = True, modules_input_dim: curiosidade.input_handlers.ModuleInputDimType = None, prune_unrelated_modules: Optional[Union[Sequence[str], Literal[infer]]] = None, device: Union[torch.device, str] = 'cpu', random_seed: Optional[int] = None) ProbingModelContainer
Attach probing models to specified base_model modules.
- Parameters
base_model (torch.nn.Module) – Pretrained base model to attach probing models to.
probing_model_factory (probers.ProbingModelFactory) – Probing model factory object.
modules_to_attach (t.Pattern[str] or str or t.Sequence[str]) – A list or regular expression pattern specifying which model modules should be probed. Use base_model.named_modules() to check available model modules for probing.
match_modules_to_attach_as_regex (bool, default=True) – If True, interpret modules_to_attach value as a regular expression in the form ^s*(?:modules_to_attach)s*$. This argument only has effect when modules_to_attach type is str, otherwise it is ignored.
modules_input_dim (dict[str, int] or None, default=None) –
Input dimension of each probing model.
If mapping (dict), should map the module name to its corresponding input dimension. Input dimension of modules not present in this mapping will be inferred.
If None, the input dimensions will be inferred from the output dimensions sequences in base_model.named_modules().
prune_unrelated_modules (t.Sequence[str] or 'infer' or None, default=None) –
Whether or not to prune pretrained modules unrelated to any probing model. This avoids unnecessary computations, speeding up the training procedure substantially if no probing model depends on final pretrained modules. Attempting to forward through pruned modules will interrupt immediately the forward flow, therefore saving computations since no further activations are required to train the probing models.
If ‘infer’, will attemp to find the first module such that no probing model depends on any further module outputs. This heuristics only works properly if the model forward flow is deterministic and ‘one-dimensional’ (no bifurcations). This strategy is expected to work most of the time for any regular pretrained model.
If list, must contain the module names to prune.
If None, no module will be pruned, and the pretrained forward flow will be computed on its entirety.
device ({'cpu', 'cuda'}, default='cpu') – Device used to train probing models.
random_seed (int or None, default=None) – If specified, this random seed will be used while instantiating the probing models and also during their training, ensuring reprodutible results.
- Returns
probers – Container with every instantiated probing model, prepared for training.
- Return type
See also
ProbingModelContainer.traintrain returned ProbingModelContainer instance.
functools.partialcreate a factory with preset named arguments.
- curiosidade.core.get_modules(pretrained_model: torch.nn.Module) List[str]
Return a list of module names available for probing from
pretrained model.This is a utility function equivalent to: >>> modules = [name for name, _ in pretrained_model.named_modules() if name] # doctest: +SKIP