spark.core.config

spark.core.config#

Attributes#

METADATA_TEMPLATE

Classes#

`SparkMetaConfig`	Metaclass that promotes class attributes to dataclass fields
`BaseSparkConfig`	Base class for module configuration.
`SparkConfig`	Default class for module configuration.

Module Contents#

spark.core.config.METADATA_TEMPLATE[source]#

class spark.core.config.SparkMetaConfig[source]#

Bases: abc.ABCMeta

Metaclass that promotes class attributes to dataclass fields

class spark.core.config.BaseSparkConfig(**kwargs)[source]#

Bases: abc.ABC

Base class for module configuration.

__config_delimiter__: str = '__'[source]#

__shared_config_delimiter__: str = '_s_'[source]#

__graph_editor_metadata__: dict[source]#

classmethod __init_subclass__(**kwargs)[source]#

merge(partial={})[source]#

Update config with partial overrides.

Parameters:: partial (dict[str, Any])
Return type:: None

diff(other)[source]#

Return differences from another config.

Parameters:: other (BaseSparkConfig)
Return type:: dict[str, Any]

validate()[source]#

Validates all fields in the configuration class.

Return type:: None

get_field_errors(field_name)[source]#

Validates all fields in the configuration class.

Parameters:: field_name (str)
Return type:: list[str]

get_metadata()[source]#

Returns all the metadata in the configuration class, indexed by the attribute name.

Return type:: dict[str, Any]

property class_ref: type[source]#

Returns the type of the associated Module/Initializer.

NOTE: It is recommended to set the __class_ref__ to the name of the associated module/initializer when defining custom configuration classes. The current class_ref solver is extremely brittle and likely to fail in many different custom scenarios.

Return type:: type

__post_init__()[source]#

to_dict()[source]#

Serialize config to dictionary

Return type:: dict[str, dict[str, Any]]

classmethod from_dict(dct)[source]#

Create config instance from dictionary.

Parameters:: dct (dict)
Return type:: BaseSparkConfig

to_file(file_path)[source]#

Export a config instance from a .scfg file.

Parameters:: file_path (str)
Return type:: None

classmethod from_file(file_path)[source]#

Create config instance from a .scfg file.

Parameters:: file_path (str)
Return type:: None

__iter__()[source]#

Custom iterator to simplify SparkConfig inspection across the entire ecosystem. This iterator excludes private fields.

Output:: field_name: str, field name field_value: tp.Any, field value

Return type:: Iterator[tuple[str, dataclasses.Field, Any]]

get_tree_structure()[source]#

Return type:: str

class spark.core.config.SparkConfig(**kwargs)[source]#

Bases: BaseSparkConfig

Default class for module configuration.

seed: int[source]#

dtype: jax.typing.DTypeLike[source]#

dt: float[source]#