nlsq.streaming.large_dataset.estimate_memory_requirements

nlsq.streaming.large_dataset.estimate_memory_requirements(n_points, n_params)

Estimate memory requirements for a dataset.

Parameters:

• n_points (int) – Number of data points

• n_params (int) – Number of parameters

Returns:

Memory requirements and processing recommendations

Return type:

DatasetStats

Examples

>>> from nlsq.streaming.large_dataset import estimate_memory_requirements
>>>
>>> # Estimate requirements for 50M points, 3 parameters
>>> stats = estimate_memory_requirements(50_000_000, 3)
>>> print(f"Estimated memory: {stats.total_memory_estimate_gb:.2f} GB")
>>> print(f"Recommended chunk size: {stats.recommended_chunk_size:,}")
>>> print(f"Number of chunks: {stats.n_chunks}")