Large Dataset API Reference¶

APIs for handling datasets that exceed GPU memory.

Overview¶

NLSQ provides three approaches for large datasets:

API	Best For	Memory Behavior
`curve_fit_large`	Datasets up to ~100M points	Automatic chunking, fits in memory
`LargeDatasetFitter`	Fine-grained control over chunking	Automatic memory management
`AdaptiveHybridStreamingOptimizer`	Huge datasets and production pipelines	Four-phase optimization

curve_fit_large¶

nlsq.curve_fit_large(f, xdata, ydata, p0=None, sigma=None, absolute_sigma=False, check_finite=True, bounds=(-inf, inf), method=None, stability=False, rescale_data=True, max_jacobian_elements_for_svd=10000000, memory_limit_gb=None, auto_size_detection=True, size_threshold=1000000, show_progress=False, chunk_size=None, multistart=False, n_starts=10, global_search=False, sampler='lhs', center_on_p0=True, scale_factor=1.0, **kwargs)[source]

Curve fitting with automatic memory management for large datasets.

Automatically selects processing strategy based on dataset size: - Small (< 1M points): Standard curve_fit - Medium (1M - 100M points): Chunked processing - Large (> 100M points): Streaming optimization

Parameters:

f (callable) – Model function f(x, *params) -> y. Must use jax.numpy operations.
xdata (array_like) – Independent variable data.
ydata (array_like) – Dependent variable data.
p0 (array_like, optional) – Initial parameter guess.
sigma (array_like, optional) – Uncertainties in ydata for weighted fitting.
absolute_sigma (bool, optional) – Whether sigma represents absolute uncertainties.
check_finite (bool, optional) – Check for finite input values.
bounds (tuple, optional) – Parameter bounds as (lower, upper).
method (str, optional) – Optimization algorithm (‘trf’, ‘lm’, or None for auto).
memory_limit_gb (float, optional) – Maximum memory usage in GB.
auto_size_detection (bool, optional) – Auto-detect dataset size for processing strategy.
size_threshold (int, optional) – Point threshold for large dataset processing (default: 1M).
show_progress (bool, optional) – Display progress bar for long operations.
chunk_size (int, optional) – Override automatic chunk size calculation.
multistart (bool, optional) – Enable multi-start optimization for global search. Default: False.
n_starts (int, optional) – Number of starting points for multi-start optimization. Default: 10.
global_search (bool, optional) – Shorthand for multistart=True, n_starts=20. Default: False.
sampler ({'lhs', 'sobol', 'halton'}, optional) – Sampling strategy for multi-start. Default: ‘lhs’.
center_on_p0 (bool, optional) – Center multi-start samples around p0. Default: True.
scale_factor (float, optional) – Scale factor for exploration region. Default: 1.0.
**kwargs (Any) – Additional optimization parameters (ftol, xtol, gtol, max_nfev, loss)

Returns:

popt (ndarray) – Fitted parameters.
pcov (ndarray) – Parameter covariance matrix.

Return type:

tuple[ndarray, ndarray] | OptimizeResult

Notes

All large datasets use streaming optimization for 100% data utilization.

Important: Model Function Requirements for Chunking

When auto_size_detection triggers chunked processing (>1M points), your model function MUST respect the size of xdata. Model output shape must match ydata shape.

INCORRECT - Fixed-size output (causes shape errors):

>>> def bad_model(xdata, a, b):
...     # WRONG: Returns fixed-size array regardless of xdata
...     t_full = jnp.arange(10_000_000)
...     return a * jnp.exp(-b * t_full)  # Shape mismatch!

CORRECT - Output matches xdata size:

>>> def good_model(xdata, a, b):
...     # CORRECT: Uses xdata as indices
...     indices = xdata.astype(jnp.int32)
...     return a * jnp.exp(-b * indices)

>>> def direct_model(xdata, a, b):
...     # CORRECT: Operates on xdata directly
...     return a * jnp.exp(-b * xdata)

Examples

Basic usage:

>>> popt, _pcov = curve_fit_large(model_func, xdata, ydata, p0=[1, 2, 3])

Large dataset with progress bar:

>>> popt, _pcov = curve_fit_large(model_func, big_xdata, big_ydata,
...                             show_progress=True, memory_limit_gb=8)

With multi-start optimization:

>>> popt, _pcov = curve_fit_large(model_func, xdata, ydata,
...                             p0=[1, 2, 3], bounds=([0, 0, 0], [10, 10, 10]),
...                             multistart=True, n_starts=10)

Using external logger for diagnostics:

>>> import logging
>>> my_logger = logging.getLogger("myapp")
>>> fitter = LargeDatasetFitter(memory_limit_gb=8, logger=my_logger)
>>> result = fitter.fit(model_func, xdata, ydata, p0=[1, 2])
>>> # Chunk failures now appear in myapp's logs

Automatically chunks data to fit within GPU memory.

Key Parameters:

memory_limit_gb: Maximum GPU memory to use
chunk_size: Override automatic chunk sizing
progress: Show progress bar during fitting

Example:

from nlsq import curve_fit_large

# 50 million data points
x = jnp.linspace(0, 100, 50_000_000)
y = 2.5 * jnp.exp(-0.1 * x) + 0.5 + noise

popt, pcov = curve_fit_large(
    model, x, y, p0=[1.0, 0.1, 0.0], memory_limit_gb=8.0  # Limit to 8 GB
)

LargeDatasetFitter¶

class nlsq.LargeDatasetFitter(memory_limit_gb=8.0, config=None, curve_fit_class=None, logger=None, multistart=False, n_starts=10, sampler='lhs')[source]

Bases: object

Large dataset curve fitting with automatic memory management and chunking.

This class handles datasets with millions to billions of points that exceed available memory through automatic chunking, progressive parameter refinement, and streaming optimization. It maintains fitting accuracy while preventing memory overflow through dynamic memory monitoring and chunk size optimization.

Core Capabilities¶

Automatic memory estimation based on data size and parameter count
Dynamic chunk size calculation considering available system memory
Sequential parameter refinement across data chunks with convergence tracking
Streaming optimization for unlimited datasets (no accuracy loss)
Real-time progress monitoring with ETA for long-running fits
Full integration with NLSQ optimization algorithms and GPU acceleration
Multi-start optimization for global search (uses full data)

Memory Management Algorithm¶

Estimates total memory requirements from dataset size and parameter count
Calculates optimal chunk sizes considering available memory and safety margins
Monitors actual memory usage during processing to prevent overflow
Uses streaming optimization for extremely large datasets (processes all data)

Processing Strategies¶

Single Pass: For datasets fitting within memory limits
Sequential Chunking: Processes data in optimal-sized chunks with parameter propagation
Streaming Optimization: Mini-batch gradient descent for unlimited datasets (no subsampling)

Multi-Start Optimization¶

For medium-sized datasets (1M-100M points), multi-start optimization explores multiple starting points on full data, and the best starting point is then used for the full chunked optimization.

Performance Characteristics¶

Maintains <1% parameter error for well-conditioned problems using chunking
Achieves 5-50x speedup over naive approaches through memory optimization
Scales to datasets of unlimited size using streaming (processes all data)
Provides linear time complexity with respect to chunk count

Model Validation Caching (Task Group 7 - 5.1a)¶

Model functions are validated once per unique function identity using a cache keyed by (id(func), id(func.__code__)). This avoids redundant validation across chunks, providing 1-5% performance gain in chunked processing.

param memory_limit_gb:: Maximum memory usage in GB. System memory is auto-detected if None.
type memory_limit_gb:: float, default 8.0
param config:: Advanced configuration for fine-tuning memory management behavior.
type config:: LDMemoryConfig, optional
param curve_fit_class:: Custom CurveFit instance for specialized fitting requirements.
type curve_fit_class:: nlsq.minpack.CurveFit, optional
param multistart:: Enable multi-start optimization for global search.
type multistart:: bool, default False
param n_starts:: Number of starting points for multi-start optimization.
type n_starts:: int, default 10
param sampler:: Sampling strategy for multi-start: ‘lhs’, ‘sobol’, or ‘halton’.
type sampler:: str, default ‘lhs’

config

Active memory management configuration

Type:: LDMemoryConfig

curve_fitter

Internal curve fitting engine with JAX acceleration

Type:: nlsq.minpack.CurveFit

logger

Internal logging for performance monitoring and debugging

Type:: Logger

fit : Main fitting method with automatic memory management

fit_with_progress : Fitting with real-time progress reporting and ETA

get_memory_recommendations : Pre-fitting memory analysis and strategy recommendations

Important: Chunking-Compatible Model Functions

-----------------------------------------------

When using chunked processing (for datasets > memory limit), your model function

MUST respect the size of xdata. During chunking, xdata will be a subset of the

full dataset, and your model must return output matching that subset size.

\*\*INCORRECT - Model ignores xdata size (will cause shape mismatch errors):**

>>> def bad_model(xdata, a, b):

... # WRONG: Always returns full array, ignoring xdata size

... t_full = jnp.arange(10_000_000) # Fixed size!

... return a * jnp.exp(-b * t_full) # Shape mismatch during chunking

\*\*CORRECT - Model respects xdata size:**

>>> def good_model(xdata, a, b):

... # CORRECT: Uses xdata as indices to return only requested subset

... indices = xdata.astype(jnp.int32)

... return a * jnp.exp(-b * indices) # Shape matches xdata

\*\*Alternative - Direct computation on xdata:**

>>> def direct_model(xdata, a, b):

... # CORRECT: Operates directly on xdata

... return a * jnp.exp(-b * xdata) # Shape automatically matches

Examples

Basic usage with automatic configuration:

>>> import numpy as np
>>> import jax.numpy as jnp
>>>
>>> # 10 million data points
>>> x = np.linspace(0, 10, 10_000_000)
>>> y = 2.5 * jnp.exp(-1.3 * x) + 0.1 + np.random.normal(0, 0.05, len(x))
>>>
>>> fitter = LargeDatasetFitter(memory_limit_gb=4.0)
>>> result = fitter.fit(
...     lambda x, a, b, c: a * jnp.exp(-b * x) + c,
...     x, y, p0=[2, 1, 0]
... )
>>> print(f"Parameters: {result.popt}")
>>> print(f"Chunks used: {result.n_chunks}")

Multi-start optimization:

>>> fitter = LargeDatasetFitter(
...     memory_limit_gb=4.0,
...     multistart=True,
...     n_starts=10,
...     sampler='lhs',
... )
>>> result = fitter.fit(
...     lambda x, a, b, c: a * jnp.exp(-b * x) + c,
...     x, y, p0=[2, 1, 0],
...     bounds=([0, 0, 0], [10, 5, 10])
... )

Advanced configuration with progress monitoring:

>>> config = LDMemoryConfig(
...     memory_limit_gb=8.0,
...     min_chunk_size=10000,
...     max_chunk_size=1000000,
...     use_streaming=True,
...     streaming_batch_size=50000
... )
>>> fitter = LargeDatasetFitter(config=config)
>>>
>>> # Fit with progress bar for long-running operation
>>> result = fitter.fit_with_progress(
...     exponential_model, x_huge, y_huge, p0=[2, 1, 0]
... )

Memory analysis before processing:

>>> recommendations = fitter.get_memory_recommendations(len(x), n_params=3)
>>> print(f"Strategy: {recommendations['processing_strategy']}")
>>> print(f"Memory estimate: {recommendations['memory_estimate_gb']:.2f} GB")
>>> print(f"Recommended chunks: {recommendations['n_chunks']}")

AdaptiveHybridStreamingOptimizer¶

class nlsq.AdaptiveHybridStreamingOptimizer(config=None)[source]

Bases: object

Adaptive hybrid streaming optimizer with four-phase optimization.

This optimizer combines parameter normalization, L-BFGS warmup, streaming Gauss-Newton, and exact covariance computation to provide:

Fast convergence for parameters with different scales
Accurate uncertainty estimates on large datasets
Memory-efficient streaming for unlimited dataset sizes
Production-ready fault tolerance

The optimization proceeds through four phases:

Phase 0: Setup parameter normalization and bounds transformation
Phase 1: L-BFGS warmup with adaptive switching to Phase 2
Phase 2: Streaming Gauss-Newton with exact J^T J accumulation
Phase 3: Denormalize parameters and transform covariance matrix

Parameters:: config (HybridStreamingConfig, optional) – Configuration for all phases of optimization. If None, uses default configuration. See HybridStreamingConfig for details.

config

Configuration object controlling all phases

Type:: HybridStreamingConfig

current_phase

Current optimization phase (0, 1, 2, or 3)

Type:: int

phase_history

History of phase transitions with timing information

Type:: list

phase_start_time

Start time of current phase (seconds since epoch)

Type:: float or None

normalized_params

Current parameters in normalized space

Type:: jax.Array or None

normalizer

Parameter normalizer instance (created in Phase 0)

Type:: ParameterNormalizer or None

normalized_model

Wrapped model function operating in normalized space

Type:: NormalizedModelWrapper or None

normalized_bounds

Bounds transformed to normalized space

Type:: tuple of jax.Array or None

normalization_jacobian

Denormalization Jacobian for covariance transform

Type:: jax.Array or None

Examples

Basic usage with default configuration:

>>> from nlsq import AdaptiveHybridStreamingOptimizer, HybridStreamingConfig
>>> import jax.numpy as jnp
>>> config = HybridStreamingConfig()
>>> optimizer = AdaptiveHybridStreamingOptimizer(config)

With bounds-based normalization:

>>> config = HybridStreamingConfig(
...     normalize=True,
...     normalization_strategy='bounds'
... )
>>> optimizer = AdaptiveHybridStreamingOptimizer(config)

With custom warmup settings:

>>> config = HybridStreamingConfig(
...     warmup_iterations=300,
...     lbfgs_initial_step_size=0.5,
...     gauss_newton_tol=1e-10
... )
>>> optimizer = AdaptiveHybridStreamingOptimizer(config)

HybridStreamingConfig¶

class nlsq.HybridStreamingConfig(normalize=True, normalization_strategy='auto', warmup_iterations=200, max_warmup_iterations=500, warmup_learning_rate=0.001, loss_plateau_threshold=0.0001, gradient_norm_threshold=0.001, active_switching_criteria=None, lbfgs_history_size=10, lbfgs_initial_step_size=0.1, lbfgs_line_search='wolfe', lbfgs_exploration_step_size=0.1, lbfgs_refinement_step_size=1.0, use_learning_rate_schedule=False, lr_schedule_warmup_steps=50, lr_schedule_decay_steps=450, lr_schedule_end_value=0.0001, gradient_clip_value=None, enable_warm_start_detection=True, warm_start_threshold=0.01, enable_adaptive_warmup_lr=True, warmup_lr_refinement=1e-06, warmup_lr_careful=1e-05, enable_cost_guard=True, cost_increase_tolerance=0.05, enable_step_clipping=True, max_warmup_step_size=0.1, gauss_newton_max_iterations=100, gauss_newton_tol=1e-08, trust_region_initial=1.0, regularization_factor=1e-10, cg_max_iterations=100, cg_relative_tolerance=0.0001, cg_absolute_tolerance=1e-10, cg_param_threshold=2000, enable_group_variance_regularization=False, group_variance_lambda=0.01, group_variance_indices=None, enable_residual_weighting=False, residual_weights=None, chunk_size=10000, gc_chunk_interval=10, loop_strategy='auto', enable_checkpoints=True, checkpoint_frequency=100, checkpoint_dir=None, resume_from_checkpoint=None, validate_numerics=True, enable_fault_tolerance=True, max_retries_per_batch=2, min_success_rate=0.5, enable_multi_device=False, callback_frequency=10, verbose=1, log_frequency=1, enable_multistart=False, n_starts=10, multistart_sampler='lhs', elimination_rounds=3, elimination_fraction=0.5, batches_per_round=50, center_on_p0=True, scale_factor=1.0)[source]

Bases: object

Configuration for adaptive hybrid streaming optimizer.

This configuration class controls all aspects of the four-phase hybrid optimizer: - Phase 0: Parameter normalization setup - Phase 1: L-BFGS warmup with adaptive switching - Phase 2: Streaming Gauss-Newton with exact J^T J accumulation - Phase 3: Denormalization and covariance transform

Parameters:

normalize (bool, default=True) – Enable parameter normalization. When True, parameters are normalized to similar scales to improve gradient signal quality and convergence speed.
normalization_strategy (str, default='auto') –
Strategy for parameter normalization. Options:
- ’auto’: Use bounds-based if bounds provided, else p0-based
- ’bounds’: Normalize to [0, 1] using parameter bounds
- ’p0’: Scale by initial parameter magnitudes
- ’none’: Identity transform (no normalization)
warmup_iterations (int, default=200) – Number of L-BFGS warmup iterations before checking switch criteria. With L-BFGS, typical values are 20-50 (5-10x fewer than Adam). More iterations allow better initial convergence before switching to Gauss-Newton.
max_warmup_iterations (int, default=500) – Maximum L-BFGS warmup iterations before forced switch to Phase 2. Safety limit to prevent indefinite warmup when loss plateaus slowly.
warmup_learning_rate (float, default=0.001) – Legacy warmup step size retained for backward compatibility. L-BFGS warmup uses lbfgs_initial_step_size and adaptive step sizes.
loss_plateau_threshold (float, default=1e-4) – Relative loss improvement threshold for plateau detection. Switch to Phase 2 if: abs(loss - prev_loss) / (abs(prev_loss) + eps) < threshold. Smaller values = stricter plateau detection = later switching.
gradient_norm_threshold (float, default=1e-3) – Gradient norm threshold for early Phase 2 switch. Switch to Phase 2 if: ||gradient|| < threshold. Indicates optimization is close to optimum and Gauss-Newton will be effective.
active_switching_criteria (list, default=['plateau', 'gradient', 'max_iter']) –
List of active switching criteria for Phase 1 -> Phase 2 transition. Available criteria:
- ’plateau’: Loss plateau detection (loss_plateau_threshold)
- ’gradient’: Gradient norm below threshold (gradient_norm_threshold)
- ’max_iter’: Maximum iterations reached (max_warmup_iterations)
Switch occurs when ANY active criterion is met.
lbfgs_history_size (int, default=10) – Number of previous gradients and updates to store for L-BFGS Hessian approximation. Standard default from SciPy, PyTorch, and Nocedal & Wright. Larger values give better Hessian approximation but use more memory.
lbfgs_initial_step_size (float, default=0.1) – Initial step size for L-BFGS during cold start (first m iterations while history buffer fills). Small value prevents overshooting when Hessian approximation is poor (identity matrix initially).
lbfgs_line_search (str, default='wolfe') –
Line search method for L-BFGS step acceptance. Options:
- ’wolfe’: Standard Wolfe conditions (default)
- ’strong_wolfe’: Strong Wolfe conditions (stricter)
- ’backtracking’: Simple backtracking line search
lbfgs_exploration_step_size (float, default=0.1) – L-BFGS initial step size for exploration mode (high relative loss). Small value prevents first “Hessian=Identity” step from overshooting.
lbfgs_refinement_step_size (float, default=1.0) – L-BFGS initial step size for refinement mode (low relative loss). Larger value leverages L-BFGS’s near-Newton convergence speed when close to optimum.
gauss_newton_max_iterations (int, default=100) – Maximum iterations for Phase 2 Gauss-Newton optimization. Typical values: 50-200.
gauss_newton_tol (float, default=1e-8) – Convergence tolerance for Phase 2 (gradient norm threshold). Optimization stops if: ||gradient|| < tol.
trust_region_initial (float, default=1.0) – Initial trust region radius for Gauss-Newton step control. Radius is adapted based on actual vs predicted reduction ratio.
regularization_factor (float, default=1e-10) – Regularization factor for rank-deficient J^T J matrices. Added to diagonal: J^T J + regularization_factor * I.
cg_max_iterations (int, default=100) – Maximum iterations for Conjugate Gradient solver in Phase 2. Used when parameter count exceeds cg_param_threshold. Higher values allow better convergence but more computation.
cg_relative_tolerance (float, default=1e-4) – Relative tolerance for CG solver convergence. Convergence check: ||r|| < cg_relative_tolerance * ||J^T r_0||. Implements Inexact Newton strategy for efficiency.
cg_absolute_tolerance (float, default=1e-10) – Absolute tolerance floor for CG solver convergence. Safety floor to prevent over-iteration on well-conditioned systems.
cg_param_threshold (int, default=2000) –
Parameter count threshold for auto-selecting CG vs materialized solver.
- p < threshold: Use materialized J^T J with SVD solve (faster for small p)
- p >= threshold: Use CG with implicit matvec (O(p) memory vs O(p^2))
Threshold balances memory savings vs additional data passes for CG.
enable_group_variance_regularization (bool, default=False) –
Enable variance regularization for parameter groups. When enabled, adds a penalty term to the loss function that penalizes variance within specified parameter groups. This is essential for preventing per-group parameter absorption in multi-component fitting.

The regularized loss becomes L = MSE + group_variance_lambda * sum(Var(group_i)) where each group_i is a slice of parameters defined by group_variance_indices.
group_variance_lambda (float, default=0.01) – Regularization strength for group variance penalty. Larger values more strongly penalize variance within parameter groups. Use 0.001-0.01 for light regularization (allows moderate group variation), 0.1-1.0 for moderate regularization (constrains groups to be similar), or 10-1000 for strong regularization (forces groups to be nearly uniform). For multi-component fits with per-group parameters, use lambda ~ 0.1 * n_data / (n_groups * sigma^2) where sigma is the expected experimental variation (~0.05 for 5%).
group_variance_indices (list of tuple, default=None) –
List of (start, end) tuples defining parameter groups for variance regularization. Each tuple specifies a slice [start:end] of the parameter vector that should have low internal variance.

Example for 23 independent groups: group_variance_indices = [(0, 23), (23, 46)] constrains contrast params [0:23] and offset params [23:46] to each have low variance, preventing them from absorbing group-dependent physical signals.

If None when enable_group_variance_regularization=True, no groups are regularized (effectively disabling the feature).
chunk_size (int, default=10000) – Size of data chunks for streaming J^T J accumulation. Larger chunks = faster but more memory. Typical: 5000-50000.
gc_chunk_interval (int, default=10) – Chunks between gc.collect() calls (FR-007). Controls how often garbage collection runs during chunked processing. Higher values reduce GC overhead but may increase memory usage. Default of 10 balances memory reclamation with performance.
enable_checkpoints (bool, default=True) – Enable checkpoint save/resume for fault tolerance.
checkpoint_frequency (int, default=100) – Save checkpoint every N iterations (across all phases).
validate_numerics (bool, default=True) – Enable NaN/Inf validation at gradient, parameter, and loss computation points.
enable_multi_device (bool, default=False) – Enable multi-GPU/TPU parallelism for Jacobian computation. Uses JAX pmap for data-parallel computation across devices.
callback_frequency (int, default=10) – Call progress callback every N iterations (if callback provided).
enable_multistart (bool, default=False) – Enable multi-start optimization with tournament selection during Phase 1. When enabled, generates multiple starting points using LHS sampling and uses tournament elimination to select the best candidate for Phase 2.
n_starts (int, default=10) – Number of starting points for multi-start optimization. Only used when enable_multistart=True.
multistart_sampler (str, default='lhs') – Sampling method for generating starting points. Options: ‘lhs’ (Latin Hypercube), ‘sobol’, ‘halton’.
elimination_rounds (int, default=3) – Number of tournament elimination rounds. Each round eliminates elimination_fraction of candidates.
elimination_fraction (float, default=0.5) – Fraction of candidates to eliminate per round. Must be in (0, 1). Default 0.5 = eliminate half each round.
batches_per_round (int, default=50) – Number of data batches to use for evaluation in each tournament round. More batches = more reliable selection but slower.

Examples

Default configuration:

>>> from nlsq import HybridStreamingConfig
>>> config = HybridStreamingConfig()
>>> config.warmup_iterations
200

Aggressive profile (faster convergence with L-BFGS):

>>> config = HybridStreamingConfig.aggressive()
>>> config.warmup_iterations
50

Conservative profile (higher quality):

>>> config = HybridStreamingConfig.conservative()
>>> config.gauss_newton_tol < 1e-8
True

Memory-optimized profile:

>>> config = HybridStreamingConfig.memory_optimized()
>>> config.chunk_size < 10000
True

Custom configuration:

>>> config = HybridStreamingConfig(
...     warmup_iterations=50,
...     lbfgs_history_size=15,
...     chunk_size=5000,
... )

With multi-start tournament selection:

>>> config = HybridStreamingConfig(
...     enable_multistart=True,
...     n_starts=20,
...     elimination_rounds=3,
...     batches_per_round=50,
... )

See also

AdaptiveHybridStreamingOptimizer: Optimizer that uses this configuration
curve_fit: High-level interface with method=’hybrid_streaming’
TournamentSelector: Tournament selection for multi-start optimization

Notes

Based on Adaptive Hybrid Streaming Optimizer specification: agent-os/specs/2025-12-18-adaptive-hybrid-streaming-optimizer/spec.md

L-BFGS replaces Adam for warmup, providing 5-10x faster convergence to the basin of attraction through approximate Hessian information.

normalize: bool

normalization_strategy: str

warmup_iterations: int

max_warmup_iterations: int

warmup_learning_rate: float

loss_plateau_threshold: float

gradient_norm_threshold: float

active_switching_criteria: list[str] | None

lbfgs_history_size: int

lbfgs_initial_step_size: float

lbfgs_line_search: Literal['wolfe', 'strong_wolfe', 'backtracking']

lbfgs_exploration_step_size: float

lbfgs_refinement_step_size: float

use_learning_rate_schedule: bool

lr_schedule_warmup_steps: int

lr_schedule_decay_steps: int

lr_schedule_end_value: float

gradient_clip_value: float | None

enable_warm_start_detection: bool

warm_start_threshold: float

enable_adaptive_warmup_lr: bool

warmup_lr_refinement: float

warmup_lr_careful: float

enable_cost_guard: bool

cost_increase_tolerance: float

enable_step_clipping: bool

max_warmup_step_size: float

gauss_newton_max_iterations: int

gauss_newton_tol: float

trust_region_initial: float

regularization_factor: float

cg_max_iterations: int

cg_relative_tolerance: float

cg_absolute_tolerance: float

cg_param_threshold: int

enable_group_variance_regularization: bool

group_variance_lambda: float

group_variance_indices: list[tuple[int, int]] | None

enable_residual_weighting: bool

residual_weights: list[float] | None

chunk_size: int

gc_chunk_interval: int

loop_strategy: Literal['auto', 'scan', 'loop']

enable_checkpoints: bool

checkpoint_frequency: int

checkpoint_dir: str | None

resume_from_checkpoint: str | None

validate_numerics: bool

enable_fault_tolerance: bool

max_retries_per_batch: int

min_success_rate: float

enable_multi_device: bool

callback_frequency: int

verbose: int

log_frequency: int

enable_multistart: bool

n_starts: int

multistart_sampler: Literal['lhs', 'sobol', 'halton']

elimination_rounds: int

elimination_fraction: float

batches_per_round: int

center_on_p0: bool

scale_factor: float

__post_init__()[source]

Validate configuration after initialization.

Delegates to specialized validator functions for each configuration group. ConfigValidationError from validators is re-raised as ValueError for backwards compatibility.

classmethod aggressive()[source]

Create aggressive profile: faster convergence with L-BFGS, looser tolerances.

This preset prioritizes speed over robustness: - L-BFGS warmup with reduced iterations (50 vs 300 with Adam) - Higher learning rate for faster progress - Looser tolerances for earlier Phase 2 switching - Larger chunks for better throughput

Returns:: Configuration with aggressive settings.
Return type:: HybridStreamingConfig

Examples

>>> config = HybridStreamingConfig.aggressive()
>>> config.warmup_learning_rate
0.003
>>> config.warmup_iterations
50

classmethod conservative()[source]

Create conservative profile: slower but robust, tighter tolerances.

This preset prioritizes solution quality over speed: - L-BFGS warmup with conservative iterations - Lower learning rate for stability - Tighter tolerances for higher quality - More Gauss-Newton iterations

Returns:: Configuration with conservative settings.
Return type:: HybridStreamingConfig

Examples

>>> config = HybridStreamingConfig.conservative()
>>> config.gauss_newton_tol
1e-10
>>> config.warmup_iterations
30

classmethod memory_optimized()[source]

Create memory-optimized profile: smaller chunks, efficient settings.

This preset minimizes memory footprint: - Smaller chunks to reduce memory usage - L-BFGS warmup with reduced iterations - Enable checkpoints for recovery (important when memory is tight) - Lower CG threshold for more aggressive CG usage (avoids O(p^2) J^T J)

Returns:: Configuration with memory-optimized settings.
Return type:: HybridStreamingConfig

Examples

>>> config = HybridStreamingConfig.memory_optimized()
>>> config.chunk_size
5000
>>> config.warmup_iterations
40
>>> config.cg_param_threshold
1000

classmethod with_multistart(n_starts=10, **kwargs)[source]

Create configuration with multi-start tournament selection enabled.

This preset enables multi-start optimization for finding global optima: - Tournament selection during Phase 1 warmup - LHS sampling for generating starting points - Progressive elimination to select best candidate

Parameters:

n_starts (int, default=10) – Number of starting points to generate.
**kwargs – Additional configuration parameters to override.

Returns:

Configuration with multi-start enabled.

Return type:

HybridStreamingConfig

Examples

>>> config = HybridStreamingConfig.with_multistart(n_starts=20)
>>> config.enable_multistart
True
>>> config.n_starts
20

classmethod defense_strict()[source]

Create strict defense layer profile for near-optimal scenarios.

This preset maximizes protection against divergence when initial parameters are expected to be close to optimal (warm starts, refinement): - Very low warm start threshold (triggers at 1% relative loss) - Ultra-conservative learning rates for refinement - Very tight cost guard tolerance (5% increase aborts) - Very small step clipping for stability

Use this when: - Continuing optimization from a previous fit - Refining parameters that are already close to optimal - Dealing with ill-conditioned problems - Prioritizing stability over speed

Returns:: Configuration with strict defense layer settings.
Return type:: HybridStreamingConfig

Examples

>>> config = HybridStreamingConfig.defense_strict()
>>> config.warm_start_threshold
0.01
>>> config.cost_increase_tolerance
0.05
>>> config.warmup_iterations
25

classmethod defense_relaxed()[source]

Create relaxed defense layer profile for exploration-heavy scenarios.

This preset reduces defense layer sensitivity for problems where significant parameter exploration is needed: - Higher warm start threshold (50% relative loss needed to skip) - More aggressive learning rates for exploration - Generous cost guard tolerance (50% increase allowed) - Larger step clipping for faster exploration

Use this when: - Starting from a rough initial guess - Exploring a wide parameter space - Problems with multiple local minima - Speed is more important than robustness

Returns:: Configuration with relaxed defense layer settings.
Return type:: HybridStreamingConfig

Examples

>>> config = HybridStreamingConfig.defense_relaxed()
>>> config.warm_start_threshold
0.5
>>> config.cost_increase_tolerance
0.5
>>> config.warmup_iterations
50

classmethod defense_disabled()[source]

Create profile with all defense layers disabled.

This preset completely disables the 4-layer defense strategy, reverting to pre-0.3.6 behavior. Use with caution as this removes protection against warmup divergence.

Use this when: - Debugging to isolate defense layer effects - Benchmarking without defense overhead - Backward compatibility with older code is required

Returns:: Configuration with all defense layers disabled.
Return type:: HybridStreamingConfig

Examples

>>> config = HybridStreamingConfig.defense_disabled()
>>> config.enable_warm_start_detection
False

classmethod scientific_default()[source]

Create profile optimized for scientific computing workflows.

This preset is tuned for scientific fitting scenarios like spectroscopy, decay curves, and other physics-based models: - Balanced defense layers that protect without being too aggressive - L-BFGS warmup with moderate iterations - Enabled checkpoints for long-running fits

Use this when: - Fitting physics-based models (spectroscopy, decay curves) - Numerical precision is important - Parameters may have multiple scales - Reproducibility is required

Returns:: Configuration optimized for scientific computing.
Return type:: HybridStreamingConfig

Examples

>>> config = HybridStreamingConfig.scientific_default()
>>> config.warmup_iterations
35

__init__(normalize=True, normalization_strategy='auto', warmup_iterations=200, max_warmup_iterations=500, warmup_learning_rate=0.001, loss_plateau_threshold=0.0001, gradient_norm_threshold=0.001, active_switching_criteria=None, lbfgs_history_size=10, lbfgs_initial_step_size=0.1, lbfgs_line_search='wolfe', lbfgs_exploration_step_size=0.1, lbfgs_refinement_step_size=1.0, use_learning_rate_schedule=False, lr_schedule_warmup_steps=50, lr_schedule_decay_steps=450, lr_schedule_end_value=0.0001, gradient_clip_value=None, enable_warm_start_detection=True, warm_start_threshold=0.01, enable_adaptive_warmup_lr=True, warmup_lr_refinement=1e-06, warmup_lr_careful=1e-05, enable_cost_guard=True, cost_increase_tolerance=0.05, enable_step_clipping=True, max_warmup_step_size=0.1, gauss_newton_max_iterations=100, gauss_newton_tol=1e-08, trust_region_initial=1.0, regularization_factor=1e-10, cg_max_iterations=100, cg_relative_tolerance=0.0001, cg_absolute_tolerance=1e-10, cg_param_threshold=2000, enable_group_variance_regularization=False, group_variance_lambda=0.01, group_variance_indices=None, enable_residual_weighting=False, residual_weights=None, chunk_size=10000, gc_chunk_interval=10, loop_strategy='auto', enable_checkpoints=True, checkpoint_frequency=100, checkpoint_dir=None, resume_from_checkpoint=None, validate_numerics=True, enable_fault_tolerance=True, max_retries_per_batch=2, min_success_rate=0.5, enable_multi_device=False, callback_frequency=10, verbose=1, log_frequency=1, enable_multistart=False, n_starts=10, multistart_sampler='lhs', elimination_rounds=3, elimination_fraction=0.5, batches_per_round=50, center_on_p0=True, scale_factor=1.0)

Configuration for the hybrid streaming optimizer.

Presets:

scientific_default(): High precision for scientific computing
memory_optimized(): Reduced memory footprint
aggressive(): Fast convergence, less conservative
conservative(): Maximum reliability
defense_strict(): Strict numerical guards

Example:

from nlsq import HybridStreamingConfig

# From preset
config = HybridStreamingConfig.scientific_default()

# Custom configuration
config = HybridStreamingConfig(
    warmup_iterations=50,
    gauss_newton_max_iterations=20,
    chunk_size=50_000,
    checkpoint_frequency=10,
)

Memory Estimation¶

nlsq.estimate_memory_requirements(n_points, n_params)[source]

Estimate memory requirements for a dataset.

Parameters:

n_points (int) – Number of data points
n_params (int) – Number of parameters

Returns:

Memory requirements and processing recommendations

Return type:

DatasetStats

Examples

>>> from nlsq.streaming.large_dataset import estimate_memory_requirements
>>>
>>> # Estimate requirements for 50M points, 3 parameters
>>> stats = estimate_memory_requirements(50_000_000, 3)
>>> print(f"Estimated memory: {stats.total_memory_estimate_gb:.2f} GB")
>>> print(f"Recommended chunk size: {stats.recommended_chunk_size:,}")
>>> print(f"Number of chunks: {stats.n_chunks}")

Estimate memory requirements before fitting:

from nlsq import estimate_memory_requirements

mem = estimate_memory_requirements(n_points=10_000_000, n_params=5, dtype="float64")

print(f"Estimated GPU memory: {mem['gpu_memory_gb']:.2f} GB")
print(f"Recommended chunk size: {mem['chunk_size']}")

Checkpointing¶

AdaptiveHybridStreamingOptimizer supports checkpointing for long-running fits:

from nlsq import AdaptiveHybridStreamingOptimizer, HybridStreamingConfig

config = HybridStreamingConfig(
    checkpoint_dir="./checkpoints",
    checkpoint_frequency=100,
)
optimizer = AdaptiveHybridStreamingOptimizer(config)

# If interrupted, resume from checkpoint
result = optimizer.fit((x, y), model, p0=p0, verbose=1)

Parallel Processing¶

For multi-GPU systems:

from nlsq import ParallelFitter

fitter = ParallelFitter(n_gpus=4, memory_per_gpu_gb=16.0)

result = fitter.fit(model, x, y, p0=p0)

Large Dataset API Reference¶

Overview¶

curve_fit_large¶

LargeDatasetFitter¶

Core Capabilities¶

Memory Management Algorithm¶

Processing Strategies¶

Multi-Start Optimization¶

Performance Characteristics¶

Model Validation Caching (Task Group 7 - 5.1a)¶

AdaptiveHybridStreamingOptimizer¶

HybridStreamingConfig¶

Memory Estimation¶

Checkpointing¶

Parallel Processing¶

See Also¶