4.4. Large Datasets

NLSQ automatically handles datasets from hundreds to 100M+ points using memory-aware strategies. You don’t need to change your code.

4.4.1. Automatic Handling

NLSQ automatically selects the best strategy:

from nlsq import fit

# Same code works for any size
popt, pcov = fit(model, x, y, p0=[...])

# 100 points: STANDARD (in-memory)
# 1M points: CHUNKED (Jacobian chunking)
# 100M points: STREAMING (batch processing)

4.4.2. Memory Strategy Selection

The system analyzes data size vs available memory:

Strategy

When Used

Characteristics

STANDARD

Data + Jacobian fit in memory

Fastest, full in-memory processing

CHUNKED

Data fits, Jacobian doesn’t

Jacobian computed in chunks

STREAMING

Data exceeds memory

Adaptive batch processing

4.4.3. Monitoring Memory Usage

Check what strategy NLSQ selects:

from nlsq import fit
from nlsq.core.workflow import MemoryBudgetSelector

# Check memory strategy before fitting
selector = MemoryBudgetSelector()
strategy, config = selector.select(
    n_points=len(x),
    n_params=3,  # Number of model parameters
)
print(f"Strategy: {strategy}")

# Fit
popt, pcov = fit(model, x, y, p0=[...])

4.4.4. Memory Override

Force a specific memory limit:

# Force chunked/streaming by limiting memory
popt, pcov = fit(model, x, y, p0=[...], memory_limit_gb=4.0)

# Useful when:
# - Running alongside other processes
# - On shared systems
# - Testing streaming behavior

4.4.5. Large Dataset Tips

1. Use the streaming optimizer for very large data:

from nlsq import curve_fit_large

popt, pcov = curve_fit_large(model, x, y, p0=p0)
# Memory-efficient chunked processing

2. Start with a subset:

# Quick test on subset
n_sample = 10000
idx = np.random.choice(len(x), n_sample, replace=False)
popt_test, _ = fit(model, x[idx], y[idx], p0=[...])

# Full fit with good initial guess
popt, pcov = fit(model, x, y, p0=popt_test)

3. Use bounds:

# Bounds help convergence with large data
popt, pcov = fit(model, x, y, p0=[...], bounds=([0, 0], [100, 10]))

4. Adjust tolerances:

# Looser tolerances for faster convergence
popt, pcov = fit(model, x, y, p0=[...], ftol=1e-6, xtol=1e-6, gtol=1e-6)

4.4.6. Performance Benchmarks

Typical performance on modern hardware (8-core CPU, 32GB RAM):

Points

Strategy

First Fit

Subsequent

Memory

10K

STANDARD

~1s

~0.1s

~100MB

100K

STANDARD

~2s

~0.3s

~1GB

1M

CHUNKED

~10s

~2s

~2GB

10M

STREAMING

~60s

~20s

~4GB

100M

STREAMING

~10min

~3min

~4GB

First fit includes JIT compilation; subsequent fits are faster.

4.4.7. GPU Acceleration

GPUs provide significant speedup for large datasets:

# Install JAX with CUDA
# pip install jax[cuda12_pip]

from nlsq import fit

# Automatically uses GPU if available
popt, pcov = fit(model, x, y, p0=[...])

GPU speedup is most significant for 100K+ points.

4.4.8. Complete Example

import numpy as np
import jax.numpy as jnp
from nlsq import fit

# Generate large dataset
np.random.seed(42)
n_points = 1_000_000  # 1 million points

x = np.linspace(0, 100, n_points)
y_true = 2.5 * np.exp(-0.05 * x) + 0.3
y = y_true + 0.2 * np.random.randn(n_points)


# Model
def exponential(x, A, k, c):
    return A * jnp.exp(-k * x) + c


# Fit - NLSQ auto-selects appropriate strategy
print(f"Fitting {n_points:,} points...")
popt, pcov = fit(exponential, x, y, p0=[2, 0.05, 0])

# Results
A, k, c = popt
perr = np.sqrt(np.diag(pcov))

print(f"\nResults:")
print(f"  A = {A:.4f} +/- {perr[0]:.4f}")
print(f"  k = {k:.5f} +/- {perr[1]:.5f}")
print(f"  c = {c:.4f} +/- {perr[2]:.4f}")

4.4.9. Troubleshooting Large Data

Out of memory:

  • Reduce memory_limit_gb

  • Use streaming optimizer

  • Close other applications

Very slow:

  • Use GPU acceleration

  • Loosen tolerances

  • Check model complexity

Poor convergence:

  • Improve initial guess (test on subset first)

  • Add bounds

  • Check data quality

4.4.10. Next Steps