4.2. DataPreprocessor

Added in version 0.6.4.

The DataPreprocessor handles input validation, type conversion, and data cleaning before optimization.

4.2.1. Basic Usage

from nlsq.core.orchestration import DataPreprocessor
import jax.numpy as jnp


def model(x, a, b):
    return a * jnp.exp(-b * x)


preprocessor = DataPreprocessor()
preprocessed = preprocessor.preprocess(
    f=model, xdata=x, ydata=y, sigma=None, check_finite=True
)

# Access preprocessed data
x_clean = preprocessed.xdata
y_clean = preprocessed.ydata
n_points = preprocessed.n_points

4.2.2. PreprocessedData

The preprocess() method returns a PreprocessedData object:

@dataclass
class PreprocessedData:
    xdata: jnp.ndarray  # Preprocessed x data
    ydata: jnp.ndarray  # Preprocessed y data
    sigma: jnp.ndarray | None  # Uncertainties
    n_points: int  # Number of data points
    is_padded: bool  # Was data padded?
    has_nans_removed: bool  # Were NaNs removed?
    original_shape: tuple  # Original data shape

4.2.3. preprocess() Parameters

preprocessed = preprocessor.preprocess(
    f,  # Model function
    xdata,  # Independent variable
    ydata,  # Dependent variable
    sigma=None,  # Measurement uncertainties
    absolute_sigma=False,  # Interpret sigma as absolute
    check_finite=True,  # Check for inf/NaN
    nan_policy="raise",  # 'raise', 'omit', 'propagate'
    stability_check=False,  # Run stability checks
    **kwargs
)

4.2.4. NaN Handling

nan_policy=’raise’ (default):

# Raises ValueError if NaN found
preprocessor.preprocess(f, x_with_nan, y, nan_policy="raise")

nan_policy=’omit’:

# Removes NaN values
preprocessed = preprocessor.preprocess(f, x_with_nan, y, nan_policy="omit")
print(f"Points after NaN removal: {preprocessed.n_points}")
print(f"NaNs removed: {preprocessed.has_nans_removed}")

nan_policy=’propagate’:

# Passes NaN through (may cause fitting issues)
preprocessed = preprocessor.preprocess(f, x_with_nan, y, nan_policy="propagate")

4.2.5. Sigma Validation

# Validate sigma has correct shape
preprocessor.validate_sigma(sigma, ydata.shape)

# With absolute_sigma
preprocessed = preprocessor.preprocess(
    f, x, y, sigma=measurement_errors, absolute_sigma=True
)

4.2.6. Type Conversion

Input types are converted to JAX arrays:

import numpy as np

# NumPy arrays → JAX arrays
x_np = np.array([1, 2, 3])
preprocessed = preprocessor.preprocess(f, x_np, y_np)
assert isinstance(preprocessed.xdata, jnp.ndarray)

# Python lists → JAX arrays
preprocessed = preprocessor.preprocess(f, [1, 2, 3], [1.0, 0.5, 0.3])

4.2.7. Complete Example

import numpy as np
import jax.numpy as jnp
from nlsq.core.orchestration import DataPreprocessor


def model(x, a, b, c):
    return a * jnp.exp(-b * x) + c


# Create data with some NaN values
np.random.seed(42)
x = np.linspace(0, 10, 100)
y = 2.5 * np.exp(-0.5 * x) + 0.3 + 0.1 * np.random.randn(100)
y[5] = np.nan  # Add NaN
y[50] = np.nan

sigma = 0.1 * np.ones(100)

# Preprocess
preprocessor = DataPreprocessor()
preprocessed = preprocessor.preprocess(
    f=model, xdata=x, ydata=y, sigma=sigma, nan_policy="omit", check_finite=True
)

print(f"Original points: 100")
print(f"After preprocessing: {preprocessed.n_points}")
print(f"NaNs removed: {preprocessed.has_nans_removed}")
print(f"Data type: {type(preprocessed.xdata)}")

4.2.8. Next Steps