2.3. TRF Optimizer

The Trust Region Reflective (TRF) algorithm is NLSQ’s core optimizer.

2.3.1. Algorithm Overview

TRF solves bounded nonlinear least squares problems:

minimize:  0.5 * ||f(x)||²
subject to: lb ≤ x ≤ ub

The algorithm:

  1. Trust Region: Approximate objective in a local region

  2. Reflective: Handle bounds via reflection at boundaries

  3. Adaptive: Adjust trust radius based on progress

2.3.2. Key Components

Located in nlsq/core/trf.py (2544 lines):

from nlsq.core.trf import TrustRegionReflective

trf = TrustRegionReflective(
    fun=residual_func,
    x0=initial_params,
    lb=lower_bounds,
    ub=upper_bounds,
    f_scale=1.0,
    ftol=1e-8,
    xtol=1e-8,
    gtol=1e-8,
    max_nfev=100,
    tr_solver="exact",
    tr_options={},
)

result = trf.solve()

2.3.3. Iteration Steps

Each iteration performs:

  1. Gradient computation: g = J^T @ f

  2. Scaling: Apply parameter scaling D

  3. Subproblem: Solve trust region subproblem

  4. Step computation: Find step direction p

  5. Reflection: Handle bound violations

  6. Ratio evaluation: ratio = actual / predicted

  7. Trust update: Adjust trust radius

ratio > 0.75  →  expand trust region (×2)
ratio > 0.25  →  keep trust region
ratio < 0.25  →  contract trust region (×0.25)

2.3.4. Trust Region Solvers

Exact (SVD-based):

# For small/medium problems
# Solves: min ||J*p + f||² s.t. ||D*p|| ≤ Δ

tr_solver = "exact"  # Uses SVD decomposition

LSMR (iterative):

# For large problems where SVD is expensive
tr_solver = "lsmr"
tr_options = {"maxiter": 100, "atol": 1e-10, "btol": 1e-10}

2.3.5. JIT-Compiled Helpers

Located in nlsq/core/trf_jit.py:

from nlsq.core.trf_jit import (
    compute_gradient_jit,
    solve_lsq_trust_region_jit,
    minimize_quadratic_1d_jit,
)

These functions are JIT-compiled for GPU acceleration.

2.3.6. Profiling

Use TRFProfiler for timing:

from nlsq.core.profiler import TRFProfiler

profiler = TRFProfiler()

# Pass to optimizer
result = optimizer.least_squares(
    fun=residuals, x0=x0, _profiler=profiler  # Internal option
)

# Get timing breakdown
profiler.print_summary()

2.3.7. Convergence Criteria

Optimization stops when any criterion is met:

# Function tolerance (relative cost reduction)
# |Δcost| / cost < ftol
ftol = 1e-8

# Parameter tolerance (relative step size)
# ||Δx|| / ||x|| < xtol
xtol = 1e-8

# Gradient tolerance (gradient norm)
# ||g||_inf < gtol
gtol = 1e-8

# Maximum evaluations
max_nfev = 100 * n_params

2.3.8. Status Codes

Status

Meaning

Action

1

ftol satisfied

Success

2

xtol satisfied

Success

3

gtol satisfied

Success

0

max_nfev reached

Increase max_nfev

-1

Improper input

Check parameters

2.3.9. Algorithm Tuning

For fast convergence:

# Start near solution
p0 = good_initial_guess

# Looser tolerances
ftol = 1e-6
xtol = 1e-6
gtol = 1e-6

For high precision:

# Tight tolerances
ftol = 1e-12
xtol = 1e-12
gtol = 1e-12

# More iterations
max_nfev = 1000

For ill-conditioned problems:

# Parameter scaling
x_scale = "jac"  # or provide manual scaling

# LSMR for stability
tr_solver = "lsmr"

2.3.10. Next Steps