LINFA Tutorial 1 - Two dimensions

This tutorial will guide you through using the functionality of LINFA to set up inference tasks involving physics-based models.

What is LINFA? LINFA is a library for variational inference with normalizing flow and adaptive annealing. LINFA accommodates computationally expensive models and difficult-to-sample posterior distributions with dependent parameters.
Why should I use LINFA? Designed as a general inference engine, LINFA allows the user to define custom input transformations, computational models, surrogates, and likelihood functions which will be discussed throughout the tutorial.

Tutorial outline

In this tutorial we will:

Define a physics-based model for a simple ballistic application.
Perform the following inference tasks:
- Variational inference with the original model.
- Variational inference with a neural network surrogate.

After going through this tutorial, users should be able to define and integrate their model with LINFA, and use its features to perform inference.

Additional Resources

Background theory and examples for LINFA

1. Wang, E.R. Cobian, J. Lee, F. Liu, J.D. Hauenstein, D.E. Schiavazzi, LINFA: a Python library for variational inference with normalizing flow and annealing
1. Wang, F. Liu and D.E. Schiavazzi, Variational Inference with NoFAS: Normalizing Flow with Adaptive Surrogate for Computationally Expensive Models
E.R. Cobian, J.D. Hauenstein, F. Liu and D.E. Schiavazzi, AdaAnn: Adaptive Annealing Scheduler for Probability Density Approximation

More about LINFA library:

LINFA library documentation.
LINFA GitHub repository

[3]:

## Import libraries ##
import os
from linfa.run_experiment import experiment
from linfa.transform import Transformation
from linfa.nofas import Surrogate
import torch
import random
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import cv2

Problem definition

Our physics-based model phys consists of a simple ballistic model. We would like to compute the quantities:
- \(x_{1}\): final location (m)
- \(x_{2}\): time of flight (s)
from the inputs:
- \(z_{1}\): starting position (m)
- \(z_{2}\): initial velocity (m/s)

The model is described by the following equations

\[x_{1} = z_{1} + \frac{z_{2}^{2}\,\sin(2\cdot 60)}{g},\,\, x_{2} = \frac{2\,z_{2}\,\sin(60)}{g}.\]

This corresponds to the situation where a target location is reached from an object launched at an horizontal angle of 60 degrees.

Model identifiability

This model is identifiable.

Implementation as a Python class

We now create a new Phys model class, having three member functions:
- __init__ - A constructor.
- genDataFile - A member function to create synthetic observations.
- solve_t - A function to perform forward model evaluations.

Please refer to the comments below for additional implementation details.

[4]:

#### Implementation of the traditional trajectory motion physics problem ####
class Phys:

    ### Define constructor function for Phys class ###
    def __init__(self):
        ## Define input parameters (True value)
        # input[] = [starting_position, initial_velocity, angle] = [1(m), 5(m/s)]
        self.defParam = torch.Tensor([[1.0, 5.0]])

        self.gConst = 9.81   # gravitational constant
        self.stdRatio = 0.05 # standard deviation ratio
        self.data = None     # data set of model sample

    ### Define data file generator function ###
    # dataSize (int): size of sample (data)
    # dataFileName (String): name of the sample data file
    # store (Boolean): True if user wish to store the generated data file; False otherwise.
    def genDataFile(self, dataSize = 50, dataFileName="data_phys_2d.txt", store=True):
        def_out = self.solve_t(self.defParam)[0]
        self.data = def_out + self.stdRatio * torch.abs(def_out) * torch.normal(0, 1, size=(dataSize, 2))
        self.data = self.data.t().detach().numpy()
        if store: np.savetxt(dataFileName, self.data)
        return self.data

    ### Define data file generator function ###
    # params (Tensor): input parameters storing starting position, initial velocity, and angle in corresponding order.
    def solve_t(self, params):
        z1, z2 = torch.chunk(params, chunks=2, dim=1) # input parameters
        z3 = torch.Tensor([[60.0]])
        z3 = z3 * (np.pi / 180)                           # convert unit from degree to radians

        ## Output value calculation
        # ouput[] = [final_location, total_time]
        x = torch.cat(( z1 + ((z2 * z2 * torch.sin(2.0 * z3)) / self.gConst),             # x2: finalLocation
            (2.0 * z2 * torch.sin(z3)) / self.gConst), 1)                                 # x3: totalTime
        return x

[5]:

## Generate phys sample file ##

# Define model
model = Phys()

# Generate Data
physData = model.genDataFile()

Now that we have our model set up, we go on to our second step, i.e., inference with the full model.

Variational inference with full model

Definition of hyperparameters

The first step is to define all options and hyperparameters for the inference task.

Additional detail for each hyperparameter can be found in the documentation or in the definition of the experiment class.

[6]:

# Experiment Setting
exp = experiment()
exp.flow_type        = 'maf'        # str: Type of flow
exp.n_blocks         = 5            # int: Number of layers
exp.hidden_size      = 100          # int: Hidden layer size for MADE in each layer
exp.n_hidden         = 1            # int: Number of hidden layers in each MADE
exp.activation_fn    = 'relu'       # str: Actication function used
exp.input_order      = 'sequential' # str: Input order for create_mask
exp.batch_norm_order = True         # boolean: Order to decide if batch_norm is used
exp.save_interval    = 5000         # int: How often to sample from normalizing flow

exp.input_size    = 2               # int: Dimensionality of input
exp.batch_size    = 200             # int: Number of samples generated
exp.true_data_num = 2               # double: number of true model evaluated
exp.n_iter        = 25001           # int: Number of iterations
exp.lr            = 0.01            # float: Learning rate
exp.lr_decay      = 0.9999          # float: Learning rate decay
exp.log_interval  = 100             # int: How often to show loss stat

exp.run_nofas          = False      # boolean: to run experiment with nofas
exp.annealing          = False      # boolean: to run experiment with annealing
exp.calibrate_interval = 1000       # int: How often to update surrogate model
exp.budget             = 260        # int: Total number of true model evaluation

exp.surr_pre_it  = 30000            # int: Number of pre-training iterations for surrogate model
exp.surr_upd_it  = 6000             # int: Number of iterations for the surrogate model update
exp.surr_folder  = "./"
exp.use_new_surr = True             # boolean: to run experiment with nofas

exp.results_file = 'results.txt'      # str: result text file name
exp.log_file     = 'log.txt'          # str: log text file name
exp.samples_file = 'samples.txt'      # str: sample text file name
exp.seed         = random.randint(0, 10 ** 9)  # int: Random seed used
exp.n_sample     = 5000               # int: Total number of iterations
exp.no_cuda      = True               # boolean: to run experiment with NO cuda

exp.optimizer    = 'RMSprop'          # str: Type of optimizer
exp.lr_scheduler = 'ExponentialLR'    # str: Type of scheduler

exp.device = torch.device('cuda:0' if torch.cuda.is_available() and not exp.no_cuda else 'cpu')

Define the transformation

Now we define the trasformation of parameters and initialize the exp.transform variable

[7]:

# Define transformation based on normalization rate
trsf_info = [['identity',0.0,0.0,0.0,0.0],
             ['identity',0.0,0.0,0.0,0.0]]
trsf = Transformation(trsf_info)
exp.transform = trsf

Model and surrogate definition

We create an instance of the Phys model and assign None to the surrogate.

[8]:

# Define model
model = Phys()
exp.model = model

# Get data
model.data = np.loadtxt('./data_phys_2d.txt')

# Run experiment without surrogate
exp.surrogate = None

Log-likelihood definiton

[12]:

## Define log density
# x: original, untransformed inputs
# model: our model
# transform: our transformation
def log_density(x, model, surrogate, transform):
    # x contains the original, untransformed inputs
    np.savetxt(exp.output_dir + '/' + exp.name + '_x', x.detach().numpy(), newline="\n")
    # Compute transformation log Jacobian
    adjust = transform.compute_log_jacob_func(x)

    # Get the absolute values of the standard deviations
    stds = torch.abs(model.solve_t(model.defParam)) * model.stdRatio
    Data = torch.tensor(model.data).to(exp.device)

    # Check for surrogate
    if surrogate:
        modelOut = exp.surrogate.forward(x)
    else:
        modelOut = model.solve_t(transform.forward(x))

    # Eval LL
    ll1 = -0.5 * np.prod(model.data.shape) * np.log(2.0 * np.pi)
    ll2 = (-0.5 * model.data.shape[1] * torch.log(torch.prod(stds))).item()
    ll3 = 0.0

    for i in range(2):
        ll3 += - 0.5 * torch.sum(((modelOut[:, i].unsqueeze(1) - Data[i, :].unsqueeze(0)) / stds[0, i]) ** 2, dim=1)

    negLL = -(ll1 + ll2 + ll3)

    # Return LL
    return -negLL.reshape(x.size(0), 1) + adjust

Launch inference task

[13]:

## Run
print('')
print('--- Temporary TEST: Physics Example - without NOFAS')
print('')

# Experiment Setting
exp.name = "phys_full_2d"        # str: Name of experiment
exp.output_dir   = './' + exp.name    # str: output directory location

# Assign logdensity
exp.model_logdensity = lambda x: log_density(x, model, exp.surrogate, trsf)

# Run VI
exp.run()


--- Temporary TEST: Physics Example - without NOFAS


--- Running on device: cpu

VI NF (t=1.000): it:     100 | loss: 5.415e+03
VI NF (t=1.000): it:     200 | loss: 2.116e+03
VI NF (t=1.000): it:     300 | loss: 1.017e+03
VI NF (t=1.000): it:     400 | loss: 5.071e+02
VI NF (t=1.000): it:     500 | loss: 2.040e+02
VI NF (t=1.000): it:     600 | loss: 7.246e+01
VI NF (t=1.000): it:     700 | loss: 3.559e+01
VI NF (t=1.000): it:     800 | loss: 3.411e+01
VI NF (t=1.000): it:     900 | loss: 3.163e+01
VI NF (t=1.000): it:    1000 | loss: 3.339e+01
VI NF (t=1.000): it:    1100 | loss: 3.128e+01
VI NF (t=1.000): it:    1200 | loss: 3.222e+01
VI NF (t=1.000): it:    1300 | loss: 3.135e+01
VI NF (t=1.000): it:    1400 | loss: 3.143e+01
VI NF (t=1.000): it:    1500 | loss: 3.133e+01
VI NF (t=1.000): it:    1600 | loss: 3.121e+01
VI NF (t=1.000): it:    1700 | loss: 3.128e+01
VI NF (t=1.000): it:    1800 | loss: 3.112e+01
VI NF (t=1.000): it:    1900 | loss: 3.153e+01
VI NF (t=1.000): it:    2000 | loss: 3.137e+01
VI NF (t=1.000): it:    2100 | loss: 3.141e+01
VI NF (t=1.000): it:    2200 | loss: 3.136e+01
VI NF (t=1.000): it:    2300 | loss: 3.148e+01
VI NF (t=1.000): it:    2400 | loss: 3.130e+01
VI NF (t=1.000): it:    2500 | loss: 3.131e+01
VI NF (t=1.000): it:    2600 | loss: 3.141e+01
VI NF (t=1.000): it:    2700 | loss: 3.132e+01
VI NF (t=1.000): it:    2800 | loss: 3.142e+01
VI NF (t=1.000): it:    2900 | loss: 3.119e+01
VI NF (t=1.000): it:    3000 | loss: 3.118e+01
VI NF (t=1.000): it:    3100 | loss: 3.131e+01
VI NF (t=1.000): it:    3200 | loss: 3.121e+01
VI NF (t=1.000): it:    3300 | loss: 3.122e+01
VI NF (t=1.000): it:    3400 | loss: 3.125e+01
VI NF (t=1.000): it:    3500 | loss: 3.111e+01
VI NF (t=1.000): it:    3600 | loss: 3.105e+01
VI NF (t=1.000): it:    3700 | loss: 3.120e+01
VI NF (t=1.000): it:    3800 | loss: 3.125e+01
VI NF (t=1.000): it:    3900 | loss: 3.113e+01
VI NF (t=1.000): it:    4000 | loss: 3.120e+01
VI NF (t=1.000): it:    4100 | loss: 3.124e+01
VI NF (t=1.000): it:    4200 | loss: 3.104e+01
VI NF (t=1.000): it:    4300 | loss: 3.123e+01
VI NF (t=1.000): it:    4400 | loss: 3.111e+01
VI NF (t=1.000): it:    4500 | loss: 3.117e+01
VI NF (t=1.000): it:    4600 | loss: 3.127e+01
VI NF (t=1.000): it:    4700 | loss: 3.134e+01
VI NF (t=1.000): it:    4800 | loss: 3.124e+01
VI NF (t=1.000): it:    4900 | loss: 3.113e+01
--- Saving results at iteration 5000
VI NF (t=1.000): it:    5000 | loss: 3.107e+01
VI NF (t=1.000): it:    5100 | loss: 3.107e+01
VI NF (t=1.000): it:    5200 | loss: 3.109e+01
VI NF (t=1.000): it:    5300 | loss: 3.121e+01
VI NF (t=1.000): it:    5400 | loss: 3.130e+01
VI NF (t=1.000): it:    5500 | loss: 3.120e+01
VI NF (t=1.000): it:    5600 | loss: 3.117e+01
VI NF (t=1.000): it:    5700 | loss: 3.114e+01
VI NF (t=1.000): it:    5800 | loss: 3.107e+01
VI NF (t=1.000): it:    5900 | loss: 3.097e+01
VI NF (t=1.000): it:    6000 | loss: 3.108e+01
VI NF (t=1.000): it:    6100 | loss: 3.111e+01
VI NF (t=1.000): it:    6200 | loss: 3.097e+01
VI NF (t=1.000): it:    6300 | loss: 3.113e+01
VI NF (t=1.000): it:    6400 | loss: 3.101e+01
VI NF (t=1.000): it:    6500 | loss: 3.109e+01
VI NF (t=1.000): it:    6600 | loss: 3.098e+01
VI NF (t=1.000): it:    6700 | loss: 3.112e+01
VI NF (t=1.000): it:    6800 | loss: 3.113e+01
VI NF (t=1.000): it:    6900 | loss: 3.124e+01
VI NF (t=1.000): it:    7000 | loss: 3.116e+01
VI NF (t=1.000): it:    7100 | loss: 3.118e+01
VI NF (t=1.000): it:    7200 | loss: 3.121e+01
VI NF (t=1.000): it:    7300 | loss: 3.121e+01
VI NF (t=1.000): it:    7400 | loss: 3.112e+01
VI NF (t=1.000): it:    7500 | loss: 3.116e+01
VI NF (t=1.000): it:    7600 | loss: 3.113e+01
VI NF (t=1.000): it:    7700 | loss: 3.107e+01
VI NF (t=1.000): it:    7800 | loss: 3.111e+01
VI NF (t=1.000): it:    7900 | loss: 3.103e+01
VI NF (t=1.000): it:    8000 | loss: 3.116e+01
VI NF (t=1.000): it:    8100 | loss: 3.101e+01
VI NF (t=1.000): it:    8200 | loss: 3.109e+01
VI NF (t=1.000): it:    8300 | loss: 3.121e+01
VI NF (t=1.000): it:    8400 | loss: 3.109e+01
VI NF (t=1.000): it:    8500 | loss: 3.103e+01
VI NF (t=1.000): it:    8600 | loss: 3.123e+01
VI NF (t=1.000): it:    8700 | loss: 3.107e+01
VI NF (t=1.000): it:    8800 | loss: 3.113e+01
VI NF (t=1.000): it:    8900 | loss: 3.105e+01
VI NF (t=1.000): it:    9000 | loss: 3.115e+01
VI NF (t=1.000): it:    9100 | loss: 3.116e+01
VI NF (t=1.000): it:    9200 | loss: 3.123e+01
VI NF (t=1.000): it:    9300 | loss: 3.113e+01
VI NF (t=1.000): it:    9400 | loss: 3.103e+01
VI NF (t=1.000): it:    9500 | loss: 3.119e+01
VI NF (t=1.000): it:    9600 | loss: 3.121e+01
VI NF (t=1.000): it:    9700 | loss: 3.113e+01
VI NF (t=1.000): it:    9800 | loss: 3.117e+01
VI NF (t=1.000): it:    9900 | loss: 3.117e+01
--- Saving results at iteration 10000
VI NF (t=1.000): it:   10000 | loss: 3.114e+01
VI NF (t=1.000): it:   10100 | loss: 3.117e+01
VI NF (t=1.000): it:   10200 | loss: 3.103e+01
VI NF (t=1.000): it:   10300 | loss: 3.114e+01
VI NF (t=1.000): it:   10400 | loss: 3.101e+01
VI NF (t=1.000): it:   10500 | loss: 3.106e+01
VI NF (t=1.000): it:   10600 | loss: 3.097e+01
VI NF (t=1.000): it:   10700 | loss: 3.132e+01
VI NF (t=1.000): it:   10800 | loss: 3.123e+01
VI NF (t=1.000): it:   10900 | loss: 3.098e+01
VI NF (t=1.000): it:   11000 | loss: 3.125e+01
VI NF (t=1.000): it:   11100 | loss: 3.115e+01
VI NF (t=1.000): it:   11200 | loss: 3.122e+01
VI NF (t=1.000): it:   11300 | loss: 3.098e+01
VI NF (t=1.000): it:   11400 | loss: 3.112e+01
VI NF (t=1.000): it:   11500 | loss: 3.116e+01
VI NF (t=1.000): it:   11600 | loss: 3.100e+01
VI NF (t=1.000): it:   11700 | loss: 3.115e+01
VI NF (t=1.000): it:   11800 | loss: 3.114e+01
VI NF (t=1.000): it:   11900 | loss: 3.100e+01
VI NF (t=1.000): it:   12000 | loss: 3.113e+01
VI NF (t=1.000): it:   12100 | loss: 3.113e+01
VI NF (t=1.000): it:   12200 | loss: 3.111e+01
VI NF (t=1.000): it:   12300 | loss: 3.116e+01
VI NF (t=1.000): it:   12400 | loss: 3.114e+01
VI NF (t=1.000): it:   12500 | loss: 3.112e+01
VI NF (t=1.000): it:   12600 | loss: 3.109e+01
VI NF (t=1.000): it:   12700 | loss: 3.108e+01
VI NF (t=1.000): it:   12800 | loss: 3.106e+01
VI NF (t=1.000): it:   12900 | loss: 3.119e+01
VI NF (t=1.000): it:   13000 | loss: 3.102e+01
VI NF (t=1.000): it:   13100 | loss: 3.114e+01
VI NF (t=1.000): it:   13200 | loss: 3.111e+01
VI NF (t=1.000): it:   13300 | loss: 3.104e+01
VI NF (t=1.000): it:   13400 | loss: 3.095e+01
VI NF (t=1.000): it:   13500 | loss: 3.102e+01
VI NF (t=1.000): it:   13600 | loss: 3.100e+01
VI NF (t=1.000): it:   13700 | loss: 3.112e+01
VI NF (t=1.000): it:   13800 | loss: 3.104e+01
VI NF (t=1.000): it:   13900 | loss: 3.107e+01
VI NF (t=1.000): it:   14000 | loss: 3.110e+01
VI NF (t=1.000): it:   14100 | loss: 3.107e+01
VI NF (t=1.000): it:   14200 | loss: 3.111e+01
VI NF (t=1.000): it:   14300 | loss: 3.120e+01
VI NF (t=1.000): it:   14400 | loss: 3.109e+01
VI NF (t=1.000): it:   14500 | loss: 3.115e+01
VI NF (t=1.000): it:   14600 | loss: 3.103e+01
VI NF (t=1.000): it:   14700 | loss: 3.117e+01
VI NF (t=1.000): it:   14800 | loss: 3.103e+01
VI NF (t=1.000): it:   14900 | loss: 3.112e+01
--- Saving results at iteration 15000
VI NF (t=1.000): it:   15000 | loss: 3.108e+01
VI NF (t=1.000): it:   15100 | loss: 3.133e+01
VI NF (t=1.000): it:   15200 | loss: 3.118e+01
VI NF (t=1.000): it:   15300 | loss: 3.104e+01
VI NF (t=1.000): it:   15400 | loss: 3.105e+01
VI NF (t=1.000): it:   15500 | loss: 3.102e+01
VI NF (t=1.000): it:   15600 | loss: 3.105e+01
VI NF (t=1.000): it:   15700 | loss: 3.120e+01
VI NF (t=1.000): it:   15800 | loss: 3.106e+01
VI NF (t=1.000): it:   15900 | loss: 3.110e+01
VI NF (t=1.000): it:   16000 | loss: 3.118e+01
VI NF (t=1.000): it:   16100 | loss: 3.108e+01
VI NF (t=1.000): it:   16200 | loss: 3.101e+01
VI NF (t=1.000): it:   16300 | loss: 3.103e+01
VI NF (t=1.000): it:   16400 | loss: 3.101e+01
VI NF (t=1.000): it:   16500 | loss: 3.103e+01
VI NF (t=1.000): it:   16600 | loss: 3.108e+01
VI NF (t=1.000): it:   16700 | loss: 3.105e+01
VI NF (t=1.000): it:   16800 | loss: 3.105e+01
VI NF (t=1.000): it:   16900 | loss: 3.126e+01
VI NF (t=1.000): it:   17000 | loss: 3.111e+01
VI NF (t=1.000): it:   17100 | loss: 3.108e+01
VI NF (t=1.000): it:   17200 | loss: 3.106e+01
VI NF (t=1.000): it:   17300 | loss: 3.109e+01
VI NF (t=1.000): it:   17400 | loss: 3.111e+01
VI NF (t=1.000): it:   17500 | loss: 3.113e+01
VI NF (t=1.000): it:   17600 | loss: 3.108e+01
VI NF (t=1.000): it:   17700 | loss: 3.115e+01
VI NF (t=1.000): it:   17800 | loss: 3.112e+01
VI NF (t=1.000): it:   17900 | loss: 3.101e+01
VI NF (t=1.000): it:   18000 | loss: 3.104e+01
VI NF (t=1.000): it:   18100 | loss: 3.121e+01
VI NF (t=1.000): it:   18200 | loss: 3.109e+01
VI NF (t=1.000): it:   18300 | loss: 3.101e+01
VI NF (t=1.000): it:   18400 | loss: 3.115e+01
VI NF (t=1.000): it:   18500 | loss: 3.111e+01
VI NF (t=1.000): it:   18600 | loss: 3.115e+01
VI NF (t=1.000): it:   18700 | loss: 3.108e+01
VI NF (t=1.000): it:   18800 | loss: 3.105e+01
VI NF (t=1.000): it:   18900 | loss: 3.109e+01
VI NF (t=1.000): it:   19000 | loss: 3.099e+01
VI NF (t=1.000): it:   19100 | loss: 3.115e+01
VI NF (t=1.000): it:   19200 | loss: 3.105e+01
VI NF (t=1.000): it:   19300 | loss: 3.112e+01
VI NF (t=1.000): it:   19400 | loss: 3.111e+01
VI NF (t=1.000): it:   19500 | loss: 3.105e+01
VI NF (t=1.000): it:   19600 | loss: 3.102e+01
VI NF (t=1.000): it:   19700 | loss: 3.096e+01
VI NF (t=1.000): it:   19800 | loss: 3.107e+01
VI NF (t=1.000): it:   19900 | loss: 3.126e+01
--- Saving results at iteration 20000
VI NF (t=1.000): it:   20000 | loss: 3.125e+01
VI NF (t=1.000): it:   20100 | loss: 3.109e+01
VI NF (t=1.000): it:   20200 | loss: 3.099e+01
VI NF (t=1.000): it:   20300 | loss: 3.115e+01
VI NF (t=1.000): it:   20400 | loss: 3.098e+01
VI NF (t=1.000): it:   20500 | loss: 3.109e+01
VI NF (t=1.000): it:   20600 | loss: 3.102e+01
VI NF (t=1.000): it:   20700 | loss: 3.108e+01
VI NF (t=1.000): it:   20800 | loss: 3.108e+01
VI NF (t=1.000): it:   20900 | loss: 3.120e+01
VI NF (t=1.000): it:   21000 | loss: 3.101e+01
VI NF (t=1.000): it:   21100 | loss: 3.114e+01
VI NF (t=1.000): it:   21200 | loss: 3.111e+01
VI NF (t=1.000): it:   21300 | loss: 3.115e+01
VI NF (t=1.000): it:   21400 | loss: 3.105e+01
VI NF (t=1.000): it:   21500 | loss: 3.106e+01
VI NF (t=1.000): it:   21600 | loss: 3.108e+01
VI NF (t=1.000): it:   21700 | loss: 3.107e+01
VI NF (t=1.000): it:   21800 | loss: 3.111e+01
VI NF (t=1.000): it:   21900 | loss: 3.104e+01
VI NF (t=1.000): it:   22000 | loss: 3.114e+01
VI NF (t=1.000): it:   22100 | loss: 3.113e+01
VI NF (t=1.000): it:   22200 | loss: 3.113e+01
VI NF (t=1.000): it:   22300 | loss: 3.108e+01
VI NF (t=1.000): it:   22400 | loss: 3.108e+01
VI NF (t=1.000): it:   22500 | loss: 3.108e+01
VI NF (t=1.000): it:   22600 | loss: 3.115e+01
VI NF (t=1.000): it:   22700 | loss: 3.110e+01
VI NF (t=1.000): it:   22800 | loss: 3.110e+01
VI NF (t=1.000): it:   22900 | loss: 3.098e+01
VI NF (t=1.000): it:   23000 | loss: 3.105e+01
VI NF (t=1.000): it:   23100 | loss: 3.109e+01
VI NF (t=1.000): it:   23200 | loss: 3.117e+01
VI NF (t=1.000): it:   23300 | loss: 3.105e+01
VI NF (t=1.000): it:   23400 | loss: 3.108e+01
VI NF (t=1.000): it:   23500 | loss: 3.095e+01
VI NF (t=1.000): it:   23600 | loss: 3.105e+01
VI NF (t=1.000): it:   23700 | loss: 3.108e+01
VI NF (t=1.000): it:   23800 | loss: 3.102e+01
VI NF (t=1.000): it:   23900 | loss: 3.109e+01
VI NF (t=1.000): it:   24000 | loss: 3.107e+01
VI NF (t=1.000): it:   24100 | loss: 3.102e+01
VI NF (t=1.000): it:   24200 | loss: 3.108e+01
VI NF (t=1.000): it:   24300 | loss: 3.112e+01
VI NF (t=1.000): it:   24400 | loss: 3.102e+01
VI NF (t=1.000): it:   24500 | loss: 3.114e+01
VI NF (t=1.000): it:   24600 | loss: 3.115e+01
VI NF (t=1.000): it:   24700 | loss: 3.100e+01
VI NF (t=1.000): it:   24800 | loss: 3.100e+01
VI NF (t=1.000): it:   24900 | loss: 3.114e+01
--- Saving results at iteration 25000
VI NF (t=1.000): it:   25000 | loss: 3.109e+01

--- Simulation completed!

Notice that the model evaluation has been successfully completed by checking at the newly created phys_nofasFree_2d folder in our current directory.

Note also that LINFA supports a post processing script to plot the results, including:

Loss profile.
Sample-based characterization of the posterior distribution.
Sample-based characterization of the posterior preditive distribution with observations.

[14]:

import linfa
! python3 -m linfa.plot_res -n phys_full_2d -i 25000 -f "./" -p 'png' -d

Plotting log...
Plotting posterior samples...
Plotting posterior predictive samples...

Variational inference with neural network surrogate model

LINFA enables the construction of an adaptively trained surrogate model to mitigate the cost of inference with computationally expensive models.

By utilizing the surrogate model, model evaluations are replaced by interrogating the surrogate at a significantly cheaper cost.

Model and surrogate definition

We create an instance of the Phys model and create an instance of a surrogate using our built-in Surrogate constructor.

[18]:

# Experiment Setting
exp = experiment()

exp.name             = "phys_surr_2d"      # str: Name of experiment
exp.output_dir       = './' + exp.name    # str: output directory location

exp.flow_type        = 'maf'        # str: Type of flow
exp.n_blocks         = 5            # int: Number of layers
exp.hidden_size      = 100          # int: Hidden layer size for MADE in each layer
exp.n_hidden         = 1            # int: Number of hidden layers in each MADE
exp.activation_fn    = 'relu'       # str: Actication function used
exp.input_order      = 'sequential' # str: Input order for create_mask
exp.batch_norm_order = True         # boolean: Order to decide if batch_norm is used
exp.save_interval    = 5000         # int: How often to sample from normalizing flow

exp.input_size    = 2               # int: Dimensionality of input
exp.batch_size    = 200             # int: Number of samples generated
exp.true_data_num = 2               # double: number of true model evaluated
exp.n_iter        = 25001           # int: Number of iterations
exp.lr            = 0.01            # float: Learning rate
exp.lr_decay      = 0.9999          # float: Learning rate decay
exp.log_interval  = 100             # int: How often to show loss stat

exp.run_nofas          = True       # boolean: to run experiment with nofas
exp.annealing          = False      # boolean: to run experiment with annealing
exp.calibrate_interval = 1000       # int: How often to update surrogate model
exp.budget             = 5000       # int: Total number of true model evaluation

exp.surr_pre_it  = 30000            # int: Number of pre-training iterations for surrogate model
exp.surr_upd_it  = 6000             # int: Number of iterations for the surrogate model update
exp.surr_folder  = "./"
exp.use_new_surr = True             # boolean: to run experiment with nofas

exp.results_file = 'results.txt'      # str: result text file name
exp.log_file     = 'log.txt'          # str: log text file name
exp.samples_file = 'samples.txt'      # str: sample text file name
exp.seed         = random.randint(0, 10 ** 9)  # int: Random seed used
exp.n_sample     = 5000               # int: Total number of iterations
exp.no_cuda      = True               # boolean: to run experiment with NO cuda

exp.optimizer    = 'RMSprop'          # str: Type of optimizer
exp.lr_scheduler = 'ExponentialLR'    # str: Type of scheduler

exp.device = torch.device('cuda:0' if torch.cuda.is_available() and not exp.no_cuda else 'cpu')

# Define transformation based on normalization rate
trsf_info = [['identity',0.0,0.0,0.0,0.0],
             ['identity',0.0,0.0,0.0,0.0]]
trsf = Transformation(trsf_info)
exp.transform = trsf

[19]:

# Define model
model = Phys()
exp.model = model

# Get data
model.data = np.loadtxt('./data_phys_2d.txt')

# Define surrogate
exp.surrogate = Surrogate(exp.name, lambda x: model.solve_t(trsf.forward(x)), exp.input_size, 2, dnn_arch = [128, 64],
                          model_folder=exp.surr_folder, limits=torch.Tensor([[0, 2], [0, 10]]),
                          memory_len=20, device=exp.device)
surr_filename = exp.surr_folder + exp.name

if exp.use_new_surr or (not os.path.isfile(surr_filename + ".sur")) or (not os.path.isfile(surr_filename + ".npz")):
    print("Warning: Surrogate model files: {0}.npz and {0}.npz could not be found. ".format(surr_filename))
    # 4 samples for each dimension
    exp.surrogate.gen_grid(gridnum=4)
    exp.surrogate.pre_train(exp.surr_pre_it, 0.01, 0.9999, 500, store=True)

# Load the surrogate
exp.surrogate.surrogate_load()

Warning: ./phys_surr_2d.npz does not found, please generate pre-grid.
Suggestion: Use Surrogate.gen_grid(input_limits=None, grid_num=5, store=True)
Warning: Surrogate model files: ./phys_surr_2d.npz and ./phys_surr_2d.npz could not be found.

--- Pre-training surrogate model

SUR: PRE: it:       0 | loss: 1.680e+00
SUR: PRE: it:     500 | loss: 1.963e-02
SUR: PRE: it:    1000 | loss: 2.407e-02
SUR: PRE: it:    1500 | loss: 1.049e-02
SUR: PRE: it:    2000 | loss: 7.563e-03
SUR: PRE: it:    2500 | loss: 6.208e-03
SUR: PRE: it:    3000 | loss: 7.349e-03
SUR: PRE: it:    3500 | loss: 6.506e-03
SUR: PRE: it:    4000 | loss: 4.128e-03
SUR: PRE: it:    4500 | loss: 3.097e-03
SUR: PRE: it:    5000 | loss: 2.454e-03
SUR: PRE: it:    5500 | loss: 1.429e-03
SUR: PRE: it:    6000 | loss: 1.274e-03
SUR: PRE: it:    6500 | loss: 1.036e-03
SUR: PRE: it:    7000 | loss: 5.934e-04
SUR: PRE: it:    7500 | loss: 3.743e-04
SUR: PRE: it:    8000 | loss: 7.861e-04
SUR: PRE: it:    8500 | loss: 1.483e-03
SUR: PRE: it:    9000 | loss: 7.878e-04
SUR: PRE: it:    9500 | loss: 6.469e-04
SUR: PRE: it:   10000 | loss: 8.619e-04
SUR: PRE: it:   10500 | loss: 4.657e-04
SUR: PRE: it:   11000 | loss: 7.060e-04
SUR: PRE: it:   11500 | loss: 3.383e-04
SUR: PRE: it:   12000 | loss: 3.328e-04
SUR: PRE: it:   12500 | loss: 4.275e-04
SUR: PRE: it:   13000 | loss: 2.478e-04
SUR: PRE: it:   13500 | loss: 2.540e-04
SUR: PRE: it:   14000 | loss: 2.588e-04
SUR: PRE: it:   14500 | loss: 2.745e-04
SUR: PRE: it:   15000 | loss: 3.259e-04
SUR: PRE: it:   15500 | loss: 1.463e-04
SUR: PRE: it:   16000 | loss: 1.860e-04
SUR: PRE: it:   16500 | loss: 1.609e-04
SUR: PRE: it:   17000 | loss: 1.595e-04
SUR: PRE: it:   17500 | loss: 2.017e-04
SUR: PRE: it:   18000 | loss: 1.159e-04
SUR: PRE: it:   18500 | loss: 1.007e-04
SUR: PRE: it:   19000 | loss: 7.205e-05
SUR: PRE: it:   19500 | loss: 7.190e-05
SUR: PRE: it:   20000 | loss: 6.536e-05
SUR: PRE: it:   20500 | loss: 1.013e-04
SUR: PRE: it:   21000 | loss: 6.878e-05
SUR: PRE: it:   21500 | loss: 4.241e-05
SUR: PRE: it:   22000 | loss: 3.532e-05
SUR: PRE: it:   22500 | loss: 3.574e-05
SUR: PRE: it:   23000 | loss: 5.258e-05
SUR: PRE: it:   23500 | loss: 4.641e-05
SUR: PRE: it:   24000 | loss: 3.958e-05
SUR: PRE: it:   24500 | loss: 2.507e-05
SUR: PRE: it:   25000 | loss: 2.378e-05
SUR: PRE: it:   25500 | loss: 1.700e-05
SUR: PRE: it:   26000 | loss: 1.842e-05
SUR: PRE: it:   26500 | loss: 2.063e-05
SUR: PRE: it:   27000 | loss: 2.230e-05
SUR: PRE: it:   27500 | loss: 2.346e-05
SUR: PRE: it:   28000 | loss: 1.618e-05
SUR: PRE: it:   28500 | loss: 1.333e-05
SUR: PRE: it:   29000 | loss: 1.124e-05
SUR: PRE: it:   29500 | loss: 8.447e-06

--- Surrogate model pre-train complete

Success: [limits] loaded.
Success: [pre_grid] loaded.
Success: [grid_record] loaded.

Launch inference task

[20]:

## Run
print('')
print('--- Temporary TEST: Physics Example - with NOFAS')
print('')

# Assign logdensity
exp.model_logdensity = lambda x: log_density(x, model, exp.surrogate, trsf)

# Run VI
exp.run()


--- Temporary TEST: Physics Example - with NOFAS


--- Running on device: cpu

VI NF (t=1.000): it:     100 | loss: 6.041e+03
VI NF (t=1.000): it:     200 | loss: 2.607e+03
VI NF (t=1.000): it:     300 | loss: 6.170e+02
VI NF (t=1.000): it:     400 | loss: 5.255e+02
VI NF (t=1.000): it:     500 | loss: 5.003e+02
VI NF (t=1.000): it:     600 | loss: 4.802e+02
VI NF (t=1.000): it:     700 | loss: 4.736e+02
VI NF (t=1.000): it:     800 | loss: 4.743e+02
VI NF (t=1.000): it:     900 | loss: 4.734e+02

--- Updating surrogate model

Std before inflation -> Std after inflation
6.111e-03 -> 1.058e-01
5.825e-03 -> 8.819e-02

SUR: UPD: it:       0 | loss: 1.071e-02
SUR: UPD: it:     500 | loss: 1.240e-03
SUR: UPD: it:    1000 | loss: 8.859e-04
SUR: UPD: it:    1500 | loss: 2.054e-04
SUR: UPD: it:    2000 | loss: 4.302e-05
SUR: UPD: it:    2500 | loss: 4.385e-05
SUR: UPD: it:    3000 | loss: 9.119e-06
SUR: UPD: it:    3500 | loss: 5.706e-06
SUR: UPD: it:    4000 | loss: 1.156e-06
SUR: UPD: it:    4500 | loss: 9.692e-07
SUR: UPD: it:    5000 | loss: 1.727e-07
SUR: UPD: it:    5500 | loss: 9.460e-08

--- Surrogate model updated

VI NF (t=1.000): it:    1000 | loss: 5.575e+02
VI NF (t=1.000): it:    1100 | loss: 1.842e+02
VI NF (t=1.000): it:    1200 | loss: 8.191e+01
VI NF (t=1.000): it:    1300 | loss: 3.252e+01
VI NF (t=1.000): it:    1400 | loss: 3.258e+01
VI NF (t=1.000): it:    1500 | loss: 3.238e+01
VI NF (t=1.000): it:    1600 | loss: 3.317e+01
VI NF (t=1.000): it:    1700 | loss: 3.371e+01
VI NF (t=1.000): it:    1800 | loss: 3.308e+01
VI NF (t=1.000): it:    1900 | loss: 3.260e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
1.216e-02 -> 1.054e-01
2.007e-02 -> 1.384e-02

SUR: UPD: it:       0 | loss: 9.643e-03
SUR: UPD: it:     500 | loss: 4.002e-03
SUR: UPD: it:    1000 | loss: 1.125e-03
SUR: UPD: it:    1500 | loss: 1.084e-03
SUR: UPD: it:    2000 | loss: 3.880e-04
SUR: UPD: it:    2500 | loss: 1.975e-04
SUR: UPD: it:    3000 | loss: 1.411e-04
SUR: UPD: it:    3500 | loss: 8.666e-05
SUR: UPD: it:    4000 | loss: 5.408e-05
SUR: UPD: it:    4500 | loss: 2.771e-05
SUR: UPD: it:    5000 | loss: 1.647e-05
SUR: UPD: it:    5500 | loss: 9.560e-06

--- Surrogate model updated

VI NF (t=1.000): it:    2000 | loss: 1.604e+02
VI NF (t=1.000): it:    2100 | loss: 7.059e+01
VI NF (t=1.000): it:    2200 | loss: 6.960e+01
VI NF (t=1.000): it:    2300 | loss: 6.971e+01
VI NF (t=1.000): it:    2400 | loss: 6.940e+01
VI NF (t=1.000): it:    2500 | loss: 6.923e+01
VI NF (t=1.000): it:    2600 | loss: 6.874e+01
VI NF (t=1.000): it:    2700 | loss: 6.854e+01
VI NF (t=1.000): it:    2800 | loss: 6.824e+01
VI NF (t=1.000): it:    2900 | loss: 6.720e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
5.759e-03 -> 2.602e-03
3.762e-03 -> 8.234e-02

SUR: UPD: it:       0 | loss: 2.198e-03
SUR: UPD: it:     500 | loss: 6.127e-03
SUR: UPD: it:    1000 | loss: 1.227e-03
SUR: UPD: it:    1500 | loss: 7.419e-04
SUR: UPD: it:    2000 | loss: 2.399e-04
SUR: UPD: it:    2500 | loss: 1.541e-04
SUR: UPD: it:    3000 | loss: 9.047e-05
SUR: UPD: it:    3500 | loss: 6.778e-05
SUR: UPD: it:    4000 | loss: 5.160e-05
SUR: UPD: it:    4500 | loss: 4.136e-05
SUR: UPD: it:    5000 | loss: 3.497e-05
SUR: UPD: it:    5500 | loss: 3.130e-05

--- Surrogate model updated

VI NF (t=1.000): it:    3000 | loss: 1.450e+02
VI NF (t=1.000): it:    3100 | loss: 3.682e+01
VI NF (t=1.000): it:    3200 | loss: 3.414e+01
VI NF (t=1.000): it:    3300 | loss: 3.407e+01
VI NF (t=1.000): it:    3400 | loss: 3.375e+01
VI NF (t=1.000): it:    3500 | loss: 3.324e+01
VI NF (t=1.000): it:    3600 | loss: 3.294e+01
VI NF (t=1.000): it:    3700 | loss: 3.284e+01
VI NF (t=1.000): it:    3800 | loss: 3.271e+01
VI NF (t=1.000): it:    3900 | loss: 3.297e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
3.889e-03 -> 8.720e-02
2.889e-02 -> 6.568e-03

SUR: UPD: it:       0 | loss: 1.959e-02
SUR: UPD: it:     500 | loss: 4.191e-03
SUR: UPD: it:    1000 | loss: 1.198e-03
SUR: UPD: it:    1500 | loss: 3.682e-04
SUR: UPD: it:    2000 | loss: 5.808e-04
SUR: UPD: it:    2500 | loss: 3.874e-04
SUR: UPD: it:    3000 | loss: 1.417e-04
SUR: UPD: it:    3500 | loss: 9.648e-05
SUR: UPD: it:    4000 | loss: 7.210e-05
SUR: UPD: it:    4500 | loss: 6.381e-05
SUR: UPD: it:    5000 | loss: 5.444e-05
SUR: UPD: it:    5500 | loss: 4.779e-05

--- Surrogate model updated

VI NF (t=1.000): it:    4000 | loss: 1.417e+02
VI NF (t=1.000): it:    4100 | loss: 3.970e+01
VI NF (t=1.000): it:    4200 | loss: 3.335e+01
VI NF (t=1.000): it:    4300 | loss: 3.361e+01
VI NF (t=1.000): it:    4400 | loss: 3.408e+01
VI NF (t=1.000): it:    4500 | loss: 3.381e+01
VI NF (t=1.000): it:    4600 | loss: 3.362e+01
VI NF (t=1.000): it:    4700 | loss: 3.366e+01
VI NF (t=1.000): it:    4800 | loss: 3.373e+01
VI NF (t=1.000): it:    4900 | loss: 3.381e+01
--- Saving results at iteration 5000

--- Updating surrogate model

Std before inflation -> Std after inflation
7.479e-03 -> 8.253e-02
3.083e-03 -> 7.861e-02

SUR: UPD: it:       0 | loss: 6.430e-03
SUR: UPD: it:     500 | loss: 6.922e-03
SUR: UPD: it:    1000 | loss: 2.836e-03
SUR: UPD: it:    1500 | loss: 1.018e-03
SUR: UPD: it:    2000 | loss: 3.940e-04
SUR: UPD: it:    2500 | loss: 5.492e-05
SUR: UPD: it:    3000 | loss: 4.510e-05
SUR: UPD: it:    3500 | loss: 3.963e-05
SUR: UPD: it:    4000 | loss: 2.579e-05
SUR: UPD: it:    4500 | loss: 2.377e-05
SUR: UPD: it:    5000 | loss: 2.022e-05
SUR: UPD: it:    5500 | loss: 1.840e-05

--- Surrogate model updated

VI NF (t=1.000): it:    5000 | loss: 7.616e+01
VI NF (t=1.000): it:    5100 | loss: 3.364e+01
VI NF (t=1.000): it:    5200 | loss: 3.445e+01
VI NF (t=1.000): it:    5300 | loss: 3.379e+01
VI NF (t=1.000): it:    5400 | loss: 3.368e+01
VI NF (t=1.000): it:    5500 | loss: 3.400e+01
VI NF (t=1.000): it:    5600 | loss: 3.400e+01
VI NF (t=1.000): it:    5700 | loss: 3.401e+01
VI NF (t=1.000): it:    5800 | loss: 3.376e+01
VI NF (t=1.000): it:    5900 | loss: 3.397e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
6.581e-03 -> 1.361e-01
6.170e-02 -> 9.207e-02

SUR: UPD: it:       0 | loss: 9.361e-03
SUR: UPD: it:     500 | loss: 5.632e-03
SUR: UPD: it:    1000 | loss: 1.237e-03
SUR: UPD: it:    1500 | loss: 1.887e-03
SUR: UPD: it:    2000 | loss: 2.834e-04
SUR: UPD: it:    2500 | loss: 2.080e-04
SUR: UPD: it:    3000 | loss: 3.206e-04
SUR: UPD: it:    3500 | loss: 1.799e-04
SUR: UPD: it:    4000 | loss: 1.234e-04
SUR: UPD: it:    4500 | loss: 9.745e-05
SUR: UPD: it:    5000 | loss: 8.481e-05
SUR: UPD: it:    5500 | loss: 7.384e-05

--- Surrogate model updated

VI NF (t=1.000): it:    6000 | loss: 4.922e+01
VI NF (t=1.000): it:    6100 | loss: 3.512e+01
VI NF (t=1.000): it:    6200 | loss: 3.469e+01
VI NF (t=1.000): it:    6300 | loss: 3.271e+01
VI NF (t=1.000): it:    6400 | loss: 3.213e+01
VI NF (t=1.000): it:    6500 | loss: 3.186e+01
VI NF (t=1.000): it:    6600 | loss: 3.198e+01
VI NF (t=1.000): it:    6700 | loss: 3.192e+01
VI NF (t=1.000): it:    6800 | loss: 3.210e+01
VI NF (t=1.000): it:    6900 | loss: 3.201e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
3.214e-02 -> 3.774e-02
2.174e-02 -> 2.128e-01

SUR: UPD: it:       0 | loss: 2.443e-03
SUR: UPD: it:     500 | loss: 2.069e-02
SUR: UPD: it:    1000 | loss: 1.371e-03
SUR: UPD: it:    1500 | loss: 1.054e-03
SUR: UPD: it:    2000 | loss: 6.096e-04
SUR: UPD: it:    2500 | loss: 4.239e-04
SUR: UPD: it:    3000 | loss: 4.691e-04
SUR: UPD: it:    3500 | loss: 3.989e-04
SUR: UPD: it:    4000 | loss: 3.880e-04
SUR: UPD: it:    4500 | loss: 3.762e-04
SUR: UPD: it:    5000 | loss: 3.714e-04
SUR: UPD: it:    5500 | loss: 3.666e-04

--- Surrogate model updated

VI NF (t=1.000): it:    7000 | loss: 7.511e+01
VI NF (t=1.000): it:    7100 | loss: 3.117e+01
VI NF (t=1.000): it:    7200 | loss: 3.106e+01
VI NF (t=1.000): it:    7300 | loss: 3.134e+01
VI NF (t=1.000): it:    7400 | loss: 3.118e+01
VI NF (t=1.000): it:    7500 | loss: 3.103e+01
VI NF (t=1.000): it:    7600 | loss: 3.114e+01
VI NF (t=1.000): it:    7700 | loss: 3.146e+01
VI NF (t=1.000): it:    7800 | loss: 3.108e+01
VI NF (t=1.000): it:    7900 | loss: 3.110e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
6.882e-02 -> 1.174e-01
6.200e-02 -> 4.097e-02

SUR: UPD: it:       0 | loss: 3.461e-04
SUR: UPD: it:     500 | loss: 6.803e-04
SUR: UPD: it:    1000 | loss: 9.166e-03
SUR: UPD: it:    1500 | loss: 1.278e-03
SUR: UPD: it:    2000 | loss: 5.025e-04
SUR: UPD: it:    2500 | loss: 3.834e-04
SUR: UPD: it:    3000 | loss: 2.359e-04
SUR: UPD: it:    3500 | loss: 2.276e-04
SUR: UPD: it:    4000 | loss: 2.142e-04
SUR: UPD: it:    4500 | loss: 2.102e-04
SUR: UPD: it:    5000 | loss: 2.085e-04
SUR: UPD: it:    5500 | loss: 2.079e-04

--- Surrogate model updated

VI NF (t=1.000): it:    8000 | loss: 3.129e+01
VI NF (t=1.000): it:    8100 | loss: 3.110e+01
VI NF (t=1.000): it:    8200 | loss: 3.125e+01
VI NF (t=1.000): it:    8300 | loss: 3.114e+01
VI NF (t=1.000): it:    8400 | loss: 3.125e+01
VI NF (t=1.000): it:    8500 | loss: 3.099e+01
VI NF (t=1.000): it:    8600 | loss: 3.109e+01
VI NF (t=1.000): it:    8700 | loss: 3.125e+01
VI NF (t=1.000): it:    8800 | loss: 3.108e+01
VI NF (t=1.000): it:    8900 | loss: 3.108e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
5.031e-03 -> 2.264e-01
1.757e-03 -> 7.609e-02

SUR: UPD: it:       0 | loss: 2.045e-04
SUR: UPD: it:     500 | loss: 1.412e-02
SUR: UPD: it:    1000 | loss: 3.641e-03
SUR: UPD: it:    1500 | loss: 1.259e-03
SUR: UPD: it:    2000 | loss: 5.400e-04
SUR: UPD: it:    2500 | loss: 3.578e-04
SUR: UPD: it:    3000 | loss: 3.110e-04
SUR: UPD: it:    3500 | loss: 2.807e-04
SUR: UPD: it:    4000 | loss: 2.633e-04
SUR: UPD: it:    4500 | loss: 2.607e-04
SUR: UPD: it:    5000 | loss: 2.587e-04
SUR: UPD: it:    5500 | loss: 2.577e-04

--- Surrogate model updated

VI NF (t=1.000): it:    9000 | loss: 3.106e+01
VI NF (t=1.000): it:    9100 | loss: 3.100e+01
VI NF (t=1.000): it:    9200 | loss: 3.107e+01
VI NF (t=1.000): it:    9300 | loss: 3.100e+01
VI NF (t=1.000): it:    9400 | loss: 3.110e+01
VI NF (t=1.000): it:    9500 | loss: 3.111e+01
VI NF (t=1.000): it:    9600 | loss: 3.114e+01
VI NF (t=1.000): it:    9700 | loss: 3.120e+01
VI NF (t=1.000): it:    9800 | loss: 3.103e+01
VI NF (t=1.000): it:    9900 | loss: 3.102e+01
--- Saving results at iteration 10000

--- Updating surrogate model

Std before inflation -> Std after inflation
1.570e-02 -> 7.066e-02
1.223e-02 -> 1.902e-01

SUR: UPD: it:       0 | loss: 2.864e-04
SUR: UPD: it:     500 | loss: 5.465e-03
SUR: UPD: it:    1000 | loss: 3.172e-03
SUR: UPD: it:    1500 | loss: 1.786e-03
SUR: UPD: it:    2000 | loss: 9.306e-04
SUR: UPD: it:    2500 | loss: 2.891e-04
SUR: UPD: it:    3000 | loss: 1.961e-04
SUR: UPD: it:    3500 | loss: 1.701e-04
SUR: UPD: it:    4000 | loss: 1.629e-04
SUR: UPD: it:    4500 | loss: 1.593e-04
SUR: UPD: it:    5000 | loss: 1.563e-04
SUR: UPD: it:    5500 | loss: 1.560e-04

--- Surrogate model updated

VI NF (t=1.000): it:   10000 | loss: 3.105e+01
VI NF (t=1.000): it:   10100 | loss: 3.113e+01
VI NF (t=1.000): it:   10200 | loss: 3.112e+01
VI NF (t=1.000): it:   10300 | loss: 3.113e+01
VI NF (t=1.000): it:   10400 | loss: 3.107e+01
VI NF (t=1.000): it:   10500 | loss: 3.123e+01
VI NF (t=1.000): it:   10600 | loss: 3.105e+01
VI NF (t=1.000): it:   10700 | loss: 3.103e+01
VI NF (t=1.000): it:   10800 | loss: 3.099e+01
VI NF (t=1.000): it:   10900 | loss: 3.103e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
1.554e-02 -> 5.137e-02
3.020e-03 -> 1.343e-01

SUR: UPD: it:       0 | loss: 1.552e-04
SUR: UPD: it:     500 | loss: 6.740e-03
SUR: UPD: it:    1000 | loss: 2.061e-03
SUR: UPD: it:    1500 | loss: 1.745e-03
SUR: UPD: it:    2000 | loss: 4.061e-04
SUR: UPD: it:    2500 | loss: 2.855e-04
SUR: UPD: it:    3000 | loss: 1.796e-04
SUR: UPD: it:    3500 | loss: 1.924e-04
SUR: UPD: it:    4000 | loss: 1.509e-04
SUR: UPD: it:    4500 | loss: 1.487e-04
SUR: UPD: it:    5000 | loss: 1.471e-04
SUR: UPD: it:    5500 | loss: 1.466e-04

--- Surrogate model updated

VI NF (t=1.000): it:   11000 | loss: 3.093e+01
VI NF (t=1.000): it:   11100 | loss: 3.082e+01
VI NF (t=1.000): it:   11200 | loss: 3.105e+01
VI NF (t=1.000): it:   11300 | loss: 3.101e+01
VI NF (t=1.000): it:   11400 | loss: 3.102e+01
VI NF (t=1.000): it:   11500 | loss: 3.106e+01
VI NF (t=1.000): it:   11600 | loss: 3.117e+01
VI NF (t=1.000): it:   11700 | loss: 3.092e+01
VI NF (t=1.000): it:   11800 | loss: 3.105e+01
VI NF (t=1.000): it:   11900 | loss: 3.107e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
8.279e-02 -> 3.688e-01
6.718e-02 -> 1.169e-01

SUR: UPD: it:       0 | loss: 1.714e-04
SUR: UPD: it:     500 | loss: 3.535e-03
SUR: UPD: it:    1000 | loss: 8.724e-04
SUR: UPD: it:    1500 | loss: 4.861e-04
SUR: UPD: it:    2000 | loss: 6.086e-05
SUR: UPD: it:    2500 | loss: 1.960e-04
SUR: UPD: it:    3000 | loss: 3.153e-05
SUR: UPD: it:    3500 | loss: 3.220e-05
SUR: UPD: it:    4000 | loss: 1.802e-05
SUR: UPD: it:    4500 | loss: 2.101e-05
SUR: UPD: it:    5000 | loss: 1.664e-05
SUR: UPD: it:    5500 | loss: 1.635e-05

--- Surrogate model updated

VI NF (t=1.000): it:   12000 | loss: 3.115e+01
VI NF (t=1.000): it:   12100 | loss: 3.103e+01
VI NF (t=1.000): it:   12200 | loss: 3.100e+01
VI NF (t=1.000): it:   12300 | loss: 3.107e+01
VI NF (t=1.000): it:   12400 | loss: 3.101e+01
VI NF (t=1.000): it:   12500 | loss: 3.113e+01
VI NF (t=1.000): it:   12600 | loss: 3.118e+01
VI NF (t=1.000): it:   12700 | loss: 3.102e+01
VI NF (t=1.000): it:   12800 | loss: 3.101e+01
VI NF (t=1.000): it:   12900 | loss: 3.125e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
5.867e-02 -> 3.129e-01
3.974e-02 -> 8.893e-02

SUR: UPD: it:       0 | loss: 1.651e-05
SUR: UPD: it:     500 | loss: 8.846e-03
SUR: UPD: it:    1000 | loss: 2.439e-03
SUR: UPD: it:    1500 | loss: 1.040e-03
SUR: UPD: it:    2000 | loss: 8.042e-04
SUR: UPD: it:    2500 | loss: 1.604e-04
SUR: UPD: it:    3000 | loss: 7.391e-05
SUR: UPD: it:    3500 | loss: 5.874e-05
SUR: UPD: it:    4000 | loss: 3.408e-05
SUR: UPD: it:    4500 | loss: 2.803e-05
SUR: UPD: it:    5000 | loss: 2.545e-05
SUR: UPD: it:    5500 | loss: 2.401e-05

--- Surrogate model updated

VI NF (t=1.000): it:   13000 | loss: 3.103e+01
VI NF (t=1.000): it:   13100 | loss: 3.101e+01
VI NF (t=1.000): it:   13200 | loss: 3.123e+01
VI NF (t=1.000): it:   13300 | loss: 3.113e+01
VI NF (t=1.000): it:   13400 | loss: 3.105e+01
VI NF (t=1.000): it:   13500 | loss: 3.111e+01
VI NF (t=1.000): it:   13600 | loss: 3.111e+01
VI NF (t=1.000): it:   13700 | loss: 3.121e+01
VI NF (t=1.000): it:   13800 | loss: 3.111e+01
VI NF (t=1.000): it:   13900 | loss: 3.110e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
6.867e-02 -> 9.649e-02
9.803e-02 -> 8.442e-02

SUR: UPD: it:       0 | loss: 2.225e-05
SUR: UPD: it:     500 | loss: 9.911e-03
SUR: UPD: it:    1000 | loss: 2.221e-03
SUR: UPD: it:    1500 | loss: 8.999e-04
SUR: UPD: it:    2000 | loss: 4.299e-04
SUR: UPD: it:    2500 | loss: 1.834e-04
SUR: UPD: it:    3000 | loss: 6.646e-05
SUR: UPD: it:    3500 | loss: 3.524e-05
SUR: UPD: it:    4000 | loss: 1.881e-05
SUR: UPD: it:    4500 | loss: 1.630e-05
SUR: UPD: it:    5000 | loss: 1.680e-05
SUR: UPD: it:    5500 | loss: 1.486e-05

--- Surrogate model updated

VI NF (t=1.000): it:   14000 | loss: 3.116e+01
VI NF (t=1.000): it:   14100 | loss: 3.102e+01
VI NF (t=1.000): it:   14200 | loss: 3.111e+01
VI NF (t=1.000): it:   14300 | loss: 3.101e+01
VI NF (t=1.000): it:   14400 | loss: 3.109e+01
VI NF (t=1.000): it:   14500 | loss: 3.118e+01
VI NF (t=1.000): it:   14600 | loss: 3.097e+01
VI NF (t=1.000): it:   14700 | loss: 3.115e+01
VI NF (t=1.000): it:   14800 | loss: 3.100e+01
VI NF (t=1.000): it:   14900 | loss: 3.106e+01
--- Saving results at iteration 15000

--- Updating surrogate model

Std before inflation -> Std after inflation
1.110e-02 -> 9.629e-02
4.179e-02 -> 9.732e-02

SUR: UPD: it:       0 | loss: 4.031e-04
SUR: UPD: it:     500 | loss: 1.200e-02
SUR: UPD: it:    1000 | loss: 2.011e-03
SUR: UPD: it:    1500 | loss: 6.918e-04
SUR: UPD: it:    2000 | loss: 3.562e-04
SUR: UPD: it:    2500 | loss: 1.094e-04
SUR: UPD: it:    3000 | loss: 9.383e-05
SUR: UPD: it:    3500 | loss: 3.949e-05
SUR: UPD: it:    4000 | loss: 3.547e-05
SUR: UPD: it:    4500 | loss: 1.806e-05
SUR: UPD: it:    5000 | loss: 1.695e-05
SUR: UPD: it:    5500 | loss: 1.533e-05

--- Surrogate model updated

VI NF (t=1.000): it:   15000 | loss: 3.103e+01
VI NF (t=1.000): it:   15100 | loss: 3.114e+01
VI NF (t=1.000): it:   15200 | loss: 3.107e+01
VI NF (t=1.000): it:   15300 | loss: 3.113e+01
VI NF (t=1.000): it:   15400 | loss: 3.095e+01
VI NF (t=1.000): it:   15500 | loss: 3.105e+01
VI NF (t=1.000): it:   15600 | loss: 3.107e+01
VI NF (t=1.000): it:   15700 | loss: 3.105e+01
VI NF (t=1.000): it:   15800 | loss: 3.117e+01
VI NF (t=1.000): it:   15900 | loss: 3.106e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
2.624e-02 -> 9.448e-02
1.401e-02 -> 5.393e-02

SUR: UPD: it:       0 | loss: 1.422e-05
SUR: UPD: it:     500 | loss: 5.328e-03
SUR: UPD: it:    1000 | loss: 7.122e-03
SUR: UPD: it:    1500 | loss: 4.442e-04
SUR: UPD: it:    2000 | loss: 4.223e-04
SUR: UPD: it:    2500 | loss: 1.169e-04
SUR: UPD: it:    3000 | loss: 5.565e-05
SUR: UPD: it:    3500 | loss: 2.981e-05
SUR: UPD: it:    4000 | loss: 3.061e-05
SUR: UPD: it:    4500 | loss: 2.245e-05
SUR: UPD: it:    5000 | loss: 1.881e-05
SUR: UPD: it:    5500 | loss: 1.849e-05

--- Surrogate model updated

VI NF (t=1.000): it:   16000 | loss: 3.116e+01
VI NF (t=1.000): it:   16100 | loss: 3.107e+01
VI NF (t=1.000): it:   16200 | loss: 3.120e+01
VI NF (t=1.000): it:   16300 | loss: 3.116e+01
VI NF (t=1.000): it:   16400 | loss: 3.118e+01
VI NF (t=1.000): it:   16500 | loss: 3.117e+01
VI NF (t=1.000): it:   16600 | loss: 3.111e+01
VI NF (t=1.000): it:   16700 | loss: 3.121e+01
VI NF (t=1.000): it:   16800 | loss: 3.117e+01
VI NF (t=1.000): it:   16900 | loss: 3.120e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
1.507e-02 -> 3.806e-02
9.023e-03 -> 1.233e-02

SUR: UPD: it:       0 | loss: 1.642e-05
SUR: UPD: it:     500 | loss: 9.687e-03
SUR: UPD: it:    1000 | loss: 3.433e-03
SUR: UPD: it:    1500 | loss: 9.443e-04
SUR: UPD: it:    2000 | loss: 5.775e-04
SUR: UPD: it:    2500 | loss: 1.633e-04
SUR: UPD: it:    3000 | loss: 7.060e-05
SUR: UPD: it:    3500 | loss: 2.870e-05
SUR: UPD: it:    4000 | loss: 1.850e-05
SUR: UPD: it:    4500 | loss: 1.465e-05
SUR: UPD: it:    5000 | loss: 1.300e-05
SUR: UPD: it:    5500 | loss: 1.217e-05

--- Surrogate model updated

VI NF (t=1.000): it:   17000 | loss: 3.114e+01
VI NF (t=1.000): it:   17100 | loss: 3.102e+01
VI NF (t=1.000): it:   17200 | loss: 3.107e+01
VI NF (t=1.000): it:   17300 | loss: 3.111e+01
VI NF (t=1.000): it:   17400 | loss: 3.114e+01
VI NF (t=1.000): it:   17500 | loss: 3.115e+01
VI NF (t=1.000): it:   17600 | loss: 3.103e+01
VI NF (t=1.000): it:   17700 | loss: 3.115e+01
VI NF (t=1.000): it:   17800 | loss: 3.108e+01
VI NF (t=1.000): it:   17900 | loss: 3.105e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
2.570e-02 -> 1.190e-01
1.993e-02 -> 6.132e-02

SUR: UPD: it:       0 | loss: 1.152e-05
SUR: UPD: it:     500 | loss: 5.580e-03
SUR: UPD: it:    1000 | loss: 1.578e-03
SUR: UPD: it:    1500 | loss: 1.647e-03
SUR: UPD: it:    2000 | loss: 4.854e-04
SUR: UPD: it:    2500 | loss: 1.530e-04
SUR: UPD: it:    3000 | loss: 1.576e-04
SUR: UPD: it:    3500 | loss: 1.219e-04
SUR: UPD: it:    4000 | loss: 4.829e-05
SUR: UPD: it:    4500 | loss: 4.021e-05
SUR: UPD: it:    5000 | loss: 3.776e-05
SUR: UPD: it:    5500 | loss: 3.609e-05

--- Surrogate model updated

VI NF (t=1.000): it:   18000 | loss: 3.112e+01
VI NF (t=1.000): it:   18100 | loss: 3.104e+01
VI NF (t=1.000): it:   18200 | loss: 3.103e+01
VI NF (t=1.000): it:   18300 | loss: 3.121e+01
VI NF (t=1.000): it:   18400 | loss: 3.098e+01
VI NF (t=1.000): it:   18500 | loss: 3.127e+01
VI NF (t=1.000): it:   18600 | loss: 3.111e+01
VI NF (t=1.000): it:   18700 | loss: 3.119e+01
VI NF (t=1.000): it:   18800 | loss: 3.108e+01
VI NF (t=1.000): it:   18900 | loss: 3.109e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
3.665e-02 -> 2.060e-02
3.909e-02 -> 4.580e-02

SUR: UPD: it:       0 | loss: 3.132e-05
SUR: UPD: it:     500 | loss: 6.733e-02
SUR: UPD: it:    1000 | loss: 1.541e-03
SUR: UPD: it:    1500 | loss: 6.137e-04
SUR: UPD: it:    2000 | loss: 3.736e-04
SUR: UPD: it:    2500 | loss: 1.979e-04
SUR: UPD: it:    3000 | loss: 9.405e-05
SUR: UPD: it:    3500 | loss: 7.938e-05
SUR: UPD: it:    4000 | loss: 7.483e-05
SUR: UPD: it:    4500 | loss: 5.132e-05
SUR: UPD: it:    5000 | loss: 3.633e-05
SUR: UPD: it:    5500 | loss: 3.246e-05

--- Surrogate model updated

VI NF (t=1.000): it:   19000 | loss: 3.114e+01
VI NF (t=1.000): it:   19100 | loss: 3.110e+01
VI NF (t=1.000): it:   19200 | loss: 3.098e+01
VI NF (t=1.000): it:   19300 | loss: 3.108e+01
VI NF (t=1.000): it:   19400 | loss: 3.116e+01
VI NF (t=1.000): it:   19500 | loss: 3.108e+01
VI NF (t=1.000): it:   19600 | loss: 3.097e+01
VI NF (t=1.000): it:   19700 | loss: 3.105e+01
VI NF (t=1.000): it:   19800 | loss: 3.120e+01
VI NF (t=1.000): it:   19900 | loss: 3.114e+01
--- Saving results at iteration 20000

--- Updating surrogate model

Std before inflation -> Std after inflation
5.500e-02 -> 6.023e-02
6.493e-02 -> 9.413e-02

SUR: UPD: it:       0 | loss: 2.807e-05
SUR: UPD: it:     500 | loss: 6.306e-02
SUR: UPD: it:    1000 | loss: 3.708e-02
SUR: UPD: it:    1500 | loss: 2.994e-03
SUR: UPD: it:    2000 | loss: 2.967e-04
SUR: UPD: it:    2500 | loss: 1.214e-04
SUR: UPD: it:    3000 | loss: 1.085e-04
SUR: UPD: it:    3500 | loss: 1.072e-04
SUR: UPD: it:    4000 | loss: 9.212e-05
SUR: UPD: it:    4500 | loss: 8.520e-05
SUR: UPD: it:    5000 | loss: 8.675e-05
SUR: UPD: it:    5500 | loss: 8.206e-05

--- Surrogate model updated

VI NF (t=1.000): it:   20000 | loss: 3.112e+01
VI NF (t=1.000): it:   20100 | loss: 3.112e+01
VI NF (t=1.000): it:   20200 | loss: 3.114e+01
VI NF (t=1.000): it:   20300 | loss: 3.111e+01
VI NF (t=1.000): it:   20400 | loss: 3.104e+01
VI NF (t=1.000): it:   20500 | loss: 3.103e+01
VI NF (t=1.000): it:   20600 | loss: 3.099e+01
VI NF (t=1.000): it:   20700 | loss: 3.112e+01
VI NF (t=1.000): it:   20800 | loss: 3.108e+01
VI NF (t=1.000): it:   20900 | loss: 3.114e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
2.027e-02 -> 1.130e-01
4.196e-04 -> 9.385e-03

SUR: UPD: it:       0 | loss: 7.135e-05
SUR: UPD: it:     500 | loss: 8.615e-03
SUR: UPD: it:    1000 | loss: 2.416e-03
SUR: UPD: it:    1500 | loss: 1.949e-03
SUR: UPD: it:    2000 | loss: 5.796e-04
SUR: UPD: it:    2500 | loss: 2.990e-04
SUR: UPD: it:    3000 | loss: 1.299e-04
SUR: UPD: it:    3500 | loss: 4.247e-05
SUR: UPD: it:    4000 | loss: 2.681e-05
SUR: UPD: it:    4500 | loss: 2.063e-05
SUR: UPD: it:    5000 | loss: 1.689e-05
SUR: UPD: it:    5500 | loss: 1.242e-05

--- Surrogate model updated

VI NF (t=1.000): it:   21000 | loss: 3.131e+01
VI NF (t=1.000): it:   21100 | loss: 3.112e+01
VI NF (t=1.000): it:   21200 | loss: 3.110e+01
VI NF (t=1.000): it:   21300 | loss: 3.096e+01
VI NF (t=1.000): it:   21400 | loss: 3.111e+01
VI NF (t=1.000): it:   21500 | loss: 3.116e+01
VI NF (t=1.000): it:   21600 | loss: 3.107e+01
VI NF (t=1.000): it:   21700 | loss: 3.117e+01
VI NF (t=1.000): it:   21800 | loss: 3.109e+01
VI NF (t=1.000): it:   21900 | loss: 3.104e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
4.258e-02 -> 1.241e-01
3.361e-02 -> 1.448e-01

SUR: UPD: it:       0 | loss: 4.185e-06
SUR: UPD: it:     500 | loss: 2.770e-02
SUR: UPD: it:    1000 | loss: 3.364e-03
SUR: UPD: it:    1500 | loss: 1.270e-03
SUR: UPD: it:    2000 | loss: 1.012e-03
SUR: UPD: it:    2500 | loss: 1.679e-04
SUR: UPD: it:    3000 | loss: 1.023e-04
SUR: UPD: it:    3500 | loss: 5.261e-05
SUR: UPD: it:    4000 | loss: 3.341e-05
SUR: UPD: it:    4500 | loss: 2.590e-05
SUR: UPD: it:    5000 | loss: 2.229e-05
SUR: UPD: it:    5500 | loss: 1.942e-05

--- Surrogate model updated

VI NF (t=1.000): it:   22000 | loss: 3.102e+01
VI NF (t=1.000): it:   22100 | loss: 3.105e+01
VI NF (t=1.000): it:   22200 | loss: 3.095e+01
VI NF (t=1.000): it:   22300 | loss: 3.112e+01
VI NF (t=1.000): it:   22400 | loss: 3.097e+01
VI NF (t=1.000): it:   22500 | loss: 3.112e+01
VI NF (t=1.000): it:   22600 | loss: 3.118e+01
VI NF (t=1.000): it:   22700 | loss: 3.111e+01
VI NF (t=1.000): it:   22800 | loss: 3.103e+01
VI NF (t=1.000): it:   22900 | loss: 3.098e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
1.858e-02 -> 1.058e-01
1.554e-02 -> 1.148e-01

SUR: UPD: it:       0 | loss: 1.150e-05
SUR: UPD: it:     500 | loss: 2.360e-03
SUR: UPD: it:    1000 | loss: 6.171e-03
SUR: UPD: it:    1500 | loss: 2.671e-03
SUR: UPD: it:    2000 | loss: 1.605e-03
SUR: UPD: it:    2500 | loss: 1.426e-04
SUR: UPD: it:    3000 | loss: 1.423e-04
SUR: UPD: it:    3500 | loss: 2.730e-05
SUR: UPD: it:    4000 | loss: 1.561e-05
SUR: UPD: it:    4500 | loss: 1.060e-05
SUR: UPD: it:    5000 | loss: 8.279e-06
SUR: UPD: it:    5500 | loss: 7.292e-06

--- Surrogate model updated

VI NF (t=1.000): it:   23000 | loss: 3.114e+01
VI NF (t=1.000): it:   23100 | loss: 3.105e+01
VI NF (t=1.000): it:   23200 | loss: 3.107e+01
VI NF (t=1.000): it:   23300 | loss: 3.108e+01
VI NF (t=1.000): it:   23400 | loss: 3.112e+01
VI NF (t=1.000): it:   23500 | loss: 3.103e+01
VI NF (t=1.000): it:   23600 | loss: 3.106e+01
VI NF (t=1.000): it:   23700 | loss: 3.100e+01
VI NF (t=1.000): it:   23800 | loss: 3.112e+01
VI NF (t=1.000): it:   23900 | loss: 3.113e+01

--- Updating surrogate model

Std before inflation -> Std after inflation
5.308e-02 -> 3.697e-02
3.758e-02 -> 9.549e-02

SUR: UPD: it:       0 | loss: 4.552e-06
SUR: UPD: it:     500 | loss: 2.124e-02
SUR: UPD: it:    1000 | loss: 6.420e-04
SUR: UPD: it:    1500 | loss: 2.259e-03
SUR: UPD: it:    2000 | loss: 7.607e-04
SUR: UPD: it:    2500 | loss: 1.122e-04
SUR: UPD: it:    3000 | loss: 4.939e-05
SUR: UPD: it:    3500 | loss: 2.542e-05
SUR: UPD: it:    4000 | loss: 1.158e-05
SUR: UPD: it:    4500 | loss: 4.708e-06
SUR: UPD: it:    5000 | loss: 2.826e-06
SUR: UPD: it:    5500 | loss: 2.281e-06

--- Surrogate model updated

VI NF (t=1.000): it:   24000 | loss: 3.111e+01
VI NF (t=1.000): it:   24100 | loss: 3.109e+01
VI NF (t=1.000): it:   24200 | loss: 3.106e+01
VI NF (t=1.000): it:   24300 | loss: 3.106e+01
VI NF (t=1.000): it:   24400 | loss: 3.118e+01
VI NF (t=1.000): it:   24500 | loss: 3.107e+01
VI NF (t=1.000): it:   24600 | loss: 3.111e+01
VI NF (t=1.000): it:   24700 | loss: 3.112e+01
VI NF (t=1.000): it:   24800 | loss: 3.112e+01
VI NF (t=1.000): it:   24900 | loss: 3.095e+01
--- Saving results at iteration 25000

--- Updating surrogate model

Std before inflation -> Std after inflation
1.051e-02 -> 7.590e-03
1.186e-03 -> 4.647e-02

SUR: UPD: it:       0 | loss: 2.387e-06
SUR: UPD: it:     500 | loss: 5.112e-03
SUR: UPD: it:    1000 | loss: 2.201e-03
SUR: UPD: it:    1500 | loss: 1.043e-03
SUR: UPD: it:    2000 | loss: 6.130e-04
SUR: UPD: it:    2500 | loss: 1.447e-04
SUR: UPD: it:    3000 | loss: 4.833e-05
SUR: UPD: it:    3500 | loss: 2.324e-05
SUR: UPD: it:    4000 | loss: 1.961e-05
SUR: UPD: it:    4500 | loss: 1.144e-05
SUR: UPD: it:    5000 | loss: 1.022e-05
SUR: UPD: it:    5500 | loss: 9.295e-06

--- Surrogate model updated

VI NF (t=1.000): it:   25000 | loss: 3.107e+01

--- Simulation completed!

Plot results in the newly created phys_surr_2d folder in our current directory.

[21]:

import linfa
! python3 -m linfa.plot_res -n phys_surr_2d -i 25000 -f "./" -p 'png' -d

Plotting log...
Plotting posterior samples...
Plotting posterior predictive samples...

Results

The plots below offer a comparison between variational inference with the full model and with the neural network surrogate.

Loss profile plots

The first set of plots represent the profile of the loss function as iterations increase.

[25]:

## log-loss plots

# create figure
fig = plt.figure(figsize=(10, 7))
plt.style.use('dark_background')

# setting row and column variables
rows = 1
columns = 2

# reading images
logLoss_phys = cv2.imread('./phys_surr_2d/log_plot.png')
logLoss_physFree = cv2.imread('./phys_full_2d/log_plot.png')

# Adds a subplot at the 1st position
fig.add_subplot(rows, columns, 1)

# Add subplots corresponding to Phys with Surrogate
plt.imshow(logLoss_phys)
plt.axis('off')
plt.title("Phys. with Surrogate")

# Add subplots corresponding to Phys without Surrogate
fig.add_subplot(rows, columns, 2)
# showing image
plt.imshow(logLoss_physFree)
plt.axis('off')
plt.title("Phys. without Surrogate")

[25]:

Text(0.5, 1.0, 'Phys. without Surrogate')

Plots of predictive posterior distribution and observations

The second set of plots are represents the compatiblity between model outputs and observations. The output values corresponding to the true model parameters are:

\(x_1\): 3.2070 \(x_2\): 0.8828

Samples from the posterior predictive distribution are represented with blue dots.

We expect to observe these samples to be compatible with the available observations (in red).

[26]:

## output plots

# create figure
fig = plt.figure(figsize=(10, 7))
plt.style.use('dark_background')

# setting row and column variables
rows = 1
columns = 2

# reading images
phys1 = cv2.imread('./phys_surr_2d/data_plot_phys_surr_2d_25000_0_1.png')
physFree1 = cv2.imread('./phys_full_2d/data_plot_phys_full_2d_25000_0_1.png')

# Add subplots corresponding to Phys with Surrogates
fig.add_subplot(rows, columns, 1)
plt.imshow(phys1)
plt.axis('off')
plt.title("Phys. with Surrogate")

# Add subplots corresponding to Phys without Surrogate
fig.add_subplot(rows, columns, 2)
plt.imshow(physFree1)
plt.axis('off')
plt.title("Phys. without Surrogate")

[26]:

Text(0.5, 1.0, 'Phys. without Surrogate')

../../_images/content_tutorial_tutorial_linfa_2d_34_1.png

Plots of posterior density

The third set of plots show samples from the posterior distribution. The true parameter set was initially selected as:

\(z_1\): 1 \(z_2\): 5

[24]:

## parameter estimation plots

# create figure
fig = plt.figure(figsize=(10, 7))
plt.style.use('dark_background')

# setting row and column variables
rows = 1
columns = 2

# reading images
phys1 = cv2.imread('./phys_surr_2d/params_plot_phys_surr_2d_25000_0_1.png')
physFree1 = cv2.imread('./phys_full_2d/params_plot_phys_full_2d_25000_0_1.png')

# Add subplots corresponding to Phys with Surrogates
fig.add_subplot(rows, columns, 1)
plt.imshow(phys1)
plt.axis('off')
plt.title("Phys. with Surrogate")

# Add subplots corresponding to Phys without Surrogate
fig.add_subplot(rows, columns, 2)
plt.imshow(physFree1)
plt.axis('off')
plt.title("Phys. without Surrogate")

[24]:

Text(0.5, 1.0, 'Phys. without Surrogate')

../../_images/content_tutorial_tutorial_linfa_2d_36_1.png

As shown in the picture, use of the full model or its surrogate lead to the same posterior distribution.

For more information, please refer to our paper Appendix B. Detailed numerical benchmarks where a few more examples are discussed.