=====================================================================
===                             QPACK                              ===
======================================================================

===== 0. Table of Contents =============================================

1. Status, bug reports, etc.
2. General information
3. Package organisation and structure
4. Getting started
5. The fields of Q
        5.1  Report levels
        5.2  Names and directories
        5.3  Atmospheres
        5.4  Tag groups
        5.5  Spectroscopy
        5.6  Hydrostatic equilibrium and RTE
        5.7  Calculation grids
        5.8  Sensor
        5.9  Data Reduction 
        5.10 Thermal noise
        5.11 Retrieval/error/random quantities
        5.12 Conditional simulations
        5.13 Retrievals
        5.14 Measurement
6. Qpack strategy
        6.1 Doing a measurement characterization
        6.2 Doing an optimal inversion
        6.3 Generating random sets

===== 1. Status, bug reports, etc. ======================================

Qpack was implemented by Patrick Eriksson and Carlos Jimenez. 

The package is shared following the philosophy of the GNU public
license, this including no warranty what so ever.

Comments, suggestions and bug reports shall be sent to:
        patrick.eriksson@chalmers.se

The present official version is Qpack 1.0. Version 1.0 meets the basic
needs for doing inversions basic error characterisation and to
generate random set of simulated spectra. The only retrieval method
handled so far is OEM.

Due to our limited time, there is no user guide for Qpack beside this
README file. But don't hesitate to contact us with questions if you
have problems with Qpack. Good luck!


===== 2. General information ============================================

Qpack is a package of Matlab functions for retrieval work using ARTS
as computational engine. Beside functions for the actual inversion and
setting up the retrieval quantities, the package includes functions to
characterise the retrieval and to optimise the calculation grids.
 
The input to ARTS is provided in two ways, by control file templates
and by a structure with settings, denoted as Q. The final control
files are created by the Qtool in AMI (ARTS Matlab interface, found at
arts/ami). All input not given by the template files must be specified
by Q. The fields of Q are defined at the end of this file.

For each retrieval/error quantity  there exist a do-field, for example
Q.TEMPERATURE_DO. If the do-field is set to zero, the quantity will be
neglected totally  and the  other fields  of the quantity  can be left
undefined. For retrieval/error  quantities,  the do-field can   have a
value between 0   and 3: 0  neglect the  quantity  totally, 1  neglect
during retrieval  but calculate error   for this quantity,  2 treat as
observation error, 3 retrieve this  quantity However, some quantities,
such as calibration error, cannot be retrieved and the  value 3 is not
allowed. See below in the section on "Qpack strategies".

Fields of Q ending with "_ON" shall be treated as booleans.  For
example, Q.CHBIN_ON=0 means that no binning of channels shall be
performed. If a on-field is set to zero, the other fields of the
quantity can be left undefined.

The Q structure shall be defined in a function taking Q as only input
and also returning Q. These functions shall not be called directly,
the Q structure is instead obtained by the function qpack which calls
the specified function. The function qpack starts with setting all do
and on variables to zero. In this way it is not necessary to change
all old files with definitions of Q when a new retrieval/error
variable is introduced. The qpack function makes also some initial
manipulations not handled elsewhere, such as setting the global
variable REPORT_LEVEL used by the out.m in AMI to the value of
Q.QP_LEVEL. See further the help of qpack.

Most input files must be in the ARTS ascii format. This is true for
all grids read directly by ARTS. Some files read by Qpack functions
can either by ascii or binary (for example, sensor definitions), but
to be safe, use ascii as this format always works. The file format for
the different quantities is commented below the first corresponding
variable. Please note that the 10-log (decades) is used as altitude
coordinate when defining covariance matrices. All covariance matrices
are read by the AMI function sFromFile (see on-line help for a general
definition of the file format).

Some quantities are expected to be calculated before doing an
inversion or characterisation of the retrieval performance. This
typically reflects the degree of influence of the atmospheric state on
the quantity. For instance, quantities such as H, Sx and Se are
pre-calculated, while Kx is not.
 
The following main quantities shall always be pre-calculated: H, Sx
and the inverse of Se. Together with these main variables, some
derived variables are created for higher flexibility. The order for
the pre-calculations is H, Sx and Seinv. The pre-calculated quantities
are stored using the Matlab format (.mat files).
See "Qpack strategy" section for more information.

See next sections for more information. They will provide more details
about the package organisation and structure, some tips about how to
get started, a description of the Q fields, and some advice about how
to run Qpack.


===== 3. Package organisation and structure ============================

Qpack is structured into several sub-folders:

=== ./ ===
The only function in the main folder is a function to start up the Qpack.

=== Templates ===
The control file templates are placed here. The extension of the
templates is ".tmplt".

=== Main ===
This folder contains the main functions of Qpack. The main functions 
(beside qpack.m) start with "qp_", while so called in-line functions 
evoked by the cfile templates starts with "qpi_".

=== CLS === The functions for doing inversions by constrained least
squares, that is, OEM and Tikhonov regularisation. Ordinary least
squares is also handled by these functions for simplicity. These
functions starts with "qpcls_", beside the function doing the actual
inversions that is called just qpcls.m.

=== Setup === 
This folder contains various help functions to create
and "optimise" the input files.  The functions in this directory
starts with "qpmake_" and "qpopt_".

=== Samples ===
This folder contains a set of sample files to get started with
Qpack. The example case corresponds to the 501.4 GHz band of Odin-SMR,
but please note that the files are only provided as examples and do
not make a perfect basis for simulating the Odin observations.
These files form a full scale, but not extremely large, test case.


===== 4. Getting started ================================================

In the directory Sample you find a definition of the Q structure matching
the files in the sub-folders. Copy this file (sample_q.m) to your test
directory. Edit the path of the directories to match your environment.
For example, if you want the pre-calculated quantities to be stored in
the folder /home/xxx/Test, you set Q.OUT to this folder.
After this, you should be ready to test Qpack.


===== 5. The fields of Q ================================================

The Qpack fields are described here. Check also Qpack/ChangeLog for an
understanding of Qpack and possible changes not reflected here. The
information here is a fairly update of the present Qpack version, but
we might have missed something. To check in the Changelog make a
search of the field from the top of the file to see in chronological
order - from most recent - all the information related to that field.
Notice also that in some cases there is an exact equivalence between
a Q field and an ARTS workspace variable. In this case more information
can be found in the ARTS user's guide (AUG).

 5.1  Report levels
        QP_LEVEL
        ARTS_LEVEL 
 5.2  Names and directories
        ARTS
        ARTS_ERROR_IGNORE
        OUT
        USE_HDF
        TMP_AREA
        SPECTRO_DIR
        CALCGRIDS_DIR
        SENSOR_DIR
        RETRIEVDEF_DIR 
 5.3  Atmospheres
        APRIORI_VMR
        APRIORI_PTZ
        SELTAGS_VMR
        SETUP_VMR
        SETUP_PTZ
 5.4  Tag groups
        RETRIEVAL_TAGS
        SELTAGS
        OTHER_TAGS
        SPECTRO_TAGS
 5.5  Spectroscopy
        LINEFORMAT
        LINEFILE
        LINESHAPE
        LINESHAPE_FACTOR
        LINESHAPE_CUTOFF
        CONTINUA
        SPECTRO_DO
        SPECTRO_TAGS 
        INTENS_ON    
        POSITION_ON  
        AGAM_ON      
        SGAM_ON      
        NAIR_ON      
        NSELF_ON     
        PSHIFT_ON    
 5.6  Hydrostatic equilibrium and RTE
        HSE_IN_ON
        HSE_RETRIEVAL_ON
        HSE_PREF
        HSE_ZREF        
        PLATFORM_ALTITUDE
        STEPLENGTH_RTE
        GROUND_ALTITUDE    
        GROUND_EMISSION    
        REFRACTION_ON
        REFR_METHOD
        REFR_LFAC
        EMISSION_ON
 5.7  Calculation grids
        P_ABS
        F_MONO
        ZA_PENCIL
 5.8  Sensor
        ANTENNA_ON
        ANTENNA_ZA
        ANTENNA_FILE
        ANTENNA_ORDER
        ANTENNA_MOVE
        DSB_ON
        DSB_FILE
        DSB_FPRIMARY
        DSB_LO
        BACKEND_ON
        BACKEND_FREQS
        BACKEND_FILE
        FREQSWITCH_ON
        FREQSWITCH_THROW
        FREQSWITCH_REVERSE
 5.9  Data Reduction 
        BINVIEW_ON
        BINVIEW_DATA
        BINNING_ON
        BINNING_FILE
        KRED_ON
        KRED_N
        KRED_DEPTH
        LRED_ON
        LRED_HISTEP
        LRED_LOSTEP
        LRED_INDTAGS
        LRED_DEPTH
        LRED_KGRIDS
        Q.LRED_N
 5.10 Thermal noise
        MEASNOISE_DO
        MEASNOISE_COVMAT 
        MEASNOISE_SCFAC
        CALINOISE_DO
        CALINOISE_COVMAT
 5.11 Retrieval/error/random quantities
        RETRIEVAL METHOD
        CLS_SPECIES_POS_ON
        CLS_NONLIN_ON
        CLS_RECALC_ABS_ON
        CLS_RECALC_WFS_NITER
        CLS_GA_START_VALUE     
        CLS_GA_FAC_WHEN_OK     
        CLS_GA_FAC_WHEN_NOT_OK 
        CLS_GA_UPPER_LIMIT  
        CLS_STOP               
        CLS_MAX_ITER
 5.12 Conditional simulations
        NUMBER_DO
 5.13 Retrievals
        RETRIEVAL METHOD
        CLS_SPECIES_POS_ON
        CLS_NONLIN_ON
        CLS_RECALC_ABS_ON
        CLS_RECALC_WFS_NITER
        CLS_GA_START_VALUE     
        CLS_GA_FAC_WHEN_OK     
        CLS_GA_FAC_WHEN_NOT_OK 
        CLS_GA_UPPER_LIMIT  
        CLS_STOP               
        CLS_MAX_ITER
 5.14 Measurement 
        VALID_CHANNELS; 
        VALID_CHANNELS_RANGE; 

=== 5.1 Report levels ===

QP_LEVEL The report level for the Qpack functions (beside messages
directly from ARTS). The number 0 means no reports, while the numbers
1-3 mean increasing level of verbosity.  The function out.m in AMI is
used for printing inside the Qpack functions.

ARTS_LEVEL 
ARTS is called with this report level for the screen.


=== 5.2 Names and directories ===

ARTS
The full path to the ARTS executable to run, e.g. 
/u/patrick/arts/src/arts

ARTS_ERROR_IGNORE
If set to 0, an ARTS error will also result in a Matlab error. That is,
the Matlab execution will stop. If set to 1, Matlab will only give a
warning and will try to continue to the execution (but this will not
always be possible). The latter can be an advantage when performing
batch inversions. For some cases qpcls can manage to continue after
an ARTS error. Default is 0.

QPACK_WARNING_IGNORE
If set to 1, Qpack will ignore the warnings Qpack is exiting on. To
date: The check if a matrix is singular exists if the matrix is
considered nearly singular. This is an somewhat arbitrary measure in
matlab.

OUT
Name or full path for storing pre-calculated quantities. For example:
        OUT = 'out'
        OUT = '/u/patrick/SMR/out'

USE_HDF
Flag to use binary files (based om HDF4) for arts output. This option
is somewhat faster but requires that arts is compiled with hdf.

TMP_AREA
All the calculations are performed in temporary sub-folders to
TMP_AREA. The name of the temporary folders is TmpXXXXXX where
XXXXXX is a random number. The sub-folder is deleted when the
calculations are finished.

SPECTRO_DIR
Full path of the directory for line files and definition of continua.

CALCGRIDS_DIR
Full path of the directory where the calculation grids are stored
(f_mono, za_pencil and p_abs).

SENSOR_DIR
Full path of the directory for sensor files.

RETRIEVDEF_DIR 
Full path of the directory for grids and definition of covariance 
matrices.


=== 5.3 Atmospheres ===

APRIORI_VMR
Full path for the a priori VMR profiles. There shall exist a file 
for all needed species. This data are only used during the actual 
inversion or when generating random data.

APRIORI_PTZ
Full path for a priori pressure, temperature and altitude file.
The full file name, including extension, shall be given, this in
contrast to APRIORI_VMR where only the main path shall be 
specified.

SELTAGS_VMR
Full path for the a priori VMR profiles for the tags specified in SELTAGS.
This is to handle 'raw_vmrsReadFromFiles' arts method. 

SETUP_VMR
SETUP_PTZ
One or several atmospheres to be used during the pre-calculations. 
The format for the VMR profiles and the PTZ files follows the a 
priori atmosphere (APRIORI_VMR_DIR and APRIORI_PTZ). Each atmosphere 
corresponds to a string in an ARTS string array (see RETRIEVAL_TAGS).
When setting up grids and similar operations, all given atmospheres
are considered to avoid "over-fitting" to a special atmospheric
condition. When only one atmospheric state can be considered, the
first atmosphere is selected. See below for its use with random data.


==== 5.4 Tag groups ===

RETRIEVAL_TAGS
The tag groups to be retrieved. The tag groups shall be given as
a Matlab string containing an ARTS string array (also called a
string-string). For example: 
        RETRIEVAL_TAGS = '"O3","ClO"'

SELTAGS
A sub group of tags which should be read from files. This corresponds 
to arts seltags variable.
        SELTAGS = '"O2"'
Default is SELTAGS empty.

OTHER_TAGS
The tag groups beside the retrieval tags. Format as above.


=== 5.5 Spectroscopy ===

Information about the spectroscopic calculations is found in the AUG
chapter "Gas absorption".

LINEFORMAT  
Format of the database. This is to handle the case when the spectral 
data is from other catalogs.
Possible competition: 'Hitran', 'Jpl','Myran2', and 'Arts'              
e.g. Q.LINEFORMAT  = 'Mytran2', reads from a Mytran format;
If no specified, then a 'Arts' format is expected.

LINEFILE
Name of file defining which transitions to consider for the line-
by-line calculations. The line file should be in the ARTS format.

LINESHAPE
LINESHAPE_FACTOR
LINESHAPE_CUTOFF
Line shape variables. Corresponds to the line-shape workspace variable
(WSV) in ARTS.

CONTINUA
The name of the file defining continua. This file shall look like the
part of an ARTS control file where the continua are described. The
cont_descriptionInit command shall not be part of the file.


=== 5.6 Hydrostatic equilibrium and RTE ===

Information about the RTE calculations is found in the AUG
chapters "Basic radiative transfer" and "Line of sight 1D".


HSE_IN_ON
Boolean to ensure that the atmosphere fulfils hydrostatic 
equilibrium. 

HSE_RETRIEVAL_ON 
Boolean to tell if hydrostatic equilibrium shall be assumed for
temperature retrievals. If this field is set to 1, temperature
weighting functions are calculated with hydrostatic eq. This field
only affects the calculation of weighting functions. If z_abs shall be
modified after an inversion to ensure that HSE is fulfilled, this is
governed by HSE_IN_ON. Notice that is possible to calculate the
temperature weighting functions without hydrostatic equilibrium but
still forcing the retrieved atmosphere to fulfil HSE.

HSE_PREF
Pressure of the reference point for hydrostatic equilibrium.

HSE_ZREF
Altitude of the reference point for hydrostatic equilibrium.

PLATFORM_ALTITUDE
Corresponds to the ARTS WSV z_plat.

STEPLENGTH_RTE
Corresponds to the ARTS WSV l_step.

GROUND_ALTITUDE 
GROUND_EMISSION 
Corresponds to the ARTS WSVs z_ground and e_ground, respectively.  
If GROUND_ALTITUDE is set to be empty, the ARTS WSM groundAtBottom is 
called.
Otherwise, groundSet is used where the ground temperature is obtained
by interpolating t_abs to the ground altitude.

REFRACTION_ON
Boolean to turn on refraction.

REFR_METHOD
The method to use for calculation of refractive index. It is assumed
that the method does not need any specific input. Allowed so far
'Unity', 'Boudoris', and 'BoudourisDryAir', see AUG for more info.
For example: 
        REFR_METHOD = 'Boudouris'

REFR_LFAC
Corresponds to the ARTS WSV with the same name.

EMISSION_ON
Boolean to include emission in the calculations.


=== 5.7 Calculation grids ===

P_ABS
F_MONO
ZA_PENCIL
The file (with extension) in GRID_DIR defining p_abs, f_mono and 
za_pencil, respectively. The files must be in ascii format.


=== 5.8 Sensor ===

Information about the sensor calculations is found in the
AUG chapter "Sensor modelling".

ANTENNA_ON
Boolean to consider the antenna pattern.

ANTENNA_ZA
The zenith angles observed by the sensor (the middle points of 
the  antenna pattern). This file is only read if an antenna is
considered. If the antenna is neglected, the viewing angles given 
by ZA_PENCIL are used to define the observation direction

ANTENNA_FILE
The file in SENSOR_DIR defining the antenna pattern. See
hAntennaFromFileAdv for file format.

ANTENNA_ORDER
Treatment of the antenna  pattern, linear (=1) or cubic (=3). 

ANTENNA_MOVE
Movement of the antenna (unit is degrees) during the integration.
A constant scanning velocity is assumed. 
The value 0 means a fixed antenna during the integration.
If the movement is the same for all zenith angles, ANTENNA_MOVE
can be set to be a vector of length 1. Otherwise ANTENNA_MOVE 
shall have the same length as ANTENNA_ZA.

DSB_ON
Boolean to consider the image sideband.

DSB_FILE
The file in SENSOR_DIR defining the response of the sideband 
filter. See hMixerFromFile for file format.

DSB_FPRIMARY
Some frequency of the primary band (to indicate if the upper or
lower band is the primary band).

DSB_LO
The LO frequency.

BACKEND_ON
Boolean to consider the backend (spectrometer).

BACKEND_FREQS
The frequencies recorded by the sensor (the middle points of 
the backend channels). This file is only read if a backend is 
considered. If no backend and mixer are included, F_MONO defines 
the frequencies of the spectral values. If no backend 
is included, but a mixer, the frequencies of F_MONO inside the 
primary band will be set as the observation frequencies.
  
BACKEND_FILE
The file in SENSOR_DIR defining the backend channel response.
The response of all channels are assumed to be identical.
See hBackendFromFile for file format.

BACKEND_IF
Boolean to determine if backend frequencies are given in RF
or IF. Default is 0, which means that RFs are given.
To use BACKEND_IF: DSB_LO must be set, and the upper band 
must be set to be primary band (DSB_FPRIMARY > DSB_LO).


FREQSWITCH_ON
Boolean to include a frequency switch procedure.
If selected, BACKEND_IF must also be selected and DSB_LO be
specified.

FREQSWITCH_THROW
The size of the frequency switch throw. This is the total size
of the throw. The actual throw is +/-FREQSWITCH_THROW/2. This means
that the two line "images" will be placed symmetrically around the 
nominal transition frequency. 
If frequency throw is included, the backend must have channels 
equally spaced in frequency. The throw must be a multiple of the
channel spacing.

FREQSWITCH_REVERSE
The default is that the line appearing at the lower frequency is 
weighted negetivaly (thus looking as an absorptionb line). This 
can be changed be setting this field to 1. 


=== 5.9 Data reduction ===

Information about the data reduction is found in the
AUG chapter "Data reduction".

BINVIEW_ON
Boolean to turn on binning of subsequent spectra.

BINVIEW_DATA
Corresponds to the input vector bins of hBinView.

BINNING_ON
Boolean to turn on binning of neighbouring channels.

BINNING_FILE
Full path of file defining the binning pattern. See hBinFile 
for file format.

KRED_ON
Boolean to turn on data reduction by the Kx-matrix version
of the Hotelling transformation. The basic idea is to
do a Hotelling transformation of the spectral space. But as
the spectral variability can be decomposed as:

                 Sy = KxSxKx' + KbSbKb' + Se

and we are interested in capture the variability of the 
parameters to be retrieved (x), the transformation is 
calculated only from that variability:  

                KxSxKx' = E V E'

where E are the eigenvectors and V the eigenvalues.
The Hotteling transformation is derived as:      
          
1. Notice that
    KxSxKx' = Kx sqrt(Sx) sqrt(Sx)' Kx' = Kx sqrt(Sx) (Kx sqrt(Sx))'
where sqrt(Sx) is one of the possible square roots of Sx.          
2. Then a SVD
             Kx sqrt(Sx) = U M V'   
where U and V are left and right singular vectors.
3. The eigenvectors are then U as from (1):
                    E V E' = U M V' (U M V')' = U M M U'

KRED_N
Number of eigenvectors to keep in the data reduction.

KRED_DEPTH
How to do the SVD of Kx sqrt(Sx) during reduction, possibilities: 
Different assumptions are implemented for sqrt(Sx):
  1.Q.KRED_DEPTH = 0 
                    sqrt(Sx) = I 
  i.e. Sx is assumed to be the identity matrix
  2.Q.KRED_DEPTH = 1 
                  sqrt(Sx) = sqrt(diag(Sx))
  i.e. Sx is assumed to be diagonal
  3.Q.KRED_DEPTH = 2 
                  sqrt(Sx) = chol(Sx)
  i.e. full Sx is used and the sqrt is done by a Choleski
  decomposition.
 
LRED_ON
Boolean to turn on data reduction by the limb version
of the Hotelling transformation. By limb it means
a special reduction based on combining the reduction based
on Kx with a prior optimisation of the spectral input 
from a limb observation regarding each specific point of a 
retrieval grid. This reduction is useful for a retrieval
method where each element of the retrieved species state 
vector is obtained independently, so the spectral input
in the limb observation can be optimised for each element.
The basic idea is to decide which elements for the spectral
input are left for each retrieval point, and then to proceed
with a normal Hotelling transformation of the new spectral
space. In practice is done by doing the Hot transf in a Kx
padded with zeros to leave only the rows (part of the scan)
and columns (the retrieval tag depending on LRED_INDTAGS) 
corresponding to each retrieval tag and point.

LRED_HISTEP
This parameter gives the upward distance from the retrieval point
to select what upper part of the scan to include in the reduction.
If a tangent altitude corresponding to a zenith angle is there, the
corresponding spectrum is kept.

LRED_LOSTEP
As above but to determine the downward distance.

LRED_INDTAGS
Flag, 1 only the Kx corresponding to the retrieval tag is
left, 0 the whole Kx is used.

LRED_DEPTH
As KRED_DEPTH
 
LRED_KGRIDS
This gives the retrieval grids where the reduction method 
(e.g neural nets) will be doing the retrieval. Notice that
they can be different from SPECIES_KGRIDS in a Q structure
use to generate rnd data (to allow the calculation of Kx
with SPECIES_KGRIDS in a fine grid, independent of the
retrieval grid).

LRED_N
As KRED_N


=== 5.10 Thermal noise ===

Some information about the thermal noise is found in the
AUG chapter "Measurement errors".

MEASNOISE_DO
Treatment of measurement thermal noise (0-2).

MEASNOISE_COVMAT 
The file specifying a covariance matrix to generate measurement
noise. The file data shall have three columns, where the first column
holds frequencies and the second column gives the standard deviations
and the third one gives correlation length (in frequency as frequency
is the abcisa here. Linear interpolation is applied to get values for
other frequencies (but no extrapolation).  See further the AMI
function sFromFile.

MEASNOISE_SCFAC
Scaling factors for the standard deviation of the measurement thermal 
noise. The thermal noise covariance values read from MEASNOISE_COVMAT 
are multiplicated with factors squared (i.e. si_new = factor * si).
The length of this vector shall be the same as for ANTENNA_ZA when
the noise shall be scaled. For no scaling, just don't define the
field or make the vector empty ([]).

CALINOISE_DO
Treatment of calibration noise (0-2). This noise is correlated 
between subsequent viewing angles, in contrast to the measurement 
noise.

CALINOISE_COVMAT
As MEASNOISE_FILE but for the calibration noise.


=== 5.11 Retrieval/error/random quantities ===

SPECIES_KGRIDS
The retrieval grid for each RETRIEVAL_TAGS as a string-string.
For example: 
        SPECIES_KGRIDS = '"grid1.aa","grid2.aa"'
The grids shall be defined in files found in RETRIEVDEF_DIR.

SPECIES_COVMATS 
Definition file of Sx for each retrieval tag as a string-string.  
The file format of these files are specified in the on-line help 
for the AMI method sFromFile. The altitude coordinate when 
specifying the covariance matrices is the 10-log of the pressure,
or decades. A decade is about 16 km. The std is given in relative
units e.g 0.3 means 30% of species apriori concentration.

TEMPERATURE_DO
Treatment of temperature (0-3).

TEMPERATURE_KGRID
The file defining the retrieval/error grid for temperature.

TEMPERATURE_COVMAT
The file defining the temperature covariance matrix. Notice that
the std is given in absolute units e.g. K and not in relative
units as the species.

TEMPERATURE_FAST
Flag to select a faster version of the function to calculate weighting
functions if hydrostatic equilibrium is turned on (HSE_RETRIEVAL_ON=1).
Default is here 1.

TEMPERATURE_RANGE
Allowed range of retrieved temperatures. A vector of length 2.
Temperatures outside this range, coming from the retrieval method, are
modified and set to the closest temperature limit.
Default is [120 32] K.

POINTING_DO
Treatment of pointing off-sets (0-3).

POINTING_STDV
Standard deviation for the pointing off-set if 'gaussian' or
half amplitude of pdf if 'uniform', realizations assumed to be
centred around the nominal zenith angle.

POINTING_DELTA
Perturbation to apply for calculation of the weighting function.
Unit is degrees.

POINTING_PDF
Treatment of pointing offset pdf, 'gaussian' and 'uniform' can be used
to fill this field. This field is only considered when generating
random data sets. For inversions and error characterisation, the PDF
is implicitly set to gaussian by the use of covariance matrices.

POINTING_COR
Set to 1, pointing error is perfectly correlated, i.e. it's a pointing
offset; set to 0, totally uncorrelated. So far no more refined schemes
set (e.g linear drifts).  Only considered for generation of random
data sets. A perfect correlation is assumed elsewhere.

FREQUENCY_DO
Treatment of frequency off-sets (0-3). This option should be used with
CLS_RECALC_ABS_ON = 1. 

FREQUENCY_STDV
Standard deviation for the frequency off-set [Hz].

FREQUENCY_DELTA
Perturbation to apply for calculation of the weighting function.
Unit is Hz.

CONTABS_DO
Treatment of continuum absorption (0-3). 

CONTABS_ORDER
Polynomial order for the continuum fitting.

CONTABS_LIMITS
A vector of length 2 specifying the frequency limits of the fit.
A -1 in the first position of this vector means that the lower limit
shall be set to the first value of F_MONO, and -1 in the second
position means that the upper limit shall be set to the last value of 
F_MONO. 

CONTABS_KGRID
The file defining the retrieval/error grid for continuum 
absorption.

CONTABS_COVMAT
The file defining the continuum absorption covariance matrix.
The standard deviations shall be given in percentages with
respect to the absorption of the species in CONTABS_REF_SPECIES. 
The given percentages are converted to absorption values [1/m]
by calculating the absorption of the reference species and
multiplicate the percentages with the absorption at the fitting
points.

CONTABS_REF_SPECIES
Reference species when defining the standard deviations in the
covariance matrix for continuum absorption. See further above
for CONTABS_COVMAT.

EGROUND_DO
Treatment of ground emission (0-3). 

EGROUND_LIMITS
A common value for the ground emission is used inside each specified
frequency range. Values outside the range of F_MONO can be given.
The values are given as a vector and gives the starting point of
each range, beside the last value that gives the end point of the 
last range. Range 1 ends where range 2 starts etc. The ranges for
EGROUND_LIMITS=[f1 f2 f3 f4] are: 
        f1 <= f < f2
        f2 <= f < f3
        f3 <= f <= f4
An example: 
        if the ground emission is assumed to be
        different in primary and image bands, but to be constant 
        inside each band, EGROUND_LIMITS can be set to [0 f_lo 1e99]
        where f_lo is the LO frequency.
If parts of F_MONO are outside the specified ranges, no retrieval of
the ground emission will be performed for those parts. If
EGROUND_LIMITS is set to be empty, this means that each frequency
shall be treated separately.

EGROUND_COVMAT
Definition file of Sx/Sb for the ground emission. Standard deviations
and correlation lengths are given as functions of frequency. The mean
of the first and last element of F_MONO inside each range given by
EGROUND_LIMITS is used when interpolating the given values.

EGROUND_MINMAX
A vector with length 2 specifying the lower and upper limit for allowed
values on the ground emission. If the vector is set to be empty, all
values are allowed.
These limits are applied only during the iterations of an inversion.
When convergence has been reached, the retrieved values are returned
even if they are outside the given min and max values. The min/max values
can accordingly be used to keep the ground emission inside the range
0 and 1 during the iterations to avoid unphysical conditions.

POLYFIT_DO
Treatment of polynomial fit to baseline ripple (0-3). 

POLYFIT_ORDER
Order of polynomial fit.

POLYFIT_COVMATS
Definition file of Sx/Sb for each polynomial coefficient, given 
as a string-string. The number of files shall be POLYFIT_ORDER+1.
The standard deviations are given in intensity unit. The
correlation length refers to the zenith angles and the unit is
accordingly degrees.

PPOLYFIT_DO
PPOLYFIT_ORDER
PPOLYFIT_COVMATS
PPOLYFIT_LIMITS
Variables for piecewise polynomial baseline fit. The variables
PPOLYFIT_DO, PPOLYFIT_ORDER and PPOLYFIT_COVMATS act as the
corresponding POLYFIT variables. All the polynomials have the
same order and a priori uncertainty. The variable PPOLYFIT_LIMITS
is a vector with the limits between the different polynomials. 
The end limits must be specified and the length of PPOLYFIT_LIMITS
is accordingly the number of polynomials + 1.
The limits are defined as for EGROUND_LIMITS.

PSINEFIT_DO     
PSINEFIT_PERIODS
PSINEFIT_LIMITS 
PSINEFIT_COVMATS
Implemented are only PSINEFIT_DO = 0 or 3 (ignore or retrieve). 
Retrieves a sinodial baseline piecewise. PSINEFIT_LIMITS corresponds to
PPOLYFIT_LIMITS and PSINEFIT_COVMATS corresponds to
PPOLYFIT_COVMATS. 
PSINEFIT_PERIODS contains the frequncies of the baseline components 
in Hz.  


TB_REFLOADS_DO
Treatment of loads used for load switching calibration (0-3).  It is
assumed that a calibration procedure is performed by observing a cold
and a hot load, and that the cold one is also used as reference load
during the observations of the atmosphere. The cold load corresponds
to load 1 and the hot load to load 2 in Equation 71 on page 250 in:
  Eriksson, P., Microwave radiometric observations of the middle
  atmosphere: Simulations and inversions, Ph.D. thesis, School of
  Environmental Sciences, Chalmers University of Technology, Sweden,
  1999.
Note that the load temperatures here correspond to V^1
and V^2 (I^1 and I^2 are the nominal values). The load are here
denoted as the cold and hot load, but the actual temperatures can be
reversed. As for PROPCAL, the baseline is not affected by calibration
imperfection.

TB_REFLOADS_NOMINAL
TB_REFLOADS_STDV
Nominal brightness temperature and a priori uncertainty for the
loads. Both variables shall be a vector of length 2. The first value
corresponds to the cold load, and the second to the hot load.

PROPCAL_DO
PROPCAL_TB_REF
PROPCAL_STDV
Treatment of proportional calibration error (0-3). These fields deal
with a calibration error that is proportional to the spectra, with a
variable zero point. The baseline is not affected by the calibration
imperfection. That is, we have
   ym = (1+c)*(y-y0) + bl, 
where bl is the baseline ripple, y0 the reference point, given by
PROPCAL_TB_REF, y is the true spectrum, c the calibration error and ym
the obtained spectrum.  The a priori uncertainty for c is given by
PROPCAL_STDV in units of brightness temperature.

Q.PSF_DO
Treatment of pressure shift parameter (0-3).

Q.PSF_FILE
Name on file defining which transitions to consider for 
pressure-shift retrieval(error analysis). The allowed number of 
species to consider is 1, but the number of transions is not 
restricted to 1. The linefile should be in the ARTS format 
and be put in SPECTRO_DIR.

Q.PSF_STDV
Standard deviation for the pressure-shift parameter.
The unit is Hz.

Q.PSF_DELTA
Perturbation to apply for calculation of the weighting function.
The unit is Hz.

Q.FREQ_CALIB_DO 
Treatment of frequency calibration uncertainty/error (0-3).
It can include translation and/or stretch of the monochromatic 
frequency grid to be fitted polynomially. 
This is a generalization of FREQUENCY_DO.
This option should be used with CLS_RECALC_ABS_ON = 1. 

Q.FREQ_CALIB_ORDER 
Order of polynomial fit.

Q.FREQ_CALIB_COVMAT
The file defining the covariance matrix. The first column 
holds frequencies and the second column gives the standard
deviations, and the third column gives the correlation length 
(This last may not be useful, but is introduced to use the 
same format as others.). The unit is Hz. See sFromFile in more 
details. The file must be put in RETRIEVEDEF directory.

Q.FREQ_CALIB_DELTA 
Perturbation to apply for calculation of the weighting function.
The unit is Hz.

SPECTRO_DO
Flag to calculate the spectroscopic parameters weighting function
SPECTRO_TAGS 
Specify the tags for whose lines the weighting function should be 
calculated. This is very useful for the case when one wants to omit 
a tag from the calculation (e.g., HNO3 for which the number of lines 
is very large), or to calculate only the weighting function for 
particular lines.
INTENS_ON         -- to calculate it for the line intensities
POSITION_ON      -- to calculate it for the line position;
AGAM_ON            -- to calculate it for agam ;
SGAM_ON            -- to calculate it for sgam ;
NAIR_ON              -- to calculate it for nair ;
NSELF_ON           -- to calculate it for nself;
PSHIFT_ON          -- to calculate it for pressure shift; 

Q.SAVE 
Additional field to save the the weighting function for different spectro parameters. This is very useful in the case when one the number of lines for which the weighting parameters should be calculated is very large and one wants to calculate the weighting function for each parameter separated. All the other quantities are calculated only once and they are save under the name given by Q.OUT.  The default value for Q:SAVE is set to Q.OUT. 

An example of the control file to calculate the weighting function and to camculate/plot the spectroscopic error is given in Samples/SampleWfss.


=== 5.12 Conditional simulations ===

NUMBER_DO 
If different from 0 a random set following the statistics given in the
_DO variables will be calculated. Those _DO variables larger than 0
are included in the calculations, but notice than so far only species,
temperature, pointing offset, polynomial of order 0 and measurement
noise can be part of random realizations.  
NUMBER_DO gives the number of rnd realizations for each variable. The
number of spectra created is also NUMBER_DO, i.e for spectrum i the i
realizations for each variable defined a state i used to generate
spectrum i. If SETUP fields are given (see below 6.3 Generating random
sets) NUMBER_DO is a vector with as meany elements as SETUP
atmospheres, each element gives the number of realizations for that
atmosphere.


=== 5.13 Retrievals ===

Retrievals by optimal estimation (OEM), Tikhonov regularization (TR) 
and ordinary (weighted) least squares (WLS) are here handled with the 
same functions as far as possible. The name constrained least squares 
is used as a common name for the methods (WLS is not a CLS method, but 
is included here for simplicity). NOTE that so far only OEM is working.

RETRIEVAL METHOD
String with the acronym of the retrieval method to use. 
Allowed options are so far:
        'oem'

CLS_SPECIES_POS_ON
Flag to include a positive constraint for the species retrieval.  The
constraint is obtained by retrieving the log of the species
profiles. This feature is only activated for non-linear retrievals or
when generating random realizations. In the last case, a lognormal
pdf is used to generate the species realizations instead of the
normal pdf.
If set to a scalar, the given value is applied to all retrieval species.
For example, [1] means a positive constrain is applied for all species.
Otherwise, the field must be a vector with length equalling the
number of retrieval tag groups (e.g. [0 0 1 0]).

CLS_NONLIN_ON
Boolean to turn on nonlinear retrievals. The Marquardt- Levenberg
scheme is used for the iterations. See qp_oem for more details.
All CLS variables below are only needed for non-linear inversions.

CLS_RECALC_ABS_ON
This flag determines if the absorption will be recalculated for
each state during non-linear inversions, or if the absorption
will only be scaled by the species amounts. The latter option is
of course faster. Default is RECALC_ABS_ON=1.
This field is neglected if temperature is retrieved. With temperature
as an inversion variable, absorption must be recalculated for each
iteration. This flag can also be used when, for example, the
self-broadening of water can change notably between the a priori and
true states.

CLS_ABS_SPECIES_ON 
Absolute vmr will be retrieved (the old Qpack
retrieved the ratio x/x_a instead) if set to one. Is this option set
to one, CLS_WFS_UNIT will be set to 'vmr'.

CLS_RECALC_WFS_NITER
This field specifies the number of iterations between recalculation of
the weighting functions. If the weighting functions are not
recalculated for an iteration, the last calculated values are used.
All values for the field <= 1 are treated as 1, which means that the
weighting functions are recalculated for each iteration. This is also
the default value.

With CLS_RECALC_WFS_NITER>1, time is saved by not calculating the
weighting functions for each iteration, but this is counteracted by
the fact that more iterations can be needed, and there will probably
be more failures (requiring an increase of gamma) during the
iterations. If time is in fact saved, and that convergence is still
obtained, must be determined by experiment.

CLS_WFS_UNIT The unit in which the weigthing function are
calculated. Allowed values are 'frac' and 'vmr'.

CLS_GA_START_VALUE     
Start value for gamma in the Marquardt-Levenberg iteration.

CLS_GA_FAC_WHEN_OK     
The factor with which gamma is decreased when an iteration is
succesful.

CLS_GA_FAC_WHEN_NOT_OK 
The factor with which gamma is increased when an iteration is not
succesful.

CLS_GA_UPPER_LIMIT  
When gamma equals or is above this value, the iterations are
stopped. 

CLS_STOP               
Stop criterion for the iteration. The quantity used to check if
convergence is reached is the change of the state vector from one
iteration to next, scaled with Sx and the length of x. See qp_oem
for details. 

CLS_MAX_ITER
An upper limit on the number of iterations to be performed.


=== 5.14 Measurement ===

VALID_CHANNELS 
Turns on rejection of unwanted channels of a measurement 
by restricting a minimum and maximum allowed brightness temperature 
or opacity. Only Option: 'Tb'. Option 'opacity' not yet implemented. 
 
VALID_CHANNELS_RANGE
Corresponding range (e.g.  = [-10., 175.]). Channels with values outside 
this range are removed from the inversion, simply by setting spectra (modeled 
and measured) and weighting functions to 0. Options: 'Tb', brightness temperature 
range in [Kelvin]. 




===== 6. Qpack strategy  ===============================================

 6.1 Doing a measurement characterisation
 6.2 Doing an optimal inversion
 6.3 Generating random sets


=== 6.1 Doing a measurement characterisation ===

The _DO fields are treated as: 
- 0 the variable is ignored, i.e. no error is calculated
- 1 calculates error as Dy K S K' Dy' but it's not
    treated as observation error, i.e it was not
    included in the derivation of Se and Dy
- 2 treats the variable as observation error, i.e
    it is included in Se and used in deriving Dy,
    and of course the error is calculated as 1.
- 3 the variable will be retrieved, so it's part
    of the vector state x and the retrieval error 
    will be calculated by adding the contribution
    from all the variables treated as 1 and 2.

The statistics for the characterisation are described by the _COVMATS
and _KGRID fields.

Use qpcls_invchar where H, Sx and Seinv must be pre-calculated. The
following sequence will do the job:
        > Q = qpack('your_q_fun');
        > qp_H(Q); 
        > qp_Sx(Q); 
        > qp_Se(Q); 
        > qpcls_invhar(Q); 


=== 6.2 Doing an OEM retrieval ===

The retrieval variables are given by _DO = 3 and the variables treated
as observation errors by _DO as 2. Variables _DO 1 or 0 are ignored
here. 

The retrieval grids are given by _KGRID fields, and the statistics to 
construct the Sx matrix are taken from _COVMATS.

The typical sequence is:
        > Q = qpack('your_q_fun');
        > qp_H(Q); 
        > qp_Sx(Q); 
        > qp_Se(Q); 
        > X = qpcls(Q,y); 
where y is the spectrum/scan, or a simulated measurement. To generate the
spectrum/scan corresponding to a Q, do the following:
        > Qy = qpack('another_q_fun');
        > qp_H(Qy); 
        > y = qp_y(Qy); 


=== 6.3 Generating random sets ====

NUMBER_DO gives the number of realizations. Variables with _DO fields
> 0 are disturbed around the a priori state with the variability given
by _COVMAT (so far only species, temperature, pointing offset,
polynomial baseline of order 0, and measurement noise). _KGRID fields
are only used if a reduction based on Kx is done, in which case the
retrieval grid is needed.

APRIORI_VMR_DIR
APRIORI_PTZ
SETUP_VMR_DIRS
SETUP_PTZ
If no SETUP fields are given the APRIORI fields give the atmosphere where
to take the species profiles to be randomly modified. If SETUP fields are
given the APRIORI fields will be ignored. The idea is to generate sets of
profiles-spectra where the profiles correspond to a more than one mean
state. This can get a bit tricky, for instance if as part of the sensor
a reduction based on Kx is done, the APRIORI fields will be used as 
a priori atmosphere when generating the Kx, so it might be a good idea if
the APRIORI atmosphere is a mean state from the SETUP atmospheres.

The typical sequence is:
        > Q = qpack('your_q_fun');
        > qp_rnd_atm(Q); 
        > qp_rnd_sensor(Q);
        > qp_rnd_atmxsensor(Q);
where qp_rnd_atm creates pencil beam spectra, qp_rnd_sensor creates
the random realizations for the sensor part, qp_rnd_atmxsensor applies
the sensor to the pencil beam spectra.

Another possible sequence is:
        > Q = qpack('your_q_fun');
        > qp_H(Q);
        > qp_rnd_atm_mem_red(Q); 
        > qp_rnd_sensor_mem_red(Q);
        > qp_rnd_atmxsensor_mem_red(Q);
where the "memory reduction" functions are used. This is recommended
to generate random realizations when the sets to generate are very
large and memory problems occur. The trick is to apply the H matrix as
soon as possible and not at the last step as it is done in
qp_rnd_atmxsensor after using qp_rnd_atm and qp_rnd_sensor. Notice
that the H matrix has to be pre-calculated. For this to be effective
of course the Hd part of H has to correspond to a large reduction,
e.g. eigenvector reduction by LRED_ON or KRED_ON. The drawback of
these new functions is that all intermediate spectra and sensor
realizations are saved in the transformed space, so it is not easy to
have a feeling about what it has been produced. Therefore, if memory
problems is not an issue it is still advisable to use the previous set
of functions.
