User's guide to interactive SOLVE

L. Petrov


Contents:
Abstract

This manual describes using the program Solve for analyzing geodetic VLBI observations in interactive mode. It contains some recipes for using this program.


Introduction

Solve is a program for analyzing geodetic VLBI observations which was developed in the 1970s at the Goddard Space Flight Center and since then it has been under continuous development through joint efforts of NASA/GSFC, the U.S. Naval Observatory, and University of Bonn. Solve is a tool for Solve supports both interactive and
batch [1] modes. It can be used in the following modes:

Customization

Before starting to use Solve the first time you have to customize your environment. Don't try to use Solve without proper customization! You will waste your time.
  1. Shell. tcsh is recommended. csh shell is also tolerable. Although Solve is able to run under either shell, this manual doesn't cover the topics of how to use Solve under other shells.

  2. Check your path variable. It should contain directories where system executables and Solve executables are located. Consult your system administrator.
  3. Environment variables. Solve has a plenty of configuration parameters. There are three levels of configuration:
    • Pre-compiled system-wide default.
    • User defaults specified by environment variables.
    • Current settings.
    Definition of user defaults are gathered in a special file. If you don't have it you should copy the file solve_env.template to your home directory, rename it, read and edit. Then add execution of this file in your startup shell file ~/.tcshrc :
    source {file_name}

    Environment variable DISPLAY should be set up properly. Refer to X11 user guide.

  4. X-resources. Some Solve programs use a PGPLOT library of graphic utilities which in turn invokes XSERV or pgxwin driver. It requires proper setting X-resources. You should add the following lines to the file ~/.Xdefaults if you use a big screen: 340x280mm pgxwin.Win.geometry: 1260x800+0+90 pgxwin.Win.maxColors: 69 pgxwin.Win.iconize: True or if you use a small screen: 300x230mm: pgxwin.Win.geometry: 1000x680+0+90 pgxwin.Win.maxColors: 69 pgxwin.Win.iconize: True

    Then you have to activate the settings defined in ~/.Xdefaults by the command

    xrdb -merge ~/.Xdefaults (NB: Environment variable DISPLAY should be set up properly before using xrdb). This operation should be done each time you start a terminal session, therefore you have to put the call of xrdb in your customization file, namely ~/.vueprofile .
  5. Solve initials. Solve creates a lot of scratch files. In order to allow more than one user to work simultaneously a two-letter code called "solve initials" is granted to each Solve user. Solve initials are defined in the file $SAVE_DIR/letok . Add a record for your initials if you don't have it.
  6. Creation of scratch files. Solve keeps intermediary results in so-called scratch files. In order to prevent abnormal termination when no free disk space left, Solve reserves space for scratch files from the very beginning and doesn't remove these files after termination. Size of these file is determined by a) maximum number of observations in the sessions to be analyzed; b) maximum number of parameters to be solved for.

    Run a program solve_reset if you don't have solve scratch files. NB: Scratch files should also be re-created after each Solve upgrade! You should also be sure that your environment variable WORK_DIR has been set before running solve_reset: echo $WORK_DIR . Usage of solve_reset:

    Usage: solve_reset xx max_obs max_parms Where xx = user's initials max_obs = maximum number of observations max_parms = maximum number of parameters Parameter max_obs should be set to twice maximum number of observations in one session if the first analysis is intended to be performed. max_param 2000 is sufficient for interactive runs.
  7. C-shell program sol. Script sol facilitates using Solve. It expects to find your Solve environment variable definition file in your home directory under the name {solve_initials}.solve_env where {solve_initials} consists of lower case symbols, e.g., px.solve_env if solve initials PX are in use.

Interactive Solve

Overview

Interactive Solve communicates with user by means of a set of menus. The majority of programs display menus in a text window and prompts you to enter a command. Each command has a one-letter code and an active field on the screen. A user has two ways to activate a command: either to position a cursor on the active field and then hit the key space bar or to hit a command code letter. Commands codes are enclosed in parentheses and emphasized as bold in this manual. Parentheses themselves are not a part of a command code. Many programs have sub-menus which operate the same way.

Solve requires hpterm window with size not less than 80x24 characters.

Solve consists of a set of programs which call each other. In order to launch interactive Solve enter a command

enter for example, enter PE The program OPTIN starts, it displays the main Solve menu and prompts for user input. You can terminate Solve by hitting a key (T) from the main menu, or by hitting a key (CNTRL/C) when Solve is waiting for user input. Menu settings remain untouched and you can resume analyzing the experiment after interruption. NB: be cautious with using (CNTRL/C) at the time when Solve is working: if you interrupt Solve during disk writing you may not be able to resume analysis. Solve expects to find VLBI observations and other intermediary data in the scratch files located in your working directory $WORK_DIR. Each scratch file has a suffix with solve user initials; this is what allows Solve to distinguish scratch files that belong to different users.

Solve scratch files contains information about one or more sessions. To learn the status of scratch files hit the key X from the main Solve menu. Solve reports the database name, version number, total number of observations (including no-detection), band and status.

Interactive Solve allows the user to fulfill the following operations:

Data loading

Solve is able to read the data in two formats: MARK-3 DBH and superfile format (the word superfile doesn't imply anything "superb" it is merely the name of a file format). The first operation in the data analysis is to read the experiment and to load it into the scratch files. Usually only one database or superfile should be loaded except the case when the data are analyzed the first time. In that case first X-band database and then S-band database of the same experiment should be loaded.

NB: If you load the experiment from the superfile then you are unable to update the database. Superfiles are loaded much faster than databases; they are used for BATCH runs and for special solutions.

Option (G) in the OPTIN menu invokes program SDBH, which loads the experiment in the scratch area. Option (X) inquires about the current status of scratch area.

Initial solution

The initial solution is carried out when the experiment is analyzed the first time. Purposes of the initial solution: It is assumed that The sequence of operations:

Initial settings

Set databases status. Hit (X) in OPTIN menu. Set YES in fields Include in solution?, Generate residuals? for X-band databases and set NO for S-band database, e.g. Databases in OBSFIL Name Vers # of Freq Include in Generate obs band solution? residuals? $93JUL21X 15 1862 X-BAND YES YES $93JUL21S 11 1914 S-BAND NO NO Set calibration status. Hit (%) in the main OPTIN menu. There are three menus: "Database calibration setup", "Flyby Station-Dependent Calibrations Status" and "Atmosphere Partials". Set P (applied) for cable calibration on the Database calibration page and V (not applied) for all others calibrations. Set P for NMFDRFLY and V for all others calibrations on the flyby station-dependent calibrations status page. Set "NMFWTFLY" on the atmosphere partials page.

Calibrations are set separately for each database. Check that calibration setup is exactly the same for both databases! Look at the database name at the upper right corner of calibration status setup. You can switch X-band and S-band databases by hitting (N) and (P) in "Database calibration setup" and "Flyby Station-Dependent Calibrations Status" menus. Atmosphere partials are always the same for both databases.

Check "Observation Dependent Calibration Status". Hit (+) in OPTIN menu. Calibrations "Pol Tide", "WobXCont", "WobYCont", "EarthTid", "Ocean" "PTideOld", "Axis 2" should be set up.

Setting initial parameterization

Go to the SETFL last page menu by hitting (L) in the OPTIN menu. Set estimation of site coordinates off by hitting (#), set all EOP flags to 0 (off), set nutation estimation flag to zero by hitting (.). Set solution type to "Group delay only" by hitting (+) and then (3). Then set clock parameters for all stations except the station taken as a clock reference. Hit the key E. Then you'll see the "site menu" of the SETFL program. It is irrelevant which station to take as a clock reference station at this step except the cases: a) there are two few good observations (say, less than 8) at that station; b) network is split onto two independent subnetworks. Set clock polynomial flags to 1 for the first three terms (clock shift, clock drift, frequency drift), f.e, Clock polynomials 93/07/21 19:59 1 1 1 0 0 * * * Be sure that all others parameters related to that station are disabled: atmosphere path delay, atmosphere gradients, axis offset. Keys (N) and (P) allow you to move to the next and the previous station.

Check once more your parameterization. Only clock polynomials of the 0,1,2 order should be setup. All other parameters should not be activated. Then run LSQ solution by hitting (Q). Look at the listing of the solution. Check once more that only clock polynomials are in the solution.

Check wrms. It should not exceed 1 microseconds. If the wrms exceeds that value it means that you have troubles, e.g. there are several very strong outliers.

Check clock offsets and rates (CL 0) and (CL 1). If there are stations with clock offsets greater by modulo than 10-4 sec -- 100 000 nsec and/or the stations with clock rate greater by modulo than 10-9 -- 100 000 D-14, you should apply an a priori clock model for that station. The number of digits in a float number presentation is not enough to handle the case when adjustments to clock parameters are too large. Rounding errors may corrupt results. To overcome this problem, an a priori clock model is added to the theoretical delays and delay rates. If you notice that clock offsets and clock rates exceed the limit for all stations, it means that the clock reference station itself has anomalous clock offset or rate. Change the clock reference station in that case. You can find preliminary values for a clock model from the correlation report if it is available.

Setting a priori clock values.

If all clock offsets and rates are below the limit you can skip this step.

Find all stations with anomalous clock offset and/or rate. Write down the values of clock and rate for these stations. Solve supports an a priori clock model for up to 4 stations. Go back to OPTIN menu. Then go to the last page of SETFL. Then hit (<). You will see the menu of SET_ACM program. Follow the SET_ACM manual [2]. Make one more LSQ solutions after applying a priori clock model. Clock parameters for the stations with applied clock model should be about zero.

Group delay ambiguity resolution.

The next step is to resolve group delay ambiguities at both bands. There are two ways to do it: manual procedure and automatic procedure GAMB. Of course, the automatic procedure should be used except the rare cases which it currently doesn't support.

Using GAMB.

GAMB is a complicated program which solves simultaneously four problems. It
  1. resolves group delay ambiguities at all baselines at both bands;

  2. determines preliminary parameters of clock polynomial model;

  3. rejects outliers;

  4. computes ionosphere calibration;

Refer to documentation of GAMB for details.

Default GAMB setup is as follows:

Before using GAMB check once more that calibration status for the X-band and S-band databases is exactly the same. Then hit key (A) in order to initiate GAMB. In many cases it is enough for resolving ambiguities. In other cases GAMB issues you a warning. If GAMB doesn't issue a warning you can be sure that ambiguities were resolved correctly. Warning usually means excessive noise or presence of observations with sub-ambiguities. But GAMB may honestly confess that ambiguity resolution was unreliable or it may refuse to process the session at all.
Troubleshooting

Manual group delay ambiguity resolution (A. Nothnagel)

Manual group delay ambiguity resolution is an alternative technique. It is rather more time consuming and it is recommended only if you failed to resolve ambiguities automatically.

Steps of manual ambiguities resolution:

Group delay ambiguities are resolved.

The next step is to inspect residuals. Set estimation of baseline-dependent clocks: hit (C) from the menu of the last SETFL page. Menu of the program BCLOK will be displayed. First set all baselines by hitting (W), then deselect a clock reference station by hitting the station code.

Make a solution by hitting (Q). Look at the listing. Normally the total wrms should be in the range [500, 1500] psec. If it exceeds 2000 psec, it means that probably either ambiguity resolution was not successful or there are clock breaks at one or more stations.

You should check estimates of baseline-dependent clocks. If the estimates exceed 1 nsec, it is an indication of remaining permanent ambiguities at that baseline, i.e there are no jumps in ambiguities among observations at all baselines but all observations at some baselines have incorrect ambiguities what causes triangle misclosures to be a multiple of the ambiguity spacing. You have to get rid of permanent ambiguities.

Manual re-distribution of permanent group delay ambiguities
First you have to decide which band is affected by permanent ambiguities. If baseline-dependent clocks has an adjustment by a multiple of the group delay ambiguity spacing at X-band -- then X-band. A permanent ambiguity at S-band will contribute by fX/fS = 12 times less. Thus, if you see baseline-dependent clock estimates less than one group delay ambiguity spacing but still larger than 1 nsec, it is an indication that there are S-band permanent ambiguities.

Set solution type X-band only or S-band only on the last SETFL page in accordance with the band affected by permanent ambiguities. Make a solution. Look at the list of baseline-dependent clocks. Find the first baseline with the estimate of the baseline-dependent clocks to be a multiple of group delay ambiguity spacing. Make plot of residuals of that baseline by hitting (P) at the OPTIN menu. Then use the function Set amb. shift. Set the correct sign of ambiguities: the sign should be the same as the sign of the baseline-dependent clock adjustments if the order of stations in the baseline both in the listing and in CNPLT is the same. The sign is opposite if the order of stations in the listing and in CNPLT is different. Then use the function Shift multi pts. CNPLT will ask you to point to the Starting point and then the End point. Click at the point in the very bottom of the plot in order to mark the "Starting point", then click the point in the very top of the plot in order to mark the "end point". Then you make a solution once more. Repeat this procedure if needed with the next baseline. Don't forget to set the solution type back to G-GXS combination after all these manipulations.

Sometime you have observations with ambiguities less than the group delay ambiguity spacing. These are sub-ambiguities and they are caused by a wring choice of maximum in the delay resolution function. There is no way to resolve sub-ambiguities in Solve. You have to suppress all observations with sub-ambiguities. It is still possible to recover at least part of such observations by running the program mk4fit for those observations, but this topic goes beyond this manual.

Inspection of residuals
Now you have to inspect residuals baseline by baseline. The purpose is to check quality of data, check whether the ambiguities were resolved correctly and check whether clock breaks have to be inserted. Call program CNPLT by hitting (P) from the OPTIN menu. (NB: CNPLT conflicts with some X-applications which grabs colors, like Netscape. You should close such applications before running CNPLT.)

If you find a jump in the plot of residuals you may try to insert a clock break. Be sure that it is not a jump in ambiguities at X- or S-band. In order to insert a clock break hit (E) in the main OPTIN menu, then list station's pages by hitting (N) or (P) till you find the station where you are going to insert clock breaks. Then hit (*) several times until you see a line insert. Then hit (C) and enter the time tag of the clock break. Then a new epoch for clock polynomial appears at the SETFL page for that station. Set the first three flags to 1 for the new clock break. Then make a new solution and check the listing and residual plots. Keep in mind that there should be enough observations between the start of the session, clock break(s) and the end of the session. There should be no less than 4 observations, otherwise your solution will be unstable or singular. If it seems to you that the session has many clock breaks, it may indicate another serious problems unrelated to clock behavior.

If you have a station with too few good observations (less than 5 at each baseline), or you have a station with postfit residual scatter larger than 5 nsec you can deselect it. But it is a last resort. In general you should try to keep as many stations/baselines as possible in the initial and intermediary solutions, and to leave the final decision to the time of the final solution. An analyst is able to select/deselect station/baseline in any time during further solutions including batch runs, however the data should be edited properly, otherwise selecting the baselines which have been suppressed during the initial solution might degrade the solution due to the presence of outliers.

Intermediary solution

Intermediary solution is carried out upon completion of the initial solution when the experiment is analyzed the first time. The purpose of the intermediary solution is to remove strong outliers at the earlier steps. The intermediary solution has incomplete parameterization. Such a parameterization facilitates suppression of outliers or observations which look like as outliers.

Settings for intermediary solution:

Now make a solution by hitting (Q). Look at the listing. Check parameterization: you should estimate a) station positions (except the reference station), b) clocks; c) atmosphere path delay; d) baseline-dependent clocks and nothing more.

Now the time came to start outlier elimination/restoration. Go to the main OPTIN page and then hit (\). You will see the menu of program ELIM/MILE for outliers elimination/restoration. There is extensive user documentation about ELIM/MILE:

  1. Description of menu items. [3]
  2. Outlier elimination algorithm. Full text. [4]
  3. ELIM/MILE. Release note. [5]
  4. Hints of using ELIM. [6]
  5. Description of observations suppression strategy. [7]
Set the following menu items for outliers elimination in the intermediary solution: (X) Maximum uncertainty: 1000. psec (A) Acceleration factor: 1 (U) Upper threshold for outlier detection: 400. psec (C) Cutoff limit for outlier detection: not specified (Y) Type: baseline
(Q) Quality code limit: 5 (D) Update residuals (-) Singularity check (') Change suppression method (V) Verbosity level: 1 (N) Confirm each action: no

Set the first line to elimination by hitting (T). Eventually hit (P) for going ahead. ELIM will suppress outliers. In order to quit ELIM with saving results hit (S).

Check your solution.

If you have a poor or bad solution you have to find the reason. First, check a) calibration; b) parameterization; c) clock breaks. Then examine the residuals. Check clock breaks and the resolution of group delay ambiguities. If you are sure that there is a malicious station which spoils the entire experiment you may try to deselect this station.

If you find that your intermediary solution is good, or at least you find the reason why it is poor, you can move to the final solution.

Final solution

Final solution is carried out either during the first analysis of the experiment or during re-analysis of the data. It is assumed that initial and intermediary solutions have already been made. The final solution may be done for different purposes. One of the objectives is to obtain a so-called quick solution. The purpose of a quick solution:
  1. Initial settings for the final solution:

  2. Make solution and look at the listing. Check whether the parameterization which you would have liked to have has actually been applied.
  3. Make the first outliers elimination. Go to the main OPTIN menu and hit (\). Set maximal uncertainty 1000 psec; upper threshold for outlier detection not specified (you should hit (U) and then enter zero); set cutoff limit for outlier detection: 3.0 sigma. Normally, parameter Acceleration factor should be 1. If you are in hurry and the database is too long (e.g., more than 5000 observations) you can increase the acceleration factor (e.g. to set 8, 16, 32 even 64). The acceleration factor specifies how frequently the residuals should be updated. E.g. acceleration factor 1 means that the residuals should be updated after elimination of each outlier (or restoration each previously suppressed observation), acceleration factor 4 means that residuals will be updated only after elimination/restoration of each 4 observations. The larger acceleration factor, the faster ELIM works, but large acceleration factor will suppress some portion of good observations. So it is better to avoid using the acceleration factor greater than 1.

    Then update weights upon ELIM completion. Hit (W) on the ELIM menu. You will see UPWEI menu. Set floor to 10.0 psec by hitting (L) and then hit (I). Then go back ELIM by hitting (O) in UPWEI menu and execute outliers elimination once more. If you notice that chi/ndg went noticeably away from 1.0 (e.g. below 0.95), update weights once more.

  4. Now after removing the main portion of outliers it is time to examine solution more carefully.

  5. Final outliers elimination/restoration. Go to the main OPTIN menu and then hit (\) in order to invoke ELIM. Set maximum uncertainty to 1000 psec; set upper threshold for outlier detection to zero (you will see not specified in the menu), set cutoff limit for outlier detection to 3.0 and set acceleration factor to 1. Hit (P) to initiate the process.

    Update weights after ELIM completion. Hit (W) at the ELIM menu. You will see UPWEI menu. Set floor to 10.0 psec by hitting (L) and then hit (I). Then go back to ELIM by hitting (O).

    Now hit (T) to set restoration mode. Set cutoff the limit of outlier detection to 4.0 sigma (or 3.5 sigma if the session has more than 5000 observations) and then proceed with restoration of the observations which were previously suppressed by hitting the key (P).

    Then set elimination mode, set the cutoff limit to 3 sigma and eliminate outliers once more. After outliers elimination update weights and proceed with outliers elimination once more.

  6. It may occur that you need to get adjustments of positions of some source(s) regardless of how bad they are. Of course, you are able to do it only if you have at least two observations. If ELIM suppressed these observations it may not restore them since their residuals are too large. In that case invoke program CNPLT by hitting (P) from the main OPTIN menu. Read the schedule file for that experiment, find all these observations and restore them manually. Then set the flag for estimation of coordinates of those sources and make a new solution. Then you may wish to eliminate outliers among these observations. One of the options is to do it manually in CNPLT, another way is to do it semi-automatically in ELIM. You can set the flag Confirm each action by hitting (N) in ELIM menu. ELIM will ask for confirmation before suppression of each observation in this mode.
  7. The last thing remaining to do is to play with cable calibration. Go to "Database calibrations status" menu by hitting (%) from the main OPTIN menu. Deselect cable calibration for all stations: hit repeatedly (/) to get Don't apply to any station status and then hit a station code.

    Set user partial program CABLE_PART: hit (<) on the main OPTIN menu and then enter the name of the program: CABLE_PART. Run a solution. Cable cal admittance will be computed for all stations in this mode. Admittance about 1.0 means that the cable cal is OK, admittance -1.0 means that the cable cal is OK but its sign is wrong. Values around zero indicates that cable cal is probably wrong. If you find that estimation of cable cal admittance improves fit by more than 2-3% (wrms is less, chi/ndg becomes less) you can decide to set the flag "not calibrate for cable cal" for some stations permanently. However as a rule of thumb you should leave cable calibration unless you have clear evidence that cable calibration at certain stations degrades fit.

    Don't forget to deactivate CABLE_PART program and to set the flag cable cal for all stations except the ones which it degrades fit. In order to deactivate user partial program CABLE_PART hit (<) from the main OPTIN menu again and them hit Return key in reply on Enter name of user partial program (Return if none):.

  8. Before the end of analysis look at the plots of segmented parameters once more. Call MDLPL_PLUS by hitting (/> from the main OPTIN menu. Look at all plots. You may wish to print them or to save as graphic gif-files for preparation of a Web analysis report. Printout is delivered in the directory /tmp/ as a default. You may change the prefix of the file name by hitting (C) in the internal menu. NB: MDLPL_PLUS merely prepends the the contents of the line Web directory before the filename. Therefore the prefix should contain a trailing "/" if it is used as a directory name.

Database update

Some parameters related to the solution, such as group and phase delay ambiguities, suppression status, clock and atmosphere parameterization, baseline-dependent clock status, reweighting parameters can be saved in the database. Database update is the last step of analysis. As a result a new version of the database will be created. Hit (U) in the main OPTIN menu in order to invoke program NEWDB for updating the database. The program NEWDB will ask you which database you are going to update. As a rule of thumb you don't need to the update S-band database. GAMB and IONO add 21 items (lcodes) used in analysis from the S-band related database to the X-band related database.

Proceed to the next menu. Hit (D) unless you didn't resolve phase delay ambiguities:

Reweighting: (G)roup (P)hase (B)oth (#)None (1) Clk & atm parms, constraints, data configuration-->Yes No (2) Group delay editing and ambiguities--------------->Yes No (3) Group ionosphere calibration:--------------------->Yes No (4) Met., cable, phase cal status:-------------------->Yes No (5) Ocean, relativity, pole tide status:-------------->Yes No (6) Phase delay editing and ambiguities--------------->Yes No (7) Phase ionosphere calibration:--------------------->Yes No

If you resolved phase delay ambiguities then you need to hit (B), then hit (6) and then (7) in order to set Reweighting to Both and to set all other items to Yes.

Then proceed to the next menu and hit the space bar. An editor will be invoked and it asks you to edit the file with name /tmp/HIST{Solve_initials}. You can write your comments about this experiment in the history file. NB: lines should not exceed 78 characters. When you finish editing, leave the editor and proceed. The contents of the file /tmp/HIST{Solve_initials} will be added to the history list of the next version of the databases.

NB: The Catalogue system prohibits you from making two database updates of the same experiment repeatedly. It is assumed that you load the new experiment after database update. If you decided to continue to analyze the same database and you would like to save your settings after you have done the database update it is till possible. Use program CATLG in order to remove the last database version. (But check that the version number of the database which you are deleting is larger than the database version which is loaded in Solve!)
NB: If the database update completed abnormally (e.g, lack of disk space) you should remove the entry of the new database version by using CATLG. (But check that the version number of the database which you are deleting is larger than the database version which is loaded in Solve!)

References

Some user documentation related to Solve.
This document was prepared by Leonid Petrov

Last update: 01-MAY-2000 12:23:00