User Tools

Site Tools


chara:trouble_shooting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revision Both sides next revision
chara:trouble_shooting [2022/01/21 20:03]
charaobs
chara:trouble_shooting [2022/02/09 19:24]
charaobs
Line 9: Line 9:
  
   * Were the clocks synced? Make sure the [SYNC CLOCKS] button on Cosmic Debris has been pushed to start the night. If the OPLE server does not display the correct CHARA time and the errors don't read (0) or (1), the clocks were not synced.   * Were the clocks synced? Make sure the [SYNC CLOCKS] button on Cosmic Debris has been pushed to start the night. If the OPLE server does not display the correct CHARA time and the errors don't read (0) or (1), the clocks were not synced.
-  * Did the Astrolib update on OPLE? If the job queue is stopped too soon after slewing on Cosmic Debris, the correct calculations for the carts will not be done by OPLE and you may be searching for fringes with the wrong star data. Hit STAR ACQUIRED on CD to update ople. The proper star can also be entered manually by typing hd #### into the OPLE server and hitting ENTER. +  * Did the Astrolib update on OPLE? If the job queue is stopped too soon after slewing on Cosmic Debris, the correct calculations for the carts will not be done by OPLE and you may be searching for fringes with the wrong star data. The star identifier will be displayed in the server window. If it is not correct, hit STAR ACQUIRED on CD to update OPLE. The proper star can also be entered manually by typing hd #### into the OPLE server and hitting ENTER. 
-  * Are the PoP's correct? After a PoP change, the PoP's are sometimes not updated in CD or ople.+  * Are the PoP's correct? After a PoP change, the PoP's are sometimes not updated in CD or ople. Compare the PoP's in the configuration tab of CD or OPLE with the PoP Overview or Popperi gtk.
   * Are the carts behaving or are there vibrations or jumps of 100 or more microns every 3-6 seconds? How are the metrology signals? Are they strong and staying white? Red signals mean one or more metrology signals may have gone too low and homing carts is necessary. The METDATA function on the ople gui can be used to sample the metrology signal and may show noise spikes that can be disruptive to the smooth cart motion. Push the METDATA button on the configure tab and then again about 6 seconds later. Plots will pop up to show the frequency and power of any noise.   * Are the carts behaving or are there vibrations or jumps of 100 or more microns every 3-6 seconds? How are the metrology signals? Are they strong and staying white? Red signals mean one or more metrology signals may have gone too low and homing carts is necessary. The METDATA function on the ople gui can be used to sample the metrology signal and may show noise spikes that can be disruptive to the smooth cart motion. Push the METDATA button on the configure tab and then again about 6 seconds later. Plots will pop up to show the frequency and power of any noise.
   * Is the target a high proper motion star? Red dwarfs are close stars and can have high proper motions. Scan a wider range to see if it is outside of the usual calculated scan range. Binaries can also have very high offsets from the expected position due to mistakenly using astromod calculations from the companion star.   * Is the target a high proper motion star? Red dwarfs are close stars and can have high proper motions. Scan a wider range to see if it is outside of the usual calculated scan range. Binaries can also have very high offsets from the expected position due to mistakenly using astromod calculations from the companion star.
-  * Do you have enough flux from each telescope or on each baseline? Is the telescope tiptilt struggling with low flux? Can the camera gain be raised or the exposure made longer to help hold the star?+  * Do you have enough flux from each telescope or on each baseline? Is the telescope tiptilt struggling with low flux? Can the camera gain be raised or the exposure made longer to help hold the star? Passing clouds or contrails can lower flux unexpectedly.
   * Did you get the same star in each telescope? Sometimes a busy star field and poor pointing of the telescopes can lead to the wrong star being acquired and locked by tiptilt. View the stars in the finder window to see if all the stars match.   * Did you get the same star in each telescope? Sometimes a busy star field and poor pointing of the telescopes can lead to the wrong star being acquired and locked by tiptilt. View the stars in the finder window to see if all the stars match.
   * Check the CHARA time on the GPS server. The "Ext-CHARA," "CHARA-Sys," and "Ext-Sys" time offsets listed on the GPS server should be small (< 0.01 sec). If there are large time offsets, then a GSYNC might be needed.   * Check the CHARA time on the GPS server. The "Ext-CHARA," "CHARA-Sys," and "Ext-Sys" time offsets listed on the GPS server should be small (< 0.01 sec). If there are large time offsets, then a GSYNC might be needed.
   * Check the time on the OPLE server. If the time is off or there are any lost ticks/seconds "Lost T/S" listed, then MSYNC ople using the GPS GUI and type "syncople" into the ople server.   * Check the time on the OPLE server. If the time is off or there are any lost ticks/seconds "Lost T/S" listed, then MSYNC ople using the GPS GUI and type "syncople" into the ople server.
   * Are the [MAN] buttons pressed (gray) for the moving carts on the OPLE Control gui? (The reference cart will remain green.)   * Are the [MAN] buttons pressed (gray) for the moving carts on the OPLE Control gui? (The reference cart will remain green.)
-  * Check that the carts are within delay line range (-1 to 44 meters) and errors are small. Were the carts homed and checked before the first slew of the night? Did any carts go to the front switch after slewing? This can cause them to lose their position. Turn off the OL button to send it back to zero. If it does not track on the home switch when displaying a zero position (0.000000), it was lost. Is the E2 cart stuck at 27.97m? See section on homing carts in __OPLE and Metrology__  section below.+  * Check that the carts are within delay line range (-1.3 to 44.25 meters) and errors are small. Were the carts homed and checked before the first slew of the night? Did any carts go to the front switch after slewing? This can cause them to lose their position. Turn off the OL button to send it back to zero. If it does not track on the home switch when displaying a zero position (0.000000), it was lost. Is the E2 cart stuck at 27.97m? See section on homing carts in __OPLE and Metrology__  section below.
   * Are the LDC's working?   * Are the LDC's working?
       * Is the glass position within allowable range (-10mm to 49mm) on all beams?       * Is the glass position within allowable range (-10mm to 49mm) on all beams?
Line 90: Line 90:
 <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>4. Move the scope a bit back toward the direction it was coming from – for example, if the scope stopped at AZ 268 while rotating clockwise, move it back to 265 or so using AZ DEC. Then press STOP.</font> <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>4. Move the scope a bit back toward the direction it was coming from – for example, if the scope stopped at AZ 268 while rotating clockwise, move it back to 265 or so using AZ DEC. Then press STOP.</font>
  
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>5. Move the scope past AZ 270 by pressing AC INC. Usally 2-4 degrees will do.  Go past the point that shows the limit is triggered.</font>+<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>5. Move the scope past AZ 270 by pressing AC INC. Usally 2-4 degrees will do. Go past the point that shows the limit is triggered.</font>
  
 <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>6. Click OVERRIDE OFF, go to the AUTO tab and press AUTO, then NEXT (also in the obsgtk; this will restore the original star's demand position to the scope).</font> <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>6. Click OVERRIDE OFF, go to the AUTO tab and press AUTO, then NEXT (also in the obsgtk; this will restore the original star's demand position to the scope).</font>
Line 106: Line 106:
 ==== Dome Server Restart ==== ==== Dome Server Restart ====
  
-Dome servers are now started using the bootlaunch_master command.  The manual process that has been superceded is archived.+Dome servers are now started using the bootlaunch_master command. The manual process that has been superceded is archived.
  
 To restart the dome server: To restart the dome server:
  
-1. Make sure the power to the drives is OFF.  Disable the scopes.+1. Make sure the power to the drives is OFF. Disable the scopes.
  
 2. Login to the relevant computer as observe. For example, type "s1" to log on to S1. 2. Login to the relevant computer as observe. For example, type "s1" to log on to S1.
Line 116: Line 116:
 3. Work out the process ID number (PID) by typing bootlaunch_master 3. Work out the process ID number (PID) by typing bootlaunch_master
  
-4. Use kill -2 PID to kill the server.  Entering the command twice will show "No such process" if it has been killed.  if it is not killed, use kill -9.+4. Use kill -2 PID to kill the server. Entering the command twice will show "No such process" if it has been killed. if it is not killed, use kill -9.
  
 5. Type bootlaunch_master again to restart the server. 5. Type bootlaunch_master again to restart the server.
Line 125: Line 125:
  
 You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scope's position. You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scope's position.
 +
 +Be aware that when restarting a dome server, the telescopes position may not be retained and the dome gui may display Az: 0.0. El: 0.0.  The gui will scroll messages that say Az: 0.0. El: 0.0 with an error in scope position.  If the scope is known to be at STOW position, you can manually enter the scope's AZ and EL using the manual tab on the dome gui and hitting the INIT button.
 +
 +Also be aware that the gains do not always return to the normal values of Slew 4-7 and Tracking 7-10 when a dome server has crashed and is restarted.  They may come up as Gain 22 for all axes and modes and they will need to be turned down or the scopes will oscillate badly.
  
 ==== Telescope is not receiving the commanded position for a target. ==== ==== Telescope is not receiving the commanded position for a target. ====
Line 296: Line 300:
   * If the top of a server goes blank, try typing "sb" (start background) on the server command line.   * If the top of a server goes blank, try typing "sb" (start background) on the server command line.
   * If the server screen fills with jibberish, try hitting CTRL-L in the server to clear it.   * If the server screen fills with jibberish, try hitting CTRL-L in the server to clear it.
 +  * If a server freezes, sometimes hitting CTRL-C and then N, for NO, will unfreeze it.
  
 ==== Server is frozen ==== ==== Server is frozen ====
Line 305: Line 310:
   * Click [REOPEN] on Cosmic Debris and relevant GUIs to re-initialize communication with the server after it is restarted.   * Click [REOPEN] on Cosmic Debris and relevant GUIs to re-initialize communication with the server after it is restarted.
   * A folder is on the desktop that has the restart commands for CTRSCRUT servers and the Shutters server restart command running on OPLE. Use it to restart servers that will not reopen from the menu.   * A folder is on the desktop that has the restart commands for CTRSCRUT servers and the Shutters server restart command running on OPLE. Use it to restart servers that will not reopen from the menu.
 +  * if a telescope server does not close after using CTRL-C when restarting them after UT midnight and the Send ON button is green on the obsgtk, turn the Send ON to OFF and the server will close. Turn the Send ON button back on to resume the setup for the night.
  
 ==== PAVO Server - Error communicating with IFW ==== ==== PAVO Server - Error communicating with IFW ====
chara/trouble_shooting.txt · Last modified: 2023/11/21 01:42 by charaobs