Differences

This shows you the differences between two versions of the page.

--- chara:trouble_shooting [2022/01/18 22:32]
charaobs
+++ chara:trouble_shooting [2022/01/21 19:07]
charaobs
@@ Line 26: / Line 26: @@
   * Check the instrument alignment. Is flux getting through to the detector? How long has it been since the last NIRO camera alignment? Classic and CLIMB programs can run for about an hour before the light will drift from the central pixel. Use the Classic or CLIMB gui to view the light on the pixels by clicking the PICTURE tab and then the PIXEL AREA button. Turn the camera off with the STOP button. Is the right dither power turned on? CLIMB 1 and Classic use different dithers. If Classic or CLIMB fringes are found in a scan, but not when in recording mode, the dither powers are likely not on. Are the camera settings correct for the seeing conditions and flux levels?
+===== The new OPLE system =====
+With the implementation of the new ople system which replaced the VME in Fall of 2021, new troubleshooting issues have arisen.  Since there are now 6 new ople computers to run the carts for each scope individually, many issues will be specific to one computer or telescope and not to the system as a whole.  Restarting the VME is now a thing of the past.
+The traditional ople server will still be used to communicate with each new ople computer, identified as OPLE 1 to OPLE 6.  When the communications are good, each active cart will be displayed in the ople server or ople gui status tab as before.  At times, an ople computer can lose communications or a server can crash and the server or comms needs to be restarted.
+If a cart cannot be started, stopped or otherwise commanded, look to the OPLESystem gui to see if a green indicator has turned red.  A message will also often pop up on the ople gui saying the command could not be sent. This may require a simple start command to restart the server or a reboot of the computer to get it back to yellow and then a start to get it back to green.  Do either of these steps with a right click of the red button and then select start or reboot.  After a reboot, select start to load the servers after the indicator has turned yellow. When this is done, the ople server will need to be connected to the newly restarted server.  Type "oo" into the ople server to open ople comms with the new ople server.  It should now reappear on the ople server display and say System Ready to indicate comms are restored.
+Some times a cart is stopped and cannot be commanded.  If the cart has gone to the front hard or back hard switch, it will not be usable until it is moved from the switch and the Ople Controller box is reset.  There are 6 silver boxes for these controllers with two green LED's for the front and back switches and two red LED's for the back hard and front hard switches.  If a red LED is lit, there will be an error displayed on the message window and the cart is disabled.  The cart will need to be moved ofo the switch and then the box can be reset with the RESET button on the front.  The error display will go away and the red LED will be off.  The cart is now controllable.
 ===== Restarting Servers =====
@@ Line 97: / Line 107: @@
 <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>6. Click OVERRIDE OFF, go to the AUTO tab and press AUTO, then NEXT (also in the obsgtk; this will restore the original star's demand position to the scope).</font>
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7. When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90.  The LED will show red when on the limit switch and is tripped, ie. limiting motion of the scope.  The LED will be yellow if it has tripped and is in the caution range, but not on a limit switch.  A fine Allen key can be used to push the internal reset button. It will turn the LED green when restored.</font>
+<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7. When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90. The LED will show red when on the limit switch and is tripped, ie. limiting motion of the scope. The LED will be yellow if it has tripped and is in the caution range, but not on a limit switch. A fine Allen key can be used to push the internal reset button. It will turn the LED green when restored.</font>
 ==== The Telescope won't track ====
@@ Line 109: / Line 119: @@
 ==== Dome Server Restart ====
-Dome servers are now started using the bootlaunch_master command.  Kill the server by finding the process ID with bootlaunch_master and restart it with bootlaunch_master also.  The manual process that has been superceded is listed below.
+Dome servers are now started using the bootlaunch_master command. Kill the server by finding the process ID with bootlaunch_master and restart it with bootlaunch_master also. The manual process that has been superceded is listed below.
 To manually start the dome server:
@@ Line 160: / Line 170: @@
 You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scopes position.
-==== Telescope clock is not correct ====
-The NTP server has obsoleted this problem and fix as of 2014.
 ==== Telescope is not receiving the commanded position for a target ====
@@ Line 185: / Line 191: @@
 ==== Telescope is tracking poorly, overshooting in slew, oscillating. ====
-This might mean that the gain for the tracking servo is wrong. Note that changing this gain can be dangerous, especially if you set it too high as that can cause the telescope to oscillated and damage the drives. Please only do this if you are very very sure that it is necessary. Symptoms of bad gain are: The scope over shoots the position while slewing. The star will be seen to move out of the window and may come back after a few seconds. This means the slewing gain is too low. The scope oscillates when tracking or after a slew. The star will be tracing an ellipse, figure eight or other looping shape. This means the tracking gain is too low. You can damp this out with the telescope or dome gui by disabling the scope, then re-enabling it. Adjust the gain upward and watch it on the next slew. In all cases if either gain is too high the scope will go into "Fog Horn" mode, which is bad. You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high. On 10-22-2016, the gain settings were:
+This might mean that the gain for the tracking servo is wrong. Note that changing this gain can be dangerous, especially if you set it too high as that can cause the telescope to oscillated and damage the drives. Please only do this if you are very very sure that it is necessary. Symptoms of bad gain are: The scope over shoots the position while slewing. The star will be seen to move out of the window and may come back after a few seconds. This means the slewing gain is too low. The scope oscillates when tracking or after a slew. The star will be tracing an ellipse, figure eight or other looping shape. This means the tracking gain is too low. You can damp this out with the telescope or dome gui by disabling the scope, then re-enabling it. Adjust the gain upward and watch it on the next slew. In all cases if either gain is too high the scope will go into "Fog Horn" mode, which is bad. This can be seen during slews on the twfs or labao as vibrating spots, usually in one axis.
-|Scope|AZ Slewing| |AZ Slewing| |AZ Tracking| |EL Tracking| |Date Updated|
+You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high.
-| |Gain|Fn|Gain|Fn|Gain|Fn|Gain|Fn| |
-|S1|7|4|4|3|16|4|10|7|10-26-2017|
-|S2|7| |7| |10| |10| |10-22-2016|
-|E1|7| |7| |13| |10| |10-22-2016|
-|E2|7| |7| |13| |10| |10-22-2016|
-|W1|13| |10| |13| |10| |10-22-2016|
-|W2|7| |7| |13| |13| |10-22-2016|
- \\
+The usual values for slewing gains are 4-7 and tracking gains are 7-10. Note that these values may change as the temperatures change rapidly. Gains are usually higher when cold and lower when warm. Be sure to set mode back to AUTO if changing the gains left it in Slewing or Tracking mode.
-Note that these values may change as the temperatures change rapidly. These are still being adjusted so this is by no means final. Be sure to set mode back to AUTO if changing the gains left it in Slewing or Tracking mode.
 ==== How to Adjust CPUMotor Gains ====
@@ Line 240: / Line 238: @@
 Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope's position. If not, it will insist on being in the wrong place. If it is not at the same AZ as the scope, manually move it until it is centered on the telescope in spycam 1. If the dome AZ does not read the same as the telescope AZ, enter the scope AZ in the position box of the DOME tab of appropriate dome server and hit the INIT POS button to tell it at what AZ it is. \\  \\
-If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker. Sometimes the drive wheel jumps in the track and cannot turn the dome, even when the motor works.  This will need to be fixed during the day.
+If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker. Sometimes the drive wheel jumps in the track and cannot turn the dome, even when the motor works. This will need to be fixed during the day.
 ==== Dome does not open ====
@@ Line 285: / Line 283: @@
   * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror.
-  * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has stopped on the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes.  The X does not guarantee the cart has retained its home position.  This can only be done by using the CHECK button on the ople gui or by typing the command "homechk S1" into the ople server.  When the error value is displayed on the ople server, hit ESC to clear the display. Click on the [OL] and [MAN] buttons and hit [TRACK] to send the cart to the desired cart position.
+  * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has stopped on the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. The X does not guarantee the cart has retained its home position. This home position can be checked by using the CHECK button on the ople gui or by typing the command "homechk S1" into the ople server. When the error value is displayed on the ople server, hit ESC to clear the display. Click on the [OL] and [MAN] buttons and hit [TRACK] to send the cart to the desired cart position.
 ===== Beam Samplers =====
@@ Line 460: / Line 458: @@
  \\
-Last updated 2017-04-12
+Last updated 2021-01-18 by Norm Vargas

The CHARA Array Wiki

User Tools

Site Tools

Differences

Page Tools