Differences

This shows you the differences between two versions of the page.

--- chara:trouble_shooting [2022/02/09 19:10]
charaobs
+++ chara:trouble_shooting [2023/02/08 19:30]
charaobs [The Telescope won't move or stopped moving]
@@ Line 18: / Line 18: @@
   * Check the time on the OPLE server. If the time is off or there are any lost ticks/seconds "Lost T/S" listed, then MSYNC ople using the GPS GUI and type "syncople" into the ople server.
   * Are the [MAN] buttons pressed (gray) for the moving carts on the OPLE Control gui? (The reference cart will remain green.)
-  * Check that the carts are within delay line range (-1.3 to 44.25 meters) and errors are small. Were the carts homed and checked before the first slew of the night? Did any carts go to the front switch after slewing? This can cause them to lose their position. Turn off the OL button to send it back to zero. If it does not track on the home switch when displaying a zero position (0.000000), it was lost. Is the E2 cart stuck at 27.97m? See section on homing carts in __OPLE and Metrology__  section below.
+  * Check that the carts are within delay line range (-1.3 to 44.25 meters) and errors are small. Were the carts homed and checked before the first slew of the night? Did any carts go to the front switch after slewing? This can cause them to lose their position. To check the homing of the carts, turn off the OL button to send it back to zero. If it does not track on the home switch when displaying a zero position (0.000000), it was lost. There is a new "CHECK" button on OPLE gtk that will send carts to home switch and report whether the is an error in the cart position. Is the E2 cart stuck at 27.97m? See section on homing carts in __OPLE and Metrology__  section below.
   * Are the LDC's working?
       * Is the glass position within allowable range (-10mm to 49mm) on all beams?
@@ Line 26: / Line 26: @@
   * Check the instrument alignment. Is flux getting through to the detector? How long has it been since the last NIRO camera alignment? Classic and CLIMB programs can run for about an hour before the light will drift from the central pixel. Use the Classic or CLIMB gui to view the light on the pixels by clicking the PICTURE tab and then the PIXEL AREA button. Turn the camera off with the STOP button. Is the right dither power turned on? CLIMB 1 and Classic use different dithers. If Classic or CLIMB fringes are found in a scan, but not when in recording mode, the dither powers are likely not on. Are the camera settings correct for the seeing conditions and flux levels?
+  * There is a script that will display offsets for all scopes against Hour Angle to help find offsets when using MIRCX and MYSTIC. You'll need to log into the MIRCX spooler computer with the command ssh spooler@mircx, then use the command fringe_predictor. A window will pop up displaying the last five nights of offsets vs. HA.
 ===== The new OPLE system =====
@@ Line 35: / Line 36: @@
 If a cart cannot be started, stopped or otherwise commanded, look to the OPLESystem gui to see if a green indicator has turned red. A message will also often pop up on the ople gui saying the command could not be sent. This may require a simple start command to restart the server or a reboot of the computer to get it back to yellow and then a start to get it back to green. Do either of these steps with a right click of the red button and then select start or reboot. After a reboot, select start to load the servers after the indicator has turned yellow. When this is done, the ople server will need to be connected to the newly restarted server. Type "oo" into the ople server to open ople comms with the new ople server. It should now reappear on the ople server display and say System Ready to indicate comms are restored.
-Some times a cart is stopped and cannot be commanded. If the cart has gone to the front hard or back hard switch, it will not be usable until it is moved from the switch and the Ople Controller box is reset. There are 6 silver boxes for these controllers with two green LED's for the front and back switches and two red LED's for the back hard and front hard switches. If a red LED is lit, there will be an error displayed on the message window and the cart is disabled. The cart will need to be moved ofo the switch and then the box can be reset with the RESET button on the front. The error display will go away and the red LED will be off. The cart is now controllable.
+Some times a cart is stopped and cannot be commanded. If the cart has gone to the front hard or back hard switch, it will not be usable until it is moved from the switch and the Ople Controller box is reset. There are 6 silver boxes for these controllers with two green LED's for the front and back switches and two red LED's for the back hard and front hard switches. If a red LED is lit, there will be an error displayed on the message window and the cart is disabled. The cart will need to be moved off the switch and then the box can be reset with the RESET button on the front. The error display will go away and the red LED will be off. The cart is now controllable.
 ===== Restarting Servers =====
@@ Line 47: / Line 49: @@
 ==== Restarting Servers using the rc.local file ====
- \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm, which may not be any at this point. \\  \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence. \\
+\\
- \\
+This procedure is applicable to servers that have not switched over to the bootlaunch paradigm, which may not be any at this point.\\
-Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn't work then type "ssh //name//" where name is the computer name. \\  \\ Find out if the server is running by typing "ps aux | grep //server_name//" where server_name is the name of the server. \\ [ctrscrut:599] ps aux | grep pico_1 \\ observe 9281 0.0 0.0 61156 692 pts/3 S+ 13:58 0:00 grep pico_1 \\ observe 12578 0.0 0.0 24524 11212 ? S Jun16 33:14 /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg \\  \\ If the entry for the dead server shows up in the process list, then identify the process identification number (12578 for the example above) and kill the server by typing "kill -9 //PID//" where PID is the process identification number. \\  \\ Look up the commands to restart the server by typing "more /etc/rc.local" (this is relevant for servers that run in the background). Press the space bar to scroll through the contents of the rc.local file. Locate the commands relevant for the server that needs to be restarted and copy and paste into a terminal window: \\  \\ #Start PICO server for PICO #1 \\ /bin/rm -f /var/lock/LCK..ttyC8 \\ /usr/local/bin/tsockman remove PICO_1 \\ /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg & \\  \\ The first command removes the lock to allow the server to restart. The second command removes the name from the socket manager listing. The last command restarts the server. Note that if you are restarting the servers as observe, you will need to remove the part of the command in the rc.local file that saves information in /var/log///server_nam//e.log file (the actual command typed should resemble the last line above). \\  \\ There are text files on the desktop with many of the restart commands. Use these files for quick access to the relevant commands. The commands are edited and can be copied exactly as written. Files include Dome servers and all servers running on ctrscrut. Many of these commands are also located on the [[:chara:restarting_servers|Restarting Servers]] page.
+\\
+If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence.\\
+\\
+Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn't work then type "ssh //name//" where name is the computer name.\\
+\\
+Find out if the server is running by typing "ps aux | grep //server_name//" where server_name is the name of the server.\\
+[ctrscrut:599] ps aux | grep pico_1\\
+observe 9281 0.0 0.0 61156 692 pts/3 S+ 13:58 0:00 grep pico_1\\
+observe 12578 0.0 0.0 24524 11212 ? S Jun16 33:14 /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg\\
+\\
+If the entry for the dead server shows up in the process list, then identify the process identification number (12578 for the example above) and kill the server by typing "kill -9 //PID//" where PID is the process identification number.\\
+\\
+Look up the commands to restart the server by typing "more /etc/rc.local" (this is relevant for servers that run in the background). Press the space bar to scroll through the contents of the rc.local file. Locate the commands relevant for the server that needs to be restarted and copy and paste into a terminal window:\\
+\\
+#Start PICO server for PICO #1\\
+/bin/rm -f /var/lock/LCK..ttyC8\\
+/usr/local/bin/tsockman remove PICO_1\\
+/usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg &\\
+\\
+The first command removes the lock to allow the server to restart. The second command removes the name from the socket manager listing. The last command restarts the server. Note that if you are restarting the servers as observe, you will need to remove the part of the command in the rc.local file that saves information in /var/log///server_nam//e.log file (the actual command typed should resemble the last line above).\\
+\\
+There are text files on the desktop with many of the restart commands. Use these files for quick access to the relevant commands. The commands are edited and can be copied exactly as written. Files include Dome servers and all servers running on ctrscrut. Many of these commands are also located on the [[:chara:restarting_servers|Restarting Servers]] page.
 Shutters Server
- \\ The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on ople, not ctrscrut. \\  \\ To start the shutter server on ople: \\  \\ Log into the ople computer and kill the process labeled shutters with the PID as described in **Restarting Servers**  above. \\ Turn off the power to the Shutters with the switch on the computer rack which is to the left of the computer desk and marked "SHUTTERS". Restart the Shutters server with the commands below. After restarting the server and testing the gui to see that it works, turn the SHUTTERS power back on with the switch. There is a printed sheet of directions in the lab to help you. \\  \\ /usr/local/bin/tsockman rm shutters \\ ctrscrut/usr/local/bin/shutter_server /ctrscrut/chara/etc/shutter.cfg &
+\\
+The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on ople, not ctrscrut.\\
+\\
+To start the shutter server on ople:\\
+\\
+Log into the ople computer and kill the process labeled shutters with the PID as described in **Restarting Servers** above.\\
+Turn off the power to the Shutters with the switch on the computer rack which is to the left of the computer desk and marked "SHUTTERS". Restart the Shutters server with the commands below. After restarting the server and testing the gui to see that it works, turn the SHUTTERS power back on with the switch. There is a printed sheet of directions in the lab to help you.\\
+\\
+/usr/local/bin/tsockman rm shutters\\
+ctrscrut/usr/local/bin/shutter_server /ctrscrut/chara/etc/shutter.cfg &
+==== Restarting Socket Manager ====
+In very rare circumstances, the socket manager might need to be restarted. This is usually needed if GUIs stop communicating (e.g., OPLE GUI stops communicating with OPLE server). Another example occurs when mircx_bootLauncher tries to remove dead server names from the socket_manager before restarting. If the socket_manager does not respond, the mircx_bootLauncher will timeout.
+Follow these instructions to restart the socket manager, but only do so if really necessary:
+ssh -X observe@ctrscrut
+killall socket_manager
+socket_manager &
 ==== MIRC-X CredoneImAcq Server ====
@@ Line 75: / Line 120: @@
 Have the powers been turned on to the drives? Are the scopes disabled? The usual state of the telescopes is disabled until enabled. This is due to the stall function of the scopes which eventually disables the scopes when they are stowed. Enable the scopes by hitting [ENABLE] on the dome gui, the telescope gui control tab, or the bottom of the obsgtk. If a scope disables itself during a slew, it may be just an overcautious stall function or the slew gains may be too high or low. Check the gains and if they are OK, Hit ENABLE and the scope should continue to slew. If it disables again without moving, there may be something wrong at the scope, ie. a hatch left open, a ladder not put away, a tool on the floor, or something else physically impeding the motion of the scope. You will need to go to the dome to see what it is. The computer in the dome will give you control of the scope to turn it away from the problem.
-Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopes. Check them for current times and continuous updates of numbers. If they are not updating, try to REOPEN them first. If that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of it. See **Dome Server Restart**  below. \\  \\ **Azimuth Limit Switches** \\  \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>As of 11-'17, the azimuth limit switches are enabled and can stop the motion of the scopes if they try to go beyond -90º or 450º. The scopes will not be movable with normal inputs so follow these instructions to return them from the out of range condition. \\
+Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopes. Check them for current times and continuous updates of numbers. If they are not updating, try to REOPEN them first. If that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of it. See **Dome Server Restart** below.\\
 \\
-. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software. \\
+**Azimuth Limit Switches** \\
+\\
+<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>As of 11-'17, the azimuth limit switches are enabled and can stop the motion of the scopes if they try to go beyond -90º or 450º. The scopes will not be movable with normal inputs so follow these instructions to return them from the out of range condition.\\
+\\
+. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software.\\
+\\
+. Make sure you understand why the limit was hit which may require a trip to the telescope. If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch causing the stall and not that the scope is actually in a wrong position.</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way.\\
+\\
+<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope.\\
+\\
+. Click ENABLE then you can move the telescope back to its normal range of operation.\\
 \\
-. Make sure you understand why the limit was hit which may require a trip to the telescope. If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch causing the stall and not that the scope is actually in a wrong position.</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way. \\  \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope. \\  \\
-. Click ENABLE then you can move the telescope back to its normal range of operation. \\  \\
 . After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again and then hit AUTO on the AUTO tab to resume normal operation.</font>
 <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>However, If you notice that a telescope will stop near AZ 90 or 270 with the scope still being ENABLED and refusing to move, this is often due to the new AZ limit switch being armed around AZ 0º at some point earlier. To get it moving again:</font>
-<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software. \\  \\
+<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software.\\
-.</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way. \\  \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope.</font>
+\\
+.</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way.\\
+\\
+<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope.</font>
 <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>4. Move the scope a bit back toward the direction it was coming from – for example, if the scope stopped at AZ 268 while rotating clockwise, move it back to 265 or so using AZ DEC. Then press STOP.</font>
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>5. Move the scope past AZ 270 by pressing AC INC. Usally 2-4 degrees will do. Go past the point that shows the limit is triggered.</font>
+<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>5. Move the scope past AZ 270 by pressing AZ INC. Usally 2-4 degrees will do. Go past the point that shows the limit is triggered.</font>
 <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>6. Click OVERRIDE OFF, go to the AUTO tab and press AUTO, then NEXT (also in the obsgtk; this will restore the original star's demand position to the scope).</font>
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7. When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90. The LED will show red when on the limit switch and is tripped, ie. limiting motion of the scope. The LED will be yellow if it has tripped and is in the caution range, but not on a limit switch. A fine Allen key can be used to push the internal reset button. It will turn the LED green when restored.</font>
+<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7. When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90. The LED will show red when on the limit switch and is tripped, ie. limiting motion of the scope. The LED will be yellow if it has tripped and is in the caution range, but not on a limit switch. A fine Allen key can be used to push the internal reset button or use the magnet and touch the box. It will turn the LED green when restored.</font>
 ==== The Telescope won't track ====
@@ Line 125: / Line 182: @@
 You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scope's position.
+Be aware that when restarting a dome server, the telescopes position may not be retained and the dome gui may display Az: 0.0. El: 0.0. The gui will scroll messages that say Az: 0.0. El: 0.0 with an error in scope position. If the scope is known to be at STOW position, you can manually enter the scope's AZ and EL using the manual tab on the dome gui and hitting the INIT button.
+Also be aware that the gains do not always return to the normal values of Slew 4-7 and Tracking 7-10 when a dome server has crashed and is restarted. They may come up as Gain 22 for all axes and modes and they will need to be turned down or the scopes will oscillate badly.
 ==== Telescope is not receiving the commanded position for a target. ====
@@ Line 200: / Line 261: @@
 [[:chara:old_lab_tiptilt_server|Instructions for the old lab tiptilit system are archived here.]]
+===== LabAO =====
+==== LabAO Shutters Not Working ====
+If the labao shutters are not working, the power button on each of the two red control boxes on the top of the table might need to be pressed. After that the other button needs to be pressed and held for 5 seconds so that the units can be controlled remotely. (Note: look for instructions on table.)
+==== LabAO DM Electronics ====
+If a labao server complains that it failed to find the EDAC40 device, try unplugging the power and ethernet cables for DM and then replugging in again.
 ===== OPLE and Metrology =====
@@ Line 281: / Line 352: @@
   * If either Phase 1 or Phase 2 states say IDLE for a given VPN, then you need to re-establish the connection. To do this, click on the "DROP" button for that VPN. Then click on the "CONNECT" button and wait for the Phase 1 and Phase 2 connections to be established.
   * Make sure to close Firefox by using CTRL-Q to quit the program and not just close the window.
+===== Network Problems =====
+==== User accounts not recognized ====
+The gps computer does the user authentication for the control side while ctrscrut provides the home directories. Yellowpages is gone, the new paradigm is LDAP (local directory access protocol). If you can't log on to computers on the control network, then make sure both ctscrut (michelson) and gps computers are running. If the login screen screen on ctrscrut says "Ubuntu Administrator" and has no user accounts, then check to make sure gps computer is on. If you restart gps, then you might have to reboot ctrscrut to get the user accounts. If ctrscrut was rebooted, you might have to reboot other control network computers if you can't log on.
+If gps computer does not start, check the power distribution unit (PDU) at the top of that rack. If any units show both "+" and "-" then you can use the arrow buttons to move through the units and cycle the units to on "+". That should automatically start the computer connected to those units.
 ===== General Problems =====
@@ Line 306: / Line 385: @@
   * Click [REOPEN] on Cosmic Debris and relevant GUIs to re-initialize communication with the server after it is restarted.
   * A folder is on the desktop that has the restart commands for CTRSCRUT servers and the Shutters server restart command running on OPLE. Use it to restart servers that will not reopen from the menu.
-  * if a telescope server does not close after using CTRL-C when restarting them after UT midnight and the Send ON button is green on the obsgtk, turn the Send ON to OFF and the server will close.  Turn the Send ON button back on to resume the setup for the night.
+  * if a telescope server does not close after using CTRL-C when restarting them after UT midnight and the Send ON button is green on the obsgtk, turn the Send ON to OFF and the server will close. Turn the Send ON button back on to resume the setup for the night.
 ==== PAVO Server - Error communicating with IFW ====

The CHARA Array Wiki

User Tools

Site Tools

Differences

Page Tools