Differences

This shows you the differences between two versions of the page.

--- chara:trouble_shooting [2021/11/09 14:03]
gail_stargazer
+++ chara:trouble_shooting [2021/11/09 15:10]
gail_stargazer
@@ Line 53: / Line 53: @@
  \\
 Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn't work then type "ssh //name//" where name is the computer name. \\  \\ Find out if the server is running by typing "ps aux | grep //server_name//" where server_name is the name of the server. \\ [ctrscrut:599] ps aux | grep pico_1 \\ observe 9281 0.0 0.0 61156 692 pts/3 S+ 13:58 0:00 grep pico_1 \\ observe 12578 0.0 0.0 24524 11212 ? S Jun16 33:14 /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg \\  \\ If the entry for the dead server shows up in the process list, then identify the process identification number (12578 for the example above) and kill the server by typing "kill -9 //PID//" where PID is the process identification number. \\  \\ Look up the commands to restart the server by typing "more /etc/rc.local" (this is relevant for servers that run in the background). Press the space bar to scroll through the contents of the rc.local file. Locate the commands relevant for the server that needs to be restarted and copy and paste into a terminal window: \\  \\ #Start PICO server for PICO #1 \\ /bin/rm -f /var/lock/LCK..ttyC8 \\ /usr/local/bin/tsockman remove PICO_1 \\ /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg & \\  \\ The first command removes the lock to allow the server to restart. The second command removes the name from the socket manager listing. The last command restarts the server. Note that if you are restarting the servers as observe, you will need to remove the part of the command in the rc.local file that saves information in /var/log///server_nam//e.log file (the actual command typed should resemble the last line above). \\  \\ There are text files on the desktop with many of the restart commands. Use these files for quick access to the relevant commands. The commands are edited and can be copied exactly as written. Files include Dome servers and all servers running on ctrscrut. Many of these commands are also located on the [[:chara:restarting_servers|Restarting Servers]] page.
-==== Shutters Server ====
- \\ The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on ople, not ctrscrut. \\  \\ To start the shutter server on ople: \\  \\ Log into the ople computer and kill the process labeled shutters with the PID as described in **Restarting Servers**  above. \\ Turn off the power to the Shutters with the switch on the computer rack which is to the left of the computer desk and marked "SHUTTERS". Restart the Shutters server with the commands below. After restarting the server and testing the gui to see that it works, turn the SHUTTERS power back on with the switch. There is a printed sheet of directions in the lab to help you. \\  \\ /usr/local/bin/tsockman rm shutters \\ ctrscrut/usr/local/bin/shutter_server /ctrscrut/chara/etc/shutter.cfg &
-==== MIRC-X CredoneImAcq Server ====
-The MIRC-X CredoneImAcq server and GUI can be opened using the "credone" icon on the wolverine desktop. Note that you can only have one of these open at a time. If you get an error message and can't bring up credone, then close the error message and follow the steps below to remove the dead processes first:
-From a terminal window, log on to mircx (old observe password): \\ ssh -X spooler@mircx
-Look up credone process: \\ ps aux | grep credone
-Kill all credone processes (including the ones for the error messages) using "kill -9 PID" where PID is the process number. Note that credone can't be restarted if there are any error processes in the list. Check "ps aux | grep credone" one more time to make sure all processes are cleared (the only one that should show up is the grep command that was just issued).
-Then restart credone using the desktop icon or by issuing the following command on spooler@mircx: \\ credoneImAcq –no-display
-(the first "–" should be two dashes: - and -).
-===== Restarting Servers =====
-==== Restarting Servers using the bootlaunch paradigm ====
-If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list".
-To restart a server, log on to the machine that runs the server and type "bootlaunch_master". This script will go through the list of executables and will check which servers are running. If a server isn't running it bootlaunch_master will remove it from socket manager, clear the lock file, and relaunch the server. The instructions below describe how to restart individual servers, but this should not be necessary anymore. \\  \\ A number of servers use an interim bootlaunch paradigm to restart. This is confined to servers that run on ubuntu machines, namely the telescope bunker computers and gps. The basic syntax is "bootlaunch_<server>" where "<server>" is replaced by the server the script is designed to address. The scripts have a number of safeties built in, so it is safe to run them even if a server is already running – they just output the process ID of the running server. The scripts also take care of the entry in socket manager as well any serial port lock files. All the pertinent information is world writeable, so one should be able to run a bootlaunch script as observe. \\  \\ One thing of note about the output of the bootlaunch scripts, they call a number of other programs which themselves have output that may be misleading in the context of bootlaunch. Chief among these is the output of "tsockman". If a server stopped unexpectedly, it may leave behind an entry in the socket manager. In order to launch a new server, one needs to clean out the socket manager entry if it is there. To do that, "tsockman remove <entry>" is called to remove "<entry>" before the new server is launched. If there is no entry, tsockman will respond with "Process by that name does not exist". THIS IS NORMAL and is not indicative of an error. The server in question launched (without fanfare) right after that output text. \\  \\ Here are the available bootlaunch scripts as of June 2017: \\  \\ gps computer:
-  * bootlaunch_beamsamp – Starts the beam sampler servers, BS1 and BS2.
-  * bootlaunch_zaber – Starts the ZABER_2 server.
- \\ telescope bunker computers:
-  * bootlaunch_hut – Starts the E1_HUT, E2_HUT, S1_HUT, S2_HUT, W1_HUT, or W2_HUT server, depending on the machine it's launched from.
-  * bootlaunch_rpc – Starts the RPC_E1, RPC_E2, RPC_S1, RPC_S2, RPC_W1, or RPC_W2 server, depending on the machine it's launched from.
-  * bootlaunch_weather – Starts the E1_WEATHER, E2_WEATHER, S1_WEATHER, S2_WEATHER, W1_WEATHER, or W2_WEATHER server, depending on the machine it's launched from.
-  * bootlaunch_lower – Starts the E1_Lower, E2_Lower, S1_Lower, S2_Lower, W1_Lower, or W2_Lower cylinder server, depending on the machine it's launched from.
-  * bootlaunch_upper – Starts the E1_Upper, E2_Upper, S1_Upper, S2_Upper, W1_Upper, or W2_Upper server, depending on the machine it's launched from.
- \\ Note: The bootlaunch scripts will not start a new server if there is an existing process running. Therefore, type "ps aux | grep //server_name//" where server_name is the name of the server. If there is a dead process running, look up the process identification number (PID) and type "kill -9 PID" to kill the process and then run the relevant bootlaunch script.
-==== Restarting Servers using the rc.local file ====
- \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm. \\  \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence. \\
- \\
-Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn't work then type "ssh //name//" where name is the computer name. \\  \\ Find out if the server is running by typing "ps aux | grep //server_name//" where server_name is the name of the server. \\ [ctrscrut:599] ps aux | grep pico_1 \\ observe 9281 0.0 0.0 61156 692 pts/3 S 13:58 0:00 grep pico_1 \\ observe 12578 0.0 0.0 24524 11212 ? S Jun16 33:14 /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg \\  \\ If the entry for the dead server shows up in the process list, then identify the process identification number (12578 for the example above) and kill the server by typing "kill -9 //PID//" where PID is the process identification number. \\  \\ Look up the commands to restart the server by typing "more /etc/rc.local" (this is relevant for servers that run in the background). Press the space bar to scroll through the contents of the rc.local file. Locate the commands relevant for the server that needs to be restarted and copy and paste into a terminal window: \\  \\ #Start PICO server for PICO #1 \\ /bin/rm -f /var/lock/LCK..ttyC8 \\ /usr/local/bin/tsockman remove PICO_1 \\ /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg & \\  \\ The first command removes the lock to allow the server to restart. The second command removes the name from the socket manager listing. The last command restarts the server. Note that if you are restarting the servers as observe, you will need to remove the part of the command in the rc.local file that saves information in /var/log///server_nam//e.log file (the actual command typed should resemble the last line above). \\  \\ There are text files on the desktop with many of the restart commands. Use these files for quick access to the relevant commands. The commands are edited and can be copied exactly as written. Files include Dome servers and all servers running on ctrscrut. Many of these commands are also located on the [[:chara:restarting_servers|Restarting Servers]] page.
 ==== Shutters Server ====
@@ Line 119: / Line 74: @@
 ===== Telescopes and Dome Servers =====
- \\ Here we discuss things that can go wrong with the telescopes.
+Here we discuss things that can go wrong with the telescopes.
 ==== The Telescope won't move or stopped moving ====
@@ Line 125: / Line 80: @@
 Have the powers been turned on to the drives? Are the scopes disabled? The usual state of the telescopes is disabled until enabled. This is due to the stall function of the scopes which eventually disables the scopes when they are stowed. Enable the scopes by hitting [ENABLE] on the dome gui, the telescope gui control tab, or the bottom of the obsgtk. If a scope disables itself during a slew, it may be just an overcautious stall function or the slew gains may be too high or low. Check the gains and if they are OK, Hit ENABLE and the scope should continue to slew. If it disables again without moving, there may be something wrong at the scope, ie. a hatch left open, a ladder not put away, a tool on the floor, or something else physically impeding the motion of the scope. You will need to go to the dome to see what it is. The computer in the dome will give you control of the scope to turn it away from the problem.
-Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopes. Check them for current times and continuous updates of numbers. If they are not updating, try to REOPEN them first. If that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of it. See **Dome Server Restart** below. \\ \\ **Azimuth Limit Switches** \\ \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>As of 11-'17, the azimuth limit switches are enabled and can stop the motion of the scopes if they try to go beyond -90º or 450º. The scopes will not be movable with normal inputs so follow these instructions to return them from the out of range condition. \\ \\ 1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software. \\ \\ 2. Make sure you understand why the limit was hit which may require a trip to the telescope. If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch causing the stall and not that the scope is actually in a wrong position. \\ \\ 3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope. \\ \\ 4. Click ENABLE then you can move the telescope back to its normal range of operation. \\ \\ 5. After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again and then hit AUTO on the AUTO tab to resume normal operation
+Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopes. Check them for current times and continuous updates of numbers. If they are not updating, try to REOPEN them first. If that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of it. See **Dome Server Restart**  below. \\  \\ **Azimuth Limit Switches** \\  \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>As of 11-'17, the azimuth limit switches are enabled and can stop the motion of the scopes if they try to go beyond -90º or 450º. The scopes will not be movable with normal inputs so follow these instructions to return them from the out of range condition. \\
-<font 16px/Calibri,sans-serif;;#333333;;white>**However, If you notice that a telescope will stop near AZ 90 or 270 with the scope still being ENABLED and refusing to move, this is often due to the new AZ limit switch being tripped at some point earlier. To get it moving again:**</font>
+\\
+. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software. \\
+\\
+. Make sure you understand why the limit was hit which may require a trip to the telescope. If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch causing the stall and not that the scope is actually in a wrong position.</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way. \\  \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope. \\  \\
+. Click ENABLE then you can move the telescope back to its normal range of operation. \\  \\
+. After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again and then hit AUTO on the AUTO tab to resume normal operation <font 16px/Calibri,sans-serif;;#333333;;white>**However, If you notice that a telescope will stop near AZ 90 or 270 with the scope still being ENABLED and refusing to move, this is often due to the new AZ limit switch being tripped at some point earlier. To get it moving again:**</font>
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;white>1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software.</font>  <font inherit/inherit;;initial;;white>2. M</font>ake sure the all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way. <font inherit/inherit;;initial;;white>3. Click the OVERRIDE ON button in domegui MANUAL tab.</font>
+<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software. \\  \\
+.</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way. \\  \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope.</font>
+<font inherit/inherit;;initial;;white>2. M</font>ake sure the all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way. <font inherit/inherit;;initial;;white>3. Click the OVERRIDE ON button in domegui MANUAL tab.</font>
 <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>4. Move the scope a bit back toward the direction it was coming from – for example, if the scope stopped at AZ 268 while rotating clockwise, move it back to 265 or so using AZ DEC. Then press STOP.</font>
@@ Line 148: / Line 111: @@
 ==== Dome Server Restart ====
-To manually start the dome server: 1. Make sure the power to the drives is OFF. 2. Login to the relevant computer as root. For example, type "s1" or "ssh s1" to log on to S1. 3. Work out the process ID number (PID), either with the command (s1:1001) tsockman get dome_S1 Name : dome_S1 Machine : s1.chara-array.org PID : 29953 Commands : -1 Data : -1 Message : 4002 Restart : /usr/local/bin/dome_server -A33.7441 S1 or with (s1:1003) ps aux | grep dome theo 4473 0.0 0.0 61188 748 pts/3 S 10:45 0:00 grep dome observe 29953 18.5 0.4 35596 9860 ? Sl Apr21 416:11 /usr/local/bin/dome_server -A33.7441 S It can also be found by pulling up the LIST on SOCKMAN and selecting the relevant dome. So in this case the PID is 29953. 4. Try and stop the server gracefully: kill -2 29953 5. You should then check that the server has indeed stopped: [s1:600] tsockman get dome_S1 Name : dome_S1 Machine : s1.chara-array.org PID : 15635 Commands : -1 Data : -1 Message : 2008 Restart : /usr/local/bin/dome_server -A41.0166 S1 If the socket manager still thinks it's running you will need to stop it forcefully: kill -9 29953; tsockman rm dome_S1 6. Restart the dome server by copying the command at the end of the /etc/rc.local file: [s1:602] more /etc/rc.local <<< Press space bar or enter to scroll through file>>> #Run the dome server /usr/local/bin/tsockman remove dome_S1 /usr/local/bin/dome_server -A41.0166 S1 & (Note: the part of the command that saves information to /var/log/dome_S1.log has been removed.) 7. Turn the power to the drives back on. 8. Hit REOPEN and ENABLE on the domegtk, and type "otcs" in the telescope server. You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scopes position.
+To manually start the dome server:
+  - Make sure the power to the drives is OFF.
+  - Login to the relevant computer as root. For example, type "s1" or "ssh s1" to log on to S1.
+  - Work out the process ID number (PID), either with the command (s1:1001) tsockman get dome_S1 Name : dome_S1 Machine : s1.chara-array.org PID : 29953 Commands : -1 Data : -1 Message : 4002 Restart : /usr/local/bin/dome_server -A33.7441 S1 or with (s1:1003) ps aux | grep dome theo 4473 0.0 0.0 61188 748 pts/3 S 10:45 0:00 grep dome observe 29953 18.5 0.4 35596 9860 ? Sl Apr21 416:11 /usr/local/bin/dome_server -A33.7441 S It can also be found by pulling up the LIST on SOCKMAN and selecting the relevant dome. So in this case the PID is 29953.
+  - Try and stop the server gracefully: kill -2 29953 5. You should then check that the server has indeed stopped: [s1:600] tsockman get dome_S1 Name : dome_S1 Machine : s1.chara-array.org PID : 15635 Commands : -1 Data : -1 Message : 2008 Restart : /usr/local/bin/dome_server -A41.0166 S1 If the socket manager still thinks it's running you will need to stop it forcefully: kill -9 29953; tsockman rm dome_S1
+  - Restart the dome server by copying the command at the end of the /etc/rc.local file: [s1:602] more /etc/rc.local <<< Press space bar or enter to scroll through file>>> #Run the dome server /usr/local/bin/tsockman remove dome_S1 /usr/local/bin/dome_server -A41.0166 S1 & (Note: the part of the command that saves information to /var/log/dome_S1.log has been removed.)
+  - Turn the power to the drives back on.
+  - Hit REOPEN and ENABLE on the domegtk, and type "otcs" in the telescope server. You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scopes position.
 ==== Telescope clock is not correct ====
@@ Line 215: / Line 186: @@
 ==== E1 Hut and Cooler Communications Recovery ====
-{{:chara:files:e1_hut_coms_recovery.pdf|E1 HuT and Cooler Communications Recovery}}  \\  \\ <font 16px/inherit;;inherit;;inherit>**S2 Acquisition Server Restart**</font>
+====   E1 HuT and Cooler Communications Recovery   \\  \\  S2 Acquisition Server Restart  ====
 S2's acquisition server resides on "S2pi4" computer. Logging on to it (ssh s2pi4) and running bootlaunch will restart it
-====== Dome issue ======
+===== Dome issue =====
 At times the domes do not rotate, open or close, or otherwise behave. Some problems are simple and others are more complex.
@@ Line 225: / Line 196: @@
 ==== Dome does not rotate ====
-Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope's position. If not, it will insist on being in the wrong place. If it is not at the same AZ as the scope, manually move it until it is centered on the telescope in spycam 1. If the dome AZ does not read the same as the telescope AZ, enter the scope AZ in the position box of the DOME tab of appropriate dome server and hit the INIT POS button to tell it at what AZ it is. \\ \\ If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker.
+Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope's position. If not, it will insist on being in the wrong place. If it is not at the same AZ as the scope, manually move it until it is centered on the telescope in spycam 1. If the dome AZ does not read the same as the telescope AZ, enter the scope AZ in the position box of the DOME tab of appropriate dome server and hit the INIT POS button to tell it at what AZ it is. \\  \\
+If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker.
 ==== Dome does not open ====
-Is the SPYCAM on and updating? An old image may show the dome is closed, but it has actually started to open. If the dome won't open, try hitting the SLIT CLOSE button on the telescope gui first. It may be that there was a computer or server issue and the computer thinks the dome is open already and won't allow it to open again. Hit the SLIT OPEN button to see if it works now. If it still won't open, go to the bunker and look to see if the power switch is on to the dome. It will be up and red if on. \\ Also look to see if the power cable is connected to the computer box. If it's connected up high to the manual dome controls, the cable needs to be brought back to the computer box connector. __Turn the power off first__ to the dome before disconnecting the power cable. Slide it onto the connector at the box and turn the collar to lock it in place. Turn the power back on and see if you can open the dome from the computer in the bunker.
+Is the SPYCAM on and updating? An old image may show the dome is closed, but it has actually started to open. If the dome won't open, try hitting the SLIT CLOSE button on the telescope gui first. It may be that there was a computer or server issue and the computer thinks the dome is open already and won't allow it to open again. Hit the SLIT OPEN button to see if it works now. If it still won't open, go to the bunker and look to see if the power switch is on to the dome. It will be up and red if on.
+Also look to see if the power cable is connected to the computer box. If it's connected up high to the manual dome controls, the cable needs to be brought back to the computer box connector. __Turn the power off first__  to the dome before disconnecting the power cable. Slide it onto the connector at the box and turn the collar to lock it in place. Turn the power back on and see if you can open the dome from the computer in the bunker.
 ==== Dome does not close ====
@@ Line 235: / Line 209: @@
 If a dome will not close after observing, it may need to be closed manually at the bunker. Turn the power to the dome off at the switch, disconnect the cable from the computer control box and connect it to the manual control switches above. Turn on the power to the dome after connecting the cable and use the two controls to close the dropout first, then the slit. The dropout must be fully closed and the slit closed over it to be shut properly. Inform the staff that the dome did not function properly so someone can look at it in the morning.
-====== HUT servers ======
+===== HUT servers =====
+==== I can't change the camera settings on the TV ====
+**CHECK THIS SECTION - THESE ARE VERY CLOSE BUT DIFFERENT PARAGRAPHS:**
 The HUT servers control functions such as beacon and dichroic movements, heater and dehumidifier usage, and various AO functions. An observer may find that the obsgtk is no longer controlling the beacon LED's, beacon flat or dichroic alignments. This happens on occasion with E2 and other scopes. The HUT server may be the cause if it has quit or lost connection or the AOB may be at fault. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the alignments can be changed from the gui, then the HUT server is ok. You can use the hut gui to continue observing. If the hut gui gives move error messages, cycle the power on the AOB and open a new hut server to restore the connection to the obsgtk. On the POWER gui, turn off the power to the AOB for the offending telescope and turn it back on. Stop the hut server by logging into the appropriate telescope computer and identifying the PID with the bootlaunch_master command and killing the process with the kill -9 #### command. Start the new server via the bootlaunch_master command. Hit REOPEN on the obsgtk to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well.
+The HUT servers control functions such as finder and acq exposure times and gains, heater and dehumidifier usage, and various AO functions. An observer may find that the camera settings do not display or are not adjustable. The HUT server may be the cause if it has quit. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the camera settings are displayed and the settings can be changed from the gui, then the HUT server is ok. Restart the telescope server to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well. If the gui is not functioning, the HUT server will need to be restarted. The HUT servers run on the computer at each telescope. Find the PID from Sockman for the server you want to shut down. Open a terminal and log into the correct telescope computer (for example, type "s1" to log on to S1). Use the command "ps aux | grep HUT" to find the process you need to shut down. The number should match the PID from Sockman. Kill the process by typing "kill -9 pid" where pid is the process number and remove the entry from Sockman with the REMOVE button on the gui or by command line. Restart the hut server using the new "bootlaunch_hut" command described in the section on Restarting Servers.
+If the server won't restart, a reboot of the power supply in the telescope bunker might be necessary. The power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunker. The power supply has green readouts of volts and current. After turning the power off for 10 seconds and back on, try restarting the server from the computer in the bunker to see if it starts cleanly. If so, then restart the telescope server, reopen the connection to the telescope gui, and hit REOPEN on Cosmic Debris. Part of the HUT server also controls the AO table. If the AOB part of the HUT server doesn't work, then the power supply on the back of the AO table in the telescope dome might need recycling. This power supply controls the actuators at M2 and the AO table. The power supply box is a 6×9 inch aluminum box on the back of the AO table, behind the keyboard and monitor. Turn it off with the power button on the bottom edge of the box, wait 5 seconds and turn it back on. The HUT server should now restart cleanly. Restart the telescope server as well to make the connections to the telescope gui. Hit REOPEN in Cosmic Debris if you are observing to make all needed connections.
 ===== Tiptilt Server. (These sections apply to the old lab tiptilt system retired in April 2021) =====
-The tiptilt server controls the CCD based tiptilt detection system. \\ \\ Before you start the tiptilt server, you must ensure that the power to the cooling system and the CCD iteslf is on. It is extremely important that the cooler be running before you turn on the CCD and is only turned off if you are sure the CCD is NOT running. You can start the server from the X windows menu or with the command xtiptilt. \\ \\ Note that there are background counts and read noise to deal with. Whenever you change the frame rate, please ensure that the bias frame is OK. The server will attempt to load an old bias frame that should work, but if things are not working, try making a new bias frame by ensuring that the detector is in the dark and typing "mkbias" into the tiptilt server. \\ \\ In the tiptilt GUI windows, the white dots represent the starlight while the green dots represent the motion applied to telescope's secondary mirror to keep the starlight centered. When tiptilt is locked the white dots will be brought to the center of the tiptilt window. The green dots should be mostly centered also. W2 and E2 telescopes have a small oscillation that show as back and forth plots of the green dots.
+[[:chara:old_lab_tiptilt_server|Instructions for the old lab tiptilit system are archived here.]]
+The tiptilt server controls the CCD based tiptilt detection system. \\  \\
+Before you start the tiptilt server, you must ensure that the power to the cooling system and the CCD iteslf is on. It is extremely important that the cooler be running before you turn on the CCD and is only turned off if you are sure the CCD is NOT running. You can start the server from the X windows menu or with the command xtiptilt. \\  \\
+Note that there are background counts and read noise to deal with. Whenever you change the frame rate, please ensure that the bias frame is OK. The server will attempt to load an old bias frame that should work, but if things are not working, try making a new bias frame by ensuring that the detector is in the dark and typing "mkbias" into the tiptilt server. \\  \\
+In the tiptilt GUI windows, the white dots represent the starlight while the green dots represent the motion applied to telescope's secondary mirror to keep the starlight centered. When tiptilt is locked the white dots will be brought to the center of the tiptilt window. The green dots should be mostly centered also. W2 and E2 telescopes have a small oscillation that show as back and forth plots of the green dots.
 ==== Tiptilt server complains about the CCD ====
-Is the CCD turned on? When the tiptilt server starts up it tries no more than five times to communicate with the CCD. If they all fail, it will give up. If this happens, try cycling the power to the CCD and try again. If this fails, connect to the tiptilt machine and type the command rtccdAPIDemo, which should return with no errors. Try this command a few times, but if it still fails, there is a more serious problem. Turn off the CCD and reboot the tiptilt computer. If it still fails, I am afraid you are in more serious trouble. \\ \\ Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine. \\ \\ Also note that cycling the power on the CCD can cause harm so be sure you need to do it before trying it. Also, it is important to wait for at least 20 seconds after turning off the power before turning it on again.
+Is the CCD turned on? When the tiptilt server starts up it tries no more than five times to communicate with the CCD. If they all fail, it will give up. If this happens, try cycling the power to the CCD and try again. If this fails, connect to the tiptilt machine and type the command rtccdAPIDemo, which should return with no errors. Try this command a few times, but if it still fails, there is a more serious problem. Turn off the CCD and reboot the tiptilt computer. If it still fails, I am afraid you are in more serious trouble. \\  \\
+Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine. \\  \\
+Also note that cycling the power on the CCD can cause harm so be sure you need to do it before trying it. Also, it is important to wait for at least 20 seconds after turning off the power before turning it on again.
 ==== Tiptilt doesn't seem to be talking to the telescopes ====
-Sometimes the telescope server will not show that TT is running. It will show 0Hz for a signal rate for TT. Running TIPTILT COMM will not get it started while other scopes do show it starting. Close and restart any telescope servers that won't connect after two tries of TIPTILT COMM. \\ Note: There is more info in the software manual on this topic, but I wasn't sure if it was still relevant.
+Sometimes the telescope server will not show that TT is running. It will show 0Hz for a signal rate for TT. Running TIPTILT COMM will not get it started while other scopes do show it starting. Close and restart any telescope servers that won't connect after two tries of TIPTILT COMM. \\
+Note: There is more info in the software manual on this topic, but I wasn't sure if it was still relevant.
 ==== Tiptilt server says the clock isn't running ====
-First check whether the clock itself is running and the other machines receive the clock signal. Look at the clock cards at the back of other computers in the rack. The clock cards have three LEDs, one yellow and two greens. If the computer is receiving the clock signal properly all three LEDs should blink, but at a different rate. If the LEDs on all the clock cards are solid then reboot the GPS computer. When the GPS computer is down, it is best to cycle the power also on the box right above the GPS computer. \\ \\ If the clock appears to be working properly on other machines and not on the tiptilt now it is time to reboot tiptilt. \\ \\ [There is a bug in the real time part of the CCD code. It is caused by the clock in the tiptilt system either not running at all or having been set to a time very different from the last time the CCD ran.] \\ \\ For the time being the only solution is to reboot tiptilt, but do so from the lab. Power OFF the CCD, then reboot the tiptilt machine and go into the BIOS. Make sure that interrupt 11 has been set to ISA legacy, save the BIOS and reboot. When the clock card LEDs in the tiptilt machine indicate proper clock signal, turn the CCD back on and start the tiptilt server. \\ \\ Also, sometimes Serial Port 3 grabs IRQ 11 which stops the clock from running. Since there is no serial port 2 it's safe to disable this in the BIOS. This problem normally comes up when there has been a power outage. \\ \\ Sometimes syncing the clock can also cause this problem, but that should be fixed soon. If it does, exit the tiptilt server, log in as root, and reload the tiptilt model using the following commands: \\ /sbin/rmmod tiptilt_rt \\ /sgin/insmod /usr/local/modules/tiptilt_rt.o \\ \\ Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine.
+First check whether the clock itself is running and the other machines receive the clock signal. Look at the clock cards at the back of other computers in the rack. The clock cards have three LEDs, one yellow and two greens. If the computer is receiving the clock signal properly all three LEDs should blink, but at a different rate. If the LEDs on all the clock cards are solid then reboot the GPS computer. When the GPS computer is down, it is best to cycle the power also on the box right above the GPS computer. \\  \\
+If the clock appears to be working properly on other machines and not on the tiptilt now it is time to reboot tiptilt. \\  \\
+[There is a bug in the real time part of the CCD code. It is caused by the clock in the tiptilt system either not running at all or having been set to a time very different from the last time the CCD ran.] \\  \\
+For the time being the only solution is to reboot tiptilt, but do so from the lab. Power OFF the CCD, then reboot the tiptilt machine and go into the BIOS. Make sure that interrupt 11 has been set to ISA legacy, save the BIOS and reboot. When the clock card LEDs in the tiptilt machine indicate proper clock signal, turn the CCD back on and start the tiptilt server. \\  \\
+Also, sometimes Serial Port 3 grabs IRQ 11 which stops the clock from running. Since there is no serial port 2 it's safe to disable this in the BIOS. This problem normally comes up when there has been a power outage. \\  \\
+Sometimes syncing the clock can also cause this problem, but that should be fixed soon. If it does, exit the tiptilt server, log in as root, and reload the tiptilt model using the following commands: \\
+/sbin/rmmod tiptilt_rt \\
+/sgin/insmod /usr/local/modules/tiptilt_rt.o \\  \\
+Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine.
 ==== Tiptilt is not locking on a star or locks, but lets the star drift away ====
@@ Line 326: / Line 324: @@
 ==== Restarting the Beam Sampler Server ====
-The most common cause for getting multiple copies of a server is using the socket manage RESTART button, which is flaky at best. Don't do this. If you need to restart a server you should do it manually. 1. Make sure you are logged into the right machine: ssh ctrscrut 2. If you are not sure see if the socket manager will tell you. If it doesn't have a look in the file: /ctrscrut/chara/etc/sock_manager/socket_manager.list 3. See if there are any ghosts running ps aux | grep esp_server Yes, it's hard to know which one it is with the esp servers. You can work out which ones are ghosts by typing the command tsockman |grep ctrscrut which will give you a list of the servers running on the machine you are interested in. Checking for non-matching PIDs will tell you which processes you need to stop. 4. Stop those processes: kill -9 PID1 PID2 …. 5. Make sure there is no sign of it in the socket manager tsockman rm BS 5. Restart the beam sampler servers (this starts both beam sampler servers BS1 and BS2): bootlaunch_beamsamp
+The most common cause for getting multiple copies of a server is using the socket manage RESTART button, which is flaky at best. Don't do this. If you need to restart a server you should do it manually.
+  - Make sure you are logged into the right machine: ssh ctrscrut
+  - If you are not sure see if the socket manager will tell you. If it doesn't have a look in the file: /ctrscrut/chara/etc/sock_manager/socket_manager.list
+  - See if there are any ghosts running \\
+    ps aux | grep esp_server \\
+    Yes, it's hard to know which one it is with the esp servers. You can work out which ones are ghosts by typing the command \\
+    tsockman |grep ctrscrut \\
+    which will give you a list of the servers running on the machine you are interested in. Checking for non-matching PIDs will tell you which processes you need to stop.
+  - Stop those processes: \\
+    kill -9 PID1 PID2 ….
+  - Make sure there is no sign of it in the socket manager \\
+    tsockman rm BS
+  - Restart the beam sampler servers (this starts both beam sampler servers BS1 and BS2): \\
+    bootlaunch_beamsamp
 ==== "Failed to request position of S1" ====
-Cosmic Debris reports "Failed to request position of S1" when trying to set the beam order. This indicates that ople is no longer talking to the beam samplers. If the beam sampler server has been restarted recently, then the ople server will also need to be restarted to re-establish the connection. \\ \\ The beam sample server runs on ctrscrut. The ople server tries to open a connection to the beam sampler when it starts. If the beam sampler is not there, or dies, \\ you need to restart the ople server as there is no command to reopen that connection.
+Cosmic Debris reports "Failed to request position of S1" when trying to set the beam order. This indicates that ople is no longer talking to the beam samplers. If the beam sampler server has been restarted recently, then the ople server will also need to be restarted to re-establish the connection. \\  \\
+The beam sample server runs on ctrscrut. The ople server tries to open a connection to the beam sampler when it starts. If the beam sampler is not there, or dies, \\
+you need to restart the ople server as there is no command to reopen that connection.
 ===== Remote Observing =====
@@ Line 349: / Line 363: @@
 ==== GUIs are frozen ====
-Check to see if Sockman is working. Click the [LIST] button on sockman. If the list doesn't come up, Sockman may be hung up. Follow the Sockman restart procedure in the text file on the desktop if it won't respond. Hit [REOPEN] on the gui to see if it reconnects with its server. if that does not fix it, the server may be dead. From the LIST on Sockman, select the server suspected of being frozen. Ping it and if it reports back as being dead, see [[:chara:trouble_shooting#server_is_frozen|Server is frozen]] section below. Telescope guis often freeze. Hit reopen on the gui to bring it back to life. Hitting [MOVE] on the telescope gui before the star has stopped moving can cause the gui to freeze. Avoid doing this. Close the dead one and open a new one to fix the problem.
+Check to see if Sockman is working. Click the [LIST] button on sockman. If the list doesn't come up, Sockman may be hung up. Follow the Sockman restart procedure in the text file on the desktop if it won't respond.
+Hit [REOPEN] on the gui to see if it reconnects with its server. if that does not fix it, the server may be dead. From the LIST on Sockman, select the server suspected of being frozen. Ping it and if it reports back as being dead, see [[:chara:trouble_shooting#server_is_frozen|Server is frozen]] section below.
+Telescope guis often freeze. Hit reopen on the gui to bring it back to life. Hitting [MOVE] on the telescope gui before the star has stopped moving can cause the gui to freeze. Avoid doing this. Close the dead one and open a new one to fix the problem.
 ==== Server display is blank or filled with jibberish ====
@@ Line 379: / Line 397: @@
 If an object is not in SIMBAD (such as a recently discovered nova), then you must enter the object manually using the instructions below.
-==== Finding a star in the CHARA database ====
+**Finding a star in the CHARA database**
 Here is an example of how to find the CHARA number of a star, nova, or AGN that is already in the CHARA database:
-[ctrscrut:502] dbadd NGC 4151 Object "NGC 4151" found as CHARA number: 320263
+[ctrscrut:502] dbadd NGC 4151 \\
+Object "NGC 4151" found as CHARA number: 320263
 Note: If the target information stored in the CHARA database does not match SIMBAD, then dbadd will provide the coordinates and magnitudes listed in SIMBAD, and it will also list the CHARA numbers of objects in the database located within 10 arcsec. Please check these nearby entries before adding a new star to the database.
@@ Line 393: / Line 412: @@
 Here is an example of how to add an object to the CHARA database:
-[ctrscrut:534] dbadd BD 41 3807 SIMBAD found this for "BD 41 3807":
+[ctrscrut:534] dbadd BD 41 3807
-ID 2MASS_J20340850 4136592 CHARA 320414 2000.0 CAT8 2MASS_J20340850 4136592 RA 20 34 08.5148 DEC 41 36 59.408 PMRA -2.32 PMDEC -3.87 PAR 0.58 RVEL -27.50 VMAG 10.03 HMAG 6.23 KMAG 5.99 SP O5.5Ifc END
+SIMBAD found this for "BD 41 3807":
-Is the above the object you are looking for? [y/n] y Added "BD 41 3807" to the database.
+ID 2MASS_J20340850 4136592 \\
+CHARA 320414 \\
+.0 \\
+CAT8 2MASS_J20340850 4136592 \\
+RA 20 34 08.5148 \\
+DEC 41 36 59.408 \\
+PMRA -2.32 \\
+PMDEC -3.87 \\
+PAR 0.58 \\
+RVEL -27.50
+VMAG 10.03 \\
+HMAG 6.23 \\
+KMAG 5.99 \\
+SP O5.5Ifc \\
+END
+Is the above the object you are looking for? [y/n] y \\
+Added "BD 41 3807" to the database.
 In the example above, the object "BD 41 3807" has been added to the database as CHARA number 320414.
@@ Line 403: / Line 440: @@
 **Targets that are not in SIMBAD**
-Some objects like recently discovered novae are not in SIMBAD. Entering dbadd <star designation> for a star not in the SIMBAD database returns the message "Simbad is unable to find an object matching <star designation>. Try using a different catalog designation, or use the "-m" switch to add the object manually." The command dbadd -m is used by entering the star name and coordinates in this format:
+Some objects like recently discovered novae are not in SIMBAD. Entering dbadd <star designation> for a star not in the SIMBAD database returns the message "Simbad is unable to find an object matching <star designation>. Try using a different catalog designation, or use the "-m" switch to add the object manually."
-Usage: dbadd -m <name> <RA> <Dec> <name>
+The command dbadd -m is used by entering the star name and coordinates in this format:
-Object ID (no spaces) <RA> : XXhXXmXX.X or XXhXX.X (no spaces) <Dec> : XXdXXmXX.X or XXdXX.X (no spaces)
+Usage: dbadd -m <name> <RA> <Dec> \\  \\
+<name>: Object ID (no spaces) \\  \\
+<RA> : XXhXXmXX.X or XXhXX.X (no spaces) \\  \\
+<Dec> : XXdXXmXX.X or XXdXX.X (no spaces) \\  \\
 There are cases where an object is not in SIMBAD and doesn't return a result in dbadd, but is in fact in the CHARA database. Novae and AGN's are the likely objects that cause this result. At times, objects have multiple entries and several CHARA numbers since the names can be so unique. A system will be developed to find these entries without knowing or having a conventional database designation.
@@ Line 421: / Line 460: @@
 ==== Adding an anchor to the wiki page ====
-To add an anchor to the wikipage, type the following where you want the anchor inserted (without the spaces between the symbols): [ [ #AnchorName ] ] Highlight the text you want to be a link to the anchor, click the link symbol above and select anchor. Select a page(such as Trouble Shooting) where the anchor is located and give the same name for the anchor.
+To add an anchor to the wikipage, type the following where you want the anchor inserted (without the spaces between the symbols): \\
+[ [ #AnchorName ] ]
+Highlight the text you want to be a link to the anchor, click the link symbol above and select anchor. Select a page(such as Trouble Shooting) where the anchor is located and give the same name for the anchor.
 ==== Displays for the MIRC-X wolverine2 monitors ====
@@ Line 427: / Line 469: @@
 Follow instructions in the [[:chara:mircx_monitors|linked page to display the MIRC-X wolverine2 monitors]] in the control room.
-\\ Last updated 2017-04-12
+ \\
+Last updated 2017-04-12

The CHARA Array Wiki

User Tools

Site Tools

Differences

Page Tools