This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
chara:trouble_shooting [2022/01/21 17:55] charaobs |
chara:trouble_shooting [2022/01/21 20:03] charaobs |
||
---|---|---|---|
Line 27: | Line 27: | ||
* Check the instrument alignment. Is flux getting through to the detector? How long has it been since the last NIRO camera alignment? Classic and CLIMB programs can run for about an hour before the light will drift from the central pixel. Use the Classic or CLIMB gui to view the light on the pixels by clicking the PICTURE tab and then the PIXEL AREA button. Turn the camera off with the STOP button. Is the right dither power turned on? CLIMB 1 and Classic use different dithers. If Classic or CLIMB fringes are found in a scan, but not when in recording mode, the dither powers are likely not on. Are the camera settings correct for the seeing conditions and flux levels? | * Check the instrument alignment. Is flux getting through to the detector? How long has it been since the last NIRO camera alignment? Classic and CLIMB programs can run for about an hour before the light will drift from the central pixel. Use the Classic or CLIMB gui to view the light on the pixels by clicking the PICTURE tab and then the PIXEL AREA button. Turn the camera off with the STOP button. Is the right dither power turned on? CLIMB 1 and Classic use different dithers. If Classic or CLIMB fringes are found in a scan, but not when in recording mode, the dither powers are likely not on. Are the camera settings correct for the seeing conditions and flux levels? | ||
- | ===== Restarting Servers | + | ===== The new OPLE system |
- | ==== Restarting | + | With the implementation of the new ople system which replaced the VME in Fall of 2021, new troubleshooting issues have arisen. Since there are now 6 new ople computers to run the carts for each scope individually, |
- | If a server | + | The traditional ople server |
- | To restart | + | If a cart cannot be started, stopped or otherwise commanded, look to the OPLESystem gui to see if a green indicator has turned red. A message |
- | * bootlaunch_beamsamp – Starts | + | Some times a cart is stopped and cannot be commanded. If the cart has gone to the front hard or back hard switch, it will not be usable until it is moved from the switch |
- | * bootlaunch_zaber – Starts | + | |
- | \\ telescope bunker computers are now using bootlaunch_master to start each of the servers listed below. | + | ===== Restarting Servers ===== |
- | * bootlaunch_hut – Starts | + | ==== Restarting Servers using the bootlaunch paradigm ==== |
- | * bootlaunch_rpc – Starts the RPC_E1, RPC_E2, RPC_S1, RPC_S2, RPC_W1, | + | |
- | * bootlaunch_weather – Starts | + | If a server |
- | * bootlaunch_lower – Starts | + | |
- | * bootlaunch_upper – Starts the E1_Upper, E2_Upper, S1_Upper, S2_Upper, W1_Upper, or W2_Upper server, depending on the machine it's launched from. | + | |
- | \\ Note: The bootlaunch scripts will not start a new server if there is an existing process running. Therefore, type "ps aux | grep // | + | To restart a server, log on to the machine that runs the server and type " |
==== Restarting Servers using the rc.local file ==== | ==== Restarting Servers using the rc.local file ==== | ||
Line 54: | Line 51: | ||
Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn' | Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn' | ||
- | ==== Shutters Server | + | Shutters Server |
\\ The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on ople, not ctrscrut. \\ \\ To start the shutter server on ople: \\ \\ Log into the ople computer and kill the process labeled shutters with the PID as described in **Restarting Servers** | \\ The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on ople, not ctrscrut. \\ \\ To start the shutter server on ople: \\ \\ Log into the ople computer and kill the process labeled shutters with the PID as described in **Restarting Servers** | ||
Line 73: | Line 70: | ||
===== Telescopes and Dome Servers ===== | ===== Telescopes and Dome Servers ===== | ||
- | |||
- | Here we discuss things that can go wrong with the telescopes. | ||
==== The Telescope won't move or stopped moving ==== | ==== The Telescope won't move or stopped moving ==== | ||
Line 86: | Line 81: | ||
2. Make sure you understand why the limit was hit which may require a trip to the telescope. If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch causing the stall and not that the scope is actually in a wrong position.</ | 2. Make sure you understand why the limit was hit which may require a trip to the telescope. If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch causing the stall and not that the scope is actually in a wrong position.</ | ||
4. Click ENABLE then you can move the telescope back to its normal range of operation. \\ \\ | 4. Click ENABLE then you can move the telescope back to its normal range of operation. \\ \\ | ||
- | 5. After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again and then hit AUTO on the AUTO tab to resume normal operation < | + | 5. After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again and then hit AUTO on the AUTO tab to resume normal operation.</font> |
+ | |||
+ | <font 14px/Arial, | ||
<font 14px/ | <font 14px/ | ||
Line 93: | Line 90: | ||
<font 14px/ | <font 14px/ | ||
- | <font 14px/ | + | <font 14px/ |
<font 14px/ | <font 14px/ | ||
Line 109: | Line 106: | ||
==== Dome Server Restart ==== | ==== Dome Server Restart ==== | ||
- | Dome servers are now started using the bootlaunch_master command. | + | Dome servers are now started using the bootlaunch_master command. |
- | To manually start the dome server: | + | To restart |
- | 1. Make sure the power to the drives is OFF. | + | 1. Make sure the power to the drives is OFF. Disable the scopes. |
- | 2. Login to the relevant computer as root. For example, type "s1" or " | + | 2. Login to the relevant computer as observe. For example, type " |
- | 3. Work out the process ID number (PID), either with the command | + | 3. Work out the process ID number (PID) by typing bootlaunch_master |
- | (s1:1001) tsockman get dome_S1 \\ | + | 4. Use kill -2 PID to kill the server. |
- | Name : dome_S1 \\ | + | |
- | Machine : s1.chara-array.org \\ | + | |
- | PID : 29953 \\ | + | |
- | Commands : -1 \\ | + | |
- | Data : -1 \\ | + | |
- | Message : 4002 \\ | + | |
- | Restart : / | + | |
- | or with \\ | + | |
- | (s1:1003) ps aux | grep dome \\ | + | |
- | theo 4473 0.0 0.0 61188 748 pts/3 S+ 10:45 0:00 grep dome \\ | + | |
- | observe 29953 18.5 0.4 35596 9860 ? Sl Apr21 416:11 / | + | |
- | It can also be found by pulling up the LIST on SOCKMAN and selecting the relevant dome. \\ \\ | + | |
- | So in this case the PID is 29953. \\ \\ | + | |
- | 4. Try and stop the server gracefully: | + | |
- | 5. You should then check that the server | + | 5. Type bootlaunch_master again to restart |
- | [s1:600] tsockman get dome_S1 \\ | + | |
- | Name : dome_S1 \\ | + | |
- | Machine : s1.chara-array.org \\ | + | |
- | PID : 15635 \\ | + | |
- | Commands : -1 \\ | + | |
- | Data : -1 \\ | + | |
- | Message : 2008 \\ | + | |
- | Restart : / | + | |
- | If the socket manager still thinks it's running you will need to stop it forcefully: kill -9 29953; tsockman rm dome_S1 | + | |
- | 6. Restart | + | 6. Turn the power to the drives back on. |
- | [s1:602] more /etc/rc.local \\ | + | 7. Hit REOPEN and ENABLE on the domegtk, and type " |
- | <<< | + | |
- | #Run the dome server | + | |
- | / | + | |
- | / | + | |
- | (Note: the part of the command that saves information to / | + | |
- | 7. Turn the power to the drives back on. | + | You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing |
- | 8. Hit REOPEN and ENABLE on the domegtk, and type " | + | ==== Telescope is not receiving |
- | You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scopes position. | + | Sometimes it happens that a telescope receives the wrong position for a target or does not receive the commanded position at all. The commanded position is listed on the telescope server in the first column under TCS Az/El; the second column lists the actual position of the telescope. Try entering the star designation directly into the telescope server, ie. hd 123456. If it does not accept the number, try closing and restarting the telescope server and hitting repoen on Cosmic Debris and the telescope gui. Try entering the star into the server again. If that does not work, it is possible that something is wrong with the dome server. To restart the dome server follow these steps above. |
- | + | ||
- | ==== Telescope is not receiving the commanded position for a target ==== | + | |
- | + | ||
- | Sometimes it happens that a telescope receives the wrong position for a target or does not receive the commanded position at all. The commanded position is listed on the telescope server in the first column under TCS Az/El; the second column lists the actual position of the telescope. Try entering the star designation directly into the telescope server, ie. hd 123456. If it does not accept the number, try closing and restarting the telescope server and hitting repoen on Cosmic Debris and the telescope gui. Try entering the star into the server again. If that does not work, it is possible that something is wrong with the dome server. To restart the dome server follow these steps: * DISABLE the telescope using either the telescope or dome GUI. | + | |
- | + | ||
- | * Turn off the power for the telescope (both AZ/EL). | + | |
- | * First shutdown the telescope server | + | |
- | * Then use SOCKMAN to select the appropriate dome server from the list (dome_E1, etc) Get the PID number for reference. | + | |
- | * The dome server will need to be restarted from the command line in a terminal. | + | |
- | * Open a terminal window and log on to the telescope computer by typing " | + | |
- | * Look for instruction in the / | + | |
- | * Locate instruction listed under "#Run the dome server" | + | |
- | * / | + | |
- | * / | + | |
- | * (Replace S1 with commands for appropriate telescope) | + | |
- | * After the dome server is running, re-open the telescope server and click [REOPEN} on the dome and telescope GUIs and on Cosmic Debris. | + | |
- | * ENABLE the telescope using the telescope or dome GUI. Check to see if the command and telescope positions are correct. If they are, then turn on the power for the telescope drives. Re-enter the star information by entering the HD number in the telescope server and click [GO NEXT] to send the telescope to the star. Make sure that the telescope is behaving as expected. The telescope might have drifted a bit, so if you can't find the star, it might be necessary to use the Telrad on a bright star to re-initialize the pointing. | + | |
- | * Restart commands for the dome servers are also listed on the desktop in a text file. | + | |
==== Telescope is tracking poorly, overshooting in slew, oscillating. ==== | ==== Telescope is tracking poorly, overshooting in slew, oscillating. ==== | ||
Line 185: | Line 136: | ||
You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high. | You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high. | ||
- | The usual values for slewing gains are 4-7 and tracking gains are 7-10. Note that these values may change as the temperatures change rapidly. Gains are usually higher when cold and lower when warm. Be sure to set mode back to AUTO if changing the gains left it in Slewing or Tracking mode. | + | The usual values for slewing gains are 4-7 and tracking gains are 7-10. Note that these values may change as the temperatures change rapidly. Gains are usually higher when cold and lower when warm. Be sure to set mode back to AUTO if changing the gains left it in Slewing or Tracking mode. |
==== How to Adjust CPUMotor Gains ==== | ==== How to Adjust CPUMotor Gains ==== | ||
Line 241: | Line 192: | ||
===== HUT servers ===== | ===== HUT servers ===== | ||
- | |||
- | ==== I can't change the camera settings on the TV ==== | ||
- | |||
- | **CHECK THIS SECTION - THESE ARE VERY CLOSE BUT DIFFERENT PARAGRAPHS: | ||
The HUT servers control functions such as beacon and dichroic movements, heater and dehumidifier usage, and various AO functions. An observer may find that the obsgtk is no longer controlling the beacon LED's, beacon flat or dichroic alignments. This happens on occasion with E2 and other scopes. The HUT server may be the cause if it has quit or lost connection or the AOB may be at fault. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the alignments can be changed from the gui, then the HUT server is ok. You can use the hut gui to continue observing. If the hut gui gives move error messages, cycle the power on the AOB and open a new hut server to restore the connection to the obsgtk. On the POWER gui, turn off the power to the AOB for the offending telescope and turn it back on. Stop the hut server by logging into the appropriate telescope computer and identifying the PID with the bootlaunch_master command and killing the process with the kill -9 #### command. Start the new server via the bootlaunch_master command. Hit REOPEN on the obsgtk to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well. | The HUT servers control functions such as beacon and dichroic movements, heater and dehumidifier usage, and various AO functions. An observer may find that the obsgtk is no longer controlling the beacon LED's, beacon flat or dichroic alignments. This happens on occasion with E2 and other scopes. The HUT server may be the cause if it has quit or lost connection or the AOB may be at fault. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the alignments can be changed from the gui, then the HUT server is ok. You can use the hut gui to continue observing. If the hut gui gives move error messages, cycle the power on the AOB and open a new hut server to restore the connection to the obsgtk. On the POWER gui, turn off the power to the AOB for the offending telescope and turn it back on. Stop the hut server by logging into the appropriate telescope computer and identifying the PID with the bootlaunch_master command and killing the process with the kill -9 #### command. Start the new server via the bootlaunch_master command. Hit REOPEN on the obsgtk to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well. | ||
- | |||
- | The HUT servers control functions such as finder and acq exposure times and gains, heater and dehumidifier usage, and various AO functions. An observer may find that the camera settings do not display or are not adjustable. The HUT server may be the cause if it has quit. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the camera settings are displayed and the settings can be changed from the gui, then the HUT server is ok. Restart the telescope server to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well. If the gui is not functioning, | ||
If the server won't restart, a reboot of the power supply in the telescope bunker might be necessary. The power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunker. The power supply has green readouts of volts and current. After turning the power off for 10 seconds and back on, try restarting the server from the computer in the bunker to see if it starts cleanly. If so, then restart the telescope server, reopen the connection to the telescope gui, and hit REOPEN on Cosmic Debris. Part of the HUT server also controls the AO table. If the AOB part of the HUT server doesn' | If the server won't restart, a reboot of the power supply in the telescope bunker might be necessary. The power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunker. The power supply has green readouts of volts and current. After turning the power off for 10 seconds and back on, try restarting the server from the computer in the bunker to see if it starts cleanly. If so, then restart the telescope server, reopen the connection to the telescope gui, and hit REOPEN on Cosmic Debris. Part of the HUT server also controls the AO table. If the AOB part of the HUT server doesn' | ||
Line 350: | Line 295: | ||
* If the top of a server goes blank, try typing " | * If the top of a server goes blank, try typing " | ||
- | * If the server screen fills with jibberish, try hitting CTRL-l in the server to clear it. | + | * If the server screen fills with jibberish, try hitting CTRL-L in the server to clear it. |
==== Server is frozen ==== | ==== Server is frozen ==== |