This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
chara:trouble_shooting [2021/11/09 15:13] gail_stargazer |
chara:trouble_shooting [2022/01/21 17:55] charaobs |
||
---|---|---|---|
Line 38: | Line 38: | ||
* bootlaunch_zaber – Starts the ZABER_2 server. | * bootlaunch_zaber – Starts the ZABER_2 server. | ||
- | \\ telescope bunker computers: | + | \\ telescope bunker computers |
* bootlaunch_hut – Starts the E1_HUT, E2_HUT, S1_HUT, S2_HUT, W1_HUT, or W2_HUT server, depending on the machine it's launched from. | * bootlaunch_hut – Starts the E1_HUT, E2_HUT, S1_HUT, S2_HUT, W1_HUT, or W2_HUT server, depending on the machine it's launched from. | ||
Line 50: | Line 50: | ||
==== Restarting Servers using the rc.local file ==== | ==== Restarting Servers using the rc.local file ==== | ||
- | \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm. \\ \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[: | + | \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm, which may not be any at this point. \\ \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[: |
\\ | \\ | ||
Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn' | Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn' | ||
Line 90: | Line 90: | ||
<font 14px/ | <font 14px/ | ||
2.</ | 2.</ | ||
- | |||
- | <font inherit/ | ||
<font 14px/ | <font 14px/ | ||
Line 99: | Line 97: | ||
<font 14px/ | <font 14px/ | ||
- | <font 14px/ | + | <font 14px/ |
==== The Telescope won't track ==== | ==== The Telescope won't track ==== | ||
Line 110: | Line 108: | ||
==== Dome Server Restart ==== | ==== Dome Server Restart ==== | ||
+ | |||
+ | Dome servers are now started using the bootlaunch_master command. Kill the server by finding the process ID with bootlaunch_master and restart it with bootlaunch_master also. The manual process that has been superceded is listed below. | ||
To manually start the dome server: | To manually start the dome server: | ||
- | - Make sure the power to the drives is OFF. | + | 1. Make sure the power to the drives is OFF. |
- | | + | |
- | | + | 2. Login to the relevant computer as root. For example, type " |
- | - Try and stop the server gracefully: kill -2 29953 5. You should then check that the server has indeed stopped: [s1:600] tsockman get dome_S1 Name : dome_S1 Machine : s1.chara-array.org PID : 15635 Commands : -1 Data : -1 Message : 2008 Restart : / | + | |
- | | + | 3. Work out the process ID number (PID), either with the command |
- | | + | |
- | - Hit REOPEN and ENABLE on the domegtk, and type " | + | (s1:1001) tsockman get dome_S1 |
+ | Name : dome_S1 | ||
+ | Machine : s1.chara-array.org | ||
+ | PID : 29953 \\ | ||
+ | Commands : -1 \\ | ||
+ | Data : -1 \\ | ||
+ | Message : 4002 \\ | ||
+ | Restart : / | ||
+ | or with \\ | ||
+ | (s1:1003) ps aux | grep dome \\ | ||
+ | theo 4473 0.0 0.0 61188 748 pts/3 S+ 10:45 0:00 grep dome \\ | ||
+ | observe 29953 18.5 0.4 35596 9860 ? Sl Apr21 416:11 / | ||
+ | It can also be found by pulling up the LIST on SOCKMAN and selecting the relevant dome. \\ \\ | ||
+ | So in this case the PID is 29953. | ||
+ | 4. Try and stop the server gracefully: kill -2 29953 | ||
+ | |||
+ | 5. You should then check that the server has indeed stopped: | ||
+ | [s1:600] tsockman get dome_S1 | ||
+ | Name : dome_S1 | ||
+ | Machine : s1.chara-array.org | ||
+ | PID : 15635 \\ | ||
+ | Commands : -1 \\ | ||
+ | Data : -1 \\ | ||
+ | Message : 2008 \\ | ||
+ | Restart : / | ||
+ | If the socket manager still thinks it's running you will need to stop it forcefully: kill -9 29953; tsockman rm dome_S1 | ||
+ | |||
+ | 6. Restart the dome server by copying the command at the end of the / | ||
+ | |||
+ | [s1:602] more / | ||
+ | <<< | ||
+ | #Run the dome server | ||
+ | / | ||
+ | / | ||
+ | (Note: the part of the command that saves information to / | ||
+ | |||
+ | 7. Turn the power to the drives back on. | ||
- | ==== Telescope clock is not correct ==== | + | 8. Hit REOPEN and ENABLE on the domegtk, and type " |
- | The NTP server has obsoleted this problem | + | You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem |
==== Telescope is not receiving the commanded position for a target ==== | ==== Telescope is not receiving the commanded position for a target ==== | ||
Line 145: | Line 181: | ||
==== Telescope is tracking poorly, overshooting in slew, oscillating. ==== | ==== Telescope is tracking poorly, overshooting in slew, oscillating. ==== | ||
- | This might mean that the gain for the tracking servo is wrong. Note that changing this gain can be dangerous, especially if you set it too high as that can cause the telescope to oscillated and damage the drives. Please only do this if you are very very sure that it is necessary. Symptoms of bad gain are: The scope over shoots the position while slewing. The star will be seen to move out of the window and may come back after a few seconds. This means the slewing gain is too low. The scope oscillates when tracking or after a slew. The star will be tracing an ellipse, figure eight or other looping shape. This means the tracking gain is too low. You can damp this out with the telescope or dome gui by disabling the scope, then re-enabling it. Adjust the gain upward and watch it on the next slew. In all cases if either gain is too high the scope will go into "Fog Horn" mode, which is bad. You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high. On 10-22-2016, the gain settings were: | Scope | AZ Slewing || EL Slewing || AZ Tracking || EL Tracking || Date Updated | | + | This might mean that the gain for the tracking servo is wrong. Note that changing this gain can be dangerous, especially if you set it too high as that can cause the telescope to oscillated and damage the drives. Please only do this if you are very very sure that it is necessary. Symptoms of bad gain are: The scope over shoots the position while slewing. The star will be seen to move out of the window and may come back after a few seconds. This means the slewing gain is too low. The scope oscillates when tracking or after a slew. The star will be tracing an ellipse, figure eight or other looping shape. This means the tracking gain is too low. You can damp this out with the telescope or dome gui by disabling the scope, then re-enabling it. Adjust the gain upward and watch it on the next slew. In all cases if either gain is too high the scope will go into "Fog Horn" mode, which is bad. This can be seen during slews on the twfs or labao as vibrating spots, usually in one axis. |
- | | |Gain|Fn|Gain|Fn|Gain|Fn|Gain|Fn| | | + | You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high. |
- | |S1|7|4|4|3|16|4|10|7|10-26-2017| | + | |
- | |S2|7| |7| |10| |10| |10-22-2016| | + | |
- | |E1|7| |7| |13| |10| |10-22-2016| | + | |
- | |E2|7| |7| |13| |10| |10-22-2016| | + | |
- | |W1|13| |10| |13| |10| |10-22-2016| | + | |
- | |W2|7| |7| |13| |13| |10-22-2016| | + | |
- | \\ | + | The usual values for slewing gains are 4-7 and tracking gains are 7-10. Note that these values may change as the temperatures change rapidly. |
- | Note that these values may change as the temperatures change rapidly. | + | |
==== How to Adjust CPUMotor Gains ==== | ==== How to Adjust CPUMotor Gains ==== | ||
Line 186: | Line 215: | ||
==== E1 Hut and Cooler Communications Recovery ==== | ==== E1 Hut and Cooler Communications Recovery ==== | ||
- | ==== E1 HuT and Cooler Communications Recovery | + | {{: |
+ | |||
+ | ==== S2 Acquisition Server Restart ==== | ||
S2's acquisition server resides on " | S2's acquisition server resides on " | ||
Line 197: | Line 228: | ||
Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope' | Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope' | ||
- | If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker. | + | If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker. Sometimes the drive wheel jumps in the track and cannot turn the dome, even when the motor works. This will need to be fixed during the day. |
==== Dome does not open ==== | ==== Dome does not open ==== | ||
Line 224: | Line 255: | ||
[[: | [[: | ||
- | |||
- | The tiptilt server controls the CCD based tiptilt detection system. \\ \\ | ||
- | Before you start the tiptilt server, you must ensure that the power to the cooling system and the CCD iteslf is on. It is extremely important that the cooler be running before you turn on the CCD and is only turned off if you are sure the CCD is NOT running. You can start the server from the X windows menu or with the command xtiptilt. \\ \\ | ||
- | Note that there are background counts and read noise to deal with. Whenever you change the frame rate, please ensure that the bias frame is OK. The server will attempt to load an old bias frame that should work, but if things are not working, try making a new bias frame by ensuring that the detector is in the dark and typing " | ||
- | In the tiptilt GUI windows, the white dots represent the starlight while the green dots represent the motion applied to telescope' | ||
- | |||
- | ==== Tiptilt server complains about the CCD ==== | ||
- | |||
- | Is the CCD turned on? When the tiptilt server starts up it tries no more than five times to communicate with the CCD. If they all fail, it will give up. If this happens, try cycling the power to the CCD and try again. If this fails, connect to the tiptilt machine and type the command rtccdAPIDemo, | ||
- | Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command " | ||
- | Also note that cycling the power on the CCD can cause harm so be sure you need to do it before trying it. Also, it is important to wait for at least 20 seconds after turning off the power before turning it on again. | ||
- | |||
- | ==== Tiptilt doesn' | ||
- | |||
- | Sometimes the telescope server will not show that TT is running. It will show 0Hz for a signal rate for TT. Running TIPTILT COMM will not get it started while other scopes do show it starting. Close and restart any telescope servers that won't connect after two tries of TIPTILT COMM. \\ | ||
- | Note: There is more info in the software manual on this topic, but I wasn't sure if it was still relevant. | ||
- | |||
- | ==== Tiptilt server says the clock isn't running ==== | ||
- | |||
- | First check whether the clock itself is running and the other machines receive the clock signal. Look at the clock cards at the back of other computers in the rack. The clock cards have three LEDs, one yellow and two greens. If the computer is receiving the clock signal properly all three LEDs should blink, but at a different rate. If the LEDs on all the clock cards are solid then reboot the GPS computer. When the GPS computer is down, it is best to cycle the power also on the box right above the GPS computer. \\ \\ | ||
- | If the clock appears to be working properly on other machines and not on the tiptilt now it is time to reboot tiptilt. \\ \\ | ||
- | [There is a bug in the real time part of the CCD code. It is caused by the clock in the tiptilt system either not running at all or having been set to a time very different from the last time the CCD ran.] \\ \\ | ||
- | For the time being the only solution is to reboot tiptilt, but do so from the lab. Power OFF the CCD, then reboot the tiptilt machine and go into the BIOS. Make sure that interrupt 11 has been set to ISA legacy, save the BIOS and reboot. When the clock card LEDs in the tiptilt machine indicate proper clock signal, turn the CCD back on and start the tiptilt server. \\ \\ | ||
- | Also, sometimes Serial Port 3 grabs IRQ 11 which stops the clock from running. Since there is no serial port 2 it's safe to disable this in the BIOS. This problem normally comes up when there has been a power outage. \\ \\ | ||
- | Sometimes syncing the clock can also cause this problem, but that should be fixed soon. If it does, exit the tiptilt server, log in as root, and reload the tiptilt model using the following commands: \\ | ||
- | /sbin/rmmod tiptilt_rt \\ | ||
- | / | ||
- | Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command " | ||
- | |||
- | ==== Tiptilt is not locking on a star or locks, but lets the star drift away ==== | ||
- | |||
- | * Are all the mirror covers open? [Note: W1 M7 cover sometimes needs two clicks to open, despite Scope-monitor indicates open.] | ||
- | * Has TIPTILT COMM been run from Cosmic Debris? | ||
- | * Check the ACQ alignment to make sure the tick marks are centered on the laser. | ||
- | * If there is plenty of starlight getting into tiptilt, then try re-initializing by clicking [TIPTILT COMM] on Cosmic Debris. | ||
- | * If there are a large number of background counts on TT, then close the M5 cover on the telescope, and click [DBIAS] on the TIPLTILT GUI to clear the background counts. Re-open the M5 cover. | ||
- | * If a star drifts even with TT locked, there could be a bright sky or light from other beams getting into the affected telescope' | ||
- | * If TT unlocks and the star drifts, the TT servo may not have engaged. This is engaged with the TIPTILT button on the telescope gui. Clicking it may not start the servo the first time. Try it again if the green dots drift on the TT plot windows or if the Servo status in the telescope gui reads None instead of Wobb 1. | ||
- | * Are the TIPTILT buttons turned on from the POWER GUI? | ||
- | |||
- | ==== Tiptilt servo oscillates ==== | ||
- | |||
- | You will see the oscillation in the green dots of the tiptilt GUI windows. Sometimes you can also see the oscillation in the white starlight dots or as an elongation of the star when looking at the ACQ field while tiptilt is locked. Some scopes have an oscillation that has not yet been diagnosed. W2 is one that usually oscillates. A diagonal motion in the tiptilt box indicates an oscillation in one axis only, while a vertical/ | ||
- | |||
- | * Type " | ||
- | * Type " | ||
- | * Read section on adjusting the Telescope Tracking Gain using the Dome Server GUI (ref). | ||
- | |||
- | ==== Tiptilt is saturating ==== | ||
- | |||
- | * Tiptilt saturates at ~ 200,000 counts. If you are near this limit, you can reduce the TT exposure time to lower the number of counts. | ||
- | * Set the TT exposure time on Cosmic Debris in the box for " | ||
- | * To change the exposure when already at a target, then click the [EXP] button on the TipTilt GUI. This will bring up a dialog box where you can enter the new exposure time in msec. Check to make sure the tiptilt frequency (in Hz) changes on the tiptilt server after changing the exposure time. If the frame rate doesn' | ||
- | * Frame rates for given exposure times: | ||
- | * ExpTime = 5 msec, Frame Rate = 157 Hz | ||
- | * ExpTime = 2 msec, Frame Rate = 299 Hz | ||
- | * ExpTime = 1 msec, Frame Rate = 427 Hz | ||
- | |||
- | ==== Tiptilt counts are way too low ==== | ||
- | |||
- | Try using a slower frame rate or increasing the NSUM. Also ensure that the acquistion is properly aligned with the laser. To change the frame rate, click the [EXP] button on the tiptilt GUI and enter a longer integration time. Remember to change " | ||
- | |||
- | ==== Tiptilt counts are negative ==== | ||
- | |||
- | The bias frame is bad. Get a new one or turn it off. | ||
===== OPLE and Metrology ===== | ===== OPLE and Metrology ===== | ||
Line 307: | Line 273: | ||
* Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror. | * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror. | ||
- | * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has homed to the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. | + | * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has stopped on the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. |
===== Beam Samplers ===== | ===== Beam Samplers ===== | ||
Line 482: | Line 448: | ||
\\ | \\ | ||
- | Last updated | + | Last updated |