Differences

This shows you the differences between two versions of the page.

--- chara:trouble_shooting [2021/11/09 15:27]
gail_stargazer
+++ chara:trouble_shooting [2022/01/18 22:32]
charaobs
@@ Line 38: / Line 38: @@
   * bootlaunch_zaber – Starts the ZABER_2 server.
- \\ telescope bunker computers:
+ \\ telescope bunker computers are now using bootlaunch_master to start each of the servers listed below.
   * bootlaunch_hut – Starts the E1_HUT, E2_HUT, S1_HUT, S2_HUT, W1_HUT, or W2_HUT server, depending on the machine it's launched from.
@@ Line 50: / Line 50: @@
 ==== Restarting Servers using the rc.local file ====
- \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm. \\  \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence. \\
+ \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm, which may not be any at this point. \\  \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence. \\
  \\
 Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn't work then type "ssh //name//" where name is the computer name. \\  \\ Find out if the server is running by typing "ps aux | grep //server_name//" where server_name is the name of the server. \\ [ctrscrut:599] ps aux | grep pico_1 \\ observe 9281 0.0 0.0 61156 692 pts/3 S+ 13:58 0:00 grep pico_1 \\ observe 12578 0.0 0.0 24524 11212 ? S Jun16 33:14 /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg \\  \\ If the entry for the dead server shows up in the process list, then identify the process identification number (12578 for the example above) and kill the server by typing "kill -9 //PID//" where PID is the process identification number. \\  \\ Look up the commands to restart the server by typing "more /etc/rc.local" (this is relevant for servers that run in the background). Press the space bar to scroll through the contents of the rc.local file. Locate the commands relevant for the server that needs to be restarted and copy and paste into a terminal window: \\  \\ #Start PICO server for PICO #1 \\ /bin/rm -f /var/lock/LCK..ttyC8 \\ /usr/local/bin/tsockman remove PICO_1 \\ /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg & \\  \\ The first command removes the lock to allow the server to restart. The second command removes the name from the socket manager listing. The last command restarts the server. Note that if you are restarting the servers as observe, you will need to remove the part of the command in the rc.local file that saves information in /var/log///server_nam//e.log file (the actual command typed should resemble the last line above). \\  \\ There are text files on the desktop with many of the restart commands. Use these files for quick access to the relevant commands. The commands are edited and can be copied exactly as written. Files include Dome servers and all servers running on ctrscrut. Many of these commands are also located on the [[:chara:restarting_servers|Restarting Servers]] page.
@@ Line 90: / Line 90: @@
 <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software. \\  \\
 .</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way. \\  \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope.</font>
-<font inherit/inherit;;initial;;white>2. M</font>ake sure the all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way. <font inherit/inherit;;initial;;white>3. Click the OVERRIDE ON button in domegui MANUAL tab.</font>
 <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>4. Move the scope a bit back toward the direction it was coming from – for example, if the scope stopped at AZ 268 while rotating clockwise, move it back to 265 or so using AZ DEC. Then press STOP.</font>
@@ Line 99: / Line 97: @@
 <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>6. Click OVERRIDE OFF, go to the AUTO tab and press AUTO, then NEXT (also in the obsgtk; this will restore the original star's demand position to the scope).</font>
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7. When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90.</font>
+<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7. When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90.  The LED will show red when on the limit switch and is tripped, ie. limiting motion of the scope.  The LED will be yellow if it has tripped and is in the caution range, but not on a limit switch.  A fine Allen key can be used to push the internal reset button. It will turn the LED green when restored.</font>
 ==== The Telescope won't track ====
@@ Line 110: / Line 108: @@
 ==== Dome Server Restart ====
+Dome servers are now started using the bootlaunch_master command.  Kill the server by finding the process ID with bootlaunch_master and restart it with bootlaunch_master also.  The manual process that has been superceded is listed below.
 To manually start the dome server:
@@ Line 227: / Line 227: @@
 ==== E1 Hut and Cooler Communications Recovery ====
-==== E1 HuT and Cooler Communications Recovery \\ \\ S2 Acquisition Server Restart ====
+{{:chara:files:e1_hut_coms_recovery.pdf|E1 HuT and Cooler Communications Recovery}}
+==== S2 Acquisition Server Restart ====
 S2's acquisition server resides on "S2pi4" computer. Logging on to it (ssh s2pi4) and running bootlaunch will restart it
@@ Line 238: / Line 240: @@
 Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope's position. If not, it will insist on being in the wrong place. If it is not at the same AZ as the scope, manually move it until it is centered on the telescope in spycam 1. If the dome AZ does not read the same as the telescope AZ, enter the scope AZ in the position box of the DOME tab of appropriate dome server and hit the INIT POS button to tell it at what AZ it is. \\  \\
-If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker.
+If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker. Sometimes the drive wheel jumps in the track and cannot turn the dome, even when the motor works.  This will need to be fixed during the day.
 ==== Dome does not open ====
@@ Line 265: / Line 267: @@
 [[:chara:old_lab_tiptilt_server|Instructions for the old lab tiptilit system are archived here.]]
-The tiptilt server controls the CCD based tiptilt detection system. \\  \\
-Before you start the tiptilt server, you must ensure that the power to the cooling system and the CCD iteslf is on. It is extremely important that the cooler be running before you turn on the CCD and is only turned off if you are sure the CCD is NOT running. You can start the server from the X windows menu or with the command xtiptilt. \\  \\
-Note that there are background counts and read noise to deal with. Whenever you change the frame rate, please ensure that the bias frame is OK. The server will attempt to load an old bias frame that should work, but if things are not working, try making a new bias frame by ensuring that the detector is in the dark and typing "mkbias" into the tiptilt server. \\  \\
-In the tiptilt GUI windows, the white dots represent the starlight while the green dots represent the motion applied to telescope's secondary mirror to keep the starlight centered. When tiptilt is locked the white dots will be brought to the center of the tiptilt window. The green dots should be mostly centered also. W2 and E2 telescopes have a small oscillation that show as back and forth plots of the green dots.
-==== Tiptilt server complains about the CCD ====
-Is the CCD turned on? When the tiptilt server starts up it tries no more than five times to communicate with the CCD. If they all fail, it will give up. If this happens, try cycling the power to the CCD and try again. If this fails, connect to the tiptilt machine and type the command rtccdAPIDemo, which should return with no errors. Try this command a few times, but if it still fails, there is a more serious problem. Turn off the CCD and reboot the tiptilt computer. If it still fails, I am afraid you are in more serious trouble. \\  \\
-Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine. \\  \\
-Also note that cycling the power on the CCD can cause harm so be sure you need to do it before trying it. Also, it is important to wait for at least 20 seconds after turning off the power before turning it on again.
-==== Tiptilt doesn't seem to be talking to the telescopes ====
-Sometimes the telescope server will not show that TT is running. It will show 0Hz for a signal rate for TT. Running TIPTILT COMM will not get it started while other scopes do show it starting. Close and restart any telescope servers that won't connect after two tries of TIPTILT COMM. \\
-Note: There is more info in the software manual on this topic, but I wasn't sure if it was still relevant.
-==== Tiptilt server says the clock isn't running ====
-First check whether the clock itself is running and the other machines receive the clock signal. Look at the clock cards at the back of other computers in the rack. The clock cards have three LEDs, one yellow and two greens. If the computer is receiving the clock signal properly all three LEDs should blink, but at a different rate. If the LEDs on all the clock cards are solid then reboot the GPS computer. When the GPS computer is down, it is best to cycle the power also on the box right above the GPS computer. \\  \\
-If the clock appears to be working properly on other machines and not on the tiptilt now it is time to reboot tiptilt. \\  \\
-[There is a bug in the real time part of the CCD code. It is caused by the clock in the tiptilt system either not running at all or having been set to a time very different from the last time the CCD ran.] \\  \\
-For the time being the only solution is to reboot tiptilt, but do so from the lab. Power OFF the CCD, then reboot the tiptilt machine and go into the BIOS. Make sure that interrupt 11 has been set to ISA legacy, save the BIOS and reboot. When the clock card LEDs in the tiptilt machine indicate proper clock signal, turn the CCD back on and start the tiptilt server. \\  \\
-Also, sometimes Serial Port 3 grabs IRQ 11 which stops the clock from running. Since there is no serial port 2 it's safe to disable this in the BIOS. This problem normally comes up when there has been a power outage. \\  \\
-Sometimes syncing the clock can also cause this problem, but that should be fixed soon. If it does, exit the tiptilt server, log in as root, and reload the tiptilt model using the following commands: \\
-/sbin/rmmod tiptilt_rt \\
-/sgin/insmod /usr/local/modules/tiptilt_rt.o \\  \\
-Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine.
-==== Tiptilt is not locking on a star or locks, but lets the star drift away ====
-  * Are all the mirror covers open? [Note: W1 M7 cover sometimes needs two clicks to open, despite Scope-monitor indicates open.]
-  * Has TIPTILT COMM been run from Cosmic Debris?
-  * Check the ACQ alignment to make sure the tick marks are centered on the laser.
-  * If there is plenty of starlight getting into tiptilt, then try re-initializing by clicking [TIPTILT COMM] on Cosmic Debris.
-  * If there are a large number of background counts on TT, then close the M5 cover on the telescope, and click [DBIAS] on the TIPLTILT GUI to clear the background counts. Re-open the M5 cover.
-  * If a star drifts even with TT locked, there could be a bright sky or light from other beams getting into the affected telescope's beam. This will show in the TT server as much lower counts than the other locked stars, but still high enough to lock TT. Use the laser to find the correct position to lock TT.
-  * If TT unlocks and the star drifts, the TT servo may not have engaged. This is engaged with the TIPTILT button on the telescope gui. Clicking it may not start the servo the first time. Try it again if the green dots drift on the TT plot windows or if the Servo status in the telescope gui reads None instead of Wobb 1.
-  * Are the TIPTILT buttons turned on from the POWER GUI?
-==== Tiptilt servo oscillates ====
-You will see the oscillation in the green dots of the tiptilt GUI windows. Sometimes you can also see the oscillation in the white starlight dots or as an elongation of the star when looking at the ACQ field while tiptilt is locked. Some scopes have an oscillation that has not yet been diagnosed. W2 is one that usually oscillates. A diagonal motion in the tiptilt box indicates an oscillation in one axis only, while a vertical/horizontal motion indicates an oscillation in both directions. Motion from the upper right to lower left corresponds to elevation axis while motion from the upper left to lower right corresponds to the azimuth axis. (I think you can check direction by typing sin into telescope server to send sine waves to the telescope.) There are a few ways you can try to correct the oscillation manually tuning the servo:
-  * Type "tune" into the tiptilt server. Select the appropriate telescope. Turning the gain down normally helps.
-  * Type "tune" into the appropriate telescope server. The default value for the proportional term is -0.5 and differential term is 0.0. Adjust these values between -0.2 to -1.0 for proportional and 0.0 to 0.2 for differential.
-  * Read section on adjusting the Telescope Tracking Gain using the Dome Server GUI (ref).
-==== Tiptilt is saturating ====
-  * Tiptilt saturates at ~ 200,000 counts. If you are near this limit, you can reduce the TT exposure time to lower the number of counts.
-  * Set the TT exposure time on Cosmic Debris in the box for "Tiptilt (mS)". This will set the TT exposure time when slewing to a new target.
-  * To change the exposure when already at a target, then click the [EXP] button on the TipTilt GUI. This will bring up a dialog box where you can enter the new exposure time in msec. Check to make sure the tiptilt frequency (in Hz) changes on the tiptilt server after changing the exposure time. If the frame rate doesn't change, then set the exposure time back to the old value and try entering the new value again. You might have to do this a few times to actually get the frame rate to change.
-  * Frame rates for given exposure times:
-      * ExpTime = 5 msec, Frame Rate = 157 Hz
-      * ExpTime = 2 msec, Frame Rate = 299 Hz
-      * ExpTime = 1 msec, Frame Rate = 427 Hz
-==== Tiptilt counts are way too low ====
-Try using a slower frame rate or increasing the NSUM. Also ensure that the acquistion is properly aligned with the laser. To change the frame rate, click the [EXP] button on the tiptilt GUI and enter a longer integration time. Remember to change "Tiptilt (mS)" on Cosmic Debris to keep the same exposure time when slewing to the next target.
-==== Tiptilt counts are negative ====
-The bias frame is bad. Get a new one or turn it off.
 ===== OPLE and Metrology =====
@@ Line 348: / Line 285: @@
   * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror.
-  * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has homed to the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. You can also find the size of the error by typing the command "homechk S1" into the ople server. Hit the [HOME] button and it should move forward and trigger the home switch. Hit [TRACK] to home it to the new home position.
+  * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has stopped on the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes.  The X does not guarantee the cart has retained its home position.  This can only be done by using the CHECK button on the ople gui or by typing the command "homechk S1" into the ople server.  When the error value is displayed on the ople server, hit ESC to clear the display. Click on the [OL] and [MAN] buttons and hit [TRACK] to send the cart to the desired cart position.
 ===== Beam Samplers =====

The CHARA Array Wiki

User Tools

Site Tools

Differences

Page Tools