Differences

This shows you the differences between two versions of the page.

--- chara:trouble_shooting [2018/09/25 16:05]
gail_stargazer
+++ chara:trouble_shooting [2021/09/28 15:02]
gail_stargazer
@@ Line 9: / Line 9: @@
   * Were the clocks synced? Make sure the [SYNC CLOCKS] button on Cosmic Debris has been pushed to start the night. If the OPLE server does not display the correct CHARA time and the errors don't read (0) or (1), the clocks were not synced. As the VME runs on its own clock and does not use the NTP server, it can drift over the course of the night and cause problems with finding fringes, even if it was synced at the start of the night. Syncing the clocks multiple time during the night may help to avoid this hidden clock problem.
-  * Did the Astrolib update on OPLE? If the job queue is stopped too soon after slewing on Cosmic Debris, the correct calculations for the carts will not be done by OPLE and you may be searching for fringes with the wrong star data. The proper star can be entered by typing hd …. into the OPLE server and hitting ENTER.
+  * Did the Astrolib update on OPLE? If the job queue is stopped too soon after slewing on Cosmic Debris, the correct calculations for the carts will not be done by OPLE and you may be searching for fringes with the wrong star data. Hit STAR ACQUIRED on CD to update ople. The proper star can also be entered by typing hd #### into the OPLE server and hitting ENTER.
+  * Are the PoP's correct? After a PoP change, the PoP's are sometimes not updated in CD or ople.
+  * Are the carts behaving or are there vibrations or jumps of 100 or more microns every 3-6 seconds? Restart the ople server if they persist.
   * Is the target a high proper motion star? Red dwarfs are close stars and can have high proper motions. Scan a wider range to see if it is outside of the usual calculated scan range. Binaries can also have very high offsets from the expected position due to mistakenly using astromod calculations from the companion star.
+  * Do you have enough flux from each telescope or on each baseline?
   * Did you get the same star in each telescope? Sometimes a busy star field and poor pointing of the telescopes can lead to the wrong star being acquired and locked by tiptilt. View the stars in the finder window to see if all the stars match.
   * Check the CHARA time on the GPS server. The "Ext-CHARA," "CHARA-Sys," and "Ext-Sys" time offsets listed on the GPS server should be small (< 0.01 sec). If there are large time offsets, then a GSYNC might be needed.
@@ Line 28: / Line 31: @@
 ==== Restarting Servers using the bootlaunch paradigm ====
- \\ A number of servers use an interim bootlaunch paradigm to restart. This is confined to servers that run on ubuntu machines, namely the telescope bunker computers and gps. The basic syntax is "bootlaunch_<server>" where "<server>" is replaced by the server the script is designed to address. The scripts have a number of safeties built in, so it is safe to run them even if a server is already running – they just output the process ID of the running server. The scripts also take care of the entry in socket manager as well any serial port lock files. All the pertinent information is world writeable, so one should be able to run a bootlaunch script as observe. \\  \\ One thing of note about the output of the bootlaunch scripts, they call a number of other programs which themselves have output that may be misleading in the context of bootlaunch. Chief among these is the output of "tsockman". If a server stopped unexpectedly, it may leave behind an entry in the socket manager. In order to launch a new server, one needs to clean out the socket manager entry if it is there. To do that, "tsockman remove <entry>" is called to remove "<entry>" before the new server is launched. If there is no entry, tsockman will respond with "Process by that name does not exist". THIS IS NORMAL and is not indicative of an error. The server in question launched (without fanfare) right after that output text. \\  \\ Here are the available bootlaunch scripts as of June 2017: \\  \\ gps computer:
+If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list".
+To restart a server, log on to the machine that runs the server and type "bootlaunch_master". This script will go through the list of executables and will check which servers are running. If a server isn't running it bootlaunch_master will remove it from socket manager, clear the lock file, and relaunch the server. The instructions below describe how to restart individual servers, but this should not be necessary anymore. \\  \\ A number of servers use an interim bootlaunch paradigm to restart. This is confined to servers that run on ubuntu machines, namely the telescope bunker computers and gps. The basic syntax is "bootlaunch_<server>" where "<server>" is replaced by the server the script is designed to address. The scripts have a number of safeties built in, so it is safe to run them even if a server is already running – they just output the process ID of the running server. The scripts also take care of the entry in socket manager as well any serial port lock files. All the pertinent information is world writeable, so one should be able to run a bootlaunch script as observe. \\  \\ One thing of note about the output of the bootlaunch scripts, they call a number of other programs which themselves have output that may be misleading in the context of bootlaunch. Chief among these is the output of "tsockman". If a server stopped unexpectedly, it may leave behind an entry in the socket manager. In order to launch a new server, one needs to clean out the socket manager entry if it is there. To do that, "tsockman remove <entry>" is called to remove "<entry>" before the new server is launched. If there is no entry, tsockman will respond with "Process by that name does not exist". THIS IS NORMAL and is not indicative of an error. The server in question launched (without fanfare) right after that output text. \\  \\ Here are the available bootlaunch scripts as of June 2017: \\  \\ gps computer:
   * bootlaunch_beamsamp – Starts the beam sampler servers, BS1 and BS2.
@@ Line 45: / Line 50: @@
 ==== Restarting Servers using the rc.local file ====
- \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm. \\  \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /chara/observe/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence. \\
+ \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm. \\  \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence. \\
  \\
 Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn't work then type "ssh //name//" where name is the computer name. \\  \\ Find out if the server is running by typing "ps aux | grep //server_name//" where server_name is the name of the server. \\ [ctrscrut:599] ps aux | grep pico_1 \\ observe 9281 0.0 0.0 61156 692 pts/3 S+ 13:58 0:00 grep pico_1 \\ observe 12578 0.0 0.0 24524 11212 ? S Jun16 33:14 /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg \\  \\ If the entry for the dead server shows up in the process list, then identify the process identification number (12578 for the example above) and kill the server by typing "kill -9 //PID//" where PID is the process identification number. \\  \\ Look up the commands to restart the server by typing "more /etc/rc.local" (this is relevant for servers that run in the background). Press the space bar to scroll through the contents of the rc.local file. Locate the commands relevant for the server that needs to be restarted and copy and paste into a terminal window: \\  \\ #Start PICO server for PICO #1 \\ /bin/rm -f /var/lock/LCK..ttyC8 \\ /usr/local/bin/tsockman remove PICO_1 \\ /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg & \\  \\ The first command removes the lock to allow the server to restart. The second command removes the name from the socket manager listing. The last command restarts the server. Note that if you are restarting the servers as observe, you will need to remove the part of the command in the rc.local file that saves information in /var/log///server_nam//e.log file (the actual command typed should resemble the last line above). \\  \\ There are text files on the desktop with many of the restart commands. Use these files for quick access to the relevant commands. The commands are edited and can be copied exactly as written. Files include Dome servers and all servers running on ctrscrut. Many of these commands are also located on the [[:chara:restarting_servers|Restarting Servers]] page.
@@ Line 52: / Line 57: @@
  \\ The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on ople, not ctrscrut. \\  \\ To start the shutter server on ople: \\  \\ Log into the ople computer and kill the process labeled shutters with the PID as described in **Restarting Servers**  above. \\ Turn off the power to the Shutters with the switch on the computer rack which is to the left of the computer desk and marked "SHUTTERS". Restart the Shutters server with the commands below. After restarting the server and testing the gui to see that it works, turn the SHUTTERS power back on with the switch. There is a printed sheet of directions in the lab to help you. \\  \\ /usr/local/bin/tsockman rm shutters \\ ctrscrut/usr/local/bin/shutter_server /ctrscrut/chara/etc/shutter.cfg &
+==== MIRC-X CredoneImAcq Server ====
+The MIRC-X CredoneImAcq server and GUI can be opened using the "credone" icon on the wolverine desktop. Note that you can only have one of these open at a time. If you get an error message and can't bring up credone, then close the error message and follow the steps below to remove the dead processes first:
+From a terminal window, log on to mircx (old observe password): \\ ssh -X spooler@mircx
+Look up credone process: \\ ps aux | grep credone
+Kill all credone processes (including the ones for the error messages) using "kill -9 PID" where PID is the process number. Note that credone can't be restarted if there are any error processes in the list. Check "ps aux | grep credone" one more time to make sure all processes are cleared (the only one that should show up is the grep command that was just issued).
+Then restart credone using the desktop icon or by issuing the following command on spooler@mircx: \\ credoneImAcq –no-display
+(the first "–" should be two dashes: - and -).
 ===== Telescopes and Dome Servers =====
@@ Line 94: / Line 113: @@
   * ENABLE the telescope using the telescope or dome GUI. Check to see if the command and telescope positions are correct. If they are, then turn on the power for the telescope drives. Re-enter the star information by entering the HD number in the telescope server and click [GO NEXT] to send the telescope to the star. Make sure that the telescope is behaving as expected. The telescope might have drifted a bit, so if you can't find the star, it might be necessary to use the Telrad on a bright star to re-initialize the pointing.
   * Restart commands for the dome servers are also listed on the desktop in a text file.
 <code>
    \
  ==== Telescope is tracking poorly, overshooting in slew, oscillating. ====
 </code>
 <code>
 This might mean that the gain for the tracking servo is wrong. Note that changing this gain can be dangerous, especially if you set it too high as that can cause the telescope to oscillated and damage the drives. Please only do this if you are very very sure that it is necessary. Symptoms of bad gain are:    The scope over shoots the position while slewing. The star will be seen to move out of the window and may come back after a few seconds. This means the slewing gain is too low.  The scope oscillates when tracking or after a slew. The star will be tracing an ellipse, figure eight or other looping shape. This means the tracking gain is too low. You can damp this out with the telescope or dome gui by disabling the scope, then re-enabling it. Adjust the gain upward and watch it on the next slew.    In all cases if either gain is too high the scope will go into "Fog Horn" mode, which is bad. You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high.    On 10-22-2016, the gain settings were:    | Scope  | AZ Slewing  || EL Slewing  || AZ Tracking  || EL Tracking  || Date Updated  |
 </code>
@@ Line 116: / Line 137: @@
 ==== How to Adjust CPUMotor Gains ====
- \\ The GAIN controls the gain of the feedback between the encoder and the drive velocity. A high gain means a "stiffer" response, but can lead to oscillations or fog-horning if it's too high. \\  \\ The Fn controls the maximum frequency of the servo response. A high Fn means higher frequencies are allowed through, which can mean correcting for faster problems but if too high can also lead to oscillations or fog-horning. \\  \\ The software will not make changes to either of these quickly as that is a dangerous thing to do. There is NO POINT to clicking the up or down buttons more than once every few seconds. Indeed it is bad to do so as you will confuse the software. The change between slewing and tracking is also slow for similar reasons. This is why sometimes the "wrong" thing seems to change. It is a sign that you are trying to do things too quickly. \\  \\ Slew and tracking mode work differently, mostly because the speeds are so different. \\  \\ IN SLEWING MODE \\  \\ - If the gain is too low you will overshoot the target. \\  \\ - If the gain is too high it will fog-horn. \\  \\ - If the Fn is too high it will also fog-horn, even at low gains. \\  \\ You need to have the lowest possible gain and Fn in slew mode that doesn't overshoot the target. Fn in slew mode should almost never be higher than 4. If it is, please turn it back down to 4. If you think this is a problem please let Theo know, along with a detailed explanation of what happened. \\  \\ IN TRACKING MODE \\  \\ - If the gain is too low it will keep moving between slewing and tracking. \\  \\ - If the gain is too high it will fog-horn. \\  \\ - The same goes for Fn. \\  \\ In tracking mode you want the highest gain and Fn that allows the telescope to track well without fog-horning. If it "oscillates", which you will see in the green dots of tiptilt oscillating, try turning up the gain, and also try turning down the Fn. \\  \\ Some final remarks: \\  \\ - The gain is temperature dependent, so when the temperature changes these things will change, but more so for tracking. \\  \\ - The tiptilt system almost never causes oscillations, it almost always shows you that the scope is oscillating. If the white dots are centered on tiptilt and the green dots are moving the tiptilt is doing it's job and correcting for scope motion. \\  \\ - If a drive gets disabled at the end of a slew, the gain is too low. At low gain and low velocity the encoder signal changes very slowly or doesn't change at all. After 5 secs the software interprets this as a stall and disables the drive. The tricky part is that increasing the gain to avoid this situation might make the telescope to oscillate during the next slew. So the gain should be low (4 or even 1) during slewing but higher 7 or 10 when the telescope is basically at the target position. \\ {{https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif?direct&|external image cleardot.gif}}
+ \\ The GAIN controls the gain of the feedback between the encoder and the drive velocity. A high gain means a "stiffer" response, but can lead to oscillations or fog-horning if it's too high. \\  \\ The Fn controls the maximum frequency of the servo response. A high Fn means higher frequencies are allowed through, which can mean correcting for faster problems but if too high can also lead to oscillations or fog-horning. \\  \\ The software will not make changes to either of these quickly as that is a dangerous thing to do. There is NO POINT to clicking the up or down buttons more than once every few seconds. Indeed it is bad to do so as you will confuse the software. The change between slewing and tracking is also slow for similar reasons. This is why sometimes the "wrong" thing seems to change. It is a sign that you are trying to do things too quickly. \\  \\ Slew and tracking mode work differently, mostly because the speeds are so different. \\  \\ IN SLEWING MODE \\  \\ - If the gain is too low you will overshoot the target. \\  \\ - If the gain is too high it will fog-horn. \\  \\ - If the Fn is too high it will also fog-horn, even at low gains. \\  \\ You need to have the lowest possible gain and Fn in slew mode that doesn't overshoot the target. Fn in slew mode should almost never be higher than 4. If it is, please turn it back down to 4. If you think this is a problem please let Theo know, along with a detailed explanation of what happened. \\  \\ IN TRACKING MODE \\  \\ - If the gain is too low it will keep moving between slewing and tracking. \\  \\ - If the gain is too high it will fog-horn. \\  \\ - The same goes for Fn. \\  \\ In tracking mode you want the highest gain and Fn that allows the telescope to track well without fog-horning. If it "oscillates", which you will see in the green dots of tiptilt oscillating, try turning up the gain, and also try turning down the Fn. \\  \\ Some final remarks: \\  \\ - The gain is temperature dependent, so when the temperature changes these things will change, but more so for tracking. \\  \\ - The tiptilt system almost never causes oscillations, it almost always shows you that the scope is oscillating. If the white dots are centered on tiptilt and the green dots are moving the tiptilt is doing it's job and correcting for scope motion. \\  \\ - If a drive gets disabled at the end of a slew, the gain is too low. At low gain and low velocity the encoder signal changes very slowly or doesn't change at all. After 5 secs the software interprets this as a stall and disables the drive. The tricky part is that increasing the gain to avoid this situation might make the telescope to oscillate during the next slew. So the gain should be low (4 or even 1) during slewing but higher 7 or 10 when the telescope is basically at the target position.
+==== E2 AOB Dichroic Recovery ====
+{{:chara:files:beaconrecoveryinstructions.pdf|E2 AOB Dichroic Recovery Document}}
+==== E1 Hut and Cooler Communications Recovery ====
+{{:chara:files:e1_hut_coms_recovery.pdf|E1 HuT and Cooler Communications Recovery}}  \\
+{{https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif?direct&|external image cleardot.gif}}
 ====== Dome issues ======
@@ Line 299: / Line 329: @@
 Is the above the object you are looking for? [y/n] y \\ Added "BD+41 3807" to the database.
-The object "BD+41 3807" has been added to the database as CHARA number 320414.
+In the example above, the object "BD+41 3807" has been added to the database as CHARA number 320414.
- \\ __Targets that are not in the CHARA database or in SIMBAD__ \\  \\ Some objects like recently discovered novae are not in SIMBAD. Entering dbadd <star designation> for a star not in the SIMBAD database returns the message "Simbad is unable to find an object matching <star designation>. Try using a different catalog designation, or use the "-m" switch to add the object manually." \\  \\ The command dbadd -m is used by entering the star name and coordinates in this format: \\  \\ Usage: dbadd -m <name> <RA> <Dec> \\  \\ <name>: Object ID (no spaces) \\  \\ <RA> : XXhXXmXX.X or XXhXX.X (no spaces) \\  \\ <Dec> : XXdXXmXX.X or XXdXX.X (no spaces) \\  \\ There are cases where an object is not in SIMBAD and doesn't return a result in dbadd, but is in fact in the CHARA database. Novae and AGN's are the likely objects that cause this result. At times, objects have multiple entries and several CHARA numbers since the names can be so unique. A system will be developed to find these entries without knowing or having a conventional database designation. \\  \\ __Binary stars__ \\  \\ Some binary stars have a common HD number with an A or B after them. These can cause problems as Cosmic Debris does not accept non numerical entries when filling in star designation. These stars are likely in CHARA's database but need to be searched in dbadd or on SIMBAD to get the CHARA number or another designation. \\ When a recognized identifier is entered, the HD number with A or B will usually display. Confirm that the coordinates, magnitudes and spectral type match the star desired. If they do not match, you may have the wrong star or the database may have the wrong info and the baseline solution will be affected. Offsets based on incorrect coordinates or misidentifications can move the fringes many thousands of microns away from the calculated position. This can happen when observing a dim companion (B) using the brighter (A) companion's coordinates. Inform the observers that noting and using the CHARA number will save time next time the target is observed. \\  \\ __Editing the database__ \\  \\ If you find a mistake in the database, please send an email to Nils to have it corrected. Identify what you believe to be the error and what is the correct information.
+ \\ __Targets that are not in SIMBAD__ \\  \\ Some objects like recently discovered novae are not in SIMBAD. Entering dbadd <star designation> for a star not in the SIMBAD database returns the message "Simbad is unable to find an object matching <star designation>. Try using a different catalog designation, or use the "-m" switch to add the object manually." \\  \\ The command dbadd -m is used by entering the star name and coordinates in this format: \\  \\ Usage: dbadd -m <name> <RA> <Dec> \\  \\ <name>: Object ID (no spaces) \\  \\ <RA> : XXhXXmXX.X or XXhXX.X (no spaces) \\  \\ <Dec> : XXdXXmXX.X or XXdXX.X (no spaces) \\  \\ There are cases where an object is not in SIMBAD and doesn't return a result in dbadd, but is in fact in the CHARA database. Novae and AGN's are the likely objects that cause this result. At times, objects have multiple entries and several CHARA numbers since the names can be so unique. A system will be developed to find these entries without knowing or having a conventional database designation. \\  \\ __Binary stars__ \\  \\ Some binary stars have a common HD number with an A or B after them. These can cause problems as Cosmic Debris does not accept non numerical entries when filling in star designation. These stars are likely in CHARA's database but need to be searched in dbadd or on SIMBAD to get the CHARA number or another designation.
+When a recognized identifier is entered, the HD number with A or B will usually display. Confirm that the coordinates, magnitudes and spectral type match the star desired. If they do not match, you may have the wrong star or the database may have the wrong info and the baseline solution will be affected. Offsets based on incorrect coordinates or misidentifications can move the fringes many thousands of microns away from the calculated position. This can happen when observing a dim companion (B) using the brighter (A) companion's coordinates. Inform the observers that noting and using the CHARA number will save time next time the target is observed. \\  \\ __Editing the database__ \\  \\ If you find a mistake in the database, please send an email to Nils to have it corrected. Identify what you believe to be the error and what is the correct information.
 ==== Adding an anchor to the wiki page ====
-To add an anchor to the wikipage, type the following where you want the anchor inserted (without the spaces between the symbols): \\ [ [ #AnchorName ] ] \\  \\ Highlight the text you want to be a link to the anchor, click the link symbol above and select anchor. Select a page(such as Trouble Shooting) where the anchor is located and give the same name for the anchor. \\  \\ Last updated 2017-04-12
+To add an anchor to the wikipage, type the following where you want the anchor inserted (without the spaces between the symbols): \\ [ [ #AnchorName ] ] \\  \\ Highlight the text you want to be a link to the anchor, click the link symbol above and select anchor. Select a page(such as Trouble Shooting) where the anchor is located and give the same name for the anchor.
+==== Displays for the MIRC-X wolverine2 monitors ====
+Follow instructions in the [[:chara:mircx_monitors|linked page to display the MIRC-X wolverine2 monitors]] in the control room.
-\\
+ \\ Last updated 2017-04-12

The CHARA Array Wiki

User Tools

Site Tools

Differences

Page Tools