User Tools

Site Tools


chara:trouble_shooting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
chara:trouble_shooting [2021/11/08 17:59]
charaobs [Dome does not close]
chara:trouble_shooting [2023/11/21 01:42] (current)
charaobs [No Power for Aperture Wheels]
Line 9: Line 9:
  
   * Were the clocks synced? Make sure the [SYNC CLOCKS] button on Cosmic Debris has been pushed to start the night. If the OPLE server does not display the correct CHARA time and the errors don't read (0) or (1), the clocks were not synced.   * Were the clocks synced? Make sure the [SYNC CLOCKS] button on Cosmic Debris has been pushed to start the night. If the OPLE server does not display the correct CHARA time and the errors don't read (0) or (1), the clocks were not synced.
-  * Did the Astrolib update on OPLE? If the job queue is stopped too soon after slewing on Cosmic Debris, the correct calculations for the carts will not be done by OPLE and you may be searching for fringes with the wrong star data. Hit STAR ACQUIRED on CD to update ople. The proper star can also be entered manually by typing hd #### into the OPLE server and hitting ENTER. +  * Did the Astrolib update on OPLE? If the job queue is stopped too soon after slewing on Cosmic Debris, the correct calculations for the carts will not be done by OPLE and you may be searching for fringes with the wrong star data. The star identifier will be displayed in the server window. If it is not correct, hit STAR ACQUIRED on CD to update OPLE. The proper star can also be entered manually by typing hd #### into the OPLE server and hitting ENTER. 
-  * Are the PoP's correct? After a PoP change, the PoP's are sometimes not updated in CD or ople. +  * Are the PoP's correct? After a PoP change, the PoP's are sometimes not updated in CD or ople. Compare the PoP's in the configuration tab of CD or OPLE with the PoP Overview or Popperi gtk
-  * Are the carts behaving or are there vibrations or jumps of 100 or more microns every 3-6 seconds? How are the metrology signals? Are they strong and staying white? Red signals mean one or more metrology signals may have gone too low and homing carts is necessary.  The METDATA function on the ople gui can be used to sample the metrology signal and may show noise spikes that can be disruptive to the smooth cart motion. Push the METDATA button on the configure tab and then again about 6 seconds later Plots will pop up to show the frequency and power of any noise. +  * Are the carts behaving or are there vibrations or jumps of 100 or more microns every 3-6 seconds? How are the metrology signals? Are they strong and staying white? Red signals mean one or more metrology signals may have gone too low and homing carts is necessary. The METDATA function on the ople gui can be used to sample the metrology signal and may show noise spikes that can be disruptive to the smooth cart motion. Set a value of 10-12 in the window and push the [MET] button on the configure tab for the cart you want to check. Plots will pop up to show the frequency and power of any noise. 
-  * Is the target a high proper motion star? Red dwarfs are close stars and can have high proper motions. Scan a wider range to see if it is outside of the usual calculated scan range. Binaries can also have very high offsets from the expected position due to mistakenly using astromod calculations from the companion star. +  * Is the target a high proper motion star? Red dwarfs are close stars and can have high proper motions. Scan a wider range to see if it is outside of the usual calculated scan range. Binaries can also have very high offsets from the expected position due to mistakenly using astromod calculations from the companion star. Make sure you are on the right star with an image from SIMBAD or Vizier
-  * Do you have enough flux from each telescope or on each baseline?  Is the telescope tiptilt struggling with low flux?  Can the camera gain be raised or the exposure made longer to help hold the star?+  * Do you have enough flux from each telescope or on each baseline? Is the telescope tiptilt struggling with low flux? Can the camera gain be raised or the exposure made longer to help hold the star? Passing clouds or contrails can lower flux unexpectedly. Remap or realign as needed.
   * Did you get the same star in each telescope? Sometimes a busy star field and poor pointing of the telescopes can lead to the wrong star being acquired and locked by tiptilt. View the stars in the finder window to see if all the stars match.   * Did you get the same star in each telescope? Sometimes a busy star field and poor pointing of the telescopes can lead to the wrong star being acquired and locked by tiptilt. View the stars in the finder window to see if all the stars match.
   * Check the CHARA time on the GPS server. The "Ext-CHARA," "CHARA-Sys," and "Ext-Sys" time offsets listed on the GPS server should be small (< 0.01 sec). If there are large time offsets, then a GSYNC might be needed.   * Check the CHARA time on the GPS server. The "Ext-CHARA," "CHARA-Sys," and "Ext-Sys" time offsets listed on the GPS server should be small (< 0.01 sec). If there are large time offsets, then a GSYNC might be needed.
   * Check the time on the OPLE server. If the time is off or there are any lost ticks/seconds "Lost T/S" listed, then MSYNC ople using the GPS GUI and type "syncople" into the ople server.   * Check the time on the OPLE server. If the time is off or there are any lost ticks/seconds "Lost T/S" listed, then MSYNC ople using the GPS GUI and type "syncople" into the ople server.
-  * Are the [MAN] buttons pressed (gray) for the moving carts on the OPLE Control gui? (The reference cart will remain green.) +  * Are the [MAN] or [FT] buttons pressed (gray) for the moving carts on the OPLE Control gui? (The reference cart will remain green.) 
-  * Check that the carts are within delay line range (-1 to 44 meters) and errors are small. Were the carts homed and checked before the first slew of the night? Did any carts go to the front switch after slewing? This can cause them to lose their position. Turn off the OL button to send it back to zero. If it does not track on the home switch when displaying a zero position (0.000000), it was lost. Is the E2 cart stuck at 27.97m? See section on homing carts in __OPLE and Metrology__  section below. +  * Check that the carts are within delay line range (-1.3 to 44.25 meters) and errors are small. Were the carts homed and checked before the first slew of the night? Did any carts go to the front switch after slewing? This can cause them to lose their position. To check the homing of the carts, turn off the OL button to send it back to zero. If it does not track on the home switch when displaying a zero position (0.000000), it was lost. Use the [CHECK] button on the OPLE gtk to send the carts to the home switch and report whether there is an error in the cart position. 
-  * Are the LDC's working? +  * See section on homing carts in __OPLE and Metrology__  section below. 
-      * Is the glass position within allowable range (-10mm to 49mm) on all beams?+  * For the visible programs, are the LDC's working? 
 +      * Is the glass position displayed on the LDC gui within allowable range (to 40) on all beams?
       * Have the commands "useldc on" and "autoldc on" been typed into the ople server?       * Have the commands "useldc on" and "autoldc on" been typed into the ople server?
-      * Look out for "-10" and "49" errors on ople server. If the LDC glass goes out of range, the LDC will need to be homed. To home the glass, stop astromod by clicking [STOP] on the OPLE gui. Make sure the LDC velocities are non-zero (~ 50… must click [VEL] button to update velocity) and click [HOME] on the LDCs in use. The LDC position number should go to 0.000 or very close to zero. After the LDCs have been homed, turn astromod on by clicking [START] on the ople gui. Then click [REF] on cosmic debris to send the LDCs to the correct positions.+      * Look out for errors on ople server. If the LDC glass goes out of range, the LDC will need to be homed. To home the glass, stop astromod by clicking [STOP] on the OPLE gui. Make sure the LDC velocities are non-zero (~ 50… must click [vel] button to update velocity) and click [HOME] on the LDCs in use. The LDC position number should go to 0.000 or very close to zero. After the LDCs have been homed, turn astromod on by clicking [START] on the ople gui. Then click [REF] on cosmic debris to send the LDCs to the correct positions.
       * If LDCs are working correctly, the LDC positions (in mm) should roughly match up with the reference cart positions (in meters).       * If LDCs are working correctly, the LDC positions (in mm) should roughly match up with the reference cart positions (in meters).
  
   * Check the instrument alignment. Is flux getting through to the detector? How long has it been since the last NIRO camera alignment? Classic and CLIMB programs can run for about an hour before the light will drift from the central pixel. Use the Classic or CLIMB gui to view the light on the pixels by clicking the PICTURE tab and then the PIXEL AREA button. Turn the camera off with the STOP button. Is the right dither power turned on? CLIMB 1 and Classic use different dithers. If Classic or CLIMB fringes are found in a scan, but not when in recording mode, the dither powers are likely not on. Are the camera settings correct for the seeing conditions and flux levels?   * Check the instrument alignment. Is flux getting through to the detector? How long has it been since the last NIRO camera alignment? Classic and CLIMB programs can run for about an hour before the light will drift from the central pixel. Use the Classic or CLIMB gui to view the light on the pixels by clicking the PICTURE tab and then the PIXEL AREA button. Turn the camera off with the STOP button. Is the right dither power turned on? CLIMB 1 and Classic use different dithers. If Classic or CLIMB fringes are found in a scan, but not when in recording mode, the dither powers are likely not on. Are the camera settings correct for the seeing conditions and flux levels?
 +  * There is a script that will display offsets for all scopes against Hour Angle to help find offsets when using MIRCX and MYSTIC. You'll need to log into the MIRCX spooler computer with the command ssh spooler@mircx, then use the command fringe_predictor. A window will pop up displaying the last five nights of offsets vs. HA.
  
 +===== Can't find the star =====
  
-===== Restarting Servers =====+Sometimes the star does not appear on the ACQ camera and there are many reasons why this can happen. Here are some of those reasons.
  
-==== Restarting Servers using the bootlaunch paradigm ====+  * Did the telescope actually slew? If the powers are off or the scope is disabled, it will not slew. 
 +  * Did the telescope disable after slewing? Check the dome gui to see if it says it has disabled. Enable to get it going again. 
 +  * Are the M1 and M3 covers open? Check with the Telescope monitor or spycam 2 to see if the covers opened. 
 +  * Did the dome rotate with the scope? Check with Spycam 1 to see if it is blocking the telescope. Use Autodome On to get it in line with the telescope. 
 +  * Is the ACQ camera showing a live view. Hit ACQ or reopen to see a live view. If the server has quit, bootlaunch on the appropriate computer to restart the ACQ server. Sockman will tell which computer the server runs on. Should be wfs computer at each scope. 
 +  * Is the ACQ exposure appropriate for the star magnitude? Turn it up as needed to show the star. 
 +  * Has the default flat or last flat been loaded after a slew? A bad DM will distort dim stars and they will not be visible. 
 +  * Is the star too low or obstructed by a tree or dome? W1 and S2 have trees nearby that can obscure the star below 40 degrees. Refer to the horizon maps to see if they are the cause. 
 +  * E1 does not track past 218 degrees azimuth. The finder camera may be needed to find the star again. It will be off in azimuth, but should still show in the finder. 
 +  * If a telescope is lost and star does not appear in the finder either, such as after a dome server crash and restart, a bright star, the moon or Jupiter can be used to get the scope pointing correctly. Anything brighter than Vmag 1 will show on the Spycam 1 view and can be used to get the scope close enough to find it in the finder, then ACQ. It should be placed just below the top bar of the telescope, centered left to right using the telescope paddle. You may have to pan up and down to see it move through the finder view. Use the obsgtk ADJUST tab scope paddle and set a step of 3000 and go up and down after centering it in the SPY 1 view below the top telescope bar. When you get it centered in the finder, manually init the scope to the AZ and EL on the MANUAL tab of the dome gui to one decimal place. Use Stellarium online or any resource to get the current position. You should then be able to slew to a star and center it in the finder, then ACQ and init on the obsgtk to get the scope back to usual.
  
-If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list". 
  
-To restart a server, log on to the machine that runs the server and type "bootlaunch_master". This script will go through the list of executables and will check which servers are running. If a server isn't running it bootlaunch_master will remove it from socket manager, clear the lock file, and relaunch the server. The instructions below describe how to restart individual servers, but this should not be necessary anymore. \\  \\ A number of servers use an interim bootlaunch paradigm to restart. This is confined to servers that run on ubuntu machines, namely the telescope bunker computers and gps. The basic syntax is "bootlaunch_<server>" where "<server>" is replaced by the server the script is designed to address. The scripts have a number of safeties built in, so it is safe to run them even if a server is already running – they just output the process ID of the running server. The scripts also take care of the entry in socket manager as well any serial port lock files. All the pertinent information is world writeable, so one should be able to run a bootlaunch script as observe. \\  \\ One thing of note about the output of the bootlaunch scripts, they call a number of other programs which themselves have output that may be misleading in the context of bootlaunch. Chief among these is the output of "tsockman". If a server stopped unexpectedly, it may leave behind an entry in the socket manager. In order to launch a new server, one needs to clean out the socket manager entry if it is there. To do that, "tsockman remove <entry>" is called to remove "<entry>" before the new server is launched. If there is no entry, tsockman will respond with "Process by that name does not exist". THIS IS NORMAL and is not indicative of an error. The server in question launched (without fanfare) right after that output text. \\  \\ Here are the available bootlaunch scripts as of June 2017: \\  \\ gps computer:+===== The new OPLE system =====
  
-  * bootlaunch_beamsamp – Starts the beam sampler serversBS1 and BS2. +With the implementation of the new ople system which replaced the VME in Fall of 2021new troubleshooting issues have arisen. Since there are now 6 new ople computers to run the carts for each scope individually, many issues will be specific to one computer or telescope and not to the system as a wholeRestarting the VME is now a thing of the past.
-  * bootlaunch_zaber – Starts the ZABER_2 server.+
  
- \\ telescope bunker computers:+The traditional ople server will still be used to communicate with each new ople computer, identified as OPLE 1 to OPLE 6. When the communications are good, each active cart will be displayed in the ople server or ople gui status tab as before. At times, an ople computer can lose communications or a server can crash and the server or comms needs to be restarted.
  
-  * bootlaunch_hut – Starts the E1_HUT, E2_HUT, S1_HUT, S2_HUT, W1_HUT, or W2_HUT serverdepending on the machine it's launched from. +If a cart cannot be startedstopped or otherwise commandedlook to the OPLESystem gui to see if a green indicator has turned red. A message will also often pop up on the ople gui saying the command could not be sentThis may require a simple start command to restart the server or a reboot of the computer to get it back to yellow and then a start to get it back to greenDo either of these steps with a right click of the red button and then select start or reboot. After a rebootselect start to load the servers after the indicator has turned yellowWhen this is done, the ople server will need to be connected to the newly restarted serverType "oo" into the ople server to open ople comms with the new ople server. It should now reappear on the ople server display and say System Ready to indicate comms are restored.
-  * bootlaunch_rpc – Starts the RPC_E1, RPC_E2, RPC_S1, RPC_S2, RPC_W1, or RPC_W2 server, depending on the machine it's launched from. +
-  * bootlaunch_weather – Starts the E1_WEATHER, E2_WEATHER, S1_WEATHER, S2_WEATHER, W1_WEATHER, or W2_WEATHER serverdepending on the machine it's launched from. +
-  * bootlaunch_lower – Starts the E1_Lower, E2_Lower, S1_Lower, S2_Lower, W1_Lower, or W2_Lower cylinder server, depending on the machine it's launched from. +
-  * bootlaunch_upper – Starts the E1_Upper, E2_Upper, S1_Upper, S2_Upper, W1_Upper, or W2_Upper server, depending on the machine it's launched from.+
  
- \\ Note: The bootlaunch scripts will not start new server if there is an existing process running. Therefore, type "ps aux | grep //server_name//" where server_name is the name of the server. If there is a dead process running, look up the process identification number (PID) and type "kill -9 PID" to kill the process and then run the relevant bootlaunch script. +Some times cart is stopped and cannot be commanded. If the cart has gone to the front hard or back hard switchit will not be usable until it is moved from the switch and the Ople Controller box is resetThere are 6 silver boxes for these controllers with two green LED'for the front and back switches and two red LED'for the back hard and front hard switchesIf a red LED is litthere will be an error displayed on the message window and the cart is disabled. The cart will need to be moved off the switch and then the box can be reset with the RESET button on the front. The error display will go away and the red LED will be offThe cart is now controllable.
- +
-==== Restarting Servers using the rc.local file ==== +
- +
- \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm. \\  \\ If a server is not running or Socket Manager reports that a server is deadthen look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence. \\ +
- \\ +
-Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn't work then type "ssh //name//" where name is the computer name. \\  \\ Find out if the server is running by typing "ps aux | grep //server_name//" where server_name is the name of the server. \\ [ctrscrut:599] ps aux | grep pico_1 \\ observe 9281 0.0 0.0 61156 692 pts/3 S+ 13:58 0:00 grep pico_1 \\ observe 12578 0.0 0.0 24524 11212 ? S Jun16 33:14 /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg \\  \\ If the entry for the dead server shows up in the process list, then identify the process identification number (12578 for the example above) and kill the server by typing "kill -9 //PID//" where PID is the process identification number\\  \\ Look up the commands to restart the server by typing "more /etc/rc.local" (this is relevant for servers that run in the background). Press the space bar to scroll through the contents of the rc.local file. Locate the commands relevant for the server that needs to be restarted and copy and paste into a terminal window: \\  \\ #Start PICO server for PICO #1 \\ /bin/rm -f /var/lock/LCK..ttyC8 \\ /usr/local/bin/tsockman remove PICO_1 \\ /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg & \\  \\ The first command removes the lock to allow the server to restartThe second command removes the name from the socket manager listing. The last command restarts the server. Note that if you are restarting the servers as observeyou will need to remove the part of the command in the rc.local file that saves information in /var/log///server_nam//e.log file (the actual command typed should resemble the last line above). \\  \\ There are text files on the desktop with many of the restart commands. Use these files for quick access to the relevant commands. The commands are edited and can be copied exactly as written. Files include Dome servers and all servers running on ctrscrut. Many of these commands are also located on the [[:chara:restarting_servers|Restarting Servers]] page. +
- +
-==== Shutters Server ==== +
- +
- \\ The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on ople, not ctrscrut. \\  \\ To start the shutter server on ople: \\  \\ Log into the ople computer and kill the process labeled shutters with the PID as described in **Restarting Servers**  above. \\ Turn off the power to the Shutters with the switch on the computer rack which is to the left of the computer desk and marked "SHUTTERS". Restart the Shutters server with the commands below. After restarting the server and testing the gui to see that it works, turn the SHUTTERS power back on with the switch. There is a printed sheet of directions in the lab to help you. \\  \\ /usr/local/bin/tsockman rm shutters \\ ctrscrut/usr/local/bin/shutter_server /ctrscrut/chara/etc/shutter.cfg & +
- +
-==== MIRC-X CredoneImAcq Server ==== +
- +
-The MIRC-X CredoneImAcq server and GUI can be opened using the "credone" icon on the wolverine desktop. Note that you can only have one of these open at a time. If you get an error message and can't bring up credone, then close the error message and follow the steps below to remove the dead processes first: +
- +
-From a terminal window, log on to mircx (old observe password): \\ ssh -X spooler@mircx +
- +
-Look up credone process: \\ ps aux | grep credone +
- +
-Kill all credone processes (including the ones for the error messages) using "kill -9 PID" where PID is the process number. Note that credone can'be restarted if there are any error processes in the listCheck "ps aux | grep credone" one more time to make sure all processes are cleared (the only one that should show up is the grep command that was just issued). +
- +
-Then restart credone using the desktop icon or by issuing the following command on spooler@mircx: \\ credoneImAcq –no-display +
- +
-(the first "–" should be two dashes: - and -).+
  
 ===== Restarting Servers ===== ===== Restarting Servers =====
Line 79: Line 61:
 If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list". If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list".
  
-To restart a server, log on to the machine that runs the server and type "bootlaunch_master". This script will go through the list of executables and will check which servers are running. If a server isn't running it bootlaunch_master will remove it from socket manager, clear the lock file, and relaunch the server. The instructions below describe how to restart individual servers, but this should not be necessary anymore.\\ +To restart a server, log on to the machine that runs the server and type "bootlaunch_master". This script will go through the list of executables and will check which servers are running. If a server isn't running it bootlaunch_master will remove it from socket manager, clear the lock file, and relaunch the server. The instructions below describe how to restart individual servers, but this should not be necessary anymore. \\  \\ A number of servers use an interim bootlaunch paradigm to restart. This is confined to servers that run on ubuntu machines, namely the telescope bunker computers and gps. The basic syntax is "bootlaunch_<server>" where "<server>" is replaced by the server the script is designed to address. The scripts have a number of safeties built in, so it is safe to run them even if a server is already running – they just output the process ID of the running server. The scripts also take care of the entry in socket manager as well any serial port lock files. All the pertinent information is world writeable, so one should be able to run a bootlaunch script as observe. \\  \\ One thing of note about the output of the bootlaunch scripts, they call a number of other programs which themselves have output that may be misleading in the context of bootlaunch. Chief among these is the output of "tsockman". If a server stopped unexpectedly, it may leave behind an entry in the socket manager. In order to launch a new server, one needs to clean out the socket manager entry if it is there. To do that, "tsockman remove <entry>" is called to remove "<entry>" before the new server is launched. If there is no entry, tsockman will respond with "Process by that name does not exist". THIS IS NORMAL and is not indicative of an error. The server in question launched (without fanfare) right after that output text. \\  \\ Note: The bootlaunch scripts will not start a new server if there is an existing process running. Therefore, type "ps aux | grep //server_name//" where server_name is the name of the server. If there is a dead process, look up the process identification number (PID) and type "kill -9 PID" to kill the process and then run the relevant bootlaunch script.
-\\ +
-A number of servers use an interim bootlaunch paradigm to restart. This is confined to servers that run on ubuntu machines, namely the telescope bunker computers and gps. The basic syntax is "bootlaunch_<server>" where "<server>" is replaced by the server the script is designed to address. The scripts have a number of safeties built in, so it is safe to run them even if a server is already running – they just output the process ID of the running server. The scripts also take care of the entry in socket manager as well any serial port lock files. All the pertinent information is world writeable, so one should be able to run a bootlaunch script as observe.\\ +
-\\ +
-One thing of note about the output of the bootlaunch scripts, they call a number of other programs which themselves have output that may be misleading in the context of bootlaunch. Chief among these is the output of "tsockman". If a server stopped unexpectedly, it may leave behind an entry in the socket manager. In order to launch a new server, one needs to clean out the socket manager entry if it is there. To do that, "tsockman remove <entry>" is called to remove "<entry>" before the new server is launched. If there is no entry, tsockman will respond with "Process by that name does not exist". THIS IS NORMAL and is not indicative of an error. The server in question launched (without fanfare) right after that output text.\\ +
-\\ +
-Here are the available bootlaunch scripts as of June 2017:\\ +
-\\ +
-gps computer:+
  
-  * bootlaunch_beamsamp – Starts the beam sampler servers, BS1 and BS2. +==== Restarting Servers using the rc.local file (likely outdated) ====
-  * bootlaunch_zaber – Starts the ZABER_2 server.+
  
- \\ telescope bunker computers:+ \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm, which may not be any at this point. \\  \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence. \\ 
 + \\ 
 +Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn't work then type "ssh //name//" where name is the computer name. \\  \\ Find out if the server is running by typing "ps aux | grep //server_name//" where server_name is the name of the server. \\ [ctrscrut:599] ps aux | grep pico_1 \\ observe 9281 0.0 0.0 61156 692 pts/3 S+ 13:58 0:00 grep pico_1 \\ observe 12578 0.0 0.0 24524 11212 ? S Jun16 33:14 /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg \\  \\ If the entry for the dead server shows up in the process list, then identify the process identification number (12578 for the example above) and kill the server by typing "kill -9 //PID//" where PID is the process identification number. \\  \\ Look up the commands to restart the server by typing "more /etc/rc.local" (this is relevant for servers that run in the background). Press the space bar to scroll through the contents of the rc.local file. Locate the commands relevant for the server that needs to be restarted and copy and paste into a terminal window: \\  \\ #Start PICO server for PICO #1 \\ /bin/rm -f /var/lock/LCK..ttyC8 \\ /usr/local/bin/tsockman remove PICO_1 \\ /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg & \\  \\ The first command removes the lock to allow the server to restart. The second command removes the name from the socket manager listing. The last command restarts the server. Note that if you are restarting the servers as observe, you will need to remove the part of the command in the rc.local file that saves information in /var/log///server_nam//e.log file (the actual command typed should resemble the last line above). \\  \\ There are text files on the desktop with many of the restart commands. Use these files for quick access to the relevant commands. The commands are edited and can be copied exactly as written. Files include Dome servers and all servers running on ctrscrut. Many of these commands are also located on the [[:chara:restarting_servers|Restarting Servers]] page.
  
-  * bootlaunch_hut – Starts the E1_HUT, E2_HUT, S1_HUT, S2_HUT, W1_HUT, or W2_HUT server, depending on the machine it's launched from. +Shutters Server
-  * bootlaunch_rpc – Starts the RPC_E1, RPC_E2, RPC_S1, RPC_S2, RPC_W1, or RPC_W2 server, depending on the machine it's launched from. +
-  * bootlaunch_weather – Starts the E1_WEATHER, E2_WEATHER, S1_WEATHER, S2_WEATHER, W1_WEATHER, or W2_WEATHER server, depending on the machine it's launched from. +
-  * bootlaunch_lower – Starts the E1_Lower, E2_Lower, S1_Lower, S2_Lower, W1_Lower, or W2_Lower cylinder server, depending on the machine it's launched from. +
-  * bootlaunch_upper – Starts the E1_Upper, E2_Upper, S1_Upper, S2_Upper, W1_Upper, or W2_Upper server, depending on the machine it's launched from.+
  
- \\ Note: The bootlaunch scripts will not start a new server if there is an existing process runningTherefore, type "ps aux | grep //server_name//where server_name is the name of the server. If there is a dead process running, look up the process identification number (PID) and type "kill -9 PID" to kill the process and then run the relevant bootlaunch script.+ \\ The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on gps, not ctrscrut. \\  \\ To start the shutter server on gps: \\  \\ Log into the gps computer and kill the process labeled shutters with the PID as described in **Restarting Servers**  above\\ Turn off the power to the Shutters with the switch on the computer rack which is marked "SHUTTERS". Restart the Shutters server with the commands below. After restarting the server and testing the gui to see that it works, turn the SHUTTERS power back on with the switchThere is a printed sheet of directions in the lab to help you. It will refer to the ople computer, but it now runs on the GPS computer\\  \\ /usr/local/bin/tsockman rm shutters \\ bootlaunch_master
  
-==== Restarting Servers using the rc.local file ====+==== Restarting Socket Manager ====
  
- \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm. \\  \\ If a server is not running or Socket Manager reports that a server is deadthen look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence\\ +In very rare circumstances, the socket manager might need to be restartedThis is usually needed if GUIs stop communicating (e.g., OPLE GUI stops communicating with OPLE server). Another example occurs when mircx_bootLauncher tries to remove dead server names from the socket_manager before restartingIf the socket_manager does not respond, the mircx_bootLauncher will timeout. 
- \\ + 
-Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn't work then type "ssh //name//" where name is the computer name. \\  \\ Find out if the server is running by typing "ps aux | grep //server_name//" where server_name is the name of the server\\ [ctrscrut:599] ps aux | grep pico_1 \\ observe 9281 0.0 0.0 61156 692 pts/3 S 13:58 0:00 grep pico_1 \\ observe 12578 0.0 0.0 24524 11212 ? S Jun16 33:14 /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg \\  \\ If the entry for the dead server shows up in the process listthen identify the process identification number (12578 for the example above) and kill the server by typing "kill -9 //PID//" where PID is the process identification number. \\  \\ Look up the commands to restart the server by typing "more /etc/rc.local" (this is relevant for servers that run in the background). Press the space bar to scroll through the contents of the rc.local file. Locate the commands relevant for the server that needs to be restarted and copy and paste into a terminal window: \\  \\ #Start PICO server for PICO #1 \\ /bin/rm -f /var/lock/LCK..ttyC8 \\ /usr/local/bin/tsockman remove PICO_1 \\ /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg & \\  \\ The first command removes the lock to allow the server to restart. The second command removes the name from the socket manager listingThe last command restarts the server. Note that if you are restarting the servers as observe, you will need to remove the part of the command in the rc.local file that saves information in /var/log///server_nam//e.log file (the actual command typed should resemble the last line above). \\  \\ There are text files on the desktop with many of the restart commands. Use these files for quick access to the relevant commands. The commands are edited and can be copied exactly as written. Files include Dome servers and all servers running on ctrscrut. Many of these commands are also located on the [[:chara:restarting_servers|Restarting Servers]] page.+Follow these instructions to restart the socket manager, but only do so if really necessary:
  
-==== Shutters Server ====+ssh -X observe@ctrscrut
  
- \\ The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on ople, not ctrscrut. \\  \\ To start the shutter server on ople: \\  \\ Log into the ople computer and kill the process labeled shutters with the PID as described in **Restarting Servers**  above. \\ Turn off the power to the Shutters with the switch on the computer rack which is to the left of the computer desk and marked "SHUTTERS". Restart the Shutters server with the commands below. After restarting the server and testing the gui to see that it works, turn the SHUTTERS power back on with the switch. There is a printed sheet of directions in the lab to help you. \\  \\ /usr/local/bin/tsockman rm shutters \\ ctrscrut/usr/local/bin/shutter_server /ctrscrut/chara/etc/shutter.cfg &+killall socket_manager 
 + 
 +socket_manager &
  
 ==== MIRC-X CredoneImAcq Server ==== ==== MIRC-X CredoneImAcq Server ====
Line 127: Line 100:
  
 ===== Telescopes and Dome Servers ===== ===== Telescopes and Dome Servers =====
- 
- \\ Here we discuss things that can go wrong with the telescopes. 
  
 ==== The Telescope won't move or stopped moving ==== ==== The Telescope won't move or stopped moving ====
-<code> 
  
-\\  Have the powers been turned on to the drives? Are the scopes disabled? The usual state of the telescopes is disabled until enabled. This is due to the stall function of the scopes which eventually disables the scopes when they are stowed. Enable the scopes by hitting [ENABLE] on the dome gui, the telescope gui control tab, or the bottom of the obsgtk. If a scope disables itself during a slew, it may be just an overcautious stall function or the slew gains may be too high or low. Check the gains and if they are OK, Hit ENABLE and the scope should continue to slew. If it disables again without moving, there may be something wrong at the scope, ie. a hatch left open, a ladder not put away, a tool on the floor, or something else physically impeding the motion of the scope. You will need to go to the dome to see what it is. The computer in the dome will give you control of the scope to turn it away from the problem.  \\    \\  Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopes. Check them for current times and continuous updates of numbers. If they are not updating, try to REOPEN them first. If that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of it. See **Dome Server Restart** below.  \\    \\  **Azimuth Limit Switches**   \\    \\  <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>As of 11-'17, the azimuth limit switches are enabled and can stop the motion of the scopes if they try to go beyond -90º or  450º. The scopes will not be movable with normal inputs so follow these instructions to return them from the out of range condition.  \\    \\  1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software.  \\    \\  2. Make sure you understand why the limit was hit which may require a trip to the telescope.  If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch causing the stall and not that the scope is actually in a wrong position. \\    \\  3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope \\    \\  4. Click ENABLE then you can move the telescope back to its normal range of operation.  \\    \\  5. After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again and then hit AUTO on the AUTO tab to resume normal operation</font>.+Have the powers been turned on to the drives? Are the scopes disabled? The usual state of the telescopes is disabled until enabled. This is due to the stall function of the scopes which eventually disables the scopes when they are stowed. Enable the scopes by hitting [ENABLE] on the dome gui, the telescope gui control tab, or the bottom of the obsgtk. If a scope disables itself during a slew, it may be just an overcautious stall function or the slew gains may be too high or low. Check the gains and if they are OK, Hit ENABLE and the scope should continue to slew. If it disables again without moving, there may be something wrong at the scope, ie. a hatch left open, a ladder not put away, a tool on the floor, or something else physically impeding the motion of the scope. You will need to go to the dome to see what it is. The computer in the dome will give you control of the scope to turn it away from the problem. Make sure the scope is in a position that allows the opening of the hatch if you need to go upstairsIt can be moved back from the bunker as needed.
  
-</code>+Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopes. Check them for current times and continuous updates of numbers. If they are not updating, try to REOPEN them first. If that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of it. See **Dome Server Restart**  below. \\  \\ **Azimuth Limit Switches** \\  \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>As of 11-'17, the azimuth limit switches are enabled and can stop the motion of the scopes if they try to go beyond -90º or 450º. The scopes will not be movable with normal inputs so follow these instructions to return them from the out of range condition. \\ 
 +\\ 
 +1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software. \\ 
 +\\ 
 +2. Make sure you understand why the limit was hit which may require a trip to the telescope. If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch incorrectly causing the stall and not that the scope is actually in a wrong position.</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way. \\  \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope. \\  \\ 
 +4. Click ENABLE then you can move the telescope back to its normal range of operation. \\  \\ 
 +5. After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again and then hit AUTO on the AUTO tab to resume normal operation.</font>
  
-<font 16px/Calibri,sans-serif;;#333333;;white>**However, If you notice that a telescope will stop near AZ 90 or 270 with the scope still being ENABLED and refusing to move, this is often due to the new AZ limit switch being tripped at some point earlier. To get it moving again:**</font>+<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>However, If you notice that a telescope will stop near AZ 90 or 270 with the scope still being ENABLED and refusing to move, this is often due to the new AZ limit switch being armed around AZ 0º at some point earlier. To get it moving again:</font>
  
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;white>1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software.</font>      <font inherit/inherit;;initial;;white>2. M</font>ake sure the all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way.    <font inherit/inherit;;initial;;white>3. Click the OVERRIDE ON button in domegui MANUAL tab.</font>+<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software. \\  \\ 
 +2.</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way. \\  \\ <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope.</font>
  
 <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>4. Move the scope a bit back toward the direction it was coming from – for example, if the scope stopped at AZ 268 while rotating clockwise, move it back to 265 or so using AZ DEC. Then press STOP.</font> <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>4. Move the scope a bit back toward the direction it was coming from – for example, if the scope stopped at AZ 268 while rotating clockwise, move it back to 265 or so using AZ DEC. Then press STOP.</font>
  
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>5. Move the scope past AZ 270 by pressing AC INC. Usally 2-4 degrees will do.</font>+<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>5. Move the scope past AZ 270 by pressing AZ INC. Usally 2-4 degrees beyond will do. Go past the point that shows the limit is triggered and click STOP.</font>
  
 <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>6. Click OVERRIDE OFF, go to the AUTO tab and press AUTO, then NEXT (also in the obsgtk; this will restore the original star's demand position to the scope).</font> <font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>6. Click OVERRIDE OFF, go to the AUTO tab and press AUTO, then NEXT (also in the obsgtk; this will restore the original star's demand position to the scope).</font>
  
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7. When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90.</font>+<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7. When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90. The LED will show red when on the limit switch and is tripped, ie. limiting motion of the scope. The LED will be yellow if it has tripped and is in the caution range, but not on a limit switch. A fine Allen key can be used to push the internal reset button or use the magnet and touch the box. It will turn the LED green when restored.</font> 
 + 
 +==== The Telescope won't track ==== 
 + 
 +If a bright star is manually found and not slewed to by number, (usually after the scope has gotten lost), the telescope will not track as it does not know it has gone to a star. When you init the EL and AZ position of the scope to the star's position, you must enable the tracking by going to the dome gui, selecting the RA/DEC tab and hitting the [GO RA/DEC] button. The scope will track, but the pointing model may still be off. Try slewing to the same star and initing on it after it is locked in Tiptilt. This can be done in the dome with the computer at the telescope.
  
 ==== The Telescope won't track ==== ==== The Telescope won't track ====
  
-  
 If a bright star is manually found and not slewed to by number, (usually after the scope has gotten lost), the telescope will not track as it does not know it has gone to a star. When you init the EL and AZ position of the scope to the star's position, you must enable the tracking by going to the dome gui, selecting the RA/DEC tab and hitting the [GO RA/DEC] button. The scope will track, but the pointing model may still be off. Try slewing to the same star and initing on it after it is locked in Tiptilt. This can be done in the dome with the computer at the telescope. If a bright star is manually found and not slewed to by number, (usually after the scope has gotten lost), the telescope will not track as it does not know it has gone to a star. When you init the EL and AZ position of the scope to the star's position, you must enable the tracking by going to the dome gui, selecting the RA/DEC tab and hitting the [GO RA/DEC] button. The scope will track, but the pointing model may still be off. Try slewing to the same star and initing on it after it is locked in Tiptilt. This can be done in the dome with the computer at the telescope.
  
 ==== The Telescope won't stop moving or moves beyond the commanded position ==== ==== The Telescope won't stop moving or moves beyond the commanded position ====
  
-  
 Disable the scope whenever its motion is outside of what it is supposed to do. Turn the powers off to the axes as well on the Power gui. The dome server is usually to blame and will need to be restarted. Disable the scope whenever its motion is outside of what it is supposed to do. Turn the powers off to the axes as well on the Power gui. The dome server is usually to blame and will need to be restarted.
 +
 +Sometimes, the gains can be too low and the scope will overshoot the star position. Disable the scope, adjust the Slewing gain up one increment and enable the scope. It should return to the star.
  
 ==== Dome Server Restart ==== ==== Dome Server Restart ====
  
-  +Dome servers are now started using the bootlaunch_master commandThe manual process that has been superceded is archived.
-To manually start the dome server:    +
-1Make sure the power to the drives is OFF.  +
-2. Login to the relevant computer as root. For example, type "s1" or "ssh s1" to log on to S1.  +
-3. Work out the process ID number (PID), either with the command    +
-(s1:1001) tsockman get dome_S1  +
-Name : dome_S1  +
-Machine : s1.chara-array.org  +
-PID : 29953  +
-Commands : -1  +
-Data : -1  +
-Message : 4002  +
-Restart : /usr/local/bin/dome_server -A33.7441 S1    +
-or with  +
-(s1:1003) ps aux | grep dome  +
-theo 4473 0.0 0.0 61188 748 pts/3 S 10:45 0:00 grep dome  +
-observe 29953 18.5 0.4 35596 9860 ? Sl Apr21 416:11 /usr/local/bin/dome_server -A33.7441 S    +
-It can also be found by pulling up the LIST on SOCKMAN and selecting the relevant dome.    +
-So in this case the PID is 29953.    +
-4. Try and stop the server gracefully: kill -2 29953  +
-5. You should then check that the server has indeed stopped:    +
-[s1:600] tsockman get dome_S1  +
-Name : dome_S1  +
-Machine : s1.chara-array.org  +
-PID : 15635  +
-Commands : -1  +
-Data : -1  +
-Message : 2008  +
-Restart : /usr/local/bin/dome_server -A41.0166 S1    +
-If the socket manager still thinks it's running you will need to stop it forcefully: kill -9 29953; tsockman rm dome_S1    +
-6. Restart the dome server by copying the command at the end of the /etc/rc.local file:    +
-[s1:602] more /etc/rc.local  +
-<<< Press space bar or enter to scroll through file>>>    +
-#Run the dome server    +
-/usr/local/bin/tsockman remove dome_S1  +
-/usr/local/bin/dome_server -A41.0166 S1 &    +
-(Note: the part of the command that saves information to /var/log/dome_S1.log has been removed.)    +
-7. Turn the power to the drives back on.  +
-8. Hit REOPEN and ENABLE on the domegtk, and type "otcs" in the telescope server.    +
-You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scopes position.+
  
-==== Telescope clock is not correct ====+To restart the dome server:
  
-  +1. Make sure the power to the drives is OFF. Disable the scopes.
-The NTP server has obsoleted this problem and fix as of 2014.+
  
-==== Telescope is not receiving the commanded position for a target ====+2. Login to the relevant computer as observe. For example, type "s1" to log on to S1.
  
-  +3Work out the process ID number (PID) by typing bootlaunch_master
-Sometimes it happens that a telescope receives the wrong position for a target or does not receive the commanded position at allThe commanded position is listed on the telescope server in the first column under TCS Az/El; the second column lists the actual position of the telescope. Try entering the star designation directly into the telescope server, ie. hd 123456. If it does not accept the number, try closing and restarting the telescope server and hitting repoen on Cosmic Debris and the telescope gui. Try entering the star into the server again. If that does not work, it is possible that something is wrong with the dome server. To restart the dome server follow these steps:  +
-* DISABLE the telescope using either the telescope or dome GUI.+
  
-  * Turn off the power for the telescope (both AZ/EL). +4Use kill -2 PID to kill the server. Entering the command twice will show "No such processif it has been killed. if it is not killed, use kill -9.
-  * First shutdown the telescope server +
-  * Then use SOCKMAN to select the appropriate dome server from the list (dome_E1, etc) Get the PID number for reference. +
-  * The dome server will need to be restarted from the command line in a terminal. +
-  * Open a terminal window and log on to the telescope computer by typing "s1", "s2", "e1", "e2", "w1", or "w2" +
-  * Look for instruction in the /etc/rc.local file by typing "more /etc/rc.local". Scroll through the file by hitting space bar or Enter. +
-  * Locate instruction listed under "#Run the dome server". Copy and paste the commands at the prompt directly on the telescope computer: +
-      * /usr/local/bin/tsockman rm dome_S1 +
-      * /usr/local/bin/dome_server -A33.7441 S1 >& /var/log/dome_S1.log +
-  * (Replace S1 with commands for appropriate telescope) +
-  * After the dome server is running, re-open the telescope server and click [REOPEN} on the dome and telescope GUIs and on Cosmic Debris. +
-  * ENABLE the telescope using the telescope or dome GUI. Check to see if the command and telescope positions are correct. If they are, then turn on the power for the telescope drives. Re-enter the star information by entering the HD number in the telescope server and click [GO NEXT] to send the telescope to the star. Make sure that the telescope is behaving as expected. The telescope might have drifted a bitso if you can't find the star, it might be necessary to use the Telrad on a bright star to re-initialize the pointing. +
-  * Restart commands for the dome servers are also listed on the desktop in a text file. +
-<code>+
  
-   \ +5. Type bootlaunch_master again to restart the server.
- ==== Telescope is tracking poorly, overshooting in slew, oscillating====+
  
-</code>+6. Turn the power to the drives back on.
  
-<code> +7Hit REOPEN and ENABLE on the domegtk, and type "otcsin the telescope server.
-This might mean that the gain for the tracking servo is wrongNote that changing this gain can be dangerous, especially if you set it too high as that can cause the telescope to oscillated and damage the drives. Please only do this if you are very very sure that it is necessary. Symptoms of bad gain are:    The scope over shoots the position while slewing. The star will be seen to move out of the window and may come back after a few seconds. This means the slewing gain is too low.  The scope oscillates when tracking or after a slew. The star will be tracing an ellipsefigure eight or other looping shape. This means the tracking gain is too low. You can damp this out with the telescope or dome gui by disabling the scope, then re-enabling it. Adjust the gain upward and watch it on the next slew.    In all cases if either gain is too high the scope will go into "Fog Hornmode, which is bad. You always want to use the lowest gain that still allows the scope to work as best as possibleIf the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high.    On 10-22-2016, the gain settings were:    | Scope  | AZ Slewing  || EL Slewing  || AZ Tracking  || EL Tracking  || Date Updated  |+
  
-</code>+You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scope's position.
  
-| |Gain|Fn|Gain|Fn|Gain|Fn|Gain|Fn| | +Be aware that when restarting a dome server, the telescopes position may not be retained and the dome gui may display Az: 0.0. El: 0.0. The gui will scroll messages that say Az: 0.0. El: 0.0 with an error in scope position. If the scope is known to be at STOW position, you can manually enter the scope's AZ and EL using the manual tab on the dome gui and hitting the INIT button.
-|S1|7|4|4|3|16|4|10|7|10-26-2017| +
-|S2|7| |7| |10| |10| |10-22-2016| +
-|E1|7| |7| |13| |10| |10-22-2016| +
-|E2|7| |7| |13| |10| |10-22-2016| +
-|W1|13| |10| |13| |10| |10-22-2016| +
-|W2|7| |7| |13| |13| |10-22-2016|+
  
- \\ +Also be aware that the gains do not always return to the normal values of Slew 4-7 and Tracking 7-10 when a dome server has crashed and is restarted. They may come up as Gain 22 for all axes and modes and they will need to be turned down or the scopes will oscillate badly. 
-Note that these values may change as the temperatures change rapidly. These are still being adjusted so this is by no means final. Be sure to set mode back to AUTO if changing the gains left it in Slewing or Tracking mode.+ 
 +==== Telescope is not receiving the commanded position for a target. ==== 
 + 
 +Sometimes it happens that a telescope receives the wrong position for a target or does not receive the commanded position at all. The commanded position is listed on the telescope server in the first column under TCS Az/El; the second column lists the actual position of the telescope. Try entering the star designation directly into the telescope server, ie. hd 123456. If it does not accept the number, try closing and restarting the telescope server and hitting repoen on Cosmic Debris and the telescope gui. Try entering the star into the server again. If that does not work, it is possible that something is wrong with the dome server. To restart the dome server follow these steps above. 
 + 
 +==== Telescope is tracking poorly, overshooting in slew, oscillating. ==== 
 + 
 +This might mean that the gain for the tracking servo is wrong. Note that changing this gain can be dangerous, especially if you set it too high as that can cause the telescope to oscillated and damage the drives. Please only do this if you are very very sure that it is necessary. Symptoms of bad gain are: The scope over shoots the position while slewing. The star will be seen to move out of the window and may come back after a few seconds. This means the slewing gain is too low. The scope oscillates when tracking or after a slew. The star will be tracing an ellipse, figure eight or other looping shape. This means the tracking gain is too low. You can damp this out with the telescope or dome gui by disabling the scope, then re-enabling it. Adjust the gain upward and watch it on the next slew. In all cases if either gain is too high the scope will go into "Fog Horn" mode, which is bad. This can be seen during slews on the twfs or labao as vibrating spots, usually in one axis. 
 + 
 +You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high. 
 + 
 +The usual values for slewing gains are 4-7 and tracking gains are 7-10. Note that these values may change as the temperatures change rapidly. Gains are usually higher when cold and lower when warm. Be sure to set mode back to AUTO if changing the gains left it in Slewing or Tracking mode.
  
 ==== How to Adjust CPUMotor Gains ==== ==== How to Adjust CPUMotor Gains ====
Line 277: Line 206:
 ==== E1 Hut and Cooler Communications Recovery ==== ==== E1 Hut and Cooler Communications Recovery ====
  
-{{:chara:files:e1_hut_coms_recovery.pdf|E1 HuT and Cooler Communications Recovery}}  \\  \\ <font 16px/inherit;;inherit;;inherit>**S2 Acquisition Server Restart**</font>+{{:chara:files:e1_hut_coms_recovery.pdf|E1 HuT and Cooler Communications Recovery}} 
 + 
 +==== S2 Acquisition Server Restart ====
  
 S2's acquisition server resides on "S2pi4" computer. Logging on to it (ssh s2pi4) and running bootlaunch will restart it S2's acquisition server resides on "S2pi4" computer. Logging on to it (ssh s2pi4) and running bootlaunch will restart it
  
-====== Dome issue ======+==== Focusing the Telescope ==== 
 + 
 +  - Fully align beacon to labao and TWFS, especially making sure the focus terms between the two agree very well. 
 +  - Open a secondary gui: mergurygtk W2_SECONDARY. 
 +  - Note the current M2 actuator positions and send them to Nic for the log. 
 +  - Turn off the beacons and lock the star on TWFS tiptilt. 
 +  - Turn the servo on in the mercurygtk gui. 
 +  - On the align tab of the wfsgtk there is a button for FOCUS SCOPE. 
 +  - Press the button. 
 +  - When done the focus term of starlight on the TWFS should be zero. 
 +  - Turn off the servo on the mercury_gtk gui. 
 + 
 +A couple of notes: 
 + 
 +  * If large moves of M2 are made, tiptilt tracking may be lost. Pay careful attention so that M2 doesn't get driven away crazily due to lost light. 
 +  * If it takes too long to start the autofocus procedure, the mercury_gtk servo will automatically turn off and nothing will happen. So just make sure the servo is on right before hitting the FOCUS SCOPE button. 
 + 
 +===== Dome issue =====
  
- \\ 
 At times the domes do not rotate, open or close, or otherwise behave. Some problems are simple and others are more complex. At times the domes do not rotate, open or close, or otherwise behave. Some problems are simple and others are more complex.
  
 ==== Dome does not rotate ==== ==== Dome does not rotate ====
  
- \\ +Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope's position. If not, it will insist on being in the wrong place. If it is not at the same AZ as the scope, turn AUTODOME [OFF] on the obsgtk, manually move it [CCW] or [CW] until it is centered on the telescope in spycam 1. If the dome AZ does not read the same as the telescope AZ, enter the scope AZ in the position box of the DOME tab of appropriate dome server and hit the INIT POS button to tell it the correct AZ. \\  \\ 
-Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope's position. If not, it will insist on being in the wrong place. If it is not at the same AZ as the scope, manually move it until it is centered on the telescope in spycam 1. If the dome AZ does not read the same as the telescope AZ, enter the scope AZ in the position box of the DOME tab of appropriate dome server and hit the INIT POS button to tell it at what AZ it is. \\  \\ +If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box instead of computer. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check the controller box just inside the door of the bunker. Sometimes the drive wheel jumps in the track and cannot turn the dome, even when the motor works. This will need to be fixed during the day. A fuse can also blow out in the dome control box and leave the dome stuck.
-If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker.+
  
 ==== Dome does not open ==== ==== Dome does not open ====
  
-\\ +Is the SPYCAM on and updating? An old image may show the dome is closed, but it has actually started to open. If the dome won't open, try hitting the SLIT CLOSE button on the telescope gui first. It may be that there was a computer or server issue and the computer thinks the dome is open already and won't allow it to open again. Hit the SLIT OPEN button to see if it works now. If it still won't open, go to the bunker and look to see if the power switch is on to the dome. It will be up and red if on.
-Is the SPYCAM on and updating?  An old image may show the dome is closed, but it has actually started to open.  If the dome won't open, try hitting the SLIT CLOSE button on the telescope gui first. It may be that there was a computer or server issue and the computer thinks the dome is open already and won't allow it to open again. Hit the SLIT OPEN button to see if it works now. If it still won't open, go to the bunker and look to see if the power switch is on to the dome. It will be up and red if on.\\ +
-Also look to see if the power cable is connected to the computer box. If it's connected up high to the manual dome controls, the cable needs to be brought back to the computer box connector. __Turn the power off first__ to the dome before disconnecting the power cable. Slide it onto the connector at the box and turn the collar to lock it in place. Turn the power back on and see if you can open the dome from the computer in the bunker.+
  
 +Also look to see if the power cable is connected to the computer box. If it's connected up high to the manual dome controls, the cable needs to be brought back to the computer box connector. __Turn the power off first,__  using the light switch with the red LED toggle, to the dome before disconnecting the power cable. Slide it onto the connector at the box and turn the collar to lock it in place. Turn the power back on and see if you can open the dome from the computer in the bunker.
  
 ==== Dome does not close ==== ==== Dome does not close ====
  
- \\ 
 If a dome will not close after observing, it may need to be closed manually at the bunker. Turn the power to the dome off at the switch, disconnect the cable from the computer control box and connect it to the manual control switches above. Turn on the power to the dome after connecting the cable and use the two controls to close the dropout first, then the slit. The dropout must be fully closed and the slit closed over it to be shut properly. Inform the staff that the dome did not function properly so someone can look at it in the morning. If a dome will not close after observing, it may need to be closed manually at the bunker. Turn the power to the dome off at the switch, disconnect the cable from the computer control box and connect it to the manual control switches above. Turn on the power to the dome after connecting the cable and use the two controls to close the dropout first, then the slit. The dropout must be fully closed and the slit closed over it to be shut properly. Inform the staff that the dome did not function properly so someone can look at it in the morning.
  
 +===== HUT servers =====
  
-====== HUT servers ======+The HUT servers control functions such as beacon and dichroic movements, heater and dehumidifier usage, and various AO functions. An observer may find that the obsgtk is no longer controlling the beacon LED's, beacon flat or dichroic alignments. This happens on occasion with E2 and other scopes. The HUT server may be the cause if it has quit or lost connection or the AOB may be at fault. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the alignments can be changed from the gui, then the HUT server is ok. You can use the hut gui to continue observing. If the hut gui gives move error messages, cycle the power on the AOB and open a new hut server to restore the connection to the obsgtk. On the POWER gui, turn off the power to the AOB for the offending telescope and turn it back on. Stop the hut server by logging into the appropriate telescope computer and identifying the PID with the bootlaunch_master command and killing the process with the kill -9 #### command. Start the new server via the bootlaunch_master command. Hit REOPEN on the obsgtk to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well.
  
-\\ +If the server won't restarta reboot of the power supply in the telescope bunker might be necessaryThe power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunkerThe power supply has green readouts of volts and currentAfter turning the power off for 10 seconds and back on, try restarting the server from the computer in the bunker to see if it starts cleanly. If so, then restart the telescope server, reopen the connection to the telescope gui, and hit REOPEN on Cosmic Debris. Part of the HUT server also controls the AO table. If the AOB part of the HUT server doesn't workthen the power supply on the back of the AO table in the telescope dome might need recyclingThis power supply controls the actuators at M2 and the AO table. The power supply box is a 6×9 inch aluminum box on the back of the AO tablebehind the keyboard and monitor. Turn it off with the power button on the bottom edge of the box, wait 5 seconds and turn it back on. The HUT server should now restart cleanly. Restart the telescope server as well to make the connections to the telescope gui. Hit REOPEN in Cosmic Debris if you are observing to make all needed connections.
-The HUT servers control functions such as beacon and dichroic movementsheater and dehumidifier usage, and various AO functionsAn observer may find that the obsgtk is no longer controlling the beacon LED's, beacon flat or dichroic alignmentsThis happens on occasion with E2 and other scopes The HUT server may be the cause if it has quit or lost connection or the AOB may be at fault. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the alignments can be changed from the gui, then the HUT server is ok. You can use the hut gui to continue observing If the hut gui gives move error messagescycle the power on the AOB and open a new hut server to restore the connection to the obsgtk On the POWER guiturn off the power to the AOB for the offending telescope and turn it back on.  Stop the hut server by logging into the appropriate telescope computer and identifying the PID with the bootlaunch_master command and killing the process with the kill -9 #### command.  Start the new server via the bootlaunch_master command.  Hit REOPEN on the obsgtk to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well.+
  
 ===== Tiptilt Server. (These sections apply to the old lab tiptilt system retired in April 2021) ===== ===== Tiptilt Server. (These sections apply to the old lab tiptilt system retired in April 2021) =====
  
-<code> +[[:chara:old_lab_tiptilt_server|Instructions for the old lab tiptilit system are archived here.]]
- \\  The tiptilt server controls the CCD based tiptilt detection system. \\    \\  Before you start the tiptilt server, you must ensure that the power to the cooling system and the CCD iteslf is on. It is extremely important that the cooler be running before you turn on the CCD and is only turned off if you are sure the CCD is NOT running. You can start the server from the X windows menu or with the command xtiptilt. \\    \\  Note that there are background counts and read noise to deal with. Whenever you change the frame rate, please ensure that the bias frame is OK. The server will attempt to load an old bias frame that should work, but if things are not working, try making a new bias frame by ensuring that the detector is in the dark and typing "mkbias" into the tiptilt server. \\    \\  In the tiptilt GUI windows, the white dots represent the starlight while the green dots represent the motion applied to telescope's secondary mirror to keep the starlight centered. When tiptilt is locked the white dots will be brought to the center of the tiptilt window. The green dots should be mostly centered also. W2 and E2 telescopes have a small oscillation that show as back and forth plots of the green dots. +
- +
-</code> +
- +
- +
-==== Tiptilt server complains about the CCD ==== +
- +
-<code> +
- \\  Is the CCD turned on? When the tiptilt server starts up it tries no more than five times to communicate with the CCD. If they all fail, it will give up. If this happens, try cycling the power to the CCD and try again. If this fails, connect to the tiptilt machine and type the command rtccdAPIDemo, which should return with no errors. Try this command a few times, but if it still fails, there is a more serious problem. Turn off the CCD and reboot the tiptilt computer. If it still fails, I am afraid you are in more serious trouble. \\    \\  Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine. \\    \\  Also note that cycling the power on the CCD can cause harm so be sure you need to do it before trying it. Also, it is important to wait for at least 20 seconds after turning off the power before turning it on again. +
- +
-</code> +
- +
-==== Tiptilt doesn't seem to be talking to the telescopes ==== +
- +
-<code> +
- \\  Sometimes the telescope server will not show that TT is running. It will show 0Hz for a signal rate for TT. Running TIPTILT COMM will not get it started while other scopes do show it starting. Close and restart any telescope servers that won't connect after two tries of TIPTILT COMM.   \\  Note: There is more info in the software manual on this topic, but I wasn't sure if it was still relevant. +
- +
-</code> +
- +
-==== Tiptilt server says the clock isn't running ==== +
- +
-<code> +
- \\  First check whether the clock itself is running and the other machines receive the clock signal. Look at the clock cards at the back of other computers in the rack. The clock cards have three LEDs, one yellow and two greens. If the computer is receiving the clock signal properly all three LEDs should blink, but at a different rate. If the LEDs on all the clock cards are solid then reboot the GPS computer. When the GPS computer is down, it is best to cycle the power also on the box right above the GPS computer. \\    \\  If the clock appears to be working properly on other machines and not on the tiptilt now it is time to reboot tiptilt. \\    \\  [There is a bug in the real time part of the CCD code. It is caused by the clock in the tiptilt system either not running at all or having been set to a time very different from the last time the CCD ran.] \\    \\  For the time being the only solution is to reboot tiptilt, but do so from the lab. Power OFF the CCD, then reboot the tiptilt machine and go into the BIOS. Make sure that interrupt 11 has been set to ISA legacy, save the BIOS and reboot. When the clock card LEDs in the tiptilt machine indicate proper clock signal, turn the CCD back on and start the tiptilt server. \\    \\  Also, sometimes Serial Port 3 grabs IRQ 11 which stops the clock from running. Since there is no serial port 2 it's safe to disable this in the BIOS. This problem normally comes up when there has been a power outage. \\    \\  Sometimes syncing the clock can also cause this problem, but that should be fixed soon. If it does, exit the tiptilt server, log in as root, and reload the tiptilt model using the following commands:   \\  /sbin/rmmod tiptilt_rt   \\  /sgin/insmod /usr/local/modules/tiptilt_rt.o \\    \\  Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine. +
- +
-</code> +
- +
-==== Tiptilt is not locking on a star or locks, but lets the star drift away ==== +
- +
-  * Are all the mirror covers open? [NoteW1 M7 cover sometimes needs two clicks to open, despite Scope-monitor indicates open.] +
-  * Has TIPTILT COMM been run from Cosmic Debris? +
-  * Check the ACQ alignment to make sure the tick marks are centered on the laser. +
-  * If there is plenty of starlight getting into tiptilt, then try re-initializing by clicking [TIPTILT COMMon Cosmic Debris. +
-  * If there are a large number of background counts on TT, then close the M5 cover on the telescope, and click [DBIASon the TIPLTILT GUI to clear the background counts. Re-open the M5 cover. +
-  * If a star drifts even with TT locked, there could be a bright sky or light from other beams getting into the affected telescope's beam. This will show in the TT server as much lower counts than the other locked stars, but still high enough to lock TT. Use the laser to find the correct position to lock TT. +
-  * If TT unlocks and the star drifts, the TT servo may not have engaged. This is engaged with the TIPTILT button on the telescope gui. Clicking it may not start the servo the first time. Try it again if the green dots drift on the TT plot windows or if the Servo status in the telescope gui reads None instead of Wobb 1. +
-  * Are the TIPTILT buttons turned on from the POWER GUI? +
- +
-==== Tiptilt servo oscillates ==== +
- +
-You will see the oscillation in the green dots of the tiptilt GUI windows. Sometimes you can also see the oscillation in the white starlight dots or as an elongation of the star when looking at the ACQ field while tiptilt is locked. Some scopes have an oscillation that has not yet been diagnosed. W2 is one that usually oscillates. A diagonal motion in the tiptilt box indicates an oscillation in one axis only, while a vertical/horizontal motion indicates an oscillation in both directions. Motion from the upper right to lower left corresponds to elevation axis while motion from the upper left to lower right corresponds to the azimuth axis. (I think you can check direction by typing sin into telescope server to send sine waves to the telescope.) There are a few ways you can try to correct the oscillation manually tuning the servo:+
  
-  * Type "tune" into the tiptilt server. Select the appropriate telescope. Turning the gain down normally helps. +===== LabAO =====
-  * Type "tune" into the appropriate telescope server. The default value for the proportional term is -0.5 and differential term is 0.0. Adjust these values between -0.2 to -1.0 for proportional and 0.0 to 0.2 for differential. +
-  * Read section on adjusting the Telescope Tracking Gain using the Dome Server GUI (ref).+
  
-==== Tiptilt is saturating ====+==== LabAO Shutters Not Working ====
  
-  * Tiptilt saturates at ~ 200,000 counts. If you are near this limityou can reduce the TT exposure time to lower the number of counts. +<font 10.5pt font-family:/inherit;;#333333;;inherit>If the labao shutters are not working, the power button on each of the two red control boxes on the top of the table might need to be pressedAfter that the other button needs to be pressed and held for 5 seconds so that the units can be controlled remotely. (Note: look for instructions on table.) A good way for this step is to manually set the shutter to the opposite state that it is in remotely, keep pressing the “enable” button until you hear a slight clickRelease. The shutter should now change state.</font>
-  * Set the TT exposure time on Cosmic Debris in the box for "Tiptilt (mS)". This will set the TT exposure time when slewing to a new target. +
-  * To change the exposure when already at a target, then click the [EXP] button on the TipTilt GUI. This will bring up a dialog box where you can enter the new exposure time in msecCheck to make sure the tiptilt frequency (in Hz) changes on the tiptilt server after changing the exposure timeIf the frame rate doesn't change, then set the exposure time back to the old value and try entering the new value againYou might have to do this a few times to actually get the frame rate to change. +
-  * Frame rates for given exposure times: +
-      * ExpTime = 5 msec, Frame Rate = 157 Hz +
-      * ExpTime = 2 msec, Frame Rate = 299 Hz +
-      * ExpTime = 1 msec, Frame Rate = 427 Hz+
  
-==== Tiptilt counts are way too low ====+<font 10.5pt font-family:/inherit;;#333333;;inherit>If this does not work, or the other CHARA shutters are not working then a reboot is required. The order of restart needs to be correct for all shutters to work again (not shutters on insturment tables which are controlled by instrument servers).</font>
  
-Try using a slower frame rate or increasing the NSUMAlso ensure that the acquistion is properly aligned with the laserTo change the frame rateclick the [EXP] button on the tiptilt GUI and enter a longer integration time. Remember to change "Tiptilt (mS)" on Cosmic Debris to keep the same exposure time when slewing to the next target.+<font 10.5pt font-family:/inherit;;#333333;;inherit>Turn off electronics box for shutters (above the GPS computer)Reboot the gps computer. Check to see that the shutter server is running by typing bootlaunch_master on gps computerIf runningturn on shutter electronics box (you just rebooted gps computer so restart gps server).</font>
  
-==== Tiptilt counts are negative ====+==== LabAO DM Electronics ====
  
-The bias frame is bad. Get new one or turn it off.+If labao server complains that it failed to find the EDAC40 device, try unplugging the power and ethernet cables for DM and then replugging in again. The boxes for these are under the respective tables.
  
 ===== OPLE and Metrology ===== ===== OPLE and Metrology =====
Line 390: Line 286:
   * This will install run_ople. Try opening the ople server again.   * This will install run_ople. Try opening the ople server again.
  
-==== Metrology and Homing carts ====+==== Homing carts ====
  
-  * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror. +  * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has stopped on the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. The X does not guarantee the cart has retained its home position. This home position can be checked by using the CHECK button on the ople gui or by typing the command “homechk S1” into the ople server. When the error value is displayed on the ople server, hit ESC to clear the display. Click on the [OL] and [MAN] buttons and hit [TRACK] to send the cart to the desired cart position. 
-  * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same timeThe X in the OPLE server under the HM will indicate it has homed to the home switchIf cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringesYou can also find the size of the error by typing the command "homechk S1" into the ople serverHit the [HOME] button and it should move forward and trigger the home switchHit [TRACK] to home it to the new home position.+ 
 +==== Metrology ==== 
 + 
 +  * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror 
 + 
 +Steps for adjusting MET2: 
 + 
 +  - Looking at the met signal while the cart is at the back. 
 +  - Peak so that the beam is aligned as well as possibleDoes not need to be perfect but not loose. 
 +  - All beams have signal between 1-2 volts while at the back. Sometimes the signal strength can change for no apparent reasons. 
 +  - Using steps of 50, try moving 3 times in one directionAfter each step, wait 5 seconds or so for things to updateIf nothing then try the other way. 
 +  - Return to starting position. 
 +  - Repeat for the other axis.
  
 ===== Beam Samplers ===== ===== Beam Samplers =====
  
-==== Beam Samplers are not moving ====+==== Beam Samplers are not moving (not a problem anymore) ====
  
 This happens most frequently with the E1 beam sampler. If you are doing alignments and are using E1, try to move it forward first to see if it dies. If it does, follow this procedure. This happens most frequently with the E1 beam sampler. If you are doing alignments and are using E1, try to move it forward first to see if it dies. If it does, follow this procedure.
Line 410: Line 318:
 ==== Restarting the Beam Sampler Server ==== ==== Restarting the Beam Sampler Server ====
  
-The most common cause for getting multiple copies of a server is using the socket manage RESTART button, which is flaky at best. Don't do this. If you need to restart a server you should do it manually.    1. Make sure you are logged into the right machine:    ssh ctrscrut    2. If you are not sure see if the socket manager will tell you. If it doesn't have a look in the file:    /ctrscrut/chara/etc/sock_manager/socket_manager.list    3. See if there are any ghosts running    ps aux | grep esp_server    Yes, it's hard to know which one it is with the esp servers. You can work out which ones are ghosts by typing the command    tsockman |grep ctrscrut    which will give you a list of the servers running on the machine you are interested in. Checking for non-matching PIDs will tell you which processes you need to stop.    4. Stop those processes:    kill -9 PID1 PID2 ….    5. Make sure there is no sign of it in the socket manager    tsockman rm BS    5. Restart the beam sampler servers (this starts both beam sampler servers BS1 and BS2):    bootlaunch_beamsamp+The most common cause for getting multiple copies of a server is using the socket manage RESTART button, which is flaky at best. Don't do this. If you need to restart a server you should do it manually.
  
-==== "Failed to request position of S1" ==== +1. Make sure you are logged into the right machine: ssh ctrscrut
-<code>+
  
- \\  Cosmic Debris reports "Failed to request position of S1" when trying to set the beam order. This indicates that ople is no longer talking to the beam samplers. If the beam sampler server has been restarted recently, then the ople server will also need to be restarted to re-establish the connection\\    \\  The beam sample server runs on ctrscrut. The ople server tries to open connection to the beam sampler when it starts. If the beam sampler is not there, or dies,   \\  you need to restart the ople server as there is no command to reopen that connection.+2. If you are not sure see if the socket manager will tell youIf it doesn't have look in the file: /ctrscrut/chara/etc/sock_manager/socket_manager.list
  
-</code>+3. See if there are any ghosts running
  
-===== Remote Observing =====+ps aux | grep esp_server
  
-==== VPN connection is not working ====+Yes, it's hard to know which one it is with the esp servers. You can work out which ones are ghosts by typing the command
  
-To re-establish the VPN connection, follow these steps:+tsockman |grep ctrscrut
  
-  * Open VPN status page from the firefox web broswer. To do this, click on the "AROC VPN reconnect" icon on the zoot desktop. Alternatively, you can open the firefox web browser and go to 192.168.3.1/vpn_sta.htm +which will give you a list of the servers running on the machine you are interested in. Checking for non-matching PIDs will tell you which processes you need to stop.
-  * The browser will prompt you for the admin user name and password. If these entries are not already filled in, then look for this information on the papers near the console. +
-  * If the browser does not open, then check if there are other instances of firefox already running. To do this, open a terminal window, type "ps auxw | grep firefox", and kill the other instances of firefox so that you can open a new window. +
-  * When the VPN connection is working, it will report "Phase 1: M-ESTABLISHED / Phase 2: ESTABLISHED" under the state column. +
-  * If either Phase 1 or Phase 2 states say IDLE for a given VPN, then you need to re-establish the connection. To do this, click on the "DROP" button for that VPN. Then click on the "CONNECT" button and wait for the Phase 1 and Phase 2 connections to be established. +
-  * Make sure to close Firefox by using CTRL-Q to quit the program and not just close the window.+
  
-===== General Problems =====+4. Stop those processes:
  
-==== GUIs are frozen ====+kill -9 PID1 PID2 ….
  
-Check to see if Sockman is working. Click the [LIST] button on sockman. If the list doesn't come up, Sockman may be hung up. Follow the Sockman restart procedure in the text file on the desktop if it won't respond.  Hit [REOPEN] on the gui to see if it reconnects with its server. if that does not fix it, the server may be dead. From the LIST on Sockman, select the server suspected of being frozen. Ping it and if it reports back as being dead, see [[:chara:trouble_shooting#server_is_frozen|Server is frozen]] section below.  +5. Make sure there is no sign of it in the socket manager
-Telescope guis often freeze. Hit reopen on the gui to bring it back to life. Hitting [MOVE] on the telescope gui before the star has stopped moving can cause the gui to freeze. Avoid doing this. Close the dead one and open a new one to fix the problem.+
  
-==== Server display is blank or filled with jibberish ====+tsockman rm BS
  
-  * If the top of a server goes blank, try typing "sb" (start backgroundon the server command line. +6. Restart the beam sampler servers (this starts both beam sampler servers BS1 and BS2):
-  * If the server screen fills with jibberish, try hitting CTRL-l in the server to clear it.+
  
-==== Server is frozen ==== 
- 
-Try quitting the server using CTRL-C and entering "Y". If that doesn't work, try the following: 
- 
-  * On Sockman, click [LIST] and select the appropriate server. It will give a PID number you can use to kill the server in a terminal. Open a terminal and follow the directions for Restarting Servers above. 
-  * If a server window does not close, click the X in the upper right corner to close it, but only after getting the PID from Sockman. 
-  * Click [REOPEN] on Cosmic Debris and relevant GUIs to re-initialize communication with the server after it is restarted. 
-  * A folder is on the desktop that has the restart commands for CTRSCRUT servers and the Shutters server restart command running on OPLE. Use it to restart servers that will not reopen from the menu. 
- 
-==== PAVO Server - Error communicating with IFW ==== 
- 
-The following error message sometimes appears when starting the PAVO server:  "Error communicating with IFW. Could not read from IFW. Is the filter wheel plugged in and on?"  Most of the time, restarting the PAVO server will clear this error message and allow PAVO to communicate with the filter wheel. If restarting the server doesn't help, then check that the cables going into the small, black IFW box on the PAVO table and into the back of the PAVO computer are plugged in securely. 
- 
-==== Adding or Finding a Star in the CHARA Database ==== 
- 
-The command "dbadd" can be used to look up the CHARA number for a star or to add a star to the CHARA database. To use dbadd, open a terminal window, and type "dbadd starname" where starname is the common name (e.g., Vega) or identifier of the star (e.g., IRC, HR, HD, SAO, FK5, HIP, GJ, or 2MASS designation). If the object is in the database, then it will return the CHARA number. 
- 
-You can also look up different identifiers for the star by using the SIMBAD database: [[http://simbad.u-strasbg.fr/simbad/sim-fbasic|http://simbad.u-strasbg.fr/simbad/sim-fbasic]]  
-  
-DBADD uses SIMBAD to look up the coordinates of the star and then searches to see if the star is already in the CHARA database. If the star is in the database, it will return the CHARA number. If the star is not in the CHARA database, it will ask if you want to add it and assign a CHARA number if you answer yes. 
- 
-If an object is not in SIMBAD (such as a recently discovered nova), then you must enter the object manually using the instructions below. 
- 
-  __Finding a star in the CHARA database__ 
- 
-Here is an example of how to find the CHARA number of a star, nova, or AGN that is already in the CHARA database: 
- 
-[ctrscrut:502] dbadd NGC 4151  Object "NGC 4151" found as CHARA number: 320263 
- 
-Note: If the target information stored in the CHARA database does not match SIMBAD, then dbadd will provide the coordinates and magnitudes listed in SIMBAD, and it will also list the CHARA numbers of objects in the database located within 10 arcsec. Please check these nearby entries before adding a new star to the database. 
- 
-  __Targets that are not in the CHARA database__    If an object is not in the CHARA database, then you can use dbadd to query SIMBAD and add the star to the CHARA database. This might need to be done for faint stars, novae, and AGN. If the object is found in SIMBAD, then dbadd will return the target's coordinates, magnitudes, and spectral type and ask if this it the object for which you are looking. If you answer yes, then dbadd will issue a CHARA number and add the object to the database. Please check the target coordinates to make sure that they are correct before adding a new star to the database. 
- 
-Here is an example of how to add an object to the CHARA database: 
- 
-[ctrscrut:534] dbadd BD 41 3807  SIMBAD found this for "BD 41 3807": 
- 
-ID 2MASS_J20340850 4136592  CHARA 320414  2000.0  CAT8 2MASS_J20340850 4136592  RA 20 34 08.5148  DEC 41 36 59.408  PMRA -2.32  PMDEC -3.87  PAR 0.58  RVEL -27.50  VMAG 10.03  HMAG 6.23  KMAG 5.99  SP O5.5Ifc  END 
- 
-Is the above the object you are looking for? [y/n] y  Added "BD 41 3807" to the database. 
- 
-In the example above, the object "BD 41 3807" has been added to the database as CHARA number 320414. 
- 
-  __Targets that are not in SIMBAD__    Some objects like recently discovered novae are not in SIMBAD. Entering dbadd <star designation> for a star not in the SIMBAD database returns the message "Simbad is unable to find an object matching <star designation>. Try using a different catalog designation, or use the "-m" switch to add the object manually."    The command dbadd -m is used by entering the star name and coordinates in this format:    Usage: dbadd -m <name> <RA> <Dec>    <name>: Object ID (no spaces)    <RA> : XXhXXmXX.X or XXhXX.X (no spaces)    <Dec> : XXdXXmXX.X or XXdXX.X (no spaces)    There are cases where an object is not in SIMBAD and doesn't return a result in dbadd, but is in fact in the CHARA database. Novae and AGN's are the likely objects that cause this result. At times, objects have multiple entries and several CHARA numbers since the names can be so unique. A system will be developed to find these entries without knowing or having a conventional database designation.    __Binary stars__    Some binary stars have a common HD number with an A or B after them. These can cause problems as Cosmic Debris does not accept non numerical entries when filling in star designation. These stars are likely in CHARA's database but need to be searched in dbadd or on SIMBAD to get the CHARA number or another designation. 
- 
-When a recognized identifier is entered, the HD number with A or B will usually display. Confirm that the coordinates, magnitudes and spectral type match the star desired. If they do not match, you may have the wrong star or the database may have the wrong info and the baseline solution will be affected. Offsets based on incorrect coordinates or misidentifications can move the fringes many thousands of microns away from the calculated position. This can happen when observing a dim companion (B) using the brighter (A) companion's coordinates. Inform the observers that noting and using the CHARA number will save time next time the target is observed.    __Editing the database__    If you find a mistake in the database, please send an email to Nils to have it corrected. Identify what you believe to be the error and what is the correct information. 
- 
-==== Adding an anchor to the wiki page ==== 
- 
-To add an anchor to the wikipage, type the following where you want the anchor inserted (without the spaces between the symbols):  [ [ #AnchorName ] ]    Highlight the text you want to be a link to the anchor, click the link symbol above and select anchor. Select a page(such as Trouble Shooting) where the anchor is located and give the same name for the anchor. 
- 
-==== Displays for the MIRC-X wolverine2 monitors ==== 
- 
-Follow instructions in the [[:chara:mircx_monitors|linked page to display the MIRC-X wolverine2 monitors]] in the control room. 
-<code> 
- 
- \\  Last updated 2017-04-12 
- 
-</code> 
- 
-==== The Telescope won't move or stopped moving ==== 
- 
-<code> 
-\\  Have the powers been turned on to the drives? Are the scopes disabled? The usual state of the telescopes is disabled until enabled. This is due to the stall function of the scopes which eventually disables the scopes when they are stowed. Enable the scopes by hitting [ENABLE] on the dome gui or in the telescope gui control tab. If a scope disables itself during a slew, it may be just an overcautious stall function. Hit ENABLE and the scope should continue to slew. If it disables again without moving, there may be something wrong at the scope, ie. a hatch left open, a ladder not put away, a tool on the floor, or something else physically impeding the motion of the scope. You will need to go to the dome to see what it is. The computer in the dome will give you control of the scope to turn it away from the problem.  \\    \\  Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopes. Check them for current times and continuous updates of numbers. If they are not updating, try to REOPEN them first. If that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of it. See **Dome Server Restart** below.  \\    \\  **Azimuth Limit Switches**   \\    \\  <font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>As of 11-'17, the azimuth limit switches are enabled and can stop the motion of the scopes if they try to go beyond -90º or  450º. The scopes will not be movable with normal inputs so follow these instructions to return them from the out of range condition.  \\    \\  1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software.  \\    \\  2. Make sure you understand why the limit was hit which may require a trip to the telescope.  \\    \\  3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope.  \\    \\  4. Click ENABLE then you can move the telescope back to its normal range of operation.  \\    \\  5. After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again</font>. 
- 
-</code> 
- 
-<font 16px/Calibri,sans-serif;;#333333;;white>**However, If you notice that a telescope will stop near AZ 90 or 270 with the scope still being ENABLED and refusing to move, this is often due to the new AZ limit switch being tripped at some point earlier. To get it moving again:**</font> 
- 
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;white>1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software.</font>      <font inherit/inherit;;initial;;white>2. M</font>ake sure the all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way.    <font inherit/inherit;;initial;;white>3. Click the OVERRIDE ON button in domegui MANUAL tab.</font> 
- 
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>4. Move the scope a bit back toward the direction it was coming from – for example, if the scope stopped at AZ 268 while rotating clockwise, move it back to 265 or so using AZ DEC. Then press STOP.</font> 
- 
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>5. Move the scope past AZ 270 by pressing AC INC. Usally 2-4 degrees will do.</font> 
- 
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>6. Click OVERRIDE OFF, go to the AUTO tab and press AUTO, then NEXT (also in the obsgtk; this will restore the original star's demand position to the scope).</font> 
- 
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7. When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90.</font> 
- 
-==== The Telescope won't track ==== 
- 
-  
-If a bright star is manually found and not slewed to by number, (usually after the scope has gotten lost), the telescope will not track as it does not know it has gone to a star. When you init the EL and AZ position of the scope to the star's position, you must enable the tracking by going to the dome gui, selecting the RA/DEC tab and hitting the [GO RA/DEC] button. The scope will track, but the pointing model may still be off. Try slewing to the same star and initing on it after it is locked in Tiptilt. This can be done in the dome with the computer at the telescope. 
- 
-==== The Telescope won't stop moving or moves beyond the commanded position ==== 
- 
-  
-Disable the scope whenever its motion is outside of what it is supposed to do. Turn the powers off to the axes as well on the Power gui. The dome server is usually to blame and will need to be restarted. 
- 
-==== Dome Server Restart ==== 
- 
-  
-To manually start the dome server:    
-1. Make sure the power to the drives is OFF.  
-2. Login to the relevant computer as root. For example, type "s1" or "ssh s1" to log on to S1.  
-3. Work out the process ID number (PID), either with the command    
-(s1:1001) tsockman get dome_S1  
-Name : dome_S1  
-Machine : s1.chara-array.org  
-PID : 29953  
-Commands : -1  
-Data : -1  
-Message : 4002  
-Restart : /usr/local/bin/dome_server -A33.7441 S1    
-or with  
-(s1:1003) ps aux | grep dome  
-theo 4473 0.0 0.0 61188 748 pts/3 S 10:45 0:00 grep dome  
-observe 29953 18.5 0.4 35596 9860 ? Sl Apr21 416:11 /usr/local/bin/dome_server -A33.7441 S    
-It can also be found by pulling up the LIST on SOCKMAN and selecting the relevant dome.    
-So in this case the PID is 29953.    
-4. Try and stop the server gracefully: kill -2 29953  
-5. You should then check that the server has indeed stopped:    
-[s1:600] tsockman get dome_S1  
-Name : dome_S1  
-Machine : s1.chara-array.org  
-PID : 15635  
-Commands : -1  
-Data : -1  
-Message : 2008  
-Restart : /usr/local/bin/dome_server -A41.0166 S1    
-If the socket manager still thinks it's running you will need to stop it forcefully: kill -9 29953; tsockman rm dome_S1    
-6. Restart the dome server by copying the command at the end of the /etc/rc.local file:    
-[s1:602] more /etc/rc.local  
-<<< Press space bar or enter to scroll through file>>>    
-#Run the dome server    
-/usr/local/bin/tsockman remove dome_S1  
-/usr/local/bin/dome_server -A41.0166 S1 &    
-(Note: the part of the command that saves information to /var/log/dome_S1.log has been removed.)    
-7. Turn the power to the drives back on.  
-8. Hit REOPEN and ENABLE on the domegtk, and type "otcs" in the telescope server.    
-You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scopes position. 
- 
-==== Telescope clock is not correct ==== 
- 
-  
-The NTP server has obsoleted this problem and fix as of 2014. 
- 
-==== Telescope is not receiving the commanded position for a target ==== 
- 
-  
-Sometimes it happens that a telescope receives the wrong position for a target or does not receive the commanded position at all. The commanded position is listed on the telescope server in the first column under TCS Az/El; the second column lists the actual position of the telescope. Try entering the star designation directly into the telescope server, ie. hd 123456. If it does not accept the number, try closing and restarting the telescope server and hitting repoen on Cosmic Debris and the telescope gui. Try entering the star into the server again. If that does not work, it is possible that something is wrong with the dome server. To restart the dome server follow these steps:  
-* DISABLE the telescope using either the telescope or dome GUI. 
- 
-  * Turn off the power for the telescope (both AZ/EL). 
-  * First shutdown the telescope server 
-  * Then use SOCKMAN to select the appropriate dome server from the list (dome_E1, etc) Get the PID number for reference. 
-  * The dome server will need to be restarted from the command line in a terminal. 
-  * Open a terminal window and log on to the telescope computer by typing "s1", "s2", "e1", "e2", "w1", or "w2" 
-  * Look for instruction in the /etc/rc.local file by typing "more /etc/rc.local". Scroll through the file by hitting space bar or Enter. 
-  * Locate instruction listed under "#Run the dome server". Copy and paste the commands at the prompt directly on the telescope computer: 
-      * /usr/local/bin/tsockman rm dome_S1 
-      * /usr/local/bin/dome_server -A33.7441 S1 >& /var/log/dome_S1.log 
-  * (Replace S1 with commands for appropriate telescope) 
-  * After the dome server is running, re-open the telescope server and click [REOPEN} on the dome and telescope GUIs and on Cosmic Debris. 
-  * ENABLE the telescope using the telescope or dome GUI. Check to see if the command and telescope positions are correct. If they are, then turn on the power for the telescope drives. Re-enter the star information by entering the HD number in the telescope server and click [GO NEXT] to send the telescope to the star. Make sure that the telescope is behaving as expected. The telescope might have drifted a bit, so if you can't find the star, it might be necessary to use the Telrad on a bright star to re-initialize the pointing. 
-  * Restart commands for the dome servers are also listed on the desktop in a text file. 
-<code> 
- 
-   \ 
- ==== Telescope is tracking poorly, overshooting in slew, oscillating. ==== 
- 
-</code> 
- 
-<code> 
-This might mean that the gain for the tracking servo is wrong. Note that changing this gain can be dangerous, especially if you set it too high as that can cause the telescope to oscillated and damage the drives. Please only do this if you are very very sure that it is necessary. Symptoms of bad gain are:    The scope over shoots the position while slewing. The star will be seen to move out of the window and may come back after a few seconds. This means the slewing gain is too low.  The scope oscillates when tracking or after a slew. The star will be tracing an ellipse, figure eight or other looping shape. This means the tracking gain is too low. You can damp this out with the telescope or dome gui by disabling the scope, then re-enabling it. Adjust the gain upward and watch it on the next slew.    In all cases if either gain is too high the scope will go into "Fog Horn" mode, which is bad. You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high.    On 10-22-2016, the gain settings were:    | Scope  | AZ Slewing  || EL Slewing  || AZ Tracking  || EL Tracking  || Date Updated  | 
- 
-</code> 
- 
-| |Gain|Fn|Gain|Fn|Gain|Fn|Gain|Fn| | 
-|S1|7|4|4|3|16|4|10|7|10-26-2017| 
-|S2|7| |7| |10| |10| |10-22-2016| 
-|E1|7| |7| |13| |10| |10-22-2016| 
-|E2|7| |7| |13| |10| |10-22-2016| 
-|W1|13| |10| |13| |10| |10-22-2016| 
-|W2|7| |7| |13| |13| |10-22-2016| 
- 
- \\ 
-Note that these values may change as the temperatures change rapidly. These are still being adjusted so this is by no means final. Be sure to set mode back to AUTO if changing the gains left it in Slewing or Tracking mode. 
- 
-==== How to Adjust CPUMotor Gains ==== 
- 
- \\ 
-The GAIN controls the gain of the feedback between the encoder and the drive velocity. A high gain means a "stiffer" response, but can lead to oscillations or fog-horning if it's too high. \\  \\ 
-The Fn controls the maximum frequency of the servo response. A high Fn means higher frequencies are allowed through, which can mean correcting for faster problems but if too high can also lead to oscillations or fog-horning. \\  \\ 
-The software will not make changes to either of these quickly as that is a dangerous thing to do. There is NO POINT to clicking the up or down buttons more than once every few seconds. Indeed it is bad to do so as you will confuse the software. The change between slewing and tracking is also slow for similar reasons. This is why sometimes the "wrong" thing seems to change. It is a sign that you are trying to do things too quickly. \\  \\ 
-Slew and tracking mode work differently, mostly because the speeds are so different. \\  \\ 
-IN SLEWING MODE \\  \\ 
-- If the gain is too low you will overshoot the target. \\  \\ 
-- If the gain is too high it will fog-horn. \\  \\ 
-- If the Fn is too high it will also fog-horn, even at low gains. \\  \\ 
-You need to have the lowest possible gain and Fn in slew mode that doesn't overshoot the target. Fn in slew mode should almost never be higher than 4. If it is, please turn it back down to 4. If you think this is a problem please let Theo know, along with a detailed explanation of what happened. \\  \\ 
-IN TRACKING MODE \\  \\ 
-- If the gain is too low it will keep moving between slewing and tracking. \\  \\ 
-- If the gain is too high it will fog-horn. \\  \\ 
-- The same goes for Fn. \\  \\ 
-In tracking mode you want the highest gain and Fn that allows the telescope to track well without fog-horning. If it "oscillates", which you will see in the green dots of tiptilt oscillating, try turning up the gain, and also try turning down the Fn. \\  \\ 
-Some final remarks: \\  \\ 
-- The gain is temperature dependent, so when the temperature changes these things will change, but more so for tracking. \\  \\ 
-- The tiptilt system almost never causes oscillations, it almost always shows you that the scope is oscillating. If the white dots are centered on tiptilt and the green dots are moving the tiptilt is doing it's job and correcting for scope motion. \\  \\ 
-- If a drive gets disabled at the end of a slew, the gain is too low. At low gain and low velocity the encoder signal changes very slowly or doesn't change at all. After 5 secs the software interprets this as a stall and disables the drive. The tricky part is that increasing the gain to avoid this situation might make the telescope to oscillate during the next slew. So the gain should be low (4 or even 1) during slewing but higher 7 or 10 when the telescope is basically at the target position. 
- 
-==== E2 AOB Dichroic Recovery ==== 
- 
-{{:chara:files:beaconrecoveryinstructions.pdf|E2 AOB Dichroic Recovery Document}} 
- 
-==== E1 Hut and Cooler Communications Recovery ==== 
- 
-{{:chara:files:e1_hut_coms_recovery.pdf|E1 HuT and Cooler Communications Recovery}}  \\  \\ <font 16px/inherit;;inherit;;inherit>**S2 Acquisition Server Restart**</font> 
- 
-S2's acquisition server resides on "S2pi4" computer. Logging on to it (ssh s2pi4) and running bootlaunch will restart it 
- 
-====== Dome issue ====== 
- 
- \\ 
-At times the domes do not rotate, open or close, or otherwise behave. Some problems are simple and others are more complex. 
- 
-==== Dome does not rotate ==== 
- 
- \\ 
-Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope's position. If not, it will insist on being in the wrong place. If it is not at the same AZ as the scope, manually move it until it is centered on the telescope in spycam 1. If the dome AZ does not read the same as the telescope AZ, enter the scope AZ in the position box of the DOME tab of appropriate dome server and hit the INIT POS button to tell it at what AZ it is. \\  \\ 
-If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker. 
- 
-==== Dome does not open ==== 
- 
- \\ 
-If the dome won't open, try hitting the SLIT CLOSE button on the telescope gui first. It may be that there was a computer or server issue and the computer thinks the dome is open already and won't allow it to open again. Hit the SLIT OPEN button to see if it works now. If it still won't open, go to the bunker and look to see if the power switch is on to the dome. It will be up and red if on. \\ 
-Also look to see if the power cable is connected to the computer box. If it's connected up high to the manual dome controls, the cable needs to be brought back to the computer box connector. __Turn the power off first__  to the dome before disconnecting the power cable. Slide it onto the connector at the box and turn the collar to lock it in place. Turn the power back on and see if you can open the dome from the computer in the bunker. 
- 
-==== Dome does not close ==== 
- 
- \\ 
-If a dome will not close after observing, it may need to be closed manually at the bunker. Turn the power to the dome off at the switch, disconnect the cable from the computer control box and connect it to the manual control switches above. Turn on the power to the dome after connecting the cable and use the two controls to close the dropout first, then the slit. The dropout must be fully closed and the slit closed over it to be shut properly. Inform the staff that the dome did not function properly so someone can look at it in the morning. 
- 
-====== HUT servers ====== 
- 
-==== I can't change the camera settings on the TV ==== 
- 
- \\ 
-The HUT servers control functions such as finder and acq exposure times and gains, heater and dehumidifier usage, and various AO functions. An observer may find that the camera settings do not display or are not adjustable. The HUT server may be the cause if it has quit. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the camera settings are displayed and the settings can be changed from the gui, then the HUT server is ok. Restart the telescope server to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well. If the gui is not functioning, the HUT server will need to be restarted. The HUT servers run on the computer at each telescope. Find the PID from Sockman for the server you want to shut down. Open a terminal and log into the correct telescope computer (for example, type "s1" to log on to S1). Use the command "ps aux | grep HUT" to find the process you need to shut down. The number should match the PID from Sockman. Kill the process by typing "kill -9 pid" where pid is the process number and remove the entry from Sockman with the REMOVE button on the gui or by command line. Restart the hut server using the new "bootlaunch_hut" command described in the section on [[:chara:trouble_shooting#restarting_servers|Restarting Servers]]. If the server won't restart, a reboot of the power supply in the telescope bunker might be necessary. The power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunker. The power supply has green readouts of volts and current. After turning the power off for 10 seconds and back on, try restarting the server from the computer in the bunker to see if it starts cleanly. If so, then restart the telescope server, reopen the connection to the telescope gui, and hit REOPEN on Cosmic Debris. Part of the HUT server also controls the AO table. If the AOB part of the HUT server doesn't work, then the power supply on the back of the AO table in the telescope dome might need recycling. This power supply controls the actuators at M2 and the AO table. The power supply box is a 6×9 inch aluminum box on the back of the AO table, behind the keyboard and monitor. Turn it off with the power button on the bottom edge of the box, wait 5 seconds and turn it back on. The HUT server should now restart cleanly. Restart the telescope server as well to make the connections to the telescope gui. Hit REOPEN in Cosmic Debris if you are observing to make all needed connections. 
- 
-===== Tiptilt Server ===== 
- 
- \\ 
-The tiptilt server controls the CCD based tiptilt detection system. \\  \\ 
-Before you start the tiptilt server, you must ensure that the power to the cooling system and the CCD iteslf is on. It is extremely important that the cooler be running before you turn on the CCD and is only turned off if you are sure the CCD is NOT running. You can start the server from the X windows menu or with the command xtiptilt. \\  \\ 
-Note that there are background counts and read noise to deal with. Whenever you change the frame rate, please ensure that the bias frame is OK. The server will attempt to load an old bias frame that should work, but if things are not working, try making a new bias frame by ensuring that the detector is in the dark and typing "mkbias" into the tiptilt server. \\  \\ 
-In the tiptilt GUI windows, the white dots represent the starlight while the green dots represent the motion applied to telescope's secondary mirror to keep the starlight centered. When tiptilt is locked the white dots will be brought to the center of the tiptilt window. The green dots should be mostly centered also. W2 and E2 telescopes have a small oscillation that show as back and forth plots of the green dots. 
- 
-==== Tiptilt server complains about the CCD ==== 
- 
- \\ 
-Is the CCD turned on? When the tiptilt server starts up it tries no more than five times to communicate with the CCD. If they all fail, it will give up. If this happens, try cycling the power to the CCD and try again. If this fails, connect to the tiptilt machine and type the command rtccdAPIDemo, which should return with no errors. Try this command a few times, but if it still fails, there is a more serious problem. Turn off the CCD and reboot the tiptilt computer. If it still fails, I am afraid you are in more serious trouble. \\  \\ 
-Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine. \\  \\ 
-Also note that cycling the power on the CCD can cause harm so be sure you need to do it before trying it. Also, it is important to wait for at least 20 seconds after turning off the power before turning it on again. 
- 
-==== Tiptilt doesn't seem to be talking to the telescopes ==== 
- 
- \\ 
-Sometimes the telescope server will not show that TT is running. It will show 0Hz for a signal rate for TT. Running TIPTILT COMM will not get it started while other scopes do show it starting. Close and restart any telescope servers that won't connect after two tries of TIPTILT COMM. \\ 
-Note: There is more info in the software manual on this topic, but I wasn't sure if it was still relevant. 
- 
-==== Tiptilt server says the clock isn't running ==== 
- 
- \\ 
-First check whether the clock itself is running and the other machines receive the clock signal. Look at the clock cards at the back of other computers in the rack. The clock cards have three LEDs, one yellow and two greens. If the computer is receiving the clock signal properly all three LEDs should blink, but at a different rate. If the LEDs on all the clock cards are solid then reboot the GPS computer. When the GPS computer is down, it is best to cycle the power also on the box right above the GPS computer. \\  \\ 
-If the clock appears to be working properly on other machines and not on the tiptilt now it is time to reboot tiptilt. \\  \\ 
-[There is a bug in the real time part of the CCD code. It is caused by the clock in the tiptilt system either not running at all or having been set to a time very different from the last time the CCD ran.] \\  \\ 
-For the time being the only solution is to reboot tiptilt, but do so from the lab. Power OFF the CCD, then reboot the tiptilt machine and go into the BIOS. Make sure that interrupt 11 has been set to ISA legacy, save the BIOS and reboot. When the clock card LEDs in the tiptilt machine indicate proper clock signal, turn the CCD back on and start the tiptilt server. \\  \\ 
-Also, sometimes Serial Port 3 grabs IRQ 11 which stops the clock from running. Since there is no serial port 2 it's safe to disable this in the BIOS. This problem normally comes up when there has been a power outage. \\  \\ 
-Sometimes syncing the clock can also cause this problem, but that should be fixed soon. If it does, exit the tiptilt server, log in as root, and reload the tiptilt model using the following commands: \\ 
-/sbin/rmmod tiptilt_rt \\ 
-/sgin/insmod /usr/local/modules/tiptilt_rt.o \\  \\ 
-Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine. 
- 
-==== Tiptilt is not locking on a star or locks, but lets the star drift away ==== 
- 
-  * Are all the mirror covers open? [Note: W1 M7 cover sometimes needs two clicks to open, despite Scope-monitor indicates open.] 
-  * Has TIPTILT COMM been run from Cosmic Debris? 
-  * Check the ACQ alignment to make sure the tick marks are centered on the laser. 
-  * If there is plenty of starlight getting into tiptilt, then try re-initializing by clicking [TIPTILT COMM] on Cosmic Debris. 
-  * If there are a large number of background counts on TT, then close the M5 cover on the telescope, and click [DBIAS] on the TIPLTILT GUI to clear the background counts. Re-open the M5 cover. 
-  * If a star drifts even with TT locked, there could be a bright sky or light from other beams getting into the affected telescope's beam. This will show in the TT server as much lower counts than the other locked stars, but still high enough to lock TT. Use the laser to find the correct position to lock TT. 
-  * If TT unlocks and the star drifts, the TT servo may not have engaged. This is engaged with the TIPTILT button on the telescope gui. Clicking it may not start the servo the first time. Try it again if the green dots drift on the TT plot windows or if the Servo status in the telescope gui reads None instead of Wobb 1. 
-  * Are the TIPTILT buttons turned on from the POWER GUI? 
- 
-==== Tiptilt servo oscillates ==== 
- 
-You will see the oscillation in the green dots of the tiptilt GUI windows. Sometimes you can also see the oscillation in the white starlight dots or as an elongation of the star when looking at the ACQ field while tiptilt is locked. Some scopes have an oscillation that has not yet been diagnosed. W2 is one that usually oscillates. A diagonal motion in the tiptilt box indicates an oscillation in one axis only, while a vertical/horizontal motion indicates an oscillation in both directions. Motion from the upper right to lower left corresponds to elevation axis while motion from the upper left to lower right corresponds to the azimuth axis. (I think you can check direction by typing sin into telescope server to send sine waves to the telescope.) There are a few ways you can try to correct the oscillation manually tuning the servo: 
- 
-  * Type "tune" into the tiptilt server. Select the appropriate telescope. Turning the gain down normally helps. 
-  * Type "tune" into the appropriate telescope server. The default value for the proportional term is -0.5 and differential term is 0.0. Adjust these values between -0.2 to -1.0 for proportional and 0.0 to 0.2 for differential. 
-  * Read section on adjusting the Telescope Tracking Gain using the Dome Server GUI (ref). 
- 
-==== Tiptilt is saturating ==== 
- 
-  * Tiptilt saturates at ~ 200,000 counts. If you are near this limit, you can reduce the TT exposure time to lower the number of counts. 
-  * Set the TT exposure time on Cosmic Debris in the box for "Tiptilt (mS)". This will set the TT exposure time when slewing to a new target. 
-  * To change the exposure when already at a target, then click the [EXP] button on the TipTilt GUI. This will bring up a dialog box where you can enter the new exposure time in msec. Check to make sure the tiptilt frequency (in Hz) changes on the tiptilt server after changing the exposure time. If the frame rate doesn't change, then set the exposure time back to the old value and try entering the new value again. You might have to do this a few times to actually get the frame rate to change. 
-  * Frame rates for given exposure times: 
-      * ExpTime = 5 msec, Frame Rate = 157 Hz 
-      * ExpTime = 2 msec, Frame Rate = 299 Hz 
-      * ExpTime = 1 msec, Frame Rate = 427 Hz 
- 
-==== Tiptilt counts are way too low ==== 
- 
-Try using a slower frame rate or increasing the NSUM. Also ensure that the acquistion is properly aligned with the laser. To change the frame rate, click the [EXP] button on the tiptilt GUI and enter a longer integration time. Remember to change "Tiptilt (mS)" on Cosmic Debris to keep the same exposure time when slewing to the next target. 
- 
-==== Tiptilt counts are negative ==== 
- 
-The bias frame is bad. Get a new one or turn it off. 
- 
-===== OPLE and Metrology ===== 
- 
-==== OPLE Server doesn't open - complains about not finding run_ople ==== 
- 
-This usually happens if someone runs a "make clean" in the cvs tree but doesn't follow up with a "make install" on ople. To generate the run_ople file, follow these steps: 
- 
-  * Open a terminal window and type "ople" to log onto the ople machine. 
-  * Change to the appropriate directory under the CVS tree by typing "cd control/cliserv/ople" 
-  * Update the CVS tree by typing "cvs update" 
-  * Run the Makefile script by typing "make" 
-  * Log on as root by typing "su" and entering password. 
-  * Type "make install" 
-  * This will install run_ople. Try opening the ople server again. 
- 
-==== Metrology and Homing carts ==== 
- 
-  * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror. 
-  * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has homed to the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. You can also find the size of the error by typing the command "homechk S1" into the ople server. Hit the [HOME] button and it should move forward and trigger the home switch. Hit [TRACK] to home it to the new home position. 
- 
-===== Beam Samplers ===== 
- 
-==== Beam Samplers are not moving ==== 
- 
-This happens most frequently with the E1 beam sampler. If you are doing alignments and are using E1, try to move it forward first to see if it dies. If it does, follow this procedure. 
- 
-  * Turn off power on Newport Universal Motion Controller/Driver for the Beam Samplers (on top of rack in computer area, labelled "BS") 
-  * Then go behind the racks and locate the serial port for BS. Look for all of the blue ethernet cables coming out. BS is labelled in prt 2. Remove the ethernet cable for BS. The step stool is available to make it easier to reach the blue ethernet cable at the back of the BS controller box. 
-  * Restart BS controller (press power button again). Wait for all axes to appear on screen. 
-  * Plug the ethernet cable back in. 
-  * Restart BS server on ctrscrut using commands in /etc/rc.local (see instructions below) 
-  * Reopen Beam Sampler GUI. You will need to home each Beam Sampler one by one. You should be able to do this from the regular Beam Sampler GUI. If that doesn't work, try opening "espgtk BS" from the command line. 
- 
-==== Restarting the Beam Sampler Server ==== 
- 
-The most common cause for getting multiple copies of a server is using the socket manage RESTART button, which is flaky at best. Don't do this. If you need to restart a server you should do it manually. \\  \\ 
-1. Make sure you are logged into the right machine: \\  \\ 
-ssh ctrscrut \\  \\ 
-2. If you are not sure see if the socket manager will tell you. If it doesn't have a look in the file: \\  \\ 
-/ctrscrut/chara/etc/sock_manager/socket_manager.list \\  \\ 
-3. See if there are any ghosts running \\  \\ 
-ps aux | grep esp_server \\  \\ 
-Yes, it's hard to know which one it is with the esp servers. You can work out which ones are ghosts by typing the command \\  \\ 
-tsockman |grep ctrscrut \\  \\ 
-which will give you a list of the servers running on the machine you are interested in. Checking for non-matching PIDs will tell you which processes you need to stop. \\  \\ 
-4. Stop those processes: \\  \\ 
-kill -9 PID1 PID2 …. \\  \\ 
-5. Make sure there is no sign of it in the socket manager \\  \\ 
-tsockman rm BS \\  \\ 
-5. Restart the beam sampler servers (this starts both beam sampler servers BS1 and BS2): \\  \\ 
 bootlaunch_beamsamp bootlaunch_beamsamp
  
 ==== "Failed to request position of S1" ==== ==== "Failed to request position of S1" ====
  
- \\ 
 Cosmic Debris reports "Failed to request position of S1" when trying to set the beam order. This indicates that ople is no longer talking to the beam samplers. If the beam sampler server has been restarted recently, then the ople server will also need to be restarted to re-establish the connection. \\  \\ Cosmic Debris reports "Failed to request position of S1" when trying to set the beam order. This indicates that ople is no longer talking to the beam samplers. If the beam sampler server has been restarted recently, then the ople server will also need to be restarted to re-establish the connection. \\  \\
 The beam sample server runs on ctrscrut. The ople server tries to open a connection to the beam sampler when it starts. If the beam sampler is not there, or dies, \\ The beam sample server runs on ctrscrut. The ople server tries to open a connection to the beam sampler when it starts. If the beam sampler is not there, or dies, \\
Line 825: Line 364:
   * If either Phase 1 or Phase 2 states say IDLE for a given VPN, then you need to re-establish the connection. To do this, click on the "DROP" button for that VPN. Then click on the "CONNECT" button and wait for the Phase 1 and Phase 2 connections to be established.   * If either Phase 1 or Phase 2 states say IDLE for a given VPN, then you need to re-establish the connection. To do this, click on the "DROP" button for that VPN. Then click on the "CONNECT" button and wait for the Phase 1 and Phase 2 connections to be established.
   * Make sure to close Firefox by using CTRL-Q to quit the program and not just close the window.   * Make sure to close Firefox by using CTRL-Q to quit the program and not just close the window.
 +
 +===== Network Problems =====
 +
 +==== User accounts not recognized ====
 +
 +The gps computer does the user authentication for the control side while ctrscrut provides the home directories. Yellowpages is gone, the new paradigm is LDAP (local directory access protocol). If you can't log on to computers on the control network, then make sure both ctscrut (michelson) and gps computers are running. If the login screen screen on ctrscrut says "Ubuntu Administrator" and has no user accounts, then check to make sure gps computer is on. If you restart gps, then you might have to reboot ctrscrut to get the user accounts. If ctrscrut was rebooted, you might have to reboot other control network computers if you can't log on.
 +
 +If gps computer does not start, check the power distribution unit (PDU) at the top of that rack. If any units show both " " and "-" then you can use the arrow buttons to move through the units and cycle the units to on " ". That should automatically start the computer connected to those units.
  
 ===== General Problems ===== ===== General Problems =====
Line 830: Line 377:
 ==== GUIs are frozen ==== ==== GUIs are frozen ====
  
-Check to see if Sockman is working. Click the [LIST] button on sockman. If the list doesn't come up, Sockman may be hung up. Follow the Sockman restart procedure in the text file on the desktop if it won't respond. \\ +Check to see if Sockman is working. Click the [LIST] button on sockman. If the list doesn't come up, Sockman may be hung up. Follow the Sockman restart procedure in the text file on the desktop if it won't respond. 
-Hit [REOPEN] on the gui to see if it reconnects with its server. if that does not fix it, the server may be dead. From the LIST on Sockman, select the server suspected of being frozen. Ping it and if it reports back as being dead, see [[:chara:trouble_shooting#server_is_frozen|Server is frozen]] section below. \\+ 
 +Hit [REOPEN] on the gui to see if it reconnects with its server. if that does not fix it, the server may be dead. From the LIST on Sockman, select the server suspected of being frozen. Ping it and if it reports back as being dead, see [[:chara:trouble_shooting#server_is_frozen|Server is frozen]] section below. 
 Telescope guis often freeze. Hit reopen on the gui to bring it back to life. Hitting [MOVE] on the telescope gui before the star has stopped moving can cause the gui to freeze. Avoid doing this. Close the dead one and open a new one to fix the problem. Telescope guis often freeze. Hit reopen on the gui to bring it back to life. Hitting [MOVE] on the telescope gui before the star has stopped moving can cause the gui to freeze. Avoid doing this. Close the dead one and open a new one to fix the problem.
  
Line 837: Line 386:
  
   * If the top of a server goes blank, try typing "sb" (start background) on the server command line.   * If the top of a server goes blank, try typing "sb" (start background) on the server command line.
-  * If the server screen fills with jibberish, try hitting CTRL-in the server to clear it.+  * If the server screen fills with jibberish, try hitting CTRL-in the server to clear it. 
 +  * If a server freezes, sometimes hitting CTRL-C and then N, for NO, will unfreeze it.
  
 ==== Server is frozen ==== ==== Server is frozen ====
Line 847: Line 397:
   * Click [REOPEN] on Cosmic Debris and relevant GUIs to re-initialize communication with the server after it is restarted.   * Click [REOPEN] on Cosmic Debris and relevant GUIs to re-initialize communication with the server after it is restarted.
   * A folder is on the desktop that has the restart commands for CTRSCRUT servers and the Shutters server restart command running on OPLE. Use it to restart servers that will not reopen from the menu.   * A folder is on the desktop that has the restart commands for CTRSCRUT servers and the Shutters server restart command running on OPLE. Use it to restart servers that will not reopen from the menu.
 +  * if a telescope server does not close after using CTRL-C when restarting them after UT midnight and the Send ON button is green on the obsgtk, turn the Send ON to OFF and the server will close. Turn the Send ON button back on to resume the setup for the night.
  
 ==== PAVO Server - Error communicating with IFW ==== ==== PAVO Server - Error communicating with IFW ====
  
-The following error message sometimes appears when starting the PAVO server: \\ +The following error message sometimes appears when starting the PAVO server: "Error communicating with IFW. Could not read from IFW. Is the filter wheel plugged in and on?" Most of the time, restarting the PAVO server will clear this error message and allow PAVO to communicate with the filter wheel. If restarting the server doesn't help, then check that the cables going into the small, black IFW box on the PAVO table and into the back of the PAVO computer are plugged in securely.
-"Error communicating with IFW. Could not read from IFW. Is the filter wheel plugged in and on?" \\ +
-Most of the time, restarting the PAVO server will clear this error message and allow PAVO to communicate with the filter wheel. If restarting the server doesn't help, then check that the cables going into the small, black IFW box on the PAVO table and into the back of the PAVO computer are plugged in securely.+
  
 ==== Adding or Finding a Star in the CHARA Database ==== ==== Adding or Finding a Star in the CHARA Database ====
Line 858: Line 407:
 The command "dbadd" can be used to look up the CHARA number for a star or to add a star to the CHARA database. To use dbadd, open a terminal window, and type "dbadd starname" where starname is the common name (e.g., Vega) or identifier of the star (e.g., IRC, HR, HD, SAO, FK5, HIP, GJ, or 2MASS designation). If the object is in the database, then it will return the CHARA number. The command "dbadd" can be used to look up the CHARA number for a star or to add a star to the CHARA database. To use dbadd, open a terminal window, and type "dbadd starname" where starname is the common name (e.g., Vega) or identifier of the star (e.g., IRC, HR, HD, SAO, FK5, HIP, GJ, or 2MASS designation). If the object is in the database, then it will return the CHARA number.
  
-You can also look up different identifiers for the star by using the SIMBAD database: [[http://simbad.u-strasbg.fr/simbad/sim-fbasic|http://simbad.u-strasbg.fr/simbad/sim-fbasic]] \\  \\ +You can also look up different identifiers for the star by using the SIMBAD database: [[http://simbad.u-strasbg.fr/simbad/sim-fbasic|http://simbad.u-strasbg.fr/simbad/sim-fbasic]] 
-DBADD uses SIMBAD to look up the coordinates of the star and then searches to see if the star is already in the CHARA database. If the star is in the database, it will return the CHARA number. If the star is not in the CHARA database, it will ask if you want to add it and assign a CHARA number if you answer yes.+ 
 +DBADD uses SIMBAD to look up the coordinates of the star and then searches to see if the star is already in the CHARA database. If the star is in the database, it will return the CHARA number. If the star is not in the CHARA database, it will ask if you want to add it and will assign a CHARA number if you answer yes.
  
 If an object is not in SIMBAD (such as a recently discovered nova), then you must enter the object manually using the instructions below. If an object is not in SIMBAD (such as a recently discovered nova), then you must enter the object manually using the instructions below.
  
- \\ __Finding a star in the CHARA database__+**Finding a star in the CHARA database**
  
 Here is an example of how to find the CHARA number of a star, nova, or AGN that is already in the CHARA database: Here is an example of how to find the CHARA number of a star, nova, or AGN that is already in the CHARA database:
Line 872: Line 422:
 Note: If the target information stored in the CHARA database does not match SIMBAD, then dbadd will provide the coordinates and magnitudes listed in SIMBAD, and it will also list the CHARA numbers of objects in the database located within 10 arcsec. Please check these nearby entries before adding a new star to the database. Note: If the target information stored in the CHARA database does not match SIMBAD, then dbadd will provide the coordinates and magnitudes listed in SIMBAD, and it will also list the CHARA numbers of objects in the database located within 10 arcsec. Please check these nearby entries before adding a new star to the database.
  
- \\ __Targets that are not in the CHARA database__ \\  \\+**Targets that are not in the CHARA database** 
 If an object is not in the CHARA database, then you can use dbadd to query SIMBAD and add the star to the CHARA database. This might need to be done for faint stars, novae, and AGN. If the object is found in SIMBAD, then dbadd will return the target's coordinates, magnitudes, and spectral type and ask if this it the object for which you are looking. If you answer yes, then dbadd will issue a CHARA number and add the object to the database. Please check the target coordinates to make sure that they are correct before adding a new star to the database. If an object is not in the CHARA database, then you can use dbadd to query SIMBAD and add the star to the CHARA database. This might need to be done for faint stars, novae, and AGN. If the object is found in SIMBAD, then dbadd will return the target's coordinates, magnitudes, and spectral type and ask if this it the object for which you are looking. If you answer yes, then dbadd will issue a CHARA number and add the object to the database. Please check the target coordinates to make sure that they are correct before adding a new star to the database.
  
 Here is an example of how to add an object to the CHARA database: Here is an example of how to add an object to the CHARA database:
  
-[ctrscrut:534] dbadd BD 41 3807 \\+[ctrscrut:534] dbadd BD 41 3807 
 SIMBAD found this for "BD 41 3807": SIMBAD found this for "BD 41 3807":
  
Line 889: Line 441:
 PMDEC -3.87 \\ PMDEC -3.87 \\
 PAR 0.58 \\ PAR 0.58 \\
-RVEL -27.50 \\+RVEL -27.50 
 VMAG 10.03 \\ VMAG 10.03 \\
 HMAG 6.23 \\ HMAG 6.23 \\
Line 901: Line 454:
 In the example above, the object "BD 41 3807" has been added to the database as CHARA number 320414. In the example above, the object "BD 41 3807" has been added to the database as CHARA number 320414.
  
- \\ __Targets that are not in SIMBAD__ \\  \\ +**Targets that are not in SIMBAD** 
-Some objects like recently discovered novae are not in SIMBAD. Entering dbadd <star designation> for a star not in the SIMBAD database returns the message "Simbad is unable to find an object matching <star designation>. Try using a different catalog designation, or use the "-m" switch to add the object manually." \\  \\ + 
-The command dbadd -m is used by entering the star name and coordinates in this format: \\  \\+Some objects like recently discovered novae are not in SIMBAD. Entering dbadd <star designation> for a star not in the SIMBAD database returns the message "Simbad is unable to find an object matching <star designation>. Try using a different catalog designation, or use the "-m" switch to add the object manually." 
 + 
 +The command dbadd -m is used by entering the star name and coordinates in this format: 
 Usage: dbadd -m <name> <RA> <Dec> \\  \\ Usage: dbadd -m <name> <RA> <Dec> \\  \\
 <name>: Object ID (no spaces) \\  \\ <name>: Object ID (no spaces) \\  \\
 <RA> : XXhXXmXX.X or XXhXX.X (no spaces) \\  \\ <RA> : XXhXXmXX.X or XXhXX.X (no spaces) \\  \\
 <Dec> : XXdXXmXX.X or XXdXX.X (no spaces) \\  \\ <Dec> : XXdXXmXX.X or XXdXX.X (no spaces) \\  \\
-There are cases where an object is not in SIMBAD and doesn't return a result in dbadd, but is in fact in the CHARA database. Novae and AGN's are the likely objects that cause this result. At times, objects have multiple entries and several CHARA numbers since the names can be so unique. A system will be developed to find these entries without knowing or having a conventional database designation. \\  \\ __Binary stars__ \\  \\+There are cases where an object is not in SIMBAD and doesn't return a result in dbadd, but is in fact in the CHARA database. Novae and AGN's are the likely objects that cause this result. At times, objects have multiple entries and several CHARA numbers since the names can be so unique. A system will be developed to find these entries without knowing or having a conventional database designation. 
 + 
 +**Binary stars** 
 Some binary stars have a common HD number with an A or B after them. These can cause problems as Cosmic Debris does not accept non numerical entries when filling in star designation. These stars are likely in CHARA's database but need to be searched in dbadd or on SIMBAD to get the CHARA number or another designation. Some binary stars have a common HD number with an A or B after them. These can cause problems as Cosmic Debris does not accept non numerical entries when filling in star designation. These stars are likely in CHARA's database but need to be searched in dbadd or on SIMBAD to get the CHARA number or another designation.
  
-When a recognized identifier is entered, the HD number with A or B will usually display. Confirm that the coordinates, magnitudes and spectral type match the star desired. If they do not match, you may have the wrong star or the database may have the wrong info and the baseline solution will be affected. Offsets based on incorrect coordinates or misidentifications can move the fringes many thousands of microns away from the calculated position. This can happen when observing a dim companion (B) using the brighter (A) companion's coordinates. Inform the observers that noting and using the CHARA number will save time next time the target is observed. \\  \\ __Editing the database__ \\  \\ +When a recognized identifier is entered, the HD number with A or B will usually display. Confirm that the coordinates, magnitudes and spectral type match the star desired. If they do not match, you may have the wrong star or the database may have the wrong info and the baseline solution will be affected. Offsets based on incorrect coordinates or misidentifications can move the fringes many thousands of microns away from the calculated position. This can happen when observing a dim companion (B) using the brighter (A) companion's coordinates. Inform the observers that saving and using the CHARA number will save time the next time the target is observed. 
-If you find a mistake in the database, please send an email to Nils to have it corrected. Identify what you believe to be the error and what is the correct information.+ 
 +**Editing the database:**  If you find a mistake in the database, please send an email to Nils to have it corrected. Identify what you believe to be the error and what is the correct information.
  
 ==== Adding an anchor to the wiki page ==== ==== Adding an anchor to the wiki page ====
  
 To add an anchor to the wikipage, type the following where you want the anchor inserted (without the spaces between the symbols): \\ To add an anchor to the wikipage, type the following where you want the anchor inserted (without the spaces between the symbols): \\
-[ [ #AnchorName ] ] \\  \\+[ [ #AnchorName ] ] 
 Highlight the text you want to be a link to the anchor, click the link symbol above and select anchor. Select a page(such as Trouble Shooting) where the anchor is located and give the same name for the anchor. Highlight the text you want to be a link to the anchor, click the link symbol above and select anchor. Select a page(such as Trouble Shooting) where the anchor is located and give the same name for the anchor.
  
Line 924: Line 485:
 Follow instructions in the [[:chara:mircx_monitors|linked page to display the MIRC-X wolverine2 monitors]] in the control room. Follow instructions in the [[:chara:mircx_monitors|linked page to display the MIRC-X wolverine2 monitors]] in the control room.
  
- \\ +===== The new OPLE system =====
-Last updated 2017-04-12+
  
 +With the implementation of the new ople system which replaced the VME in Fall of 2021, new troubleshooting issues have arisen. Since there are now 6 new ople computers to run the carts for each scope individually, many issues will be specific to one computer or telescope and not to the system as a whole. Restarting the VME is now a thing of the past.
  
-==== I can't change the camera settings on the TV ====+The traditional ople server will still be used to communicate with each new ople computer, identified as OPLE 1 to OPLE 6. When the communications are good, each active cart will be displayed in the ople server or ople gui status tab as before. At times, an ople computer can lose communications or a server can crash and the server or comms needs to be restarted.
  
- \\ +If a cart cannot be startedstopped or otherwise commandedlook to the OPLESystem gui to see if a green indicator has turned redA message will also often pop up on the ople gui saying the command could not be sentThis may require a simple start command to restart the server or reboot of the computer to get it back to yellow and then a start to get it back to greenDo either of these steps with a right click of the red button and then select start or rebootAfter a reboot, select start to load the servers after the indicator has turned yellowWhen this is done, the ople server will need to be connected to the newly restarted server. Type "oo" into the ople server to open ople comms with the new ople serverIt should now reappear on the ople server display and say System Ready to indicate comms are restored.
-The HUT servers control functions such as finder and acq exposure times and gainsheater and dehumidifier usageand various AO functions. An observer may find that the camera settings do not display or are not adjustable. The HUT server may be the cause if it has quitTo see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the camera settings are displayed and the settings can be changed from the gui, then the HUT server is okRestart the telescope server to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well. If the gui is not functioning, the HUT server will need to be restarted. The HUT servers run on the computer at each telescope. Find the PID from Sockman for the server you want to shut down. Open terminal and log into the correct telescope computer (for example, type "s1" to log on to S1). Use the command "ps aux | grep HUT" to find the process you need to shut downThe number should match the PID from Sockman. Kill the process by typing "kill -9 pid" where pid is the process number and remove the entry from Sockman with the REMOVE button on the gui or by command lineRestart the hut server using the new "bootlaunch_hut" command described in the section on [[:chara:trouble_shooting#restarting_servers|Restarting Servers]]. If the server won't restart, a reboot of the power supply in the telescope bunker might be necessaryThe power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunker. The power supply has green readouts of volts and current. After turning the power off for 10 seconds and back ontry restarting the server from the computer in the bunker to see if it starts cleanly. If so, then restart the telescope server, reopen the connection to the telescope gui, and hit REOPEN on Cosmic Debris. Part of the HUT server also controls the AO tableIf the AOB part of the HUT server doesn't work, then the power supply on the back of the AO table in the telescope dome might need recycling. This power supply controls the actuators at M2 and the AO table. The power supply box is a 6×9 inch aluminum box on the back of the AO table, behind the keyboard and monitor. Turn it off with the power button on the bottom edge of the box, wait 5 seconds and turn it back onThe HUT server should now restart cleanly. Restart the telescope server as well to make the connections to the telescope gui. Hit REOPEN in Cosmic Debris if you are observing to make all needed connections.+
  
-===== Tiptilt Server =====+Some times a cart is stopped and cannot be commanded. If the cart has gone to the front hard or back hard switch, it will not be usable until it is moved from the switch and the Ople Controller box is reset. There are 6 silver boxes for these controllers with two green LED's for the front and back switches and two red LED's for the back hard and front hard switches. If a red LED is lit, there will be an error displayed on the message window and the cart is disabled. The cart will need to be moved off the switch and then the box can be reset with the RESET button on the front. The error display will go away and the red LED will be off. The cart is now controllable.
  
- \\ +===== Restarting Servers =====
-The tiptilt server controls the CCD based tiptilt detection system. \\  \\ +
-Before you start the tiptilt server, you must ensure that the power to the cooling system and the CCD iteslf is on. It is extremely important that the cooler be running before you turn on the CCD and is only turned off if you are sure the CCD is NOT running. You can start the server from the X windows menu or with the command xtiptilt. \\  \\ +
-Note that there are background counts and read noise to deal with. Whenever you change the frame rate, please ensure that the bias frame is OK. The server will attempt to load an old bias frame that should work, but if things are not working, try making a new bias frame by ensuring that the detector is in the dark and typing "mkbias" into the tiptilt server. \\  \\ +
-In the tiptilt GUI windows, the white dots represent the starlight while the green dots represent the motion applied to telescope's secondary mirror to keep the starlight centered. When tiptilt is locked the white dots will be brought to the center of the tiptilt window. The green dots should be mostly centered also. W2 and E2 telescopes have a small oscillation that show as back and forth plots of the green dots.+
  
-==== Tiptilt server complains about the CCD ====+==== Restarting Servers using the bootlaunch paradigm ====
  
- \\ +If a server is not running or Socket Manager reports that a server is deadthen look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]])You can also look at the up-to-date file by opening a terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list".
-Is the CCD turned on? When the tiptilt server starts up it tries no more than five times to communicate with the CCD. If they all fail, it will give up. If this happens, try cycling the power to the CCD and try again. If this fails, connect to the tiptilt machine and type the command rtccdAPIDemo, which should return with no errors. Try this command few times, but if it still fails, there is a more serious problem. Turn off the CCD and reboot the tiptilt computer. If it still fails, I am afraid you are in more serious trouble. \\  \\ +
-Note that it is never good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptiltother than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command lineIf this says the clock is working do not reboot the machine\\  \\ +
-Also note that cycling the power on the CCD can cause harm so be sure you need to do it before trying itAlso, it is important to wait for at least 20 seconds after turning off the power before turning it on again.+
  
-==== Tiptilt doesn'seem to be talking to the telescopes ====+To restart a server, log on to the machine that runs the server and type "bootlaunch_master". This script will go through the list of executables and will check which servers are running. If a server isn'running it bootlaunch_master will remove it from socket manager, clear the lock file, and relaunch the server. The instructions below describe how to restart individual servers, but this should not be necessary anymore.\\ 
 +\\ 
 +A number of servers use an interim bootlaunch paradigm to restart. This is confined to servers that run on ubuntu machines, namely the telescope bunker computers and gps. The basic syntax is "bootlaunch_<server>" where "<server>" is replaced by the server the script is designed to address. The scripts have a number of safeties built in, so it is safe to run them even if a server is already running – they just output the process ID of the running server. The scripts also take care of the entry in socket manager as well any serial port lock files. All the pertinent information is world writeable, so one should be able to run a bootlaunch script as observe.\\ 
 +\\ 
 +One thing of note about the output of the bootlaunch scripts, they call a number of other programs which themselves have output that may be misleading in the context of bootlaunch. Chief among these is the output of "tsockman". If a server stopped unexpectedly, it may leave behind an entry in the socket manager. In order to launch a new server, one needs to clean out the socket manager entry if it is there. To do that, "tsockman remove <entry>" is called to remove "<entry>" before the new server is launched. If there is no entry, tsockman will respond with "Process by that name does not exist". THIS IS NORMAL and is not indicative of an error. The server in question launched (without fanfare) right after that output text.\\ 
 +\\ 
 +Note: The bootlaunch scripts will not start a new server if there is an existing process running. Therefore, type "ps aux | grep //server_name//" where server_name is the name of the server. If there is a dead process, look up the process identification number (PID) and type "kill -9 PID" to kill the process and then run the relevant bootlaunch script. 
 +==== Restarting Servers using the rc.local file (likely outdated) ====
  
- \\ +\\ 
-Sometimes the telescope server will not show that TT is runningIt will show 0Hz for signal rate for TTRunning TIPTILT COMM will not get it started while other scopes do show it starting. Close and restart any telescope servers that won'connect after two tries of TIPTILT COMM. \\ +This procedure is applicable to servers that have not switched over to the bootlaunch paradigm, which may not be any at this point.\\ 
-NoteThere is more info in the software manual on this topic, but I wasn't sure if it was still relevant.+\\ 
 +If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[:chara:socket_manager_list_file|socket_manager.list]]). You can also look at the up-to-date file by opening terminal window and typing "less /ctrscrut/chara/etc/socket_manager/socket_manager.list" Note that servers can be running fine, but if the Socket Manager drops the connection to them, they are as good as dead when it comes to functioning with other servers or as part of a larger sequence.\\ 
 +\\ 
 +Log on to the relevant computer by typing the computer name (ctrscrut, ople, s1, …). If the shortcut doesn'work then type "ssh //name//" where name is the computer name.\\ 
 +\\ 
 +Find out if the server is running by typing "ps aux | grep //server_name//" where server_name is the name of the server.\\ 
 +[ctrscrut:599] ps aux | grep pico_1\\ 
 +observe 9281 0.0 0.0 61156 692 pts/3 S 13:58 0:00 grep pico_1\\ 
 +observe 12578 0.0 0.0 24524 11212 ? S Jun16 33:14 /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg\\ 
 +\\ 
 +If the entry for the dead server shows up in the process list, then identify the process identification number (12578 for the example above) and kill the server by typing "kill -9 //PID//" where PID is the process identification number.\\ 
 +\\ 
 +Look up the commands to restart the server by typing "more /etc/rc.local" (this is relevant for servers that run in the background). Press the space bar to scroll through the contents of the rc.local file. Locate the commands relevant for the server that needs to be restarted and copy and paste into a terminal window:\\ 
 +\\ 
 +#Start PICO server for PICO #1\\ 
 +/bin/rm -f /var/lock/LCK..ttyC8\\ 
 +/usr/local/bin/tsockman remove PICO_1\\ 
 +/usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg &\\ 
 +\\ 
 +The first command removes the lock to allow the server to restart. The second command removes the name from the socket manager listing. The last command restarts the server. Note that if you are restarting the servers as observe, you will need to remove the part of the command in the rc.local file that saves information in /var/log///server_nam//e.log file (the actual command typed should resemble the last line above).\\ 
 +\\ 
 +There are text files on the desktop with many of the restart commands. Use these files for quick access to the relevant commands. The commands are edited and can be copied exactly as written. Files include Dome servers and all servers running on ctrscrut. Many of these commands are also located on the [[:chara:restarting_servers|Restarting Servers]] page.
  
-==== Tiptilt server says the clock isn't running ====+Shutters Server
  
- \\ +\\ 
-First check whether the clock itself is running and the other machines receive the clock signalLook at the clock cards at the back of other computers in the rackThe clock cards have three LEDs, one yellow and two greensIf the computer is receiving the clock signal properly all three LEDs should blink, but at a different rate. If the LEDs on all the clock cards are solid then reboot the GPS computer. When the GPS computer is downit is best to cycle the power also on the box right above the GPS computer. \\  \\ +The Shutters server can become unresponsive or disconnected from the Socket ManagerThis server must be restarted from the lab and not from the Control RoomFollow these instructions to restart itNote that Shutters runs on gpsnot ctrscrut.\\ 
-If the clock appears to be working properly on other machines and not on the tiptilt now it is time to reboot tiptilt. \\  \\ +\\ 
-[There is a bug in the real time part of the CCD code. It is caused by the clock in the tiptilt system either not running at all or having been set to a time very different from the last time the CCD ran.] \\  \\ +To start the shutter server on gps:\\ 
-For the time being the only solution is to reboot tiptilt, but do so from the labPower OFF the CCD, then reboot the tiptilt machine and go into the BIOSMake sure that interrupt 11 has been set to ISA legacy, save the BIOS and reboot. When the clock card LEDs in the tiptilt machine indicate proper clock signal, turn the CCD back on and start the tiptilt server\\  \\ +\\ 
-Also, sometimes Serial Port 3 grabs IRQ 11 which stops the clock from running. Since there is no serial port 2 it's safe to disable this in the BIOSThis problem normally comes up when there has been a power outage. \\  \\ +Log into the gps computer and kill the process labeled shutters with the PID as described in **Restarting Servers** above.\\ 
-Sometimes syncing the clock can also cause this problem, but that should be fixed soon. If it does, exit the tiptilt server, log in as root, and reload the tiptilt model using the following commands: \\ +Turn off the power to the Shutters with the switch on the computer rack which is marked "SHUTTERS"Restart the Shutters server with the commands belowAfter restarting the server and testing the gui to see that it works, turn the SHUTTERS power back on with the switchThere is a printed sheet of directions in the lab to help youIt will refer to the ople computer, but it now runs on the GPS computer.\\ 
-/sbin/rmmod tiptilt_rt \\ +\\ 
-/sgin/insmod /usr/local/modules/tiptilt_rt.o \\  \\ +/usr/local/bin/tsockman rm shutters\\ 
-Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine.+bootlaunch_master
  
-==== Tiptilt is not locking on a star or locks, but lets the star drift away ====+==== Restarting Socket Manager ====
  
-  * Are all the mirror covers open? [Note: W1 M7 cover sometimes needs two clicks to opendespite Scope-monitor indicates open.] +In very rare circumstances, the socket manager might need to be restartedThis is usually needed if GUIs stop communicating (e.g., OPLE GUI stops communicating with OPLE server)Another example occurs when mircx_bootLauncher tries to remove dead server names from the socket_manager before restarting. If the socket_manager does not respond, the mircx_bootLauncher will timeout.
-  * Has TIPTILT COMM been run from Cosmic Debris? +
-  * Check the ACQ alignment to make sure the tick marks are centered on the laser. +
-  * If there is plenty of starlight getting into tiptilt, then try re-initializing by clicking [TIPTILT COMM] on Cosmic Debris. +
-  * If there are a large number of background counts on TTthen close the M5 cover on the telescope, and click [DBIAS] on the TIPLTILT GUI to clear the background counts. Re-open the M5 cover. +
-  * If a star drifts even with TT locked, there could be a bright sky or light from other beams getting into the affected telescope's beam. This will show in the TT server as much lower counts than the other locked stars, but still high enough to lock TTUse the laser to find the correct position to lock TT. +
-  * If TT unlocks and the star drifts, the TT servo may not have engagedThis is engaged with the TIPTILT button on the telescope gui. Clicking it may not start the servo the first time. Try it again if the green dots drift on the TT plot windows or if the Servo status in the telescope gui reads None instead of Wobb 1. +
-  * Are the TIPTILT buttons turned on from the POWER GUI?+
  
-==== Tiptilt servo oscillates ====+Follow these instructions to restart the socket manager, but only do so if really necessary:
  
-You will see the oscillation in the green dots of the tiptilt GUI windows. Sometimes you can also see the oscillation in the white starlight dots or as an elongation of the star when looking at the ACQ field while tiptilt is locked. Some scopes have an oscillation that has not yet been diagnosed. W2 is one that usually oscillates. A diagonal motion in the tiptilt box indicates an oscillation in one axis only, while a vertical/horizontal motion indicates an oscillation in both directions. Motion from the upper right to lower left corresponds to elevation axis while motion from the upper left to lower right corresponds to the azimuth axis. (I think you can check direction by typing sin into telescope server to send sine waves to the telescope.) There are a few ways you can try to correct the oscillation manually tuning the servo:+ssh -X observe@ctrscrut
  
-  * Type "tune" into the tiptilt server. Select the appropriate telescope. Turning the gain down normally helps. +killall socket_manager
-  * Type "tune" into the appropriate telescope server. The default value for the proportional term is -0.5 and differential term is 0.0. Adjust these values between -0.2 to -1.0 for proportional and 0.0 to 0.2 for differential. +
-  * Read section on adjusting the Telescope Tracking Gain using the Dome Server GUI (ref).+
  
-==== Tiptilt is saturating ====+socket_manager &
  
-  * Tiptilt saturates at ~ 200,000 counts. If you are near this limit, you can reduce the TT exposure time to lower the number of counts. +==== MIRC-X CredoneImAcq Server ====
-  * Set the TT exposure time on Cosmic Debris in the box for "Tiptilt (mS)". This will set the TT exposure time when slewing to a new target. +
-  * To change the exposure when already at a target, then click the [EXP] button on the TipTilt GUI. This will bring up a dialog box where you can enter the new exposure time in msec. Check to make sure the tiptilt frequency (in Hz) changes on the tiptilt server after changing the exposure time. If the frame rate doesn't change, then set the exposure time back to the old value and try entering the new value again. You might have to do this a few times to actually get the frame rate to change. +
-  * Frame rates for given exposure times: +
-      * ExpTime 5 msec, Frame Rate 157 Hz +
-      * ExpTime 2 msec, Frame Rate 299 Hz +
-      * ExpTime 1 msec, Frame Rate 427 Hz+
  
-==== Tiptilt counts are way too low ====+The MIRC-X CredoneImAcq server and GUI can be opened using the "credone" icon on the wolverine desktop. Note that you can only have one of these open at a time. If you get an error message and can't bring up credone, then close the error message and follow the steps below to remove the dead processes first:
  
-Try using slower frame rate or increasing the NSUM. Also ensure that the acquistion is properly aligned with the laser. To change the frame rateclick the [EXP] button on the tiptilt GUI and enter a longer integration time. Remember to change "Tiptilt (mS)" on Cosmic Debris to keep the same exposure time when slewing to the next target.+From terminal windowlog on to mircx (old observe password):\\ 
 +ssh -X spooler@mircx
  
-==== Tiptilt counts are negative ====+Look up credone process:\\ 
 +ps aux | grep credone
  
-The bias frame is badGet a new one or turn it off.+Kill all credone processes (including the ones for the error messages) using "kill -9 PID" where PID is the process numberNote that credone can't be restarted if there are any error processes in the list. Check "ps aux | grep credone" one more time to make sure all processes are cleared (the only one that should show up is the grep command that was just issued).
  
-===== OPLE and Metrology =====+Then restart credone using the desktop icon or by issuing the following command on spooler@mircx:\\ 
 +credoneImAcq –no-display
  
-==== OPLE Server doesn't open complains about not finding run_ople ====+(the first "–" should be two dashes: and -).
  
-This usually happens if someone runs a "make clean" in the cvs tree but doesn't follow up with a "make install" on ople. To generate the run_ople file, follow these steps:+===== Telescopes and Dome Servers =====
  
-  * Open a terminal window and type "ople" to log onto the ople machine. +==== The Telescope won't move or stopped moving ====
-  * Change to the appropriate directory under the CVS tree by typing "cd control/cliserv/ople" +
-  * Update the CVS tree by typing "cvs update" +
-  * Run the Makefile script by typing "make" +
-  * Log on as root by typing "su" and entering password. +
-  * Type "make install" +
-  * This will install run_ople. Try opening the ople server again.+
  
-==== Metrology and Homing carts ====+Have the powers been turned on to the drives? Are the scopes disabled? The usual state of the telescopes is disabled until enabled. This is due to the stall function of the scopes which eventually disables the scopes when they are stowed. Enable the scopes by hitting [ENABLE] on the dome gui, the telescope gui control tab, or the bottom of the obsgtk. If a scope disables itself during a slew, it may be just an overcautious stall function or the slew gains may be too high or low. Check the gains and if they are OK, Hit ENABLE and the scope should continue to slew. If it disables again without moving, there may be something wrong at the scope, ie. a hatch left open, a ladder not put away, a tool on the floor, or something else physically impeding the motion of the scope. You will need to go to the dome to see what it is. The computer in the dome will give you control of the scope to turn it away from the problem. Make sure the scope is in a position that allows the opening of the hatch if you need to go upstairs. It can be moved back from the bunker as needed.
  
-  * Good metrology signals are important to the proper positioning of the cartsMonitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are usingThey should show white sine waves that are around the height of the window. Erraticfluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too lowRed waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal backDo not adjust the MET1 mirror+Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopesCheck them for current times and continuous updates of numbersIf they are not updatingtry to REOPEN them firstIf that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of itSee **Dome Server Restart** below.\\ 
-  To home the cartsturn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switchIf the cart has no issueit will arrive at the target position of 0m and the home switch at the same timeThe X in the OPLE server under the HM will indicate it has homed to the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringesYou can also find the size of the error by typing the command "homechk S1" into the ople serverHit the [HOME] button and it should move forward and trigger the home switchHit [TRACK] to home it to the new home position.+\\ 
 +**Azimuth Limit Switches** \\ 
 +\\ 
 +<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>As of 11-'17, the azimuth limit switches are enabled and can stop the motion of the scopes if they try to go beyond -90º or 450º. The scopes will not be movable with normal inputs so follow these instructions to return them from the out of range condition.\\ 
 +\\ 
 +1. On the domegui MANUAL tabclick STOP so pulses won't be sent to the drive by the control software.\\ 
 +\\ 
 +2. Make sure you understand why the limit was hit which may require a trip to the telescope. If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch incorrectly causing the stall and not that the scope is actually in a wrong position.</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way.\\ 
 +\\ 
 +<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope.\\ 
 +\\ 
 +4. Click ENABLE then you can move the telescope back to its normal range of operation.\\ 
 +\\ 
 +5. After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again and then hit AUTO on the AUTO tab to resume normal operation.</font>
  
-===== Beam Samplers =====+<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>However, If you notice that a telescope will stop near AZ 90 or 270 with the scope still being ENABLED and refusing to move, this is often due to the new AZ limit switch being armed around AZ 0º at some point earlier. To get it moving again:</font>
  
-==== Beam Samplers are not moving ====+<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software.\\ 
 +\\ 
 +2.</font><font inherit/inherit;;initial;;white>M</font>ake sure all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way.\\ 
 +\\ 
 +<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope.</font>
  
-This happens most frequently with the E1 beam samplerIf you are doing alignments and are using E1try to move it forward first to see if it diesIf it does, follow this procedure.+<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>4Move the scope a bit back toward the direction it was coming from – for example, if the scope stopped at AZ 268 while rotating clockwise, move it back to 265 or so using AZ DECThen press STOP.</font>
  
-  * Turn off power on Newport Universal Motion Controller/Driver for the Beam Samplers (on top of rack in computer arealabelled "BS"+<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>5Move the scope past AZ 270 by pressing AZ INCUsally 2-4 degrees beyond will doGo past the point that shows the limit is triggered and click STOP.</font>
-  * Then go behind the racks and locate the serial port for BSLook for all of the blue ethernet cables coming outBS is labelled in prt 2. Remove the ethernet cable for BS. The step stool is available to make it easier to reach the blue ethernet cable at the back of the BS controller box. +
-  * Restart BS controller (press power button again). Wait for all axes to appear on screen. +
-  * Plug the ethernet cable back in. +
-  * Restart BS server on ctrscrut using commands in /etc/rc.local (see instructions below) +
-  * Reopen Beam Sampler GUI. You will need to home each Beam Sampler one by oneYou should be able to do this from the regular Beam Sampler GUI. If that doesn't work, try opening "espgtk BS" from the command line.+
  
-==== Restarting the Beam Sampler Server ====+<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>6. Click OVERRIDE OFF, go to the AUTO tab and press AUTO, then NEXT (also in the obsgtk; this will restore the original star's demand position to the scope).</font>
  
-The most common cause for getting multiple copies of a server is using the socket manage RESTART buttonwhich is flaky at bestDon't do this. If you need to restart a server you should do it manually. \\  \\ +<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90The LED will show red when on the limit switch and is tripped, ielimiting motion of the scopeThe LED will be yellow if it has tripped and is in the caution range, but not on a limit switchA fine Allen key can be used to push the internal reset button or use the magnet and touch the boxIt will turn the LED green when restored.</font>
-1. Make sure you are logged into the right machine: \\  \\ +
-ssh ctrscrut \\  \\ +
-2. If you are not sure see if the socket manager will tell you. If it doesn't have a look in the file: \\  \\ +
-/ctrscrut/chara/etc/sock_manager/socket_manager.list \\  \\ +
-3See if there are any ghosts running \\  \\ +
-ps aux | grep esp_server \\  \\ +
-Yes, it's hard to know which one it is with the esp serversYou can work out which ones are ghosts by typing the command \\  \\ +
-tsockman |grep ctrscrut \\  \\ +
-which will give you a list of the servers running on the machine you are interested inChecking for non-matching PIDs will tell you which processes you need to stop. \\  \\ +
-4. Stop those processes: \\  \\ +
-kill -9 PID1 PID2 …. \\  \\ +
-5. Make sure there is no sign of it in the socket manager \\  \\ +
-tsockman rm BS \\  \\ +
-5Restart the beam sampler servers (this starts both beam sampler servers BS1 and BS2): \\  \\ +
-bootlaunch_beamsamp+
  
-==== "Failed to request position of S1" ====+==== The Telescope won't track ====
  
- \\ +If a bright star is manually found and not slewed to by number, (usually after the scope has gotten lost), the telescope will not track as it does not know it has gone to a star. When you init the EL and AZ position of the scope to the star's position, you must enable the tracking by going to the dome gui, selecting the RA/DEC tab and hitting the [GO RA/DEC] button. The scope will track, but the pointing model may still be offTry slewing to the same star and initing on it after it is locked in TiptiltThis can be done in the dome with the computer at the telescope.
-Cosmic Debris reports "Failed to request position of S1" when trying to set the beam order. This indicates that ople is no longer talking to the beam samplers. If the beam sampler server has been restarted recentlythen the ople server will also need to be restarted to re-establish the connection\\  \\ +
-The beam sample server runs on ctrscrutThe ople server tries to open a connection to the beam sampler when it startsIf the beam sampler is not there, or dies, \\ +
-you need to restart the ople server as there is no command to reopen that connection.+
  
-===== Remote Observing =====+==== The Telescope won't track ====
  
-==== VPN connection is not working ====+If a bright star is manually found and not slewed to by number, (usually after the scope has gotten lost), the telescope will not track as it does not know it has gone to a star. When you init the EL and AZ position of the scope to the star's position, you must enable the tracking by going to the dome gui, selecting the RA/DEC tab and hitting the [GO RA/DEC] button. The scope will track, but the pointing model may still be off. Try slewing to the same star and initing on it after it is locked in Tiptilt. This can be done in the dome with the computer at the telescope.
  
-To re-establish the VPN connection, follow these steps:+==== The Telescope won't stop moving or moves beyond the commanded position ====
  
-  * Open VPN status page from the firefox web broswer. To do this, click on the "AROC VPN reconnect" icon on the zoot desktopAlternatively, you can open the firefox web browser and go to 192.168.3.1/vpn_sta.htm +Disable the scope whenever its motion is outside of what it is supposed to do. Turn the powers off to the axes as well on the Power guiThe dome server is usually to blame and will need to be restarted.
-  * The browser will prompt you for the admin user name and password. If these entries are not already filled in, then look for this information on the papers near the console. +
-  * If the browser does not open, then check if there are other instances of firefox already running. To do this, open a terminal window, type "ps auxw | grep firefox", and kill the other instances of firefox so that you can open a new window. +
-  * When the VPN connection is working, it will report "Phase 1: M-ESTABLISHED / Phase 2: ESTABLISHED" under the state column. +
-  * If either Phase 1 or Phase 2 states say IDLE for a given VPN, then you need to re-establish the connection. To do this, click on the "DROP" button for that VPN. Then click on the "CONNECT" button and wait for the Phase 1 and Phase 2 connections to be established. +
-  * Make sure to close Firefox by using CTRL-Q to quit the program and not just close the window.+
  
-===== General Problems =====+Sometimes, the gains can be too low and the scope will overshoot the star position. Disable the scope, adjust the Slewing gain up one increment and enable the scope. It should return to the star.
  
-==== GUIs are frozen ====+==== Dome Server Restart ====
  
-Check to see if Sockman is working. Click the [LIST] button on sockmanIf the list doesn't come up, Sockman may be hung up. Follow the Sockman restart procedure in the text file on the desktop if it won't respond. \\ +Dome servers are now started using the bootlaunch_master commandThe manual process that has been superceded is archived.
-Hit [REOPEN] on the gui to see if it reconnects with its server. if that does not fix it, the server may be dead. From the LIST on Sockman, select the server suspected of being frozen. Ping it and if it reports back as being dead, see [[:chara:trouble_shooting#server_is_frozen|Server is frozen]] section below. \\ +
-Telescope guis often freeze. Hit reopen on the gui to bring it back to life. Hitting [MOVE] on the telescope gui before the star has stopped moving can cause the gui to freeze. Avoid doing this. Close the dead one and open a new one to fix the problem.+
  
-==== Server display is blank or filled with jibberish ====+To restart the dome server:
  
-  * If the top of a server goes blank, try typing "sb" (start background) on the server command line. +1. Make sure the power to the drives is OFFDisable the scopes.
-  * If the server screen fills with jibberish, try hitting CTRL-l in the server to clear it.+
  
-==== Server is frozen ====+2. Login to the relevant computer as observe. For example, type "s1" to log on to S1.
  
-Try quitting the server using CTRL-C and entering "Y"If that doesn't work, try the following:+3Work out the process ID number (PID) by typing bootlaunch_master
  
-  * On Sockman, click [LIST] and select the appropriate serverIt will give a PID number you can use to kill the server in a terminalOpen a terminal and follow the directions for Restarting Servers above. +4Use kill -2 PID to kill the server. Entering the command twice will show "No such process" if it has been killedif it is not killed, use kill -9.
-  * If a server window does not close, click the X in the upper right corner to close it, but only after getting the PID from Sockman. +
-  * Click [REOPEN] on Cosmic Debris and relevant GUIs to re-initialize communication with the server after it is restarted. +
-  * A folder is on the desktop that has the restart commands for CTRSCRUT servers and the Shutters server restart command running on OPLE. Use it to restart servers that will not reopen from the menu.+
  
-==== PAVO Server - Error communicating with IFW ====+5. Type bootlaunch_master again to restart the server.
  
-The following error message sometimes appears when starting the PAVO server: \\ +6Turn the power to the drives back on.
-"Error communicating with IFWCould not read from IFW. Is the filter wheel plugged in and on?" \\ +
-Most of the time, restarting the PAVO server will clear this error message and allow PAVO to communicate with the filter wheel. If restarting the server doesn't help, then check that the cables going into the small, black IFW box on the PAVO table and into the back of the PAVO computer are plugged in securely.+
  
-==== Adding or Finding a Star in the CHARA Database ====+7. Hit REOPEN and ENABLE on the domegtk, and type "otcs" in the telescope server.
  
-The command "dbadd" can be used to look up the CHARA number for a star or to add a star to the CHARA database. To use dbaddopen a terminal window, and type "dbadd starname" where starname is the common name (e.g., Vega) or identifier of the star (e.g., IRC, HR, HD, SAO, FK5, HIP, GJ, or 2MASS designation). If the object is in the database, then it will return the CHARA number.+You may have to reinitialize the scope on bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finderIf notyou may need to go out to the telescope to find the bright star to reacquire the scope's position.
  
-You can also look up different identifiers for the star by using the SIMBAD database[[http://simbad.u-strasbg.fr/simbad/sim-fbasic|http://simbad.u-strasbg.fr/simbad/sim-fbasic]] \\  \\ +Be aware that when restarting a dome server, the telescopes position may not be retained and the dome gui may display Az0.0. El0.0The gui will scroll messages that say Az0.0El: 0.0 with an error in scope position. If the scope is known to be at STOW positionyou can manually enter the scope's AZ and EL using the manual tab on the dome gui and hitting the INIT button.
-DBADD uses SIMBAD to look up the coordinates of the star and then searches to see if the star is already in the CHARA database. If the star is in the databaseit will return the CHARA number. If the star is not in the CHARA database, it will ask if you want to add it and assign a CHARA number if you answer yes.+
  
-If an object is not in SIMBAD (such as a recently discovered nova), then you must enter the object manually using the instructions below.+Also be aware that the gains do not always return to the normal values of Slew 4-7 and Tracking 7-10 when a dome server has crashed and is restarted. They may come up as Gain 22 for all axes and modes and they will need to be turned down or the scopes will oscillate badly.
  
- \\ __Finding a star in the CHARA database__+==== Telescope is not receiving the commanded position for a target. ====
  
-Here is an example of how to find the CHARA number of star, novaor AGN that is already in the CHARA database:+Sometimes it happens that a telescope receives the wrong position for a target or does not receive the commanded position at all. The commanded position is listed on the telescope server in the first column under TCS Az/El; the second column lists the actual position of the telescope. Try entering the star designation directly into the telescope serverie. hd 123456. If it does not accept the numbertry closing and restarting the telescope server and hitting repoen on Cosmic Debris and the telescope gui. Try entering the star into the server again. If that does not work, it is possible that something is wrong with the dome server. To restart the dome server follow these steps above.
  
-[ctrscrut:502] dbadd NGC 4151 \\ +==== Telescope is tracking poorly, overshooting in slew, oscillating. ====
-Object "NGC 4151" found as CHARA number: 320263+
  
-Note: If the target information stored in the CHARA database does not match SIMBADthen dbadd will provide the coordinates and magnitudes listed in SIMBAD, and it will also list the CHARA numbers of objects in the database located within 10 arcsecPlease check these nearby entries before adding new star to the database.+This might mean that the gain for the tracking servo is wrong. Note that changing this gain can be dangerousespecially if you set it too high as that can cause the telescope to oscillated and damage the drives. Please only do this if you are very very sure that it is necessary. Symptoms of bad gain are: The scope over shoots the position while slewing. The star will be seen to move out of the window and may come back after a few secondsThis means the slewing gain is too low. The scope oscillates when tracking or after slew. The star will be tracing an ellipse, figure eight or other looping shape. This means the tracking gain is too low. You can damp this out with the telescope or dome gui by disabling the scope, then re-enabling it. Adjust the gain upward and watch it on the next slew. In all cases if either gain is too high the scope will go into "Fog Horn" mode, which is bad. This can be seen during slews on the twfs or labao as vibrating spots, usually in one axis.
  
- \\ __Targets that are not in the CHARA database__ \\  \\ +You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high.
-If an object is not in the CHARA database, then you can use dbadd to query SIMBAD and add the star to the CHARA database. This might need to be done for faint stars, novae, and AGN. If the object is found in SIMBAD, then dbadd will return the target's coordinates, magnitudes, and spectral type and ask if this it the object for which you are looking. If you answer yes, then dbadd will issue a CHARA number and add the object to the database. Please check the target coordinates to make sure that they are correct before adding a new star to the database.+
  
-Here is an example of how to add an object to the CHARA database:+The usual values for slewing gains are 4-7 and tracking gains are 7-10. Note that these values may change as the temperatures change rapidly. Gains are usually higher when cold and lower when warm. Be sure to set mode back to AUTO if changing the gains left it in Slewing or Tracking mode.
  
-[ctrscrut:534] dbadd BD 41 3807 \\ +==== How to Adjust CPUMotor Gains ====
-SIMBAD found this for "BD 41 3807": +
- +
-ID 2MASS_J20340850 4136592 \\ +
-CHARA 320414 \\ +
-2000.0 \\ +
-CAT8 2MASS_J20340850 4136592 \\ +
-RA 20 34 08.5148 \\ +
-DEC 41 36 59.408 \\ +
-PMRA -2.32 \\ +
-PMDEC -3.87 \\ +
-PAR 0.58 \\ +
-RVEL -27.50 \\ +
-VMAG 10.03 \\ +
-HMAG 6.23 \\ +
-KMAG 5.99 \\ +
-SP O5.5Ifc \\ +
-END +
- +
-Is the above the object you are looking for? [y/n] y \\ +
-Added "BD 41 3807" to the database. +
- +
-In the example above, the object "BD 41 3807" has been added to the database as CHARA number 320414. +
- +
- \\ __Targets that are not in SIMBAD__ \\  \\ +
-Some objects like recently discovered novae are not in SIMBAD. Entering dbadd <star designation> for a star not in the SIMBAD database returns the message "Simbad is unable to find an object matching <star designation>. Try using a different catalog designation, or use the "-m" switch to add the object manually." \\  \\ +
-The command dbadd -m is used by entering the star name and coordinates in this format: \\  \\ +
-Usage: dbadd -m <name> <RA> <Dec> \\  \\ +
-<name>: Object ID (no spaces) \\  \\ +
-<RA> : XXhXXmXX.X or XXhXX.X (no spaces) \\  \\ +
-<Dec> : XXdXXmXX.X or XXdXX.X (no spaces) \\  \\ +
-There are cases where an object is not in SIMBAD and doesn't return a result in dbadd, but is in fact in the CHARA database. Novae and AGN's are the likely objects that cause this result. At times, objects have multiple entries and several CHARA numbers since the names can be so unique. A system will be developed to find these entries without knowing or having a conventional database designation. \\  \\ __Binary stars__ \\  \\ +
-Some binary stars have a common HD number with an A or B after them. These can cause problems as Cosmic Debris does not accept non numerical entries when filling in star designation. These stars are likely in CHARA's database but need to be searched in dbadd or on SIMBAD to get the CHARA number or another designation. +
- +
-When a recognized identifier is entered, the HD number with A or B will usually display. Confirm that the coordinates, magnitudes and spectral type match the star desired. If they do not match, you may have the wrong star or the database may have the wrong info and the baseline solution will be affected. Offsets based on incorrect coordinates or misidentifications can move the fringes many thousands of microns away from the calculated position. This can happen when observing a dim companion (B) using the brighter (A) companion's coordinates. Inform the observers that noting and using the CHARA number will save time next time the target is observed. \\  \\ __Editing the database__ \\  \\ +
-If you find a mistake in the database, please send an email to Nils to have it corrected. Identify what you believe to be the error and what is the correct information. +
- +
-==== Adding an anchor to the wiki page ==== +
- +
-To add an anchor to the wikipage, type the following where you want the anchor inserted (without the spaces between the symbols): \\ +
-[ [ #AnchorName ] ] \\  \\ +
-Highlight the text you want to be a link to the anchor, click the link symbol above and select anchor. Select a page(such as Trouble Shooting) where the anchor is located and give the same name for the anchor. +
- +
-==== Displays for the MIRC-X wolverine2 monitors ==== +
- +
-Follow instructions in the [[:chara:mircx_monitors|linked page to display the MIRC-X wolverine2 monitors]] in the control room. +
- +
- \\ +
-Last updated 2017-04-12 +
- +
- +
-==== The Telescope won't move or stopped moving ====+
  
 \\ \\
-Have the powers been turned on to the drives? Are the scopes disabled? The usual state of the telescopes is disabled until enabled. This is due to the stall function of the scopes which eventually disables the scopes when they are stowed. Enable the scopes by hitting [ENABLE] on the dome gui or in the telescope gui control tabIf scope disables itself during a slewit may be just an overcautious stall function. Hit ENABLE and the scope should continue to slew. If it disables again without moving, there may be something wrong at the scope, ie. a hatch left open, a ladder not put away, a tool on the floor, or something else physically impeding the motion of the scope. You will need to go to the dome to see what it is. The computer in the dome will give you control of the scope to turn it away from the problem.\\+The GAIN controls the gain of the feedback between the encoder and the drive velocityA high gain means "stiffer" responsebut can lead to oscillations or fog-horning if it's too high.\\
 \\ \\
-Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopesCheck them for current times and continuous updates of numbers. If they are not updatingtry to REOPEN them first. If that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of it. See **Dome Server Restart** below.\\+The Fn controls the maximum frequency of the servo responseA high Fn means higher frequencies are allowed throughwhich can mean correcting for faster problems but if too high can also lead to oscillations or fog-horning.\\
 \\ \\
-**Azimuth Limit Switches** \\+The software will not make changes to either of these quickly as that is a dangerous thing to do. There is NO POINT to clicking the up or down buttons more than once every few seconds. Indeed it is bad to do so as you will confuse the software. The change between slewing and tracking is also slow for similar reasons. This is why sometimes the "wrong" thing seems to change. It is a sign that you are trying to do things too quickly.\\
 \\ \\
-<font 14px/Arial,Helvetica,sans-serif;;inherit;;inherit>As of 11-'17, the azimuth limit switches are enabled and can stop the motion of the scopes if they try to go beyond -90º or +450º. The scopes will not be movable with normal inputs so follow these instructions to return them from the out of range condition.\\+Slew and tracking mode work differentlymostly because the speeds are so different.\\
 \\ \\
-1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software.\\+IN SLEWING MODE\\
 \\ \\
-2. Make sure you understand why the limit was hit which may require a trip to the telescope.\\+- If the gain is too low you will overshoot the target.\\
 \\ \\
-3. Click the OVERRIDE ON button in domegui MANUAL tab. After this, the hardware doesn't care about the limits switches and you're free to move the telescope.\\+- If the gain is too high it will fog-horn.\\
 \\ \\
-4. Click ENABLE then you can move the telescope back to its normal range of operation.\\+- If the Fn is too high it will also fog-horn, even at low gains.\\
 \\ \\
-5. After the telescope is back in it normal rangeclick OVERRIDE OFF which makes the hardware aware of the limits again</font>+You need to have the lowest possible gain and Fn in slew mode that doesn't overshoot the target. Fn in slew mode should almost never be higher than 4. If it isplease turn it back down to 4. If you think this is a problem please let Theo know, along with a detailed explanation of what happened.\\ 
- +\\ 
-<font 16px/Calibri,sans-serif;;#333333;;white>**However, If you notice that a telescope will stop near AZ 90 or 270 with the scope still being ENABLED and refusing to move, this is often due to the new AZ limit switch being tripped at some point earlier. To get it moving again:**</font> +IN TRACKING MODE\\ 
- +\\ 
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;white>1On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software.</font>  \\+- If the gain is too low it will keep moving between slewing and tracking.\\ 
 +\\ 
 +If the gain is too high it will fog-horn.\\ 
 +\\ 
 +- The same goes for Fn.\\ 
 +\\ 
 +In tracking mode you want the highest gain and Fn that allows the telescope to track well without fog-horning. If it "oscillates", which you will see in the green dots of tiptilt oscillating, try turning up the gain, and also try turning down the Fn.\\ 
 +\\ 
 +Some final remarks:\\
 \\ \\
-<font inherit/inherit;;initial;;white>2. M</font>ake sure the all the scopes’ demand positions agree – for example, sometimes bringing a scope to a configuration that’s already on sky and issuing a slew command will make the additional scope go around North the “wrong” way.\\+- The gain is temperature dependent, so when the temperature changes these things will change, but more so for tracking.\\
 \\ \\
-<font inherit/inherit;;initial;;white>3Click the OVERRIDE ON button in domegui MANUAL tab.</font>+- The tiptilt system almost never causes oscillations, it almost always shows you that the scope is oscillating. If the white dots are centered on tiptilt and the green dots are moving the tiptilt is doing it's job and correcting for scope motion.\\ 
 +\\ 
 +- If a drive gets disabled at the end of a slew, the gain is too low. At low gain and low velocity the encoder signal changes very slowly or doesn't change at all. After 5 secs the software interprets this as a stall and disables the drive. The tricky part is that increasing the gain to avoid this situation might make the telescope to oscillate during the next slew. So the gain should be low (4 or even 1) during slewing but higher 7 or 10 when the telescope is basically at the target position.
  
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>4. Move the scope a bit back toward the direction it was coming from – for example, if the scope stopped at AZ 268 while rotating clockwise, move it back to 265 or so using AZ DEC. Then press STOP.</font>+==== E2 AOB Dichroic Recovery ====
  
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>5Move the scope past AZ 270 by pressing AC INC. Usally 2-4 degrees will do.</font>+{{:chara:files:beaconrecoveryinstructions.pdf|E2 AOB Dichroic Recovery Document}}
  
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>6. Click OVERRIDE OFF, go to the AUTO tab and press AUTO, then NEXT (also in the obsgtk; this will restore the original star's demand position to the scope).</font>+==== E1 Hut and Cooler Communications Recovery ====
  
-<font 14px/Arial,Helvetica,sans-serif;;#333333;;inherit>7When you have time, go to the scope and reset the limit switch; otherwise, it will stop each time you pass AZ 270/90.</font>+{{:chara:files:e1_hut_coms_recovery.pdf|E1 HuT and Cooler Communications Recovery}}
  
 +==== S2 Acquisition Server Restart ====
  
-==== The Telescope won't track ====+S2's acquisition server resides on "S2pi4" computer. Logging on to it (ssh s2pi4) and running bootlaunch will restart it
  
- \\ If a bright star is manually found and not slewed to by number, (usually after the scope has gotten lost), the telescope will not track as it does not know it has gone to a star. When you init the EL and AZ position of the scope to the star's position, you must enable the tracking by going to the dome gui, selecting the RA/DEC tab and hitting the [GO RA/DEC] button. The scope will track, but the pointing model may still be off. Try slewing to the same star and initing on it after it is locked in Tiptilt. This can be done in the dome with the computer at the telescope.+==== Focusing the Telescope ====
  
-==== The Telescope won't stop moving or moves beyond the commanded position ====+  - Fully align beacon to labao and TWFS, especially making sure the focus terms between the two agree very well. 
 +  - Open a secondary gui: mergurygtk W2_SECONDARY. 
 +  - Note the current M2 actuator positions and send them to Nic for the log. 
 +  - Turn off the beacons and lock the star on TWFS tiptilt. 
 +  - Turn the servo on in the mercurygtk gui. 
 +  - On the align tab of the wfsgtk there is a button for FOCUS SCOPE. 
 +  - Press the button. 
 +  - When done the focus term of starlight on the TWFS should be zero. 
 +  - Turn off the servo on the mercury_gtk gui.
  
- \\ Disable the scope whenever its motion is outside of what it is supposed to do. Turn the powers off to the axes as well on the Power gui. The dome server is usually to blame and will need to be restarted.+A couple of notes:
  
-==== Dome Server Restart ====+  * If large moves of M2 are made, tiptilt tracking may be lost. Pay careful attention so that M2 doesn't get driven away crazily due to lost light. 
 +  * If it takes too long to start the autofocus procedure, the mercury_gtk servo will automatically turn off and nothing will happen. So just make sure the servo is on right before hitting the FOCUS SCOPE button.
  
- \\ To manually start the dome server: \\  \\ 1. Make sure the power to the drives is OFF. \\ 2. Login to the relevant computer as root. For example, type "s1" or "ssh s1" to log on to S1. \\ 3. Work out the process ID number (PID), either with the command \\  \\ (s1:1001) tsockman get dome_S1 \\ Name : dome_S1 \\ Machine : s1.chara-array.org \\ PID : 29953 \\ Commands : -1 \\ Data : -1 \\ Message : 4002 \\ Restart : /usr/local/bin/dome_server -A33.7441 S1 \\  \\ or with \\ (s1:1003) ps aux | grep dome \\ theo 4473 0.0 0.0 61188 748 pts/3 S+ 10:45 0:00 grep dome \\ observe 29953 18.5 0.4 35596 9860 ? Sl Apr21 416:11 /usr/local/bin/dome_server -A33.7441 S \\  \\ It can also be found by pulling up the LIST on SOCKMAN and selecting the relevant dome. \\  \\ So in this case the PID is 29953. \\  \\ 4. Try and stop the server gracefully: kill -2 29953 \\ 5. You should then check that the server has indeed stopped: \\  \\ [s1:600] tsockman get dome_S1 \\ Name : dome_S1 \\ Machine : s1.chara-array.org \\ PID : 15635 \\ Commands : -1 \\ Data : -1 \\ Message : 2008 \\ Restart : /usr/local/bin/dome_server -A41.0166 S1 \\  \\ If the socket manager still thinks it's running you will need to stop it forcefully: kill -9 29953; tsockman rm dome_S1 \\  \\ 6. Restart the dome server by copying the command at the end of the /etc/rc.local file: \\  \\ [s1:602] more /etc/rc.local \\ <<< Press space bar or enter to scroll through file>>> \\  \\ #Run the dome server \\  \\ /usr/local/bin/tsockman remove dome_S1 \\ /usr/local/bin/dome_server -A41.0166 S1 & \\  \\ (Note: the part of the command that saves information to /var/log/dome_S1.log has been removed.) \\  \\ 7. Turn the power to the drives back on. \\ 8. Hit REOPEN and ENABLE on the domegtk, and type "otcs" in the telescope server. \\  \\ You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scopes position.+===== Dome issue =====
  
-==== Telescope clock is not correct ==== 
- 
- \\ The NTP server has obsoleted this problem and fix as of 2014. 
- 
-==== Telescope is not receiving the commanded position for a target ==== 
- 
- \\ Sometimes it happens that a telescope receives the wrong position for a target or does not receive the commanded position at all. The commanded position is listed on the telescope server in the first column under TCS Az/El; the second column lists the actual position of the telescope. Try entering the star designation directly into the telescope server, ie. hd 123456. If it does not accept the number, try closing and restarting the telescope server and hitting repoen on Cosmic Debris and the telescope gui. Try entering the star into the server again. If that does not work, it is possible that something is wrong with the dome server. To restart the dome server follow these steps: \\ * DISABLE the telescope using either the telescope or dome GUI. 
- 
-  * Turn off the power for the telescope (both AZ/EL). 
-  * First shutdown the telescope server 
-  * Then use SOCKMAN to select the appropriate dome server from the list (dome_E1, etc) Get the PID number for reference. 
-  * The dome server will need to be restarted from the command line in a terminal. 
-  * Open a terminal window and log on to the telescope computer by typing "s1", "s2", "e1", "e2", "w1", or "w2" 
-  * Look for instruction in the /etc/rc.local file by typing "more /etc/rc.local". Scroll through the file by hitting space bar or Enter. 
-  * Locate instruction listed under "#Run the dome server". Copy and paste the commands at the prompt directly on the telescope computer: 
-      * /usr/local/bin/tsockman rm dome_S1 
-      * /usr/local/bin/dome_server -A33.7441 S1 >& /var/log/dome_S1.log 
-  * (Replace S1 with commands for appropriate telescope) 
-  * After the dome server is running, re-open the telescope server and click [REOPEN} on the dome and telescope GUIs and on Cosmic Debris. 
-  * ENABLE the telescope using the telescope or dome GUI. Check to see if the command and telescope positions are correct. If they are, then turn on the power for the telescope drives. Re-enter the star information by entering the HD number in the telescope server and click [GO NEXT] to send the telescope to the star. Make sure that the telescope is behaving as expected. The telescope might have drifted a bit, so if you can't find the star, it might be necessary to use the Telrad on a bright star to re-initialize the pointing. 
-  * Restart commands for the dome servers are also listed on the desktop in a text file. 
-<code> 
- 
-   \ 
- ==== Telescope is tracking poorly, overshooting in slew, oscillating. ==== 
- 
-</code> 
- 
-<code> 
-This might mean that the gain for the tracking servo is wrong. Note that changing this gain can be dangerous, especially if you set it too high as that can cause the telescope to oscillated and damage the drives. Please only do this if you are very very sure that it is necessary. Symptoms of bad gain are:    The scope over shoots the position while slewing. The star will be seen to move out of the window and may come back after a few seconds. This means the slewing gain is too low.  The scope oscillates when tracking or after a slew. The star will be tracing an ellipse, figure eight or other looping shape. This means the tracking gain is too low. You can damp this out with the telescope or dome gui by disabling the scope, then re-enabling it. Adjust the gain upward and watch it on the next slew.    In all cases if either gain is too high the scope will go into "Fog Horn" mode, which is bad. You always want to use the lowest gain that still allows the scope to work as best as possible. If the tiptilt tells you the scope is oscillating slowly, the gain may be too low. If it is oscillating quickly it may be too high.    On 10-22-2016, the gain settings were:    | Scope  | AZ Slewing  || EL Slewing  || AZ Tracking  || EL Tracking  || Date Updated  | 
- 
-</code> 
- 
-| |Gain|Fn|Gain|Fn|Gain|Fn|Gain|Fn| | 
-|S1|7|4|4|3|16|4|10|7|10-26-2017| 
-|S2|7| |7| |10| |10| |10-22-2016| 
-|E1|7| |7| |13| |10| |10-22-2016| 
-|E2|7| |7| |13| |10| |10-22-2016| 
-|W1|13| |10| |13| |10| |10-22-2016| 
-|W2|7| |7| |13| |13| |10-22-2016| 
- 
- \\ Note that these values may change as the temperatures change rapidly. These are still being adjusted so this is by no means final. Be sure to set mode back to AUTO if changing the gains left it in Slewing or Tracking mode. 
- 
-==== How to Adjust CPUMotor Gains ==== 
- 
- \\ The GAIN controls the gain of the feedback between the encoder and the drive velocity. A high gain means a "stiffer" response, but can lead to oscillations or fog-horning if it's too high. \\  \\ The Fn controls the maximum frequency of the servo response. A high Fn means higher frequencies are allowed through, which can mean correcting for faster problems but if too high can also lead to oscillations or fog-horning. \\  \\ The software will not make changes to either of these quickly as that is a dangerous thing to do. There is NO POINT to clicking the up or down buttons more than once every few seconds. Indeed it is bad to do so as you will confuse the software. The change between slewing and tracking is also slow for similar reasons. This is why sometimes the "wrong" thing seems to change. It is a sign that you are trying to do things too quickly. \\  \\ Slew and tracking mode work differently, mostly because the speeds are so different. \\  \\ IN SLEWING MODE \\  \\ - If the gain is too low you will overshoot the target. \\  \\ - If the gain is too high it will fog-horn. \\  \\ - If the Fn is too high it will also fog-horn, even at low gains. \\  \\ You need to have the lowest possible gain and Fn in slew mode that doesn't overshoot the target. Fn in slew mode should almost never be higher than 4. If it is, please turn it back down to 4. If you think this is a problem please let Theo know, along with a detailed explanation of what happened. \\  \\ IN TRACKING MODE \\  \\ - If the gain is too low it will keep moving between slewing and tracking. \\  \\ - If the gain is too high it will fog-horn. \\  \\ - The same goes for Fn. \\  \\ In tracking mode you want the highest gain and Fn that allows the telescope to track well without fog-horning. If it "oscillates", which you will see in the green dots of tiptilt oscillating, try turning up the gain, and also try turning down the Fn. \\  \\ Some final remarks: \\  \\ - The gain is temperature dependent, so when the temperature changes these things will change, but more so for tracking. \\  \\ - The tiptilt system almost never causes oscillations, it almost always shows you that the scope is oscillating. If the white dots are centered on tiptilt and the green dots are moving the tiptilt is doing it's job and correcting for scope motion. \\  \\ - If a drive gets disabled at the end of a slew, the gain is too low. At low gain and low velocity the encoder signal changes very slowly or doesn't change at all. After 5 secs the software interprets this as a stall and disables the drive. The tricky part is that increasing the gain to avoid this situation might make the telescope to oscillate during the next slew. So the gain should be low (4 or even 1) during slewing but higher 7 or 10 when the telescope is basically at the target position. 
- 
-==== E2 AOB Dichroic Recovery ==== 
- 
-{{:chara:files:beaconrecoveryinstructions.pdf|E2 AOB Dichroic Recovery Document}} 
- 
-==== E1 Hut and Cooler Communications Recovery ==== 
- 
-{{:chara:files:e1_hut_coms_recovery.pdf|E1 HuT and Cooler Communications Recovery}}  \\ 
- \\ 
-<font 16px/inherit;;inherit;;inherit>**S2 Acquisition Server Restart**</font> 
- 
-S2's acquisition server resides on "S2pi4" computer. Logging on to it (ssh s2pi4) and running bootlaunch will restart it 
- 
-====== Dome issue ====== 
- 
- \\ 
 At times the domes do not rotate, open or close, or otherwise behave. Some problems are simple and others are more complex. At times the domes do not rotate, open or close, or otherwise behave. Some problems are simple and others are more complex.
  
 ==== Dome does not rotate ==== ==== Dome does not rotate ====
  
- \\ +Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope's position. If not, it will insist on being in the wrong place. If it is not at the same AZ as the scope, turn AUTODOME [OFF] on the obsgtk, manually move it [CCW] or [CW] until it is centered on the telescope in spycam 1. If the dome AZ does not read the same as the telescope AZ, enter the scope AZ in the position box of the DOME tab of appropriate dome server and hit the INIT POS button to tell it the correct AZ. \\  \\ 
-Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope's position. If not, it will insist on being in the wrong place. If it is not at the same AZ as the scope, manually move it until it is centered on the telescope in spycam 1. If the dome AZ does not read the same as the telescope AZ, enter the scope AZ in the position box of the DOME tab of appropriate dome server and hit the INIT POS button to tell it at what AZ it is. \\  \\ +If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box instead of computer. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check the controller box just inside the door of the bunker. Sometimes the drive wheel jumps in the track and cannot turn the dome, even when the motor works. This will need to be fixed during the day. A fuse can also blow out in the dome control box and leave the dome stuck.
-If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check to see that control of the dome rotation is in the computer position and not manual on the dome rotation controller box just inside the door of the bunker.+
  
 ==== Dome does not open ==== ==== Dome does not open ====
  
- \\ +Is the SPYCAM on and updating? An old image may show the dome is closed, but it has actually started to open. If the dome won't open, try hitting the SLIT CLOSE button on the telescope gui first. It may be that there was a computer or server issue and the computer thinks the dome is open already and won't allow it to open again. Hit the SLIT OPEN button to see if it works now. If it still won't open, go to the bunker and look to see if the power switch is on to the dome. It will be up and red if on. 
-If the dome won't open, try hitting the SLIT CLOSE button on the telescope gui first. It may be that there was a computer or server issue and the computer thinks the dome is open already and won't allow it to open again. Hit the SLIT OPEN button to see if it works now. If it still won't open, go to the bunker and look to see if the power switch is on to the dome. It will be up and red if on. \\ + 
-Also look to see if the power cable is connected to the computer box. If it's connected up high to the manual dome controls, the cable needs to be brought back to the computer box connector. __Turn the power off first__  to the dome before disconnecting the power cable. Slide it onto the connector at the box and turn the collar to lock it in place. Turn the power back on and see if you can open the dome from the computer in the bunker.+Also look to see if the power cable is connected to the computer box. If it's connected up high to the manual dome controls, the cable needs to be brought back to the computer box connector. __Turn the power off first,__  using the light switch with the red LED toggle, to the dome before disconnecting the power cable. Slide it onto the connector at the box and turn the collar to lock it in place. Turn the power back on and see if you can open the dome from the computer in the bunker.
  
 ==== Dome does not close ==== ==== Dome does not close ====
  
- \\ 
 If a dome will not close after observing, it may need to be closed manually at the bunker. Turn the power to the dome off at the switch, disconnect the cable from the computer control box and connect it to the manual control switches above. Turn on the power to the dome after connecting the cable and use the two controls to close the dropout first, then the slit. The dropout must be fully closed and the slit closed over it to be shut properly. Inform the staff that the dome did not function properly so someone can look at it in the morning. If a dome will not close after observing, it may need to be closed manually at the bunker. Turn the power to the dome off at the switch, disconnect the cable from the computer control box and connect it to the manual control switches above. Turn on the power to the dome after connecting the cable and use the two controls to close the dropout first, then the slit. The dropout must be fully closed and the slit closed over it to be shut properly. Inform the staff that the dome did not function properly so someone can look at it in the morning.
  
-====== HUT servers ======+===== HUT servers =====
  
-==== I can't change the camera settings on the TV ====+The HUT servers control functions such as beacon and dichroic movements, heater and dehumidifier usage, and various AO functions. An observer may find that the obsgtk is no longer controlling the beacon LED's, beacon flat or dichroic alignments. This happens on occasion with E2 and other scopes. The HUT server may be the cause if it has quit or lost connection or the AOB may be at fault. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the alignments can be changed from the gui, then the HUT server is ok. You can use the hut gui to continue observing. If the hut gui gives move error messages, cycle the power on the AOB and open a new hut server to restore the connection to the obsgtk. On the POWER gui, turn off the power to the AOB for the offending telescope and turn it back on. Stop the hut server by logging into the appropriate telescope computer and identifying the PID with the bootlaunch_master command and killing the process with the kill -9 #### command. Start the new server via the bootlaunch_master command. Hit REOPEN on the obsgtk to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well.
  
- \\ +If the server won't restart, a reboot of the power supply in the telescope bunker might be necessary. The power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunker. The power supply has green readouts of volts and current. After turning the power off for 10 seconds and back on, try restarting the server from the computer in the bunker to see if it starts cleanly. If so, then restart the telescope server, reopen the connection to the telescope gui, and hit REOPEN on Cosmic Debris. Part of the HUT server also controls the AO table. If the AOB part of the HUT server doesn't work, then the power supply on the back of the AO table in the telescope dome might need recycling. This power supply controls the actuators at M2 and the AO table. The power supply box is a 6×9 inch aluminum box on the back of the AO table, behind the keyboard and monitor. Turn it off with the power button on the bottom edge of the box, wait 5 seconds and turn it back on. The HUT server should now restart cleanly. Restart the telescope server as well to make the connections to the telescope gui. Hit REOPEN in Cosmic Debris if you are observing to make all needed connections.
-The HUT servers control functions such as finder and acq exposure times and gains, heater and dehumidifier usage, and various AO functions. An observer may find that the camera settings do not display or are not adjustable. The HUT server may be the cause if it has quit. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the camera settings are displayed and the settings can be changed from the gui, then the HUT server is ok. Restart the telescope server to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well. If the gui is not functioning, the HUT server will need to be restarted. The HUT servers run on the computer at each telescope. Find the PID from Sockman for the server you want to shut down. Open a terminal and log into the correct telescope computer (for example, type "s1" to log on to S1). Use the command "ps aux | grep HUT" to find the process you need to shut down. The number should match the PID from Sockman. Kill the process by typing "kill -9 pid" where pid is the process number and remove the entry from Sockman with the REMOVE button on the gui or by command line. Restart the hut server using the new "bootlaunch_hut" command described in the section on [[:chara:trouble_shooting#restarting_servers|Restarting Servers]]. If the server won't restart, a reboot of the power supply in the telescope bunker might be necessary. The power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunker. The power supply has green readouts of volts and current. After turning the power off for 10 seconds and back on, try restarting the server from the computer in the bunker to see if it starts cleanly. If so, then restart the telescope server, reopen the connection to the telescope gui, and hit REOPEN on Cosmic Debris. Part of the HUT server also controls the AO table. If the AOB part of the HUT server doesn't work, then the power supply on the back of the AO table in the telescope dome might need recycling. This power supply controls the actuators at M2 and the AO table. The power supply box is a 6×9 inch aluminum box on the back of the AO table, behind the keyboard and monitor. Turn it off with the power button on the bottom edge of the box, wait 5 seconds and turn it back on. The HUT server should now restart cleanly. Restart the telescope server as well to make the connections to the telescope gui. Hit REOPEN in Cosmic Debris if you are observing to make all needed connections.+
  
-===== Tiptilt Server ===== +===== Tiptilt Server. (These sections apply to the old lab tiptilt system retired in April 2021) =====
- +
- \\ +
-The tiptilt server controls the CCD based tiptilt detection system\\  \\ +
-Before you start the tiptilt server, you must ensure that the power to the cooling system and the CCD iteslf is on. It is extremely important that the cooler be running before you turn on the CCD and is only turned off if you are sure the CCD is NOT running. You can start the server from the X windows menu or with the command xtiptilt. \\  \\ +
-Note that there are background counts and read noise to deal with. Whenever you change the frame rate, please ensure that the bias frame is OK. The server will attempt to load an old bias frame that should work, but if things are not working, try making a new bias frame by ensuring that the detector is in the dark and typing "mkbias" into the tiptilt server. \\  \\ +
-In the tiptilt GUI windows, the white dots represent the starlight while the green dots represent the motion applied to telescope's secondary mirror to keep the starlight centered. When tiptilt is locked the white dots will be brought to the center of the tiptilt window. The green dots should be mostly centered also. W2 and E2 telescopes have a small oscillation that show as back and forth plots of the green dots. +
- +
-==== Tiptilt server complains about the CCD ==== +
- +
- \\ +
-Is the CCD turned on? When the tiptilt server starts up it tries no more than five times to communicate with the CCD. If they all fail, it will give up. If this happens, try cycling the power to the CCD and try again. If this fails, connect to the tiptilt machine and type the command rtccdAPIDemo, which should return with no errors. Try this command a few times, but if it still fails, there is a more serious problem. Turn off the CCD and reboot the tiptilt computer. If it still fails, I am afraid you are in more serious trouble. \\  \\ +
-Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine. \\  \\ +
-Also note that cycling the power on the CCD can cause harm so be sure you need to do it before trying it. Also, it is important to wait for at least 20 seconds after turning off the power before turning it on again. +
- +
-==== Tiptilt doesn't seem to be talking to the telescopes ==== +
- +
- \\ +
-Sometimes the telescope server will not show that TT is running. It will show 0Hz for a signal rate for TT. Running TIPTILT COMM will not get it started while other scopes do show it starting. Close and restart any telescope servers that won't connect after two tries of TIPTILT COMM. \\ +
-Note: There is more info in the software manual on this topic, but I wasn't sure if it was still relevant. +
- +
-==== Tiptilt server says the clock isn't running ==== +
- +
- \\ +
-First check whether the clock itself is running and the other machines receive the clock signal. Look at the clock cards at the back of other computers in the rack. The clock cards have three LEDs, one yellow and two greens. If the computer is receiving the clock signal properly all three LEDs should blink, but at a different rate. If the LEDs on all the clock cards are solid then reboot the GPS computer. When the GPS computer is down, it is best to cycle the power also on the box right above the GPS computer. \\  \\ +
-If the clock appears to be working properly on other machines and not on the tiptilt now it is time to reboot tiptilt. \\  \\ +
-[There is a bug in the real time part of the CCD code. It is caused by the clock in the tiptilt system either not running at all or having been set to a time very different from the last time the CCD ran.] \\  \\ +
-For the time being the only solution is to reboot tiptilt, but do so from the lab. Power OFF the CCD, then reboot the tiptilt machine and go into the BIOS. Make sure that interrupt 11 has been set to ISA legacy, save the BIOS and reboot. When the clock card LEDs in the tiptilt machine indicate proper clock signal, turn the CCD back on and start the tiptilt server. \\  \\ +
-Also, sometimes Serial Port 3 grabs IRQ 11 which stops the clock from running. Since there is no serial port 2 it's safe to disable this in the BIOS. This problem normally comes up when there has been a power outage. \\  \\ +
-Sometimes syncing the clock can also cause this problem, but that should be fixed soon. If it does, exit the tiptilt server, log in as root, and reload the tiptilt model using the following commands: \\ +
-/sbin/rmmod tiptilt_rt \\ +
-/sgin/insmod /usr/local/modules/tiptilt_rt.o \\  \\ +
-Note that it is never a good idea to reboot machines unless you are very very sure it is necessary. The only reason to reboot tiptilt, other than a lock up of some kind, is that the clock interrupt has failed. You can test this by running the command "testclock" on a tiptilt command line. If this says the clock is working do not reboot the machine. +
- +
-==== Tiptilt is not locking on a star or locks, but lets the star drift away ==== +
- +
-  * Are all the mirror covers open? [Note: W1 M7 cover sometimes needs two clicks to open, despite Scope-monitor indicates open.] +
-  * Has TIPTILT COMM been run from Cosmic Debris? +
-  * Check the ACQ alignment to make sure the tick marks are centered on the laser. +
-  * If there is plenty of starlight getting into tiptilt, then try re-initializing by clicking [TIPTILT COMM] on Cosmic Debris. +
-  * If there are a large number of background counts on TT, then close the M5 cover on the telescope, and click [DBIAS] on the TIPLTILT GUI to clear the background counts. Re-open the M5 cover. +
-  * If a star drifts even with TT locked, there could be a bright sky or light from other beams getting into the affected telescope's beam. This will show in the TT server as much lower counts than the other locked stars, but still high enough to lock TT. Use the laser to find the correct position to lock TT. +
-  * If TT unlocks and the star drifts, the TT servo may not have engaged. This is engaged with the TIPTILT button on the telescope gui. Clicking it may not start the servo the first time. Try it again if the green dots drift on the TT plot windows or if the Servo status in the telescope gui reads None instead of Wobb 1. +
-  * Are the TIPTILT buttons turned on from the POWER GUI? +
- +
-===Tiptilt servo oscillates ====+
  
-You will see the oscillation in the green dots of the tiptilt GUI windowsSometimes you can also see the oscillation in the white starlight dots or as an elongation of the star when looking at the ACQ field while tiptilt is locked. Some scopes have an oscillation that has not yet been diagnosed. W2 is one that usually oscillates. A diagonal motion in the tiptilt box indicates an oscillation in one axis only, while a vertical/horizontal motion indicates an oscillation in both directions. Motion from the upper right to lower left corresponds to elevation axis while motion from the upper left to lower right corresponds to the azimuth axis. (I think you can check direction by typing sin into telescope server to send sine waves to the telescope.) There are a few ways you can try to correct the oscillation manually tuning the servo:+[[:chara:old_lab_tiptilt_server|Instructions for the old lab tiptilit system are archived here.]]
  
-  * Type "tune" into the tiptilt server. Select the appropriate telescope. Turning the gain down normally helps. +===== LabAO =====
-  * Type "tune" into the appropriate telescope server. The default value for the proportional term is -0.5 and differential term is 0.0. Adjust these values between -0.2 to -1.0 for proportional and 0.0 to 0.2 for differential. +
-  * Read section on adjusting the Telescope Tracking Gain using the Dome Server GUI (ref).+
  
-==== Tiptilt is saturating ====+==== LabAO Shutters Not Working ====
  
-  * Tiptilt saturates at ~ 200,000 counts. If you are near this limityou can reduce the TT exposure time to lower the number of counts. +<font 10.5pt font-family:/inherit;;#333333;;inherit>If the labao shutters are not working, the power button on each of the two red control boxes on the top of the table might need to be pressedAfter that the other button needs to be pressed and held for 5 seconds so that the units can be controlled remotely. (Note: look for instructions on table.) A good way for this step is to manually set the shutter to the opposite state that it is in remotely, keep pressing the “enable” button until you hear a slight clickRelease. The shutter should now change state.</font>
-  * Set the TT exposure time on Cosmic Debris in the box for "Tiptilt (mS)". This will set the TT exposure time when slewing to a new target. +
-  * To change the exposure when already at a target, then click the [EXP] button on the TipTilt GUI. This will bring up a dialog box where you can enter the new exposure time in msecCheck to make sure the tiptilt frequency (in Hz) changes on the tiptilt server after changing the exposure timeIf the frame rate doesn't change, then set the exposure time back to the old value and try entering the new value againYou might have to do this a few times to actually get the frame rate to change. +
-  * Frame rates for given exposure times: +
-      * ExpTime = 5 msec, Frame Rate = 157 Hz +
-      * ExpTime = 2 msec, Frame Rate = 299 Hz +
-      * ExpTime = 1 msec, Frame Rate = 427 Hz+
  
-==== Tiptilt counts are way too low ====+<font 10.5pt font-family:/inherit;;#333333;;inherit>If this does not work, or the other CHARA shutters are not working then a reboot is required. The order of restart needs to be correct for all shutters to work again (not shutters on insturment tables which are controlled by instrument servers).</font>
  
-Try using a slower frame rate or increasing the NSUMAlso ensure that the acquistion is properly aligned with the laserTo change the frame rateclick the [EXP] button on the tiptilt GUI and enter a longer integration time. Remember to change "Tiptilt (mS)" on Cosmic Debris to keep the same exposure time when slewing to the next target.+<font 10.5pt font-family:/inherit;;#333333;;inherit>Turn off electronics box for shutters (above the GPS computer)Reboot the gps computer. Check to see that the shutter server is running by typing bootlaunch_master on gps computerIf runningturn on shutter electronics box (you just rebooted gps computer so restart gps server).</font>
  
-==== Tiptilt counts are negative ====+==== LabAO DM Electronics ====
  
-The bias frame is bad. Get new one or turn it off.+If labao server complains that it failed to find the EDAC40 device, try unplugging the power and ethernet cables for DM and then replugging in again. The boxes for these are under the respective tables.
  
 ===== OPLE and Metrology ===== ===== OPLE and Metrology =====
Line 1400: Line 792:
   * This will install run_ople. Try opening the ople server again.   * This will install run_ople. Try opening the ople server again.
  
-==== Metrology and Homing carts ====+==== Homing carts ====
  
-  * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror. +  * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has stopped on the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. The X does not guarantee the cart has retained its home position. This home position can be checked by using the CHECK button on the ople gui or by typing the command homechk S1” into the ople server. When the error value is displayed on the ople server, hit ESC to clear the display. Click on the [OL] and [MAN] buttons and hit [TRACK] to send the cart to the desired cart position.
-  * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has homed to the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. You can also find the size of the error by typing the command "homechk S1into the ople server. Hit the [HOMEbutton and it should move forward and trigger the home switch. Hit [TRACK] to home it to the new home position.+
  
 +==== Metrology ====
 +
 +  * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror
 +
 +Steps for adjusting MET2:
 +
 +  - Looking at the met signal while the cart is at the back.
 +  - Peak so that the beam is aligned as well as possible. Does not need to be perfect but not loose.
 +  - All beams have a signal between 1-2 volts while at the back. Sometimes the signal strength can change for no apparent reasons.
 +  - Using steps of 50, try moving 3 times in one direction. After each step, wait 5 seconds or so for things to update. If nothing then try the other way.
 +  - Return to starting position.
 +  - Repeat for the other axis.
  
 ===== Beam Samplers ===== ===== Beam Samplers =====
  
-==== Beam Samplers are not moving ====+==== Beam Samplers are not moving (not a problem anymore) ====
  
 This happens most frequently with the E1 beam sampler. If you are doing alignments and are using E1, try to move it forward first to see if it dies. If it does, follow this procedure. This happens most frequently with the E1 beam sampler. If you are doing alignments and are using E1, try to move it forward first to see if it dies. If it does, follow this procedure.
Line 1421: Line 824:
 ==== Restarting the Beam Sampler Server ==== ==== Restarting the Beam Sampler Server ====
  
-The most common cause for getting multiple copies of a server is using the socket manage RESTART button, which is flaky at best. Don't do this. If you need to restart a server you should do it manually. \\  \\ +The most common cause for getting multiple copies of a server is using the socket manage RESTART button, which is flaky at best. Don't do this. If you need to restart a server you should do it manually. 
-1. Make sure you are logged into the right machine: \\  \\ + 
-ssh ctrscrut \\  \\ +1. Make sure you are logged into the right machine: ssh ctrscrut 
-2. If you are not sure see if the socket manager will tell you. If it doesn't have a look in the file: \\  \\ + 
-/ctrscrut/chara/etc/sock_manager/socket_manager.list \\  \\ +2. If you are not sure see if the socket manager will tell you. If it doesn't have a look in the file: /ctrscrut/chara/etc/sock_manager/socket_manager.list 
-3. See if there are any ghosts running \\  \\ + 
-ps aux | grep esp_server \\  \\ +3. See if there are any ghosts running 
-Yes, it's hard to know which one it is with the esp servers. You can work out which ones are ghosts by typing the command \\  \\ + 
-tsockman |grep ctrscrut \\  \\ +ps aux | grep esp_server 
-which will give you a list of the servers running on the machine you are interested in. Checking for non-matching PIDs will tell you which processes you need to stop. \\  \\ + 
-4. Stop those processes: \\  \\ +Yes, it's hard to know which one it is with the esp servers. You can work out which ones are ghosts by typing the command 
-kill -9 PID1 PID2 …. \\  \\ + 
-5. Make sure there is no sign of it in the socket manager \\  \\ +tsockman |grep ctrscrut 
-tsockman rm BS \\  \\ + 
-5. Restart the beam sampler servers (this starts both beam sampler servers BS1 and BS2): \\  \\+which will give you a list of the servers running on the machine you are interested in. Checking for non-matching PIDs will tell you which processes you need to stop. 
 + 
 +4. Stop those processes: 
 + 
 +kill -9 PID1 PID2 …. 
 + 
 +5. Make sure there is no sign of it in the socket manager 
 + 
 +tsockman rm BS 
 + 
 +6. Restart the beam sampler servers (this starts both beam sampler servers BS1 and BS2): 
 bootlaunch_beamsamp bootlaunch_beamsamp
  
 ==== "Failed to request position of S1" ==== ==== "Failed to request position of S1" ====
  
- \\ 
 Cosmic Debris reports "Failed to request position of S1" when trying to set the beam order. This indicates that ople is no longer talking to the beam samplers. If the beam sampler server has been restarted recently, then the ople server will also need to be restarted to re-establish the connection. \\  \\ Cosmic Debris reports "Failed to request position of S1" when trying to set the beam order. This indicates that ople is no longer talking to the beam samplers. If the beam sampler server has been restarted recently, then the ople server will also need to be restarted to re-establish the connection. \\  \\
 The beam sample server runs on ctrscrut. The ople server tries to open a connection to the beam sampler when it starts. If the beam sampler is not there, or dies, \\ The beam sample server runs on ctrscrut. The ople server tries to open a connection to the beam sampler when it starts. If the beam sampler is not there, or dies, \\
Line 1457: Line 870:
   * If either Phase 1 or Phase 2 states say IDLE for a given VPN, then you need to re-establish the connection. To do this, click on the "DROP" button for that VPN. Then click on the "CONNECT" button and wait for the Phase 1 and Phase 2 connections to be established.   * If either Phase 1 or Phase 2 states say IDLE for a given VPN, then you need to re-establish the connection. To do this, click on the "DROP" button for that VPN. Then click on the "CONNECT" button and wait for the Phase 1 and Phase 2 connections to be established.
   * Make sure to close Firefox by using CTRL-Q to quit the program and not just close the window.   * Make sure to close Firefox by using CTRL-Q to quit the program and not just close the window.
 +
 +===== Network Problems =====
 +
 +==== User accounts not recognized ====
 +
 +The gps computer does the user authentication for the control side while ctrscrut provides the home directories. Yellowpages is gone, the new paradigm is LDAP (local directory access protocol). If you can't log on to computers on the control network, then make sure both ctscrut (michelson) and gps computers are running. If the login screen screen on ctrscrut says "Ubuntu Administrator" and has no user accounts, then check to make sure gps computer is on. If you restart gps, then you might have to reboot ctrscrut to get the user accounts. If ctrscrut was rebooted, you might have to reboot other control network computers if you can't log on.
 +
 +If gps computer does not start, check the power distribution unit (PDU) at the top of that rack. If any units show both " " and "-" then you can use the arrow buttons to move through the units and cycle the units to on " ". That should automatically start the computer connected to those units.
  
 ===== General Problems ===== ===== General Problems =====
Line 1462: Line 883:
 ==== GUIs are frozen ==== ==== GUIs are frozen ====
  
-Check to see if Sockman is working. Click the [LIST] button on sockman. If the list doesn't come up, Sockman may be hung up. Follow the Sockman restart procedure in the text file on the desktop if it won't respond. \\ +Check to see if Sockman is working. Click the [LIST] button on sockman. If the list doesn't come up, Sockman may be hung up. Follow the Sockman restart procedure in the text file on the desktop if it won't respond. 
-Hit [REOPEN] on the gui to see if it reconnects with its server. if that does not fix it, the server may be dead. From the LIST on Sockman, select the server suspected of being frozen. Ping it and if it reports back as being dead, see [[:chara:trouble_shooting#server_is_frozen|Server is frozen]] section below. \\+ 
 +Hit [REOPEN] on the gui to see if it reconnects with its server. if that does not fix it, the server may be dead. From the LIST on Sockman, select the server suspected of being frozen. Ping it and if it reports back as being dead, see [[:chara:trouble_shooting#server_is_frozen|Server is frozen]] section below. 
 Telescope guis often freeze. Hit reopen on the gui to bring it back to life. Hitting [MOVE] on the telescope gui before the star has stopped moving can cause the gui to freeze. Avoid doing this. Close the dead one and open a new one to fix the problem. Telescope guis often freeze. Hit reopen on the gui to bring it back to life. Hitting [MOVE] on the telescope gui before the star has stopped moving can cause the gui to freeze. Avoid doing this. Close the dead one and open a new one to fix the problem.
  
Line 1469: Line 892:
  
   * If the top of a server goes blank, try typing "sb" (start background) on the server command line.   * If the top of a server goes blank, try typing "sb" (start background) on the server command line.
-  * If the server screen fills with jibberish, try hitting CTRL-in the server to clear it.+  * If the server screen fills with jibberish, try hitting CTRL-in the server to clear it. 
 +  * If a server freezes, sometimes hitting CTRL-C and then N, for NO, will unfreeze it.
  
 ==== Server is frozen ==== ==== Server is frozen ====
Line 1479: Line 903:
   * Click [REOPEN] on Cosmic Debris and relevant GUIs to re-initialize communication with the server after it is restarted.   * Click [REOPEN] on Cosmic Debris and relevant GUIs to re-initialize communication with the server after it is restarted.
   * A folder is on the desktop that has the restart commands for CTRSCRUT servers and the Shutters server restart command running on OPLE. Use it to restart servers that will not reopen from the menu.   * A folder is on the desktop that has the restart commands for CTRSCRUT servers and the Shutters server restart command running on OPLE. Use it to restart servers that will not reopen from the menu.
 +  * if a telescope server does not close after using CTRL-C when restarting them after UT midnight and the Send ON button is green on the obsgtk, turn the Send ON to OFF and the server will close. Turn the Send ON button back on to resume the setup for the night.
  
 ==== PAVO Server - Error communicating with IFW ==== ==== PAVO Server - Error communicating with IFW ====
  
-The following error message sometimes appears when starting the PAVO server: \\ +The following error message sometimes appears when starting the PAVO server: "Error communicating with IFW. Could not read from IFW. Is the filter wheel plugged in and on?" Most of the time, restarting the PAVO server will clear this error message and allow PAVO to communicate with the filter wheel. If restarting the server doesn't help, then check that the cables going into the small, black IFW box on the PAVO table and into the back of the PAVO computer are plugged in securely.
-"Error communicating with IFW. Could not read from IFW. Is the filter wheel plugged in and on?" \\ +
-Most of the time, restarting the PAVO server will clear this error message and allow PAVO to communicate with the filter wheel. If restarting the server doesn't help, then check that the cables going into the small, black IFW box on the PAVO table and into the back of the PAVO computer are plugged in securely.+
  
 ==== Adding or Finding a Star in the CHARA Database ==== ==== Adding or Finding a Star in the CHARA Database ====
Line 1490: Line 913:
 The command "dbadd" can be used to look up the CHARA number for a star or to add a star to the CHARA database. To use dbadd, open a terminal window, and type "dbadd starname" where starname is the common name (e.g., Vega) or identifier of the star (e.g., IRC, HR, HD, SAO, FK5, HIP, GJ, or 2MASS designation). If the object is in the database, then it will return the CHARA number. The command "dbadd" can be used to look up the CHARA number for a star or to add a star to the CHARA database. To use dbadd, open a terminal window, and type "dbadd starname" where starname is the common name (e.g., Vega) or identifier of the star (e.g., IRC, HR, HD, SAO, FK5, HIP, GJ, or 2MASS designation). If the object is in the database, then it will return the CHARA number.
  
-You can also look up different identifiers for the star by using the SIMBAD database: [[http://simbad.u-strasbg.fr/simbad/sim-fbasic|http://simbad.u-strasbg.fr/simbad/sim-fbasic]] \\  \\ +You can also look up different identifiers for the star by using the SIMBAD database: [[http://simbad.u-strasbg.fr/simbad/sim-fbasic|http://simbad.u-strasbg.fr/simbad/sim-fbasic]] 
-DBADD uses SIMBAD to look up the coordinates of the star and then searches to see if the star is already in the CHARA database. If the star is in the database, it will return the CHARA number. If the star is not in the CHARA database, it will ask if you want to add it and assign a CHARA number if you answer yes.+ 
 +DBADD uses SIMBAD to look up the coordinates of the star and then searches to see if the star is already in the CHARA database. If the star is in the database, it will return the CHARA number. If the star is not in the CHARA database, it will ask if you want to add it and will assign a CHARA number if you answer yes.
  
 If an object is not in SIMBAD (such as a recently discovered nova), then you must enter the object manually using the instructions below. If an object is not in SIMBAD (such as a recently discovered nova), then you must enter the object manually using the instructions below.
  
- \\ __Finding a star in the CHARA database__+**Finding a star in the CHARA database**
  
 Here is an example of how to find the CHARA number of a star, nova, or AGN that is already in the CHARA database: Here is an example of how to find the CHARA number of a star, nova, or AGN that is already in the CHARA database:
Line 1504: Line 928:
 Note: If the target information stored in the CHARA database does not match SIMBAD, then dbadd will provide the coordinates and magnitudes listed in SIMBAD, and it will also list the CHARA numbers of objects in the database located within 10 arcsec. Please check these nearby entries before adding a new star to the database. Note: If the target information stored in the CHARA database does not match SIMBAD, then dbadd will provide the coordinates and magnitudes listed in SIMBAD, and it will also list the CHARA numbers of objects in the database located within 10 arcsec. Please check these nearby entries before adding a new star to the database.
  
- \\ __Targets that are not in the CHARA database__ \\  \\+**Targets that are not in the CHARA database** 
 If an object is not in the CHARA database, then you can use dbadd to query SIMBAD and add the star to the CHARA database. This might need to be done for faint stars, novae, and AGN. If the object is found in SIMBAD, then dbadd will return the target's coordinates, magnitudes, and spectral type and ask if this it the object for which you are looking. If you answer yes, then dbadd will issue a CHARA number and add the object to the database. Please check the target coordinates to make sure that they are correct before adding a new star to the database. If an object is not in the CHARA database, then you can use dbadd to query SIMBAD and add the star to the CHARA database. This might need to be done for faint stars, novae, and AGN. If the object is found in SIMBAD, then dbadd will return the target's coordinates, magnitudes, and spectral type and ask if this it the object for which you are looking. If you answer yes, then dbadd will issue a CHARA number and add the object to the database. Please check the target coordinates to make sure that they are correct before adding a new star to the database.
  
 Here is an example of how to add an object to the CHARA database: Here is an example of how to add an object to the CHARA database:
  
-[ctrscrut:534] dbadd BD+41 3807 \\ +[ctrscrut:534] dbadd BD 41 3807
-SIMBAD found this for "BD+41 3807":+
  
-ID 2MASS_J20340850+4136592 \\+SIMBAD found this for "BD 41 3807": 
 + 
 +ID 2MASS_J20340850 4136592 \\
 CHARA 320414 \\ CHARA 320414 \\
 2000.0 \\ 2000.0 \\
-CAT8 2MASS_J20340850+4136592 \\+CAT8 2MASS_J20340850 4136592 \\
 RA 20 34 08.5148 \\ RA 20 34 08.5148 \\
-DEC +41 36 59.408 \\+DEC 41 36 59.408 \\
 PMRA -2.32 \\ PMRA -2.32 \\
 PMDEC -3.87 \\ PMDEC -3.87 \\
 PAR 0.58 \\ PAR 0.58 \\
-RVEL -27.50 \\+RVEL -27.50 
 VMAG 10.03 \\ VMAG 10.03 \\
 HMAG 6.23 \\ HMAG 6.23 \\
Line 1529: Line 956:
  
 Is the above the object you are looking for? [y/n] y \\ Is the above the object you are looking for? [y/n] y \\
-Added "BD+41 3807" to the database.+Added "BD 41 3807" to the database.
  
-In the example above, the object "BD+41 3807" has been added to the database as CHARA number 320414.+In the example above, the object "BD 41 3807" has been added to the database as CHARA number 320414. 
 + 
 +**Targets that are not in SIMBAD** 
 + 
 +Some objects like recently discovered novae are not in SIMBAD. Entering dbadd <star designation> for a star not in the SIMBAD database returns the message "Simbad is unable to find an object matching <star designation>. Try using a different catalog designation, or use the "-m" switch to add the object manually." 
 + 
 +The command dbadd -m is used by entering the star name and coordinates in this format:
  
- \\ __Targets that are not in SIMBAD__ \\  \\ 
-Some objects like recently discovered novae are not in SIMBAD. Entering dbadd <star designation> for a star not in the SIMBAD database returns the message "Simbad is unable to find an object matching <star designation>. Try using a different catalog designation, or use the "-m" switch to add the object manually." \\  \\ 
-The command dbadd -m is used by entering the star name and coordinates in this format: \\  \\ 
 Usage: dbadd -m <name> <RA> <Dec> \\  \\ Usage: dbadd -m <name> <RA> <Dec> \\  \\
 <name>: Object ID (no spaces) \\  \\ <name>: Object ID (no spaces) \\  \\
 <RA> : XXhXXmXX.X or XXhXX.X (no spaces) \\  \\ <RA> : XXhXXmXX.X or XXhXX.X (no spaces) \\  \\
 <Dec> : XXdXXmXX.X or XXdXX.X (no spaces) \\  \\ <Dec> : XXdXXmXX.X or XXdXX.X (no spaces) \\  \\
-There are cases where an object is not in SIMBAD and doesn't return a result in dbadd, but is in fact in the CHARA database. Novae and AGN's are the likely objects that cause this result. At times, objects have multiple entries and several CHARA numbers since the names can be so unique. A system will be developed to find these entries without knowing or having a conventional database designation. \\  \\ __Binary stars__ \\  \\+There are cases where an object is not in SIMBAD and doesn't return a result in dbadd, but is in fact in the CHARA database. Novae and AGN's are the likely objects that cause this result. At times, objects have multiple entries and several CHARA numbers since the names can be so unique. A system will be developed to find these entries without knowing or having a conventional database designation. 
 + 
 +**Binary stars** 
 Some binary stars have a common HD number with an A or B after them. These can cause problems as Cosmic Debris does not accept non numerical entries when filling in star designation. These stars are likely in CHARA's database but need to be searched in dbadd or on SIMBAD to get the CHARA number or another designation. Some binary stars have a common HD number with an A or B after them. These can cause problems as Cosmic Debris does not accept non numerical entries when filling in star designation. These stars are likely in CHARA's database but need to be searched in dbadd or on SIMBAD to get the CHARA number or another designation.
  
-When a recognized identifier is entered, the HD number with A or B will usually display. Confirm that the coordinates, magnitudes and spectral type match the star desired. If they do not match, you may have the wrong star or the database may have the wrong info and the baseline solution will be affected. Offsets based on incorrect coordinates or misidentifications can move the fringes many thousands of microns away from the calculated position. This can happen when observing a dim companion (B) using the brighter (A) companion's coordinates. Inform the observers that noting and using the CHARA number will save time next time the target is observed. \\  \\ __Editing the database__ \\  \\ +When a recognized identifier is entered, the HD number with A or B will usually display. Confirm that the coordinates, magnitudes and spectral type match the star desired. If they do not match, you may have the wrong star or the database may have the wrong info and the baseline solution will be affected. Offsets based on incorrect coordinates or misidentifications can move the fringes many thousands of microns away from the calculated position. This can happen when observing a dim companion (B) using the brighter (A) companion's coordinates. Inform the observers that saving and using the CHARA number will save time the next time the target is observed. 
-If you find a mistake in the database, please send an email to Nils to have it corrected. Identify what you believe to be the error and what is the correct information.+ 
 +**Editing the database:**  If you find a mistake in the database, please send an email to Nils to have it corrected. Identify what you believe to be the error and what is the correct information.
  
 ==== Adding an anchor to the wiki page ==== ==== Adding an anchor to the wiki page ====
  
 To add an anchor to the wikipage, type the following where you want the anchor inserted (without the spaces between the symbols): \\ To add an anchor to the wikipage, type the following where you want the anchor inserted (without the spaces between the symbols): \\
-[ [ #AnchorName ] ] \\  \\+[ [ #AnchorName ] ] 
 Highlight the text you want to be a link to the anchor, click the link symbol above and select anchor. Select a page(such as Trouble Shooting) where the anchor is located and give the same name for the anchor. Highlight the text you want to be a link to the anchor, click the link symbol above and select anchor. Select a page(such as Trouble Shooting) where the anchor is located and give the same name for the anchor.
  
Line 1555: Line 990:
  
 Follow instructions in the [[:chara:mircx_monitors|linked page to display the MIRC-X wolverine2 monitors]] in the control room. Follow instructions in the [[:chara:mircx_monitors|linked page to display the MIRC-X wolverine2 monitors]] in the control room.
 +
 +==== No Power for Aperture Wheels ====
 +
 +To minimize heat sources in the lab, the power for the apwheels is turned off. If you need to use aperture wheels during the night, the operator will need to plug the power into the outlet next to the metrology table (see photo below). If there are any communication issues after plugging in the power, try restarting the server using bootlaunch on ctrscrut. It's probably a good idea to home the apwheels after turning on the power.
 +
 +{{:chara:d859af0efb2dab690d3a44526d7b8afc.png}}
  
  \\  \\
-Last updated 2017-04-12+Last updated 2023-11-20 by Norm Vargas END HERE
  
  
chara/trouble_shooting.1636412361.txt.gz · Last modified: 2021/11/08 17:59 by charaobs