The CHARA Array Wiki

User Tools

Site Tools

Trace:
chara:trouble_shooting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
chara:trouble_shooting [2018/07/05 11:12]
jones [**I can't change the camera settings on the TV**] 		chara:trouble_shooting [2018/07/05 11:16]
jones [GUIs are frozen]
Line 95: Line 95:
 \\  \\ 
 ====I can't change the camera settings on the TV==== ====I can't change the camera settings on the TV====
-\\  The HUT servers control functions such as finder and acq exposure times and gains, heater and dehumidifier usage, and various AO functions. An observer may find that the camera settings do not display or are not adjustable. The HUT server may be the cause if it has quit. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the camera settings are displayed and the settings can be changed from the gui, then the HUT server is ok. Restart the telescope server to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well. If the gui is not functioning, the HUT server will need to be restarted. The HUT servers run on the computer at each telescope. Find the PID from Sockman for the server you want to shut down. Open a terminal and log into the correct telescope computer (for example, type "s1" to log on to S1). Use the command "ps aux | grep HUT" to find the process you need to shut down. The number should match the PID from Sockman. Kill the process by typing "kill -9 pid" where pid is the process number and remove the entry from Sockman with the REMOVE button on the gui or by command line. Restart the hut server using the new "bootlaunch_hut" command described in the section on [[chara:trouble_shooting#chapter.restart.servers|Restarting Servers]]. If the server won't restart, a reboot of the power supply in the telescope bunker might be necessary. The power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunker. The power supply has green readouts of volts and current. After turning the power off for 10 seconds and back on, try restarting the server from the computer in the bunker to see if it starts cleanly. If so, then restart the telescope server, reopen the connection to the telescope gui, and hit REOPEN on Cosmic Debris. Part of the HUT server also controls the AO table. If the AOB part of the HUT server doesn't work, then the power supply on the back of the AO table in the telescope dome might need recycling. This power supply controls the actuators at M2 and the AO table. The power supply box is a 6x9 inch aluminum box on the back of the AO table, behind the keyboard and monitor. Turn it off with the power button on the bottom edge of the box, wait 5 seconds and turn it back on. The HUT server should now restart cleanly. Restart the telescope server as well to make the connections to the telescope gui. Hit REOPEN in Cosmic Debris if you are observing to make all needed connections.\\ \\ \\ +\\  The HUT servers control functions such as finder and acq exposure times and gains, heater and dehumidifier usage, and various AO functions. An observer may find that the camera settings do not display or are not adjustable. The HUT server may be the cause if it has quit. To see if it is the server, open the HUT gui for the desired telescope from the CHARA menu. If the camera settings are displayed and the settings can be changed from the gui, then the HUT server is ok. Restart the telescope server to reopen the connection to the HUT server and hit reopen on Cosmic Debris as well. If the gui is not functioning, the HUT server will need to be restarted. The HUT servers run on the computer at each telescope. Find the PID from Sockman for the server you want to shut down. Open a terminal and log into the correct telescope computer (for example, type "s1" to log on to S1). Use the command "ps aux | grep HUT" to find the process you need to shut down. The number should match the PID from Sockman. Kill the process by typing "kill -9 pid" where pid is the process number and remove the entry from Sockman with the REMOVE button on the gui or by command line. Restart the hut server using the new "bootlaunch_hut" command described in the section on [[chara:trouble_shooting#restarting_servers|Restarting Servers]]. If the server won't restart, a reboot of the power supply in the telescope bunker might be necessary. The power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunker. The power supply has green readouts of volts and current. After turning the power off for 10 seconds and back on, try restarting the server from the computer in the bunker to see if it starts cleanly. If so, then restart the telescope server, reopen the connection to the telescope gui, and hit REOPEN on Cosmic Debris. Part of the HUT server also controls the AO table. If the AOB part of the HUT server doesn't work, then the power supply on the back of the AO table in the telescope dome might need recycling. This power supply controls the actuators at M2 and the AO table. The power supply box is a 6x9 inch aluminum box on the back of the AO table, behind the keyboard and monitor. Turn it off with the power button on the bottom edge of the box, wait 5 seconds and turn it back on. The HUT server should now restart cleanly. Restart the telescope server as well to make the connections to the telescope gui. Hit REOPEN in Cosmic Debris if you are observing to make all needed connections.\\ \\ \\ 
 ===== Tiptilt Server ===== ===== Tiptilt Server =====
 \\  The tiptilt server controls the CCD based tiptilt detection system.\\ \\  Before you start the tiptilt server, you must ensure that the power to the cooling system and the CCD iteslf is on. It is extremely important that the cooler be running before you turn on the CCD and is only turned off if you are sure the CCD is NOT running. You can start the server from the X windows menu or with the command xtiptilt.\\ \\  Note that there are background counts and read noise to deal with. Whenever you change the frame rate, please ensure that the bias frame is OK. The server will attempt to load an old bias frame that should work, but if things are not working, try making a new bias frame by ensuring that the detector is in the dark and typing "mkbias" into the tiptilt server.\\ \\  In the tiptilt GUI windows, the white dots represent the starlight while the green dots represent the motion applied to telescope's secondary mirror to keep the starlight centered. When tiptilt is locked the white dots will be brought to the center of the tiptilt window. The green dots should be mostly centered also. W2 and E2 telescopes have a small oscillation that show as back and forth plots of the green dots.\\ \\ \\  \\  The tiptilt server controls the CCD based tiptilt detection system.\\ \\  Before you start the tiptilt server, you must ensure that the power to the cooling system and the CCD iteslf is on. It is extremely important that the cooler be running before you turn on the CCD and is only turned off if you are sure the CCD is NOT running. You can start the server from the X windows menu or with the command xtiptilt.\\ \\  Note that there are background counts and read noise to deal with. Whenever you change the frame rate, please ensure that the bias frame is OK. The server will attempt to load an old bias frame that should work, but if things are not working, try making a new bias frame by ensuring that the detector is in the dark and typing "mkbias" into the tiptilt server.\\ \\  In the tiptilt GUI windows, the white dots represent the starlight while the green dots represent the motion applied to telescope's secondary mirror to keep the starlight centered. When tiptilt is locked the white dots will be brought to the center of the tiptilt window. The green dots should be mostly centered also. W2 and E2 telescopes have a small oscillation that show as back and forth plots of the green dots.\\ \\ \\ 
Line 135: Line 135:
  Try using a slower frame rate or increasing the NSUM. Also ensure that the acquistion is properly aligned with the laser. To change the frame rate, click the [EXP] button on the tiptilt GUI and enter a longer integration time. Remember to change "Tiptilt (mS)" on Cosmic Debris to keep the same exposure time when slewing to the next target.\\ \\   Try using a slower frame rate or increasing the NSUM. Also ensure that the acquistion is properly aligned with the laser. To change the frame rate, click the [EXP] button on the tiptilt GUI and enter a longer integration time. Remember to change "Tiptilt (mS)" on Cosmic Debris to keep the same exposure time when slewing to the next target.\\ \\ 
 ==== Tiptilt counts are negative ==== ==== Tiptilt counts are negative ====
- The bias frame is bad. Get a new one or turn it off.\\ \\ [[/Trouble%20Shooting#page.troubleshooting|Back to Main Menu]]\\ \\  + The bias frame is bad. Get a new one or turn it off.\\ 
-===== __OPLE and Metrology__ =====+=====OPLE and Metrology=====
 \\  \\ 
 ==== OPLE Server doesn't open - complains about not finding run_ople ==== ==== OPLE Server doesn't open - complains about not finding run_ople ====
Line 148: Line 148:
   * Type "make install"   * Type "make install"
   * This will install run_ople. Try opening the ople server again.   * This will install run_ople. Try opening the ople server again.
-==== **Metrology and Homing carts** ====+====Metrology and Homing carts====
  
   * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror.   * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror.
   * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has homed to the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. Hit the [HOME] button and it should move forward and trigger the home switch. Hit [TRACK] to home it to the new home position.   * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has homed to the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. Hit the [HOME] button and it should move forward and trigger the home switch. Hit [TRACK] to home it to the new home position.
-\\ [[/Trouble%20Shooting#page.troubleshooting|Back to Main Menu]]\\ \\  +=====Beam Samplers=====
-===== __Beam Samplers__ =====+
 \\  \\ 
 ==== Beam Samplers are not moving ==== ==== Beam Samplers are not moving ====
Line 168: Line 167:
  The most common cause for getting multiple copies of a server is using the socket manage RESTART button, which is flaky at best. Don't do this. If you need to restart a server you should do it manually.\\ \\  1. Make sure you are logged into the right machine:\\ \\  ssh ctrscrut\\ \\  2. If you are not sure see if the socket manager will tell you. If it doesn't have a look in the file:\\ \\  /ctrscrut/chara/etc/sock_manager/socket_manager.list\\ \\  3. See if there are any ghosts running\\ \\  ps aux | grep esp_server\\ \\  Yes, it's hard to know which one it is with the esp servers. You can work out which ones are ghosts by typing the command\\ \\  tsockman |grep ctrscrut\\ \\  which will give you a list of the servers running on the machine you are interested in. Checking for non-matching PIDs will tell you which processes you need to stop.\\ \\  4. Stop those processes:\\ \\  kill -9 PID1 PID2 ....\\ \\  5. Make sure there is no sign of it in the socket manager\\ \\  tsockman rm BS\\ \\  5. Restart the beam sampler servers (this starts both beam sampler servers BS1 and BS2):\\ \\  bootlaunch_beamsamp\\ \\ \\   The most common cause for getting multiple copies of a server is using the socket manage RESTART button, which is flaky at best. Don't do this. If you need to restart a server you should do it manually.\\ \\  1. Make sure you are logged into the right machine:\\ \\  ssh ctrscrut\\ \\  2. If you are not sure see if the socket manager will tell you. If it doesn't have a look in the file:\\ \\  /ctrscrut/chara/etc/sock_manager/socket_manager.list\\ \\  3. See if there are any ghosts running\\ \\  ps aux | grep esp_server\\ \\  Yes, it's hard to know which one it is with the esp servers. You can work out which ones are ghosts by typing the command\\ \\  tsockman |grep ctrscrut\\ \\  which will give you a list of the servers running on the machine you are interested in. Checking for non-matching PIDs will tell you which processes you need to stop.\\ \\  4. Stop those processes:\\ \\  kill -9 PID1 PID2 ....\\ \\  5. Make sure there is no sign of it in the socket manager\\ \\  tsockman rm BS\\ \\  5. Restart the beam sampler servers (this starts both beam sampler servers BS1 and BS2):\\ \\  bootlaunch_beamsamp\\ \\ \\ 
 ==== "Failed to request position of S1" ==== ==== "Failed to request position of S1" ====
-\\  Cosmic Debris reports "Failed to request position of S1" when trying to set the beam order. This indicates that ople is no longer talking to the beam samplers. If the beam sampler server has been restarted recently, then the ople server will also need to be restarted to re-establish the connection.\\ \\  The beam sample server runs on ctrscrut. The ople server tries to open a connection to the beam sampler when it starts. If the beam sampler is not there, or dies,\\  you need to restart the ople server as there is no command to reopen that connection.\\ \\ [[/Trouble%20Shooting#page.troubleshooting|Back to Main Menu]]\\ \\  +\\  Cosmic Debris reports "Failed to request position of S1" when trying to set the beam order. This indicates that ople is no longer talking to the beam samplers. If the beam sampler server has been restarted recently, then the ople server will also need to be restarted to re-establish the connection.\\ \\  The beam sample server runs on ctrscrut. The ople server tries to open a connection to the beam sampler when it starts. If the beam sampler is not there, or dies,\\  you need to restart the ople server as there is no command to reopen that connection.\\ 
-===== __Remote Observing__ =====+=====Remote Observing=====
 \\  \\ 
 ==== VPN connection is not working ==== ==== VPN connection is not working ====
Line 180: Line 179:
   * If either Phase 1 or Phase 2 states say IDLE for a given VPN, then you need to re-establish the connection. To do this, click on the "DROP" button for that VPN. Then click on the "CONNECT" button and wait for the Phase 1 and Phase 2 connections to be established.   * If either Phase 1 or Phase 2 states say IDLE for a given VPN, then you need to re-establish the connection. To do this, click on the "DROP" button for that VPN. Then click on the "CONNECT" button and wait for the Phase 1 and Phase 2 connections to be established.
   * Make sure to close Firefox by using CTRL-Q to quit the program and not just close the window.   * Make sure to close Firefox by using CTRL-Q to quit the program and not just close the window.
-\\ [[/Trouble%20Shooting#page.troubleshooting|Back to Main Menu]]\\ \\  +=====General Problems=====
-===== __General Problems__ =====+
 \\  \\ 
 ==== GUIs are frozen ==== ==== GUIs are frozen ====
- Check to see if Sockman is working. Click the [LIST] button on sockman. If the list doesn't come up, Sockman may be hung up. Follow the Sockman restart procedure in the text file on the desktop if it won't respond.\\  Hit [REOPEN] on the gui to see if it reconnects with its server. if that does not fix it, the server may be dead. From the LIST on Sockman, select the server suspected of being frozen. Ping it and if it reports back as being dead, see [[/Trouble%20Shooting#chapter.general.frozenserver|Server is frozen]] section below.\\  Telescope guis often freeze. Hit reopen on the gui to bring it back to life. Hitting [MOVE] on the telescope gui before the star has stopped moving can cause the gui to freeze. Avoid doing this. Close the dead one and open a new one to fix the problem.\\ \\ + Check to see if Sockman is working. Click the [LIST] button on sockman. If the list doesn't come up, Sockman may be hung up. Follow the Sockman restart procedure in the text file on the desktop if it won't respond.\\  Hit [REOPEN] on the gui to see if it reconnects with its server. if that does not fix it, the server may be dead. From the LIST on Sockman, select the server suspected of being frozen. Ping it and if it reports back as being dead, see [[chara:trouble_shooting#server_is_frozen|Server is frozen]] section below.\\  Telescope guis often freeze. Hit reopen on the gui to bring it back to life. Hitting [MOVE] on the telescope gui before the star has stopped moving can cause the gui to freeze. Avoid doing this. Close the dead one and open a new one to fix the problem.\\ \\ 
 ==== Server display is blank or filled with jibberish ==== ==== Server display is blank or filled with jibberish ====
  
chara/trouble_shooting.txt · Last modified: 2024/06/18 00:21 by charaobs

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki