The CHARA Array Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
======Restarting Servers using the bootlaunch paradigm======
\\  A number of servers use an interim bootlaunch paradigm to restart. This is confined to servers that run on ubuntu machines, namely the telescope bunker computers and gps. The basic syntax is "bootlaunch_<server>" where "<server>" is replaced by the server the script is designed to address. The scripts have a number of safeties built in, so it is safe to run them even if a server is already running -- they just output the process ID of the running server. This scripts also take care of the entry in socket manager as well any serial port lock files. All the pertinent information is world writable, so one should be able to run a bootlaunch script as observe.\\ \\  One thing of note about the output of the bootlaunch scripts, they call a number of other programs which themselves have output that may be misleading in the context of bootlaunch. Chief among these is the output of "tsockman". If a server stopped unexpectedly, it may leave behind an entry in the socket manager. In order to launch a new server, one needs to clean out the socket manager entry if it is there. To do that, "tsockman remove <entry>" is called to remove "<entry>" before the new server is launched. If there is no entry, tsockman will respond with "Process by that name does not exist". THIS IS NORMAL and is not indicative of an error. The server in question launched (without fanfare) right after that output text.\\ \\  Here are the available bootlaunch scripts as of June 2017:\\ \\  gps computer:\\ \\ 

  * bootlaunch_beamsamp -- Starts the beam sampler servers, BS1 and BS2.
  * bootlaunch_zaber -- Starts the ZABER_2 server.
\\  telescope bunker computers:\\ \\ 

  * bootlaunch_hut -- Starts the E1_HUT, E2_HUT, S1_HUT, S2_HUT, W1_HUT, or W2_HUT server, depending on the machine it's launched from.
  * bootlaunch_rpc -- Starts the RPC_E1, RPC_E2, RPC_S1, RPC_S2, RPC_W1, or RPC_W2 server, depending on the machine it's launched from.
  * bootlaunch_weather -- Starts the E1_WEATHER, E2_WEATHER, S1_WEATHER, S2_WEATHER, W1_WEATHER, or W2_WEATHER server, depending on the machine it's launched from.
  * bootlaunch_lower -- Starts the E1_Lower, E2_Lower, S1_Lower, S2_Lower, W1_Lower, or W2_Lower cylinder server, depending on the machine it's launched from.
  * bootlaunch_upper -- Starts the E1_Upper, E2_Upper, S1_Upper, S2_Upper, W1_Upper, or W2_Upper server, depending on the machine it's launched from.
\\  Note: The bootlaunch scripts will not start a new server if there is an existing process running. Therefore, type "ps aux | grep //server_name//" where server_name is the name of the server. If there is a dead process running, look up the process identification number (PID) and type "kill -9 PID" to kill the process and then run the relevant bootlaunch script again.\\ \\ 
====== Restarting Servers using the rc.local file ======
\\  This procedure is applicable to servers that have not switched over to the bootlaunch paradigm.\\ \\  Sometimes servers get locked up, crash, or stop working in some way and need to be restarted. The socket manager is no longer used to restart them, so it is necessary to restart them manually using a terminal. Here's how:\\ \\  1. Open a terminal window and connect to the computer that runs the server. These are listed at the bottom of this page. For example, to connect to the S1 telescope control computer type "s1".\\ \\  2. See if there is a process running on the machine. For example, if we were concerned about the RPC (remote Power Control) server on s1 we would use (**NOTE: The telescope RPC servers are no longer started in this way, see bootlaunch command above - these instructions are still relevant for servers not running under the bootlaunch paradigm**):\\ \\  s1:1001) ps aux | grep rpc_server\\  theo 15219 0.0 0.0 61188 752 pts/6 S+ 13:56 0:00 grep rpc_server\\ \\  So there is one running and its PID (process ID) is 15219\\ \\  3. Try and stop this gracefully by using "kill -2 15219"\\ \\  4. Redo the "ps aux | grep ....." command to see if that worked. If the PID is gone, the shut down was successful. If not force it to stop with "kill -9 15219".\\ \\  5. Make sure there is nothing left in the socket manager by using the GUI or the text command\\ \\  tscokman rm RPC_S1\\ \\  6. Find the restart command in the file /etc/rc.local or in the text files on the desktop.\\ \\  (s1:1003) grep rpc_server rc.local\\  /usr/local/bin/rpc_server /dev/ttyC1 /ctrscrut/chara/etc/s1_scope.cfg >& /var/log/rpc.log &\\ \\  7. Use that command to restart the server, but remove everything from the ">&" onwards and the > symbol:\\ \\  /usr/local/bin/rpc_server /dev/ttyC1 /ctrscrut/chara/etc/s1_scope.cfg &\\ \\  8. Sometimes it may complain about a lockfile. This can be removed with the command\\ \\  (s1:1003) rm /var/lock/LCK..ttyC1\\ \\  and you may have to remove the socket manger entry again.\\ \\  Some servers have to run as root, for example the wave front sensors, but most are best run as observe.\\ \\ 
===== The following servers run on CTRSCRUT. =====
\\ 
===== There is also a text file with these restart commands on the desktop. Just copy and paste the restart command after confirming that all dead processes are removed. =====
\\ 

  - Start the esp server for RETRO
\\  /usr/local/bin/tsockman remove RETRO\\  rm -f /var/lock/LCK..ttyC0\\  echo "Starting esp_retro"\\  /usr/local/bin/esp_retro /dev/ttyC0 retro >& /var/log/retro.log &\\ \\ 

  - Start the esp server for BS
**NOTE: We're currently not using BS since early 2017, instead there are BS1 and BS2 on GPS**.\\  See bootlaunch above.\\ \\ /usr/local/bin/tsockman remove BS\\ rm -f /var/lock/LCK..ttyC1\\ echo "Starting esp_bs BS"\\ /usr/local/bin/esp_bs /dev/ttyC1 bs >& /var/log/bs.log &\\ \\ 

  - Start the esp server for PERI1
/usr/local/bin/tsockman remove PERI1\\  rm -f /var/lock/LCK..ttyC2\\  echo "Starting esp_peri1 peri1"\\  /usr/local/bin/esp_peri1 /dev/ttyC2 peri1 >& /var/log/peri1.log &\\ \\ 

  - Start the esp server for PERI2
/usr/local/bin/tsockman remove PERI2\\  rm -f /var/lock/LCK..ttyC3\\  echo "Starting esp_peri2 peri2"\\  /usr/local/bin/esp_peri2 /dev/ttyC3 peri2 >& /var/log/peri2.log &\\ \\ 

  - Start the esp server for LDC1
/usr/local/bin/tsockman remove LDC1\\  rm -f /var/lock/LCK..ttyC4\\  echo "Starting esp_ldc1 ldc1"\\  /usr/local/bin/esp_ldc1 /dev/ttyC4 ldc1 >& /var/log/ldc1.log &\\ \\ 

  - Start the esp server for LDC2
/usr/local/bin/tsockman remove LDC2\\  rm -f /var/lock/LCK..ttyC5\\  echo "Starting esp_ldc2 ldc2"\\  /usr/local/bin/esp_ldc2 /dev/ttyC5 ldc2 >& /var/log/ldc2.log &\\ \\ 

  - Start the esp server for BC1
/usr/local/bin/tsockman remove BC1\\  rm -f /var/lock/LCK..ttyC7\\  echo "Starting esp_bc1 bc1"\\  /usr/local/bin/esp_bc1 /dev/ttyC7 bc1 >& /var/log/bc1.log &\\ \\ 

  - Start PICO server for PICO #1
/bin/rm -f /var/lock/LCK..ttyC8\\  /usr/local/bin/tsockman remove PICO_1\\  echo "Starting pico_server pico_1"\\  /usr/local/bin/pico_server /dev/ttyC8 /ctrscrut/chara/etc/pico_1.cfg >& /var/log/pico_1.log &\\ \\ 

  - Start PICO server for PICO #2
/bin/rm -f /var/lock/LCK..ttyC9\\  /usr/local/bin/tsockman remove PICO_2\\  echo "Starting pico_server pico_2"\\  /usr/local/bin/pico_server /dev/ttyC9 /ctrscrut/chara/etc/pico_2.cfg >& /var/log/pico_2.log &\\ \\ 

  - Start PICO server for PICO #3
/bin/rm -f /var/lock/LCK..ttyC10\\  /usr/local/bin/tsockman remove PICO_3\\  echo "Starting pico_server pico_3"\\  /usr/local/bin/pico_server /dev/ttyC10 /ctrscrut/chara/etc/pico_3.cfg >& /var/log/pico_3.log &\\ \\ 

  - Start PICO server for PICO #4
/bin/rm -f /var/lock/LCK..ttyC11\\  /usr/local/bin/tsockman remove PICO_4\\  echo "Starting pico_server pico_4"\\  /usr/local/bin/pico_server /dev/ttyC11 /ctrscrut/chara/etc/pico_4.cfg >& /var/log/pico_4.log &\\ \\ 

  - Start PICO server for PICO #5
/bin/rm -f /var/lock/LCK..ttyC12\\  /usr/local/bin/tsockman remove PICO_5\\  echo "Starting pico_server pico_5"\\  /usr/local/bin/pico_server /dev/ttyC12 /ctrscrut/chara/etc/pico_5.cfg >& /var/log/pico_5.log &\\ \\ 

  - And the server controling CCD power and other things
/bin/rm -f /var/lock/LCK..ttyC13\\  /usr/local/bin/tsockman remove RPC_LABPOW\\  echo "Starting rpc_server labpow"\\  /usr/local/bin/rpc_server /dev/ttyC13 /ctrscrut/chara/etc/labpow.cfg >& /var/log/labpow.log &\\ \\ 

  - Laser filter wheel
/bin/rm -f /var/lock/LCK..ttyC14\\  /usr/local/bin/tsockman remove Laser_Filters\\  echo "Starting thorlabs_fw102_server"\\  /usr/local/bin/thorlabs_fw102_server /dev/ttyC14 /ctrscrut/chara/etc/nd_wheel.cfg >& /var/log/filter.log &\\ \\ 

  - the server controlling the POPs
/bin/rm -f /var/lock/LCK..ttyC15\\  /usr/local/bin/tsockman remove pop\\  echo "Starting pop_server"\\  /usr/local/bin/pop_server /dev/ttyC15 >& /var/log/pop.log &\\ \\ 

  - Now start up the METLAS server
/bin/rm -f /var/lock/LCK..ttyC16\\  /usr/local/bin/tsockman remove metlas\\  echo "Starting metlas_server"\\  /usr/local/bin/metlas_server /dev/ttyC16 >& /var/log/metlas.log &\\ \\ 

  - Start the esp server for BC2
/usr/local/bin/tsockman remove BC2\\  rm -f /var/lock/LCK..ttyC17\\  echo "Starting esp_bc2 bc2"\\  /usr/local/bin/esp_bc2 /dev/ttyC17 bc2 >& /var/log/bc2.log &\\ \\ 

  - Start weather server 1 in LAB
/bin/rm -f /var/lock/LCK..ttyC18\\  /usr/local/bin/tsockman remove L1_WEATHER\\  /usr/local/bin/weather_server /dev/ttyC18 L1_WEATHER >& /var/log/lab1_weather.log &\\ \\ 

  - Server for M10 targets
/bin/rm -f /var/lock/LCK..ttyC20\\  /usr/local/bin/tsockman remove target_M10\\  /usr/local/bin/target_server /dev/ttyC20 /ctrscrut/chara/etc/target_M10.cfg >& /var/log/target_M10.log &\\ \\ 

  - And the server controling RACK3 Power
/bin/rm -f /var/lock/LCK..ttyC21\\  /usr/local/bin/tsockman remove RPC_RACK_3\\  echo "Starting rpc_server rack_3"\\  /usr/local/bin/rpc_server /dev/ttyC21 /ctrscrut/chara/etc/rack_3.cfg >& /var/log/rack_3.log &\\ \\ 

  - Start Vega Periscope server
/bin/rm -f /var/lock/LCK..ttyC24\\  /usr/local/bin/tsockman remove vega_periscope\\  echo "Starting Vega Periscope"\\  /usr/local/bin/vega_periscope_server /dev/ttyC24 >& /var/log/vega_periscope.log &\\ \\ 

  - Start PICO server for PICO #6
/bin/rm -f /var/lock/LCK..ttyC25\\  /usr/local/bin/tsockman remove PICO_6\\  echo "Starting pico_server pico_6"\\  /usr/local/bin/pico_server /dev/ttyC25 /ctrscrut/chara/etc/pico_6.cfg >& /var/log/pico_6.log &\\ \\ 

  - Start the first RPC unit
\\  /bin/rm -f /var/lock/LCK..ttyC26\\  /usr/local/bin/tsockman remove RPC_RACK_1\\  /usr/local/bin/rpc_server /dev/ttyC26 /ctrscrut/chara/etc/rack_1.cfg >& /var/log/rack_1.log &\\ \\ 

  - Start the iris server
/bin/rm -f /var/local/LCK..ttyC27\\  /usr/local/bin/tsockman remove iris\\  /usr/local/bin/iris_server /dev/ttyC27 >& /var/log/iris.log &\\ \\ 

  - Start PICO server for PICO #7
/bin/rm -f /var/lock/LCK..ttyC28\\  /usr/local/bin/tsockman remove PICO_7\\  echo "Starting pico_server pico_7"\\  /usr/local/bin/pico_server_875x /dev/ttyC28 /ctrscrut/chara/etc/pico_7.cfg >& /var/log/pico_7.log &\\ \\ 

  - Start the esp server for BC3
/usr/local/bin/tsockman remove BC3\\  rm -f /var/lock/LCK..ttyC29\\  echo "Starting esp_bc3 bc3"\\  /usr/local/bin/esp_bc3 /dev/ttyC29 bc3 >& /var/log/bc3.log &\\ \\ 

  - Start the esp server for APWHEELS
/usr/local/bin/tsockman remove apwheels\\  rm -f /var/lock/LCK..ttyC30\\  echo "Starting Apwheels"\\  /usr/local/bin/apwheels_server /dev/ttyC30 /ctrscrut/chara/etc/apwheels.cfg >& /var/log/apwheels.log &\\ \\ 

  - Start the second RPC unit
\\  /bin/rm -f /var/lock/LCK..ttyC31\\  /usr/local/bin/tsockman remove RPC_RACK_2\\  /usr/local/bin/rpc_server /dev/ttyC31 /ctrscrut/chara/etc/rack_2.cfg >& /var/log/rack_2.log &\\ \\  #Run Andor Technology Startup Script\\  sh /etc/andordrvlx_load DMA_MODE=1\\ \\ 
===== Each telescope runs the same servers on its computer: =====
\\ **NOTE: The RPC, weather, cylinder, and HUT servers should be restarted using the bootlaunch commands described above**\\ \\ 

  - Now start up the RPC server
\\ /bin/rm -f /var/lock/LCK..ttyC1\\ /usr/local/bin/tsockman remove RPC_S1\\ /usr/local/bin/rpc_server /dev/ttyC1 /ctrscrut/chara/etc/s1_scope.cfg >& /var/log/rpc.log &\\ \\ 

  - And the weather server
\\ /bin/rm -f /var/lock/LCK..ttyC7\\ /usr/local/bin/tsockman remove S1_WEATHER\\ /bin/nice /usr/local/bin/weather_server /dev/ttyC7 S1_WEATHER >& /var/log/weather.log &\\ \\ 

  - And the cylinder servers
\\ /bin/rm -f /var/lock/LCK..ttyC3\\ /usr/local/bin/tsockman remove S1_Lower\\ /usr/local/bin/cylinder_server /dev/ttyC3 /ctrscrut/chara/etc/S1_lower_cylinder.cfg >& /var/log/cyl_lower.log &\\ \\ #/bin/rm -f /var/lock/LCK..ttyC4\\ #/usr/local/bin/tsockman remove S1_Upper\\ #/usr/local/bin/cylinder_server /dev/ttyC4 /ctrscrut/chara/etc/S1_upper_cylinder.cfg >& /var/log/cyl_upper.log &\\ \\ #\\ 

  - Dome Humidity/Temperature Sensor * See below for additional instructions.
#\\ \\ #/chara/observe/huT-1.0/huT 7 30 1 S1huT.dat 0 &\\ \\ rm -f /var/lock/LCK..ttyC15\\ /usr/local/bin/tsockman remove S1_HUT\\ /usr/local/bin/new_huT_server /dev/ttyC15 S1_HUT /ctrscrut/chara/etc/S1_aob.cfg >& /var/log/S1_huT.log &\\ \\ 

  - Run the dome server
\\  /usr/local/bin/tsockman remove dome_S1\\  /usr/local/bin/dome_server -A41.0166 S1 >& /var/log/dome_S1.log &\\ \\ 

  - Make sure our clock is good
/usr/local/bin/ntp2linux\\ \\ 

  * Sometimes the HUT server fails to update the camera settings such as FINDER and ACQ exposure times and gains. Most of the time, this can be fixed by restarting the HUT server (see instructions above - log onto the telescope computer, kill any dead process, remove from sockman, and restart using the bootlaunch_hut command). If the server won't restart, a reboot of the power supply in the telescope bunker might be necessary. The power supply that controls the acquisition and finder cameras as well as their controllers is located on top of the computer rack in each bunker. The power supply has green readouts of volts and current. After turning the power off for 10 seconds and back on, try restarting the server from the computer in the bunker to see if it starts cleanly. Part of the HUT server also controls the AO table. If the AOB part of the HUT server doesn't work, then the power supply on the back of the AO table in the telescope dome might need recycling. This power supply controls the actuators at M2 and the AO table. The power supply box is a 6x9 inch aluminum box on the back of the AO table, behind the keyboard and monitor. Turn it off with the power button on the bottom edge of the box, wait 5 seconds and turn it back on. The HUT server should now restart cleanly. Restart the telescope server as well to make the connections to the telescope gui. Hit REOPEN in Cosmic Debris if you are observing to make all needed connections.
\\ 
===== The BRT, ZABER_1, and PICO_8 servers run on irimage: =====
\\  #\\ 

  - Start Zaber 1 server
#\\ \\  /usr/local/bin/tsockman remove ZABER_1\\  /bin/rm -f /var/lock/LCK..ttyS0\\  echo "Starting zaber_server ZABER_1"\\  /usr/local/bin/zaber_server /dev/ttyS0 /ctrscrut/chara/etc/zaber_1.cfg >& /var/log/zaber_1.log &\\ \\ 

  - Start BRT ESP servers
\\  /usr/local/bin/tsockman remove BRTA\\  rm -f /var/lock/LCK..ttyS1\\  echo "Starting esp_brta BRTA"\\  /usr/local/bin/esp_brta /dev/ttyS1 /ctrscrut/chara/etc/brta.cfg >& /var/log/brta.log &\\ \\  /usr/local/bin/tsockman remove BRTB\\  rm -f /var/lock/LCK..ttyS2\\  echo "Starting esp_brtb BRTB"\\  /usr/local/bin/esp_brtb /dev/ttyS2 /ctrscrut/chara/etc/brtb.cfg >& /var/log/brtb.log &\\ \\ 

  - Start PICO server for PICO #8
\\  /bin/rm -f /var/lock/LCK..ttyS3\\  /usr/local/bin/tsockman remove PICO_8\\  echo "Starting pico_server pico_8"\\  /usr/local/bin/pico_server /dev/ttyS3 /ctrscrut/chara/etc/pico_8.cfg >& /var/log/pico_2.log &\\ \\ 
===== The ZABER_2 server runs on GPS: =====
\\ **NOTE: ZABER_2 on GPS should be started using the bootlaunch command above**.\\ \\ #\\ 

  - Start the ZABER_2 server
#\\ \\ /usr/local/bin/tsockman rm ZABER_2\\ /bin/rm -f /var/lock/LCK..ttyS1\\ 

  * /usr/local/bin/zaber_server /dev/ttyS1 /ctrscrut/chara/etc/zaber_2.cfg >& /var/log/zaber_2.log &
The CHARA Array Wiki

User Tools

Site Tools

Page Tools
