chara:trouble_shooting
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
chara:trouble_shooting [2023/03/25 01:06] cyprien |
chara:trouble_shooting [2023/07/06 07:45] gail_stargazer |
||
|---|---|---|---|
| Line 11: | Line 11: | ||
| * Did the Astrolib update on OPLE? If the job queue is stopped too soon after slewing on Cosmic Debris, the correct calculations for the carts will not be done by OPLE and you may be searching for fringes with the wrong star data. The star identifier will be displayed in the server window. If it is not correct, hit STAR ACQUIRED on CD to update OPLE. The proper star can also be entered manually by typing hd #### into the OPLE server and hitting ENTER. | * Did the Astrolib update on OPLE? If the job queue is stopped too soon after slewing on Cosmic Debris, the correct calculations for the carts will not be done by OPLE and you may be searching for fringes with the wrong star data. The star identifier will be displayed in the server window. If it is not correct, hit STAR ACQUIRED on CD to update OPLE. The proper star can also be entered manually by typing hd #### into the OPLE server and hitting ENTER. | ||
| * Are the PoP's correct? After a PoP change, the PoP's are sometimes not updated in CD or ople. Compare the PoP's in the configuration tab of CD or OPLE with the PoP Overview or Popperi gtk. | * Are the PoP's correct? After a PoP change, the PoP's are sometimes not updated in CD or ople. Compare the PoP's in the configuration tab of CD or OPLE with the PoP Overview or Popperi gtk. | ||
| - | * Are the carts behaving or are there vibrations or jumps of 100 or more microns every 3-6 seconds? How are the metrology signals? Are they strong and staying white? Red signals mean one or more metrology signals may have gone too low and homing carts is necessary. The METDATA function on the ople gui can be used to sample the metrology signal and may show noise spikes that can be disruptive to the smooth cart motion. | + | * Are the carts behaving or are there vibrations or jumps of 100 or more microns every 3-6 seconds? How are the metrology signals? Are they strong and staying white? Red signals mean one or more metrology signals may have gone too low and homing carts is necessary. The METDATA function on the ople gui can be used to sample the metrology signal and may show noise spikes that can be disruptive to the smooth cart motion. |
| - | * Is the target a high proper motion star? Red dwarfs are close stars and can have high proper motions. Scan a wider range to see if it is outside of the usual calculated scan range. Binaries can also have very high offsets from the expected position due to mistakenly using astromod calculations from the companion star. | + | * Is the target a high proper motion star? Red dwarfs are close stars and can have high proper motions. Scan a wider range to see if it is outside of the usual calculated scan range. Binaries can also have very high offsets from the expected position due to mistakenly using astromod calculations from the companion star. Make sure you are on the right star with an image from SIMBAD or Vizier. |
| - | * Do you have enough flux from each telescope or on each baseline? Is the telescope tiptilt struggling with low flux? Can the camera gain be raised or the exposure made longer to help hold the star? Passing clouds or contrails can lower flux unexpectedly. | + | * Do you have enough flux from each telescope or on each baseline? Is the telescope tiptilt struggling with low flux? Can the camera gain be raised or the exposure made longer to help hold the star? Passing clouds or contrails can lower flux unexpectedly. Remap or realign as needed. |
| * Did you get the same star in each telescope? Sometimes a busy star field and poor pointing of the telescopes can lead to the wrong star being acquired and locked by tiptilt. View the stars in the finder window to see if all the stars match. | * Did you get the same star in each telescope? Sometimes a busy star field and poor pointing of the telescopes can lead to the wrong star being acquired and locked by tiptilt. View the stars in the finder window to see if all the stars match. | ||
| * Check the CHARA time on the GPS server. The " | * Check the CHARA time on the GPS server. The " | ||
| * Check the time on the OPLE server. If the time is off or there are any lost ticks/ | * Check the time on the OPLE server. If the time is off or there are any lost ticks/ | ||
| - | * Are the [MAN] buttons pressed (gray) for the moving carts on the OPLE Control gui? (The reference cart will remain green.) | + | * Are the [MAN] or [FT] buttons pressed (gray) for the moving carts on the OPLE Control gui? (The reference cart will remain green.) |
| - | * Check that the carts are within delay line range (-1.3 to 44.25 meters) and errors are small. Were the carts homed and checked before the first slew of the night? Did any carts go to the front switch after slewing? This can cause them to lose their position. To check the homing of the carts, turn off the OL button to send it back to zero. If it does not track on the home switch when displaying a zero position (0.000000), it was lost. There is a new "CHECK" | + | * Check that the carts are within delay line range (-1.3 to 44.25 meters) and errors are small. Were the carts homed and checked before the first slew of the night? Did any carts go to the front switch after slewing? This can cause them to lose their position. To check the homing of the carts, turn off the OL button to send it back to zero. If it does not track on the home switch when displaying a zero position (0.000000), it was lost. Use the [CHECK] button on the OPLE gtk to send the carts to the home switch and report whether |
| - | * Are the LDC's working? | + | * See section on homing carts in __OPLE and Metrology__ |
| - | * Is the glass position within allowable range (-10mm to 49mm) on all beams? | + | * For the visible programs, are the LDC's working? |
| + | * Is the glass position | ||
| * Have the commands " | * Have the commands " | ||
| - | * Look out for " | + | * Look out for errors on ople server. If the LDC glass goes out of range, the LDC will need to be homed. To home the glass, stop astromod by clicking [STOP] on the OPLE gui. Make sure the LDC velocities are non-zero (~ 50… must click [vel] button to update velocity) and click [HOME] on the LDCs in use. The LDC position number should go to 0.000 or very close to zero. After the LDCs have been homed, turn astromod on by clicking [START] on the ople gui. Then click [REF] on cosmic debris to send the LDCs to the correct positions. |
| * If LDCs are working correctly, the LDC positions (in mm) should roughly match up with the reference cart positions (in meters). | * If LDCs are working correctly, the LDC positions (in mm) should roughly match up with the reference cart positions (in meters). | ||
| Line 44: | Line 45: | ||
| If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[: | If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[: | ||
| - | To restart a server, log on to the machine that runs the server and type " | + | To restart a server, log on to the machine that runs the server and type " |
| - | ==== Restarting Servers using the rc.local file ==== | + | ==== Restarting Servers using the rc.local file (likely outdated) |
| \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm, which may not be any at this point. \\ \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[: | \\ This procedure is applicable to servers that have not switched over to the bootlaunch paradigm, which may not be any at this point. \\ \\ If a server is not running or Socket Manager reports that a server is dead, then look at the socket manager list to find out what computer the server runs on ([[: | ||
| Line 54: | Line 55: | ||
| Shutters Server | Shutters Server | ||
| - | \\ The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on ople, not ctrscrut. \\ \\ To start the shutter server on gps: \\ \\ Log into the gps computer and kill the process labeled shutters with the PID as described in **Restarting Servers** | + | \\ The Shutters server can become unresponsive or disconnected from the Socket Manager. This server must be restarted from the lab and not from the Control Room. Follow these instructions to restart it. Note that Shutters runs on gps, not ctrscrut. \\ \\ To start the shutter server on gps: \\ \\ Log into the gps computer and kill the process labeled shutters with the PID as described in **Restarting Servers** |
| ==== Restarting Socket Manager ==== | ==== Restarting Socket Manager ==== | ||
| Line 86: | Line 87: | ||
| ==== The Telescope won't move or stopped moving ==== | ==== The Telescope won't move or stopped moving ==== | ||
| - | Have the powers been turned on to the drives? Are the scopes disabled? The usual state of the telescopes is disabled until enabled. This is due to the stall function of the scopes which eventually disables the scopes when they are stowed. Enable the scopes by hitting [ENABLE] on the dome gui, the telescope gui control tab, or the bottom of the obsgtk. If a scope disables itself during a slew, it may be just an overcautious stall function or the slew gains may be too high or low. Check the gains and if they are OK, Hit ENABLE and the scope should continue to slew. If it disables again without moving, there may be something wrong at the scope, ie. a hatch left open, a ladder not put away, a tool on the floor, or something else physically impeding the motion of the scope. You will need to go to the dome to see what it is. The computer in the dome will give you control of the scope to turn it away from the problem. | + | Have the powers been turned on to the drives? Are the scopes disabled? The usual state of the telescopes is disabled until enabled. This is due to the stall function of the scopes which eventually disables the scopes when they are stowed. Enable the scopes by hitting [ENABLE] on the dome gui, the telescope gui control tab, or the bottom of the obsgtk. If a scope disables itself during a slew, it may be just an overcautious stall function or the slew gains may be too high or low. Check the gains and if they are OK, Hit ENABLE and the scope should continue to slew. If it disables again without moving, there may be something wrong at the scope, ie. a hatch left open, a ladder not put away, a tool on the floor, or something else physically impeding the motion of the scope. You will need to go to the dome to see what it is. The computer in the dome will give you control of the scope to turn it away from the problem. Make sure the scope is in a position that allows the opening of the hatch if you need to go upstairs. It can be moved back from the bunker as needed. |
| Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopes. Check them for current times and continuous updates of numbers. If they are not updating, try to REOPEN them first. If that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of it. See **Dome Server Restart** | Sometimes the dome guis get hung up and can cause erratic motion or no motion of the scopes. Check them for current times and continuous updates of numbers. If they are not updating, try to REOPEN them first. If that does not work, close the gui and open a new one. If a new one does not open, the dome server may be dead or Sockman lost track of it. See **Dome Server Restart** | ||
| Line 92: | Line 93: | ||
| 1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software. \\ | 1. On the domegui MANUAL tab, click STOP so pulses won't be sent to the drive by the control software. \\ | ||
| \\ | \\ | ||
| - | 2. Make sure you understand why the limit was hit which may require a trip to the telescope. If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch causing the stall and not that the scope is actually in a wrong position.</ | + | 2. Make sure you understand why the limit was hit which may require a trip to the telescope. If the azimuth positions on all telescope servers and dome guis match, it is likely the limit switch |
| 4. Click ENABLE then you can move the telescope back to its normal range of operation. \\ \\ | 4. Click ENABLE then you can move the telescope back to its normal range of operation. \\ \\ | ||
| 5. After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again and then hit AUTO on the AUTO tab to resume normal operation.</ | 5. After the telescope is back in it normal range, click OVERRIDE OFF which makes the hardware aware of the limits again and then hit AUTO on the AUTO tab to resume normal operation.</ | ||
| Line 103: | Line 104: | ||
| <font 14px/ | <font 14px/ | ||
| - | <font 14px/ | + | <font 14px/ |
| <font 14px/ | <font 14px/ | ||
| Line 201: | Line 202: | ||
| ==== Dome does not rotate ==== | ==== Dome does not rotate ==== | ||
| - | Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope' | + | Sometimes when observing, the dome will not follow the telescope during a slew. This can happen when the Autodome feature is not turned on. Click the ON button on the MAIN tab of the telescope gui to enable it. This may happen after a server restart so always check the dome position with the spycam during a slew after a server restart. Also make sure the target position of the dome matches the telescope' |
| If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box instead of computer. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check the controller box just inside the door of the bunker. Sometimes the drive wheel jumps in the track and cannot turn the dome, even when the motor works. This will need to be fixed during the day. A fuse can also blow out in the dome control box and leave the dome stuck. | If the dome does not turn at all, even with the manual controls on the telescope or dome guis, the control may be set to manual on the control box instead of computer. This can happen if there was work done at the dome during the day. If the dome opens, but does not turn, check the controller box just inside the door of the bunker. Sometimes the drive wheel jumps in the track and cannot turn the dome, even when the motor works. This will need to be fixed during the day. A fuse can also blow out in the dome control box and leave the dome stuck. | ||
| Line 232: | Line 233: | ||
| ==== LabAO DM Electronics ==== | ==== LabAO DM Electronics ==== | ||
| - | If a labao server complains that it failed to find the EDAC40 device, try unplugging the power and ethernet cables for DM and then replugging in again. | + | If a labao server complains that it failed to find the EDAC40 device, try unplugging the power and ethernet cables for DM and then replugging in again. The boxes for these are under the respective tables. |
| ===== OPLE and Metrology ===== | ===== OPLE and Metrology ===== | ||
| Line 248: | Line 249: | ||
| * This will install run_ople. Try opening the ople server again. | * This will install run_ople. Try opening the ople server again. | ||
| - | ==== Metrology and Homing carts ==== | + | ==== Homing carts ==== |
| - | * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror. | + | * |
| - | | + | |
| + | To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has stopped on the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. The X does not guarantee the cart has retained its home position. This home position can be checked by using the CHECK button on the ople gui or by typing the command | ||
| + | |||
| + | ==== Metrology ==== | ||
| + | |||
| + | * | ||
| + | |||
| + | Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror | ||
| + | |||
| + | Steps for adjusting MET2: | ||
| + | |||
| + | - Looking at the met signal while the cart is at the back. | ||
| + | - Peak so that the beam is aligned as well as possible. Does not need to be perfect but not loose. | ||
| + | - All beams have a signal between 1-2 volts while at the back. Sometimes the signal strength can change for no apparent reasons. | ||
| + | - Using steps of 50, try moving 3 times in one direction. After each step, wait 5 seconds or so for things to update. If nothing then try the other way. | ||
| + | - Return to starting position. | ||
| + | - Repeat for the other axis. | ||
| ===== Beam Samplers ===== | ===== Beam Samplers ===== | ||
| - | ==== Beam Samplers are not moving ==== | + | ==== Beam Samplers are not moving |
| This happens most frequently with the E1 beam sampler. If you are doing alignments and are using E1, try to move it forward first to see if it dies. If it does, follow this procedure. | This happens most frequently with the E1 beam sampler. If you are doing alignments and are using E1, try to move it forward first to see if it dies. If it does, follow this procedure. | ||
| Line 359: | Line 376: | ||
| You can also look up different identifiers for the star by using the SIMBAD database: [[http:// | You can also look up different identifiers for the star by using the SIMBAD database: [[http:// | ||
| - | DBADD uses SIMBAD to look up the coordinates of the star and then searches to see if the star is already in the CHARA database. If the star is in the database, it will return the CHARA number. If the star is not in the CHARA database, it will ask if you want to add it and assign a CHARA number if you answer yes. | + | DBADD uses SIMBAD to look up the coordinates of the star and then searches to see if the star is already in the CHARA database. If the star is in the database, it will return the CHARA number. If the star is not in the CHARA database, it will ask if you want to add it and will assign a CHARA number if you answer yes. |
| If an object is not in SIMBAD (such as a recently discovered nova), then you must enter the object manually using the instructions below. | If an object is not in SIMBAD (such as a recently discovered nova), then you must enter the object manually using the instructions below. | ||
| Line 436: | Line 453: | ||
| \\ | \\ | ||
| - | Last updated 2023-02-09 by Norm Vargas | + | Last updated 2023-02-09 by Norm Vargas |
| - | + | ||
| - | ==== Dome Server Restart ==== | + | |
| - | + | ||
| - | Dome servers are now started using the bootlaunch_master command. The manual process that has been superceded is archived. | + | |
| - | + | ||
| - | To restart the dome server: | + | |
| - | + | ||
| - | 1. Make sure the power to the drives is OFF. Disable the scopes. | + | |
| - | + | ||
| - | 2. Login to the relevant computer as observe. For example, type " | + | |
| - | + | ||
| - | 3. Work out the process ID number (PID) by typing bootlaunch_master | + | |
| - | + | ||
| - | 4. Use kill -2 PID to kill the server. Entering the command twice will show "No such process" | + | |
| - | + | ||
| - | 5. Type bootlaunch_master again to restart the server. | + | |
| - | + | ||
| - | 6. Turn the power to the drives back on. | + | |
| - | + | ||
| - | 7. Hit REOPEN and ENABLE on the domegtk, and type " | + | |
| - | + | ||
| - | You may have to reinitialize the scope on a bright star. If the powers were turned off quickly when the problem was noticed, the position of the scope should be retained and slewing to a bright star will get it in the finder. If not, you may need to go out to the telescope to find the bright star to reacquire the scope' | + | |
| - | + | ||
| - | Be aware that when restarting a dome server, the telescopes position may not be retained and the dome gui may display Az: 0.0. El: 0.0. The gui will scroll messages that say Az: 0.0. El: 0.0 with an error in scope position. If the scope is known to be at STOW position, you can manually enter the scope' | + | |
| - | + | ||
| - | Also be aware that the gains do not always return to the normal values of Slew 4-7 and Tracking 7-10 when a dome server has crashed and is restarted. They may come up as Gain 22 for all axes and modes and they will need to be turned down or the scopes will oscillate badly. | + | |
| ==== Telescope is not receiving the commanded position for a target. ==== | ==== Telescope is not receiving the commanded position for a target. ==== | ||
| Line 563: | Line 554: | ||
| * This will install run_ople. Try opening the ople server again. | * This will install run_ople. Try opening the ople server again. | ||
| - | ==== Metrology and Homing carts ==== | + | ==== Homing carts ==== |
| - | * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror. | ||
| * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has stopped on the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. The X does not guarantee the cart has retained its home position. This home position can be checked by using the CHECK button on the ople gui or by typing the command " | * To home the carts, turn off the [OL] and [MAN] buttons on each cart and it will automatically return to the front switch. If the cart has no issue, it will arrive at the target position of 0m and the home switch at the same time. The X in the OPLE server under the HM will indicate it has stopped on the home switch. If a cart does not reach the home switch when it returns to position 0m, it was lost and likely the cause of the difficulty in finding fringes. The X does not guarantee the cart has retained its home position. This home position can be checked by using the CHECK button on the ople gui or by typing the command " | ||
| + | |||
| + | ==== Metrology ==== | ||
| + | |||
| + | * Good metrology signals are important to the proper positioning of the carts. Monitor the signal strength by running [RUN MULTIPLE] on the Metrology Monitor. Place the windows above the TV windows for each scope you are using. They should show white sine waves that are around the height of the window. Erratic, fluctuating waves indicate self-interference or a weak signal. This may cause the carts to lose their place as the signal strength falls too low. Red waves indicate that some displayed signal has gone too low and the carts will all need to be homed. A careful adjustment of the MET2 mirror can sometimes bring the signal back. Do not adjust the MET1 mirror | ||
| + | |||
| + | Steps for adjusting MET2: | ||
| + | |||
| + | - Looking at the met signal while the cart is at the back. | ||
| + | - Peak so that the beam is aligned as well as possible. Does not need to be perfect but not loose. | ||
| + | - All beams have a signal between 1-2 volts while at the back. Sometimes the signal strength can change for no apparent reasons. | ||
| + | - Using steps of 50, try moving 3 times in one direction. After each step, wait 5 seconds or so for things to update. If nothing then try the other way. | ||
| + | - Return to starting position. | ||
| + | - Repeat for the other axis. | ||
| ===== Beam Samplers ===== | ===== Beam Samplers ===== | ||
| Line 751: | Line 754: | ||
| \\ | \\ | ||
| - | Last updated | + | Last updated |
chara/trouble_shooting.txt · Last modified: 2024/06/18 00:21 by charaobs
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
