Differences

This shows you the differences between two versions of the page.

--- chara:new_operating_procedures [2025/06/13 19:48]
gail_stargazer
+++ chara:new_operating_procedures [2025/06/18 20:29] (current)
kkubiak [Setting Threshold for WFS]
@@ Line 922: / Line 922: @@
 For programs with bright IR stars, skip align beacon, but do the M7 and labao focus alignments. Then lock the star on the blue boxes and use the STST to fine tune any misalignment of the beacon flat. Check the beacon focus on the twfs occasionally to make sure it stays near zero, every 2-3 stars.
-==== Trouble Shooting - AO Performance ====
+==== Trouble Shooting - AO Performance - Obsolete ====
 If AO performance is bad try these steps in the following order:
@@ Line 934: / Line 934: @@
   * If X/Y and higher order terms are still not going to zero after checking alignment, making a new flat, and zeroing centroids, then this likely means a new reconstructor needs to be made. Notify charatech (maybe try taking onsky reconstructor).
-==== Trouble Shooting - Locking TT and AO Loops ====
+==== Trouble Shooting - Locking TT and AO Loops - Obsolete ====
   - If **not locking**, make sure red beacon is off, make sure on blue boxes, load def flat or last flat, make sure no clouds
@@ Line 947: / Line 948: @@
       - Select [FAINT], wait til wfs starts moving, relock TT DM.
   - After TT DM, Zdark if gain is not 300
 ===== Optimizing AO Performance =====
@@ Line 962: / Line 964: @@
 For the VIS dichroics, the WFS gets only 20% of the light, but 1 msec should still get to magnitude 5.0.
-==== Setting Threshold for WFS ====
+==== Setting Threshold for WFS ==== - Obsolete
 If the starlight is missing from the boxes on the TWFS on S2, you might be able to improve performance on by applying a threshold to ignore the TWFS boxes without flux This is likely caused by an alignment problem at the telescope/AOB, so notify charatech if you apply a threshold. Here are the steps to try this approach again on sky:
@@ Line 974: / Line 976: @@
   - NOTE: <font 11pt/inherit;;inherit;;inherit>The thresholding works by setting everything below the threshold to</font><font 11pt/inherit;;inherit;;inherit>zero, so if it is too high it will black out the TWFS. You want</font><font 11pt/inherit;;inherit;;inherit>the threshold to be as low as possible while still actually helping out.</font>
-==== Offloading Telescope Aberrations ====
+===== Going on Sky =====
+==== Opening the Telescope Domes and enclosures ====
+  - Before opening the domes make sure to check the current weather conditions and the forecast to make sure conditions are safe to open and are likely to be stable as you wait for it to get dark.
+  - Open the dome slit if haven't already (high humidity).
+      - **obsgtk, **Control Tab, click **[UNL SL]**  and **[SLIT]**
+  - Monitor the progress of opening the domes:
+      * **Obsgtk, [ON], [SPY1]**
+==== Opening Telescope Optics ====
+  - **Obsgtk**, Control Tab, open mirror covers. Click only one cover at a time and wait for it to open. Click each button only once, do not click repeatedly:
+      - [M1 OPEN] - opens the primary mirror cover (takes ~ 3 minutes to open) Only open after 20 minutes after sunset.
+      - [M3 OPEN] Same as above.
+      - [M5 OPEN] - if not already open
+      - [M7 OPEN] - if not already open
+      - [FIND OPEN] if you want the finder. Note that the W2 and E1? finder covers require two clicks of the [OPEN] button to fully open.
+  - Monitor progress of the M1 and M3 covers by using [SPY2] camera on the telescope TV screens. Wait for all covers to open.
+  - The Telescope Monitor reports the status of all mirror covers. Close and reopen the Telescope Monitor if it reads UNKNOWN for a telescope, even after hitting REOPEN.
+==== Final Alignment Before Slewing ====
+  - While still at stow, align red and blue beacon and bring focus terms to zero. Be aware that SPICA sees the red beacon and may want it off when they do STS recording.
+  - If the aberration terms are high, zero out the centroids by clicking** [ZCEN]**  on all scopes. Doing zero centroids will help with the labao corrections once on sky.
+==== Enabling The Telescopes ====
+  - Make sure telescopes are at stow and have no demand positions
+      - El: 90 for all
+        - [SPY1] should show the end ring of the telescope framing the top of the open dome.
+      - Az:
+        - S’s: 82.0
+        - E’s: 55.9
+        - W’s: 99.3.
+  - if at 0,0, the numbers are fake and must not enable until it has correct stow positions.
+      - *confirm/update* need to get an idea of where telescope is by going to the dome
+        - behind the scope on the ground, a black tape markes stow position.
+      - then in dome GUI, manual tab, init, then stow.
+  - Make sure there are no demand positions that are different from stow positions (on telescope servers make sure it shows stow position) or the scopes may begin moving without being commanded. Hit **[STOW]**  if any demand positions are not at the stow positions.
+  - Enable the telescopes only when ready to slew.
+      - **Obsgtk**, **[ENABLE]**
+      - Make sure that the telescopes do not move away from their stow positions when enabling the scopes!
+      - If they begin moving, **obsgtk,**  control tab, **[STOW] **and make sure that they arrive at the correct stow position.
+      - Check for problems with the dome servers by watching each telescope move to the precise stow position, ie. 89.990º and moving to 90.000º.
+      - If it stays off the precise stow position, the dome severs may need restarting and the Gains/Fn's will be wrong and need to be brought down to normal.
+      - Click **[REOPEN] **on the dome gui if the clock is not the same as the CHARA time.
+===== Slewing and Locking on a Target =====
+It is typically dark enough to lock on your first star roughly 25 minutes after sunset or the time of twilight reported by Cosmic Debris.
+If the first science target/calibrator is faint in V-band, slew to a brighter star to initialize the telescopes, make flats, and do STST alignments.
+Slewing to a Target Using Cosmic Debris
+  - Enter sci and cals into CD
+      - If a star or new target is not recognized by Cosmic Debris, [[:chara:trouble_shooting#adding_or_finding_a_star_in_the_chara_database|follow the procedure for DBADD]].
+      - Check to make sure that EL/AZ reported by Cosmic Debris are correct and safe for pointing
+      - **Never point below 15-20 degrees elevation.**
+      - It is recommended to initialize the telescopes on a target between 40-70 degrees elevation.
+  - Hit the **WHEN**  button for your first star to see a plot of cart positions and when it is in delay.
+  - Control Tab, Enter ideal reference cart pos into the REF text window, click **[REF]**
+      - Press **[Opt delay] **about 5-10 sec after star acq and enter those values
+  - On the Star/Calibrator Tab, click **[Slew To]**
+  - **Verify that the telescopes are moving**  by checking the azimuth and elevation status in the telescope servers
+      - //The left set of numbers show the commanded EL/AZ of the target while the right set of numbers show the current location of the telescope (where)//
+      - If the scopes do not move, check the following:
+        - check that the powers are on
+        - check that they are enabled. The telescope server will show RA and DEC coordinates if enabled and will show DISABLED if disabled.
+        - check that its in the config
+        - press reopen on CD and ople gui
+        - restart scope server then reopen guis
+      - Also view “SPY1” on **obsgtk**  to check that the telescopes and domes are moving.
+      - If the domes do not rotate, **Dome **GUI, click the** [AUTODOME ON] **button to enable the domes
+      - Scopes will usually arrive at the elevation of the target first and then continue in azimuth until the star appears in the finder or ACQ window.
+      - When the telescope slew finishes, check the SPY 1 to make sure that the domes are aligned with the telescope if no star shows up in the ACQ or Finder cameras.
+        - If the star is not in the ACQ window, go to the FIND window and do the same, then return to the ACQ window and repeat.
+  - Make sure all carts are tracking to values between 0 and 44m.
+      - Make sure** [TRACK]**  is on for all active carts, including the reference cart.
+      - The** [OL] buttons should be gray for ALL active carts**  (including the reference cart).
+        - **MIRCX/MYSTIC/SPICA**: The** [FT]**  buttons should be gray for all active scopes BUT the mircx reference cart (W2)
+        - **PAVO/CLASSIC**: The **[MAN]**  buttons should be depressed gray for the moving carts
+          - The [MAN] button for the reference cart should be green.
+==== Locking on a Target ====
+Locking on a star involves aligning the beacons, locking the star on tiptilt and tel AO, and locking the blue beacon on labAO. TelAO performs fast correction to compensate for atmospheric turbulence. LabAO performs slow correction to compensate for non-common path length errors along the beam path from the telescope to the lab.
+The path of the visible and IR light into the lab is different because of atmospheric refraction and dispersion in the AO dichroic. Below we outline alignment steps separately for visible and IR combiners. For IR combiners we use the Six Telescope Star Tracker to align the IR light in the lab.
+While tracking on a star, labao will offload X,Y tilt to M7 to keep the the blue beacon centroids aligned to labAO WFS. There are two methods for doing this:
+**Old method**: DMTILT is gray (reconstructor ignores tilt). **M7AUTO**  will adjust M7 to keep blue beacon centroids aligned to labAO WFS.
+**New method**: **DMTILT**  is green (uses tilt when computing reconstructor - "Not ignoring tilt"). **DMAUTO**  will use labao DM to correct for tilt and offload to M7 to avoid reaching its tilt limit.
+**STOP A: **If needed, the 3 STOP A buttons on obsgtk will stop any alignment sequences in progress.
+==== Alignment Sequence for Visible Combiners: SPICA and PAVO ====
+For the visible combiners the red beacon and starlight will be coaligned, so you only need to use the blue starlight boxes on the telescope WFS.
+**Obsgtk, **main tab, **[ZERO BCN]**  at the start of the night to clear any offset between the position of the beacon boxes and the starlight boxes to avoid any confustion
+For **SPICA**, make sure to do "**useldc **on" in ople server if you haven't done so already.
+  - Slew to star.
+  - Main tab, turn on **[M7AUTO] **to hold the blue beacon during the slew and avoid drifting. It will turn off at the end of the slew.
+  - CD, **[Star Acquired],**  and set new ref pos with **[Opt delay]**
+      - **FOR PAVO: **Do **<font inherit/inherit;;#e74c3c;;inherit>__NOT __</font>**hit **[Star Acquired] **before aligning for PAVO, only do **[STAR ACQUIRED]**  after alignment is finished.
+  - Obsgtk, set EXP to 0.001 ms and GAIN to 800
+      - for spica, or bright stars (should get a preset soon)
+  - **[ALN BCN], **Align red beacon to the blue starlight boxes on the telescope WFS using the beacon flat.
+      - Wfs tab, Use beacon buttons if needed on W1 if beacon gets out of the boxes
+      - or hit RIGHT with 500 stepsize to get them back in.
+  - LABAO section, **[ALN M7], **Align blue beacon to lab WFS using M7.
+      - WFS tab, use the yellow M7 buttons to get centered if beacon is not in boxes, then **[ALN M7]**
+      - 150,000 step size moves 1 full row. Directions are normal.
+  - LABAO section, **[FOCUS] **blue beacon on labao WFS if the focus term is high (moves beacon parabola).
+  - TELWFS section, **[FOCUS] **red beacon on TWFS if focus term is high
+  - **[RED OFF] **Turn off red beacon.
+  - Click **[MOVE]**  button then click on star in ACQ window. This will move star into ACQ hole.
+  - Turn off blue beacon and lock star on **[TT]**  only
+  - Move TWFS boxes (**ADJ**  popup button on obsgtk) to get **star centered and evenly illuminated on ACQ hole**. (Note: you might need to close DM AO loop during this step.)
+  - Lock star on telescope AO servo (**[****DM]**  button on obsgtk). Both TT and DM servos should be on now.
+  - Turn on the blue beacon
+  - Make sure **DMTilt is green**  (stays on all the time - the message from DM tilt “NOT Ignoring DM tilt…” when it is on. This may take 2 clicks to confirm this message.)
+  - Turn on **DM AUTO**, **LABAO**.
+  - **For PAVO: **make sure carts are in position, then hit **[STAR ACQUIRED**] once alignments are finished. Star acq. starts the camera on pavo.
+  - **For SPICA**:
+      - **ip_det_gtk **  (blue chara menu)
+      - **[Start ACQ] **
+      - can adjust display with 200 - 2000 in the max, then hit **[SET]**.
+      - press **[BF/M7 CONTROL] **to open pop up control to align.
+        - use step size of about 32 if on the edge to bring to center. Directions are normal (ex. right is right)
+      - Use **STST **as a reference, if spots are far on ipdet display, stst is probably really off too. Use stst to bring spots close to center and it should help bring spots close to center on ipdet, then use BF/M7 to finish aligning.
+      - let SPICA team know, they may or may not want you to turn on FTT/SERVO.
+      - Once aligned, on STST GUI, press **[CenBox]**  to mark new reference position to monitor any drift.
+      - Make sure data acquired line comes in on CD, otherwise do **[DATA ACQ]. **
+        - or make sure to **[REOPEN]**  CD during shutters
+  - If the coma and astigmatism terms are high on the aberration plots, create a new sky flat
+      - Turn off **[DM]**, **[SV FLT]**, **[FLATEN]**, then relock the **[DM]**.
+      - If the signal is good on the tel WFS, it is a good idea to make a new sky flat on every star.
+  - **Check aberration plots.**  If X/Y terms are not going to zero, try stopping servo loops, recentering alignment, and turning back on servo loops. If X/Y and higher order terms still not going to zero after checking alignment and making a new flat, then this likely means a new reconstructor needs to be made.
+  - SPICA uses MIRCX as a fringe tracker and MYSTIC as a fringe finder so having the STST open to monitor the IR beam quality is helpful even if you will not be doing the additional STST beacon flat alignment.
+      - Center the star flux in the Crop boxes with the CenBox button and use the drift from center, along with SPICA team input, to make beacon flat corrections as requested.
+  - At times, Mircx will have such poor flux that a beacon flat correction to align to the STST reference position will be needed. This will occur on **E1 pop 4 ****and 5**  and **S1 pop 5**  due to their extreme distances from the telescope to the lab.
+      - Follow the SPICA team guidance in making this correction to improve the flux to keep Mircx usable as a fringe tracker with those baselines.
+==== Alignment Sequence for IR Combiners: MIRC-X/MYSTIC, CLASSIC/CLIMB, SILMARIL ====
+The path of the IR light will be different from the visible light because of dispersion in the AO dichroic and atmospheric refraction. To compensate for this offset, we have **two sets of boxes**  on the telescope WFS.
+The **__blue boxes__  are for starlight**  and the **__green boxes__  are for aligning the red beacon**. The offset between these sets of boxes will change depending on the azimuth and elevation of the target.
+The green beacon boxes help keep track of the offset red beacon motion. These are defined as an offset to the starlight position, so if you move the starlight boxes, the beacon boxes will also move. If you move the beacon boxes only they will move.
+Before the start of the night clear any offset between the position of the beacon boxes and the starlight boxes on the telescope WFS using the “ZERO BCN” button on obsgtk.
+Faint stars (H > 5 mag) will not be easily visible in STST. Therefore, before slewing to the first star of the night or when slewing to a different region of the sky, slew to to a nearby bright IR star to align the beacon flat and set the beacon box offset on the TWFS. Then slew to faint star, keeping the red beacon at the same fixed position.
+**NOTE:**  The STST centroiding/servo'ing now works for longer exposure times. If you change exposure time, you need to: stop the camera, config the camera, restart the camera, and retake backgrounds.
+For **H=6 mag**, try changing coadds to 5 or 10 for 1 sec integration.
+For **H=8 mag,**  centroiding has been shown to work using 50 sec exposure (1 coadd).
+*Please update here* if you determine settings that work good for the using the auto STST alignments at different magnitudes.
+**Aligning the scopes before STST:**
+  - On shutters GUI, make sure **[STST]**  is in.
+  - Slew to a star of Hmag 5 or brighter (or adjust exposure time/coadds on STST as needed to see fainter stars).
+      - The Find IRC function on CD can do this quickly.
+        - Enter the target on a empty line, press **[HD]**  once (or appropriate designation), and **[Find IRC] **
+          - will automatically give you a bright star.
+  - Close IR shutters as needed to avoid saturating the combiners.
+  - **[M7AUTO]**  can help keep the blue beacon aligned during the slew.
+  - On the first slew only,
+      - **[ALN BCN], **align red beacon to the __blue star boxes __on the telescope WFS using the beacon flat.
+  - On subsequent slews,
+      - if the star is bright enough to see in STST (Hmag 5 or brighter), then go to step **12 **
+      - if you can't see the star on STST, then align red beacon to the __offset green beacon boxes__  (only after green boxes have been set using STST - see below).
+  - **[ALN M7] **Align blue beacon to lab WFS using M7.
+      - If beacon is not in boxes, use the yellow M7 buttons on the WFS tab to get centered
+      - 150,000 step size moves 1 full row. Directions are normal.
+  - Labao section,** [FOCUS]**  blue beacon If focus term is high
+  - TWFS section,** [FOCUS]**  red beacon If focus term is high
+      - (ONLY DO THIS STEP IF RED BEACON IS ALIGNED TO STARLIGHT BOXES - e.g., first slew)
+  - **[RED OFF]**. Make sure you have the blue starlight boxes on telescope WFS.
+  - Click **[MOVE]**  button then click on star in ACQ window. This will move star into ACQ hole.
+  - Turn off blue beacon and lock star on **[TT]**  only.
+  - Move TWFS boxes (use popup from **ADJ**  button on obsgtk) to** get star centered and evenly illuminated on ACQ hole.**
+      - (Note: you might need to close DM AO loop during this step.)
+  - Lock star on telescope AO servo (**DM**  button on obsgtk). Both TT and DM servos should be on.
+  - Turn on the blue beacon.
+  - Make sure **DMTilt is green**  (stays on all the time - the message from DM tilt “NOT Ignoring DM tilt…” when it is on. This may take 2 clicks to confirm this message.)
+  - Turn on **DM AUTO**, **LABAO servo**.
+  - If the coma and astigmatism terms are high on the aberration plots, create a new sky flat
+      - Turn off **[DM]**, **[SV FLT]**, **[FLATEN]**, then relock the **[DM]**.
+      - **If the signal is good on the tel WFS, it is a good idea to make a new sky flat on every star.**
+**Aligning STST:**
+  - With the star locked, move the beacon to align the starlight on STST.
+  - On STST GUI, use either Full or Crop view.
+      - The green cross marks the center of the box and is your target, the red cross marks the centroid of star.
+      - If the star is within the STST reference box,
+        - **obsgtk**, main tab, very bottom, click **[STST]**  to automatically move the beacon and M7 simultaneously to align the star to STST. You can do all six at the same time.
+      - If the star is outside of the STST reference box,
+        - Try increasing the size of the STST reference boxes or move the beacon manually using the WFS tab.
+          - Use the rotation matrix wheel printout on the desk to know which way to move.
+            - point arrow to the objects azimuth.
+            - the direction on the printout corresponds to the direction you want to move on stst.
+            - the direction printed on the wheel is the beacon button to press on obsgtk wfs tab.
+            - So if 'LEFT' is pointed up, press the left button to move stst up.
+          - The first few clicks may not move at all due to hysteresis in the actuators. Do this process slowly, as the labao DM TILT will offload to M7 slowly to compensate for the beacon shift and keep the blue beacon aligned to labao.
+          - If the IR flux did not move when clicking the beacon flat, click again as it was taking up slack in the movement. the labao DMTilt can handle up to 3 units of tilt to offload to the M7.
+          - If the tilt values reported on the obsgtk labao tilt are lower than 2 units, you can click again. Steps of 333-500 are typical for the first moves.
+            - If the required move is large, ~1000 steps on the beacon flat, or about 25 pixels on STST, it may help to turn the Labao Servo and DM AUTO off and manually move the blue beacon with M7 back to center on labao until the flux is closer to the STS center. AUTO M7 can also be used for medium sized moves, ~333-500 steps or so of the beacon flat.
+  - **If this is the first STST alignment star,**  then set the offset position of green beacon boxes on TWFS.
+      - When the star is aligned on STST
+        - turn off the DM servo and TT
+        - move the star away with NEXT
+        - turn on the Red Beacon.
+        - Turn on the boxes
+        - Click “TOGG” on obsgtk (or “TOG BEACON” on WFS gtk) to toggle from the blue starlight boxes to the green beacon boxes on telescope WFS.
+        - Click “SET” button on obsgtk (or “ALIGN BOXES” on WFS gtk) to move the green beacon boxes to the position of the red beacon. This will center the boxes over the beacon and save the new beacon flat positions in a few seconds. Watch the X,Y values of the twfs go to zero. Alternatively, you can use the buttons on the obsgtk ADJ tab pop up window to manually move them.
+        - TOGG back to blue starlight boxes when finished.
+  - **If you will record fringes on this star**
+      - turn off the red beacon
+      - toggle back to the starlight (blue) boxes
+      - MOVE the star to center on ACQ
+      - Turn on **TT**  and **DM**  servo for TWFS.
+      - Turn on **DM AUTO**  (**DM Tilt**  should already be on) and **LABAO**  servo.
+  - **If this is an STST alignment star only**
+      - slew to the first target and align as needed there, which shouldn't be much or at all.
+  - **Check aberration plots. **If X/Y terms are not going to zero, try stopping servo loops, recentering alignment, and turning back on servo loops. If X/Y and higher order terms still not going to zero after checking alignment and making a new flat, then this likely means a new reconstructor needs to be made.
+  - **When slewing to the subsequent stars:**
+      - If you will see the star on STST, then align red beacon using **STST**  auto-alignment button after locking the star.
+      - If you won't see the star on STST, then align the red beacon using the new positions for the green beacon boxes (not the blue starlight boxes), then switch to the starlight boxes when locking starlight. If the slew is close on the sky, you can use the beacon alignment from the previous star. Go back to a nearby bright star if you need to check the beacon alignment on STST (e.g. if flux is lower or fibermaps have degraded on Mircx/Mystic
+      - Always lock the star using the blue star boxes.
+  - When you finish slewing and locking the star on TT, telAO, labAO, then click **[STAR ACQUIRED]**  on Cosmic Debris.
+For programs with bright IR stars, skip align beacon, but do the M7 and labao focus alignments. Then lock the star on the blue boxes and use the STST to fine tune any misalignment of the beacon flat. Check the beacon focus on the twfs occasionally to make sure it stays near zero, every 2-3 stars.
+==== Trouble Shooting - AO Performance - Obsolete ====
+If AO performance is bad try these steps in the following order:
+  * Check aberration plots. If X/Y terms are not going to zero, try stopping servo loops, recentering alignment, and turning back on servo loops.
+  * Turn off the blue beacon and check the centering of the star on the ACQ hole. Adjust box positions to center star (ADJ popup on obsgtk). Hint - zoom in on ACQ display.
+  * Look at red beacon on tel WFS and try zeroing the centroids using ZCEN on obsgtk. Note if you are using IR combiner, then align red beacon to blue starlight boxes (not the offset green beacon boxes) before clicking ZCEN. Align beacon back to STST reference after.
+  * Check alignment of blue/red beacon on labao - make sure they are aligned properly with labao boxes. Use red beacon to find center of pattern then turn on blue to be sure the spots are in correct boxes. If not, move M7 to center with correct boxes. Note that the blue/red beacons are offset by about half a box.
+  * If IR flux is low or spots are maps are poor on MIRCX/MYSTIC, then go to bright IR star and check beacon alignment again with STST.
+  * Go back to stow and check the full beacon/dichroic alignment using the start of the night procedure. Remember to load all defaults before doing the alignment.
+  * If X/Y and higher order terms are still not going to zero after checking alignment, making a new flat, and zeroing centroids, then this likely means a new reconstructor needs to be made. Notify charatech (maybe try taking onsky reconstructor).
+==== Trouble Shooting - Locking TT and AO Loops - Obsolete ====
+  - If **not locking**, make sure red beacon is off, make sure on blue boxes, load def flat or last flat, make sure no clouds
+  - If **not locking**, make sure DM current is low or around 1.2
+      - If not, Obsgtk, main tab, LD DEF
+  - If **drifting**  after being locked:
+      - Unlock TT DM, last flat, make drk, next, lock again, TT
+      - DM sometimes makes it hard to hold on to star if using flat.
+  - if star is **flaring**  and really bright,
+      - lower the gain, may need to go lower than 100, maybe even 30 or lower. Lower until star is no longer flaring and spot is compact.
+  - May also need to change to higher gain for dimmer stars, Vmag 8 or higher
+      - Select [FAINT], wait til wfs starts moving, relock TT DM.
+  - After TT DM, Zdark if gain is not 300
+===== Optimizing AO Performance =====
+==== Setting Threshold for WFS - Obsolete ====
+If the starlight is missing from the boxes on the TWFS on S2, you might be able to improve performance on by applying a threshold to ignore the TWFS boxes without flux This is likely caused by an alignment problem at the telescope/AOB, so notify charatech if you apply a threshold. Here are the steps to try this approach again on sky:
+  - If light is missing in some of the TWFS boxes, set a threshold to ignore those boxes.
+  - Open: **wfs_gtk S2**
+  - The threshold is set on the WFS/TT tab. Enter a number in the box to the left of the **THRESHLD**  button. Then click **THRESHLD**  to set.
+  - A threshold of 0.5 to 1.0 seems to work well.
+  - Look at DM tab to make sure the edge actuators are not at or close to 1. If they are, try iterating between the default flat and new sky flat so that the edge values are not saturated.
+  - The thresholding is based on the standard deviation across the full frame (with starlight). The statistics might be better if you make a dark for the WFS. Note that darks should be made after the sky is fully dark, otherwise the dark will include residual sky light.
+  - NOTE: <font 11pt/inherit;;inherit;;inherit>The thresholding works by setting everything below the threshold to</font><font 11pt/inherit;;inherit;;inherit>zero, so if it is too high it will black out the TWFS. You want</font><font 11pt/inherit;;inherit;;inherit>the threshold to be as low as possible while still actually helping out.</font>
+==== Offloading Telescope Aberrations - Obsolete ====
 Theo added a routine to offload aberrations from the telescope WFS on to the labAO DM. This could help remove static aberrations that are not corrected by telAO. Follow these instructions:
@@ Line 996: / Line 1281: @@
       - Note: ADDAB now works for astigmatism and coma terms.
       - This must be repeated after every slew.
+===== Guiding with STST =====
+For stars that are bright enough to see on STST, we can actively guide to keep the IR flux centered in the lab. This will minimize the number of fiber maps that are needed to keep the light centered on mircx/mystic. Check with the PI to ask if they want active guiding.
+There are three methods for keeping starlight aligned on STST:
+  * **STST ALIGN**: The [STST] button on obsgtk takes the star offset (delX, delY) from STST and applies it to the BEACON-FLAT and M7 in a single move (open loop). During this operation, the auto-guiding of M7 is turned off to avoid conflicts. This command is very fast, a few seconds.
+  * **STST GUIDIN****G using obsgtk**: The [GUIDE] button on obsgtk is a continuously running background process that runs every 10 seconds. It takes the star offset from STST and moves the BEACON-FLAT in small increments of 400 units. This method relies on the Labao M7 auto-guiding corrections against the BEACON-FLAT moves (closed loop). This mode added more short-term variations to the flux trend plots because of backlash/overshooting of the beacon flat. We need a more precise way to control the beacon flat for this guiding mode to work.
+  * **STST GUIDING using ststrtd_gtk**: The [GUIDING] button on ststrtd_gtk<font inherit/inherit;;black;;inherit>moves the labao WFS boxes to keep the star centered on STST</font>. In turn, the DMAUT/M7AUT tracking will move M7 to keep the beacon centered in the labao boxes. The motion of the labao boxes can be controlled more precisely than the moving the beacon flat. This is the preferred method for guiding with STST. If using this mode on sky, make sure to send labao boxes to default before slewing to the next star, otherwise the labao reconstructor might fail if the labao boxes move too much.
+Follow this procedure to use active guiding with STST:
+  * <font inherit/inherit/inherit;;inherit;;inherit>Move the beacon flat M7 using the **[STST]**</font> button on obsgtk to center the star close to the STST reference pixel. This step cannot correct the STST error below 5 pixels.
+  * After the star is centered on STST, click the boxes next to the active scope (S1 S2 E1 E2 W1 W2) and turn on the **[GUIDING]**  button on ststrtd_gtk. This will move the labao boxes to keep the star centered on STST. You can unclick the box to turn off guiding for a particular telescope.
+  * The STST-LABAO-BOXES guiding will correct errors above 5 pixels and any slow beam drift.
+  * If the LABAO boxes move too much, the LABAO reconstructor can have problems (tests show that it can remain stable for over 30 minutes). If you see flux dropping on mircx/mystic, it might mean that the LABAO-BOXES have moved too far. To fix this, switch off guiding, load default labao boxes, correct the STST error with the beacon M7 button (STST button on obsgtk), and then restart the STST-LABAO-BOXES guiding. Narsi will try to automate that by checking the difference between the labao default boxes and the current boxes – if they too much difference, stop the guiding offload to the beacon flat.
+  * **IMPORTANT:**  Turn off GUIDING when sky shutters start. After the the data/shutter sequence is complete, load the default labao boxes **BEFORE**  slewing to the next star.
+===== The Next Slew - Avoid Triggering AZ Limit Switches: =====
+Always pay attention to the AZ you are slewing from or to, to avoid passing the sensors incorrectly and dealing with scopes getting stuck near 90° or 270°.
+__Az Limit Switch Near 0º:__
+There are two sensors negative and positive near AZ 0.
+It is okay to slew completely past AZ 0 in either direction.
+However, if you slew close to zero from the west or away from zero to the east, it is possible you will hit just one sensor.
+  * If your next slew is continuing past zero in the SAME direction that you came from, both switches will get hit, and all is fine.
+  * If your next slew is going back away from zero, you will not hit the second limit switch and will run into problems later.
+      * before going to your next slew away from zero, slew to a different star past zero to hit the second limit switch.
+        * can use obsgtk,** [F**i**nd *]**
+          * Az: ~355 or 5º, keep el the same, mag 6
+        * now you can slew back the other way to the next target.
+If you don't pay attention and only hit one sensor at AZ 0, some or all telescopes will continue to stall near 90 or 270 degrees and it is a pain to fix. Will have to trip the limit switch at the telescope.
+__TCS Az limit beyond -90 and 450º: __
+On the telescope server, keep an eye on the TCS az to not go beyond -90º and 450º. If you are getting close, telescopes will have to go the other way around to avoid hitting the limit.
+  * can send to STOW or to another star using FIND* on obsgtk.
+Got to [[:chara:trouble_shooting#the_telescope_won_t_move_or_stopped_moving|Troubleshooting]], "Azimuth Limit Switches" Section
+===== Additional Observing Procedures =====
+==== Manually Slewing a Telescope: ====
+If you need to send a telescope to a different star while keeping the others at their current position:
+  * scope server, type in the star name (use a space instead of an underscore), press enter, then type "next" and enter.
+  * Obsgtk, Control tab, Dome section, press [ON] to the left of AUTO].
+==== Initialize pointing ====
+Initialize the pointing of the telescopes on your first target by going to the MAIN tab on the obsgtk and clicking the red [INIT] button. Make sure you are on the correct star before initing the scopes or you will have problems. This may have to be repeated if the pointing drifts during the night. This will allow more consistent and accurate pointing for this part of the sky. You can also INIT during the SKY frames on MIRCX and MYSTIC since the star is in a good spot for finishing a slew.
+==== Data Acquired ====
+  - Mircx/mystic doesn't always give the Data Acquired line in CD:
+      - CD, Control tab, Data Acquired after shutter sequence
+      - Make sure combiner(s) are selected or you won't get combiner in data entry for the log
+or
+  - Hit reopen on CD after data sequence/before shutter foregrounds end
+      - Sometimes data acquired will show after shutter sequence finishes
+==== Monitoring MIRCX/MYSTIC observing ====
+When running MIRCX/MYSTIC, the operator may want to have the windows open to follow the fiber mapping, scanning for fringes and data recording. The windows can be opened with the command mircx_launch_all_guis on a desktop terminal. It will open 5 windows for each combiner, but they do not need to all be open. Close what you don't want to monitor. The fiberexplorer, gdt, rtd, and super_gtk windows are most helpful.
+==== Adjusting ACQ Camera Settings at dusk and dawn ====
+  - When it is light out, can lower the exposure to make ACQ darker
+      - Adjust tab, exp down.
+==== Acquisition Laser Alignment if the star does not lock in Tiptilt (Rarely used) ====
+  - Set VISBEAMS to correct beams.
+  - Laser Filter on ND 3.0
+  - If the beacon and fiber covers are open, use the SHUTTERS GUI to open the laser shutter and the corresponding B1-B6 shutters.
+  - You can also put in the corner cube with the [CCIN] button.
+  - You will see at least two stars and two laser spots in the ACQ window. There is a reflection, hence two images of each. The desired spots are the right ones.
+  - Use the “ORIGIN” paddle on the ADJUST Tab of the Telescope GUI or obsgtk to align the TV cross-hairs to coincide with the laser spot.
+  - Close the laser shutter and the B1-B6 shutters.
+  - * Move the corner cube out with the [CCOUT] button on the Main tab or Control tab on the telescope gui if you used it.
+      - It will take about 10 seconds to move out of the the beam.
+  - Hit [MOVE] under the TV tracking menu on the Main Tab of the Telescope GUI and then click the right star to bring the star to the cross-hairs.
+      - If it doesn't center well, click [MOVE] again and then the star again until the TipTilt plots show that it is getting light.
+  - Lock Tiptilt as shown above. Maintain the ACQ laser alignment during the night by adjusting the crosshairs to the star after locking Tiptilt.
+==== Changing Ref Cart position: ====
+  - Manually
+      - Take difference of the Targets pos and Laser position for cart farthest to the back (~44.2 m) or closest to the front (-1.2 m)
+      - If w1 is close to front also, adjust w1, then the next farthest one.
+      - Then make up the difference on the ref
+  - Opt Delay (check for ref cart pos)
+      - Wait like 5 seconds after star acq, CD, Opt Delay,
+      - WHEN plot gives starting and ending ref cart position. May not be correct for S1 as ref.
+  - If W2 get's dropped from observations but is the ref cart (ex. going from 6T to 5T with no W2):
+      - keep W2 in the config
+      - turn cart off when its near the back
+      - set ref between 44-70 to get delay on other carts.
+==== POP changes later in the night ====
+  - Once shutters are finished, let carts know where to go
+      - CD → Main tab, Data Acquired → slew to new sci/cal
+      - Obsgtk → M7Auto
+      - Change configuration to new pops, send
+        - Make sure ople reads new star
+      - Star acquired
+  - Do pop changes
+      - Obsgtk, Control, **[On]**, M5targ, close M5
+      - Shutters, Turn on laser, open vis shutters for beams,
+        - If using SPICA, do [SPICA PERI OUT] to see laser. Reopen on shutters gui if it gives an error message.
+      - Visbeams, go to beams you want, Laser filters, ND 0.3 or OPEN
+      - Popperi, turn on power to pops you want to change, change first pop
+        - turn off pop when it finishes on popperi
+      - Align pop on obsgtk
+        - While aligning one pop, start the other
+      - When finished aligning, open M5, ACQ
+      - Turn off Laser, close beam shutters,
+        - If using SPICA, do [SPICA PERI IN]
+  - If staying on the same star, turn off labao servo and load the default flat first.
+==== Changing Beam Configuration During the Night ====
+If you need to realign the lab dichroic or the IR mirror on the beam sampler stage during the night, make sure to remove STST stage (using the shutter gtk) before doing any alignments. Put STST back in the path after you are finished with the lab alignments and are ready to go on back on sky.
+STST gets the beam information from the telemetry server. Try clicking RECONNECT button on ststrtd_gtk. Also try resending the updated configuration from Cosmic Debris and the OPLE GTK (yes try both!). If that doesn't work try restarting the telemetry server by typing the following into ctrscrut:
+  * killall telemetry_server
+  * bootlaunch_master
+Reopen Cosmic Debris and resend configuration again. If that doesn't work then try killing and restarting mircx_stst_server on spooler@stst and running mircx_bootLaunch_master with N option (don't restart all servers - it will mess up mircx/mystic observer).
+==== Locking Lab Tiptilt (only used in special cases) ====
+With the star aligned with the acquisition ticks, check to make sure there are counts on the TipTilt server or the white plots in the TT windows have condensed.
+  - Obsgtk, Main Tab, [TIP/TILT] under the Pointing Servo menu.
+      - If this button is not pressed, then the green dots on the tiptilt windows will drift and eventually drop TT lock.
+      - The [STAR ACQUIRED] button on Cosmic Debris also starts the Tiptilt servo for all active scopes with one push of a button.
+      - On the Tiptilt Servo Control GUI, turn tiptilt [ON] for each telescope or use the [TT ON] button on Cosmic Debris to perform this function for all active scopes.
+===== Procedure for Shutting Down at the End of the Night =====
+==== End Night Sequence ====
+The End Night sequence on Cosmic Debris can be used to end observing and stow the telescopes, carts, and domes and close the mirror covers. It will only stow the active scopes, carts, and domes, so if there are other scopes open, make them active in the Configure tab of CD or stow them manually with the procedure below.
+  - Cosmic Debris, **[CLEAR JOB QUEUE], [END NIGHT], **and **[REPORT]**.
+      - Automatically:
+        - Stops twfs cameras and closes shutters
+        - stow active telescopes
+        - Close M1 (primary) and M3 mirror covers,
+        - send the active OPLE carts to the back switch
+        - close the OPLE socket
+        - archive the accumulated data for the night.
+      - Does **NOT**:
+        - close the dome slits
+        - send inactive scopes or carts back.
+        - Update the configuration with any inactive scopes to make sure those carts can be sent back manually as well.
+        - Doesn't close M5, M7, fiber, beacon, led (red), or laser (blue) laser beacons,
+        - Doesn't disable scopes, shut off powers, turn off labao camera,
+      - Wait for CD to say end night sequence is complete.
+        - Can close Cosmic Debris after you have sent out the Observing Report.
+  - On the Report, add the Observers and Baselines entries, add comments to the bottom of the report and put your name at the end.
+      - Cut and paste this into an email to CHARA Obs and send it.
+      - To archive the report, you must also hit the **[Email]**  button at the bottom of the report to save it.
+  - Obsgtk, MAIN→ Labao STOP OFF
+  - Obsgtk, CONTROL→ unlock then close slits, M7, M5, Red, Fiber, Beacon
+  - Beacon Laser Control → Turn off red and blue lasers
+  - Obsgtk, WFS → WFS COOL OFF, DM POWER OFF
+      - Never do camerlink off
+  - Shutdown STST
+      - Click the “Start Camera” and "Start RTD" button to turn off the camera (the button will go from green to gray).
+      - Change the filter wheel to the closed position.
+      - Using the shutter gtk, remove the STST stage from the beams.
+        - Reopen if needed to get stage to show it is moving out.
+      - For now, the daytime alignment is done without STST in the path so it is important to remove STST at the end of the night.
+  - Confirm everything is at stow before turning powers off.
+  - Power GUI
+      - Turn off the power for [TEL AZ], [TEL EL] and [TIP/TILT] for all scopes once stowed
+  - obsgtk → main tab, [DISABLE] for all scopes
+      - Visually check all telescopes in the spycams (1) to make sure all covers and slits are closed.
+        - make sure finder lid closed all the way (sometimes doesn't).
+  - Obsgtk → Turn the TV OFF
+      - Leave the telescope GUIs open until the End Night Sequence is finished. (Cosmic Debris will turn on the TVs and the Spy Cam 2 during the End Night Sequence.)
+  - Double check carts are at the backswitch, then turn them off
+  - Shut down the ople control system
+      - Close ople server
+      - Press the shutdown button on the OPLE System Control gui, wait until all 6 dots are grey.
+      - Press the Metro button, and wait until it is grey.
+      - Close the gui on Wazoo.
+      - Turn off the metrology laser (METLAS → Laser Power)
+  - Close labao servers
+  - If using Classic or CLIMB:
+      - Check to make sure the dither is off by looking at the dither status in the CLASSIC/CLIMB server.
+        - If necessary, this can be turned off from the dither tab on the Classic/CLIMB GUI. They are off when gray.
+      - Turn off [NIRO CPU], and [CLS-DITH], on the POWER gui.
+      - Close CLIMB/CLASSIC GUIs and CLIMB/CLASSIC servers (type CTRL C, then Y)
+  - Visually check all telescopes in the spycams to make sure all covers and slits are closed. Turn the TV OFF.
+  - Leave GPS, Metrol, and Vacmon servers and the Telescope servers open and running
+====   ====
+==== End Night Tech Report ====
+At the end of the night, write a tech report detailing any issues that happened throughout the night. Helpful to keep an open email draft to note issues as they happen instead of waiting until the end of the night to write everything down.
+  - Include as much detail as possible (i.e. what did you try, did it work/not work, az/el if there was a scope issue, etc.)
+  - If a problem keeps happening (ex. cart keeps getting stuck at the same location), keep mentioning it in the tech report until it gets fixed.
+  - Add your name at the end and send to charatech.
+==== Manually stowing the telescopes ====
+  - Unlock the tiptilt and AO loops.
+  - Obsgtk, → SPY1
+  - Control Tab, →[STOW].
+  - When the telescopes reach an elevation above 50 degrees, you can begin closing the mirror covers (one at a time, click each button only once).
+      - M1 CLOSE,
+      - M3 CLOSE,
+      - M5 CLOSE,
+      - M7 CLOSE,
+        - W1, W2, and S2 M7 covers often read MOVING, don't click them again
+      - Finder CLOSE
+        - W2 and E1 finder covers need a second click to close
+  - Use SPY2 to make sure mirror covers close properly.
+  - Check the Scope Monitor for mirror cover status.
+==== Shutting down the Lab ====
+  - Close the vacuum valves for all telescope lines
+  - Go to pump shack, close valve, shut off the pump.
+      - Note any unusual noises or excessive oil temps (above 54 C) reported by the thermometer. Let Craig, Victor or Steve know of any concerns. Lock the padlock.
+  - Go inside the lab with booties:
+      - Turn off the key for the green alignment laser and turn the power switch off.
+      - Turn off Pico 3 controller above the tiptilt camera.
+      - Turn off the blue amplifiers for the metrology laser (button is labeled “line”) *
+      - Turn off the key for the metrology laser and place it on top of the laser box.
+      - Shut off lights when exiting the lab building.
+  - Check for Any signs on the power controllers indicate power was left on.
+      - E1 computer is usually on.
+        - Push the Off button to turn off the component that is on.
+        - Scroll left and right using the UP/DOWN buttons to move to each sign. Make sure all doors are closed as you leave.
+==== Back in the Control Room: ====
+  - If you run into any problems during the shutdown procedures, send an email to charamnt or directly to Craig and Steve so that they can work to resolve any problems that need to be resolved promptly.
+  - Please feel free to record any details in the Observers Notebook.
+  - Email a Tech Report if you have other technical information to send out that is not related to the nightly observing runs.
+  - Lock front and side doors of Operations Center if you are the last one in the building.
+  - Close OPLE building doors; they tend to stick open, please push firmly.
+===== Useful Commands for Errors =====
+Labao server:
+  - “oscope”
+      - Type in Labao server
+      - For **hut and labao error**. Reconnects labao with hut.
+Scope Server
+  - “otema”
+      - Type in Telescope server
+      - If **no connection to tema**, then type “otema”
+  - “otcs”
+      - Type in Telescope server
+      - Opens telescope control system, use when **“NO DOME SERVER”**  error in scope server, esp. after dome server restart
+Ople Server
+  - “ref [some #] 0”
+      - Type in primary ople server
+      - If **carts aren’t moving to ref**  after entering in CD, type command in server.
+  - “oo”
+      - Open ople
+      - Use if **ople server crashes**  or if you **open ople before ople system controller**
+  - HD 1234 type HD or CHARA number of star if Astromod cannot be set by Cosmic Debris
+Generic
+  - “sb”
+      - Start Backgrounds
+      - Use when any **server display goes blank**
+  - ctrl-L
+      - Clears **server error messages**  that block display
+===== Common Issues =====
+=== Carts ===
+  - cart is stuck
+      - if you notice the laser position is not moving and does not match the laser position on the ople server, it is likely stuck.
+        - turn the cart **[OFF]**, then try sending it the opposite direction with either **[FRONT] **or **[BACK] **button, then **[TRACK]. **it may move fast enough to not get stuck again.
+        - if that doesn't work, go to the lab and try giving it a soft nudge to get it unstuck.
+        - if it doesn't move, check that the ribbon and cable are on the track and not caught somewhere. Also check the cable spooler is not loose and wind it up if it is.
+  - cart error is jumping about 100 microns every 4-6 seconds. On mircx/mystic, fringes will show an oscillating pattern that corresponds to the jump in cart error. In the lab, a faint ticking sound may also be heard.
+      - First try restarting ople server.
+        - Turn off the carts
+        - Restart ople server
+        - Track carts.
+      - if problem persists, restart ople system controller.
+        - On Ople System Controller, restart the ople computer for the cart.
+        - When its done, rehome and check the cart. Issue should go away.
+=== Hut Error ===
+Obsgtk: "**Hut….Error: Relative move error -12"**
+If you start to notice that buttons aren't doing anything (usually on E2) such as moving the beacon or turning on the red beacon, try doing one ~1100 step move to the beacon. If you get a "Hut….Error: Relative move error -12", need to recycle the aob and restart hut.
+  - Power Gui, turn off the AOB power (ex. E2WFS → AOB)
+  - Obsgtk, Ajust tab,** [RESTART PIC]**
+old way:
+  - Recycle AOB power,
+  - open terminal, 'Tel'wfs (ex. E2wfs)
+      - bootlaunch_master
+      - kill hut twice to make sure it no longer exists.
+  - Turn the AOB back on
+  - bootlaunch_master twice again to make the dead hut server has been removed and restarted.
+  - reopen CD and obsgtk
+=== WFS server crashes ===
+On obsgtk, if the telescope wfs goes black, the wfs server has crashed.
+  - log in the 'tel'wfs, bootlaunch, kill wfs
+  - Reopen Obsgtk or open a new one.
+  - If obsgtk displays a message "Telao has not started" (this needs to be updated, see narsi's message in "nighttime technical tracking" channel, search "wfs_cameralink_server" on 11/03/2023
+      - wfs tab,
+        - open shutters
+        - WFS Cool ON
+        - Send On On
+        - DM Power On
+      - on Main tab, wait for cooling to reach -50. Will need to reopen to see it update.
+      - once cooled, turn on the beacon, press STAND, and pres REDON.
+=== Beacontune not connected ===
+  - ssh 'tel'zwo
+      - if "no route to host" message, need to go to dome
+      - turn off raspberry pi and unplug both cables (black and orange)
+      - plug back in and turn back on
+      - log in on computer
+  - bootlaunch_master
+      - make sure its okay or removed from sockman and bootlaunch again.
+=== Telescopes ===
+Sometimes S2 unlocks and won't move in elevation. Will notice star will only move side to side and drift diagonally and never be able to move straight to acquisition hole. Try the following to get it to move in elevation again:
+  - Try hitting **[ENABLE] **one or two times only.
+  - Switch to Manual mode and back to Auto mode
+      - Dome GUI, notice that the Mode says Auto near the bottom.
+      - Manual tab, press** [MANUAL]**
+      - Auto tab, **[AUTO]**
+  - Restart dome server
+      - open terminal, type “S2”
+      - bootlaunch_master, kill dome_S2 PID, bootlaunch_master twice
+===== –END– =====
 ===== Guiding with STST =====

The CHARA Array Wiki

User Tools

Site Tools

Differences

Page Tools