The CHARA Array Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== New Operating Procedures ======

Last updated 04/22/2024

[[:chara:operating_procedures|Link for Old Operating Procedures Page]]

**Observing Procedures**

   * [[:chara:new_operating_procedures#observing_conditions|Observing Conditions]]
  * [[:chara:new_operating_procedures#lab_protocol|Lab Protocol]]
  * [[:chara:new_operating_procedures#sunset_prep|Sunset Prep]]
  * [[:chara:new_operating_procedures#preparations_in_the_lab_prior_to_observing|Preparations in the Lab Prior to Observing]]
  * [[:chara:new_operating_procedures#setting_up_in_control_room|Setting up in Control Room]]
  * [[:chara:new_operating_procedures#pop_changes_and_m10_alignments|POP changes and M10 Alignments:]]
  * [[:chara:new_operating_procedures#beacon_alignments_at_stow|Beacon Alignments at Stow]]
  * [[:chara:new_operating_procedures#setting_up_the_six_telescope_star_tracker|Setting up the Six Telescope Star Tracker]]
  * [[:chara:new_operating_procedures#homing_the_carts|Homing the Carts]]
  * [[:chara:new_operating_procedures#going_on_sky|Going on Sky]]
  * [[:chara:new_operating_procedures#slewing_and_locking_on_a_target|Slewing and Locking on a Target]]
  * [[:chara:new_operating_procedures#alignment_sequence_for_visible_combinersspica_and_pavo|Alignment Sequence for Visible Combiners: SPICA and PAVO]]
  * [[:chara:new_operating_procedures#alignment_sequence_for_ir_combinersmirc-x_mystic_classic_climb_silmaril|Alignment Sequence for IR Combiners: MIRC-X/MYSTIC, CLASSIC/CLIMB, SILMARIL]]
  * [[:chara:new_operating_procedures#optimizing_ao_performance_on_w2_and_s2|Optimizing AO Performance on W2 and S2]]
  * [[:chara:new_operating_procedures#the_next_slew_-_avoid_hitting_the_limit_switch_near_az_0|Next Slew - avoid hitting the limit switch near AZ 0]]
  * [[:chara:new_operating_procedures#procedure_for_shutting_down_at_the_end_of_the_night|Procedure for Shutting Down at the End of the Night]]

**Supplemental Links**

[[:chara:discussion_of_alignment_procedures|Discussion of Alignment Procedures]] \\
[[:chara:new_alignment_procedures|New Alignment Procedures]] - no longer updated - see alignment sections on this page

[[:chara:night_time_pop_changes|Night-time POP Changes]] - includes how to trouble shoot pop not moving \\
[[:chara:remote_observing|Remote Observing]] \\
[[:chara:power_outage|Shutting Down prior to a Power Outage]] \\
[[:chara:power_recovery|Recovering from a Power Outage]] \\
[[:chara:trouble_shooting|Trouble Shooting]] \\
[[:chara:text_commands_for_chara_guis|Text Commands for CHARA GUIs]] \\
{{:chara:files:dailylabcheck.pdf|Daily Alignment Check Procedure}}  \\
{{:chara:files:co-phase2019.pdf|Co-phasing with the CHARA phase reference}}  \\
[[:chara:cophasing_with_the_new_chancam|Cophasing with the New ChanCam ]] \\
[[https://docs.google.com/document/d/1w1Ppi21b0Uh37R52btCgwsX6dNYUfEE5hUhncDlNvYQ/edit#|Instructions for using the Six Telescope Simulator (STS)]] \\
[[https://docs.google.com/document/d/1Ok1e0omwCD1teKlX2sW9izxA3x-IxH91jFQieh2OtSs|Six Telescope Star Tracker (STST) Manual]] \\
{{:chara:files:tiptilt_splitters_change.pdf|Tip-tilt Splitters Change}}  \\
{{:chara:files:e1_hut_coms_recovery.pdf|E1 HuT and Cooler Communications Recovery}}  \\
[[:chara:coude_alignment_procedure|Coude alignment procedure]] - Updated 2021Jun01 \\
[[:chara:horizons|Telescope Horizon Limits]] \\
[[:chara:restarting_servers|Restarting Servers]] \\
[[http://www.astro.gsu.edu/~jones/chara_aberration_plots|Daily Aberration Plots]]

 \\ __**ADAPTIVE OPTICS DOCUMENTATION**__

[[:chara:tunable_beacons|Tunable beacons]] \\
[[:chara:start|Tuning the AO Systems]] \\
{{:chara:files:beaconrecoveryinstructions.pdf|E2 AOB Dichroic Recovery Document}}  \\
{{:chara:files:labao_with_starlight_v4.pdf|Using Labao with Starlight}}  \\
[[https://docs.google.com/document/d/1b1f0pgeeKOx5NO-jUOCbFM-e7b9GL8iBMFijD1B3Qc8/edit#|Telescope AO User Manual]] - an engineering manual, not for observing \\
[[https://docs.google.com/document/d/1TnLxWaAw42DZwh7l6_wPfe_CpIfT935_aShK0ymBX1o|WFS TT Alignment and Usage]] \\
{{:chara:files:twfs_faint_objects.pdf|TWFS Faint Object Instructions}}

{{:chara:files:chara_telao_calibration_source_6.pdf|Telescope Adaptive Optics Calibration Procedure}}  \\
[[:chara:labao_alignment_procedure|LabAO Alignment Procedure]]

__**INSTRUMENTS**__  \\  \\ [[:chara:instruments|Instrument Manuals and Observing Logs]] \\
[[:chara:fringe_offsets|Recent fringe offsets]] \\
[[:chara:glass_offsets|Recent glass offsets]] \\
[[:chara:instruments|Software setup for specific instruments]] \\
[[:chara:classic_faint_object_procedures|Classic Faint Object Procedures]]

===== Observing Conditions =====

Check the current conditions and weather forecast for Mount Wilson before opening any of the domes or telescope covers. The telescopes can be closed in about 5 minutes from the control room. With this in mind, don’t open up the telescopes if conditions can change for the worse rapidly enough that you don't have time to close down safely.

Weather stations are installed at each bunker; readings are listed on the **weathergtk**. The HPWREN cameras are also useful in monitoring local conditions. However, nothing beats going outside for a gander and giving the sky a good old fashioned assessment.

Weather variations do exist from telescope to telescope, as much as 20% in RH at times. Most of the time this just means something is blowing through and opening is not a good idea. These circumstances need to be dealt with cautiously depending on the given weather conditions. Allowing extra time to close is advised in case of any problems.

==== Humidity ====

  * Only open the telescopes if the humidity is steady or dropping, at 75% or below. If RH has been high (90-100%) when starting, do not open unless it drops below 75% and stays there for 30 minutes.
  * If the RH is low at the beginning of the night, and rises after you are already open, the array operator should inform the PI around 70-75%, and start closing between 80-85%.
  * If the roofs outside the control room are ever dripping, then the telescopes should not be opened or should be closed immediately.
  * Be aware that if the RH is at 100% for many hours, things will be very wet, even if the RH drops dramatically below 75%.

Other humidity warning signs are: 1) water dripping oﬀ the OPLE building (or wet spots on the asphalt straight down from the eaves); 2) cold, clammy feel to metal objects such as railings; and 3) dew forming on the parked cars. It is important to note that the humidity can rise from 50% to 80% in as little as 10 minutes, so keep a close eye on the rate of change of the humidity plots. It can also rain or hail from small, passing clouds when the humidity is quite low, even 30% RH or lower. Operators have been surprised before by small puffy clouds in unsettled air with low humidity. If in doubt, stay closed. The HPWREN webcams are a good resource to view the bottoms of clouds. If there is any virga, the clouds are holding moisture and try to rain out. Stay closed if you see anything streaming below the clouds.

==== Dust ====

Gauging the airborne dust and pollen can be problematic because you need a bright light and a relatively dark night. In general, you want to look in the bottom 0.5 meter of the light column. There will be a diﬀuse column lit up by the beam – don’t worry about it, just follow the dust glints. If you can count the dust glints, the conditions are safe to observe. If it looks like a blizzard, close up. If any of the dust glints shine with an orange or a peach hue, or the dust glints look abnormally large, the dust is probably ash – close up immediately. Ash seriously degrades bare aluminum coatings. Dust and wind go together. But, just because it is calm, the dust conditions might still be bad. If it was windy a few days earlier, it could have kicked up a lot of dust, which can take several days to settle. Finally, during late spring to early summer, conifer tree pollen can be problematic. If particulates are borderline please email the CHARA day staﬀ so they can clean the optics as soon as possible.

==== Wind ====

  * Use caution if wind gusts > 15-20 kph
  * Close up if wind gusts > 30 kph

Wind will degrade the seeing and kick up dust. The wind eﬀects are ampliﬁed in the tunnel between the OPLE building and the oﬃce building, so look for other places to gauge the wind conditions. The intersection of the road and the trail to the eastern telescopes is a good place to gauge conditions. If you can hear the gusts from your desk, whistling from the light pipe supports, or the chain banging, it is too windy to observe. Occasionally you will hear the wind rustling the tops of the tall trees, but it is calm at ground level. This is a symptom of chaotic wind conditions, and almost always happens during windy episodes (Santa Anas, onshore ﬂows, etc.), during which you shouldn’t observe – the seeing will be bad anyway. Be cautious about opening if wind gusts are above 15-20 kph; winds this high are usually correlated with bad seeing and poor data quality. Telescopes and domes should be closed if the wind gusts get up to 30 kph. This will protect the optics from branches and debris that can be blown in from the surrounding trees.

==== Snow and cold weather ====

If there is snow stuck to the telescope dome itself, don’t open up. If there is snow still on the trees and branches procede with caution and make a visual assessment.

Cold temperatures around freezing also can be problematic. Several components are not rated to operate at freezing temps. The DM is temperature sensitive with a 5° C lower limit. Do not open unless it is above freezing and keep an eye on the temps if they are still dropping.

  * Do not open if T_out is near freezing and trending downward.
  * If open when scopes were above freezing, close when T_tel reaches 3ºC and trending downward.
  * If it is below freezing, do not open.

[Outdated] Here are limits for observing close to freezing:

  * Temperature warning: T_tel < 2C or T_outside < -3C
  * Hard limit for closing: T_tel < 0C or T_outside < -5C


==== Essential Observing Links ====

  * Mount Wilson HP Wren Cameras ([[http://hpwren.ucsd.edu/cameras/S/SD/wilson.html|http://hpwren.ucsd.edu/cameras/S/SD/wilson.html]]) 360º views of mountain in color and B&W IR modes
  * National Weather Service ([[http://forecast.weather.gov/MapClick.php?site=lox&textField1=34.2231&textField2=-118.0587&smap=1#.VKsuq2TF8rO|http://forecast.weather.gov]]) Mount Wilson weather and 5 day forecast
  * Mount Wilson Ambient station [[https://ambientweather.net/dashboard/c68efe875521e576a9694913bdd92e65|https://ambientweather.net/dashboard/c68efe875521e576a9694913bdd92e65]]
  * Jet stream forecast [[https://weatherstreet.com/models/gfs-jetstream-wind-forecast.php|https://weatherstreet.com/models/gfs-jetstream-wind-forecast.php]] Monitors high level winds which degrade seeing
  * IR satellite loop [[https://www.star.nesdis.noaa.gov/GOES/sector_band.php?sat=G17&sector=psw&band=GEOCOLOR&length=12|https://www.star.nesdis.noaa.gov/GOES/sector_band.php?sat=G17§or=psw&band=GEOCOLOR&length=12]] Keeps an eye on storms and clouds coming in. Use Band 13 for viewing clouds at night.
  * Clear Sky Clock ([[http://cleardarksky.com/c/MtWilsonOBCAkey.html|http://cleardarksky.com/c/MtWilsonOBCAkey.html]]) a forecasting tool, but be aware that the humidity forecast is often incorrect on this site as it incorporates the marine layer forecast for the valley below, which does not affect us 80-90% of the time

===== Lab Protocol =====

==== Metrology Laser ====

A high-powered infrared laser is used for the delay line metrology. This is an eye hazard, so before entering the lab check to make sure the red light above the door is not on. If the laser is activated, you must wear IR-protective goggles. Please note that the METLAS gui or OPLESystem Controller gui may not show the correct state of the laser. Hit reopen on the METLAS gui if it does not agree with the others.

==== General Lab Rules ====

Lab booties/shoe covers must be worn inside the lab. Throw out booties that have holes in them. The traction tape on the steps are particularly rough on booties so examine them before and after using them. Proper clothing is also important. Loose hanging jackets or garments can snag on mirrors or other exposed equipment and are better left outside of the lab. Please be alert and move slowly and cautiously. When entering the lab make sure to walk across the sticky ﬂoor mat. If the mat isn’t sticky, remove the top layer. If this is close to the ﬁnal layer please notify the staff.

Minimize the number of people in the lab, especially if within a few hours before observing time. Many bodies can create unwanted atmospheric eﬀects. With too many people, it is too easy to bump into each other and the equipment. If you bump, hit, move or just tap anything in the slightest way, let staff know. Never touch anything you have not been trained to use. If ever in any doubt, leave it alone and get someone who knows. Any unusual sounds, sights, movement of equipment, etc., should be noted. You must know the path of the laser beams, metrology or alignment.

When leaving the lab any time of the day, remember to turn off all lights (fluorescent and incandescent). Lights left on generate unnecessary heat in the lab.

===== Sunset Prep =====

==== Opening the telescope domes and enclosures ====

Begin opening telescope domes and enclosures **at least two hours before sunset (when the sun is below 40º).**

Before opening the domes, make sure to check the current weather conditions and the forecast to make sure conditions are safe to open and are likely to be stable as you wait for it to get dark.

__To vent the domes (open partially, usually during Summertime):__

  - On **obsgtk**, control tab, **[UNL SL], [VENT] **
      - Open them fully when the sun has reached 40º for maximum cooling. The time you can fully open is on the white board,
  - Then** Dome GUI, RA/DEC tab, ANTI-SUN**  setting for **AUTODOME**  (usually just the E’s are the ones that need to face away from the Sun).

__To open the dome slit fully:__

  - **obsgtk**  → Control Tab, click **[UNL SL]**  and **[SLIT]**
  * Can open the M5 mirror cover early while doing lab alignments to help cool the AO box
  * Monitor the progress of opening the domes:
      * **Obsgtk, [ON], [SPY1]**
=====   =====

===== Preparations in the Lab Prior to Observing =====

==== Starting the vacuum pump and opening valves ====

  - Go to vacuum pump shack outside.
      - Flip on power switch for pump and wait for the blower to go on in about 7 seconds.
      - 5 seconds later, open the valve slowly (parallel to pipe means open)

  * Note if fan cords are plugged into wall outlet to keep pumps cool in the summer. Turn on the fan that is located at the blower if it is hot. Turn off at the end of the night.
  * Note oil temperature when you start up. Oil temps are usually kept above 20º C by a block heater on the north side of the pump and a space heater on the south side. Inform Craig or Larry if the oil temp is below 20º C and the heaters did not run to bring the temp up before starting.

  - Go to lab building
      - check pressures on Vacmon display
        - To open gui: Chara blue menu, **Vacuum Monitor, [Status]**
      - S1 and S2 lines leak the most, so open first until they reach the level of the next highest pressure line (usually E1, then W2). Can open all the valves once they are low enough ( ~3 torr or so).
      - If pressure is higher than 100 torr in any of the lines, pump those lines only one at a time.


==== Turn on Electronics Inside the Lab ====

  - Put on booties and go inside lab
  - Turn on **green alignment laser**  in the beam combining area (black box nearest niro book)
      - Turn on power switch
      - Turn the key.
  - Turn on **Pico 3 **controller (or make sure its on)
      - On visbeams table - above tiptilt camera.
  - Turn on blue **metrology amplifier boxes**, button is facing you (one flashes, the other doesn’t, will hear it start up)
  - Turn on **met laser**  next to blue boxes by putting in the key, will say "Standby".
  - Filling NIRO with LN2 [ONLY IF NIRO is COOL]
      - Fill both chambers on the NIRO camera. Use the gloves and glasses provided on the shelf with the funnels and the thermos. It will normally take 1 liter of LN2 in the outer chamber and 1/2 liter in the inner chamber. Fill it until the LN2 bubbles out of the top.
      - Record that you filled the camera in the log book, noting time and camera filled. Write FF in the NIRO column to show both chambers were filled. Note that there is a morning and evening filling so follow the pattern.
      - If you do not observe due to weather, fire or closure, inform someone on the mountain before 2:30 pm so they can fill it for you.

==== If lab alignments haven’t been done ====

Day time staff will align the lab during the week, but operator is responsible for alignments on the weekends.

Before doing alignments,** home LDC's **  (as of 10/12/23 - update if no longer needed)

  - Blue chara menu, ESP Controllers, open both LDC's
  - Turn on each scope ldc (red button will turn green)
  - change 'vel' to 50 and 'acc' to 200
  - **[HOME]**  on all.
If alignments have been done, check the following:

  - corner cubes are out
  - LDC's are in or out as needed
  - IR target is out
  - Skip down to the Ople and Metrology section.

=== East Table Alignments: ===

  - Do alignments on the E table target at North wall of Beam Switching area on laptop
      - On **Laser Filter gui**, select **[Open]**
      - On **SHUTTERS gui**, open **LASER**  shutter and open **shutters**  for all beams.
      - If SPICA has been used and no laser is showing, press **[SPICA PERI OUT]**
      - On **VISBEAMS gui**, move the laser to **[B1&2]**.
      - On **Beam Samplers gui**, move all beams out of the way
        - In "All" blue box section, move drop down menu to **[B5]**  and** [set]**
      - Use **Pico3 gui**  to center laser dots on cross hairs. Alignment windows are usually left open on the laptop.
        - **Iris gui, [Beam]**  it to adjust iris and check if its centered.
      - When ready to do Beams 3&4, change **VISBEAMS**  to **[B3&4]**  and align.
      - When ready to do Beams 5&6, **Beams Samplers gui**, ALL section, dropdown menu to **[B4]**  and **[Set]**.
      - NOTE: For PAVO and SILMARIL, align Beams 1&2 and 2&3. For CLIMB 2, align 4&5 and 5&6. For Classic, align 5&6.
        -
      - When finished, correct beams may already be selected above ALL section (double check), hit** [Set]**  to send them to the correct positions.
        - MIRC-X/MYSTIC/SPICA beam order: E1-W2-W1-S2-S1-E2
        - If SPICA is being used, make sure the LDC's are all in place before doing the lab dichroic alignments.
      - Close the laser shutter.
  - Or use Automatic Alignments (New)
      - TBD
      - Note that with the new automated alignment setup installed, the beams to the right of the beams selected will also be illuminated. For instance, B2 is illuminated by B1 if both shutters are open and both Beam Samplers are moved away. Close B1 shutter to see B2 illuminated by the proper beam.
  - Turn off all lights and exit.

=== Lab Dichroic Alignments ===

  - Make sure corner cubes are out.
  - Go to the computer outside of lab
  - Turn on labao servers if not turned on in the control room after UT 0.
  - Open up **labao guis**
      - Blue Chara menu, "labao guis", "all labao guis”
  - Make sure on appropriate beam pair on **Visbeams**
  - On **labao guis**, go to WFS tab
      - **[Start WFS]**
      - **[Start DISPLAY]**
      - **[MAKE DARK]**
  - Laser Filter gui, select **[ND 5.0]**  - Do not open WFS shutters until lowering laser intensity.
  - **Shutters gui**, open Laser shutter, open all beams, IR beams, and WFS shutters
      - If you see double spots, make sure corner cubes or sts vis stage is out
  - If the configuration has been changed and laser flux is not in all the boxes, open **Pico1**, select dichroic for the scope, and align the flux into the boxes.
      - use large steps ~700 cts
      - if pico isn't moving the flux, need to manually move the vis dichroic using the knobs on the picos until flux is in the boxes.
      - S1 and S2 are most likely to be misaligned.
  - **Labao gui**
      - ALIGN tab,
        - **[LAB DICH]**
          - Wait for it to center near 0 (x & y numbers)
            - If it fails, do it again
        - **[TOGGLE REF]**  to switch boxes back to laser reference
          - Checks if errors are small, repeat alignment if steadily higher than 0.015.
      - WFS tab
        - **[STOP WFS]**
        - **[STOP DISPLAY]**
        - **[Quit]**  to close GUI
  - Repeat for next beams (3&4, 1&2)
  - **Shutters, **turn off **[Laser]**.
  - Close labao servers

=== IR Target ===

Pavo does not need IR target alignments.

  - Go back in lab
  - Put corner cubes in place
  - Put IR target into the mounts on either MIRC-X/MYSTIC, SILMARIL/CLASSIC/CLIMB2 IR table
      - If laser isnt showing, check that **[STS IR FOLD]**  is out on shutters gui and that IR shutters are open
        - May need to reopen Shutters gui to move STS IR fold
      - Make sure STST stage is out
  - Refresh guis with **[REOPEN]**  buttons
      - if guis are not open, blue chara menu,
        - Pico 2, dropdown menu, E1IR and so on
        - Laser Filter
        - Shutters
        - VISBEAMS
        - Beam Samplers
        - Iris
      - **Visbeams**, set to **[B1&2]**
      - **Laser_Filters**, Open
      - **Shutters**, turn on **[Laser]**
      - Align (note: buttons on IR Picos move reversed from what they are labeled)
        - if really misaligned (add example pic), check placement of the corner cube
        - **iris, [Beam]**  to make sure its centered, second dark ring can be used to get to center of target circle
        - move on to next beam pair.
  - when finished aligning all beams,
      - **Shutters,**  turn off** [Laser]**
      - remove corner cubes
      - remove IR target
      - turn off all lights.
  - Confirm the correct Prism or Grism is in MIRC-X as requested in the setup request email or be prepared to change it after the flux is found on the currently installed prism.
      - MIRCX users differ in wanting that done before or after they come online. Some prefer centering STS flux into fibers before changing modes.

=== Aperture Wheels: ===

If the observer will be observing bright targets and requires the use of the aperture wheels, the power cord needs to be plugged in.

  - go inside lab to where STS is set up. On the wall outlet, plug in the black cord; it's the only cord by the outlet.
  - use GUI "aperture wheels" on blue chara menu once in the control room to change the aperture size for the beams.

==== Starting OPLE System and Metrology ====

  - Go back to computer outside
  - Check sockman for old OPLE“X” servers (where X is 1-6) and remove any that may be on the sockman list.
  - Turn on metrology laser
      - Make sure no one is in the lab or needing to go back in.
      - Chara menu, **Metrology Laser**
        - select** [Laser Power]**, wait for red light above lab door to go on, wait for button to turn green on gui or **[Reopen]**
  - Start OPLE System Controller -
      - Chara menu,** OPLESystem Controller**  (towards bottom)
      - If you need to start from command line: ssh -Y ople /opt/bin/OPLERemCtrlApp
  - Wait for laser to turn green. When the laser is on, the circle marked Laser will be green and the wattage listed will be 65-80mw (instead of ~20mW when off).
  - All the circle indicators start out grey.
  - After a 1 minute warmup of the laser, press the **[Metro]**  button
  - Press the **[start]**  button near the bottom.
      - This takes about 90 seconds to complete. The grey buttons will turn yellow with a blue icon when the computers have started up. Then they will then turn solid yellow when they are ready for the next step.
      - Note: E1 has often failed to start and will need to be powered on manually by the rocker switch behind the right, front cover of the E1 ople computer. Manually turn on any computer that does not have a blue Led showing it is on.
  - Once all 6 are solid yellow, right click on each yellow dot, and select** [start]**. Buttons will turn green.
      - be careful to not accidentally select Stop or Reboot.
  - Once all 6 are green, quit the GUI.
  - Turn off all lights before exiting building
  - Inform MIRCX/MYSTIC PI lab is done and ok to do any alignments/STS

**NOTE:**  If using W2 E2 S2 for Classic or PAVO, then W1 E1 S1 ople computers have to be turned on as well in order to turn on the electronics for the pair, even if you don't plan to use them.

===== Setting up in Control Room =====

There are six computer screens. Top Left is screen 1, bottom left is screen 2, top middle is screen 3 and so on. Window layouts can be saved and reloaded to suit each operator. Use the third icon from the left in the toolbar (Operator Layout) to open a window to save or load a layout.

M10 and red beacon alignments can get started before ople system controller is on and before UT 0 because the scope servers are always on. Will need labao servers (after UT 0) to do labao alignments. Remember to restart scope servers. Ople system controller is needed for the carts, not alignments

==== Open servers required for observing after UT 0 ====

Servers can be opened from the black menu at bottom of screen 1

  - **GPS, METROL, and VACMON**  servers should always stay on and remain on screen 1.
  - Close existing **Telescope Servers**  on screen 2 after UT 0 (because of tiptilt connection issues)
      - Ctrl-c, y
  - Open new **Telescope Servers**
      - Black menu, "ALL SCOPE SERVERS" (place on screen 2)
  - Open **Labao Servers**
      - Black menu, “ALL LABAO SERVERS" (place on screen 5)
  - Open **Ople System Controller**  if not already open
      - blue chara menu, "OPLESystem controller" (place on screen 1)
      - if gui isn't opening, open terminal:
        - ctrscrut
        - ps aux | grep ople
        - remove PID's associated with "…controller app"
  - Open **Primary Ople server**
      - Black menu, "Primary Ople Server" (place on screen 3)
      - if using SPICA, type "useldc on" in server.
        - always do useldc on when reopening ople.
  - Open **Classic, Climb1, or Climb2**  servers if needed. **MIRCX, MYSTIC, PAVO, and SPICA**  servers are started by the observer, not the operator.
  - To reorganize the servers - White menu, load layout, press mouse wheel, enter, all servers will go to saved positions.
      - Or highlight the last entry on 2nd column and press the scroll wheel to paste, then enter.

==== Setting up GUIs for observing: ====

  - Reopen **obsgtks**
  - **SCOPE MONITOR**  often doesn’t update, so good to reopen a new one.
      - Blue menu, Telescope monitor (towards top)
  - Turn on the following from the **Power GUI**:
      - Unlock the telescopes by unchecking the tick-box
      - For each telescope, turn on **[TIP/TILT], [TEL AZ], **and** [TEL EL].**
      - If observing with CLASSIC or CLIMB2, turn on **[NIRO CPU]**  and **[CLS-DITH]**  on RACK_2.
  - Open **Primary Ople**  GUI
      - Blue menu, GTKs, "Primary Ople"
  - Open **Primary Cosmic Debris**  (always open after OPLE server)
      - If PAVO is observing, PAVO always uses primary Cosmic Debris.
      - If CLASSIC is observing simultaneously, open secondary CD and secondary ople for CLASSIC
  - Open **Classic**  or **Climb2**  gui if needed.
  - Open **LDC1**  and **LDC2**  if observing with PAVO or SPICA
      - Turn on power for each telecope (will go from red to green) for LDCs in use
      - Press** [Home]**  LDCs - make sure VEL is 50.
  - for **SPICA**,
      - on ople server, type '**useldc on**'
      - ssh -Y observe@spica-control
        - **ipdet_gtk_gui**  &
        - **[Start ACQ] **
        - can adjust display with 2000 in the max, then hit **[SET]**.
  - The following windows should be opened on computer screen 1 (these are usually kept open all the time):
      - Telescope monitor, Beam Samplers and PoPs, Visbeams, Shutters, Laser Filter Wheel, Iris, Metrology Laser (METLAS), Metrology Monitor (METROL), Sockman, Pico 2, Power, Weather gui and graph, terminal for opening popperigtk, aomon and beacon tune.


==== Setting up Cosmic Debris Sequencer ====

  - Enter PI and program (ex. 2023B-M10)
  - On the **Control tab**, check instrument boxes (e.g., Mircx and Mystic)
      - If using CLIMB or CLASSIC, select the filter (H, K) and make sure it agrees with the NIRO Filter gui
  - Synchronize all clocks by clicking **[SYNC CLOCKS]**
      - A series of messages will display on CD to indicate that systems and servers are synced.
      - Check the top of OPLE server to confirm that the OPLE Tm clock is the same as CHARA time and that the error in the parentheses is (0) or (1).
        - A reading of (0) and (-1) may cause problems as the clock is synced one second off from usual. Use **[SYNC CLOCKS] **again to get the OPLE errors to (0) or (1).
      - A reading of a different time with (!!) behind it means the clocks have not been synced.
  - On **Configure tab**
      - Select active telescopes
      - Check which is ref cart
        - W2 is usually the reference cart for 6T observations
        - **OLD…Bad Calibration Zone **- If targets will be in bad calibration zone (az 40-120 and el < 60), S1 needs to be ref. **No longer the case. **
      - Make sure beams are correct, check and set pops to first configuration in setup email
        - If beams are not correct, move using the **BeamSampler GUI**  and set on** Cosmic Debris**.
      - After everything is set correctly, then click **[SEND] **on **Cosmic Debris**  to send the active scope information to Ople.

===== POP changes and M10 Alignments: =====

See also [[:chara:night_time_pop_changes|Night-time POP Changes]] which includes trouble shooting notes if a pop mirror does not move.

Check that pops are set as the same pops as the setup request email before homing the carts or after homing and sending them back to 20m.

  - **PoPS Gui**,** [Overview**] button to bring up the **POP Overview**** gui**
      - If nonsense comes up in the overview
        - **PoPs**** gui**, click** [FLUSH]**
        - **PoPs Overview**, click **[UPDATE]**
  - Change PoPs using the **Popperi**  gui
      - Open in terminal if not open - command: popperigtk
      - Popperi cannot do more than one pop change at a time
      - On **Obsgtk**, turn **[ON]**, **[M5targ]**, Open **[M7]**, (close M5 if already open)
      - On **Shutters**, open **[L****aser], **make sure beams are open.
      - On **Laser Filters**, **[0.3]**  or** [open]. **
        - Open works best if doing pops after domes are opened.
      - On **VIS Beams**, check if on 1&2, 3&4, or 5&6 and work from there.
      - if you don't see the m5target after steps 2c-f, corner cubes may have been left in.
      - On **Popperi**, turn on the beam that needs to change (will go from red to green), change pop, wait, and turn off the beam
      - Start the next pop change while aligning the current one
  - Align M10
      - Pop change will open Pico 2 guis and M10 alignment guis
      - If not, on** Pico2 (P2) gui**, dropdown menu, (ex. E2M10), **[MOVE]**
        - If pico menu isn’t working, open a new pico
          - Blue chara menu, Pico Controllers, No.2 or Lab align
      - Beginning of night, align __all __M10's whether you change the pop or not.
        - Mid of night, align just the scopes that need pop change.
      - Use M10 Alignments cheat sheet for directions that buttons will move at the beginning of night
      - Directions will be different on sky.
        - On sky, W's and E's will be the same directions, S's will be opposite of the W's and E's
        - Make sure to stop Labao servo and load the default flat before changing pops once on sky.
  - Tips:
      - On **iris**,
        - **[BEAM] **cycles the iris from open to closed and gives a better view of where the center of the spot is
        - **[DEC] / [INC]**  makes the rings more or less defined.
      - On **Obsgtk**, **[ADJ] **  button, (top right, not the Adjust tab)
        - Can change contrast of the M5 camera with **up/down**  buttons on the right top half
      - S2: align to double spot
      - Use large steps ~ 100 or bigger (E2 ~300 to start, W2 ~500 or bigger)
  - When the spot is centered, close the M10 pop up window
  - On **Obsgtk**, Control tab, OPEN **[M5]**
  - On **Visbeams**, Change to next beam pair
  - When finished with all M10 alignments,
      - On **Shutters**, Close **[laser] **
      - On **Obsgtk**, **[OFF]**, **[Acq]**.
  - Double check all pops are correct and match the setup email.
  - On **CD**, Configure tab, check/change the pops,** [Send]**.

Manually Moving PoPs during Popperi movement

  - Can manually move out & move in pops with **POPS gui**  while popperi is changing to be quicker
      - Select pop mirror from scroll down menu
      - MIRROR in or out
      - Make sure to change on popperi bc that changes the periscope position as well.

===== Beacon Alignments at Stow =====

The initial beacons alignments are done with the telescope at stow with M7 in the default position. Because we are no longer sending M7 to the default position during the slew sequence, this is the __only time that we should adjust the dichroic__.

Make sure the carts are **not near the front**  (0-5 meters) when performing these alignments or the labao alignments will be off.

  * Move the carts to 5m or farther back from the home switch.

**IMPORTANT NOTE:**  There are now **two**  sets of boxes on the TWFS.

  - The **blue boxes**  are for locking on starlight and are the default reference position.
  - The** green boxes**  can be offset from these to save the beacon positions after the STST alignment.

At the beginning of the night, the **red beacon**  must be **aligned to the blue**  starlight boxes with the telescope** at stow.**  You can zero out any offsets between the green beacon boxes and the blue starlight boxes on the telescope WFS using [ZERO] on obsgtk or [ZERO BCN] on wfsgtk.

==== Turning on Beacons and Starting WFS Cameras ====

  - On **Obsgtk**, control tab
      - Turn on **[RED]**  beacons and open **[FIBER]**  covers on all scopes in OPEN/ON section
      - Open** [BEACN] **covers after fiber covers open
  - Open Beacon Laser Control
      - terminal command: beacontune_gtk, usually left open on desktop
      - **[Blue On]**  on all but S2
      - **[Red On] **on S2. (The S2 blue beacon is currently running on the red channel.)
  - Open M7 covers (or check that they are open)
  - On **Obsgtk**, WFS tab
      - Camera SHUTTER **[OPEN]**
      - WFS COOL **[ON]**  (doesn’t hurt even if its already on)
      - SEND **[ON] **Button will be green when on.
      - DM POWER** [ON]**
      - The CAMERALINK and WFS server are now part of Bootlaunch script and the ANDOR camera power should always be on in the POWER gui.
  - On **Obsgtk**, Main tab
      - Set the gain
        - use **[STAND]**  (Gain 300) for IR programs
          - [Stand] will automatically turn on the camera
        - use Gain 100 for VIS programs, W2 and S1 can stay at Gain 300 as they are very dim.
          - have to manually set
            - **[USBOFF]**, Set gain,** [SEND]**, then **[USBON]**.
      - **[RED ON]**.
        - Make sure TELWFS says COOLING ON (to -50C) and says CL 440.
        - If W1 shows no daylight or beacon, the shutter may be stuck,
          - Main tab, **[USBOFF]**
          - WFS tab, click Shutters **[CLOSE]**/**[OPEN]**  a few times, more times in cold weather.
          - **[USBON]**
      - Under the LABAO section, turn camera **[ON]**  and start display** [START]**
        - If labao window is frozen on **obsgtk**, confirm labao is connected at bottom of gui, blue labao label
          - If it is red, **[REOPEN] **on obsgtk
          - If "server is missing" displays, type "sb" in server and enter. If still not connecting, kill and restart server
          - REOPEN on obsgtk and Cosmic Debris
        - If labao window is pale grey (like an inverted image) and shows a circular pattern or spots, it needs a new dark
          - Turn off beacon
          - Control tab, Close M5,
          - Main tab, labao section, **[MKDRK]**
          - Turn on beacon, open M5
  - Turn on the acquisition camera** [ACQ]**
      - If the acquistition view ever gets stuck (not changing) and gets red “failed to open server” message on **obsgtk **or shows red acq at the bottom:
        - Open terminal
        - > s2wfs (or other scope wfs), E1 server runs on e1zwo. Use ssh e1zwo in terminal. Sockman will display where each server runs.
        - > bootlaunch_master
        - Kill acq PID twice to make sure its killed
        - Bootlaunch_master twice to make sure it is removed from sockman and a new acq server started
        - Reopen obsgtk and turn on Acq view

==== Beacon Alignment/Focus Sequence ====

On Obsgtk,

  - Send M7 to default. The End Night sequence usually does this, but it is good to confirm.
      - MAIN tab [M7 DFLT] or WFS tab, **[M7 to DFLT]**
  - Load default boxes on tel WFS
      - Main tab, **[****DEFBX].**  Blue boxes may move to default position from on sky position.
  - Load default flat on __labao__
      - Main tab, **[****LD DEF]**. Default is loaded with every server restart.
  - Load default flat on __tel WFS__
      - Main tab, **[****LD DEF].**
        - Confirm DM curent changes when loading the default. Should range between 0.76-1.61A
  - Make sure DMTILT is on
      - Main tab, **[DMTILT]**, make sure message says "Not ignoring tilt"

Red Beacon:

  - WFS tab, use Beacon buttons to center the **Blue beacon**  in the ACQ hole to fully illuminate the twfs with the red beacon.
      - If aligning before sunset with domes open, you will need to turn the ACQ camera exposure way down to see the beacon.
        - Adjust tab,** [EXP DOWN] **to around 3-10ms.
  - Use the beacon spikes as a guide to center the blue beacon on the cross in the ACQ hole. Zoom in 3-4 clicks using the ZOOM button pop up window to see the cross.
  - A preliminary alignment may be done to confirm the beacons are reaching the cameras, but a precise alignment of the beacons must be done before going on sky
  - If blue beacon is in the ACQ hole, but spots aren't showing in the TWFS, do **[RED ON]**  again until it shows or check TWFS shutter to make sure it opened
  - **Do NOT align the beacon [ALN BCN]**  to the boxes in this step as the beacon is the target for the boxes, the boxes are not the target for the beacon.
  - Use blue starlight boxes with the red beacon or zero beacon offset for the green beacon boxes (ZERO button on obsgtk)
  - You may need to **move the blue starlight boxes**  to center the boxes on the beacon when the beacon is centered on the ACQ hole. Use the Adjust pop up window to move the boxes.
      - Click** [ADJ]**  button at top to bring up ** TV ADJUST popup**
      - Use UP DOWN LEFT RIGHT buttons to center boxes over the beacon and **make the X,Y terms zero**  on the TWFS display at the bottom of the obsgtk.

Blue Beacon:

  - WFS tab, Align blue beacon on labao using **dichroic **buttons.
      - use cheat sheet to get directions of buttons.
  - If spots aren't in the labao display:
      - Turn up blue beacon on beacon laser control.
      - May need to do one big (~1100) click in each direction to find it. Down and Right are the usual directions in the evening.
        - Go one left, if not there, back to center, repeat other direction.
      - Or use the alignment laser
        - **VIS Beams, **move to beam pair.
        - **Laser_filter, **select nd 2.3 or 4.0
        - **Shutters, **turn on laser
        - Move laser spot into hole with dichroic buttons on wfs tab.
          - UP button on dichroic moves up and right on ACQ camera
          - Use large steps, ~2500
        - Turn off laser when done.
  - Toggle boxes to make sure flux is in/touching boxes.
  - On the Main tab, click **ALN DIC**  (only do this at stow!)

Focus:

  - Main tab,** [Focus]**  blue beacon on labao, then **[Focus]**  red on twfs.
  - Can repeat as you wait for the sky to get dark.
  - Make sure to repeat alignments (ALN BCN, ALN DIC, FOCUS, FOCUS) __before slewing__. The optics will drift as the AO board thermally cools after the domes are opened.

Check higher order terms - look at abberation plots or at terms displayed on the wfs gui.

  - Open AO monitor by typing command: aomon
      - Or bring up “TOG ABER” plots (on WFS/TT tab for wfsgtk, on ALIGN tab for labao).
  - If the astigmatism and coma terms reported by the telescope WFS are high
      - obsgtk, main tab, **[****ZCEN] **to update the relative box positions for the red beacon obsgtk to zero out higher order terms
        - or **[ZERO_CENT] **button on telescope wfsgtk -E for engineering version.
      - Only perform the ZERO CENT function when the X,Y and focus terms have been brought to near zero.
  - If aberration terms are high on labao, try making a new flat on labao using the blue beacon. (Note that the terms on labao might not go to zero because of the uneven illumination of the beacon.)
      - LABAO section, SV FLT, FLATEN

Checks to make sure beacons are aligned properly to TWFS:

  * Change tel WFS to 5 msec (default 2 msec; 10 msec on W2) - make sure we see all spots on TWFS. Change back to standard observing values after.
  * Check alignment of blue/red beacon on labao - make sure they are aligned properly with labao boxes. Change FPS as needed so that you see the red beacon. Use red beacon to find center of pattern then turn on blue to be sure the spots are in correct boxes. If not, move dichroic to center with correct boxes. Note that the blue/red beacons are offset by about half a box.

**Aberration plots**: Keep aomon open during the night to montior the aberration terms. If twfs terms get large, do the ZERO CENT procedure.

The path of the IR light will be different from the visible light because of dispersion in the AO dichroic and atmospheric refraction. To compensate for this offset, we have two sets of boxes on the telescope WFS. The blue boxes are for starlight and the green boxes are for aligning the red beacon. The offset between these sets of boxes will change depending on the azimuth and elevation of the target.

The green beacon boxes save the beacon position after the STST alignment. These are defined as an offset to the starlight position, so if you move the starlight boxes, the beacon boxes will also move. If you move the beacon boxes, only they will move.

===== Setting up the Six Telescope Star Tracker =====

We use the Six Telescope Star Tracker (STST) to align the IR starlight in the lab.</font>

Can either align STST during the first slew (see [[:https:chara.gsu.edu:wiki:doku.php?id=chara:new_operating_procedures#alignment_sequence_for_ir_combinersmirc-x_mystic_classic_climb_silmaril|Alignment Sequence for IR Combiners]]) or with STS (see below).

Before the start of the night, use the Six Telescope Simulator (STS) to set the reference boxes on the Six Telecope Star Tracker (STST). Here are brief instructions:

[[https://docs.google.com/document/d/1Ok1e0omwCD1teKlX2sW9izxA3x-IxH91jFQieh2OtSs/edit#|(see STST manual for more details)]]

  - On shutgtk, click on **[STST] **to put the stage into the beam path.
  - On mircx_super_gtk, click **[****STS] **to move stage into beam path and turn on Halogen lamp. (Use stsgtk if you need to adjust halogen flux.)
  - Open the Star Tracker gui by typing **“ststrtd_gtk”**in a terminal.
  - Config camera needed. The default settings are good to Hmag 5 or so.
  - press **[Start Camera]**
  - Click **[Start RTD] **to start the display.
      - if gui is not responding, restart stst server
        - ssh -X spooler@stst
        - killall mircx_stst_server
        - mircx_bootLaunch, type '**no**'
  - Take backgrounds if the display is not evenbackly gray. Close shutter using FILTERS menu. Click Background, wait until finished, about 10-12 seconds (look at message at the bottom of the STST RTD to see status). The “Use Bkg” button should be green after taking backgrounds.
  - Select “Image” from filter menu.
      - if you don't see anything
        - close shutter using Filters menu
        - stop exposure by clicking Start Cam and Start RTD
        - click Config Camera
        - turn on exposure and change filter wheel to Image.
  - Select “Load ref” to read in last saved ref box positions.
  - Set the box size to 50 pixels by scrolling in the box size window (centroiding algorithm now works for boxes larger than 20 pixels).
  - Adjust ref box positions using “Set ref” if STS light is within the boxes or “Adjust ref” buttons if STS light is outside boxes. Adjust ref will open a pop up window that allows you to select a beam and move the green box up, down, left or right with some step size.
  - When ref boxes are well centered on STS, click “Save ref” to save these positions.
  - Keep these reference box positions fixed during the night. The IR combiners are aligned with STS, so we need to align the starlight to the same reference positions.
  - On mircx_super_gtk, turn off Halogen and take out STS.

If the STST camera is already running, you can open a second GUI by typing “ststrtd_gtk” in a terminal and click the “Start RTD” button to start the display. You can adjust the contrast of the display by changing the number in “Flux” box in the upper right of the GUI.


===== Homing the Carts =====

  - Ople GUI, control tab, all the buttons will be green
  - HOME the carts
      - Click **[HOME]**  on each of the active carts.
        - Laser values will go negative.
        - Don't home carts while doing labao alignments
        - When finished, ople server will say “HOMESETDONE”
  - CHECK the carts:
      - Click **[CHECK]**  for each cart, wait about 2-3 seconds before clicking CHECK on the next cart
        - A value of a few microns is usually returned after the cart is checked.
      - If any cart has large values above 20 microns, send back a bit, then home the cart, then check until it shows low values.
  - TRACK the carts at home:
      - Click **[TRACK]**  for each cart
      - Make sure the target position settles at “0.000000” on the OPLE Server *
      - Make sure the “X” lights up and stays steady under the HM column on the Primary OPLE Server for each cart
        - If a cart doesn't have an "X", first send back a bit (0.5-1m) from homeswitch and track again.
      - Make sure that the errors between the laser and target position are small, (0.001-0.040µm is typical), when tracking at the home switch
  - If not opening yet (high humidity, clouds)
      - Send carts back to ~20-30 m.
      - Click **[Off]**  to stop them from moving
  - Or send carts to first object
      - **CD**, Star/Calibrator tab, enter sci/check/cal, click **[Object, Check or Cal 1]**,
      - Control tab, **[STAR ACQUIRED]**
      - Enter position for reference cart (Ref) on Cosmic Debris
        - Stars/Calibrators tab, Bring up** [WHEN]**  plot, ref cart position will be displayed in messages section of CD even if not in delay
        - Or click** [OPT DELAY] **at the bottom of CD for CD to tell you optimal position of reference cart, only works when in delay
        - Set reference postion
          - Control tab, enter value next to REF box, and click **[REF]**
      - On Ople_1 gui, Press **[Start]**  (on bottom row) to start astromod
        - Ople will now get positions for the first object, saves time, makes alignments more accurate.
        - **[Track]**  carts to new positions when values have updated, about 10 seconds.
  - Get carts ready to find fringes.
      - The **[OL] **buttons should be **gray **for **ALL **active carts (including the reference cart). START button does this above.
        - Open Loop sets the Target Position to the expected position computed by ASTROMOD
      - __MIRCX/MYSTIC/SPICA:__  The **[FT]**  buttons should be **gray for all **active scopes **BUT **the mircx reference cart (**W2**)
        - The FT button allows the beam combiner to send offsets to the carts directly through the metrology box.
      - __PAVO/CLASSIC__: The **[MAN]**  button should be depressed **gray for the moving cart**
        - The MAN button allows the beam combiner to send manual offsets to the cart through OPLE.
        - The **[MAN] **button for **the reference cart**  should be **green**.
      - Click **[TRACK] **on the OPLE Control Tab **for all active carts,**  including the reference cart.

===== Going on Sky =====

==== Opening the Telescope Domes and enclosures ====

  - Before opening the domes make sure to check the current weather conditions and the forecast to make sure conditions are safe to open and are likely to be stable as you wait for it to get dark.
  - Open the dome slit if haven't already (high humidity).
      - **obsgtk, **Control Tab, click **[UNL SL]**  and **[SLIT]**
  - Monitor the progress of opening the domes:
      * **Obsgtk, [ON], [SPY1]**
==== Opening Telescope Optics ====

  - **Obsgtk**, Control Tab, open mirror covers. Click only one cover at a time and wait for it to open. Click each button only once, do not click repeatedly:
      - [M1 OPEN] - opens the primary mirror cover (takes ~ 3 minutes to open)
      - [M3 OPEN]
      - [M5 OPEN] - if not already open
      - [M7 OPEN] - if not already open
      - [FIND OPEN] if you want the finder. Note that the W2 and E1? finder covers require two clicks of the [OPEN] button to fully open.
  - Monitor progress of the M1 and M3 covers by using [SPY2] camera on the telescope TV screens. Wait for all covers to open.
  - The Telescope Monitor reports the status of all mirror covers. Close and reopen the Telescope Monitor if it reads UNKNOWN for a telescope, even after hitting REOPEN.

==== Final Alignment Before Slewing ====

  - While still at stow, align red and blue beacon and bring focus terms to zero.
  - **[ZCEN]**  on all scopes but W2. Doing zero centroid will help with the labao corrections once on sky.
==== Enabling The Telescopes ====

  - Make sure telescopes are at stow and have no demand positions
      - El: 90 for all
        - [SPY1] should show the end ring of the telescope framing the top of the open dome.
      - Az:
        - S’s: 82.0
        - E’s: 55.9
        - W’s: 99.3.
  - if at 0,0, the numbers are fake and must not enable until it has correct stow positions.
      - *confirm/update* need to get an idea of where telescope is by going to the dome
        - behind the scope on the ground, a black tape markes stow position.
      - then in dome GUI, manual tab, init, then stow.
  - Make sure there are no demand positions that are different from stow positions (on telescope servers make sure it shows stow position) or the scopes may begin moving without being commanded. Hit **[STOW]**  if any demand positions are not at the stow positions.
  - Enable the telescopes only when ready to slew.
      - **Obsgtk**, **[ENABLE]**
      - Make sure that the telescopes do not move away from their stow positions when enabling the scopes!
      - If they begin moving, **obsgtk,**  control tab, **[STOW] **and make sure that they arrive at the correct stow position.
      - Check for problems with the dome servers by watching each telescope move to the precise stow position, ie. 89.990º and moving to 90.000º.
      - If it stays off the precise stow position, the dome severs may need restarting and the Gains/Fn's will be wrong and need to be brought down to normal.
      - Click **[REOPEN] **on the dome gui if the clock is not the same as the CHARA time.


===== Slewing and Locking on a Target =====

It is typically dark enough to lock on your first star roughly 25 minutes after sunset or the time of twilight reported by Cosmic Debris.

If the first science target/calibrator is faint in V-band, slew to a brighter star to initialize the telescopes, make flats, and do STST alignments.

Slewing to a Target Using Cosmic Debris

  - Enter sci and cals into CD
      - If a star or new target is not recognized by Cosmic Debris, [[:chara:trouble_shooting#adding_or_finding_a_star_in_the_chara_database|follow the procedure for DBADD]].
      - Check to make sure that EL/AZ reported by Cosmic Debris are correct and safe for pointing
      - **Never point below 15-20 degrees elevation.**
      - It is recommended to initialize the telescopes on a target between 40-70 degrees elevation.
  - Hit the **WHEN**  button for your first star to see a plot of cart positions and when it is in delay.
  - Control Tab, Enter ideal reference cart pos into the REF text window, click **[REF]**
      - Press **[Opt delay] **about 5-10 sec after star acq and enter those values
  - On the Star/Calibrator Tab, click **[Slew To]**
  - **Verify that the telescopes are moving**  by checking the azimuth and elevation status in the telescope servers
      - //The left set of numbers show the commanded EL/AZ of the target while the right set of numbers show the current location of the telescope (where)//
      - If the scopes do not move, check that they are enabled. The telescope server will show RA and DEC coordinates if enabled and will show DISABLED if disabled.
      - Also view “SPY1” on **obsgtk**  to check that the telescopes and domes are moving.
      - If the domes do not rotate, **Dome **GUI, click the** [AUTODOME ON] **button to enable the domes
      - Scopes will usually arrive at the elevation of the target first and then continue in azimuth until the star appears in the finder or ACQ window.
      - When the telescope slew finishes, check the SPY 1 to make sure that the domes are aligned with the telescope if no star shows up in the ACQ or Finder cameras.
        - If the star is not in the ACQ window, go to the FIND window and do the same, then return to the ACQ window and repeat.
  - Make sure all carts are tracking to values between 0 and 44m.
      - Make sure** [TRACK]**  is on for all active carts, including the reference cart.
      - The** [OL] buttons should be gray for ALL active carts**  (including the reference cart).
        - **MIRCX/MYSTIC/SPICA**: The** [FT]**  buttons should be gray for all active scopes BUT the mircx reference cart (W2)
        - **PAVO/CLASSIC**: The **[MAN]**  buttons should be depressed gray for the moving carts
          - The [MAN] button for the reference cart should be green.

==== Locking on a Target ====

Locking on a star involves aligning the beacons, locking the star on tiptilt and tel AO, and locking the blue beacon on labAO. TelAO performs fast correction to compensate for atmospheric turbulence. LabAO performs slow correction to compensate for non-common path length errors along the beam path from the telescope to the lab.

Note: W2 does not have a DM installed at the telescope yet (no telAO, only labAO).

The path of the visible and IR light into the lab is different because of atmospheric refraction and dispersion in the AO dichroic. Below we outline alignment steps separately for visible and IR combiners. For IR combiners we use the Six Telescope Star Tracker to align the IR light in the lab.

While tracking on a star, labao will offload X,Y tilt to M7 to keep the the blue beacon centroids aligned to labAO WFS. There are two methods for doing this:

**Old method**: DMTILT is gray (reconstructor ignores tilt). **M7AUTO**  will adjust M7 to keep blue beacon centroids aligned to labAO WFS.

**New method**: **DMTILT**  is green (uses tilt when computing reconstructor - "Not ignoring tilt"). **DMAUTO**  will use labao DM to correct for tilt and offload to M7 to avoid reaching its tilt limit.

**STOP A: **If needed, the 3 STOP A buttons on obsgtk will stop any alignment sequences in progress.

**New procedure (as of 10/12/2023)**  to offload aberrations from the W2 telescope WFS on to the labAO DM. This should provide better corrected images on W2 until we get the DM installed at the telescope. Must be done after every slew. Follow these instructions:

  - Open an engineering version of w2 wfsgtk:
      - wfsgtk -E W2 &
  - WFS/TT tab, toward the bottom, set terms to ~0.5 on Foc, A1, A2, C1, C2 (text boxes to the left of the terms)
  - after slewing, before doing beacon alignments and focus, **[ZEROAB]**
  - When done aligning and locked on star, **[ZEROAB] and [ADDAB] **
  - This must be repeated after every slew for W2.

==== Alignment Sequence for Visible Combiners: SPICA and PAVO ====

For the visible combiners the red beacon and starlight will be coaligned, so you only need to use the blue starlight boxes on the telescope WFS.

**Obsgtk, **main tab, **[ZERO BCN]** at the start of the night to clear any offset between the position of the beacon boxes and the starlight boxes to avoid any confustion

For **SPICA**, make sure to do "**useldc **on" in ople server if you haven't done so already.

   - Slew to star.
  - Main tab, turn on **[M7AUTO] **to hold the blue beacon during the slew and avoid drifting. It will turn off at the end of the slew.
  - CD, **[Star Acquired],**  and set new ref pos with **[Opt delay]**
      - **FOR PAVO: **Do **<font inherit/inherit;;#e74c3c;;inherit>__NOT __</font>**hit **[Star Acquired] **before aligning for PAVO, only do **[STAR ACQUIRED]**  after alignment is finished.
  - **[ALN BCN], **Align red beacon to the blue starlight boxes on the telescope WFS using the beacon flat.
      - Wfs tab, Use beacon buttons if needed on W1 if beacon gets out of the boxes
      - or hit RIGHT with 500 stepsize to get them back in.
  - LABAO section, **[ALN M7], **Align blue beacon to lab WFS using M7.
      - WFS tab, use the yellow M7 buttons to get centered if beacon is not in boxes, then **[ALN M7]**
      - 150,000 step size moves 1 full row. Directions are normal.
  - LABAO section, **[FOCUS] **blue beacon on labao WFS if the focus term is high (moves beacon parabola).
  - TELWFS section, **[FOCUS] **red beacon on TWFS if focus term is high
  - **[RED OFF] **Turn off red beacon.
  - Click **[MOVE]**  button then click on star in ACQ window. This will move star into ACQ hole.
  - Turn off blue beacon and lock star on **[TT]**  only
  - Move TWFS boxes (**ADJ**  popup button on obsgtk) to get **star centered and evenly illuminated on ACQ hole**. (Note: you might need to close DM AO loop during this step.)
  - Lock star on telescope AO servo (**[****DM]**  button on obsgtk). Both TT and DM servos should be on now.
  - Turn on the blue beacon
  - Make sure **DMTilt is green**  (stays on all the time - the message from DM tilt “NOT Ignoring DM tilt…” when it is on. This may take 2 clicks to confirm this message.)
  - Turn on **DM AUTO**, **LABAO**.
  - **For PAVO: **make sure carts are in position, then hit **[STAR ACQUIRED**] once alignments are finished. Star acq. starts the camera on pavo.
  - **For SPICA**:
      - on obsgtk, WFS, use beacon buttons to align beams on ipdet gtk.
        - can use rotation wheel. spica is 90 degrees CW from azimuth of star.
        - use steps 222 - 500 and wait for dm to adjust before clicking again.
      - when finished, on ip_det, press **[FTT SERVO] **and** [OFFLOAD]**.
  - Make sure data acquired line comes in on CD, otherwise do **[DATA ACQ]. **
      - or make sure to **[REOPEN]**  CD during shutters
  - If the coma and astigmatism terms are high on the aberration plots, create a new sky flat
      - Turn off **[DM]**, **[SV FLT]**, **[FLATEN]**, then relock the **[DM]**.
      - If the signal is good on the tel WFS, it is a good idea to make a new sky flat on every star.
  - **Check aberration plots.**  If X/Y terms are not going to zero, try stopping servo loops, recentering alignment, and turning back on servo loops. If X/Y and higher order terms still not going to zero after checking alignment and making a new flat, then this likely means a new reconstructor needs to be made.
  - SPICA uses MIRCX as a fringe tracker and MYSTIC as a fringe finder so having the STST open to monitor the IR beam quality is helpful even if you will not be doing the additional STST beacon flat alignment.
      - Center the star flux in the Crop boxes with the CenBox button and use the drift from center, along with SPICA team input, to make beacon flat corrections as requested.
  - At times, Mircx will have such poor flux that a beacon flat correction to align to the STST reference position will be needed. This will occur on **E1 pop 4 ****and 5**  and **S1 pop 5**  due to their extreme distances from the telescope to the lab.
      - Follow the SPICA team guidance in making this correction to improve the flux to keep Mircx usable as a fringe tracker with those baselines.


==== Alignment Sequence for IR Combiners: MIRC-X/MYSTIC, CLASSIC/CLIMB, SILMARIL ====

The path of the IR light will be different from the visible light because of dispersion in the AO dichroic and atmospheric refraction. To compensate for this offset, we have **two sets of boxes**  on the telescope WFS.

The **__blue boxes__  are for starlight**  and the** __green boxes__  are for aligning the red beacon**. The offset between these sets of boxes will change depending on the azimuth and elevation of the target.

The green beacon boxes help keep track of the offset red beacon motion. These are defined as an offset to the starlight position, so if you move the starlight boxes, the beacon boxes will also move. If you move the beacon boxes only they will move.

Before the start of the night clear any offset between the position of the beacon boxes and the starlight boxes on the telescope WFS using the “ZERO BCN” button on obsgtk.

Faint stars (H > 5 mag) will not be easily visible in STST. Therefore, before slewing to the first star of the night or when slewing to a different region of the sky, slew to to a nearby bright IR star to align the beacon flat and set the beacon box offset on the TWFS. Then slew to faint star, keeping the red beacon at the same fixed position.

**NOTE:**  The STST centroiding/servo'ing now works for longer exposure times. If you change exposure time, you need to: stop the camera, config the camera, restart the camera, and retake backgrounds.

For **H=6 mag**, try changing coadds to 5 or 10 for 1 sec integration.

For **H=8 mag,**  centroiding has been shown to work using 50 sec exposure (1 coadd).

*Please update here* if you determine settings that work good for the using the auto STST alignments at different magnitudes.

**Aligning the scopes before STST:**

  - On shutters GUI, make sure **[STST]**  is in.
  - Slew to a star of Hmag 4 or brighter (or adjust exposure time/coadds on STST as needed to see fainter stars).
      - The Find IRC function on CD can do this quickly.
        - Enter the target on a empty line, press **[HD]**  once (or appropriate designation), and **[Find IRC] **
          - will automatically give you a bright star.
  - Close IR shutters as needed to avoid saturating the combiners.
  - **[M7AUTO]**  can help keep the blue beacon aligned during the slew.
  - On the first slew only,
      - **[ALN BCN], **align red beacon to the __blue star boxes __on the telescope WFS using the beacon flat.
  - On subsequent slews,
      - if the star is bright enough to see in STST (Hmag 5 or brighter), then go to step **12 **
      - if you can't see the star on STST, then align red beacon to the __offset green beacon boxes__  (only after green boxes have been set using STST - see below).
  - **[ALN M7] **Align blue beacon to lab WFS using M7.
      - If beacon is not in boxes, use the yellow M7 buttons on the WFS tab to get centered
      - 150,000 step size moves 1 full row. Directions are normal.
  - Labao section,** [FOCUS]**  blue beacon If focus term is high
  - TWFS section,** [FOCUS]**  red beacon If focus term is high
      - (ONLY DO THIS STEP IF RED BEACON IS ALIGNED TO STARLIGHT BOXES - e.g., first slew)
  - **[RED OFF]**. Make sure you have the blue starlight boxes on telescope WFS.
  - Click **[MOVE]**  button then click on star in ACQ window. This will move star into ACQ hole.
  - Turn off blue beacon and lock star on **[TT]**  only.
  - Move TWFS boxes (use popup from **ADJ**  button on obsgtk) to** get star centered and evenly illuminated on ACQ hole.**
      - (Note: you might need to close DM AO loop during this step.)
  - Lock star on telescope AO servo (**DM**  button on obsgtk). Both TT and DM servos should be on.
  - Turn on the blue beacon.
  - Make sure **DMTilt is green**  (stays on all the time - the message from DM tilt “NOT Ignoring DM tilt…” when it is on. This may take 2 clicks to confirm this message.)
  - Turn on **DM AUTO**, **LABAO servo**.
  - If the coma and astigmatism terms are high on the aberration plots, create a new sky flat
      - Turn off **[DM]**, **[SV FLT]**, **[FLATEN]**, then relock the **[DM]**.
      - **If the signal is good on the tel WFS, it is a good idea to make a new sky flat on every star.**
**Aligning STST:**

  - With the star locked, WFS tab, align STST with the beacon buttons.
  - on STST GUI, use wither Full or Crop view.
      - The green cross marks the center of the box and is your target, the red cross marks the centroid of star.
      - If the star is within the STST reference box,
        - **obsgtk**, main tab, very bottom, click **[STST]**  - automatically aligns the beacon to stst (use step size close to 200). This may be slow, but you can do all six at the same time.
      - If the star is outside of the STST reference box,
        - try increasing the box size or move the beacon manually.
          - Use the rotation matrix wheel printout on the desk to know which way to move.
            - point arrow to the objects azimuth.
            - the direction on the printout corresponds to the direction you want to move on stst.
            - the direction printed on the wheel is the beacon button to press on obsgtk wfs tab.
            - So if 'LEFT' is pointed up, press the left button to move stst up.
          - The first few clicks may not move at all due to hysteresis in the actuators. Do this process slowly, as the labao DM TILT will offload to M7 slowly to compensate for the beacon shift and keep the blue beacon aligned to labao.
          - If the IR flux did not move when clicking the beacon flat, click again as it was taking up slack in the movement. the labao DMTilt can handle up to 3 units of tilt to offload to the M7.
          - If the tilt values reported on the obsgtk labao tilt are lower than 2 units, you can click again. Steps of 333-500 are typical for the first moves.
            - If the required move is large, ~1000 steps on the beacon flat, or about 25 pixels on STST, it may help to turn the Labao Servo and DM AUTO off and manually move the blue beacon with M7 back to center on labao until the flux is closer to the STS center. AUTO M7 can also be used for medium sized moves, ~333-500 steps or so of the beacon flat.
  - **If this is the first STST alignment star,**  then set the offset position of green beacon boxes on TWFS.
      - When the star is aligned on STST
        - turn off the DM servo and TT
        - move the star away with NEXT
        - turn on the Red Beacon.
        - Turn on the boxes
        - Click “TOGG” on obsgtk (or “TOG BEACON” on WFS gtk) to toggle from the blue starlight boxes to the green beacon boxes on telescope WFS.
        - Click “SET” button on obsgtk (or “ALIGN BOXES” on WFS gtk) to move the green beacon boxes to the position of the red beacon. This will center the boxes over the beacon and save the new beacon flat positions in a few seconds. Watch the X,Y values of the twfs go to zero. Alternatively, you can use the buttons on the obsgtk ADJ tab pop up window to manually move them.
  - **If you will record fringes on this star**
      - turn off the red beacon
      - toggle back to the starlight (blue) boxes
      - MOVE the star to center on ACQ
      - Turn on **TT**  and **DM**  servo for TWFS.
      - Turn on **DM AUTO**  (**DM Tilt**  should already be on) and **LABAO**  servo.
  - **If this is an STST alignment star only**
      - slew to the first target and align as needed there, which shouldn't be much or at all.
  - **Check aberration plots. **If X/Y terms are not going to zero, try stopping servo loops, recentering alignment, and turning back on servo loops. If X/Y and higher order terms still not going to zero after checking alignment and making a new flat, then this likely means a new reconstructor needs to be made.
  - **When slewing to the subsequent stars:**
      - If you will see the star on STST, then align red beacon using **STST**  auto-alignment button after locking the star.
      - If you won't see the star on STST, then align the red beacon using the new positions for the green beacon boxes (not the blue starlight boxes), then switch to the starlight boxes when locking starlight. If the slew is close on the sky, you can use the beacon alignment from the previous star. Go back to a nearby bright star if you need to check the beacon alignment on STST (e.g. if flux is lower or fibermaps have degraded on Mircx/Mystic
      - Always lock the star using the blue star boxes.
  - When you finish slewing and locking the star on TT, telAO, labAO, then click **[STAR ACQUIRED]**  on Cosmic Debris.

For programs with bright IR stars, skip align beacon, but do the M7 and labao focus alignments. Then lock the star on the blue boxes and use the STST to fine tune any misalignment of the beacon flat. Check the beacon focus on the twfs occasionally to make sure it stays near zero, every 2-3 stars.

==== Trouble Shooting - AO Performance ====

If AO performance is bad try these steps in the following order:

  * Check aberration plots. If X/Y terms are not going to zero, try stopping servo loops, recentering alignment, and turning back on servo loops.
  * Turn off the blue beacon and check the centering of the star on the ACQ hole. Adjust box positions to center star (ADJ popup on obsgtk). Hint - zoom in on ACQ display.
  * Look at red beacon on tel WFS and try zeroing the centroids using ZCEN on obsgtk. Note if you are using IR combiner, then align red beacon to blue starlight boxes (not the offset green beacon boxes) before clicking ZCEN. Align beacon back to STST reference after.
  * Check alignment of blue/red beacon on labao - make sure they are aligned properly with labao boxes. Use red beacon to find center of pattern then turn on blue to be sure the spots are in correct boxes. If not, move M7 to center with correct boxes. Note that the blue/red beacons are offset by about half a box.
  * If IR flux is low or spots are maps are poor on MIRCX/MYSTIC, then go to bright IR star and check beacon alignment again with STST.
  * Go back to stow and check the full beacon/dichroic alignment using the start of the night procedure. Remember to load all defaults before doing the alignment.
  * If X/Y and higher order terms are still not going to zero after checking alignment, making a new flat, and zeroing centroids, then this likely means a new reconstructor needs to be made. Notify charatech (maybe try taking onsky reconstructor).

==== Trouble Shooting - Locking TT and AO Loops ====

  - If **not locking**, make sure red beacon is off, make sure on blue boxes, load def flat or last flat, make sure no clouds
  - If **not locking**, make sure DM current is low or around 1.2
      - If not, Obsgtk, main tab, LD DEF
  - If **drifting**  after being locked:
      - Unlock TT DM, last flat, make drk, next, lock again, TT
      - DM sometimes makes it hard to hold on to star if using flat.
  - if star is **flaring** and really bright,
      - lower the gain, may need to go lower than 100, maybe even 30 or lower. Lower until star is no longer flaring and spot is compact.
  - May also need to change to higher gain for dimmer stars, Vmag 8 or higher
      - Select [FAINT], wait til wfs starts moving, relock TT DM.
  - After TT DM, Zdark if gain is not 300


===== Optimizing AO Performance on W2 and S2 =====

==== Setting Threshold for S2 WFS ====

The starlight is typically missing from the edge rows of the TWFS on S2. This is likely caused by an alignment problem at the telescope/AOB. We are able to get improved performance on S2 by applying a threshold to ignore the TWFS boxes on the edges without flux. Here are the steps to try this approach again on sky:

  - If light is missing in the left/right rows of S2 TWFS, set a threshold to ignore those boxes.
  - Open: **wfs_gtk S2**
  - The threshold is set on the WFS/TT tab. Enter a number in the box to the left of the **THRESHLD**  button. Then click **THRESHLD**  to set.
  - A threshold of 0.5 to 1.0 seems to work well.
  - Look at DM tab to make sure the edge actuators are not at or close to 1. If they are, try iterating between the default flat and new sky flat so that the edge values are not saturated.
  - The thresholding is based on the standard deviation across the full frame (with starlight). The statistics might be better if you make a dark for the WFS. Note that darks should be made after the sky is fully dark, otherwise the dark will include residual sky light.
  - NOTE: <font 11pt/inherit;;inherit;;inherit>The thresholding works by setting everything below the threshold to</font><font 11pt/inherit;;inherit;;inherit>zero, so if it is too high it will black out the TWFS. You want</font><font 11pt/inherit;;inherit;;inherit>the threshold to be as low as possible while still actually helping out.</font>

==== Offloading Telescope Aberrations on W2 ====

Theo added a routine to offload aberrations from the W2 telescope WFS on to the labAO DM. This should provide better corrected images on W2 until we get the DM installed at the telescope. Follow these instructions:

  - Open an engineering version of the telescope wfsgtk:
      - wfsgtk -E W2 &
  - Toward the bottom of the WFS/TT tab, there is a line with the buttons** [ADDAB]**  and **[ZEROAB]**.
      - The numbers to the left of the aberration terms are the gain for each term.
        - Gain = 0 means no correction.
        - Gain = 1.0 means full correction
      - Clicking **[ADDAB]**  will apply the correction to the terms with non-zero gain.
      - Clicking **[ZEROAB]**  will zero out the corrections that were added to the labao DM.
  - Follow these procedures to offload the W2 aberrations to labao DM:
      - Click **[ZEROAB]**  to zero out any aberrations currently added to the DM.
      - Set **Gain to 0**  on X, Y.
      - Set **Gain to 1 **on Focus, A1, A2, C1, and C2.
        - Note: Norm found that **Gain=0.5**  seems to work well.
      - After locked on the star, click **[ADDAB]**  to add the aberrations.
      - When slewing to a new star, click **[ZEROAB]**  before aligning/focusing the beacons.
      - Note: You can try adding Focus, A1, A2, C1, and C2 one at a time by changing the gain values one at a time.
      - Note: You can try using smaller/larger gains to see what works best.
      - Note: ADDAB now works for astigmatism and coma terms.

===== The Next Slew - Avoid Limit Switch Issues Near AZ 0º: =====

Always pay attention to the AZ you are slewing from or to, to avoid passing the sensors incorrectly and dealing with scopes getting stuck near 90° or 270°.

There are two sensors negative and positive near AZ 0.

It is okay to slew completely past AZ 0 in either direction.

However, if you slew close to zero from the west or away from zero to the east, it is possible you will hit just one sensor.

  * If your next slew is continuing past zero in the SAME direction that you came from, both switches will get hit, and all is fine.
  * If your next slew is going back away from zero, you will not hit the second limit switch and will run into problems later.
      * before going to your next slew away from zero, slew to a different star past zero to hit the second limit switch.
        * can use obsgtk,** [F**i**nd *]**
          * Az: ~355 or 5º, keep el the same, mag 6
        * now you can slew back the other way to the next target.

If you don't pay attention and only hit one sensor at AZ 0, some or all telescopes will continue to stall near 90 or 270 degrees and it is a pain to fix.

Got to [[:chara:trouble_shooting#the_telescope_won_t_move_or_stopped_moving|Troubleshooting]], "Azimuth Limit Switches" Section

===== Additional Observing Procedures =====

==== Manually Slewing a Telescope: ====

If you need to send a telescope to a different star while keeping the others at their current position, in the scope server, type in the star name (use a space instead of an underscore), press enter, then type "next" and enter.

==== Initialize pointing ====

Initialize the pointing of the telescopes on your first target by going to the MAIN tab on the obsgtk and clicking the red [INIT] button. Make sure you are on the correct star before initing the scopes or you will have problems. This may have to be repeated if the pointing drifts during the night. This will allow more consistent and accurate pointing for this part of the sky. You can also INIT during the SKY frames on MIRCX and MYSTIC since the star is in a good spot for finishing a slew.

==== Data Acquired ====

  - Mircx/mystic doesn't always give the Data Acquired line in CD:
      - CD, Control tab, Data Acquired after shutter sequence
      - Make sure combiner(s) are selected or you won't get combiner in data entry for the log

or

  - Hit reopen on CD after data sequence/before shutter foregrounds end
      - Sometimes data acquired will show after shutter sequence finishes

==== Monitoring MIRCX/MYSTIC observing ====

When running MIRCX/MYSTIC, the operator may want to have the windows open to follow the fiber mapping, scanning for fringes and data recording. The windows can be opened with the command mircx_launch_all_guis on a desktop terminal. It will open 5 windows for each combiner, but they do not need to all be open. Close what you don't want to monitor. The fiberexplorer, gdt, rtd, and super_gtk windows are most helpful.

==== Adjusting ACQ Camera Settings at dusk and dawn ====

  - When it is light out, can lower the exposure to make ACQ darker
      - Adjust tab, exp down.

==== Acquisition Laser Alignment if the star does not lock in Tiptilt (Rarely used) ====

  - Set VISBEAMS to correct beams.
  - Laser Filter on ND 3.0
  - If the beacon and fiber covers are open, use the SHUTTERS GUI to open the laser shutter and the corresponding B1-B6 shutters.
  - You can also put in the corner cube with the [CCIN] button.
  - You will see at least two stars and two laser spots in the ACQ window. There is a reflection, hence two images of each. The desired spots are the right ones.
  - Use the “ORIGIN” paddle on the ADJUST Tab of the Telescope GUI or obsgtk to align the TV cross-hairs to coincide with the laser spot.
  - Close the laser shutter and the B1-B6 shutters.
  - * Move the corner cube out with the [CCOUT] button on the Main tab or Control tab on the telescope gui if you used it.
      - It will take about 10 seconds to move out of the the beam.
  - Hit [MOVE] under the TV tracking menu on the Main Tab of the Telescope GUI and then click the right star to bring the star to the cross-hairs.
      - If it doesn't center well, click [MOVE] again and then the star again until the TipTilt plots show that it is getting light.
  - Lock Tiptilt as shown above. Maintain the ACQ laser alignment during the night by adjusting the crosshairs to the star after locking Tiptilt.

==== Changing Ref Cart position: ====

  - Manually
      - Take difference of the Targets pos and Laser position for cart farthest to the back (~44.2) or closest to the front (0.0)
      - If w1 is close to front also, adjust w1, then the next farthest one.
      - Then make up the difference on the ref
  - Opt Delay (check for ref cart pos)
      - Wait like 5 seconds after star acq, CD, Opt Delay,
      - WHEN plot gives starting and ending ref cart position. May not be correct for S1 as ref.
  - If W2 get's dropped from observations but is the ref cart (ex. going from 6T to 5T with no W2):
      - keep W2 in the config
      - turn cart off when its near the back
      - set ref between 44-70 to get delay on other carts.

==== POP changes later in the night ====

  - Once shutters are finished, let carts know where to go
      - CD → Main tab, Data Acquired → slew to new sci/cal
      - Obsgtk → M7Auto
      - Change configuration to new pops, send
        - Make sure ople reads new star
      - Star acquired
  - Do pop changes
      - Obsgtk, Control, **[On]**, M5targ, close M5
      - Shutters, Turn on laser, open vis shutters for beams,
        - If using SPICA, do [SPICA PERI OUT] to see laser. Reopen on shutters gui if it gives an error message.
      - Visbeams, go to beams you want, Laser filters, ND 0.3 or OPEN
      - Popperi, turn on power to pops you want to change, change first pop
        - turn off pop when it finishes on popperi
      - Align pop on obsgtk
        - While aligning one pop, start the other
      - When finished aligning, open M5, ACQ
      - Turn off Laser, close beam shutters,
        - If using SPICA, do [SPICA PERI IN]
  - If staying on the same star, turn off labao servo and load the default flat first.

==== Changing Beam Configuration During the Night ====

If you need to realign the lab dichroic or the IR mirror on the beam sampler stage during the night, make sure to remove STST stage (using the shutter gtk) before doing any alignments. Put STST back in the path after you are finished with the lab alignments and are ready to go on back on sky.

STST gets the beam information from the telemetry server. Try clicking RECONNECT button on ststrtd_gtk. Also try resending the updated configuration from Cosmic Debris and the OPLE GTK (yes try both!). If that doesn't work try restarting the telemetry server by typing the following into ctrscrut:

  * killall telemetry_server
  * bootlaunch_master

Reopen Cosmic Debris and resend configuration again. If that doesn't work then try killing and restarting mircx_stst_server on spooler@stst and running mircx_bootLaunch_master with N option (don't restart all servers - it will mess up mircx/mystic observer).

==== Locking Lab Tiptilt (only used in special cases) ====

With the star aligned with the acquisition ticks, check to make sure there are counts on the TipTilt server or the white plots in the TT windows have condensed.

  - Obsgtk, Main Tab, [TIP/TILT] under the Pointing Servo menu.
      - If this button is not pressed, then the green dots on the tiptilt windows will drift and eventually drop TT lock.
      - The [STAR ACQUIRED] button on Cosmic Debris also starts the Tiptilt servo for all active scopes with one push of a button.
      - On the Tiptilt Servo Control GUI, turn tiptilt [ON] for each telescope or use the [TT ON] button on Cosmic Debris to perform this function for all active scopes.

===== Procedure for Shutting Down at the End of the Night =====

==== End Night Sequence ====

The End Night sequence on Cosmic Debris can be used to end observing and stow the telescopes, carts, and domes and close the mirror covers. It will only stow the active scopes, carts, and domes, so if there are other scopes open, make them active in the Configure tab of CD or stow them manually with the procedure below.

  - Cosmic Debris, [CLEAR JOB QUEUE], [END NIGHT], and [REPORT].
      - Automatically:
        - Stops twfs cameras and closes shutters
        - stow active telescopes
        - Close M1 (primary) and M3 mirror covers,
        - send the active OPLE carts to the back switch
        - close the OPLE socket
        - archive the accumulated data for the night.
      - Does **NOT**:
        - close the dome slits
        - send inactive scopes or carts back.
        - Update the configuration with any inactive scopes to make sure those carts can be sent back manually as well.
        - Doesn't close M5, M7, fiber, beacon, led (red), or laser (blue) laser beacons,
        - Doesn't disable scopes, shut off powers, turn off labao camera,
      - Wait for CD to say end night sequence is complete.
        - Can close Cosmic Debris after you have sent out the Observing Report.
  - On the Report, add the Observers and Baselines entries, add comments to the bottom of the report and put your name at the end.
      - Cut and paste this into an email to CHARA Obs and send it.
      - To archive the report, you must also hit the [Email] button at the bottom of the report to save it.
  - Obsgtk → unlock then close slits, M7, M5, Red, Fiber, Beacon
  - Beacon Laser Control → Turn of red and blue lasers
  - Obsgtk → WFS → Shutters CLOSE, SEND OFF, DM POWER OFF
      - Turn off WFS COOL during when Matt requests it.
      - Never do camerlink off
  - Obsgtk → main → RED OFF, USBOFF, Labao STOP OFF
  - Shutdown STST
      - Click the “Start Camera” button to turn off the camera (the button will go from green to gray).
      - Change the filter wheel to the closed position.
      - Using the shutter gtk, remove the STST stage from the beams.
        - Reopen if needed to get stage to show it is moving out.
      - For now, the daytime alignment is done without STST in the path so it is important to remove STST at the end of the night.
  - Confirm everything is at stow before turning powers off.
  - Power GUI
      - Turn off the power for [TEL AZ], [TEL EL] and [TIP/TILT] for all scopes once stowed
  - obsgtk → main tab, [DISABLE] for all scopes
      - Visually check all telescopes in the spycams (1) to make sure all covers and slits are closed.
  - Obsgtk → Turn the TV OFF
      - Leave the telescope GUIs open until the End Night Sequence is finished. (Cosmic Debris will turn on the TVs and the Spy Cam 2 during the End Night Sequence.)
  - Double check carts are at the backswitch, then turn them off
  - Shut down the ople control system
      - Close ople server
      - Press the shutdown button on the OPLE System Control gui, wait until all 6 dots are grey.
      - Press the Metro button, and wait until it is grey.
      - Close the gui on Wazoo.
      - Turn off the metrology laser (METLAS → Laser Power)
  - Close labao servers
  - If using Classic or CLIMB:
      - Check to make sure the dither is off by looking at the dither status in the CLASSIC/CLIMB server.
        - If necessary, this can be turned off from the dither tab on the Classic/CLIMB GUI. They are off when gray.
      - Turn off [NIRO CPU], and [CLS-DITH], on the POWER gui.
      - Close CLIMB/CLASSIC GUIs and CLIMB/CLASSIC servers (type CTRL C, then Y)
  - Visually check all telescopes in the spycams to make sure all covers and slits are closed. Turn the TV OFF.
  - Leave GPS, Metrol, and Vacmon servers and the Telescope servers open and running

==== Manually stowing the telescopes ====

  - Unlock the tiptilt and AO loops.
  - Obsgtk, → SPY1
  - Control Tab, →[STOW].
  - When the telescopes reach an elevation above 50 degrees, you can begin closing the mirror covers (one at a time, click each button only once).
      - M1 CLOSE,
      - M3 CLOSE,
      - M5 CLOSE,
      - M7 CLOSE,
        - W1, W2, and S2 M7 covers often read MOVING, don't click them again
      - Finder CLOSE
        - W2 and E1 finder covers need a second click to close
  - Use SPY2 to make sure mirror covers close properly.
  - Check the Scope Monitor for mirror cover status.

==== Shutting down the Lab ====

  - Close the vacuum valves for all telescope lines
  - Go to pump shack, close valve, shut off the pump.
      - Note any unusual noises or excessive oil temps (above 54 C) reported by the thermometer. Let Craig, Victor or Steve know of any concerns. Lock the padlock.
  - Go inside the lab with booties:
      - Turn off the key for the green alignment laser and turn the power switch off.
      - Turn off Pico 3 controller above the tiptilt camera.
      - Turn off the blue amplifiers for the metrology laser (button is labeled “line”) *
      - Turn off the key for the metrology laser and place it on top of the laser box.
      - Shut off lights when exiting the lab building.
  - Check for Any signs on the power controllers indicate power was left on.
      - E1 computer is usually on.
        - Push the Off button to turn off the component that is on.
        - Scroll left and right using the UP/DOWN buttons to move to each sign. Make sure all doors are closed as you leave.

==== Back in the Control Room: ====

  - If you run into any problems during the shutdown procedures, send an email to charamnt or directly to Craig and Steve so that they can work to resolve any problems that need to be resolved promptly.
  - Please feel free to record any details in the Observers Notebook.
  - Email a Tech Report if you have other technical information to send out that is not related to the nightly observing runs.
  - Lock front and side doors of Operations Center if you are the last one in the building.
  - Close OPLE building doors; they tend to stick open, please push firmly.

==== Shutting down the Lab ====

  - Close the vacuum valves for all telescope lines
  - Go to pump shack, close valve, shut off the pump.
      - Note any unusual noises or excessive oil temps (above 54 C) reported by the thermometer. Let Craig, Victor or Steve know of any concerns. Lock the padlock.
  - Go inside the lab with booties:
      - Turn off the key for the green alignment laser and turn the power switch off.
      - Turn off Pico 3 controller above the tiptilt camera.
      - Turn off the blue amplifiers for the metrology laser (button is labeled “line”) *
      - Turn off the key for the metrology laser and place it on top of the laser box.
      - Shut off lights when exiting the lab building.
  - Check for Any signs on the power controllers indicate power was left on.
      - E1 computer is usually on.
        - Push the Off button to turn off the component that is on.
        - Scroll left and right using the UP/DOWN buttons to move to each sign. Make sure all doors are closed as you leave.

==== Back in the Control Room: ====

  - If you run into any problems during the shutdown procedures, send an email to charamnt or directly to Craig and Steve so that they can work to resolve any problems that need to be resolved promptly.
  - Please feel free to record any details in the Observers Notebook.
  - Email a Tech Report if you have other technical information to send out that is not related to the nightly observing runs.
  - Lock front and side doors of Operations Center if you are the last one in the building.
  - Close OPLE building doors; they tend to stick open, please push firmly.

===== Useful Commands for Errors =====

Labao server:

  - “oscope”
      - Type in Labao server
      - For **hut and labao error**. Reconnects labao with hut.

Scope Server

  - “otema”
      - Type in Telescope server
      - If **no connection to tema**, then type “otema”
  - “otcs”
      - Type in Telescope server
      - Opens telescope control system, use when **“NO DOME SERVER”**  error in scope server, esp. after dome server restart

Ople Server

  - “ref [some #] 0”
      - Type in primary ople server
      - If **carts aren’t moving to ref**  after entering in CD, type command in server.
  - “oo”
      - Open ople
      - Use if **ople server crashes**  or if you **open ople before ople system controller**
  - HD 1234 type HD or CHARA number of star if Astromod cannot be set by Cosmic Debris

Generic

  - “sb”
      - Start Backgrounds
      - Use when any **server display goes blank**
  - ctrl-L
      - Clears **server error messages**  that block display

===== Common Issues =====

=== Carts ===

  - cart is stuck
      - if you notice the laser position is not moving and does not match the laser position on the ople server, it is likely stuck.
        - turn the cart **[OFF]**, then try sending it the opposite direction with either **[FRONT] **or **[BACK] **button, then **[TRACK]. **it may move fast enough to not get stuck again.
        - if that doesn't work, go to the lab and try giving it a soft nudge to get it unstuck.
        - if it doesn't move, check that the ribbon and cable are on the track and not caught somewhere. Also check the cable spooler is not loose and wind it up if it is.
  - cart error is jumping about 100 microns every 4-6 seconds. On mircx/mystic, fringes will show an oscillating pattern that corresponds to the jump in cart error. In the lab, a faint ticking sound may also be heard.
      - First try restarting ople server.
        - Turn off the carts
        - Restart ople server
        - Track carts.
      - if problem persists, restart ople system controller.
        - On Ople System Controller, restart the ople computer for the cart.
        - When its done, rehome and check the cart. Issue should go away.

=== Hut Error ===

Obsgtk: "**Hut….Error: Relative move error -12"**

If you start to notice that buttons aren't doing anything (usually on E2) such as moving the beacon or turning on the red beacon, try doing one ~1100 step move to the beacon. If you get a "Hut….Error: Relative move error -12", need to recycle the aob and restart hut.

  - Power Gui, turn off the AOB power (ex. E2WFS → AOB)
  - open terminal, 'Tel'wfs (ex. E2wfs)
      - bootlaunch_master
      - kill hut twice to make sure it no longer exists.
  - Turn the AOB back on
  - bootlaunch_master twice again to make the dead hut server has been removed and restarted.
  - reopen CD and obsgtk

=== WFS server crashes ===

On obsgtk, if the telescope wfs goes black, the wfs server has crashed.

  - log in the 'tel'wfs, bootlaunch, kill wfs
  - Reopen Obsgtk or open a new one.
  - If obsgtk displays a message "Telao has not started" (this needs to be updated, see narsi's message in "nighttime technical tracking" channel, search "wfs_cameralink_server" on 11/03/2023
      - wfs tab,
        - open shutters
        - WFS Cool ON
        - Send On On
        - DM Power On
      - on Main tab, wait for cooling to reach -50. Will need to reopen to see it update.
      - once cooled, turn on the beacon, press STAND, and pres REDON.

=== Beacontune not connected ===

  - ssh 'tel'zwo
      - if "no route to host" message, need to go to dome
      - turn off raspberry pi and unplug both cables (black and orange)
      - plug back in and turn back on
      - log in on computer
  - bootlaunch_master
      - make sure its okay or removed from sockman and bootlaunch again.
The CHARA Array Wiki

User Tools

Site Tools

Page Tools
