The CHARA Array Wiki

October 27, 2020

Things Judit had to do in the lab to recover after power outage:

• Shutters - cycled the power using the manual switch on the SHUTTER box in the rack
• Homed Newport stage VIS beams (same as Esp BC2 gui VISB axis)
• Bootlaunch_master on ctrscrut to get the BS servers running
• Homed all 6 Newport BS stages (using guis Esp BS1 and BS2)
• Labao DM electronics were all fine, all those UPSes kept the power
• Started up labao computer
• Pushed power button on the USB hub for labao cameras (on top of the shelf above W1) before starting labao servers so that the servers could find the cameras
• Labao shutters were OK, on same UPSes as labao DM
• Cycled ApertureWheels power and inited all 6 Aperture Wheels
• Started up tiptilt computer → check was forced due to not clean shutdown
• Soft rebooted tiptilt computer with entering BIOS and RE-SAVED it for proper clock start
• Homed all axes of tiptilt zabers (ZABER 2) using ENG gui, because they lost their position. [Best to do this when the beams are aligned with labao and corner cubes are in, so that the movements can be seen in tiptilt gui. Laser ND=6. One click on Home button on any axis homes all axes. After homing, for each axes GO TO SETPOS takes it to the latest position in the cfg file. It is likely that the zaber positions need to be adjusted and fresh set positions saved.]
• All pops were out and PERI positions lost → All 6 PERI axes had to be homed and AFTER THAT set pop5 first with POPPERIGTK and adjusted M10 as needed.

Additional Notes After Power Outage on August, 17, 2021, June 2022, Feb 2024.

• Reinitialize the NIRO Filter Wheel. Open the niro_filter_gtk and click REINIT. Check to see if the indicator wheel on the NIRO camera is pointing to the correct filter after it is homed. Do not turn the wheel by hand. If it points to the wrong filter, home the filter wheel again, and compare counts for each filter (or look for metrology light in J-band) to confirm the correct filters are in position. Then loosen the screw on the indicator wheel and align so that it points to the correct filter label that is currently in.
• Leave plenty of time to do a full check of the alignment on all beams.
• Check alignment of MIRC-X and MYSTIC using STS. Make sure the fiber stages and internal delays are consistent with the positions recorded in the manual/headers. All of the MIRC-X and MYSTIC components are on UPS, so they should retain their positions, but it's always good to check in case one of the UPS fails.
• Home the engineering beam using BC1. A list of engineering beam positions for each beam is on a post-it note near the stage. Note that CLIMB_B1 and CLIMB_B2 are usually not homed and just left in their last good positions. If they are homed, the fringe positions for the CLIMB1 stages: CL1= 7.0565 mm and CL2 = 5.3224 mm. For the CLIMB2 stages: CL4= 2.9630 mm and CL5= 7.8500 mm. (See email from Judit on Aug 8, 2021.)
• Home and re-align lab tiptilt zaber stages. See the link here for tiptilt zaber_2 positions and instructions for aligning the beams.
• Check the PAVO ENG GUI to see if the PAVO zaber stages lost position. If so, prior to the next run, follow instructions in the pavo manual for homing the zaber stages. See special notes for homing the lenslet stage - after aligning, check to make sure you can see light in the camera in both the lenslet and image plane (the homing of the lenslet is not precise so the position might need to be fine tuned).
• Restart network connection on spica-console:
  • The Intel I219 network interface is not natively supported by Linux (something about shipping with an invalid checksum which Windows drivers ignore, but Linux takes to heart). Nils downloaded the driver source code which ignores the checksum problem and placed it at /root/e1000e.x.x.x.x (where x.x.x.x is the version).
  • You should be able to access it with the following command sequence (the observe account should have sudo access):
  • To find out the version:
    sudo ls /root
  • enter that directory:
    sudo cd /root/e1000e.<version>/src
  • remove the old module (if it is still loaded):
    sudo rmmod e1000e
  • build the new driver:
    sudo make install
  • load the new module:
    sudo modprobe e1000e
  • ensure the module is persistent across a reboot:
    sudo dpkg-reconfigure linux-image-$(uname -r)
• The STST chiller will power back on after an outage, but the cooler does not start automatically. Press the settings button to start the cooling again.

Karolina's Notes from Power outage 07/23/2024 - 1 min, scheduled because some electricians work.

After today's power outage, I followed our recover after power outage list and checked in the lab.

• Shutters - were working no power cycling needed.
• Homed Newport stage VIS beams (same as Esp BC2 gui VISB axis) - done
• Bootlaunch_master on ctrscrut to get the BS servers running - done
• Homed all 6 Newport BS stages (using guis Esp BS1 and BS2) - done
• Labao DM electronics were all fine, all those UPSes kept the power - were fine
• Started up labao computer - was running, labao servers started without any problem and detected all cameras
• Pushed power button on the USB hub for labao cameras (on top of the shelf above W1) before starting labao servers so that the servers could find the cameras - no need
• Labao shutters were OK, on same UPSes as labao DM - were ok.
• Cycled ApertureWheels power and inited all 6 Aperture Wheels - done
• Started up tiptilt computer→check was forced due to not clean shutdown - tiptilt computer is off since we did not use it recently
• All pops were out and PERI positions lost→All 6 PERI axes had to be homed and AFTER THAT set pop5 first with POPPERIGTK and adjusted M10 as needed. The Periscopes were ok, didn't lose the positions, I checked them by changing positions.

I did not check any instruments.

Notes added Sept 20, 2024:

• Check the Exhibit Hall Displays. If the slides are not displaying correctly, log on and reboot each computer:
  • ssh -X pi@ehpi01
  • Enter wifi password
  • sudo reboot
• The ehpi's go from 01 to 10.
• You can reboot all 10 of the displays in the exhibit hall using the following command sequence:
  • Log onto the control network (any machine) as user “observe”
  • Issue the command: “eh-raspberry-pi-reboot”

Facitilities Checks:

• The HVAC unit in the pump house looses its memory during a power outage. Power cycle the HVAC unit to reconnect with the thermostat.