The following pdf file hasn't been updated according to recent changes made to the operating procedures on the wiki page. However, it contains useful photos to guide through the power down instructions:
Operating Procedures
Chapter 6: Control System Power Down 16
6 Control System Power Down
6.1 Control System Power Down Intro
In the summer of 2006, the CHARA Array experienced a number of extended power outages.
This was due to Southern California Edison converting the Mount Wilson main line power
from 4 kV to 16 kV in preparation for the higher powered HDTV broadcasts. On several
occasions, inexperienced visiting astronomers were on duty when the outages occurred,
leading to frustration and scientifically lost nights. While the “horses are out of the barn”
at this point, should it be required that less experienced personnel need to power down the
computer systems in the Array, this document will help.
ARRAY POWER DOWN ORDER
6.2 Power Down Order
The CHARA Array control system involves over two dozen computers, four operating
systems, four flavors of real-time control system, hundreds of computer-controlled opto-
mechanical components, and thousands of meters of cabling – electrical, optical fiber, and
network. All of this doesn’t include the office, trailer, and electronic shop computer network.
Needless to say, shutting down the beast is an intricate task.
Since this controlled shutdown may occur while observing (or observing preparations) are
underway, one should follow the normal end-of-night shutdown procedures (see Chapter 5
[Shutdown Checklist], page 13). After this is done, one can begin the computer shutdown
process. It is best to start with the computers that are least inter-connected and then move
up to the “server” class machines, those that all the other machines depend upon. Start by
powering down the computers in the bunkers, then the office, trailer, and electronic shop
computers, then the non-server computers at the west end of the OPLE building, and finally
the servers at the west end of the OPLE building.
6.2.1 Bunker Computers
Each bunker has two computers, a system running a real-time version of Linux, and a
system running DOS. Start with the Linux box. Log onto the machine as ‘observe’, and
Chapter 6: Control System Power Down 17
type the command, startx. Look for the RPC GUI. Make sure all of the devices are off
except ‘TEMA’, the bottom one.
This figure shows a typical RPC GUI at a telescope.
Now go back to the DOS computer. Shut down TCS. Use the arrow keys to move over
to the far right menu item shutDown and select the Manual method of shut down. The item
is selected if the background is blue. Press ⟨ESC⟩ for all queries except the last, ‘Abandon
TCS for System?’ For that, hit the ⟨ENTER⟩ key.
Chapter 6: Control System Power Down 18
This figure shows a typical TCS screen. Note in the lower right that TCS is
about to be manually shut down.
The result of this procedure will drop you into the DOS prompt. At this point, perform
the “three-fingered salute”, ⟨Ctrl⟩-⟨Alt⟩-⟨Del⟩. When the BIOS screen appears, power off the
DOS box. Power off the dome shutter power. This is the switch on the control panel
mounted to the wall above the bunker light switch. The dome shutter power switch has a
power light on the casing and two protective plates on either side of the switch.
Now go back to the Linux box. Exit ‘X’. Perform the “three-fingered salute”, ⟨Ctrl⟩-⟨Alt⟩-
⟨Del⟩. When the BIOS screen appears, power off the Linux box. Now power off the tan ISA
expansion chassis. Power off the UPS unit at the bottom of the computer rack. Finally,
power off the computer monitor and make sure the clock cable is disconnected from the
junction box mounted on the wall.
6.2.2 Office Computers
After shutting down the computers in all six bunkers, move onto the office computers. Cur-
rently, not including personal laptops, there are 13 computers in the Operations Center (the
main office building), a Mac and a Windows XP machine in Bob Cadman/Steve Golden’s of-
fice, an XP machine in Judit Sturmann’s office, another XP machine in Laszlo Sturmann’s
office, yet another XP machine (a laptop) in the conference room, a Linux box in Theo
ten Brummelaar’s office, two Linux boxes in Nils Turner/Antoine M´erand/PJ Goldfinger’s
office, and four Linux boxes and one Mac machine in the control room.
For the Mac computers, shutdown is straightforward. Move the mouse to the upper left
corner of the screen and click on the Apple icon. This will activate a menu in which one of
the selections is ‘Shut Down’. Select this. You’re done with the Macs.
Next, move onto the XP machines. They are also easy to shut down. Click on the ‘Start’
icon at the lower left of the screen. Select ‘Turn Off Computer’ and confirm it in the pop-up
window. You’re done with the XP machines. Make sure you turn off the monitors.
Chapter 6: Control System Power Down 19
The machines ‘allures’ (in Theo’s office) and ‘lem’ (in Nils’ office) have display man-
agers running. If someone is logged in, log him or her out to get back to the display manager.
On the display manager, there will be a button allowing one to shutdown the computer.
Do so. ‘candler’ (also in Nils’ office) doesn’t run a display manager. Power it down by
logging out whoever is logged on and perform the “three-fingered salute”, ⟨Ctrl⟩-⟨Alt⟩-⟨Del⟩.
When the BIOS screen appears, power off ‘candler’. Make sure you turn off the monitors.
This will leave four computers remaining, all Linux boxes, all in the control room.
‘lothlorien’, the computer in the far right corner when looking at the four-screen control
console. That’s a MIRC computer, and we don’t have the root password for that computer,
so just push and hold in the power button on the case to shut it down. Make sure the
BenQ monitor is powered down. The computers ‘pakyderm’ and ‘elefant’, to the left of
‘lothlorien’, are the archive computers. Neither of them has a keyboard, mouse or moni-
tor. To shut them down, log onto ‘zoot’, the control console computer, as ‘observe’. Open
up an ‘xterm’, and in that terminal, type ssh pakyderm. Once logged onto ‘pakyderm’,
su to root and type poweroff. If you don’t know the root password on ‘pakyderm’, don’t
worry about it. Simply log out of ‘pakyderm’ and just power it off in a way similar to
‘lothlorien’. Shutting down ‘elefant’ is done in a similar fashion to ‘pakyderm’. In the
terminal type ssh pease. When logged in, type ssh elefant. At this point the shutdown
is just like that of ‘pakyderm’. At this point, you can log out of ‘zoot’ and perform the
“three-fingered salute”, ⟨Ctrl⟩-⟨Alt⟩-⟨Del⟩. When the BIOS screen appears, power off ‘zoot’.
Make sure you turn off all four monitors.
Finally, power down both printers (the HP 4000TN in Bob’s office and the HP 1120c in
the control room) and go around to all the offices and shut off the UPS units. In the control
room, the UPS units are located on little shelfs below the desk top.
6.2.3 Trailer Computers
Currently, there are only two computers in the trailer office annex, ‘decatur’ and ‘garnett’.
Neither runs a display manager, so you shut them down in a manner similar to ‘candler’
(see [candler], page 19). Once these computers are powered down, turn off the monitors,
the printer, and the UPS units, one located below the desk housing ‘decatur’, and one on
the desk housing ‘garnett’.
6.2.4 Electronic Shop Computers
There are two computers in the electronics shop, one XP box, one Linux box. The Linux
box powers down in a manner similar to ‘candler’ (see [candler], page 19), while the XP
machine powers off, well like an XP box (see [powering off an XP machine], page 18). Make
sure the monitors are off and the UPS unit has been powered down.
6.2.5 Server Farm
At the east end of the OPLE building, within the storage area, is located a server farm.
There are up to eight computers. This is a grab-bag of computers. Some will shut down
in a manner similar to ‘allures’ (see [allures], page 18) and some will follow the ‘candler’
procedure (see [candler], page 19). Shut down all these computers, power off the monitors,
and power off the outlet strip at the back of the shelf unit.
Chapter 6: Control System Power Down 20
6.2.6 Computer Room Computers
Prior to shutting down the (non-server) computers in the computer room, there are a number
of hardware controllers that need to be shut down first. Start with the New Focus 8732
picomotor controllers. There are five of them, and they are all inside the inner enclosure.
Units ‘#1’ and ‘#2’ are located just in front (east) of the W1/W2 BRT/beam switcher table.
Unit ‘#3’ is above the CHARA Classic beam combiner table. Unit ‘#4’ is at the far eastern
end of the inner enclosure, straddling the W1/W2 PoP pipes. Finally, unit ‘#5’ is near the
ceiling, just over the W1/W2 periscopes. Simply turning off the units is sufficient.
Next, shut down the Newport ESP units. There are seven of these, four inside the inner
enclosure, and three outside. Start with the ones inside. Over the visible fringe tracking
table (the 6x8ft table), power down the ‘VIS DITH’ unit. Over the CHARA Classic table
(next to picomotor unit ‘#3’), power down the ‘DITH’ unit. Over the W1/W2 periscopes
(just below picomotor unit ‘#5’), power down units ‘PERI1’ and ‘PERI2’. Moving outside
the inner enclosure, power down the ‘LDC1’, ‘LDC2’, and ‘BS’ units, located on top of the
OPLE cart control racks (the far left racks when you look at the racks from the west). Like
the picomotor control units, simply turning off the ESP units is sufficient.
There are several laptops in that reside in the BC area. These need to be shut down.
Start with ‘lab1’ which lives on the E1/E2 BRT/beam switcher table, near the visible
beams target. This is a Linux box. It doesn’t have a display manager running, so follow
the same shutdown procedure as ‘candler’ (see [candler], page 19). Next move to ‘wild’
which lives on the CHARA Classic table. It shuts down in the same way. Next are ‘dotty’
and ‘daisy’ which live on the metrology table and light source table, respectively. Both
of these run display managers, so following the power down procedure for ‘allures’ (see
[allures], page 18) will be the thing to do.
Make sure the metrology laser controller is off, and if so, power down the UPS unit that
sits on top of the metrology table. Over at the light source table, make sure the alignment
laser is off. The switch for it is a keyed switch on the table cover above the laser. Finally,
make sure the paper cup covers the ob jective to the tip/tilt CCD.
Move back outside of the inner enclosure. At this point you can sit at the console. To the
left of the monitors, you will notice an Oriel white-light source power supply. Make sure it is
off. The console here controls 8 machines: ‘gps’, ‘irimage’, ‘merrill’, ‘michelson’, ‘ople’,
‘pease’, ‘tiptilt’, and ‘wobble’. You can switch between these machines by typing, in
rapid succession, ⟨Ctrl⟩-⟨Ctrl⟩. The arrow keys will then allow you to select to which machine
you wish the keyboard, video display, and mouse to connect. Note that ‘michelson’ and
‘pease’ are not actual machines, but are used to differentiate between the control-side and
office-side ‘ctrscrut’ machines. ‘michelson’ is the master machine. It controls the power
supply to most of the other machines by means of several RPC units. This is done in order
to be able to remotely cycle power on the computer room real-time Linux boxes in case of
runaway servers.
Chapter 6: Control System Power Down 21
This figure shows the full RPC GUI.
If the KVM selection is not already at ‘michelson’, perform the ⟨Ctrl⟩-⟨Ctrl⟩ sequence and
select ‘michelson’. This computer will actually drive both displays. The vacuum monitor
server is usually run from this machine. Shut it down by typing quit in the server window
and following the subsequent directives. At this point, it is safe to shut off the electronics
by means of the RPC GUI (‘RACK_2’, ‘VACMON’).
Next, you want to power down the dither mirror. It is important that the dither mirror
power not be cut while it is in motion. Verify that the dither status is off by launching the
irimage server (left mouse click on the background, follow the ‘Servers’ menu directive,
and select ‘IRIMAGE server’). In the server window, 9th line from the top, far right, you
will see ‘DITH STS:’. If it reports ‘Off’, you can power down the dither mirror electronics
by means of the RPC GUI (‘RACK_2’, ‘DITHER’), and quit out of the irimage server.
If a shutter GUI is not already up, launch one (left mouse click on the background and
select ‘Shutters’). Close all of them and power down the shutter electronics by means of
the RPC GUI (‘RACK_2’, ‘SHUTTERS’).
This figure shows shutter GUI.
At this point, you can start shutting down machines. Start with ‘wobble’. Perform the
⟨Ctrl⟩-⟨Ctrl⟩ sequence and select ‘wobble’. Reboot the machine by typing the “three-fingered
salute”, ⟨Ctrl⟩-⟨Alt⟩-⟨Del⟩. When the BIOS screen appears, perform the ⟨Ctrl⟩-⟨Ctrl⟩ sequence
and select ‘michelson’. Cut the power to ‘wobble’ by means of the RPC GUI (‘RACK_1’,
‘WOBBLE’). Follow a similar sequence of events to power down ‘irimage’, ‘ople’, ‘tiptilt’,
and ‘gdt’ (‘LABPOW’, ‘GDT’).
‘mirkwood’ is a MIRC computer for which we don’t have the root password. Just shut
it down by cutting its power from the RPC GUI (‘RACK_3’, ‘CHAN1’). Finally, power down
the ‘FLUOR G3’ (see [powering off Macs], page 18).
Chapter 6: Control System Power Down 22
6.2.7 Computer Room Servers
You’re almost done. Power down ‘gps’. Perform the ⟨Ctrl⟩-⟨Ctrl⟩ sequence and select ‘gps’.
Reboot the machine by typing the “three-fingered salute”, ⟨Ctrl⟩-⟨Alt⟩-⟨Del⟩. When the BIOS
screen appears, perform the ⟨Ctrl⟩-⟨Ctrl⟩ sequence and select ‘michelson’. Cut the power to
‘gps’ by means of the RPC GUI (‘RACK_1’, ‘GPS’).
Power down ‘pease’. ‘pease’ runs a display manager, so shut it down in a manner
similar to ‘allures’ (see [allures], page 18). When the BIOS screen appears, perform the
⟨Ctrl⟩-⟨Ctrl⟩ sequence and select ‘michelson’. Cut the power to ‘pease’ by means of the RPC
GUI (‘RACK_1’, ‘PEASE’).
Power down ‘michelson’. ‘michelson’ runs a display manager, so shut it down in a
manner similar to ‘allures’ (see [allures], page 18).
Power down the mail server, ‘merrill’. Perform the ⟨Ctrl⟩-⟨Ctrl⟩ sequence and select
‘merrill’. Reboot the machine by typing the “three-fingered salute”, ⟨Ctrl⟩-⟨Alt⟩-⟨Del⟩. When
the BIOS screen appears, reach over to the computer in the rack and press the power button
in the upper right corner of the case. A non-ATX power supply – nostalgic, eh? Finally,
shut off the UPS units. There are three large UPS units at the bottom of several of the
racks, and one smaller unit near the bottom of the rightmost rack.
Revel in the quiet.
Chapter 7: TCS Shutdown and Startup
7 TCS Shutdown and Startup
When it comes to the telescope control system (TCS) computers, there are a few tricks of
the trade that will minimize pain in shutting them down and starting them back up. This
document is designed to pass along that lore.
The TCS computers run PC-TCS, a telescope control software product from Comsoft 1.
It runs best under native DOS, but will run in the DOS emulation mode under Windows
95/98. Typically, we run under DOS, but we do make use of the emulation mode under
Windows for certain tasks, notably, taking data for a pointing model.
The TCS shutdown technique was described previously, in detail, in the discussion of
completely powering down the Array, so it won’t be discussed further here. See [TCS
Shutdown], page 17, for a description.
Starting TCS back up can be tricky. The two things to watch out for are the hand
paddle/drive connector and the dome shutter. The following figure shows you the hand
paddle/drive connector.
This figure shows the hand paddle/drive connector (the top connector with the
3 round cables coming out of it) disconnected from the pulse generator card
(the bottom connector with the flat ribbon cable).
When starting up TCS, it is best to disconnect the hand paddle/drive connector from
the ribbon cable connector.
1
1552 West Chapala Drive, Tucson AZ 85704, 520-621-3648, comsoft@primenet.com
Chapter 7: TCS Shutdown and Startup 24
Next, we have to make sure the dome shutter power is off. The location of the power
switch is shown in the following figure.
This figure shows the location of the dome shutter switch.
Frequently, one has to shut down and restart TCS while the dome shutter is open.
Ideally, one would close everything up tight and then shut down TCS, but the dome shutter
takes so long to close and then open back up. With the power off while TCS starts, one
doesn’t have to worry about the shutter state being in limbo.
To start TCS:
• Make sure the hand paddle/drive connector is disconnected.
• Make sure the dome shutter power is off.
• Power up the TCS computer. Once DOS starts, it will automatically go into TCS.
• Once the TCS command screen comes up, and the menu bar appears along the bottom,
reconnect the hand paddle/drive connector to the ribbon cable connector.
• Synchronize the shutter menu entry with the actual state of the shutter (see the de-
scription and figure below).
• Restore power to the shutters.
• Home the dome.
Synchronizing the Dome Shutter:
The figure below shows TCS with the ‘Motion’ menu enabled.
Chapter 7: TCS Shutdown and Startup 25
This figure shows the TCS screen with the ‘Motion’ menu enabled.
The sub-menu entry of interest is ‘Open Shutter’. If the dome shutter is actually open
to the night air, there should be a ‘»’ in front of the ‘Open Shutter’. If they are not there,
put them there by using the arrow keys to highlight the ‘Open Shutter’ entry and pressing
⟨Enter⟩. If the dome shutter is actually closed, make sure that there are no ‘»’ next to
‘Open Shutter’. If they are there, remove them by using the arrow keys to highlight the
‘»Open Shutter’ entry and pressing ⟨Enter⟩. Once the ‘Open Shutter’ entry matches the
actual shutter state, it is safe to restore power to the dome shutter.
Chapter 8: Enclosure Cylinder Reinitialization 26
8 Enclosure Cylinder Reinitialization
Each telescope enclosure has 2 cylinders, the upper (sometimes referred to as the outer ) and
the lower (sometimes referred to as the inner ). Each cylinder is individually controlled by
a digital servo motor and amplifier combo, sold as the Kol lmorgen ServoSTAR CD, where
the motor is out at the enclosure while the amplifier is in the bunker. The current state
of the system, settings, etc., are kept in the EEPROM and RAM of the amplifier unit. As
long as the amplifier power is not cut, the amplifier knows all. However, if there is a power
outage . . .
8.1 Hardware
The ServoSTAR CD amps are wired to keep tabs on 3 limit switches: an upper limit, a lower
limit, and a home switch. The upper and lower limits work in a “normally closed” mode,
i.e., motion of the cylinder is allowed as long as the limit switch circuit is closed. Triggering
the limit (or cutting the limit switch wiring) will open the circuit and the amplifier will stop
the motion of the cylinder. The home switch operates in a “normally open” mode, with
an inverse behavior to the “normally closed” mode. In both the upper and lower cylinders,
the home switch is between the upper and lower limits. In the case of the upper cylinders,
the home switch is 20-40cm below the upper limit. In the case of the lower cylinders, the
home switch is 10-20cm above the lower limit. The system will allow one to “back out” of
a limit switch trip without having to disable the limit switches.
When the ServoSTAR CD amps are first powered up, approximately 100kB of informa-
tion is loaded from the EEPROM to the RAM. This information contains things such as
default acceleration rates, velocities, motor voltages, current limits, torque values, etc. The
amp also assumes that the cylinder is located at the home switch, so it sets the “motor en-
coder count” to zero. For information, 6.3 million counts will move a cylinder approximately
1cm.
The ServoSTAR CD amps take their commands from the bunker Linux box. The Linux
OS (Red Hat 6.2) starts a server at boot time which controls both cylinders via 3 serial
ports and the joystick port. When a client makes a request of the server to move a cylinder,
the server queries the amp for its current position. If the server is told that the cylinder is
near zero, it checks whether the home switch is closed. If the home switch is still open, the
server assumes that amp has lost its way, and it issues a homing command. By default, the
lower cylinders home by moving up, and the upper cylinders by moving down. Once the
homing is complete, the server will honor the clients motion request. This is how things
are supposed to work. Sometimes, the situation gets gnarled. We have found this gnarling
happens most often when there are several clients open on various machines, each with
conflicting commands in their buffers. Sometimes, one has to resort to manual homing,
described in the following section.
8.2 Manual Homing Procedure
8.2.1 Error Codes
These observations need to be verified.
• In an upper cylinder, ‘L2’ indicates that the CW limit has been tripped. This happens
if the upper cylinder goes into its closed limit. It can be moved off of the limit by
Chapter 8: Enclosure Cylinder Reinitialization 27
incrementally moving the cylinder in a negative direction. There’s no need to disable
the limits.
• In a lower cylinder, ‘L1’ indicates that the CCW limit has been tripped. This happens
if the lower cylinder goes into its closed limit. It can be moved off of the limit by
incrementally moving the cylinder in a positive direction. There’s no need to disable
the limits.
8.2.2 Serial Ports
The lower cylinder is controlled by serial port ‘/dev/ttyC3’, while the upper cylinder is
controlled by ‘/dev/ttyC4’. The comm parameters are 9600-8-N-1. ‘/dev/ttyC5’ controls
the Practical Instruments DIO port which does I/O for the alarm, emergency stop, and
joystick key control.
8.2.3 Kollmorgen Terminal Mode
The following table lists the basic commands that will be needed in manually homing the
cylinders.
‘STOP’ Stop the motion of the motor, with the appropriate deceleration.
‘MH’ Perform the homing procedure, starting in the default direction.
‘HOMESTATE’
Report on the status of the homing procedure. Important here is the value ‘10’,
which indicates that the homing is complete.
‘MA <pos> <vel>’
Move to the absolute motor encoder position represented by ‘<pos>’, at a ve-
locity represented by ‘<vel>’.
‘MI <pos> <vel>’
Move, incrementally, ‘<pos>’ motor encoder counts from the current position,
at a velocity represented by ‘<vel>’.
‘PFB’ Print the current motor encoder position.
‘LIMDIS’ Indicate the status of al l the end-of-travel limit switches. ‘0’ indicates that
the limit switches are enabled, while ‘1’ indicates that they are disabled. In
addition, disabled switches will make their status know by blinking the LED
dot on the segmented display of the amplifier.
‘LIMDIS 1’ Disable all end-of-travel limit switches. The LED dot on the segmented display
of the amplifier will blink.
‘LIMDIS 0’ Enable all end-of-travel limit switches.
8.2.4 Procedure
• Make sure you are at the telescope bunker of the telescope in question before performing
a manual homing procedure.
• Disable the encld server by issuing the command (as root), ‘service encld stop’.
• Depending on whether the cylinder in question is an upper or a lower, issue the com-
mand (as user observe), ‘minicom upper’ or ‘minicom lower’. Press enter a few times
until you get the ‘→’ prompt.
Chapter 8: Enclosure Cylinder Reinitialization 28
• Move incrementally (‘MI’ command) in 4 million step increments (about 5 mm actual
cylinder motion) away from the limit (see [Error Codes], page 26 for information on
the correct direction to move) until the LED indicator on the drive unit shows ‘8.’ (the
LED dot should be steady).
• Issue the command, ‘MH’.
• You can monitor the homing progress with the command, ‘HOMESTATE’.
• Once the homing is done, exit minicom with the key sequence, ‘C-a; x; <ENTER>’.
• Reenable the encld server by issuing the command (as root), ‘service encld start’.
———————————————————————————