Don’t it always seem to go That you don’t know what you’ve got ‘till it’s gone Joni Mitchell
In July of1984, I started working as a Systems Analyst at the Zion Nuclear Power Station. After being there for about a year, one of my responsibilities was system administrator for the Process Computers. The Process Computers were these old, 1960s technology computers from Westinghouse that took inputs from throughout the plant and converted them to values for monitoring the plant’s performance. An entire room was dedicated for each of these computers (there was one for each reactor, two in total), with the majority of the room being used for termination points for the cables that ran from sensors throughout the plant. The actual computer consisted of four small CPUs (the primary CPU, a backup CPU, a Process I/O CPU, and a Data CPU if memory serves me correctly) that each had three panels which displayed the flashing hexadecimal code for which instruction it was executing and what memory address it was accessing at the time. There were three black and white workstations hooked up to it, one in the computer room, one in the control room, and one in the Technical Support Center, which was where management would huddle if there was ever a significant event or nuclear accident. There were also printers in the control room that printed key events in the sequence they occurred and time and date stamped alarms when pre-defined limits were reached on the data points.
The data points that fed the process computer were split between digital (on or off) values (for example, breaker switches on a pump) , and analog values (which typically sent a signal of one to five volts), which would use a pre-defined and calibrated polynomial equation to convert the voltages to a value that could be monitored. The Westinghouse process computers were so old that they didn’t have any mass storage capabilities; for that, corporate Comed IT engineers had designed an interface with the Prime mini-computers that, every minute, sent snapshots of all the Process Computer data points. I wrote FORTRAN 77 programs against the Prime Computer databases that retrieved this data, and eventually assumed system administration responsibilities for the Prime Computers as well.
The plant was designed in such a fashion that it could operate safely and fully without either the Process or the Prime computers. However, the nature of technology is once it is available, new uses are found for it, and it becomes relied upon in ways that the designers didn’t foresee. So it was with the Process and Prime computers. The operations and engineering staffs became more and more reliant on the data that the systems served up. This meant that the system administrator had to be available on call should the systems ever fail.
Note that the Process Computers, being 1960s technology, were already, in the mid 80s, approximately 20 years old. This meant that they failed quite frequently. I quickly learned that about 90% of the failures occurred with one of the several same hexadecimal values displayed on the CPUs; I learned what the related cause was and more often than not was able to get the system back up and running pretty quickly. Sometimes I had to call one of the hardware guys, either Dan Z. or Pat M – they had been there a long time and more often than not were able to quickly resolve the problem. When things got sticky, they had diagnostic programs (on Hollerith punch cards!) that they’d run to help pin point the problem. I learned a lot from Dan and Pat, and also from my I.T. cohorts, Denis and Terry.
Note also that Nuclear Power Plants are 24 X 7 operations. This meant that these failures occurred, more often than not, in the night, usually not long after I had fallen into a deep sleep, and my name was at the top of the call-in list. Here’s a little secret: no matter how much I complained about getting roused from a sound sleep, I actually loved it. It was my favorite part of the job.
The reason I liked being called in says something to the nature of my ego. I enjoyed being the guy everybody was counting on, I enjoyed the challenge. I also enjoyed driving the seven miles in through the sleepy town and the empty streets, and the sight and the feeling of the plant at night. I’d get in the almost empty parking lot and the containment buildings would be all lit up with yellow lights. I’d go thru security and enter the service building and take the elevator to the third floor, and walk across the turbine deck to our office in the Prime computer room. The turbines would be loudly humming as they turned, and I’d enter the computer room where our desks were, and it’d be empty and quiet. Then I’d exit into the Unit One Process Computer room, and from there go into the control room, and talk to the operator who called me in. Then I’d get to work on the problem, which 9 times out of 10 was a very easy and quick fix, and I’d talk a bit to the operator, telling him it is fixed and engaging in a moment or two of idle chit chat. I loved the sights and sounds of the plant at night, I loved the warmth of the lights and the humming of the turbine, and I loved the fact that it was just a skeleton crew on duty. Most of all, I loved being needed, being the guy that was counted on, and more than anything, solving the problem.
I was 25 years old when I started working at Zion. About five months later, my wife and I bought the house we still live in to this day. A little more than a year later, in September of 1985, our first child, Jon, was born. In May of 1989, our second son, Nick was born, and in 1994, our daughter Hannah arrived.
I left Zion in January of 1996, after working there for 11 ½ years, to pursue other opportunities. About a year and a half later, they announced plans to shut the plant down, and now, 16 years later, I think the handful of employees left are involved in moving what remains of the spent fuel off of the plant.
There were plenty of times that I hated my job at Zion. There were plenty of times when we wondered just what the Hell management was thinking, and there were plenty of times where it was a grind, and time seemed to crawl. But there were also just as many times where we laughed and fought and triumphed and failed, and looking back on it now, those 11 ½ years, when I was young and starting a family, flew by, and I didn’t realize, I didn’t appreciate, just how good the good times were, and how good it felt to be young and healthy. I think the Joni Mitchell song “Big Yellow Taxi” pretty much sums up how we all experience our youth – we don’t know what we’ve got till it’s gone.
These days I sleep pretty good – five or six hours uninterrupted on good nights. It’s been years since my phone has rung in the middle of the night. But if it were to wake me tonight, it’d be a thrill to go in and walk across the turbine deck and hear the hum of the generator and the rumble of the turbines one last time.
I like this Dave. Parts of it remind me of my nights on call for Wallace Computer Services (now a part of R.R. Donnelley) and when I would drive from Niles, IL to Hillside, IL to walk into their computer room after midnight to solve a problem in a COBOL program. My most memorable on-call accomplishment was when I found that a COBOL program was using a subscript value to point into a table that was indexed (not subscripted). I too enjoyed being the guy they counted on to solve the problem, and I too enjoyed visting with the 3rd shift operators. Those guys are much more intersting than the daytime corporate people. And, I like your wrap-up about how we sometimes miss the good times as they go by.
Nice Post Dave,
I’m the Corporate IT Guy that lead the team of 3 guys that built the OS and System that you are refering to, If you still have contact with Dan and Pat they might still remeber my name. The “1960’s technology was actually 1970 tecchnology BTW. The same system went into Byron and Braidwood Plants. YOu left off the cool Ramtek Colorgraphics display system that we also integrated with this syste and provided rael time colorgraphics to control room operators.
PRK
I remember you, PRK! Denis and I came down to see you on Adams St. a couple of times. You created “WPORT”, among other things, if I recall correctly. And you are right, I forgot about the Ramteks and SPDS.
Sadly, Pat passed away a few years ago – Dan is retired and living in Northern Wis last time I heard.
Glad you liked the post!
Wonderful trip down memory lane. Seems sad that such a huge building, complicated production of energy and hard work by thousands of people in the construction, operation and maintenance of the Zion Nuclear Station is gone. Memories will always bind the men and women who gave Zion it’s soul.