Saturday, March 1, 2014

Mainframe Memories

At the time of writing, it is approaching the 50th anniversary of the announcement of the IBM System/360 aka "The Mainframe". By now, anyone who's been even tangentially involved with this beast is familiar with the number of times it's been written off: most notably in 1991 when the prediction was made that the last mainframe would be switched off in 1996.

Of course, that never happened. There have been notable losses, including one site dear to my heart, but the mainframe has one thing in common with Mark Twain: "rumors of my death have been greatly exaggerated". Indeed since Linux became available on the platform the mainframe has shaken off its "legacy" (how I loath that term) badge and is at the leading edge of innovation.

I started with my first mainframe in 1981, when I received a cadetship at the New South Wales Totalizator Agency Board (aka TAB). This was the state government's off-track betting agency which had set up a computer science cadetship scheme where high school graduates would work part time and do university part time for two years before doing the final two years of a Computer Science degree full time. This was a wonderful scheme, the brainchild of its former CEO, the late Allen Windross.
Allen Windross
Although 1981 was the first time I ever got to touch a mainframe, it was a few years earlier that I was first exposed to the technology. I started my high school's basketball program when I was in year 9 and as part of getting it off the ground I scrounged for equipment and supplies wherever I could. One lunch time I was given the okay to go into the sports' storeroom whereupon I came across a box full of manuals and correspondence.

I don't know how they ended up where they did but in that box were IBM manuals including an introduction to System/360 (S/360), a self-paced assembler course, functional manuals of the S/360 model 20 (sometimes referred to a S/360 in name only), a 1401 autocoder manual, and the minutes of the local S/360 user group.




Over the next couple of years I taught myself assembler and COBOL but had no way of seeing if the programs would even assemble or compile. No matter, I was smitten with the technology.

As part of the cadetship I was required to work all three shifts in the computer room or three months during my first year. In those days the shifts were 3 x 8 hours starting at 8am, 4pm and midnight. Consequently, I was able to work with a variety of equipment.

For the hard-core geek here's a run down of the equipment I encountered during my 18 year stay with the TAB. Thanks go to Phil Steele who compiled the list of equipment.

S/360


S/360 Model 44 - we had two of these, the only other two in the country were at the Pine Gap facility in the Northern Territory. These were grand beasts, complete with flashing light front panel and golf ball typewriter console. These were acquired in 1971.
Front Panel of S/360 44

The configuration included:

  • Each processor had 128K memory storage and operated in dual, master/slave, modes. During the system set-up the storage was extended to 256K each. The software used the IBM Assembler language and eventually ran a specially customised version of DOS Release 19. (Programmers had to get special permission to use storage-to-storage instructions as these were emulated on the '44.)
  • Four IBM 3967 telecommunications processors (which I think were special purpose S/360 processors, but I could be wrong. In any event they had that remarkable TROS technology). 
  • 12 IBM 3970 telecommunications adapters. 
  • 5 IBM 2314 disk drives for each central processor. 
  • One IBM 2415 magnetic tape drive for each central processor. 
  • One IBM 1403-N1 and One 1443-N1 printer (shared via 2914 switch) 
  • One IBM 2501 card reader for each central processor. 
  • One IBM 1442 Card Punch (shared via 2914 switch - note this picture's from one on sale on eBay!) 
  • One IBM 2848 visual display unit controller for each central processor supporting six 2260 VDUs
  • Two IBM 1053 golf ball type printers.
  • Model 29 card punch
2501 Card Reader
2260 VDU


















The 44's were fun to operate: front panel lights always made you feel like that real work was being done. 

One story I only came across recently, states that a ruggedized redundant version of the 360/44, the System/4 pi, flew on the space shuttle. Supposedly they whacked an axe into it for a demonstration and it kept running.

A novel feature of the 2314 was that their addressing could be changed by moving magnetic plugs on the front of the drive.
2314 - The address plugs are the circular entities in the grey area of the top panels

The 2315 tape drives also had a personality all their own. There was nothing quite like the noise they used to make when you loaded a tape and the vacuum started.

The 1443 printer was sometimes a real pain in the arse. Night shift was a good experience. Most of the time offline processing was finished by 4am so as a junior member of the team would often be able to go home rather than stay until the next shift started. Often we'd start the offline jobs and head out to "Pancakes on the Rocks" for a very early breakfast and then watch the sun rise over Sydney harbour and the Opera House - spectacular. How I wish we had cell 'phones with cameras then: Instagram would be overflowing with such pictures.

The only fly in the ointment was the weekly (or was it monthly) credit union statement printing jobs. For some reason, the output was directed to the 1443 rather than the infinitely more reliable 1403. The 1443 in operation was a sight to behold. You could watch the hammers rise and fall as they impacted the ribbon and be amazed that they didn't get caught up with each other. Unfortunately they sometimes did. Usually when we were on our second batch of pancakes! Many times we'd return and the print job had failed because the printer hammers had become stuck. There was no going home early on these days.

By 1985, the 360/44s were retired. The front panel of one was salvaged but the rest and the peripherals were unceremoniously dumped on the back of a truck that was 60cm lower than the loading dock. There was so much gold in those boxes that they were destined for the smelters instead of a museum. 

System/370


By 1977, the S/360 machines had been joined by S/370. The equipment included:

In our point of sales branches and agencies there were:
Of particular interest to me was the 3410 tape drives which looked like roll-top desks, and the 3704 looked like some sort of mutant washing machine. (For the uber-geeks amongst us the 3705/3704 Principles of Operation makes compelling reading.) 
3705 Front Panel

For the Star Trek fan, the 3340 data module was always fun with which to work.
3340 Data Module - aka NCC-1701

By the time I arrived at TAB, the S/370 135 had been replaced but still in the computer room and was used by the head operator as his own VM/370 host. Robert Bayliss is still involved with mainframe and, like me, is an ex-pat Aussie living in the USA. His choice of password for the user MAINT, however, was the very definition of a weak password (being his favourite 4 letter expletive). So I had fun logging on to a class A user and exploring the ins and outs of the world's best virtualization technology.



By 1980, the S/370 145 was joined by additional equipment:

You will note that we started to refer to the processors by colour. From this point on whenever we acquired additional hardware, one of the requirements was that its doors had to be a different colour to what was currently installed. Consequently, our VM systems were given system identifiers such as "BLUE", "RED", "GREY", "GREEN", and "BROWN". At one point, IBM had to spray paint the doors for us!

The 3031 was the beginning of the "sterile" mainframe. Gone were the flashing lights, the IPL device dials, and the "stop", "reset" and "load" buttons. There was nothing quite like IPL'ing the earlier boxes: it really felt as if you were doing something as you spun dials and clicked buttons. The 3031 did have an early form of the "System Activity Display" aka "SAD" that let you see what the system was up to, but it just wasn't the same.

As for human interfaces: Now, I don't know about you, but the 3278 was my all-time favourite terminal. Especially if it had the silk screen protection. There was nothing quite like the feel of the keyboard. Not even the 3279, 3179 etc. came close to the ergonomic perfection that was the 3278. 
3278

The TAB took a slightly backward step in 1982, when it added capacity in the form of a S/370-148 for the new Telephone Betting system. It was bought second hand from a Western Australia-based Iron Ore producer. It was still covered in ore dust when we received it. The box also came with microcode listings which made fascinating reading to a computer-tragic such as me.
IBM S/370 148

The early 1980s also saw the acquisition of the 4341 to replace the older S/370 boxes. Whereas the older boxes looked like "real" computers, the 4341 looked like a rather oversized coffee or card table. However, under the hood it packed a punch for its day with its use of ECPS microcode. It did, however, have one problem for which IBM developed a special RPQ: the stop button on the console is where the PA2 is on a normal 3278. Consequently, operators would sometimes accidentally stop the system when they wanted to clear the screen,

A 4341-L1 (2MB and 0.4 MIPs) was purchased for exclusive use of the Management Information System (MIS).  Two 4341-K1 (2MB and 0.9 MIPs) were acquired for our cash betting system and for test and development. Quite soon, owing to growth in our betting systems, these were all upgraded to 4341-L12 (4MB and 1.7 MIPs).
IBM 4341
Like our S/360 systems, the TAB online systems (Tabmark 1, Tabmark 2, Telephone Betting, and Race Day Control) were all home-grown transaction systems: written in assembler for speed running on a customized DOS and DOS/VSE operating system. Tabmark 1 and Telebet both had home-grown telecom access methods and we programmed the 3704/3705 devices ourselves. Tabmark 2 used VTAM but it had been modified such that it skipped a couple of the upper levels (which we called VTAM fast-path). 

By the mid-1980s, with the upgrade to 4341-P02 (16MB and 2.8 MIPS) processor speed had caught up to our requirements and we eliminated the VTAM "fast path" and made the decision to use a high-level language (PL/I - still one of my favourites!) for the re-write of our Telephone Betting application which was to be called PhoneTAB.

Also, at this time we commissioned a second site for disaster/recovery and split operations. As part of this, the decision was made to run our DOS systems under VM. We had been a VM shop since 1974 and it became an integral part of our system operations and D/R strategy. Communication between these sites was via microwave using Hyperchannel technology to connect the online systems. 

By this stage we were running VM/SP Release 4: easily my favourite of that family of operating systems. We now had VTAM that enabled connection between our VM systems and our DOS (VSE) based betting systems. The Group Control System (GCS) that was introduced to enable VTAM, being a subset of MVS, also enabled me to write an online system in PL/I that communicated via NPSI to the race courses throughout the state of New South Wales. This coincided with the introduction of the 3725 and, soon after, 3745 communications controllers which replace the aging 3704/3705 devices.
IBM 3745 Communications Controller

The 3340 devices were retired and the 3380 was introduced. Once again, the more aesthetically pleasing hardware was replaced with a bland box: technologically superior but not as pleasing to the eye. 
3380 Drive Assembly

Efficiency Drives Growth which Drives Efficiency

Growth of TAB continued at a rapid rate throughout the late 80s and into the 90s. The new PhoneTAB system was commissioned and soon it was apparent that additional capacity was required. First the 4381-P03 (16MB and 3.5 MIPs) was introduced, but soon an architectural leap forward was undertaken with the introduction of the 3090-150E (32MB and 10.5 MIPs). This leap necessitated the introduction of VM/HPO 4.2 to take advantage of the additional memory. 
IBM 3090 Complex

One of my favourite memories from this time was when we commissioned one of the 3090 boxes located at our disaster recovery site. My wife was with friends so I had charge of my 18 month old daughter Kate. I had to set up and initiate the IOCDS of the "BROWN" system so had to take Kate with me. I sat down at the console and did some work before noticing she was no longer with me. A quick search of the complex revealed her hiding in the cabinet where the bus-and-tag channel cables plugged into the processor. She was quite pleased with herself at finding such a clever hiding spot. Fortunately the statute of limitations has expired on any child endangerment charges I may have faced, besides she's now a 24 year old college graduate - so no harm-no foul!

Another pleasing result of the 3090 acquisitions was that we made a deal with IBM that they would stop bugging us to get off VSE and onto MVS!

The rest of 1989-1990 was dominated by re-writes of the Tabmark systems called "Cashbet" and the splitting of the state between the central and remote sites. The intent was that if there were two outlets in close proximity the loss of one site would only affect one of those outlets. This was to be a demonstrated on the TAB's busiest day of the year.


Melbourne Cup Day 1991


For those readers outside Australia, Melbourne Cup Day, which occurs on the first Tuesday in November, is an event that really does stop the nation. Just after 3pm a 3200m race is run. During the lead up to the race hundreds of millions of dollars are waged. Our systems would take a pounding but would make it through the ordeal with great aplomb. 
Melbourne Cup

Unfortunately, in 1991 this was not to be the case. A long-standing but undiscovered bug in the Cashbet system revealed itself 15 minutes before race start time, taking down half of the state of New South Wales' betting outlets. I had spent my day, as normal, collecting one minutes statistics from our RTM/SF system (aka SMART). (In the old days we'd go down and watch the select light on the 3420 to see how rapidly data was being logged.) On this day the figures were coming in as expected. However, when the bug revealed itself I watched as processor utilization fell through the floor. There's nothing to compare to the feeling in the pit of the stomach as the realization hits you that several million dollars has just been lost. 

The next 24-48 hours were a blur as the follow-on affects were felt. It was a miserable time. I even made the evening news as background material: my boss, Peter Turner, and I pointed at some GDDM/PGF graphs I called up on the 3279-3 while the CEO spoke to the cameras in front of us.

The only upside was that I and one of my co-workers, Brook Mathews, were rewarded for our efforts at recovery by becoming part of the crew of "Kookaburra 3" on Sydney Harbour with the (ultimately unsuccessful) Australian America's Cup crew. (This boat had lost to Stars and Stripes in 1987.)


Evolution and Sophistication


The remainder of the 1990s was a study in evolution rather than revolution. Our 3380 devices were upgraded to 3390s and RAMAC, our 3090-150Es were upgraded to 3090-17T (64MB and 18MIPs), then to 9021-330s (96MB real/128MB expanded and 23.5MIPS) and then replaced entirely by the 9672-RB5 (Dual processor with 2GB and 89MIPs). The 3420 devices were replaced with 3480s and eventually by 3590s. 1993 saw the introduction of ESCON technology at the TAB with the installation of 2 x 9033 directors.
9672

3590
Operating systems-wise, by the end of the 1990s (and the end of my time at TAB) we were running VM/ESA and VSE/ESA. TCP/IP had been introduced in the early 1990s with our first e-commerce site selling its first bet in March of 1997.

I should also mention that we got our systems running on the P/370, which provided me a great excuse to learn FBA channel programming. We had two of the prototype cards and did the early testing of the SDLC support. 

We were also very early users of SQL/DS and based our management information systems (and our credit union) on this technology.

The NSW TAB also set up operations for the Northern Territory TAB as well internationalizing the code when we set up the Hungary on and off-course operations in 1991 running on a small 9021. 


So Long and Thanks for the Processors


By this time, the New South Wales government had privatized the former Statutory Authority. I left TAB in October, 1999. Unfortunately, the management team that replaced CEO Allen Windross were the case study for the riddle: "Q: How do you start someone off in a small business? A: You give them a big one." and relinquished the TABs former number one position such that its former rival, VicTAB - now known as TabCorp, found it ripe for the picking. 

To the victors go the spoils and that includes the choice of betting system and hardware platform. Over a period of years the old NSW TAB systems were decommissioned and its 30+ year association with IBM was ended. I'm so glad I wasn't there to see it.

The people working at the NSW TAB of the 60s, 70s, 80s, and 90s were innovative, collegial, and masters of doing a lot with very little. I remarked at the time of my leaving that I hadn't encountered any team quite like the group I worked with (and that goes outside the technology group). I haven't changed my opinion in the 14+ years that have intervened.

Incontinent Nostalgia 

The 18 years at TAB were a great ride that I took along with the mainframe. But you should understand that while all the model numbers evoke a certain nostalgia, they don't tell the whole story. It was the people and experiences that made my time with the TAB and its mainframes memorable. 

[Caveat: I'm going to offend people by my poor memory for names, so apologies in advance if you are not mentioned - it's not intentional!]

From working shifts in the computer room with people like Mark Loftus, John Charlton, John Hay, Laurie Sparks, Mark Dimento, Ziggy, Dave Cowperthwaite (and Patsy from PhoneTAB), the Doctor (yes he had a scarf like Tom Baker), Edie Fiallos, Andrew Cantwell and people whose names are on the tip of my tongue but I can't recall. 

To my 18 years in the technology group where life long friendships were forged: Brook and Annette Mathews, Chris and Alison Brooks, Bob Peoples, Phil Steele, Tony and Louise Simon, Peter Turner, Bernie Wilkson, Linda and Gonzo, Robyn Lynch, Renata Koch, Greg Conroy, Victor Voros, and scores of others. [By the way Gonzo I finally worked out why NPSI has an error called "Illegal Procedure" - it appears one of the architects was an American Football fan where that's a type of violation.]

Our IBM account manager, CSRs, and SEs were part of our team - not just guns for hire. Glen Boreham, Tony Best, Peter Allen, Don Krone were amongst of the best of them.
A get together of TAB and IBM old timers
It was people like Dot, Betty and Elaine - our tea ladies; Gordon Williams, Tony Hayes and Peter Brennan from Control Centre; Peter De Low, Peter Harris and so many others, that made my work on the mainframe systems worthwhile. 

It was the childcare centre where all three of my kids went, where I had the pleasure of dropping them off or dropping in on them that made it nice to go to work.

It was the Glasgow Arms every Friday afternoon where stories were told and retold until they no longer resembled the facts from which they were derived. It was also the place where many a million dollar design idea began as a scribble on a coaster.
Glasgow Arms Hotel


Thanks for the Memories and the Seeds of Destruction


So with this being the 50th anniversary of the mainframe I hope IBM also realizes that it's the people and not just its machines or Nobel prizes that have made it what it is. That these same people love what they do and will put up with a lot in order to do it, but there are limits. The mainframe is a great technology but it won't take much to resurrect the low point of 1991 in the interest of short term gain. It's superiority in reliability and virtualization requires cultivation and not slash and burn to squeeze the last cent out of it.


Postscript


I created a video based on this blog:



19 comments:

  1. Fascinating history lesson, Neale. My first brush with a mainframe was in 1984 at Waverley in Bondi Junction. We librarians were allowed to look and admire throught the windows of the computer room.

    ReplyDelete
  2. a jolly good read bring back lots of memories
    I remember the PA2 'feature' really well. it was worse at the comm
    bank. We were using AF/Remote which provides a virtial 30xx hardware
    console hosted on a PC to do multi-hosting, auto IPL and other good stuff
    and while on the real console you had to press ALT then PA2 to stop the
    system, you only needed to press PA2 on AF/Remote (until they changed it
    shortly after our experience). One day at 11am the IBM rep was working
    on the console and a book on a shelf above the keyboard fell off and
    pressed the PA2 key , stopping one of the 4 live mainframes, the one
    running VTAM. result was all transational activity across australia
    halted.When we realised what had happened the head of systems support
    came to the console and couldnt beleieve you could stop the VTAM system by pressing a single
    button and so he pressed it and the system came down a second time on the same day! You couldn't
    have written a movie plot like that.

    ReplyDelete
  3. Thanks for the flashbacks! My 360 days started in 1967 at Florida Power where they had a 360-30 connected to tape and an 1287 (I think) optical scanner for meter readings and payment stubs running BOS/TOS card OS. They also still had 1401 and I got to run FORTRAN compiler on it from cards. As a system programmer starting in 1970 I replaced 360-20 and two 360-30;s with 360-50 running MFT. Then over 25 years moving up through 370-158 and air cooled Hitatchi compatibles, and MVT, VS1, VS2, OS, MVS, HASP, JES, QTAM, VTAM, CICS, ISAM, DB2, etc.

    ReplyDelete
  4. Ah, the good (?) old days. We had a 360/75 with 3MB (one core, two solid-state) and a /44 and a 370/158 at UofWaterloo. Lots of big iron for the time. Then 4 4341s in an SSI complex, which was also really neat, if a bit ahead of its time (FRE016 ABENDs galore--though the 200+ local mods in addition to SSI didn't help!).

    The 3278 was indeed the ne plus ultra of terminals. I fell in love with them in 1980 and wish I could still use one!

    ReplyDelete
  5. 3278s ARE an iconic design. I prefer the mainline production models (without the system console modifications) but for sheer usability, there's been nothing better since. The 3279-3G came close, but the whole integrated experience of the 3278 physical unit engineering was it.

    ReplyDelete
  6. Thanks for bringing back many memories that are in common with mine; I started on a 360/44 at Indiana University / Purdue University in Indianapolis (IUPUI) in 1973, and have also been through most of the machinery you've cited in your article.

    ReplyDelete
    Replies
    1. What's surprised me has been the number of 360/44s there were. I thought they were a rare box, but I've heard from lots of people who had them.

      Delete
  7. Ah, to be one of the forgotten names... As a contemporary of Rob Bayliss, I was a year or 2 ahead of you in the TAB cadetship program.

    I also tested the power of the PA2 key when, in an idle moment at the console, I took the button off to see the mechanism used for the button to change colours. Putting it back on involved a teensy bit too much pressure, and the 375/145 restarted. The dev folks upstairs were none too pleased.

    I also had a Melbourne Cup moment in the TAB computer room, in 1979. Whilst sitting at the console, it sprang to life and wrote something like "TF has reached end of extents". This was fortunately *after* the race had been run so most of the transactions had made it on to the Transaction File, but I still clearly remember the computer room quickly filling up with all the hierarchy.
    Nic Townsend

    ReplyDelete
    Replies
    1. Oops, meant to reply but added a new comment by mistake... see below

      Delete
  8. Hi Nic,
    I think the last time I saw you was at the Deep Purple concert at the Entertainment Centre when George Harrison came on stage.
    After I wrote this blog a lot more names came flooding back including yours, Lorraine, Linda, Georgina etc.
    I caught up with Rob at SHARE in Anaheim a couple of weeks ago.
    Hope you're doing well.

    ReplyDelete
  9. Wow how the time flies. I started in 1972 in a 360/30 at a well known philadelphia PA company. Thank you for the stroll down memory lane. I too worked on almost the same hardware and IBM OS. I will always love cp370 - AKA VM.

    ReplyDelete
  10. Fascinating recount Neale. So many memories.

    I understand why you are glad that you were not around at the end of 2007 when we were removing the last applications from the mainframe, allowing it to be switched off for good. I was tasked with dealing with the last users of the mainframe systems, having to explain to them why the VM applications had to be replaced. I heard many emotional comments from staff who could not imagine how they could continue working without their VM systems, sales outlet support staff, customer management and the others.

    There could be another 50th soon but it won't. The 495 Harris Street building where you started your mainframe adventure would have housed NSW totalizator operations for 50 years in July 2015. That won't happen. We're moving out later this year.
    Renata

    ReplyDelete
    Replies
    1. Hi Renata,
      Yes, Peter de Lowe told me that 495 was not going to be around for much longer. Where are they moving to?

      Neale

      Delete
  11. I agree with your statement regarding the IBM 3278 terminal, the best typing tactile feedback for a touch typist.

    ReplyDelete
  12. Does anybody know why the old 3270 terminals with the metal cases had a cicle etched into the metal on the side? It looked like it was cut part way through so you could punch it out to make a ventilation hole, but I never saw one with a hole there and I never thought to ask.

    ReplyDelete
  13. Enjoyable read, thank you! Words and pics from 80s come flooding back....z EOD.

    ReplyDelete