Saturday, March 1, 2014

Mainframe Memories

At the time of writing, it is approaching the 50th anniversary of the announcement of the IBM System/360 aka "The Mainframe". By now, anyone who's been even tangentially involved with this beast is familiar with the number of times it's been written off: most notably in 1991 when the prediction was made that the last mainframe would be switched off in 1996.

 Of course, that never happened. There have been notable losses, including one site dear to my heart, but the mainframe has one thing in common with Mark Twain: "rumors of my death have been greatly exaggerated". Indeed since Linux became available on the platform the mainframe has shaken off its "legacy" (how I loath that term) badge and is at the leading edge of innovation.

 I started with my first mainframe in 1981, when I received a cadetship at the New South Wales Totalizator Agency Board (aka TAB). This was the state government's off-track betting agency which had set up a computer science cadetship scheme where high school graduates would work part time and do university part time for two years before doing the final two years of a Computer Science degree full time. This was a wonderful scheme, the brainchild of its former CEO, the late Allen Windross.
Allen Windross
Although 1981 was the first time I ever got to touch a mainframe, it was a few years earlier that I was first exposed to the technology. I started my high school's basketball program when I was in year 9 and as part of getting it off the ground I scrounged for equipment and supplies wherever I could. One lunch time I was given the okay to go into the sports' storeroom whereupon I came across a box full of manuals and correspondence.

 I don't know how they ended up where they did but in that box were IBM manuals including an introduction to System/360 (S/360), a self-paced assembler course, functional manuals of the S/360 model 20 (sometimes referred to a S/360 in name only), a 1401 autocoder manual, and the minutes of the local S/360 user group.




Over the next couple of years I taught myself assembler and COBOL but had no way of seeing if the programs would even assemble or compile. No matter, I was smitten with the technology.

 As part of the cadetship I was required to work all three shifts in the computer room or three months during my first year. In those days the shifts were 3 x 8 hours starting at 8am, 4pm and midnight. Consequently, I was able to work with a variety of equipment.

 For the hard-core geek here's a run down of the equipment I encountered during my 18 year stay with the TAB. Thanks go to Phil Steele who compiled the list of equipment.

S/360


S/360 Model 44 - we had two of these, the only other two in the country were at the Pine Gap facility in the Northern Territory. These were grand beasts, complete with flashing light front panel and golf ball typewriter console. These were acquired in 1971.
Front Panel of S/360 44

 The configuration included:

  • Each processor had 128K memory storage and operated in dual, master/slave, modes. During the system set-up the storage was extended to 256K each. The software used the IBM Assembler language and eventually ran a specially customised version of DOS Release 19. (Programmers had to get special permission to use storage-to-storage instructions as these were emulated on the '44.)
  • Four IBM 3967 telecommunications processors (which I think were special purpose S/360 processors, but I could be wrong. In any event they had that remarkable TROS technology). 
  • 12 IBM 3970 telecommunications adapters. 
  • 5 IBM 2314 disk drives for each central processor. 
  • One IBM 2415 magnetic tape drive for each central processor. 
  • One IBM 1403-N1 and One 1443-N1 printer (shared via 2914 switch) 
  • One IBM 2501 card reader for each central processor. 
  • One IBM 1442 Card Punch (shared via 2914 switch - note this picture's from one on sale on eBay!) 
  • One IBM 2848 visual display unit controller for each central processor supporting six 2260 VDUs. 
  • Two IBM 1053 golf ball type printers.
  • Model 29 card punch
2501 Card Reader
2260 VDU













The 44's were fun to operate: front panel lights always made you feel like that real work was being done. 

 One story I only came across recently, states that a ruggedized redundant version of the 360/44, the System/4 pi, flew on the space shuttle. Supposedly they whacked an axe into it for a demonstration and it kept running.

 A novel feature of the 2314 was that their addressing could be changed by moving magnetic plugs on the front of the drive.
2314 - The address plugs are the circular entities in the grey area of the top panels

The 2315 tape drives also had a personality all their own. There was nothing quite like the noise they used to make when you loaded a tape and the vacuum started.

 The 1443 printer was sometimes a real pain in the arse. Night shift was a good experience. Most of the time offline processing was finished by 4am so as a junior member of the team would often be able to go home rather than stay until the next shift started. Often we'd start the offline jobs and head out to "Pancakes on the Rocks" for a very early breakfast and then watch the sun rise over Sydney harbour and the Opera House - spectacular. How I wish we had cell 'phones with cameras then: Instagram would be overflowing with such pictures.

 The only fly in the ointment was the weekly (or was it monthly) credit union statement printing jobs. For some reason, the output was directed to the 1443 rather than the infinitely more reliable 1403. The 1443 in operation was a sight to behold. You could watch the hammers rise and fall as they impacted the ribbon and be amazed that they didn't get caught up with each other. Unfortunately they sometimes did. Usually when we were on our second batch of pancakes! Many times we'd return and the print job had failed because the printer hammers had become stuck. There was no going home early on these days.

 By 1985, the 360/44s were retired. The front panel of one was salvaged but the rest and the peripherals were unceremoniously dumped on the back of a truck that was 60cm lower than the loading dock. There was so much gold in those boxes that they were destined for the smelters instead of a museum. 

System/370


 By 1977, the S/360 machines had been joined by S/370. The equipment included:

In our point of sales branches and agencies there were:
Of particular interest to me was the 3410 tape drives which looked like roll-top desks, and the 3704 looked like some sort of mutant washing machine. (For the uber-geeks amongst us the 3705/3704 Principles of Operation makes compelling reading.) 
3705 Front Panel

For the Star Trek fan, the 3340 data module was always fun with which to work.
3340 Data Module - aka NCC-1701

By the time I arrived at TAB, the S/370 135 had been replaced but still in the computer room and was used by the head operator as his own VM/370 host. Robert Bayliss is still involved with mainframe and, like me, is an ex-pat Aussie living in the USA. His choice of password for the user MAINT, however, was the very definition of a weak password (being his favourite 4 letter expletive). So I had fun logging on to a class A user and exploring the ins and outs of the world's best virtualization technology.

By 1980, the S/370 145 was joined by additional equipment:

You will note that we started to refer to the processors by colour. From this point on whenever we acquired additional hardware, one of the requirements was that its doors had to be a different colour to what was currently installed. Consequently, our VM systems were given system identifiers such as "BLUE", "RED", "GREY", "GREEN", and "BROWN". At one point, IBM had to spray paint the doors for us!

 The 3031 was the beginning of the "sterile" mainframe. Gone were the flashing lights, the IPL device dials, and the "stop", "reset" and "load" buttons. There was nothing quite like IPL'ing the earlier boxes: it really felt as if you were doing something as you spun dials and clicked buttons. The 3031 did have an early form of the "System Activity Display" aka "SAD" that let you see what the system was up to, but it just wasn't the same.

 As for human interfaces: Now, I don't know about you, but the 3278 was my all-time favourite terminal. Especially if it had the silk screen protection. There was nothing quite like the feel of the keyboard. Not even the 3279, 3179 etc. came close to the ergonomic perfection that was the 3278. 
3278

The TAB took a slightly backward step in 1982, when it added capacity in the form of a S/370-148 for the new Telephone Betting system. It was bought second hand from a Western Australia-based Iron Ore producer. It was still covered in ore dust when we received it. The box also came with microcode listings which made fascinating reading to a computer-tragic such as me.
IBM S/370 148

The early 1980s also saw the acquisition of the 4341 to replace the older S/370 boxes. Whereas the older boxes looked like "real" computers, the 4341 looked like a rather oversized coffee or card table. However, under the hood it packed a punch for its day with its use of ECPS microcode. It did, however, have one problem for which IBM developed a special RPQ: the stop button on the console is where the PA2 is on a normal 3278. Consequently, operators would sometimes accidentally stop the system when they wanted to clear the screen,

 A 4341-L1 (2MB and 0.4 MIPs) was purchased for exclusive use of the Management Information System (MIS).  Two 4341-K1 (2MB and 0.9 MIPs) were acquired for our cash betting system and for test and development. Quite soon, owing to growth in our betting systems, these were all upgraded to 4341-L12 (4MB and 1.7 MIPs).
IBM 4341
Like our S/360 systems, the TAB online systems (Tabmark 1, Tabmark 2, Telephone Betting, and Race Day Control) were all home-grown transaction systems: written in assembler for speed running on a customized DOS and DOS/VSE operating system. Tabmark 1 and Telebet both had home-grown telecom access methods and we programmed the 3704/3705 devices ourselves. Tabmark 2 used VTAM but it had been modified such that it skipped a couple of the upper levels (which we called VTAM fast-path). 

 By the mid-1980s, with the upgrade to 4341-P02 (16MB and 2.8 MIPS) processor speed had caught up to our requirements and we eliminated the VTAM "fast path" and made the decision to use a high-level language (PL/I - still one of my favourites!) for the re-write of our Telephone Betting application which was to be called PhoneTAB.

 Also, at this time we commissioned a second site for disaster/recovery and split operations. As part of this, the decision was made to run our DOS systems under VM. We had been a VM shop since 1974 and it became an integral part of our system operations and D/R strategy. Communication between these sites was via microwave using Hyperchannel technology to connect the online systems. 

 By this stage we were running VM/SP Release 4: easily my favourite of that family of operating systems. We now had VTAM that enabled connection between our VM systems and our DOS (VSE) based betting systems. The Group Control System (GCS) that was introduced to enable VTAM, being a subset of MVS, also enabled me to write an online system in PL/I that communicated via NPSI to the race courses throughout the state of New South Wales. This coincided with the introduction of the 3725 and, soon after, 3745 communications controllers which replace the aging 3704/3705 devices.
IBM 3745 Communications Controller

The 3340 devices were retired and the 3380 was introduced. Once again, the more aesthetically pleasing hardware was replaced with a bland box: technologically superior but not as pleasing to the eye. 
3380 Drive Assembly

Efficiency Drives Growth which Drives Efficiency

Growth of TAB continued at a rapid rate throughout the late 80s and into the 90s. The new PhoneTAB system was commissioned and soon it was apparent that additional capacity was required. First the 4381-P03 (16MB and 3.5 MIPs) was introduced, but soon an architectural leap forward was undertaken with the introduction of the 3090-150E (32MB and 10.5 MIPs). This leap necessitated the introduction of VM/HPO 4.2 to take advantage of the additional memory. 
IBM 3090 Complex

One of my favourite memories from this time was when we commissioned one of the 3090 boxes located at our disaster recovery site. My wife was with friends so I had charge of my 18 month old daughter Kate. I had to set up and initiate the IOCDS of the "BROWN" system so had to take Kate with me. I sat down at the console and did some work before noticing she was no longer with me. A quick search of the complex revealed her hiding in the cabinet where the bus-and-tag channel cables plugged into the processor. She was quite pleased with herself at finding such a clever hiding spot. Fortunately the statute of limitations has expired on any child endangerment charges I may have faced, besides she's now a 24 year old college graduate - so no harm-no foul!

 Another pleasing result of the 3090 acquisitions was that we made a deal with IBM that they would stop bugging us to get off VSE and onto MVS!

 The rest of 1989-1990 was dominated by re-writes of the Tabmark systems called "Cashbet" and the splitting of the state between the central and remote sites. The intent was that if there were two outlets in close proximity the loss of one site would only affect one of those outlets. This was to be a demonstrated on the TAB's busiest day of the year.


Melbourne Cup Day 1991


 For those readers outside Australia, Melbourne Cup Day, which occurs on the first Tuesday in November, is an event that really does stop the nation. Just after 3pm a 3200m race is run. During the lead up to the race hundreds of millions of dollars are waged. Our systems would take a pounding but would make it through the ordeal with great aplomb. 
Melbourne Cup

Unfortunately, in 1991 this was not to be the case. A long-standing but undiscovered bug in the Cashbet system revealed itself 15 minutes before race start time, taking down half of the state of New South Wales' betting outlets. I had spent my day, as normal, collecting one minutes statistics from our RTM/SF system (aka SMART). (In the old days we'd go down and watch the select light on the 3420 to see how rapidly data was being logged.) On this day the figures were coming in as expected. However, when the bug revealed itself I watched as processor utilization fell through the floor. There's nothing to compare to the feeling in the pit of the stomach as the realization hits you that several million dollars has just been lost. 

 The next 24-48 hours were a blur as the follow-on affects were felt. It was a miserable time. I even made the evening news as background material: my boss, Peter Turner, and I pointed at some GDDM/PGF graphs I called up on the 3279-3 while the CEO spoke to the cameras in front of us.

 The only upside was that I and one of my co-workers, Brook Mathews, were rewarded for our efforts at recovery by becoming part of the crew of "Kookaburra 3" on Sydney Harbour with the (ultimately unsuccessful) Australian America's Cup crew. (This boat had lost to Stars and Stripes in 1987.)


Evolution and Sophistication


 The remainder of the 1990s was a study in evolution rather than revolution. Our 3380 devices were upgraded to 3390s and RAMAC, our 3090-150Es were upgraded to 3090-17T (64MB and 18MIPs), then to 9021-330s (96MB real/128MB expanded and 23.5MIPS) and then replaced entirely by the 9672-RB5 (Dual processor with 2GB and 89MIPs). The 3420 devices were replaced with 3480s and eventually by 3590s. 1993 saw the introduction of ESCON technology at the TAB with the installation of 2 x 9033 directors.
9672

3590
Operating systems-wise, by the end of the 1990s (and the end of my time at TAB) we were running VM/ESA and VSE/ESA. TCP/IP had been introduced in the early 1990s with our first e-commerce site selling its first bet in March of 1997.

 I should also mention that we got our systems running on the P/370, which provided me a great excuse to learn FBA channel programming. We had two of the prototype cards and did the early testing of the SDLC support. 

 We were also very early users of SQL/DS and based our management information systems (and our credit union) on this technology.

 The NSW TAB also set up operations for the Northern Territory TAB as well internationalizing the code when we set up the Hungary on and off-course operations in 1991 running on a small 9021. 


So Long and Thanks for the Processors


 By this time, the New South Wales government had privatized the former Statutory Authority. I left TAB in October, 1999. Unfortunately, the management team that replaced CEO Allen Windross were the case study for the riddle: "Q: How do you start someone off in a small business? A: You give them a big one." and relinquished the TABs former number one position such that its former rival, VicTAB - now known as TabCorp, found it ripe for the picking. 

 To the victors go the spoils and that includes the choice of betting system and hardware platform. Over a period of years the old NSW TAB systems were decommissioned and its 30+ year association with IBM was ended. I'm so glad I wasn't there to see it.

 The people working at the NSW TAB of the 60s, 70s, 80s, and 90s were innovative, collegial, and masters of doing a lot with very little. I remarked at the time of my leaving that I hadn't encountered any team quite like the group I worked with (and that goes outside the technology group). I haven't changed my opinion in the 14+ years that have intervened.

Incontinent Nostalgia 

The 18 years at TAB were a great ride that I took along with the mainframe. But you should understand that while all the model numbers evoke a certain nostalgia, they don't tell the whole story. It was the people and experiences that made my time with the TAB and its mainframes memorable. 

 [Caveat: I'm going to offend people by my poor memory for names, so apologies in advance if you are not mentioned - it's not intentional!]

 From working shifts in the computer room with people like Mark Loftus, John Charlton, John Hay, Laurie Sparks, Mark Dimento, Ziggy, Dave Cowperthwaite (and Patsy from PhoneTAB), the Doctor (yes he had a scarf like Tom Baker), Edie Fiallos, Andrew Cantwell and people whose names are on the tip of my tongue but I can't recall. 

 To my 18 years in the technology group where life long friendships were forged: Brook and Annette Mathews, Chris and Alison Brooks, Bob Peoples, Phil Steele, Tony and Louise Simon, Peter Turner, Bernie Wilkson, Linda and Gonzo, Robyn Lynch, Renata Koch, Greg Conroy, Victor Voros, and scores of others. [By the way Gonzo I finally worked out why NPSI has an error called "Illegal Procedure" - it appears one of the architects was an American Football fan where that's a type of violation.]

 Our IBM account manager, CSRs, and SEs were part of our team - not just guns for hire. Glen Boreham, Tony Best, Peter Allen, Don Krone were amongst of the best of them.
A get together of TAB and IBM old timers
It was people like Dot, Betty and Elaine - our tea ladies; Gordon Williams, Tony Hayes and Peter Brennan from Control Centre; Peter De Low, Peter Harris and so many others, that made my work on the mainframe systems worthwhile. 

 It was the childcare centre where all three of my kids went, where I had the pleasure of dropping them off or dropping in on them that made it nice to go to work.

 It was the Glasgow Arms every Friday afternoon where stories were told and retold until they no longer resembled the facts from which they were derived. It was also the place where many a million dollar design idea began as a scribble on a coaster.
Glasgow Arms Hotel


Thanks for the Memories and the Seeds of Destruction


 So with this being the 50th anniversary of the mainframe I hope IBM also realizes that it's the people and not just its machines or Nobel prizes that have made it what it is. That these same people love what they do and will put up with a lot in order to do it, but there are limits. The mainframe is a great technology but it won't take much to resurrect the low point of 1991 in the interest of short term gain. It's superiority in reliability and virtualization requires cultivation and not slash and burn to squeeze the last cent out of it.


Postscript


 I created a video based on this blog:
Posted by at 19 comments:
Labels: , , , , , , , ,
Subscribe to: Posts (Atom)