mainframe

File organization in really old COBOL code.

May 7, 2020 Mainframe No comments , , , , , , , , , , , , ,

I encountered customer COBOL code today with a file declaration of the following form:

000038   SELECT AUSGABE ASSIGN TO UR-S-AUSGABE            
000039    ACCESS IS SEQUENTIAL.                   
...
000056 FD  AUSGABE                                                     
000057     RECORDING F                                                  
000058     BLOCK 0 RECORDS                                              
000059     LABEL RECORDS OMITTED.                                       

where the program’s JCL used an AUSGABE (German “output”) DDNAME of the following form:

//AUSGABE   DD    DUMMY

The SELECT looked completely wrong to me, as I thought that SELECT is supposed to have the form:

SELECT cobol-file-variable-name ASSIGN TO ddname

That’s the syntax that my Murach’s Mainframe COBOL uses, and also what I’d seen in big-blue’s documentation.

However, in this customer’s code, the identifier UR-S-AUSGABE is longer than 8 characters, so it sure didn’t look like a DDNAME. I preprocessed the code looking to see if UR-S-AUSGABE was hiding in a copybook (mainframe lingo for an include file), but it wasn’t. How on Earth did this work when it was compiled and run on the original mainframe?

It turns out that [LABEL-]S- or [LABEL]-AS- are ways that really old COBOL code used to specify file organization (something like PL/I’s ENV(ORGANIZATION) clauses for FILEs). This works on the mainframe because a “modern” mainframe COBOL compiler strips off the LABEL- prefix if specified and the organization prefix S- as well, essentially treating those identifier fragments as “comments”.

For anybody reading this who has only programmed in a sane programming language, on sane operating systems, this all probably sounds like verbal diarrhea.  What on earth is a file organization and ddname?  Do I really have to care about those just to access a file?  Well, on the mainframe, yes, you do.

These mysterious dependencies highlight a number of reasons why COBOL code is hard to migrate. It isn’t just a programming language, but it is tied to the mainframe with lots of historic baggage in ways that are very difficult to extricate.  Even just to understand how to open a file in mainframe COBOL you have a whole pile of obstacles along the learning curve:

  • You don’t just run the program in a shell, passing in arguments, but you have to construct a JCL job step to do so.  This specifies parameters, environment variables, file handles, and other junk.
  • You have to know what a DDNAME is.  This is like a HANDLE in the JCL code that refers to a file.  The file has a filename (DSNAME), but you don’t typically use that.  Instead the JCL’s job step declares an arbitrary DDNAME to refer to that handle, and the program that is run in that job step has to always refer to the file using that abstract handle.
  • The file has all sorts of esoteric attributes that you have to know about to access it properly (fixed, variable, blocked, record length, block size, …).  The program that accesses the file typically has to make sure that these attributes are all encoded with the equivalent language specific syntax.
  • Files are not typically just byte streams on the mainframe but can have internal structure that can be as complicated as a simple database (keyed records, with special modes to access them to initialize vs access/modify.)
  • To make life extra “fun”, files are found in a variety of EBCDIC code pages.  In some cases these can’t be converted to single byte iso-8859-X code pages, so you have to use utf-8, and can get into trouble if you want to do round trip conversions.
  • Because of the internal structure of a mainframe file, you may not be able to transfer it to a sane operating system unless special steps are taken.  For example, a variable format file with binary data would typically have to be converted to a fixed format representation so that it’s possible to seek from record to record.
  • Within the (COBOL) code you have three sets of attributes that you have to specify to “declare” a file, before you can even attempt to open it: the DDNAME to COBOL-file-name mapping (SELECT), the FD clause (file properties), and finally record declarations (global variables that mirror the file data record structure that you have to use to read and write the file.)

You can’t just learn to program COBOL, like you would any sane programming language, but also have to learn all the mainframe concepts that the COBOL code is dependent on.  Make sure you are close enough to your eyewash station before you start!

Interesting z/OS (clang based) compiler release notes.

December 13, 2019 C/C++ development and debugging. No comments , , , ,

The release notes for the latest z/OS C/C++ compiler are interesting.  When I was at IBM they were working on “clangtana”, a clang frontend melded with the legacy TOBY backend.  This really surprised me, but was consistent with the fact that the IBM compiler guys kept saying that they were continually losing their internal funding — that project was a clever way to do more with less resources.  I think they’d made the clangtana switch for zLinux by the time I left, with AIX to follow once they had resolved some ABI incompatibility issues.  At the time, I didn’t know (nor care) about the status of that project on z/OS.

Well, years later, it looks like they’ve now switched to a clang based compiler frontend on z/OS too.  This major change appears to have a number of side effects that I can imagine will be undesirable to existing mainframe customers:

  • Compiler now requires POSIX(ON) and Unix System Services.  No more compilation using JCL.
  • Compiler support for 31-bit applications appears to be dropped (64-bit only!)
  • Support for C, FASTLINK, and OS linkage conventions has been dropped (XPLINK only.)
  • Only ibm-1047 is supported for both source and runtime character set encoding.
  • C89 support appears to have been dropped.
  • Hex floating support has been dropped.
  • No decimal floating point support.
  • SIMD support isn’t implemented.
  • Metal C support has been dropped.

i.e. if you want C++14, you have to be willing to give up a lot to get it.  They must be using an older clang, because this “new” compiler doesn’t include C++17 support.  I’m surprised that they didn’t even manage multiple character set support for this first compiler release.

It is interesting that they’ve also dropped IPA and PDF support, and that the optimization options have changed.  Does that mean that they’ve actually not only dropped the old Montana frontend, but also gutted the whole backend, switching to clang exclusively?

JCL equivalent of head -1.

June 12, 2019 Mainframe 1 comment , , , ,

Suppose you wanted to do the equivalent of the following Unix shell code on the mainframe in JCL:

head -1 < UT128.SYSOUT.EXPECTED > $TID.$CID.SYSOUT.ACT
head -1 < UT128.COBPRINT.EXPECTED > $TID.$CID.COBPRINT.ACT

Here’s the JCL equivalent of this pair of one-liners:

There are probably shorter ways to do this, but the naive way weighs in at 22:2 lines for JCL:Unix — damn!

I can’t help but to add a punny comment that knowing JCL must have once been really good JOB security.

Mainframe development: a story, chapter 1.

April 19, 2018 Mainframe No comments , , , , , , , , , , , , ,

Once upon a time, in a land far from any modern developers, were languages named COBOL and PL/I, which generated programs that were consumed by a beast known as Mainframe. Developers for those languages compiled and linked their applications huddled around strange luminous green screens and piles of hole filled papers while chanting vaguely latin sounding incantations like “Om-padre-JCL-beget-loadmodule-pee-dee-ess.”

In these ancient times, version control tools like git were not available. There was no notion of makefiles, so compilation and link was a batch process, with no dependency tracking, and no parallelism. Developers used printf-style debugging, logging trace information to files.  In order to keep the uninitiated from talking to the Mainframe, files were called datasets.  In order to use graphical editors, developers had to repeatedly feed their source to the Mainframe using a slave named ftp, while praying that the evil demon EBCDIC-conversion didn’t mangle their work. The next day, they could go back and see if Mainframe accepted their offering.

[TO BE CONTINUED.]

Incidentally, as of a couple days ago, I’ve now been working for LzLabs for 2 years.  My work is not yet released, nor announced, so I can’t discuss it here yet, but it can be summarized as really awesome.  I’m still having lots of fun with my development work, even if I have to talk in languages that the beast understands.

Propublica’s IBM age discrimination investigation

March 22, 2018 Incoherent ramblings No comments , , , , ,

Not too long after I quit IBM for LzLabs in 2016, I was sent a copy of Pro-publica’s survey about age discrimination based firing and forced retirement at IBM. It appears that this survey was just the start of a very long investigation, and they’ve now published their story.

I wasn’t forced out of IBM, and am only ~45 years old, but at the time I had close to 20 years at IBM (including my student internship), and could see the writing on the wall. Technically skilled people with experience were expendable, and being fired or retired with gusto. To me it looked like 25 years at IBM was the firing threshold, unless you took the management path or did a lot of high visibility customer facing work.

IBM’s treatment of employees in the years leading up to when I quit was a major part of my decision to leave. I considered my position at IBM vulnerable for a number of reasons. One was my part time status (80% pay and hours), as I’d been slowly studying physics at UofT with a plan of a future science based job change. Another was that I was a work in the trenches kind of person that did not have the high visibility that looked like it was required for job security in the new IBM where the quarterly firing had gotten so pervasive that you could trip on the shrapnel.

Even after two years I still use “we” talking about my time as an IBMer working on DB2 LUW, as I worked with people that were awesome (some of which I still work with at LzLabs.) Despite now competing with IBM, I hope they stop shooting themselves in the gut by disposing of their skilled employees, and by treating people as rows in resource spreadsheets. It is hard to imagine that this will end well, and it’s too easy to visualize an IBM headstone sharing a plot with HP and Sun.

When I was recruited for LzLabs, my options seemed like continue working for IBM for <= 5 more years before I too got the ax, or to ride into the wild west working as a contractor for a company that was technically still a “startup”. Many startups don’t make it 5 years before folding, so even in the worst case it looked like no bigger risk than IBM, but I thought I was going to have a lot of fun on the ride. LzLabs was just coming out of stealth mode when I was interviewed, but had an astounding ~100 people working at that point! Salaries add up, so it was clear to me that LzLabs was not really a startup in the conventional sense of the word.

It is amusing to read the Pro-publica article now, as most of LzLabs employees are probably over 65. At 45 I’ve been singled out in staff meetings as the “young guy”. Many of the LzLabs employees are technically scary, and know the mainframe cold. I once wrote a simple PowerPC disassembler, but that’s a different game than “disassembling” 390 hex listings by chunking it into various fixed size blocks hex sequences in an editor so it can be “read” by eye!

In less than one month I’ll have been working for LzLabs for 2 years, about six months of which was a contractor before LzLabs Canada was incorporated. Two years ago, if you had mentioned JCL, LE, PL/I, COBOL, QSAM or VSAM (to name a few) to me, I’d have known that seeing COBOL is a good reason to get to an eye wash station pronto (it still is), but would not have even recognized the rest. It’s been fun learning along the way, and I continually impress myself with the parts that I’ve been adding to the LzLabs puzzle. Our technology is amazing and I think that we are going to really kick some butt in the marketplace.

A JCL sample, writing IO to a DATASET (mainframe filename)

November 19, 2016 Mainframe 1 comment , , , , ,

The mainframe and it’s scripting language is a weird beast. Here’s a sample of the scripting language

//LZIOTEST JOB
//A EXEC PGM=IOTEST
//SYSOUT DD SYSOUT=*
//*SYSPRINT DD SYSOUT=*
//SYSPRINT DD DSN=PJOOT.OUT6,DISP=(MOD,KEEP,KEEP),
// DCB=(DSORG=PS,LRECL=80,RECFM=FB,BLKSIZE=800)

When I see this, my gut feeling is to ignore it all, since it looks like a comment. The comments are actually the //* lines, so that is equivalent to:

//LZIOTEST JOB
//A EXEC PGM=IOTEST
//SYSOUT DD SYSOUT=*
//SYSPRINT DD DSN=PJOOT.OUT6,DISP=(MOD,KEEP,KEEP),
// DCB=(DSORG=PS,LRECL=80,RECFM=FB,BLKSIZE=800)

If I understand things properly, the commands in this particular JCL are:

  • JOB (with parameter value LZIOTEST, that identifies the job output in the spool reader)
  • EXEC (with parameter A, which I believe is a step name), and a specification of what to execute for that step (my IOTEST code in this case).
  • DD (define an alias for a file (called a DATASET in mainframe-ese)

The SYSPRINT line associates a DDNAME “SYSPRINT” with a file (i.e. a DATASET) named PJOOT.OUT6. Creating a file seems very painful to do, requiring specification of blocksize, filetype (DSORG), record length, record format (Fixed Blocked in this case), and a DISPosition (whether to create/modify/access-shared/…, and what action to take if the JCL script succeeds or fails). Once that file is created it can then accessed by DDNAME in fopen (i.e. fopen( “DD:SYSPRINT”, …).  I have the feeling that REXX, COBOL, AND PL/1 operate on the DDNAME exclusively, and don’t require it to be prefixed with DD: as the Z/OS C/C++ runtime docs for fopen suggest.

Another oddity with JCL is that it appears to have an 80 character line limitation.  For example, the following produces JCL syntax errors.

//SYSPRINT DD DSN=PJOOT.OUT6,DISP=(MOD,KEEP,KEEP),DCB=(DSORG=PS,LRECL=80,RECFM=FB,BLKSIZE=800)

A trailing comma appears to be the required continuation character.  I don’t know if the indenting I used for DCB= matters, but suspect not.