eye wash station

Unexpected COBOL implicit operator distribution!

May 27, 2020 COBOL , , , , , , , , , , ,

Another day, another surprise from COBOL.  I was debugging a failure in a set of COBOL programs, and it seemed that the place things started going wrong was a specific IF check, which basically looked like:

The original code was triple incomprehensible, as it:

  • Was in German.
  • Was in COBOL.
  • Was generated by DELTA and was completely disgusting spaghetti code.  A map of the basic blocks would have looked like it was colored by a three year old vigorously scribbling with a crayon.

It turns out that there was a whole pile of error handling code that happened after the IF check, and I correctly guessed that there was something wrong with how our compiler handled the IF statement.

What I didn’t guess was what the actual operator precedence in this IF check was.  Initially, my C programmer trained mind looked at that IF condition, and said “what the hell is that!?”  I then guessed, incorrectly, that it meant:

if ( X != SPACES and X = ZERO)

where X is the array slice expression.  That interpretation did not explain the runtime failure, so I was hoping that I was both wrong about what it meant, but right that there was a compiler bug.

It turns out that in COBOL the implicit operator for the second part of the IF statement is  ‘NOT =’.  i.e. the NOT= distributes over the AND, so this IF actually meant:

if ( X != SPACES and X != ZERO)

In the original program, that IF condition actually makes sense.  After some reflection, I see there is some sense to this distribution, but it certainly wasn’t intuitive after programming C and C++ for 27 years. I’d argue that the root cause of the confusion is COBOL itself. A real programming language would use a temporary variable for the really long array slice expression, which would obliterate the need for counter-intuitive operator distribution requirements. Imagine something like:

  VAR X = PAYLOAD-DATA(PAYLOAD-START(TALLY): PAYLOAD-END(TALLY))

  IF (X NOT = SPACES) AND (X NOT = LOW-VALUE)
     NEXT SENTENCE ELSE GO TO CHECK-IT-DONE.

(Incidentally LOW-VALUE means binary-zero, not a ‘0’ character that has a 0xF0 value in EBCDIC).

COBOL is made especially incomprehensible since you can’t declare an in-line temporary in COBOL.  If you want one, you have to go thousands of lines up in the code to the WORKING-STORAGE section, and put a variable in there somewhere.  Such a variable is global to the whole program, and you have to search to determine it’s usage scope.  You probably also need a really long name for it because of namespace collision with all the other global variables.  Basically, you are better off not using any helper variables, unless you want to pay an explicit cost in overall code complexity.

In my test program that illustrated the compiler bug, I made other COBOL errors. I blame the fact that I was using gross GOTO ridden code like the original. Here was my program:

Because I misinterpreted the NOT= distribution, I thought this should produce:

000000001: !(not space and low-value.)
000000002: !(not space and low-value.)
000000003: !(not space and low-value.)
000000003: not space and low-value.

Once the subtle compiler bug was fixed, the actual SYSOUT from the program was:

000000001: not space and low-value.
000000001: !(not space and low-value.)
000000002: !(not space and low-value.)
000000003: !(not space and low-value.)

See how both the TRUE and FALSE basic blocks executed in my code. That didn’t occur in the original code, because it used an additional dummy EXIT paragraph to end the PERFORM range, and had a GOTO out of the first paragraph.

There is more modern COBOL syntax that can avoid this GOTO hell, but I hadn’t used it, as I kept the reproducer somewhat like the original code.

COBOL spaghetti code: EXIT does nothing!

May 20, 2020 COBOL , , , , , , , , , ,

I was staring down COBOL code of the following form:

       LOOP-COUNTER-INCREMENT.
           ADD 1 TO J.
       LOOP-PREDICATE-CHECK.   
           IF J GREATER 10 GO TO MYSTERIOUS-LABEL-1.
           
           IF ARRAY-1 (J)      NOT = ZERO
           NEXT SENTENCE ELSE GO TO MYSTERIOUS-LABEL-1.
           
           IF ARRAY-2 (J) = MYSTERIOUS-MAGIC-NUMBER-CONSTANT
           NEXT SENTENCE ELSE GO TO COUNTER-INCREMENT-SPAGGETTIFI.
           
     *     ...MORE STUFF...                                        
     
           GO TO MYSTERIOUS-LABEL-3.
           
       COUNTER-INCREMENT-SPAGGETTIFI.
           GO TO LOOP-COUNTER-INCREMENT.
           
       MYSTERIOUS-LABEL-1.
                       EXIT.
       MYSTERIOUS-LABEL-2.
                       EXIT.
       MYSTERIOUS-LABEL-3.
                       EXIT.

I had to get some guru help understanding what this was about (thanks Roger!). I didn’t understand why somebody would code a GOTO LABEL, when the the code at that LABEL just did an EXIT. If my intuition could be trusted, I would have assumed that this code was equivalent to the much simpler:

       LOOP-COUNTER-INCREMENT.
           ADD 1 TO J.
       LOOP-PREDICATE-CHECK.   
           IF J GREATER 10 EXIT.
           
           IF ARRAY-1 (J)      NOT = ZERO
           NEXT SENTENCE ELSE EXIT.
           
           IF ARRAY-2 (J) = MYSTERIOUS-MAGIC-NUMBER-CONSTANT
           NEXT SENTENCE ELSE GO TO LOOP-COUNTER-INCREMENT.
           
     *     ...MORE STUFF...                                        
     
           EXIT.

It turns out that intuition is not much use when looking at COBOL code. In this case, that intuition failure is because EXIT doesn’t actually do anything. It is not like a return, which is what I assumed, but is just something that you can put in a paragraph at the end of the section so that the code can exit the section (or at the end of a sequence of paragraphs invoked by PERFORM THRU, so that the code can return to the caller.)  The EXIT in such a paragraph is just a comment, and you could use an empty paragraph to do the same thing.

In my transformation of the code the EXIT would do nothing, and execution would just fall through to the next sentence!

Some of the transformations I made are valid. In particular, the spaghettification-indirection used to increment the loop counter, by using a goto to goto the target location instead of straight there, has no reason to exist.

The code in question was an edited version of a program that was generated by a 4GL language (DELTA), so some of the apparent stupidity can be blamed on the code generator. I also assume DELTA can also be blamed for the multiple EXIT paragraphs, when it would seem more natural to just have one per section.

This code also uses EXIT after other paragraph labels too. The first paragraph in the following serving of horror has such an example:

            PERFORM TRANSFER-CHECK THRU TRANSFER-CHECK-EXIT.

            [snip]

       TRANSFER-CHECK.
                       EXIT.
       MEANINGLESS-LABEL-1.
           IF [A COMPOUND PREDICATE CHECK]
           NEXT SENTENCE ELSE GO TO MEANINGLESS-LABEL-2.
                 [SNIP]
           PERFORM [MORE STUFF]
           GO TO MEANINGLESS-LABEL-100.
       MEANINGLESS-LABEL-2.
           [STUFF]
           GO TO MEANINGLESS-LABEL-4.
       MEANINGLESS-LABEL-3.
           [increment loop counter, and fall through]
       MEANINGLESS-LABEL-4.
           [loop body]
...
       MEANINGLESS-LABEL-50.
           GO TO MEANINGLESS-LABEL-3.
           [SNIP]
...
       MEANINGLESS-LABEL-99.                            
                       EXIT.                               
       MEANINGLESS-LABEL-100.                                       
                       EXIT. 
       TRANSFER-CHECK-EXIT.
                       EXIT.

Nothing ever branches to MEANINGLESS-LABEL-1 directly, so why even have that there? Using my new found knowledge that EXIT doesn’t do anything, I’m pretty sure that you could just write:

            PERFORM TRANSFER-CHECK THRU TRANSFER-CHECK-EXIT.

            [snip]

       TRANSFER-CHECK.
       
           IF [A COMPOUND PREDICATE CHECK]

Is there some subtle reason that this first no-op paragraph was added? My guess is that the programmer was either being paid per line of code, or the code generator is to blame.

I’m not certain about the flow-control in the TRUE evaluation above. My intuition about the THRU use above is that if we have a GOTO that bypasses one of the paragraphs, then all the preceding paragraphs are counted as taken (i.e. if you get to the final paragraph in the THRU evaluation, no matter how you get there, then you are done.) I’ll have to do an experiment to determine if that’s actually the case.

Reverse engineering a horrible COBOL structure initialization

May 16, 2020 Mainframe , , , ,

The COBOL code that I was looking at used a magic value 999, and I couldn’t see where it could be coming from.  After considerable head scratching, I managed to figure out that all the array structure instantiations in the code are initialized using strings.  That seems to be the origin of the magic (standalone) 999’s scattered through the code.

To share the horror, here is an (anonymized) example of the offending array structure initialization

where I added in the block comment that points out each of the interesting regions of the initialization strings.

Here’s what’s going on.  We have a global variable array (effectively unnamed) that has three fields:

  • two-characters (numeric only)
  • dummy-structure-name, containing a 3 character field and a pad.
  • nine-more-characters

If you add up all the characters in this data structure we have: 2 + 1 + 4 * (3 + 1) + 9 = 28, so this array initialization is effectively done by aliasing the array elements with the memory containing a char[7][28].

My eyes are burning!

As far as I can tell, COBOL has no notion of a structure type, you just have instances of structures everywhere (they are probably called something different — a level 01 declaration, or something like that).  A lot of the PL/I code I’ve seen is also like that, although in PL/I you can declare your structure types if you want to.

The display’s above make use of the fact that COBOL variables don’t have to use all the high level qualifiers (unless there is ambiguity).  My SYSOUT shows that, sure enough, the (5) element of the array (COBOL arrays are one’s counted) has the values I expected:

1 22
2 999
3 1/2
4
5
6 SF

Basically, the horrendous initialization above, is as if you if declared your structure as:

struct arrayname
{                   
   char numeric2[2];
   char filler1[1];
   struct               
   {                 
      char threemore[3];
      char filler2[1];
   } threepluspad[4];

   char ninemore[9];     
};

and then initialized it with:

char globalmemory[7][28] = {
   // n2       f    x    x    x    y    x    x    x    y    x    x    x    y    x    x    x    y    'K', 'l', 'a', 's', 's', 'e', ' ', ' ', ' '},
   { '0', '1', ' ', ' ', ' ', '0', ' ', ' ', ' ', '0', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'K', 'l', 'a', 's', 's', 'e', ' ', ' ', ' '},
   { '0', '2', ' ', ' ', ' ', '0', ' ', ' ', ' ', '0', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'K', 'l', 'a', 's', 's', 'e', ' ', ' ', ' '},
   { '1', '3', ' ', '9', '9', '9', ' ', '9', '9', '9', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'K', 'l', 'a', 's', 's', 'e', ' ', ' ', ' '},
   { '2', '1', ' ', '9', '9', '9', ' ', '1', '/', '2', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'S', 'F', ' ', ' ', ' ', ' ', ' ', ' ', ' '},
   { '2', '2', ' ', '9', '9', '9', ' ', '1', '/', '2', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'S', 'F', ' ', ' ', ' ', ' ', ' ', ' ', ' '},
   { '2', '3', ' ', '1', '/', '2', ' ', '1', '/', '2', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'S', 'F', ' ', ' ', ' ', ' ', ' ', ' ', ' '},
   { '3', '1', ' ', ' ', ' ', '1', ' ', ' ', ' ', '1', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'S', 'F', ' ', ' ', ' ', ' ', ' ', ' ', ' '},
};

struct arrayname * p = (struct arrayname*)globalmemory;

and then and then printed:

   printf( "1 %.2s\n", p[4].numeric2 );
   printf( "2 %.3s\n", p[4].threepluspad[0].threemore );
   printf( "3 %.3s\n", p[4].threepluspad[1].threemore );
   printf( "4 %.3s\n", p[4].threepluspad[2].threemore );
   printf( "5 %.3s\n", p[4].threepluspad[3].threemore );
   printf( "6 %.9s\n", p[4].ninemore );

Of course, the use of fixed length strings without a null terminator wouldn’t ever be done in C, so a more natural equivalent (assuming one doesn’t care about the specific memory equivalence of the two representations, and can tolerate null terminators instead of spaces) would just be something like:

struct arrayname
{
   char numeric2[3];
   struct 
   {
      char threemore[4];
   } threepluspad[4];

   char ninemore[9];
};

struct arrayname g[7] = {
   { "01", {"  0", "  0", "   ", "   "}, "Klasse  " },
   { "02", {"  0", "  0", "   ", "   "}, "Klasse  " },
   { "13", {"999", "999", "   ", "   "}, "Klasse  " },
   { "21", {"999", "1/2", "   ", "   "}, "SF      " },
   { "22", {"999", "1/2", "   ", "   "}, "SF      " },
   { "23", {"1/2", "1/2", "   ", "   "}, "SF      " },
   { "31", {"  1", "  1", "   ", "   "}, "SF      " }
};

You could argue that the COBOL way isn’t so bad once you’ve seen the pattern, and is only cosmetically different from the natural C analogue. That is, if you ignore the fact that there is no separation of fields in the initializer strings, and that you have to name a whole bunch of dummy initializer objects and fill characters, and the fact that any semblance of typing is completely obliterated.

The code in question is also complete spaghetti, with GOTO all over the place.  Perhaps COBOL versions after COBOL77, which is what I assume I’m looking at, added loops and better initialization syntax?

Computing “offsetof” in COBOL

May 15, 2020 Mainframe , , , , ,

I couldn’t find a way to compute something like C offsetof in COBOL code.  What I could manage to figure out how to do is compare addresses of a runtime instantiation of the structure, effectively doing this indirectly.  Here’s the ugly mess that I cooked up:

I couldn’t figure out the right syntax to do a single compute statement that was just the difference of addresses, as I got numeric/pointer compare errors from the compiler, no matter what I tried.  I think that ‘USAGE IS POINTER’ may be required on my variables, but that would still require a temporary.  I’m probably either doing this the hard way, or there is no easy way in COBOL.

This program was run with the following simple JCL

//TESTPROG JOB
//A EXEC PGM=TESTPROG
//SYSOUT DD SYSOUT=*
//STEPLIB DD DSN=COBRC.NATIVE.TESTPROG,
// DISP=SHR

and produced the following SYSOUT

address of TESTPROG-STRUCT = 0016800264
offsetof(ARRAY-NAME,RUECK-BKL) = 0000000002
offsetof(ARRAY-NAME,RUECK-BS) = 0000000004
offsetof(ARRAY-NAME,RUECK-SF) = 0000000007
sizeof(ARRAY-NAME(1)) = 0000000019

Looking at that output, we can conclude the following:

  • PIC S9(3) COMP-3 is effectively horrible eye-burning syntax for a “short”
  • There is no alignment padding between fields, nor end of array-member padding to force natural alignment of the next array element, should the structure start have been aligned.

I knew the latter, but wasn’t sure what size the first field was, and thought that trying to figure it out with COBOL code would be a good learning exercise.

File organization in really old COBOL code.

May 7, 2020 Mainframe , , , , , , , , , , , , ,

I encountered customer COBOL code today with a file declaration of the following form:

000038   SELECT AUSGABE ASSIGN TO UR-S-AUSGABE            
000039    ACCESS IS SEQUENTIAL.                   
...
000056 FD  AUSGABE                                                     
000057     RECORDING F                                                  
000058     BLOCK 0 RECORDS                                              
000059     LABEL RECORDS OMITTED.

where the program’s JCL used an AUSGABE (German “output”) DDNAME of the following form:

//AUSGABE   DD    DUMMY

The SELECT looked completely wrong to me, as I thought that SELECT is supposed to have the form:

SELECT cobol-file-variable-name ASSIGN TO ddname

That’s the syntax that my Murach’s Mainframe COBOL uses, and also what I’d seen in big-blue’s documentation.

However, in this customer’s code, the identifier UR-S-AUSGABE is longer than 8 characters, so it sure didn’t look like a DDNAME. I preprocessed the code looking to see if UR-S-AUSGABE was hiding in a copybook (mainframe lingo for an include file), but it wasn’t. How on Earth did this work when it was compiled and run on the original mainframe?

It turns out that [LABEL-]S- or [LABEL]-AS- are ways that really old COBOL code used to specify file organization (something like PL/I’s ENV(ORGANIZATION) clauses for FILEs). This works on the mainframe because a “modern” mainframe COBOL compiler strips off the LABEL- prefix if specified and the organization prefix S- as well, essentially treating those identifier fragments as “comments”.

For anybody reading this who has only programmed in a sane programming language, on sane operating systems, this all probably sounds like verbal diarrhea.  What on earth is a file organization and ddname?  Do I really have to care about those just to access a file?  Well, on the mainframe, yes, you do.

These mysterious dependencies highlight a number of reasons why COBOL code is hard to migrate. It isn’t just a programming language, but it is tied to the mainframe with lots of historic baggage in ways that are very difficult to extricate.  Even just to understand how to open a file in mainframe COBOL you have a whole pile of obstacles along the learning curve:

  • You don’t just run the program in a shell, passing in arguments, but you have to construct a JCL job step to do so.  This specifies parameters, environment variables, file handles, and other junk.
  • You have to know what a DDNAME is.  This is like a HANDLE in the JCL code that refers to a file.  The file has a filename (DSNAME), but you don’t typically use that.  Instead the JCL’s job step declares an arbitrary DDNAME to refer to that handle, and the program that is run in that job step has to always refer to the file using that abstract handle.
  • The file has all sorts of esoteric attributes that you have to know about to access it properly (fixed, variable, blocked, record length, block size, …).  The program that accesses the file typically has to make sure that these attributes are all encoded with the equivalent language specific syntax.
  • Files are not typically just byte streams on the mainframe but can have internal structure that can be as complicated as a simple database (keyed records, with special modes to access them to initialize vs access/modify.)
  • To make life extra “fun”, files are found in a variety of EBCDIC code pages.  In some cases these can’t be converted to single byte iso-8859-X code pages, so you have to use utf-8, and can get into trouble if you want to do round trip conversions.
  • Because of the internal structure of a mainframe file, you may not be able to transfer it to a sane operating system unless special steps are taken.  For example, a variable format file with binary data would typically have to be converted to a fixed format representation so that it’s possible to seek from record to record.
  • Within the (COBOL) code you have three sets of attributes that you have to specify to “declare” a file, before you can even attempt to open it: the DDNAME to COBOL-file-name mapping (SELECT), the FD clause (file properties), and finally record declarations (global variables that mirror the file data record structure that you have to use to read and write the file.)

You can’t just learn to program COBOL, like you would any sane programming language, but also have to learn all the mainframe concepts that the COBOL code is dependent on.  Make sure you are close enough to your eyewash station before you start!