LzLabs

COBOL code! Where’s the eyewash station?

March 20, 2017 Mainframe , , , , , , ,

In code that I am writing for work, I’m calling into COBOL code from C, and in order to setup the parameters and interpret the results, I have to know a little bit about how variables are declared in COBOL. I got an explanation of a little bit of COBOL syntax today that takes some of the mystery away.

Here’s the equivalent of something like a declaration of compile time constant variables in COBOL, a hierarchical beast something akin to a structure:

004500 01  CONSTANT-VALUES.                                             ORIG_SRC
004600     02  AN-CONSTANT PIC X(5) VALUE "IC104".                      ORIG_SRC
004700     02  NUM-CONSTANT PIC 99V9999 VALUE 0.7654.                   ORIG_SRC

This is roughly the equivalent of the following pseudo-c++11:

struct
{
   char AN_CONSTANT[5]{'I','C','1','0','4'};
   struct {
      char digits1[2]{'0', '0'};
      char decimalpoint{ '.' };
      char digits2[4]{'7', '6', '5', '4'};
   } NUM_CONSTANT;
} ;

Some points:

  • The first 6 characters are source sequence numbers.  They aren’t line numbers like in BASIC (ie. you wouldn’t do a ‘goto 004500’), but were related to punch cards to make sure that out of sequence cards weren’t inserted into the card reader, or a card wasn’t fed into the reader by the operator by accident.
  • The ‘ORIG_SRC’ stuff in column 73+ are ignored.  These columns are also related to punch cards, as an additional card sequence number could be encoded in those locations.
  • The 01 indicates the first level of the ‘structure’.
  • The 02 means a second level.  I don’t know if the indenting of the 01, 02 is significant, but I suspect not.
  • PIC or PICTURE basically means the structure line is a variable and not the name of a new level.
  • A sequence of 9’s means that the variable takes numeric digits in those locations, whereas the V means that location is a period.
  • A sequence of X’s (or the X(5) here that means XXXXX), means that those characters can be alphanumeric.
  • There is no reference to ‘CONSTANT-VALUES’ when the variables are referenced.  That is like a namespace of some sort.
  • The level indicators 01, 02 are arbitrary, but have to be less than 77 (why that magic number? … who knows).  For example 05, 10 could have been used, so that another level could have been inserted in between without renumbering things.

The 01, 02 level indicators are also used for global variable declarations, also somewhat struct like:

004900 01  GRP-01.                                                      ORIG_SRC
005000     02  AN-FIELD PICTURE X(5).                                   ORIG_SRC
005100     02  NUM-DISPLAY PIC 99.                                      ORIG_SRC
005200     02  GRP-LEVEL.                                               ORIG_SRC
005300         03  A-FIELD PICTURE A(3).                                ORIG_SRC

This might be considered equivalent to:

struct
{
   char AN_FIELD[5];
   char NUM_DISPLAY[2];
   struct {
      char A_FIELD[3];
   } GRP_LEVEL;
} GRP_01;

Here:

  • A(3), equivalent to AAA, means the field can have ASCII values.
  • The name ‘GRP-LEVEL’ header for the 03 structure level is not referenced in the code.

It is also possible to declare a variable as binary, like so:

005400 77  ELEM-01 PIC  V9(4) COMPUTATIONAL.                            ORIG_SRC
  • Here 77 is a special magic level number, that really means what follows is a variable and not a “structure”.
  • The V here means an implied decimal place in the interpretation of the value.
  • The 9(4), equivalent to 9999, means the variable must be able to hold 4 numeric digits.
  • The COMPUTATIONAL means the underlying variable must be able to hold a value as big as 9999.  i.e. a short or unsigned short must be used, and not a char or unsigned char.

The final variable group in the code I was looking at was:

005500 01  GRP-02.                                                      ORIG_SRC
005600     02  GRP-03.                                                  ORIG_SRC
005700         03  NUM-ITEM PICTURE S99.                                ORIG_SRC
005800         03  EDITED-FIELD  PIC XXBX0X.                            ORIG_SRC

which is roughly equivalent to:

struct
{  
   struct {
      char NUM_ITEM[2];
      struct
      {
         char digits1[2];
         char blank1[1]{' '};
         char digits2[1];
         char zero1[1]{'0'};
         char digits3[1];
      } EDITED_FIELD;
   } GRP_03;
} GRP_02;

Here

  • EDITED-FIELD includes fixed blank and zero markers (B, 0 respectively).  When a four character variable is copied into this field, only the characters in the non-blank and non-zero values are touched.
  • NUM-ITEM is a signed numeric value.  It’s representation is strange:

The signed representation is also char based, and uses what is referred to as an “over-punch” to encode the sign.  The normal (EBCDIC) encoding of a two digit variable 42 without a sign, would be:

‘4’, ‘2’ == 0xF4, 0xF2

when the S modifier is used in the PICTURE declaration, the F in the EDCDIC encoding range is changed to either C or D for unsigned and signed respectively.  That means the ‘4’, ‘2’ is encoded as:

0xF4, 0xC2

whereas the signed value “-42” is encoded as:

0xF4, 0xD2

The procedure prototype, specifically, what the parameters to the function are, are given in a ‘PROCEDURE DIVISION’ block, like so:

005900 PROCEDURE DIVISION USING GRP-01 ELEM-01 GRP-02.

Here

  • The first 6 characters are still just punch card junk.
  • Three variables are passed to and from the function: GRP-01, ELEM-01, GRP-02.  These are, respectively, 10, 4, and 8 bytes respectively.
  • On the mainframe the COBOL function could be called with R1 something like:

struct parms {
    void * pointers[3];
    char ten[10];
    uint16_t h;
    char eight[8];
};

//...
struct parms p;

p.pointers[0] = &p.ten[0];
p.pointers[1] = &p.h;
p.pointers[2] = &p.eight | 0x80000000;

strncpy( p.ten, "XXXXX00ZZZ", 10 );
p.h = 0;
strncpy( p.eight, "99XXBX0X" );

setregister( R1, &p );

The 0x80000000 is the mainframe “31-bit” way of indicating the end of list. It relies on the fact that virtual memory addresses in 32-bit z/OS processes have only 31-bits of addressable space available, so you can hack one extra bit into a pointer to indicate end of list of pointers.

Suppose the program has statements like the following to populate its output fields

006400 MOVE AN-CONSTANT TO AN-FIELD. 
006500 ADD 25 TO NUM-DISPLAY. 
006600 MOVE "YES" TO A-FIELD. 
006700 MOVE NUM-CONSTANT TO ELEM-01. 
006800 MOVE NUM-DISPLAY TO NUM-ITEM. 
006900 MOVE "ABCD" TO EDITED-FIELD.

The results of this are roughly:

strncpy( p.ten, "IC104", 5 ); // MOVE AN-CONSTANT TO AN-FIELD (GRP-01)
strcpy( p.ten + 5, "25", 2 ); // ADD 25 TO NUM-DISPLAY (GRP-01): since the initial value was "00"
strncpy( p.ten + 7, "YES", 3 ); // MOVE "YES" TO A-FIELD. 
p.h = 7654 // MOVE NUM-CONSTANT TO ELEM-01. 
strcpy( p.eight, "25", 2 ); // MOVE NUM-DISPLAY TO NUM-ITEM. 
strncpy( p.eight + 2, "AB C0D", 6 ); // MOVE "ABCD" TO EDITED-FIELD.

It appears that the the assignment of NUM-CONSTANT, a number of the form 99.9999 to the numeric value ELEM-01 which is of the form .9999, just truncates any whole portion of the number.

A JCL sample, writing IO to a DATASET (mainframe filename)

November 19, 2016 Mainframe , , , , ,

The mainframe and it’s scripting language is a weird beast. Here’s a sample of the scripting language

//LZIOTEST JOB
//A EXEC PGM=IOTEST
//SYSOUT DD SYSOUT=*
//*SYSPRINT DD SYSOUT=*
//SYSPRINT DD DSN=PJOOT.OUT6,DISP=(MOD,KEEP,KEEP),
// DCB=(DSORG=PS,LRECL=80,RECFM=FB,BLKSIZE=800)

When I see this, my gut feeling is to ignore it all, since it looks like a comment. The comments are actually the //* lines, so that is equivalent to:

//LZIOTEST JOB
//A EXEC PGM=IOTEST
//SYSOUT DD SYSOUT=*
//SYSPRINT DD DSN=PJOOT.OUT6,DISP=(MOD,KEEP,KEEP),
// DCB=(DSORG=PS,LRECL=80,RECFM=FB,BLKSIZE=800)

If I understand things properly, the commands in this particular JCL are:

  • JOB (with parameter value LZIOTEST, that identifies the job output in the spool reader)
  • EXEC (with parameter A, which I believe is a step name), and a specification of what to execute for that step (my IOTEST code in this case).
  • DD (define an alias for a file (called a DATASET in mainframe-ese)

The SYSPRINT line associates a DDNAME “SYSPRINT” with a file (i.e. a DATASET) named PJOOT.OUT6. Creating a file seems very painful to do, requiring specification of blocksize, filetype (DSORG), record length, record format (Fixed Blocked in this case), and a DISPosition (whether to create/modify/access-shared/…, and what action to take if the JCL script succeeds or fails). Once that file is created it can then accessed by DDNAME in fopen (i.e. fopen( “DD:SYSPRINT”, …).  I have the feeling that REXX, COBOL, AND PL/1 operate on the DDNAME exclusively, and don’t require it to be prefixed with DD: as the Z/OS C/C++ runtime docs for fopen suggest.

Another oddity with JCL is that it appears to have an 80 character line limitation.  For example, the following produces JCL syntax errors.

//SYSPRINT DD DSN=PJOOT.OUT6,DISP=(MOD,KEEP,KEEP),DCB=(DSORG=PS,LRECL=80,RECFM=FB,BLKSIZE=800)

A trailing comma appears to be the required continuation character.  I don’t know if the indenting I used for DCB= matters, but suspect not.

Note to IBMers re: LzLabs employment.

August 22, 2016 Incoherent ramblings , ,

When contemplating the decision to leave IBM for LzLabs, I found it helpful to enumerate the pros and cons of that decision, which I shared in a Leaving IBM: A causal analysis blog post after I’d formally left IBM.

Who could have predicted that a blog post that wasn’t about programming arcana, physics or mathematics would have been my most popular ever.  There’s no accounting for the taste of the reader;)

In response to that blog post, I’ve been contacted by a few IBMers who were interested in potential LzLabs employment.

Please note that I left IBM with band 9 status, which means that there is a one year restriction (expiring ~May 2017) against me having any involvement with hiring, or recruitment of IBM employees.  An IBM lawyer was very careful to point that the band 9 contract I signed in 2006 has such a non-solicitation agreement.  I don’t think that anybody told the IBM lawyer that IBM appears to be trying really hard to throw away technical staff, but that does not change the fact that I am bound by such an agreement.

If you contacted me, and I was to respond, it could probably be argued that this would not count as solicitation.  However, I don’t feel inclined to pick a fight with IBM lawyers, who I imagine to have very sharp teeth and unlimited legal budgets.  So, if you are working for IBM, and asking about LzLabs employment, please don’t be offended that I did not reply.  I will try to remember to respond to you next spring, when the sharks are swimming elsewhere.

Leaving IBM: A causal analysis.

May 3, 2016 Incoherent ramblings , , , , , , , ,

EDIT: fixed the title; casual -> causal. I am clearly not ever going to be headhunted by any editing or proofreading companies.

I was not looking for a job change, but one found me. There has been persistent headhunter (Google, Microsoft, Facebook, Amazon, …) interest over the years, but any interesting job available also required relocation. I also wanted something compatible with my part time studies, as I have been working 80% part time at IBM to accommodate those studies.

I was contacted by the founder of a new startup. As a long time salaried IBM employee, I am surprised that the possible instability of a startup operation was of interest, but they made a convincing case for the future success of their product ideas and company. Their salary offer was also significantly higher than what I make at IBM, which sure didn’t hurt.

With interesting work as a prospect at this company, an attractive salary, no relocation required, and none of the negatives of current IBM HR practices, I took this offer after a week of deliberation.

Pros of working in DB2

One week of deliberation, especially when I wasn’t looking for a change, is a short amount of time to decide to say goodbye to a company that I’ve worked for about 20 years, which is almost half my life! Both my kids were born while working for IBM, and are now almost grown. I have a roots with the people and work that I am leaving behind.

I did a lot of really fun work in my years with DB2:

– Implemented multithread support and associated reentrancy retrofit of DB2’s Unix client code.
– DB2 Linux porting work, including portability reengineering.
– DB2 64-bit port. This was a project of massive scope. We can now eat terrabytes of RAM for breakfast.
– Development and maintenance of DB2’s internal mutex and atomic implementations, and associated memory ordering mechanisms.
– Lots of fun parts of our platform stackwalk and related post mortem factilities.
– AIX/TPCC performance and exploitation liason.
– DB2 contact for xlC and other compiler coordination.
– Lock-free reimplementation of DB2’s reader-writer mutex. The performance of our original reader-writer mutex code sucked for a number of reasons (including use of an mutex within the mutex.) This was a from scratch implementation where we used a single atomic to track the reader and writer linked lists (indirectly), the reader count, the writer held bit, and a writer reserved bit. This code has stood the test of time and remains one of my proudest creations.
– Implementation of DB2’s asynchronous IO abstraction layer. It’s hard to believe that it wasn’t that long ago that we did all our IO synchronously. This bit of code hides an impressive amount of operating system centric code from our high level development consumers, while squeezing maximum performance from each system.
– Development and maintenance of many other aspects of DB2’s operating system abstraction layer.
– Lead of project branch integration team during internal transition from the CMVC version control system to clearcase.
– Ad-hoc build tooling and makefile maintenance as required.
– Technical owner of DB2’s coding standards
– DB2 pureScale project (distributed shared disk database): Implemented various duplexing, failover and reconstruct aspects of the communications between the DB2 engine and the shared buffer pool and lock manager component.
– Lots of other stuff along the way that took me into various components of DB2.

I started straight from school in ’97 with a low salary (somewhere around $50K CAD). My salary and band rating both progressed very rapidly from there. There were also frequent bonuses in those early days. I was clearly perceived as being of value.

I also really enjoyed the people that I worked with in IBM. I’ve worked with so many very competent and skilled software developers over the years.

So, the pros include:
– Lots of fun and challenging work.
– What seemed like a decent salary.
– Almost nobody else to work for in the Toronto region for whom I could do systems and low level programming.
– Part time options for my educational project and extra time off with the kids.
– Awesome fellow developers. It is humbling and rewarding to work with so many really smart people.
– Working for a startup carries the risk of complete failure.

So why leave?

Many of the reasons to leave DB2 mirror the reasons to stay:

– Fun and challenging work.
– Better salary.
– Flexibility offered for part time work if desired.
– Opportunity to continue to work without relocation, and without having to do something boring like web programming, phone app development, java coding, …
– A collection of really smart developers.
– Staying at IBM can pigeonhole me and leave me viewed as a one trick pony.
– Expectation of a stock distribution and/or bonuses with success.
– The chance to work for a small company, shaping things instead of being a nameless drone.
– No inhuman IBM corporate HR policies to have to observe or be impacted by.

Fun and challenging work.

Now that I have started with LzLabs, I am starting to get a glimpse at just how aggressive and visionary this project is. The scope of it is very impressive, and I’m going to have a lot of fun working on it. My work is likely going to be some combination of concurrency, porting, and build & test infrastructure. These are all things that I am comfortable with and enjoy working on.

Better salary

I was very satisfied with the rate that I achieved my maximum full time equivalent salary at IBM.

I was not satisfied with the way that it stagnated after that. I had not had a raise in a long time. Living in Canada it seems that food costs have 2x’ed in recent years, as have gas prices, hydro bills, and many others. My IBM salary was clearly not even tracking inflation.

The offer I got from LZ certainly made up for that stagnation.

That said, I probably could have asked for a lot more. I got heavily berated by an ex-IBM buddy for accepting the offer I got from LZ, which he said was way too low. He had the good fortune to have been canned by IBM. He’s since moved around and found just how low IBM Canada pays in comparison to others, and is now making $200K USD on the west coast. While my new LZ salary is significantly better than my IBM salary, he said that I shouldn’t have accepted anything less than $240K USD given my skills, and would have easily gotten that, even without requiring relocation. This was surprising to hear. I was clearly pretty clueless about the going rates for the sort of work that I’ve been doing.

My buddy recommends that I get away from LZ in short order (i.e. 6 months) if it shows any signs of not becoming a superstar player. It sounds like I could definitely profit by such a move, but I’ve never been primarily motivated by money. I’d rather play this new game long term and see where it goes.

IBM Canada capitalizes on the sort of salary comparison cluelessness that I had. It also capitalizes on what used to be a monopoly on systems and low level programming work. Without anybody else local to work for they have been able to keep salaries low.

With the ease of working remotely becoming so pervasive, IBM won’t be able to play this game as effectively anymore. This is clearly evident by the mass migrations that seem to be occurring (not even counting the frequent IBM purges).

One trick pony.

Loyalty to an employer used to be considered a virtue. I am not sure that is the case anymore. I left on my own terms, but should I have been fired from IBM, I think that the fact that I worked in only one place for 20 years would have hurt, not helped. Leaving IBM has a positive optics impact on my resume for future work.

Part time options.

I’ve started full time with LZ. I’ve been offered the chance to go back to part time in the fall when school starts again. The LZ founder has said of this

“You are going to have so much fun that you won’t have any desire to continue with an attempt to become a failed physicist. We have lots of those working for us already.”

Although this sounds like it is condescending, he is proud of his collection of very smart people, including all his “failed physicists”, which he said in a way that it sounded like a complement to their intelligence.

One of the LZ hiring strategies is to ask new hires “who is the best/smartest developer that you have ever worked with” (I can’t play my role in that game for a year since I was an IBM band 9). Judging by the interactions that I have had so far at the company, I can confirm that I am in with a really good group.

Inhuman IBM HR

I’ve mentioned that I really enjoyed the people that I worked with in DB2 and IBM. There are so many truly on the ball experienced developers in the pool. It was a real loss to leave IBM and not be able to continue working with these people anymore.

However, in recent years, I have seen IBM treat many of those skilled and experienced developers as disposable. Again and again and again, the most experienced developers are tossed like trash when the firing purges occur. These are in many cases guru level developers with irreplaceable knowledge. Some call this agism, but it may be more accurate to call it salarism, because I think their true crime was getting paid too much in the view of HR that is eying three new hire recent grads for the same total salary (or perhaps three employees in an offshore IBM lab where salaries are uniformly less).

As a 20 year employee, I had only 5 years before I also hit the apparent disposability threshold. Given that, the risks of a joining a startup that might fail are severely minimized. Worst case, if the start up fails, then I am left looking for a job, which is probably also an inevitability if staying on at IBM. The only real loss in that case is the IBM severance that I walk away from by leaving voluntarily, and the new larger salary should compensate for that, provided I sack some of it away for a rainy day.

When I did a final walk around of the lab, saying bye to so many people I had worked with for so long (some that I had never even met in person!), I observed many that had reached or surpassed what I thought of as the 25 year purge threshold criteria. There is no apparent sane rationale for who gets to stay and who gets the knife. That uncertainty for those who remain must be very hard to deal with. Some who I talked to did not even have this perception of disposability, which was interesting to observe.

Tossing employees who have expired is not the only example of cutthroat IBM HR I have seen. In one case, a co-worker who had become the goto guy for a complex component of our code had the bad fortune of getting a promotion too close to one of the firing purges. There is an internal ranking scheme within IBM, and in recent years, anybody with a rank three was toast when the purges ran. Unfortunately, a promotion usually means you are dropped temporarily in rank, since you have to compete with more experienced developers once you have your new band rating. In this case, the promotion in question, reducing the employee rank from 2 to 3, which put him into the automatic firing bucket. It seems to me completely insane to spend five years training somebody, and then essentially fire him for a promotion.

I have also observed two cases where IBMers were fired within what seemed like months of returning to work after having recovered from cancer. That’s a callous action that demonstrates the people with their fingers on the triggers truly don’t think of employees as people. They are resources. Just are just numbers to be shuffled in spreadsheets. Observe that I refuse to use the cowardly term “Resource Action” (or worse RA, or RAed) for firing that IBM HR employs. I am guessing that this a term invented so that HR employees feel less sadistic when they have to run a spreadsheet computation to see who gets the knife. I won’t miss that sort of corporate sadism, and it was a strong factor associated with my choice to leave IBM.

Weighing all the factors.

Looking at all the pros and cons, it seemed clear to me that joining LZ and leaving IBM was in my best interest.

It was certainly a scary move. I don’t even have a salary or real job offer since LZ is not yet incorporated in Canada, and I am currently working as a contractor. Now that I have started, things are no longer quite so scary.

This was a huge change, and has happened really fast. All said, I am happy to have been able to leave on my own terms, and am going to have fun playing a new game at LZ.

Would I work for IBM again in the future?

I had trouble with the binary “Would you work for IBM again” question that was part of leaving IBM feedback form. Having enjoyed my work and the people I worked with over these years, my first instinct was to answer yes, and I did so. However, corporate IBM would have to reform the way it treats its people significantly to be attractive again. I suspect IBM is too large for that to ever occur.