Month: March 2017

LLVM IR Null pointer constants and function pointers. A wild goose chase after a bad assumption.

March 30, 2017 clang/llvm No comments , , , , , ,

With ELLCC, you can easily check out the LLVM IR for code like:

typedef void ( *f )( void );
void foo( void );

f bar() {
    return (f)foo;
}

That code is:

define nonnull void ()* @bar() local_unnamed_addr {
  ret void ()* @foo
}

declare void @foo()

I was trying to use @foo in a “struct” object, and was getting an error attempting this:

llvm/lib/IR/Constants.cpp:879:llvm::ConstantAggregate::ConstantAggregate(
 llvm::CompositeType*, llvm::Value::ValueTy, llvm::ArrayRef<llvm::Constant*>):
 Assertion `V[I]->getType() == T->getTypeAtIndex(I) &&
 "Initializer for composite element doesn't match!"' failed.

After adding:

fooFunc->dump();

where it shows the whole function body of foo(), I thought that’s where the error was coming from, and that I needed some other method to obtain just “@foo”, a global variable reference to the function, and not the function body itself.

The actual story is much simpler. Here the LLVM code to generate the IR for a foo() with this interface:

//------------
// void foo(){ }
//
auto vt = m_builder.getVoidTy();
auto voidFuncVoidType = FunctionType::get( vt, false /* varargs */ );

Function *fooFunc = Function::Create(
    voidFuncVoidType, Function::InternalLinkage, "foo",
    m_module );
BasicBlock *fooBB =
    BasicBlock::Create( m_context, "", fooFunc );
m_builder.SetInsertPoint( fooBB );
m_builder.CreateRetVoid();

My clue that the error is something else is that I am able to build a function that returns a foo function pointer:

//------------
// void(*)() bar() { return foo ; }
//
auto fpRetFuncType = FunctionType::get( voidFuncVoidType->getPointerTo(), false /* varargs */ );

Function *barFunc = Function::Create(
    fpRetFuncType, Function::ExternalLinkage, "bar",
    m_module );
BasicBlock *barBB =
    BasicBlock::Create( m_context, "", barFunc );
m_builder.SetInsertPoint( barBB );
m_builder.CreateRet( fooFunc );

The module at this point looks like:

define internal void @foo() {
   ret void
}

define void ()* @bar() {
   ret void ()* @foo
}

So why can I used fooFunc in a return statement, but don’t appear to be able to use it in a structure object? Here’s the code that created that structure type

//------------
//
// struct { int, void (*)(), char * }
auto i8t = m_builder.getInt8Ty();
auto i32t = m_builder.getInt32Ty();
std::vector<Type *> consStructMembers{
    i32t, voidFuncVoidType->getPointerTo(), i8t->getPointerTo()};
auto consStructType =
    StructType::create( m_context, consStructMembers, "" );

and my attempt to populate an object of this type:

//
// %struct { int, void (*)(), char * } = { 65535, foo, null };
//
auto consPriority = ConstantInt::get( i32t, 65535 );
auto consDataZero = ConstantInt::get( i8t->getPointerTo(), 0 );

std::vector<Constant *> v{consPriority, fooFunc, consDataZero};
Constant *g = ConstantStruct::get( consStructType, v );

The actual error was in the third struct member initialization, and had nothing to do with the function pointer value. In retrospect, this makes sense since llvm::Function is derived from llvm::Constant, so there shouldn’t logically be a mismatch there.

What actually fixed the error was simply:

auto consDataZero = ConstantPointerNull::get( i8t->getPointerTo() );

It appears that the numeric zero value isn’t the same thing as an LLVM ‘null’. With that corrected, my variable declaration is:

%"type 0x10ea0c0" { i32 65535, void ()* @foo, i8* null }

… so I should now be able to proceed with the actual task at hand.

COBOL code! Where’s the eyewash station?

March 20, 2017 Mainframe No comments , , , , , , ,

In code that I am writing for work, I’m calling into COBOL code from C, and in order to setup the parameters and interpret the results, I have to know a little bit about how variables are declared in COBOL. I got an explanation of a little bit of COBOL syntax today that takes some of the mystery away.

Here’s the equivalent of something like a declaration of compile time constant variables in COBOL, a hierarchical beast something akin to a structure:

004500 01  CONSTANT-VALUES.                                             ORIG_SRC
004600     02  AN-CONSTANT PIC X(5) VALUE "IC104".                      ORIG_SRC
004700     02  NUM-CONSTANT PIC 99V9999 VALUE 0.7654.                   ORIG_SRC

This is roughly the equivalent of the following pseudo-c++11:

struct
{
   char AN_CONSTANT[5]{'I','C','1','0','4'};
   struct {
      char digits1[2]{'0', '0'};
      char decimalpoint{ '.' };
      char digits2[4]{'7', '6', '5', '4'};
   } NUM_CONSTANT;
} ;

Some points:

  • The first 6 characters are source sequence numbers.  They aren’t line numbers like in BASIC (ie. you wouldn’t do a ‘goto 004500’), but were related to punch cards to make sure that out of sequence cards weren’t inserted into the card reader, or a card wasn’t fed into the reader by the operator by accident.
  • The ‘ORIG_SRC’ stuff in column 73+ are ignored.  These columns are also related to punch cards, as an additional card sequence number could be encoded in those locations.
  • The 01 indicates the first level of the ‘structure’.
  • The 02 means a second level.  I don’t know if the indenting of the 01, 02 is significant, but I suspect not.
  • PIC or PICTURE basically means the structure line is a variable and not the name of a new level.
  • A sequence of 9’s means that the variable takes numeric digits in those locations, whereas the V means that location is a period.
  • A sequence of X’s (or the X(5) here that means XXXXX), means that those characters can be alphanumeric.
  • There is no reference to ‘CONSTANT-VALUES’ when the variables are referenced.  That is like a namespace of some sort.
  • The level indicators 01, 02 are arbitrary, but have to be less than 77 (why that magic number? … who knows).  For example 05, 10 could have been used, so that another level could have been inserted in between without renumbering things.

The 01, 02 level indicators are also used for global variable declarations, also somewhat struct like:

004900 01  GRP-01.                                                      ORIG_SRC
005000     02  AN-FIELD PICTURE X(5).                                   ORIG_SRC
005100     02  NUM-DISPLAY PIC 99.                                      ORIG_SRC
005200     02  GRP-LEVEL.                                               ORIG_SRC
005300         03  A-FIELD PICTURE A(3).                                ORIG_SRC

This might be considered equivalent to:

struct
{
   char AN_FIELD[5];
   char NUM_DISPLAY[2];
   struct {
      char A_FIELD[3];
   } GRP_LEVEL;
} GRP_01;

Here:

  • A(3), equivalent to AAA, means the field can have ASCII values.
  • The name ‘GRP-LEVEL’ header for the 03 structure level is not referenced in the code.

It is also possible to declare a variable as binary, like so:

005400 77  ELEM-01 PIC  V9(4) COMPUTATIONAL.                            ORIG_SRC
  • Here 77 is a special magic level number, that really means what follows is a variable and not a “structure”.
  • The V here means an implied decimal place in the interpretation of the value.
  • The 9(4), equivalent to 9999, means the variable must be able to hold 4 numeric digits.
  • The COMPUTATIONAL means the underlying variable must be able to hold a value as big as 9999.  i.e. a short or unsigned short must be used, and not a char or unsigned char.

The final variable group in the code I was looking at was:

005500 01  GRP-02.                                                      ORIG_SRC
005600     02  GRP-03.                                                  ORIG_SRC
005700         03  NUM-ITEM PICTURE S99.                                ORIG_SRC
005800         03  EDITED-FIELD  PIC XXBX0X.                            ORIG_SRC

which is roughly equivalent to:

struct
{  
   struct {
      char NUM_ITEM[2];
      struct
      {
         char digits1[2];
         char blank1[1]{' '};
         char digits2[1];
         char zero1[1]{'0'};
         char digits3[1];
      } EDITED_FIELD;
   } GRP_03;
} GRP_02;         

Here

  • EDITED-FIELD includes fixed blank and zero markers (B, 0 respectively).  When a four character variable is copied into this field, only the characters in the non-blank and non-zero values are touched.
  • NUM-ITEM is a signed numeric value.  It’s representation is strange:

The signed representation is also char based, and uses what is referred to as an “over-punch” to encode the sign.  The normal (EBCDIC) encoding of a two digit variable 42 without a sign, would be:

‘4’, ‘2’ == 0xF4, 0xF2

when the S modifier is used in the PICTURE declaration, the F in the EDCDIC encoding range is changed to either C or D for unsigned and signed respectively.  That means the ‘4’, ‘2’ is encoded as:

0xF4, 0xC2

whereas the signed value “-42” is encoded as:

0xF4, 0xD2

The procedure prototype, specifically, what the parameters to the function are, are given in a ‘PROCEDURE DIVISION’ block, like so:

005900 PROCEDURE DIVISION USING GRP-01 ELEM-01 GRP-02. 

Here

  • The first 6 characters are still just punch card junk.
  • Three variables are passed to and from the function: GRP-01, ELEM-01, GRP-02.  These are, respectively, 10, 4, and 8 bytes respectively.
  • On the mainframe the COBOL function could be called with R1 something like:
struct parms {
    void * pointers[3];
    char ten[10];
    uint16_t h;
    char eight[8];
};

//...
struct parms p;

p.pointers[0] = &p.ten[0];
p.pointers[1] = &p.h;
p.pointers[2] = &p.eight | 0x80000000;

strncpy( p.ten, "XXXXX00ZZZ", 10 );
p.h = 0;
strncpy( p.eight, "99XXBX0X" );

setregister( R1, &p );

The 0x80000000 is the mainframe “31-bit” way of indicating the end of list. It relies on the fact that virtual memory addresses in 32-bit z/OS processes have only 31-bits of addressable space available, so you can hack one extra bit into a pointer to indicate end of list of pointers.

Suppose the program has statements like the following to populate its output fields

006400 MOVE AN-CONSTANT TO AN-FIELD. 
006500 ADD 25 TO NUM-DISPLAY. 
006600 MOVE "YES" TO A-FIELD. 
006700 MOVE NUM-CONSTANT TO ELEM-01. 
006800 MOVE NUM-DISPLAY TO NUM-ITEM. 
006900 MOVE "ABCD" TO EDITED-FIELD. 

The results of this are roughly:

strncpy( p.ten, "IC104", 5 ); // MOVE AN-CONSTANT TO AN-FIELD (GRP-01)
strcpy( p.ten + 5, "25", 2 ); // ADD 25 TO NUM-DISPLAY (GRP-01): since the initial value was "00"
strncpy( p.ten + 7, "YES", 3 ); // MOVE "YES" TO A-FIELD. 
p.h = 7654 // MOVE NUM-CONSTANT TO ELEM-01. 
strcpy( p.eight, "25", 2 ); // MOVE NUM-DISPLAY TO NUM-ITEM. 
strncpy( p.eight + 2, "AB C0D", 6 ); // MOVE "ABCD" TO EDITED-FIELD. 

It appears that the the assignment of NUM-CONSTANT, a number of the form 99.9999 to the numeric value ELEM-01 which is of the form .9999, just truncates any whole portion of the number.

Notes compilation for ECE1505, Convex Optimization

March 18, 2017 ece1505 No comments

I’ve now posted a notes compilation for the subset of the Convex Optimization (ECE1505H) course I was taking in the winter 2017 session.

This course was taught by Prof. S. Draper.

These convex optimization notes are incomplet, covering only the first 9 lectures. The unredacted notes include my solution to problem set 1 (149 pages, vs. 131 pages).

I initially enrolled on this optimization course because I needed a specific quota of ECE courses to satisfy the M.Eng graduation requirements, and the electromagnetics group wasn’t offering enough courses.  I remembered liking linear programming in high school, and always wanted to understand the rational for some of the assumptions that was based on that were never proven in class.  Specifically, I recall that it was stated, but not proved in that high school class, that the extreme values were always found at the vertices of the optimization region.  So, my thought was, I’ll have fun learning the basis for those assumptions, and also learn about optimization theory in general.

It turns out that optimization theory, at least as presented in this course, is very very dry.  It was an endless seeming sequence of definition and proof, with the end goal so far away that it was very difficult to see the big picture.  I worked through the a number of weeks of this particular course before I had enough and bailed.  Work is too fun right now to torture myself and spend the time on an academic course that I am not enjoying, so I dropped it and am back to full time work at LzLabs (from 80%) until the next session at UofT starts again.

The reason I enrolled on the M.Eng in the first place was to study material that I was interested in.  Ideally I would have done that in a part time physics grad context, but that was not available, so I found that the M.Eng allowed me to take an interesting (but constrained) mix of physics and engineering electromagnetism courses.  However, when I enrolled, the electromagnetism course selection was a lot better, and now unfortunately it is sparse and includes only courses that I’d already taken.  I don’t want the M.Eng degree paper badly enough to torture myself with a course that I’m not actually interested in.

I now actually have a plan to satisfy both the degree requirements and my interests (using a project “course”).  That will involve independent study on Geometric Algebra applications to engineering electromagnetism.  I am irked that I have to pay a part time engineering program fee next year to self study, but it does seem worthwhile to come out of the M.Eng study with an actual degree as a side effect, so I am going to go ahead and do it anyways.

gdb pretty print of structures

March 9, 2017 C/C++ development and debugging. No comments , , , ,

Here’s a nice little gdb trick for displaying structure contents in a less compact format

(gdb) set print pretty on
(gdb) p dd[0]
$4 = {
  jfcb = {
    datasetName = "PJOOT.NVS1", ' ' <repeats 34 times>,
    .
    .
    .
    vols = {"<AAAiW", "\000\000\000\000\000", "\000\000\000\000\000", "\000\000\000\000\000", "\000\000\000\000\000"},
  },
  block_size = 800,
  device_class = 32 '\040',
  device_type = 15 '\017',
  disp_normal = 8 '\010',
  disp_cond = 8 '\010',
  volsers = 0x7fb71801ecd6 "<AAAiW",
} 

compare this to the dense default

(gdb) set print pretty on
(gdb) p dd[0]
$5 = {jfcb = {datasetName = "PJOOT.NVS1", ... vols = {"<AAAiW", "\000\000\000\000\000", "\000\000\000\000\000", "\000\000\000\000\000", "\000\000\000\000\000"}, block_size = 800, block_size_limit = 0, device_class = 32 '\040', device_type = 15 '\017', disp_normal = 8 '\010', disp_cond = 8 '\010', volsers = 0x7fb71801ecd6 "<AAAiW"}

For really big structures (this one actually is, but I’ve pruned a bunch of stuff), this makes the structure print display a whole lot more readable. Additionally, if you combine this with ‘(gdb) set logging on’, then with pretty print enabled you can prune the output by line easily to see just what you want.

A really dumb DNS lookup for my internal network

March 8, 2017 perl and general scripting hackery No comments , , , , ,

The new Hitron cable modem in the house cowardly refuses to let me cache mac and ip address pairs, which is really annoying because my ip addresses now change on me over a couple days. The old router (also a Hitron) allowed that, so putting it on a UPS was generally enough to let me have a static IP table, provided I didn’t have to reboot it.

Here’s a hack using nmap that I just cobbled together to fill in the /etc/hosts entries on the couple machines that I want to talk to each other (mac and Linux machines, so all are unix like).

my %hostnameByMacAddr = (
'B8:4E:3F:C4:04:02' => 'router',
'E4:5C:89:C2:0F:4B' => 'macbookw # wireless',
'10:C2:C6:A0:20:58' => 'nuc2w',
'10:C2:C6:CA:93:6A' => 'nuc1w',
'A8:AE:ED:EB:39:86' => 'nuc1',
'A8:AE:ED:7D:CE:5A' => 'nuc2',
'28:C9:86:46:A8:15' => 'macbookt # thunderbolt monitor connected',
'10:24:2B:A1:7B:F7' => 'brother # printer',
'BC:87:A3:34:1A:FF' => 'macbooke # ethernet cable connected',
);

open my $h, "sudo nmap -n -p 22 192.168.0.1/24 2>&1 | grep -e '192' -e '^MAC' |" or die;

my $ip;
while ( <$h> )
{  
   if ( /scan report for.*(192\.\d+\.\d+\.\d+)/ )
   {  
      $ip = $1 ;
   }

   if ( /MAC Address: (.*) / ) {
      my $mac = $1;

      if ( defined $hostnameByMacAddr{$mac} ) {
         print "$ip $hostnameByMacAddr{$mac} # $mac\n" ;
      }
      else {
         print "# $ip $mac # unknown\n" ;
      }
   }
}

close $h or die;

If anybody knows how to set up an actual DNS server for internal networks, I’d be interested to see what is involved, since it looked very hard when I googled it.

VSAM creation and population with JCL and IDCAMS

March 7, 2017 Mainframe No comments , , , , , , , ,

I learned a few JCL DATASET related things yesterday that seemed notable, at least for a JCL newbie.

Delete a DATASET, and ignore any error.

Each time I’ve wanted a DATASET cleanup step in JCL I’ve been using a separate script, and running that first.  A better way of doing this is to include a IDCAMS job step in the script, and have that do the deletion

//CLEANUP EXEC PGM=IDCAMS
//SYSIN DD *
  DELETE PJOOT.XXXXX005
  SET MAXCC = 0
/*
//SYSPRINT DD SYSOUT=*
//SYSOUT   DD SYSOUT=*
//*SYSTERM  DD SYSOUT=*

This deletes the file PJOOT.XXXXX005, which in this case was a VSAM file. In case that file (a DATASETs in mainframe-eze) did not exist, the error code for that DELETE is ignored by setting MAXCC=0. If you have multiple things that you want to do with IDCAMS, you can do things like DELETE and then ALLOCATE immediately, such as

//REALLOC EXEC PGM=IDCAMS
//SYSIN DD *
  DELETE PJOOT.XXXXX005
  SET MAXCC = 0
  DEFINE CLUSTER (NAME(PJOOT.XXXXX005) -
               CYLINDERS(1) VOLUMES(LZ0000) -
               INDEXED -
               KEYS(4 0) -
               RECORDSIZE(240 240) -
               ) -
         DATA (NAME(PJOOT.XXXXX005.DATA)) -
         INDEX (NAME(PJOOT.XXXXX005.INDEX))
/*
//SYSPRINT DD SYSOUT=*
//SYSOUT   DD SYSOUT=*
//*SYSTERM  DD SYSOUT=*

This does the DELETE, ignores any error, and then proceeds to do the new ALLOCATE for the VSAM file. I haven’t seen any way described of ALLOCATING a VSAM file other than using IDCAMS, except in 3270 screens. I think I’ve seen that LzLabs has 3270 capabilities for this sort of stuff, but I’m not inclined to try to figure out how to use it. I’d rather use our much more intuitive GUI or do it in script with JCL like this.

Copy a DATASET.

Here is some JCL to copy an (INLINE) dataset into the VSAM file created above

//COPY2VS EXEC PGM=IDCAMS
//TARGET DD DSN=PJOOT.XXXXX005,DISP=(OLD,KEEP,KEEP)
//INLINEDD DD DATA,DCB=(BLKSIZE=240,LRECL=240,RECFM=F)
a
brown
fox
quick
/*
//SYSIN DD *
REPRO -
  INFILE(INLINEDD) -
  OUTFILE(TARGET)
/*
//SYSPRINT DD SYSOUT=*
//SYSOUT   DD SYSOUT=*
//SYSTERM  DD SYSOUT=*

There are two quirks that are noteworthy here.

  1. The VSAM file requires the input be sorted, which is why the words from ‘a quick brown fox’ are in the explicitly sorted order above.
  2. The VSAM file was created with RECORDSIZE 240, so the input file had to be forced to LRECL=240 to match.

Omission of either sort or the LRECL matching causes the VSAM load to fail.

This was the first time that I’d seen this specific INLINE DD syntax, with explicit parameters.  The way I’d seen it before was how SYSIN was specified above with ‘NAME DD *’, ending with C “comment start” /* sequence.  It turns out the default end of file delimiter can also be specified, for example, this also works:

//INLINEDD DD DATA,DLM=@@,DCB=(BLKSIZE=240,LRECL=240,RECFM=F)
a
brown
fox
quick
@@

Cat a file to spool

Because IDCAMS can copy files, this can also be used to cat a file to SPOOL if desired.  Here’s an example:

//CATVS JOB
//CATVS EXEC PGM=IDCAMS
//TARGET DD DSN=PJOOT.XXXXX005,DISP=(OLD,KEEP,KEEP)
//SYSIN DD *
REPRO -
  INFILE(TARGET) -
  OUTFILE(SYSOUT)
/*
//SYSPRINT DD SYSOUT=*
//SYSOUT   DD SYSOUT=*
//SYSTERM  DD SYSOUT=*

If I include a step like this, I’m able to see the file contents in our nice GUI spool browser along with the JCL script and all the other output.