decoding some powerpc rotate-mask instructions

November 4, 2014 C/C++ development and debugging. , , , , , , ,

I’m looking at what might be an issue in optimized code, where we are seeing some signs that expected byteswapping is not occuring when optimization is enabled. Trying to find exactly where the code is that does this swapping was been a bit challenging, so I decided to look a simple 2-byte swap sequence in a standalone program first. For such code I see the following in the listing file (xlC compiler)

  182| 00002C rlwinm   5405C23E   1     SRL4      gr5=gr0,8
  182| 000030 rlwinm   5400063E   1     RN4       gr0=gr0,0,0xFF
  182| 000034 rldimi   7805402C   1     RI8       gr5=gr0,8,gr5,0xFFFFFF00

The SRL4, RN4, and RI8 “mnemonics” are internal compiler codes meant to be “intuitive”, but I didn’t find them intuitive until I figured out what the instructions actually did. Here’s a couple reverse engineering notes for that task. Tackling the first rlwinm instruction, note that the raw instruction corresponds to:

0x5405C23E == rlwinm r5,r0,24,8,31

Here’s what the Power ISA says about rlwinm:

Rotate Left Word Immediate then AND with Mask 

rlwinm RA,RS,SH,MB,ME

n  <= SH
r  <= ROTL_32 ((RS)_32:63 , n)
m  <= MASK(MB+32, ME+32)
RA <= r & m

The contents of register RS are rotated 32 left SH bits. A
mask is generated having 1-bits from bit MB+32
through bit ME+32 and 0-bits elsewhere. The rotated
data are ANDed with the generated mask and the
result is placed into register RA.

To interpret this we have to look up the meanings of all the MASK and ROTL operations. My first attempt at that, I got the meaning of MASK() wrong, since I was counting bits from the wrong end. I resorted to the following to figure out the instruction, using gcc inline asm immediate operand constraints “i”, to build up the instruction I wanted to examine the effects of:

#include <stdio.h>

#define rlwinm( output, input, sh, mb, me ) \
   __asm__( "rlwinm %0, %1, %2, %3, %4" \
          : "=r"(output) \
          : "r"(input), "i"(sh), "i"(mb), "i"(me) \
          : )

int main()
   long x = 0x1122334455667788L ;
   long y ;

   rlwinm( y, x, 24, 8, 31 ) ;

   printf("0x%016lX -> 0x%016lX\n", x, y ) ;

   return 0 ;

This generates an rlwinm instruction with the SH,MB,ME=24,8,31 triplet that I’d found in the listing. This code produces:

0x1122334455667788 -> 0x0000000000556677

Observing the effects of the instruction in this concrete example makes it easier to interpret the effects of the instruction. That seems to be:

   long y = ((int)x << 24) | (char)(x >> 8) ;
   y  |= (y << 32) ; (* The ROTL_32 operation in the ISA appears to have this effect, but it is killed in the mask application *)
   y &= 0xFFFFFF ;

Now the internal mnemonic “SRL4 …,8” has a specific meaning. It looks like it means Shift-Right-Lower4ByteWord 8 bits. It’s intuitive when you know that the L here means Lower. I didn’t guess that, and wondered what the hell shift RightLeft meant.

What does RN4 mean? That instruction was:

0x5400063E == rlwinm r0,r0,0,24,31

This has no shift, but applies a mask, and that mask has 16 bits less ones in it. This appears to be an AND with 0xFF. A little test program using “rlwinm( y, x, 0, 24, 31 )” this time confirms this, as it produces:

0x1122334455667788 -> 0x0000000000000088

What could the R and N have meant? Knowing what the instruction does, I’d now guess RotateNone(andMask).

Finally, how about the RI8 operation? This time we have

0x7805402C == rldimi r5,r0,8,32

The PowerISA says of this:

Rotate Left Doubleword Immediate then Mask Insert  

rldimi RA,RS,SH,MB 

n  <= sh_5 || sh_0:4
r  <= ROTL_64 ((RS), n)
b  <= mb_5 || mb_0:4
m  <= MASK(b, ¬n)
RA  <= r&m | (RA) & ¬ m
The contents of register RS are rotated 64 left SH bits. A
mask is generated having 1-bits from bit MB through bit
63-SH and 0-bits elsewhere. The rotated data are
inserted into register RA under control of the generated

Let’s also see if an example makes this easier to understand. This time a read/write modifier + is required on the output operand

#include <stdio.h>

#define rldimi( inout, input, sh, mb ) \
   __asm__( "rldimi %0, %1, %2, %3" \
          : "+r"(inout) \
          : "r"(input), "i"(sh), "i"(mb) \
          : )

int main()
   long x = 0x1122334455667788L ;
   long y = 0x99aabbccddeeff12L ;
   long yo = y ;

   rldimi( y, x, 8, 32 ) ;

   printf("0x%016lX,0x%016lX -> 0x%016lX\n", x, yo, y ) ;

   return 0 ;

This produces:

0x1122334455667788,0x99AABBCCDDEEFF12 -> 0x99AABBCC66778812

It appears that the effect is:

y = (y & ~0xFFFFFF00L) | ((x << 8) & 0xFFFFFF00L) ;

I find it tricky to understand this from the PowerISA description, so if I encountered different values of SH,MB I’d probably run them through this little reverse engineering program. That said, at least the meaning of RI8 in -qlist output is now clear.

A debugger walk through an out of module call on AIX

August 29, 2014 C/C++ development and debugging. , , , , , ,

We were seeing the following powerpc instruction sequence mess up, ending up
with the CTR (counter) register containing zero. The CTR register, which can
be used for computed gotos and other stuff, is one of the only registers that I believe can be
both loaded and branched to easily on powerpc, and is one of the volatile
regisers in the ABI (one that doesn’t have to be spilled to and from the stack
before another call).

0x090000000A9E8B78 : E8040000 ld r0,0(r4)
0x090000000A9E8B7C : 7C0903A6 mtctr r0
0x090000000A9E8B80 : E8440008 ld r2,8(r4)
0x090000000A9E8B84 : 4E800421 bctrl # 20,bit0

I had a recollection that this sequence was a call through a function pointer,
but thought it may also be what we get for a plain old out of module call
(calling something in a shared library from the main text segment or a call to some other shared
library function from a shared library function). Let’s see what an out of module looks like, for a simple call like

#include <stdio.h>

int main(int argc, char ** argv)
   printf( "out of module call\n" ) ;

   return 0;

I set an instruction breakpoint at the bl (branch and link) instruction for
printf, and then step into that

(dbx) stopi at 0x100000788
[1] stopi at 0x100000788 (main+0x28)
(dbx) c
[1] stopped in main at 0x100000788
0x100000788 (main+0x28) 48000059 bl 0x1000007e0 (printf)
(dbx) stepi
stopped in glink64.printf at 0x1000007e0
0x1000007e0 (printf) e98200d8 ld r12,0xd8(r2)

Observe that the debugger is letting us know that we aren’t actually in printf
yet, but are in the glue code for printf. Looking at the instruction sequence
for this glue code we see that it matches the type of code we saw in out NULL
CTR trap sequence above

stopped in glink64.printf at 0x1000007e4
0x1000007e4 (printf+0x4) f8410028 std r2,0x28(r1)

stopped in glink64.printf at 0x1000007e8
0x1000007e8 (printf+0x8) e80c0000 ld r0,0x0(r12)

stopped in glink64.printf at 0x1000007ec
0x1000007ec (printf+0xc) e84c0008 ld r2,0x8(r12)

stopped in glink64.printf at 0x1000007f0
0x1000007f0 (printf+0x10) 7c0903a6 mtctr r0

We save our TOC register (GR2, the table of contents register) to the stack,
load a new value into GR0 to copy to the CTR register, and load the TOC
register (GR2) for the module that we are calling.

Now if we look at what just got put in the TOC register, we see that it’s the
address that we find the actual code for printf at

(dbx) p $r0
(dbx) listi 0x0900000000004f80
0x900000000004f80 (printf) fbe1fff8 std r31,-8(r1)
0x900000000004f84 (printf+0x4) fbc1fff0 std r30,-16(r1)
0x900000000004f88 (printf+0x8) 7c0802a6 mflr r0
0x900000000004f8c (printf+0xc) fba1ffe8 std r29,-24(r1)
0x900000000004f90 (printf+0x10) ebe20dd0 ld r31,0xdd0(r2)
0x900000000004f94 (printf+0x14) f8010010 std r0,0x10(r1)
0x900000000004f98 (printf+0x18) 8002000c lwz r0,0xc(r2)
0x900000000004f9c (printf+0x1c) f821ff71 stdu r1,-144(r1)
0x900000000004fa0 (printf+0x20) 60000000 ori r0,r0,0x0
0x900000000004fa4 (printf+0x24) 2c000000 cmpi cr0,0x0,r0,0x0

The glue code, a branch table for out of module calls, gets us to there, but
we have to pay a number of instruction penalty for this call, in addition to
the normal function call overhead.

What does this mean for the trap scenerio? One implication is that this isn’t
neccessarily as simple as a NULL function pointer. That instruction sequence
is probably different (but I don’t recall exactly how at the moment). Perhaps
this means that the jump table for the currently exectuting shared library got
corrupted? It is probably writable since the run time loader must be able to
modify it. I’d guess that it remains writable throughout execution to support
lazy runtime loader address fixups. This is likely not going to be an easy
problem to solve.