C/C++ development and debugging.

splitting the last git commit into two

April 23, 2020 C/C++ development and debugging. No comments , ,

In the blog post, Split a commit in two with Git, Emmanuel provides a super clear explanation of how to split an old commit into multiple commits, separating that commit into different commits, each with a subset of the files initially committed.

It took me a while before I could figure out how to apply this to the very last commit.  Here’s the required git magic:

git log -n 1 > m
git reset HEAD^
git add ...
git commit -m "First part"
git add ...
git commit -m "Second part"

The differences are really to just skip the first and last rebase steps (don’t do an interactive rebase, and don’t continue that rebase when done.) This was probably obvious to the author of the more general instructions.

Note that before resetting HEAD to the previous commit, I collect the current commit message, under the assumption that portions of it will be used in either of the two (or more) new commit messages.  If you don’t do that, you can fish it out of your history by looking at ‘git reflog’ to see what the message was before mucking around with HEAD.

Interesting z/OS (clang based) compiler release notes.

December 13, 2019 C/C++ development and debugging. No comments , , , ,

The release notes for the latest z/OS C/C++ compiler are interesting.  When I was at IBM they were working on “clangtana”, a clang frontend melded with the legacy TOBY backend.  This really surprised me, but was consistent with the fact that the IBM compiler guys kept saying that they were continually losing their internal funding — that project was a clever way to do more with less resources.  I think they’d made the clangtana switch for zLinux by the time I left, with AIX to follow once they had resolved some ABI incompatibility issues.  At the time, I didn’t know (nor care) about the status of that project on z/OS.

Well, years later, it looks like they’ve now switched to a clang based compiler frontend on z/OS too.  This major change appears to have a number of side effects that I can imagine will be undesirable to existing mainframe customers:

  • Compiler now requires POSIX(ON) and Unix System Services.  No more compilation using JCL.
  • Compiler support for 31-bit applications appears to be dropped (64-bit only!)
  • Support for C, FASTLINK, and OS linkage conventions has been dropped (XPLINK only.)
  • Only ibm-1047 is supported for both source and runtime character set encoding.
  • C89 support appears to have been dropped.
  • Hex floating support has been dropped.
  • No decimal floating point support.
  • SIMD support isn’t implemented.
  • Metal C support has been dropped.

i.e. if you want C++14, you have to be willing to give up a lot to get it.  They must be using an older clang, because this “new” compiler doesn’t include C++17 support.  I’m surprised that they didn’t even manage multiple character set support for this first compiler release.

It is interesting that they’ve also dropped IPA and PDF support, and that the optimization options have changed.  Does that mean that they’ve actually not only dropped the old Montana frontend, but also gutted the whole backend, switching to clang exclusively?

An example of Linux/glibc symbol versioning.

September 20, 2019 C/C++ development and debugging. 3 comments , , , , , , ,

Here’s a little introduction to Linux/glibc symbol versioning.  The sources, linker version scripts, and makefile from this post can be found here in github.

The definitive reference for this topic is Ulrich Drepper’s dsohowto document.

Linux symbol versioning is a very logical way to construct a definitive API for your product.  If you are careful, you can have a single version of your shared library that is binary compatible with all previous versions of code that have linked it, and additionally hide all symbols that you don’t want your API consumer to be able to use (as if all non-exported symbols have a “static” like scope within the library itself).

That visibility hiding can be done in other ways (such as using static, or special compiler options), but utilizing a symbol version linker script to do so also has all the additional binary compatibility related benefits.

Suppose that we have a “v1.1” API with the following implementation:

#include <stdio.h>

void internal_function_is_not_visible()
   printf( "You can't call me directly.\n" );

void foo( int x )

   printf( "foo@@V1: %d\n", x );

and we build this into a shared library like so:

$ make libfoo.1.so
rm -f libfoo.1.so
cc foo.c -g -Wl,--version-script=symver.map -fpic -o libfoo.1.so -Wl,-soname,libfoo.so -shared
rm -f libfoo.so
ln -s libfoo.1.so libfoo.so

Everything here is standard for building a shared library except for the –version-script option that is passed into the linker with -Wl,. That version script file has the following contents:

$ cat symver.map
	  local: *;

This defines a “V1.1” API where all the symbols that are exported with a symbol version @@MYSTUFF_1.1. Note that internal_function_is_not_visible is not in that list. It’s covered in the local: catch-all portion of the symbol version file. Code that calls foo does not look out of the ordinary:

void foo(int);

int main()

   return 0;

Compiling and linking that code is also business as usual:

$ make caller1
cc caller.c -g -Wl,-rpath,`pwd` -o caller1 -lfoo -L.

However, look at the foo symbol reference that we have for this program:

$ nm caller1 | grep foo
                 U foo@@MYSTUFF_1.1

If we run this, we get:

$ ./caller1
You can't call me directly.
foo@@V1: 1

If you add in a call to internal_function_is_not_visible() you’ll see that compilation fails:

void foo(int);
void internal_function_is_not_visible();

int main()

   return 0;

$ make caller1
cc caller.c -g -Wl,-rpath,`pwd` -o caller1 -lfoo -L.
/run/user/1002/ccqEPYcu.o: In function `main':
/home/pjoot/symbolversioning/caller.c:7: undefined reference to `internal_function_is_not_visible'
collect2: error: ld returned 1 exit status
make: *** [caller1] Error 1

This is because internal_function_is_not_visible is not a visible symbol. Cool. We now have versioned symbols and symbol hiding. Suppose that we now want to introduce a new binary incompatible change too our foo API, but want all existing binaries to still work unchanged. We can do so by introducing a symbol alias, and implementations for both the new and the OLD API.

#include <stdio.h>

void internal_function_is_not_visible()
   printf( "You can't call me directly.\n" );

void foo2( int x, int y )
   if ( y < 2 )
   printf( "foo@@V2: %d %d\n", x, y );

void foo1( int x )
   printf( "foo@@V1: %d\n", x );

This is all standard C up to this point, but we now add in a little bit of platform specific assembler directives (using gcc specific compiler sneaks) :

#define V_1_2 "MYSTUFF_1.2"
#define V_1_1 "MYSTUFF_1.1"

#define SYMVER( s ) \
    __asm__(".symver " s )

SYMVER( "foo1,foo@" V_1_1 );
SYMVER( "foo2,foo@@" V_1_2 );

We’ve added a symbol versioning alias for foo@@MYSTUFF_1.2 and foo@MYSTUFF_1.1. The @@ one means that it applies to new code, whereas the @MYSTUFF_1.1 is a load only function, and no new code can use that symbol. In the symbol version script we now introduce a new version stanza:

$ cat symver.2.map
	  local: *;

$ make libfoo.2.so
rm -f libfoo.2.so
cc foo.2.c -g -Wl,--version-script=symver.2.map -fpic -o libfoo.2.so -Wl,-soname,libfoo.so -shared
rm -f libfoo.so
ln -s libfoo.2.so libfoo.so

If we call the new V1.2 API program, like so:

void foo(int, int);
int main()
   foo(1, 2);
   return 0;

our output is now:

$ ./caller2
foo@@V2: 1 2

Our “binary incompatible changes” are two fold. We don’t call internal_function_is_not_visible if our new parameter is >= 2, and we print a different message.

If you look at the symbols that are referenced by the new binary, you’ll see that it now explicitly has a v1.2 dependency:

$ nm caller2 | grep foo
                 U foo@@MYSTUFF_1.2

The lldb TUI (text user interface)

August 26, 2019 C/C++ development and debugging. No comments , , ,

It turns out, like GDB, that lldb has a TUI mode too, but it’s really simplistic.  You enter with

(lldb) gui

at which point you get a full screen of code or assembly, and options for register exploration, thread and stack exploration, and a variable view.  The startup screen looks like:

If you tab over to the Threads window, you can space select the process, and drill into the stack traces for any of the running threads

You can also expand the regsiters by register class:

I’d like to know how to resize the various windows.  If you resize the terminal, the size of the stack view pane seems to remain fixed, so the symbol names always end up truncated.

Apparently this code hasn’t been maintained or developed since it was added.  Because there is no console pane, you have to set all desired breakpoints and continue, then pop into the GUI to look at stuff, and then <F1> to get back to the console prompt.  It’s nice that it gives you a larger view of the code, but given that lldb already displays context around each line, the lldb TUI isn’t that much of a value add in that respect.

This “GUI” would actually be fairly usable if it just had a console pane.

gdb pretty print of structures

March 9, 2017 C/C++ development and debugging. No comments , , , ,

Here’s a nice little gdb trick for displaying structure contents in a less compact format

(gdb) set print pretty on
(gdb) p dd[0]
$4 = {
  jfcb = {
    datasetName = "PJOOT.NVS1", ' ' <repeats 34 times>,
    vols = {"<AAAiW", "\000\000\000\000\000", "\000\000\000\000\000", "\000\000\000\000\000", "\000\000\000\000\000"},
  block_size = 800,
  device_class = 32 '\040',
  device_type = 15 '\017',
  disp_normal = 8 '\010',
  disp_cond = 8 '\010',
  volsers = 0x7fb71801ecd6 "<AAAiW",

compare this to the dense default

(gdb) set print pretty on
(gdb) p dd[0]
$5 = {jfcb = {datasetName = "PJOOT.NVS1", ... vols = {"<AAAiW", "\000\000\000\000\000", "\000\000\000\000\000", "\000\000\000\000\000", "\000\000\000\000\000"}, block_size = 800, block_size_limit = 0, device_class = 32 '\040', device_type = 15 '\017', disp_normal = 8 '\010', disp_cond = 8 '\010', volsers = 0x7fb71801ecd6 "<AAAiW"}

For really big structures (this one actually is, but I’ve pruned a bunch of stuff), this makes the structure print display a whole lot more readable. Additionally, if you combine this with ‘(gdb) set logging on’, then with pretty print enabled you can prune the output by line easily to see just what you want.

peeking into relocation of function static in shared library

February 27, 2017 C/C++ development and debugging. 2 comments , , , , , , , ,

Here’s GUI (TUI) output of a function with a static variable access:

B+ |0x7ffff7616800 <st32>           test   %edi,%edi                                                                                                                     |
   |0x7ffff7616802 <st32+2>         je     0x7ffff7616811 <st32+17>                                                                                                      |
   |0x7ffff7616804 <st32+4>         mov    %edi,%eax                                                                                                                     |
   |0x7ffff7616806 <st32+6>         bswap  %eax                                                                                                                          |
   |0x7ffff7616808 <st32+8>         mov    %eax,0x200852(%rip)        # 0x7ffff7817060 <st32.yst32>                                                                        |
   |0x7ffff761680e <st32+14>        mov    %edi,%eax                                                                                                                     |
   |0x7ffff7616810 <st32+16>        retq                                                                                                                                 |
  >|0x7ffff7616811 <st32+17>        mov    0x200849(%rip),%edi        # 0x7ffff7817060 <st32.yst32>                                                                        |
   |0x7ffff7616817 <st32+23>        bswap  %edi                                                                                                                          |
   |0x7ffff7616819 <st32+25>        mov    %edi,%eax                                                                                                                     |
   |0x7ffff761681b <st32+27>        retq                                                                                                                                 |
   |0x7ffff761681c <_fini>          sub    $0x8,%rsp                                                                                                                     |
   |0x7ffff7616820 <_fini+4>        add    $0x8,%rsp                                                                                                                     |
   |0x7ffff7616824 <_fini+8>        retq                                                                                                                                 |
   |0x7ffff7616825                  add    %al,(%rcx)                                                                                                                    |
   |0x7ffff7616827 <x16+1>          add    (%rcx),%al                                                                                                                    |

The associated code is:

int st32( int v ) {
    static int yst32 = 0x1a2b3c4d;

    if ( v ) {
        yst32 = v;

    return yst32;

The object code dump (prior to relocation) just has zeros in the offset for the variable:

$ objdump -d g.bs.o | grep -A12 '<st32>'
0000000000000050 <st32>:
  50:   85 ff                   test   %edi,%edi
  52:   74 0d                   je     61 <st32+0x11>
  54:   89 f8                   mov    %edi,%eax
  56:   0f c8                   bswap  %eax
  58:   89 05 00 00 00 00       mov    %eax,0x0(%rip)        # 5e <st32+0xe>
  5e:   89 f8                   mov    %edi,%eax
  60:   c3                      retq   
  61:   8b 3d 00 00 00 00       mov    0x0(%rip),%edi        # 67 <st32+0x17>
  67:   0f cf                   bswap  %edi
  69:   89 f8                   mov    %edi,%eax
  6b:   c3                      retq   

The linker has filled in the real offsets in question, and the dynamic loader has collaborated to put the data segment in the desired location.

The observant reader may notice bwsap instructions in the listings above that don’t make sense for x86_64 code. That is because this code is compiled with an LLVM pass that performs byte swapping at load and store points, making it big endian in a limited fashion.

The book Linkers and Loaders has some nice explanation of how relocation works, but I wanted to see the end result first hand in the debugger. It turned out that my naive expectation that the sum of $rip and the constant relocation factor is the address of the global variable (actually static in this case) is incorrect. Check that out in the debugger:

(gdb) p /x 0x200849+$rip
$1 = 0x7ffff781705a

(gdb) x/10 $1
0x7ffff781705a <gy+26>: 0x22110000      0x2b1a4433      0x00004d3c      0x00000000
0x7ffff781706a: 0x00000000      0x00000000      0x00000000      0x30350000
0x7ffff781707a: 0x20333236      0x64655228

My magic value 0x1a2b3c4d looks like it is 6 bytes into the $rip + 0x200849 location that the disassembly appears to point to, and that is in fact the case:

(gdb) x/10 $1+6
0x7ffff7817060 <st32.yst32>:      0x4d3c2b1a      0x00000000      0x00000000      0x00000000
0x7ffff7817070 <y32>:   0x00000000      0x00000000      0x32363035      0x52282033
0x7ffff7817080: 0x48206465      0x34207461

My guess was the mysterious offset of 6 required to actually find this global address was the number of bytes in the MOV instruction, and sure enough that MOV is 6 bytes long:

(gdb) disassemble /r
Dump of assembler code for function st32:
   0x00007ffff7616800 <+0>:     85 ff   test   %edi,%edi
   0x00007ffff7616802 <+2>:     74 0d   je     0x7ffff7616811 <st32+17>
   0x00007ffff7616804 <+4>:     89 f8   mov    %edi,%eax
   0x00007ffff7616806 <+6>:     0f c8   bswap  %eax
   0x00007ffff7616808 <+8>:     89 05 52 08 20 00       mov    %eax,0x200852(%rip)        # 0x7ffff7817060 <st32.yst32>
   0x00007ffff761680e <+14>:    89 f8   mov    %edi,%eax
   0x00007ffff7616810 <+16>:    c3      retq
=> 0x00007ffff7616811 <+17>:    8b 3d 49 08 20 00       mov    0x200849(%rip),%edi        # 0x7ffff7817060 <st32.yst32>
   0x00007ffff7616817 <+23>:    0f cf   bswap  %edi
   0x00007ffff7616819 <+25>:    89 f8   mov    %edi,%eax
   0x00007ffff761681b <+27>:    c3      retq
End of assembler dump.

So, it appears that the %rip reference in the disassembly is really the value of the instruction pointer after the instruction executes, which is curious.

Note that this 4 byte relocation requires the shared library code segment and the shared library data segment be separated by no more than 4G. The linux dynamic loader has put all the segments back to back so that this is the case. This can be seen from /proc/PID/maps for the process:

$ ps -ef | grep maindl
pjoot    17622 17582  0 10:50 pts/3    00:00:00 /home/pjoot/workspace/pass/global/maindl libglobtestbs.so

$ grep libglob /proc/17622/maps
7ffff7616000-7ffff7617000 r-xp 00000000 fc:00 2492653                    /home/pjoot/workspace/pass/global/libglobtestbs.so
7ffff7617000-7ffff7816000 ---p 00001000 fc:00 2492653                    /home/pjoot/workspace/pass/global/libglobtestbs.so
7ffff7816000-7ffff7817000 r--p 00000000 fc:00 2492653                    /home/pjoot/workspace/pass/global/libglobtestbs.so
7ffff7817000-7ffff7818000 rw-p 00001000 fc:00 2492653                    /home/pjoot/workspace/pass/global/libglobtestbs.so

We’ve got a read-execute mmap region, where the code lies, and a read-write mmap region for the data. There’s a read-only segment which I presume is for read only global variables (my shared lib has one such variable and we have one page worth of space allocated for read only memory).

I wonder what the segment that has none of the read, write, nor execute permissions set is?

New book for work: Linkers and Loaders

February 24, 2017 C/C++ development and debugging. No comments , ,

Fresh off the press:

I got this book to get some background on relocation of ELF globals, and was surprised to find a bit on z/OS (punch card compatible!) object format layout:

… an interesting bonus that’s topical.

gdb set target-charset

January 9, 2017 C/C++ development and debugging. No comments , , ,

I was looking for a way to convert ASCII and EBCDIC strings in gdb debugging sessions and was experimenting with gdb python script extensions. I managed to figure out how to add my own command that read a gdb variable, and print it out, but it failed when I tried to run a character conversion function. In the process of debugging that char encoding error, I found that there’s a built in way to do exactly what I wanted to do:

(gdb) p argv[0]
$16 = 0x7fd8fbda0108 "\323\326\303\301\323\305\303\326"
(gdb) set target-charset EBCDIC-US
(gdb) p argv[0]
$17 = 0x7fd8fbda0108 "LOCALECO"
(gdb) set target-charset ASCII
(gdb) p argv[0]
$18 = 0x7fd8fbda0108 "\323\326\303\301\323\305\303\326"

A wierd way to invoke the compiler

November 18, 2016 C/C++ development and debugging. 2 comments ,

Did you know that you can run the compiler from sources specified in stdin?  Here’s an example:

// m.c
#include <stdio.h>

int main()
    int x = 3;
    printf("%d\n", x);
    return 0;

You have to specify the language for the code explicitly, since that can’t be inferred from the filename when that file data is coming from stdin:

$ cat m.c | clang -g -x c - -o f
$ ./f

This fact came up in conversation the other day. The result is something that is completely undebuggable, but you can do it! I’m curious if there’s actually a use case for this?

Another Linux shared library trace facility

October 27, 2016 C/C++ development and debugging. No comments , ,

I previously blogged about a way to force ltrace to show some shared memory trace records that didn’t show up by default.

Where that fails to be useful, is when you don’t have a guess about what shared library the code in question lives in. I just blundered on the latrace command that uses a Linux dynamic loader audit facility to give a complete trace of all the function-name/library-name pairs that are executed!

Here’s an example invocation:

latrace \
clang xx.c -c 2>&1 | c++filt

without output like:

 9022     std::operator&(std::memory_order, std::__memory_order_modifier) [/home/pjoot/clang/be.5e0ac1f.lz31/bin/../lib/libLLVMAnalysis.so]
 9022     std::operator&(std::memory_order, std::__memory_order_modifier) [/home/pjoot/clang/be.5e0ac1f.lz31/bin/../lib/libLLVMAnalysis.so]
 9022     strlen [/lib64/libc.so.6]
 9022     strlen [/lib64/libc.so.6]
 9022     strlen [/lib64/libc.so.6]
 9022     llvm::cl::basic_parser::basic_parser(llvm::cl::Option&) [/home/pjoot/clang/be.5e0ac1f.lz31/bin/../lib/libLLVMSupport.so]
 9022     strlen [/lib64/libc.so.6]
 9022     llvm::cl::Option::setArgStr(llvm::StringRef) [/home/pjoot/clang/be.5e0ac1f.lz31/bin/../lib/libLLVMSupport.so]
 9022     strlen [/lib64/libc.so.6]
 9022     std::pair::__type, std::__decay_and_strip::__type> std::make_pair(void const**&&, bool&&) [/home/pjoot/clang/be.5e0ac1f.lz31/bin/../lib/libLLVMX86CodeGen.so]
 9022       void const**&& std::forward(std::remove_reference::type&) [/home/pjoot/clang/be.5e0ac1f.lz31/bin/../lib/libLLVMX86CodeGen.so]
 9022       bool&& std::forward(std::remove_reference::type&) []
 9022       void const**&& std::forward(std::remove_reference::type&) [/home/pjoot/clang/be.5e0ac1f.lz31/bin/../lib/libLLVMX86CodeGen.so]
 9022       bool&& std::forward(std::remove_reference::type&) []
 9022     void const**&& std::forward(std::remove_reference::type&) [/home/pjoot/clang/be.5e0ac1f.lz31/bin/../lib/libLLVMX86CodeGen.so]

With this latrace command, we get all the shared library function call names and their corresponding shared library names. Using that info we can dig into a specific shared library with ltrace or the debugger, once a point of interest is determined.