linux

A great tip for throttling IO intensive processes.

July 9, 2022 Linux , , ,

If I do anything too IO intensive on my Vmware Linux installation (running on MacOSX), it often crashes my VM and the MacOS host along with it, and I have no choice but to powercycle the mac.

This used to happen with my previous Mac too (both work machines, intel generation cpus, not M1/M2.)

Today I crashed my VM/host pair trying to unpack a 4G tar.gz file, and found this great way to throttle the unpacking enough that my machine stayed alive.

My use case was:

# cat master.tar.gz | pv -L 50M | tar zxf -       

I’m running RHEL8 on my VM, but was able to find a EPEL pv rpm (pv-1.6.6-7.el8.x86_64.rpm) to install on rpmsearch.

Reimaged Windows laptop for dual Windows-Fedora boot.

June 6, 2021 Incoherent ramblings , , , , , , , ,

I used my funky usb-C external drive enclosure to facilitate an operating system switcheroo today.  I wanted to try a Windows/Linux dual boot configuration, something that I haven’t done in a long time.  There was enough room on the original 1Tb SSD, but I’d bought myself a 2Tb SSD for backups (and crypto-mining experimentation) and decided to repurpose that as a replacement drive for my Windows (xpg) laptop.

Since both my new and old drives were both M2 SSD drives, I was able to pop the old drive and put it into my external enclosure.  If I messed up reinstalling either Windows or Linux, then at least theoretically, I could have recovered by just putting the old drive back in.

I didn’t mess up the installation and now have a brand new dual boot configuration with Windows-10 and Fedora-34.  I’ve got WSL-2 + ubuntu-20 on the Windows side:

and a grub boot selector for the operating systems:

and the Fedora-34 desktop configuration for the native Linux installation:

I opted for Fedora over Ubuntu, since I figured both track the state of the art fairly well, but with Fedora I don’t have to keep looking up the dpkg equivalents of the rpm commands I’ve used for so many years and have memorized.

I have a whole bunch of install fine tuning to do still, but have all the bare bones now installed on both sides.

Having an external enclosure for the SSD made migration really easy.  I was able to mount it after my Windows reinstall was done, and just move my old \Users\peete directory.  I’ll have to weed out the bits that I don’t care about, but I’m now ready to blast away the partitions from the original laptop installation, and then use that 1Tb drive for backups and file transfers between machines.

Introducing a fixed capacity directory (Linux).

March 23, 2021 Linux , , , ,

I have a dump directory that is too easily filled up (with cores and dumps) if a programming error makes our system misbehave catastrophically.

Here’s a nice trick to create a small filesystem with fixed capacity so that the owning filesystem (in my case /var) can’t be filled up:

dd if=/dev/zero of=dump.loopback bs=1M count=10
mke2fs ./dump.loopback
mount -o loop dump.loopback ./dump
chmod 777 ./dump

A minimally configured Windows laptop

November 22, 2020 Windows , , , , , ,

I’ve now installed enough that my new Windows machine is minimally functional (LaTex, Linux, and Mathematica), with enough installed that I can compile any of my latex based books, or standalone content for blog posts.  My list of installed extras includes:

  • Brother HL-2170W (printer driver)
  • Windows Terminal
  • GPL Ghostscript (for MaTeX, latex labels in Mathematica figures.)
  • Wolfram Mathematica
  • Firefox
  • Chrome
  • Visual Studio
  • Python
  • Julia
  • Adobe Acrobat Reader
  • Discord
  • OBS Studio
  • MikTeX
  • SumatraPDF
  • GVim
  • Git
  • PowerShell (7)
  • Ubuntu
  • Dropbox

Some notes:

  • On Windows, for my LaTeX work, I used to use MikTex + cygwin.  The cygwin dependency was for my makefile dependencies (gnu-make+perl).  With this new machine, I tried WSL2.  I’m running my bash shells within the new Windows Terminal, which is far superior to the old cmd.
  • Putty is no longer required.  Windows Terminal does the job very nicely.  It does terminal emulation well enough that I can even ssh into a Linux machine and use screen within my Linux session, and my .screenrc just works.  Very nice.
  • SumatraPDF is for latex reverse tex lookup.  i.e. I can double click on pdf content, and up pops the editor with the latex file.  Last time I used Sumatra, I had to configure it to use GVim (notepad used to be the default I think.)  Now it seems to be the default (to my suprise.)
  • I will probably uninstall Git, as it seems superfluous given all the repos I want to access are cloned within my bash file system.
  • I used to use GVim extensively on Windows, but most of my editing has been in vim in the bash shell.  I expect I’ll now only use it for reverse tex (–synctex) lookup editing.

WSL2 has very impressive integration.  A really nice demo of that was access of synctex lookup.  Here’s a screenshot that shows it in action:

I invoked the windows pdf viewer within a bash shell in the Ubuntu VM, using the following:

 
pjoot@DESKTOP-6J7L1NS:~/project/blogit$ alias pdfview
alias pdfview='/mnt/c/Users/peete/AppData/Local/SumatraPDF/SumatraPDF.exe'
pjoot@DESKTOP-6J7L1NS:~/project/blogit$ pdfview fibonacci.pdf

The Ubuntu filesystem directory has the fibonacci.synctex.gz reverse lookup index that Summatra is able to read. Note that this file, after unzipping, has only Linux paths (/home/pjoot/…), but Summatra is able to use those without any trouble, and pops up the (Windows executable) editor on the files after I double click on the file. This sequence is pretty convoluted:

  • Linux bash ->
  • invoke Windows pdf viewer ->
  • that program reading Linux files ->
  • it invokes a windows editor (presumably using the Linux path), and that editor magically knows the path to the Linux file that it has to edit.

Check out the very upper corner of that GVim window, where it shows the \\wsl$\Ubuntu\home\pjoot\project\blogit\fibonacci.tex path

As well as full Linux access to the Windows filesystem, we have full Windows access to the Linux filesystem.

Not all applications know how to access files with UNC paths (for example, the old crappy cmd.exe cannot), but so far all the ones I have cared about have been able to do so.

An example of Linux/glibc symbol versioning.

September 20, 2019 C/C++ development and debugging. , , , , , , ,

Here’s a little introduction to Linux/glibc symbol versioning.  The sources, linker version scripts, and makefile from this post can be found here in github.

The definitive reference for this topic is Ulrich Drepper’s dsohowto document.

Linux symbol versioning is a very logical way to construct a definitive API for your product.  If you are careful, you can have a single version of your shared library that is binary compatible with all previous versions of code that have linked it, and additionally hide all symbols that you don’t want your API consumer to be able to use (as if all non-exported symbols have a “static” like scope within the library itself).

That visibility hiding can be done in other ways (such as using static, or special compiler options), but utilizing a symbol version linker script to do so also has all the additional binary compatibility related benefits.

Suppose that we have a “v1.1” API with the following implementation:

#include <stdio.h>

void internal_function_is_not_visible()
{
   printf( "You can't call me directly.\n" );
}

void foo( int x )
{
   internal_function_is_not_visible();

   printf( "foo@@V1: %d\n", x );
}

and we build this into a shared library like so:

$ make libfoo.1.so
rm -f libfoo.1.so
cc foo.c -g -Wl,--version-script=symver.map -fpic -o libfoo.1.so -Wl,-soname,libfoo.so -shared
rm -f libfoo.so
ln -s libfoo.1.so libfoo.so

Everything here is standard for building a shared library except for the –version-script option that is passed into the linker with -Wl,. That version script file has the following contents:

$ cat symver.map
MYSTUFF_1.1 {
	  global:
foo;
	  local: *;
	};

This defines a “V1.1” API where all the symbols that are exported with a symbol version @@MYSTUFF_1.1. Note that internal_function_is_not_visible is not in that list. It’s covered in the local: catch-all portion of the symbol version file. Code that calls foo does not look out of the ordinary:

void foo(int);

int main()
{
   foo(1);

   return 0;
}

Compiling and linking that code is also business as usual:

$ make caller1
cc caller.c -g -Wl,-rpath,`pwd` -o caller1 -lfoo -L.

However, look at the foo symbol reference that we have for this program:

$ nm caller1 | grep foo
                 U foo@@MYSTUFF_1.1

If we run this, we get:

$ ./caller1
You can't call me directly.
foo@@V1: 1

If you add in a call to internal_function_is_not_visible() you’ll see that compilation fails:

void foo(int);
void internal_function_is_not_visible();

int main()
{
   foo(1);
   internal_function_is_not_visible();

   return 0;
}

$ make caller1
cc caller.c -g -Wl,-rpath,`pwd` -o caller1 -lfoo -L.
/run/user/1002/ccqEPYcu.o: In function `main':
/home/pjoot/symbolversioning/caller.c:7: undefined reference to `internal_function_is_not_visible'
collect2: error: ld returned 1 exit status
make: *** [caller1] Error 1

This is because internal_function_is_not_visible is not a visible symbol. Cool. We now have versioned symbols and symbol hiding. Suppose that we now want to introduce a new binary incompatible change too our foo API, but want all existing binaries to still work unchanged. We can do so by introducing a symbol alias, and implementations for both the new and the OLD API.

#include <stdio.h>

void internal_function_is_not_visible()
{
   printf( "You can't call me directly.\n" );
}

void foo2( int x, int y )
{
   if ( y < 2 )
   {
      internal_function_is_not_visible();
   }
   printf( "foo@@V2: %d %d\n", x, y );
}

void foo1( int x )
{
   internal_function_is_not_visible();
   printf( "foo@@V1: %d\n", x );
}

This is all standard C up to this point, but we now add in a little bit of platform specific assembler directives (using gcc specific compiler sneaks) :

#define V_1_2 "MYSTUFF_1.2"
#define V_1_1 "MYSTUFF_1.1"

#define SYMVER( s ) \
    __asm__(".symver " s )

SYMVER( "foo1,foo@" V_1_1 );
SYMVER( "foo2,foo@@" V_1_2 );

We’ve added a symbol versioning alias for foo@@MYSTUFF_1.2 and foo@MYSTUFF_1.1. The @@ one means that it applies to new code, whereas the @MYSTUFF_1.1 is a load only function, and no new code can use that symbol. In the symbol version script we now introduce a new version stanza:

$ cat symver.2.map
MYSTUFF_1.2 {
	  global:
foo;
};
MYSTUFF_1.1 {
	  global:
foo;
	  local: *;
};

$ make libfoo.2.so
rm -f libfoo.2.so
cc foo.2.c -g -Wl,--version-script=symver.2.map -fpic -o libfoo.2.so -Wl,-soname,libfoo.so -shared
rm -f libfoo.so
ln -s libfoo.2.so libfoo.so

If we call the new V1.2 API program, like so:

void foo(int, int);
int main()
{
   foo(1, 2);
   return 0;
}

our output is now:

$ ./caller2
foo@@V2: 1 2

Our “binary incompatible changes” are two fold. We don’t call internal_function_is_not_visible if our new parameter is >= 2, and we print a different message.

If you look at the symbols that are referenced by the new binary, you’ll see that it now explicitly has a v1.2 dependency:

$ nm caller2 | grep foo
                 U foo@@MYSTUFF_1.2