perl and general scripting hackery

Why does touch include a utimensat() syscall?

February 25, 2015 perl and general scripting hackery , , , , , , , , ,

I’m seeing odd time sequencing of files when using clearcase version 8 dynamic views, which makes me wonder about an aspect of the (gnu) touch command. Running:

> cat u
rm -f touchedEarlier touchedLater

perl -e "open (F, '> touchedEarlier') || die"
touch touchedLater

ls --full-time touchedEarlier touchedLater

produces:

> ./u
-rw-r--r-- 1 peeterj pdxdb2 0 2015-02-25 11:42:05.833044000 -0500 touchedEarlier
-rw-r--r-- 1 peeterj pdxdb2 0 2015-02-25 11:42:05.000000000 -0500 touchedLater

Notice that the file that is touched by doing a perl “open” ends up with a later time, despite the fact that it was done logically earlier than the touch.

Running this command outside of a clearcase dynamic view shows zeros only in the subsecond times (also the behaviour of clearcase V7). Needless to say, this difference in file times from their creation sequence wreaks havoc on make.

I was curious how the two touch methods differed, and stracing them shows that the touch differs by including a utimesat() syscall. The perl touch is:

open(“touchedEarlier”, O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff29cd29f0) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=0, …}) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
close(3) = 0
exit_group(0) = ?

whereas the touch command has:

open(“touchedLater”, O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = 3
dup2(3, 0) = 0
close(3) = 0
dup2(0, 0) = 0
utimensat(0, NULL, NULL, 0) = 0
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(0) = ?

It appears that the touch command explicitly zeros the subsecond portion of the files’ timestamp.

I also see that perl’s File::Touch module does the same thing, but uses a different mechanism. I see the following in a strace of such a Touch() call:

stat(“xxyyzz”, 0x656060)                = -1 ENOENT (No such file or directory)
open(“xxyyzz”, O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff16bb3c60) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=0, …}) = 0
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
close(3)                                = 0
utimes(“xxyyzz”, {{1424884553, 0}, {1424884553, 0}}) = 0

I am very curious why touch and perl’s File::Touch() both explicitly zero the subsecond modification time for the file (using utimensat() or utimes() syscalls)?

Some Unix command line one liners

July 24, 2014 perl and general scripting hackery , , , , , , , , , ,

Here’s a couple one-liner shell commands collected over the last couple months when it occurred to me to record them.  Each of these I thought were somewhat notable at the time I did so.

Nested “backquotes”

I often have to run commands where it is convenient to have the parameters of the commands in a file.  A simple example is to edit all the files in a list of files, say:

vim `cat c`
# or:
vim $(cat c)

A useful variation of this is to do the same using the output of a command that also takes its input from a file. Here’s one to edit all the “ancestor” files in the version control system, assuming a command vcsancestor that produces such filenames

vim `vcsancestor $(cat c)`
# or
vim $(vcsancestor $(cat c))

Observe how two different methods of embedding shell commands can be combined into one command. In the past I often used for loops for something like this, say:

for i in `cat c` ; do vcsancestor $i ; done > f
vim `cat f`

(because backquotes can’t be nested). It only recently occurred to me that this isn’t a limitation if $() style subshells are used.

Batching commands with xargs

When working in a version control system, it’s often useful to do a batch checkout of all the files that have compilation errors.  Suppose that you made changes that produced the following compilation error output:

$ cat compile.errors
"satauth.C", line 978.30: 1540-0274 (S) The name lookup for "sqlorest" did not find a declaration.
"scrutil.C", line 142.52: 1540-0274 (S) The name lookup for "SQLNLS_SAME_STRING" did not find a declaration.
"testdrv.C", line 1146.16: 1540-0274 (S) The name lookup for "SQLO_OK" did not find a declaration.
"testdrv.C", line 183.15: 1540-0274 (S) The name lookup for "SQLO_OK" did not find a declaration.

Here’s a one liner to checkout all the files in this list of compilation errors (this is AIX xlC error output):

cut -f2 -d'"' x | sort -u | xargs cleartool checkout -nc

The cut command selects just the (first) double-quote delimited text, then dups are removed with sort -u, and finally xargs is used to run a command on each of the files in the resulting output

Looking for a subset of information delimited by markers on separate lines

grep works nicely for matching patterns that are constrained to a single line.  If you are using gnu-grep you can use the -A and -B options to find stuff after and before the pattern of interest.  As an example, in our stacktrace files (a post mortem crash dump format), we have output that includes:

<pre>

<StackTrace>
—–FUNC-ADDR—- ——FUNCTION + OFFSET——
0x00002AAAC74EF263 ossDumpStackTraceInternal(unsigned long, OSSTrapFile&, int, siginfo*, void*, unsigned long, unsigned long) + 0x06e3
0x00002AAAC74EFE89 ossDumpStackTraceV98 + 0x007f
0x00002AAAC74E5C5F OSSTrapFile::dumpEx(unsigned long, int, siginfo*, void*, unsigned long) + 0x04db
0x00002AAABA6EB313 sqlo_trce + 0x0a6f
0x00002AAABA9C52B5 sqloDumpDiagInfoHandler + 0x047b
0x00002AAAAABD5E00 address: 0x00002AAAAABD5E00 ; dladdress: 0x00002AAAAABC8000 ; offset in lib: 0x000000000000DE00 ;
0x00002AAAAABD30A5 pthread_kill + 0x0035
0x00002AAAB5D828DF ossPthreadKill(unsigned long, unsigned int) + 0x0053
0x00002AAABA9C6CA1 sqloDumpEDU + 0x0091
0x00002AAABED7A853 sqlzerdm + 0x149b
0x00002AAAB5D7D745 sqle_remap_errors(int, sqlca*, sqeAgent*) + 0x01c9
0x00002AAAB5DE8717 sqeApplication::AppStopUsing(sqeAgent*, unsigned char, sqlca*) + 0x10b1
0x00002AAAB5D46FF5 address: 0x00002AAAB5D46FF5 ; dladdress: 0x00002AAAAACE1000 ; offset in lib: 0x000000000B065FF5 ;
0x00002AAAB5D4073F address: 0x00002AAAB5D4073F ; dladdress: 0x00002AAAAACE1000 ; offset in lib: 0x000000000B05F73F ;
0x00002AAAB5D44F35 sqleIndCoordProcessRequest(sqeAgent*) + 0x3959
0x00002AAAB5DA8E55 sqeAgent::RunEDU() + 0x061b
0x00002AAABEDAC2C7 sqzEDUObj::EDUDriver() + 0x035d
0x00002AAABEDABBD7 sqlzRunEDU(char*, unsigned int) + 0x0053
0x00002AAABA9BFC62 sqloEDUEntry + 0x1460
0x00002AAAAABCE2A3 address: 0x00002AAAAABCE2A3 ; dladdress: 0x00002AAAAABC8000 ; offset in lib: 0x00000000000062A3 ;
0x00002AAAC7F376DD __clone + 0x006d
</StackTrace>

</pre>

Here’s a one-liner to grab just the portions of these files within the delimiters (with some other filtering that isn’t of terrible interest to describe)

for i in *stack* ; do grep -A40 ‘<StackTrace’ $i | grep -v ‘(/’ | grep -B40 ‘/StackTrace’ | c++filt ; done | less

Unix to Windows path separator switching

Suppose we have some unix filenames

</pre>
$ head -5 f
/vbs/bin/AEDefines.pm
/vbs/bin/AEMacro.pm
/vbs/bin/bld_shared_lib_Darwin
/vbs/bin/chglibpaths
/vbs/bin/chglibpaths_Darwin



and want the Windows paths for the same

</pre>
$ head -5 f | tr / '\\'
bin\AEDefines.pm
bin\AEMacro.pm
bin\bld_shared_lib_Darwin
bin\chglibpaths
bin\chglibpaths_Darwin
<pre>

The tr command above looks a bit like ascii barf, and will translate forward slashes to backward slashes (perhaps for input that’s a list of files).

I didn’t understand the requirement to both single quote the backslash as well as escaping it, but Darin explained it for me:

Quotes allow the backslashes to go through the shell to tr.  And tr has its own backslash escape mechanism (so you can do things like transform \n into \r or something – where you’d then specify ‘\n’ or just \\n and ‘\r’ or \\r).

Vim: replace search results with contents from a file

Probably related to merging conflicting changes, I wanted to completely replace the implementation of a particular function:

void foo() {
 ...
}

This was an easy way one liner method to do that replacement, deleting the implementation of foo, and replacing it with the one that was found in the file ‘foo’

:,/^}/ !cat foo

file:line: delimited output for a single file

The grep -n command is very handy for producing file:line:content delimited output.  In particular, you can iterate over such output with vim -q.  When you want to do this for a single file, grep -n doesn’t include the filename, defeating a subsequent vim -q (since vim then doesn’t know what file to open).  Here’s an example

$ cat my_file_to_search
blah patternOfInterest hi
foo goo
patternOfInterest bye
blah patternOfInterest hi
blah patternOfInterest hi
foo goo
patternOfInterest bye
foo goo
patternOfInterest bye

$ grep -n patternOfInterest my_file_to_search | tee v
1:blah patternOfInterest hi
3:patternOfInterest bye
4:blah patternOfInterest hi
5:blah patternOfInterest hi
7:patternOfInterest bye
9:patternOfInterest bye

To get vim -q’able output, just include a second non-existent dummy file in the search

grep -n patternOfInterest my_file_to_search a_file_that_doesnt_exist | tee v
vim -q v

I usually use a very-short filename for the “does not exist file”, say, .u (which presumes I also don’t create little hidden files .u in my day-to-day work).

vim hacking: move set of lines into new file without buffer switching

May 30, 2014 perl and general scripting hackery , , , ,

 

Yesterday I was faced with a task where I wanted to repeatedly edit the first N lines of a file, then when they were in the form I wanted insert them into another file to be consumed by another script.  A braindead way to do this would be something like:

:0, d
:w
:e t
:$
:p
:0, d
:w
:e#

This is:

  • delete the lines from the beginning of the file to the current line.
  • save my file with these lines deleted
  • switch to my destination file to be overwritten and go to the end
  • paste in the new content
  • delete the old content
  • save this new destination file
  • return to my original file

If you only need to do this once or twice, it’s not that many keystrokes, and can be done really quickly.  However, it occurred to me that this can all be done with a one liner instead, cool enough to share:

:0,!cat > t

Here’s what happens:

  • Filter the lines from the beginning of the file to the current line through a script
  • This script has no stdout, so those lines are deleted from the current file.
  • That script uses cat to read from stdin and write the lines to a file, overwriting the destination file as desired

This is a pretty slick technique, but involves a few subtleties.  As it happens, my consumer of the moved lines was also a vim editing session, so I’m sure this was probably also possible with a split screen vim.  However, I like my screen real estate, and didn’t opt to go that route (and am not actually sure of the keystrokes required to do the same task in a split screen vim session).