Reticularium - Linux

Unixtime to date and back conversion

2011-11-22T00:04:00+01:00

Cause of many mistakes. Applies anywhere, I am going to provide a shell examples.

So this is what can twist your brain:

1234	$ date +"%s"1321920962$ date -d "1970/01/01 +1321920962 sec" +"%s"1321917362

Unixtime (POSIX time) is a number of seconds passed since the beginning of year 1970. So in the example above I get this number with date +"%s", then add this number to the epoch (1970/01/01) and get a different number!

The reason of this is very simple:

1234	$ date -d "1970/01/01" +"%s"-3600$ date -u -d "1970/01/01" +"%s"0

Unixtime is always UTC

It is a well-known fact, but it’s so easy to miss:

$ date ; date -u
Tue Nov 22 01:31:42 CET 2011
Tue Nov 22 00:31:42 UTC 2011
$ date +"%s" ; date -u +"%s"
1321921903
1321921903

That is, for example, you might have local and UTC datetime variables defined in your code, perfectly distinguishable. And then at some point you decide to just format it with +"%s" to unixtime, and they become equal :)

BUT:

$ dt='Tue Nov 22 01:31:42'
$ date -d "$dt" +"%s"
1321921902
$ date -u -d "$dt" +"%s"
1321925502

Note that the second number is bigger. Seems wrong because my local CET time is one hour ahead. But it is absolutely correct. It is bigger because in the first case to be converted to unixtime this variable must be converted to UTC first (minus 1 hour), so it becomes ‘Tue Nov 22 00:31:42 UTC 2011’ while in the second case it is considered as ‘Tue Nov 22 01:31:42 UTC 2011’. In other words, $dt is a string, not a date. This is why the -u option converts it to date like this:

1234	$ date -d "$dt"Tue Nov 22 01:31:42 CET 2011$ date -u -d "$dt"Tue Nov 22 01:31:42 UTC 2011

String can’t be converted to unixtime, so it is converted to date first, then the result is converted to UTC if needed and this result is then converted to unixtime.

M-Script's new home

2011-07-31T11:16:00+02:00

M-Script project has been moved to its new home: http://m-script.org

It’s been under heavy development lately: old scripts have been renewed and enhanced, folders reorganized, files moved to proper locations, default monitors made more informative.

It’s also been published on Github

Gem list -> gem install conversion one-liner

2011-06-04T21:51:00+02:00

Useful to sync gems across servers. Converts gem list --local output to a bunch of gem install commands. Rails related gems are removed from the initial list.

gem list --local | grep -v '\*' | grep -v ^$ | grep -v ^action | grep -v ^active | grep -v ^rails | grep -v ^rack | grep -v ^rake | sed 's|(|-v=|' | sed 's|)||' | sed 's|^\(.*\),|\1\n\1|' | sed 's|=.*[[:space:]]|=|' | sed 's|^|gem install |'

Why Linux is a pain?

2011-05-31T12:28:00+02:00

In fact, it isn’t. Let me explain.

When you see the next time a complaint like this, note what exactly that complaint is related to. I mean operating system. I am sure you’ll find out that it’s some sort of binary distributives like Centos or Ubuntu. But they are not exactly Linux! They are what they are: Linux-based operating systems distributable in binary form. You can’t refer to them as to Linux, because 1) Linux is a kernel, not an operation system, and 2) being a monolithic kernel, it’s supposed to be carefully configured and compiled natively for each system. And operating system (GNU), should be compiled against this kernel. And the whole toolchain, too.

You might say, well, why does it matter? Is there any difference?

Yes, there is, and it’s huge. Let me show you.

In fact, it isn’t. Let me explain.

You might say, well, why does it matter? Is there any difference?

Yes, there is, and it’s huge. Let me show you.

There are several aspects of why binary distros are kind of compromizing the idea of GNU/Linux: performance, security, manageability. Security part is pretty obvious: if you have an operating system that is not just binary compatible, but binary identical with millions of others, it is much easier to create a virus or rootkit infecting such systems. Creating virus for your unique system is just impossible: to create it, attacker needs access to your system. There is a chance of course that a virus will work on your particular system, and this chance is not very small, but attackers don’t want a chance, they rather distribute their product via certain binary distro related channels to be sure that it runs just fine on every system it gets into.

Performance is a long story. I’d like to get some proper benchmark results first, and I am going to get them. But this is for another article. In short, the more your hardware differs from the one binary distro packages are compiled on, and the more your usage scenario differs from some average scenario, the more performance difference you’ll get.

Manageability looks like the advantage of the binary approach. Pre-compiled software packages are easy to install (just download) and easy to manage via package management system. This is what I want to compare now: how much easier it is, if it is at all.

I am going to provide three detailed step-by-step algorithms for setting up a simple ROR website on a freshly installed systems: Linux From Scratch¹, Gentoo² and Ubuntu³.

So I am going to start from the point where I have a ROR website copied over to a new server that has a newly installed operating system. This is a common situation: you have your new server just booted up and running and you are going to move your website over to the new server. To shorten the article, I assume that database is already prepared or located on another server. Neither I am going to describe HTTP server related steps. All I need is to start the application. So the further steps basically are:

install Ruby
install Rubygems
install Rails gem
check if the application starts (I’ll be using script/console for this)
fix problems, if any

Pure GNU/Linux

Download the source code tarball of required version from ruby-lang.org:
wget ftp://ftp.ruby-lang.org/pub/xxxxxx

(if you don't know a certain software download site, Google is your friend, with opensource 
software it's never a problem to find the site and get the download link)

Unpack:

tar xf <tarball>

cd to directory and run:

./configure --help

this command shows all available options, I consider three of them needed:

./configure --prefix=/usr --enable-shared --enable-pthread

Now compile and install:

make
make install

Now I need Rubygems:

1234	Download it using the link provided on http://rubyonrails.org/downloadUnpack, cd to its directory and run:ruby setup.rb

And finally Rails of required version:

gem install rails -v <version>

Now I can switch to the directory where my website is located and run:

script/console

It says there are missing gems. It’s OK, there is a rake task for this:

rake gems:install

Completes succesfully. That’s all – now script/console loads OK.

Gentoo

With Gentoo, you never need to choose configure script options. Instead, you need to configure the whole system. You only need to do it once, though no problem to adjust any settings later and re-compile affected packages. Basically, you just read the description provided and enable or disable options according to your needs. For example, if you are never going to use X server, you add it to the system-wide configuration with the minus: -X. From now on no package will ever need X server or X-related libraries as its dependencies. This initial step is quite time-consuming, also it assumes that you are familiar with various software pieces and their relations. But the further management of properly configured system is just like a dream.

So here is what my task looks like on Gentoo:

Compiling and installing Ruby of required version:

emerge =ruby-<version>

(or just 'emerge ruby', if you want the latest stable version)

Now Rubygems:

emerge rubygems

The rest goes as easy as with pure GNU:

Rails first. It's available via Portage system, but I prefer to use Rubygems for this:

gem install rails -v <version>

Checking if it works:

script/console

Installing missing gems:

rake gems:install

Done.

Ubuntu

Installing Ruby and Rubygems is easy:

apt-get install ruby1.8 rubygems

Now Rails:

gem install rails -v <version>

Easy. Testing the result:

script/console
  irb not found
???

ruby --version
  ruby 1.8.7
irb
  irb: command not found

WTH? Irb is a Ruby shell, it’s the part of Ruby!

In such occasions I use aptitude – it’s an interface to Debian package management system. Inside there is a convenient search option. This way, probing various patterns, you can eventually find what package you need. In this case it’s pretty obvious: the needed package name is irb. But! There is also a package named ruby-full. It’s a “virtual” package made for convenience. You can install it and have everything relating to Ruby. But if you list all the packages it is going to install, you’ll probably be shocked: it’s 111 packages, including, for example, hicolor-icon-theme (Gnome desktop icons collection), several libavahi packages (it’s an automatic network configuration tool), libgl1-mesa-dri (Direct Rendering Infrastructure support for Mesa OpenGL), libxinerama1 (multiple displays support for X server), a lot of X rendering-related libraries including even libxcomposite1 (desktop visual effects support library). I understand that they are needed if you are going to use Ruby for desktop applications development, but using this virtual package to get Ruby properly installed on a web server doesn’t sound like a good idea.

I mention this package for a good reason though: this is a shortest way to find out if there are any other parts of Ruby missing. And yes, reading through those 111 packages I find several other important commands that are parts of Ruby initially, but apt-get install ruby doesn’t install them, so I install them now, in a single row:

apt-get install rake irb rdoc ri

Now I run script/console and it complains about missing gems, as expected.

Trying to install them I get:

123	rake gems:install rake aborted! no such file to load -- rspec

Ohh, it’s yet another part of Ruby that has not been installed. Searching by ‘rspec’ keyword I find it out:

apt-get install librspec-ruby1.8

Trying again:

rake gems:install
  rake aborted!
  no such file to load -- mime/types

# OK, found:

apt-get install libmime-types-ruby

and again:

rake gems:install
  rake aborted!
  no such file to load -- net/https

# found, too:

apt-get install libopenssl-ruby1.8

At last it starts installing gems:

123	gem install nokogiri ERROR: Failed to build gem native extension. extconf.rb:5:in `require': no such file to load -- mkmf (LoadError)

With Google help I find that mkmf is a part of ruby-dev package. OK, OK, I suspected I would need it.

apt-get install ruby-dev

gem install nokogiri
  ERROR: Failed to build gem native extension.
  checking for libxml/parser.h... no

If you think that the system has no libxml installed, you are not familiar enough with binary distros:

12	apt-get install libxml2 libxml2 is already the newest version.

What it really wants is:

apt-get install libxml2-dev

OK, next round:

123	gem install nokogiri ERROR: Failed to build gem native extension. checking for libxslt/xslt.h... no

but I already know what it wants:

1234	apt-get install libxslt-devgem install nokogiri success

Ooophhh!

Do you see now how you may come to the conclusion that Linux is a pain? All you need to come to this is:

to believe that those distros are Linux
to need something that the distro of your choice doesn’t include, or to want this installed your way.

Yes, I could install that ruby-full package and I would have everything I need, but with such approach I would get all systems I am maintaining growing in an uncontrollable manner, let alone unneeded services that might get installed as dependencies, that need to be removed later, otherwise they will be consuming resources and keeping open ports, and each open port is a potential security hole and so on and so on.

There exists another point I am argued with from time to time: Red Hat (Ubuntu, whatever) is a well-known and stable brand, and we trust it.

Well, yes, I can understand this. They are commercial companies and they care about their brand. But think about this: they rely on something, too. They rely on their upstreams: the developers of open source software. First of all, they rely on Linux and GNU. Linux and GNU are the only two brands in Linux world that you can trust absolutely. Otherwise you wouldn’t use Linux, would you? So what’s the problem using its pure form rather than something modified?

If you think I found a particularly hard case, try to find out how to install rmagick gem on Debian/Ubuntu properly :) This is a real pain.

If you think Debian/Ubuntu is bad, it’s not true. I prefer Debian-based distros over RedHat-based distros. On RedHat and its derivatives you just can’t find needed packages at all. Or can find packages, but their versions are too old to be usable. So you end up with getting packages from unofficial repositories or even worse – you have to compile from sources. The latter is a bad idea, because it’s very probable that some package (similar to abovementioned ruby-full) will want to install the software you have already installed from source as a dependency and you know what? – it will install it! Package management system doesn’t know about anything you install without its help. So from its point of view everything will be alright, but in fact the system will be messed up. Of course it’s not this stupid, there are proper ways: you will need to learn how to use SRPM and how to create your own packages. But tell me one thing: does it sound easier than using pure GNU/Linux or Gentoo at least?

¹ LFS is not a distribution, it’s a book of instructions about compiling pure GNU/Linux OS from scratch.

² Gentoo is more like distribution, but it doesn’t provide binary packages except those needed as a start point for compiling the whole system. It provides compilation scripts bound by full-featured management system named Portage.

³ Ubuntu is a very popular Debian-based binary distribution. Its goal is desktop systems primarily, but it’s being widely used as a server system too. As a Debian derivative, it is a bit more flexible compared to most RPM-based distros.

Unmounting rbind mounts

2011-05-31T07:58:00+02:00

Command mount --rbind mounts recursively, which is handy, but umount has no recursive capabilities, so several umount commands are needed to unmount. The easiest way to do this is to look into /proc/mounts :

cat /proc/mounts

Even better:

cat /proc/mounts | awk '{print $2}'

Take a look and if you can grep needed mountpoints out of it, just add |grep <something> | sort -r | xargs umount to the above line.

What time is it in.. ?

2011-04-24T15:38:00+02:00

I am used to ask Linux utility date about everything related to date/time, like date -d "next friday" or date -d "+245 min", but it doesn’t answer one question I have pretty often: what time is it now in certain timezone?

So I had to create a simple script for this:

#!/bin/bash

localtime=`date +"%x %X"`
zonediff=`date -d "$1" +"%H:%M"`
hours="`echo $zonediff | cut -d':' -f1` hours"
minutes="`echo $zonediff | cut -d':' -f2` minutes"
date -d "-$hours -$minutes" +"%x %X"

Zone here is represented by alphabetic abbreviation, that is if you named this script e.g. “timein”, you use it like this:

timein EDT

Gentoo Coreutils/touch problem hack

2011-01-31T00:28:00+01:00

Old-time problem comes back: coreutils doesn’t roll back to calls supported by your older kernel. This results in errors like:

Failed 'touch .unpacked': No such file or directory

or:

Failed 'touch .unpacked': Bad address

Funny little helpful hack taken here

1234	mv /bin/touch /bin/oldtouchecho '#!/bin/sh' > /bin/touchecho 'echo -n >> "$1"' >> /bin/touchchmod +x /bin/touch

Don’t forget to restore it back later.

Mounting a logical volume (LVM)

2010-09-26T19:50:00+02:00

Taken from the Fedora forum

..........

#modprobe dm-mod
#vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "VolGroup00" using metadata type lvm2
# vgchange -a y VolGroup00
# lvdisplay
  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVol01
  VG Name                VolGroup00
. . .
  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVol00
  VG Name                VolGroup00
# mkdir /mnt/VolGroup00/01
# mount /dev/VolGroup00/LogVol01 /mnt/VolGroup00/01

Reference the man pages for lvm, pvdisplay, vgdisplay, lvcreate, lvdisplay, lvextend

..........

As you can see, you need two things: dm-mod kernel module and LVM2 package (apt-get install lvm2 on Debian, emerge lvm2 on Gentoo and so on)

My personal note: LVM provides several useful features but has one huge drawback. If your disk or file system crashes, you won’t be able to access your LVM partitions using any liveCD. At least I don’t know any that would have both dm-mod module and LVM2 package.

CLI scripts templates

2010-05-22T21:18:00+02:00

Well, quite self-explanatory. Only Bash and Ruby. Options “folder” and “name” are examples, replace them with what you need… Shared in hope this can help someone.

Well, quite self-explanatory. Only Bash and Ruby. Options “folder” and “file” are examples, replace them with what you need… Shared in hope this can help someone.

Bash template:

#!/usr/bin/env bash

possible_options="folder, file"
necessary_options="file"

# The 3 lines below are needed if the script needs other files located in the same folder.
# This finds the real folder even if the script is symlinked.
[ -h $0 ] && xcommand=`readlink $0` || xcommand=$0
rcommand=${xcommand##*/}   # The real script
rpath=${xcommand%/*}   # The real folder
# Replace the message below with appropriate usage info"
[ "X$*" == "X" ] && echo "Can't run without options. Possible options are: ${possible_options}" && exit 1

IFS1=$IFS
IFS='--'
for s_option in "${@}"
do
  found=0
  case ${s_option} in
  --*=*)
    s_optname=`expr "X$s_option" : 'X[^-]*-*\([^=]*\)'`  
    s_optarg=`expr "X$s_option" : 'X[^=]*=\(.*\)'` 
    ;;
  --*)
    s_optname=`expr "X$s_option" : 'X[^-]*-*\([^=]*\)'`    
    s_optarg='yes' 
    ;;
  *=*)
    echo "Wrong syntax: options must start with a double dash"
    exit 1
    ;;
  *)
    s_param=${s_option}
    s_optname=''
    s_optarg=''
    ;;
  esac

  for option in `echo $possible_options | sed 's/ /--/g'`; do 
    [ -n "$s_optname" ] && [ "X$s_optname" == "X$option" ] && eval "$option=\"${s_optarg}\"" && found=1
  done
  [ "X$s_option" == "X$s_param" ] && found=1
  if [[ found -ne 1 ]]; then 
    echo "Unknown option: $s_optname"
    exit 1
  fi
done
found=0

for option in `echo $necessary_options | sed 's/,//g'`; do
  [ "X$(eval echo \$$option)" == "X" ] && missing_options="${missing_options}, --${option}" && found=1
done
if [[ found -eq 1 ]]; then
  missing_options=${missing_options#*,}
  echo "Necessary options: ${missing_options} not found"
  exit 1
fi
IFS=$IFS1


### Here goes your code
#echo "option folder: ${folder}"
#echo "option file: ${file}"
#echo "parameter: ${s_param}"

Ruby template:

#!/usr/bin/env ruby

possible_options = "folder, file"
necessary_options = "file"
options = Hash.new
possible_options.split(',').each { |po| options[po.strip] = '' }
File.symlink?(__FILE__) ? xcommand = File.readlink(__FILE__) : xcommand = File.expand_path(__FILE__)
rcommand = File.basename(xcommand)
rpath = File.dirname(xcommand)
begin
  if ARGV.empty?
    # Replace the message below with appropriate usage info"
    raise(ArgumentError, "Can't run without options. Possible options are: #{possible_options}")
  else
    for s_option in ARGV
      ((s_option =~ /^--/).nil? and not (s_option =~ /=/).nil?) and raise(ArgumentError, "Wrong syntax: options must start with a double dash")
      ((s_option =~ /^--/).nil? and (s_option =~ /=/).nil?) and s_param = s_option
      (s_option =~ /=/).nil? ? s_optarg = "yes" : s_optarg = s_option.split("=")[1]
      s_optname = s_option.split('=')[0].gsub('--','')
      if options.include?(s_optname)
        options[s_optname] = s_optarg
      else
        if not s_param == s_optname
          raise(ArgumentError, "Unknown option: #{s_optname}")
          exit 1
        end
      end
    end
    necessary_options.split(',').each do |o|
      if options[o.strip].empty?
        raise(ArgumentError, "Option --#{o.strip} is necessary")
        exit 1
      end
    end
  end

### Here goes your code
### note that command-line options are in hash! Address them as e.g. options["file"]
#puts "option folder: #{options["folder"]}"
#puts "option file: #{options["file"]}"
#puts "parameter: #{s_param}"

rescue Exception => e
#  puts e.backtrace
  puts e.message
  exit 1
end

I’d only recommend to expand usage/help information.

Update: you can use method path instead of expand_path if it’s supported by your Ruby version.

Update: modified shell template, allowing arguments with spaces, but without non-option parameters:

#!/usr/bin/env bash

possible_options="folder, file"
necessary_options="file"

# The 3 lines below are needed if the script needs other files located in the same folder.
# This finds the real folder even if the script is symlinked.
[ -h $0 ] && xcommand=`readlink $0` || xcommand=$0
rcommand=${xcommand##*/}   # The real script
rpath=${xcommand%/*}   # The real folder
# Replace the message below with appropriate usage info"
[ "X$*" == "X" ] && echo "Can't run without options. Possible options are: ${possible_options}" && exit 1

ARGV=`echo ${@} | sed 's/^--//' | sed 's/ --/|/g'`

IFS1=$IFS
IFS='|'
for s_option in $ARGV
do
  s_optname=${s_option%%=*}
  s_optarg=${s_option##*=}
  [ "X$s_optarg" == "X$s_optname" ] && s_optarg="yes"
  for option in `echo $possible_options | sed 's/,//g'`; do 
    [ "X$s_optname" == "X$option" ] && eval "$s_optname=\"$s_optarg\"" && found=1
  done
  if [[ found -ne 1 ]]; then 
    echo "Unknown option: $s_optname"
    exit 1
  fi
done
IFS=$IFS1

found=0
for option in `echo $necessary_options | sed 's/ /|/g'`; do
  [ "X$(eval echo \$$option)" == "X" ] && missing_options="${missing_options}, --${option}" && found=1
done
if [[ found -eq 1 ]]; then
  missing_options=${missing_options#*,}
  echo "Necessary options: ${missing_options} not found"
  exit 1
fi

### Here goes your code
#echo "option folder: ${folder}"
#echo "option file: ${file}"
#echo "parameter: ${s_param}"

Vim auto-comment feature

2010-05-03T09:11:00+02:00

The subject is extremely annoying. I am not that lazy to add a comment symbol myself. This would have been a good idea though, if it wasn’t sometimes needed to insert blocks of code via clipboard. With auto-comment on, if I insert

some line
# some comment
some line

I get

some line
# some comment
# some line

because there is a CR at the end of commented line and the auto-comment function adds the # symbol to the next line automatically.

The real problem begins when you try to insert many lines with some of them commented. You end up with each line after the first comment commented, the next comment commented twice so the next lines all commented twice and so on.

The solution:

Add this to your vimrc file:

au FileType * setl fo-=cro

where:

au = autocmd
FileType * – any file type
setl = setlocal
fo = formatoptions
-=cro disables c, r and o options ( see documentation )

Bash and whitespaces in filenames

2010-04-24T00:41:00+02:00

This is a really annoying problem we meet pretty often. You can meet it even if you don’t use Bash. For example, you upload an image named “my lovely cat.jpg” to your website and find no image there. Or it has no thumbnail. This may happen because you are using some program for making thumbnails that is being called from, say, PHP via backtick operator (that is, using unix shell) and that program interprets the file name passed to it as three different command line options: my, lovely and cat.jpg.

Well, cases like this are simple to avoid, just add double quotes, like this: yourthumbnailer "$filename", and your “my lovely bash.jpg” will be processed. Err, I meant “my lovely cat.jpg”, not going to correct it, it’s funny :)

But there may be more complex issues. And fortunately there are plenty of methods to deal with those issues.

I am far from an idea to describe all possible methods, so just a few examples, hopefully useful. Anyways, all methods are based on same ideas, and I proudly know the three of them:

null character termination built into some utilities (e.g. find) to avoid special treatment of all special characters including whitespaces
using double quotes where applicable
using IFS variable

Here’s the examples.

Idea #1

$ mkdir "test 1"
$ touch test\ 1/some\ file   # created "test 1" folder with file "some file" inside
# here is the problem:
$ find -name "test*" | xargs ls
ls: cannot access ./test: No such file or directory
ls: cannot access 1: No such file or directory
# checking if the find part works:
$ find -name "test*"
./test 1   # works perfectly
# trying the -0 option for xargs that enables null character termination:
$ find -name "test*" | xargs -0 ls
ls: cannot access ./test 1    # doesn't work, but note the difference:
: No such file or directory   # now it considers "test 1" a single folder
# what is needed is null character termination on both ends of the pipe:
$ find -name "test*" -print0 | xargs -0 ls
some file   # works perfectly now
# using find's built-in -exec feature is often preferrable:
$ find -name "test*" -exec ls '{}' \;
some file   # works just fine without null character termination

Idea #2

$ for dir in test*; do echo $dir; ls $dir; done
test 1    # note that echo $dir works, so it's assigned correctly
ls: cannot access test: No such file or directory   # ls $dir doesn't work
ls: cannot access 1: No such file or directory      # double quotes are needed
$ for dir in test*; do echo "$dir"; ls "$dir"; done
test 1
some file

Idea #3

IFS is a system variable, it stands for “Internal Field Separator”. “I Feel Stupid” is a good match, too, because the idea to use IFS often comes after a long time spent for escaping and quoting and many lines of unneeded code written. By default, IFS is set to a tab, newline and space bound together so any of these works as a field separator.

123	$ (IFS='';for dir in test*; do echo $dir; ls $dir; done) # parenthesis neededtest 1 # to run this in a subshell leavingsome file # the original IFS untouched

More complex example:

$ mypath="test 1/test 2/test 3"  # some recursive folders, each needs to
$ for folder in $mypath; do echo $folder; done         # be processed
test                                           # I am simulating processing
1/test                                       # these folders with the echo
2/test                                       # command. Apparently the result
3                                              # is not what we wanted
$ IFS1=$IFS; IFS='/'; for folder in $mypath; do echo $folder; done; IFS=$IFS1
test 1
test 2
test 3

I used IFS reassigning here instead of subshell, this way you can use it in scripts reassigning at the beginning and assigning back at the end or for some part of code where needed.

Strings splitting in Bash

2010-04-08T21:22:00+02:00

It’s not easy to remember, so here’s the reference.

var=aaa.bbb.ccc

%%%### .* *.

`${var%.*} => aaa.bbb`	`${var%%.*} => aaa`	`${var#.*} => aaa.bbb.ccc` (doesn’t work)	`${var##.*} => aaa.bbb.ccc` (doesn’t work)
`${var%*.} => aaa.bbb.ccc` (doesn’t work)	`${var%%*.} => aaa.bbb.ccc` (doesn’t work)	`${var#*.} => bbb.ccc`	`${var##*.} => ccc`

It’s not easy to remember, so here’s the reference.

% means “right”

# means “left”

(look at the keyboard if you forgot, these signs relative positions can remind their meaning)

single sign means “the first”, double sign means “the last”.

So, a single % means “the first starting from right”, a double ## means “the last starting from left”.

.* means “everything on the right from the dot”. Instead of the dot any other symbol can be used. Nothing can be used instead of the asterisk symbol it seems, at least I couldn’t find anything.

Similarly, *. means “everything on the left form the dot”

Let’s assign var=aaa.bbb.ccc

There are 8 possible combinations, but not all of them work:

%%%### .* *.

`${var%.*} => aaa.bbb`	`${var%%.*} => aaa`	`${var#.*} => aaa.bbb.ccc` (doesn’t work)	`${var##.*} => aaa.bbb.ccc` (doesn’t work)
`${var%*.} => aaa.bbb.ccc` (doesn’t work)	`${var%%*.} => aaa.bbb.ccc` (doesn’t work)	`${var#*.} => bbb.ccc`	`${var##*.} => ccc`

Here’s the example.

The code below cuts off sub-domains and returns domain name (2nd level domain), e.g. abc.def.example.com -> example.com

domain2l=`a12=${domain%.*} ; a3=${domain##*.} ; a2=${a12##*.} ; echo "$a2.$a3"`

where domain is any domain name with any number of sub-domains, and domain2l is a resulting 2nd level domain.

find vs ls

2010-02-26T10:47:00+01:00

I was doing a simple test of pwgen (handy utility to generate passwords) and since this test was creating a lot of files in a folder I decided to combine it with a test of ls speed vs find speed:

homemain ~ # mkdir test
homemain ~ # for ((i=1; i<=300; i++)); do for pw in `pwgen -C`; do touch test/${pw}; done; done
homemain ~ # time (ls test|wc -l)
47685

real  0m0.152s
user  0m0.148s
sys  0m0.008s
homemain ~ # time (find test -type f|wc -l)
47685

real  0m0.072s
user  0m0.024s
sys  0m0.048s

Well, find is faster in mere files listing. On folders with files having more complicated names and differing in size and other attributes the difference is much more significant.

I repeated the time measurement part several times to check if any caching involved, but found no difference.

Ahh, about pwgen: there should be 48000 files. So 48000 – 47685 = 315 filenames appeared to be not unique. Not that bad given that it is a default simple passwords that are easy to remember (e.g. doo5maiJ) without additional symbols etc. The utility has options to make password more secure, I didn’t test it. This pwgen test didn’t have any practical meaning, just curiosity :)

The ‘find vs ls’ test has much more practical meaning though. The result means that in scripts it is much better to use find rather than ls. The ‘find’ utility at the first glance is something more complicated than a “simple” ls. But ls checks and outputs more information about each file and therefore is slower.

Another practical outcome is a simple wrapper equivalent to ‘ls -l’. I found it useful to read e.g. mail folders containing a huge amount of messages:

12	#!/bin/shfind $1 -maxdepth 1 -ls

or copy and paste my command sequence:

cat > /usr/local/bin/fls << "EOF"
#!/bin/sh
find $1 -maxdepth 1 -ls
EOF
chmod 755 /usr/local/bin/fls

Doing a test:

homemain ~ # time (fls test|wc -l)
47686

real  0m0.218s
user  0m0.140s
sys  0m0.076s
homemain ~ # time (ls -l test|wc -l)
47686

real  0m0.366s
user  0m0.228s
sys  0m0.136s

Enabling Gnome composite extension

2010-02-07T12:15:00+01:00

For some reason there is no obvious way to enable it (Gnome 2.26.3)

Open GConf editor: Applications -> System Tools -> Configuration Editor
apps -> metacity -> general -> compositing manager (check box there)

That’s all. Enjoy numerous visual effects like shadows, real transparency (underlying windows become visible) etc.

MySQL export to CSV

2010-02-02T22:12:00+01:00

Pretty easy:

mysql -u<username> -p<password> <dbname> -B -e "select * from <tablename>" | sed 's/\t/","/g;s/^/"/;s/$/"/;s/\n//g' > <filename.csv>