Gem list -> gem install conversion one-liner

2011-06-04T21:51:00+02:00

Useful to sync gems across servers. Converts gem list --local output to a bunch of gem install commands. Rails related gems are removed from the initial list.

gem list --local | grep -v '\*' | grep -v ^$ | grep -v ^action | grep -v ^active | grep -v ^rails | grep -v ^rack | grep -v ^rake | sed 's|(|-v=|' | sed 's|)||' | sed 's|^\(.*\),|\1\n\1|' | sed 's|=.*[[:space:]]|=|' | sed 's|^|gem install |'

Why Linux is a pain?

2011-05-31T12:28:00+02:00

In fact, it isn’t. Let me explain.

When you see the next time a complaint like this, note what exactly that complaint is related to. I mean operating system. I am sure you’ll find out that it’s some sort of binary distributives like Centos or Ubuntu. But they are not exactly Linux! They are what they are: Linux-based operating systems distributable in binary form. You can’t refer to them as to Linux, because 1) Linux is a kernel, not an operation system, and 2) being a monolithic kernel, it’s supposed to be carefully configured and compiled natively for each system. And operating system (GNU), should be compiled against this kernel. And the whole toolchain, too.

You might say, well, why does it matter? Is there any difference?

Yes, there is, and it’s huge. Let me show you.

In fact, it isn’t. Let me explain.

You might say, well, why does it matter? Is there any difference?

Yes, there is, and it’s huge. Let me show you.

There are several aspects of why binary distros are kind of compromizing the idea of GNU/Linux: performance, security, manageability. Security part is pretty obvious: if you have an operating system that is not just binary compatible, but binary identical with millions of others, it is much easier to create a virus or rootkit infecting such systems. Creating virus for your unique system is just impossible: to create it, attacker needs access to your system. There is a chance of course that a virus will work on your particular system, and this chance is not very small, but attackers don’t want a chance, they rather distribute their product via certain binary distro related channels to be sure that it runs just fine on every system it gets into.

Performance is a long story. I’d like to get some proper benchmark results first, and I am going to get them. But this is for another article. In short, the more your hardware differs from the one binary distro packages are compiled on, and the more your usage scenario differs from some average scenario, the more performance difference you’ll get.

Manageability looks like the advantage of the binary approach. Pre-compiled software packages are easy to install (just download) and easy to manage via package management system. This is what I want to compare now: how much easier it is, if it is at all.

I am going to provide three detailed step-by-step algorithms for setting up a simple ROR website on a freshly installed systems: Linux From Scratch¹, Gentoo² and Ubuntu³.

So I am going to start from the point where I have a ROR website copied over to a new server that has a newly installed operating system. This is a common situation: you have your new server just booted up and running and you are going to move your website over to the new server. To shorten the article, I assume that database is already prepared or located on another server. Neither I am going to describe HTTP server related steps. All I need is to start the application. So the further steps basically are:

install Ruby
install Rubygems
install Rails gem
check if the application starts (I’ll be using script/console for this)
fix problems, if any

Pure GNU/Linux

Download the source code tarball of required version from ruby-lang.org:
wget ftp://ftp.ruby-lang.org/pub/xxxxxx

(if you don't know a certain software download site, Google is your friend, with opensource 
software it's never a problem to find the site and get the download link)

Unpack:

tar xf <tarball>

cd to directory and run:

./configure --help

this command shows all available options, I consider three of them needed:

./configure --prefix=/usr --enable-shared --enable-pthread

Now compile and install:

make
make install

Now I need Rubygems:

1234	Download it using the link provided on http://rubyonrails.org/downloadUnpack, cd to its directory and run:ruby setup.rb

And finally Rails of required version:

gem install rails -v <version>

Now I can switch to the directory where my website is located and run:

script/console

It says there are missing gems. It’s OK, there is a rake task for this:

rake gems:install

Completes succesfully. That’s all – now script/console loads OK.

Gentoo

With Gentoo, you never need to choose configure script options. Instead, you need to configure the whole system. You only need to do it once, though no problem to adjust any settings later and re-compile affected packages. Basically, you just read the description provided and enable or disable options according to your needs. For example, if you are never going to use X server, you add it to the system-wide configuration with the minus: -X. From now on no package will ever need X server or X-related libraries as its dependencies. This initial step is quite time-consuming, also it assumes that you are familiar with various software pieces and their relations. But the further management of properly configured system is just like a dream.

So here is what my task looks like on Gentoo:

Compiling and installing Ruby of required version:

emerge =ruby-<version>

(or just 'emerge ruby', if you want the latest stable version)

Now Rubygems:

emerge rubygems

The rest goes as easy as with pure GNU:

Rails first. It's available via Portage system, but I prefer to use Rubygems for this:

gem install rails -v <version>

Checking if it works:

script/console

Installing missing gems:

rake gems:install

Done.

Ubuntu

Installing Ruby and Rubygems is easy:

apt-get install ruby1.8 rubygems

Now Rails:

gem install rails -v <version>

Easy. Testing the result:

script/console
  irb not found
???

ruby --version
  ruby 1.8.7
irb
  irb: command not found

WTH? Irb is a Ruby shell, it’s the part of Ruby!

In such occasions I use aptitude – it’s an interface to Debian package management system. Inside there is a convenient search option. This way, probing various patterns, you can eventually find what package you need. In this case it’s pretty obvious: the needed package name is irb. But! There is also a package named ruby-full. It’s a “virtual” package made for convenience. You can install it and have everything relating to Ruby. But if you list all the packages it is going to install, you’ll probably be shocked: it’s 111 packages, including, for example, hicolor-icon-theme (Gnome desktop icons collection), several libavahi packages (it’s an automatic network configuration tool), libgl1-mesa-dri (Direct Rendering Infrastructure support for Mesa OpenGL), libxinerama1 (multiple displays support for X server), a lot of X rendering-related libraries including even libxcomposite1 (desktop visual effects support library). I understand that they are needed if you are going to use Ruby for desktop applications development, but using this virtual package to get Ruby properly installed on a web server doesn’t sound like a good idea.

I mention this package for a good reason though: this is a shortest way to find out if there are any other parts of Ruby missing. And yes, reading through those 111 packages I find several other important commands that are parts of Ruby initially, but apt-get install ruby doesn’t install them, so I install them now, in a single row:

apt-get install rake irb rdoc ri

Now I run script/console and it complains about missing gems, as expected.

Trying to install them I get:

123	rake gems:install rake aborted! no such file to load -- rspec

Ohh, it’s yet another part of Ruby that has not been installed. Searching by ‘rspec’ keyword I find it out:

apt-get install librspec-ruby1.8

Trying again:

rake gems:install
  rake aborted!
  no such file to load -- mime/types

# OK, found:

apt-get install libmime-types-ruby

and again:

rake gems:install
  rake aborted!
  no such file to load -- net/https

# found, too:

apt-get install libopenssl-ruby1.8

At last it starts installing gems:

123	gem install nokogiri ERROR: Failed to build gem native extension. extconf.rb:5:in `require': no such file to load -- mkmf (LoadError)

With Google help I find that mkmf is a part of ruby-dev package. OK, OK, I suspected I would need it.

apt-get install ruby-dev

gem install nokogiri
  ERROR: Failed to build gem native extension.
  checking for libxml/parser.h... no

If you think that the system has no libxml installed, you are not familiar enough with binary distros:

12	apt-get install libxml2 libxml2 is already the newest version.

What it really wants is:

apt-get install libxml2-dev

OK, next round:

123	gem install nokogiri ERROR: Failed to build gem native extension. checking for libxslt/xslt.h... no

but I already know what it wants:

1234	apt-get install libxslt-devgem install nokogiri success

Ooophhh!

Do you see now how you may come to the conclusion that Linux is a pain? All you need to come to this is:

to believe that those distros are Linux
to need something that the distro of your choice doesn’t include, or to want this installed your way.

Yes, I could install that ruby-full package and I would have everything I need, but with such approach I would get all systems I am maintaining growing in an uncontrollable manner, let alone unneeded services that might get installed as dependencies, that need to be removed later, otherwise they will be consuming resources and keeping open ports, and each open port is a potential security hole and so on and so on.

There exists another point I am argued with from time to time: Red Hat (Ubuntu, whatever) is a well-known and stable brand, and we trust it.

Well, yes, I can understand this. They are commercial companies and they care about their brand. But think about this: they rely on something, too. They rely on their upstreams: the developers of open source software. First of all, they rely on Linux and GNU. Linux and GNU are the only two brands in Linux world that you can trust absolutely. Otherwise you wouldn’t use Linux, would you? So what’s the problem using its pure form rather than something modified?

If you think I found a particularly hard case, try to find out how to install rmagick gem on Debian/Ubuntu properly :) This is a real pain.

If you think Debian/Ubuntu is bad, it’s not true. I prefer Debian-based distros over RedHat-based distros. On RedHat and its derivatives you just can’t find needed packages at all. Or can find packages, but their versions are too old to be usable. So you end up with getting packages from unofficial repositories or even worse – you have to compile from sources. The latter is a bad idea, because it’s very probable that some package (similar to abovementioned ruby-full) will want to install the software you have already installed from source as a dependency and you know what? – it will install it! Package management system doesn’t know about anything you install without its help. So from its point of view everything will be alright, but in fact the system will be messed up. Of course it’s not this stupid, there are proper ways: you will need to learn how to use SRPM and how to create your own packages. But tell me one thing: does it sound easier than using pure GNU/Linux or Gentoo at least?

¹ LFS is not a distribution, it’s a book of instructions about compiling pure GNU/Linux OS from scratch.

² Gentoo is more like distribution, but it doesn’t provide binary packages except those needed as a start point for compiling the whole system. It provides compilation scripts bound by full-featured management system named Portage.

³ Ubuntu is a very popular Debian-based binary distribution. Its goal is desktop systems primarily, but it’s being widely used as a server system too. As a Debian derivative, it is a bit more flexible compared to most RPM-based distros.

Passenger memory stats decolorifying

2011-01-18T09:44:00+01:00

Just to save for future use..

passenger-memory-stats | sed 's|[^[:print:]]||g' | sed 's|\[44m\[1m\[33m||g' | sed 's|\[37m||g' | sed 's|\[0m||g'

I wonder why at all they use colors for the output. Since it’s the only way for Passenger based setups to get some info about memory usage, its output should have at least plain text option to be used in other scripts.

Git plugin support doesn't work in Rails 2.3.4

2009-10-11T16:58:00+02:00

It’s a 2.3 bug still not fixed in 2.3.4 gems according to this thread

On my new server with Rails 2.3.4 installed it looks like this:

I run script/plugin install git://github.com/whatever/plugin.git and then just nothing happens.

Recipe from above mentioned thread works for me, so I have to run instead:

script/plugin install http://github.com/whatever/plugin.git/

Note slash at the end!

RJS and PrototypeHelper methods

2009-08-19T11:01:00+02:00

Quite a common mistake.

Never use :update option of PrototypeHelper methods (like link_to_remote) together with RJS templates. Simply remove :update and RJS will care about it. Another option is to avoid using RJS if possible.

Multiple conditions for find (ActiveRecord::Base)

2009-08-10T21:36:00+02:00

Had to spend some time to figure out:

Person.find :all, :conditions => [ "user_name = ? AND surname = ? AND address = ?", user_name, surname, address]

Reticularium - RoR