Archive for February, 2013

wget, busybox, ipv6…

Thursday, February 28th, 2013

In our work environment, we use pxe and preseed to roll out Ubuntu 12.04 installations. In the late command, we even splat puppet standalone and a subversion repository on it. Works nicely.

Last week however, this tested routine stopped working in a very early stage – when wget in busybox tries to figure out which release of Ubuntu to pull from the mirror server.

The actual code line was: wget -q ftp://somehost.domain.local/file -O – | grep -E ‘^(Suite|Codename):’
This line didn’t return anything, so the installation got stuck. On my pc, the same line returned something useful.

In the end, the problem was that ipv4 and ipv6 were configured automatically and wget in busybox prefers ipv6 over ipv4 but if wget fails, it does not try again with ipv4. And since it comes with busybox, you can’t force a protocol family (see ‘man wget’ for further infos).

The solution was to supply another kernel parament in the pxe configuration: ipv6.disable=1

There are some users who report that this line does not work for them. Maybe they got the spelling slightly wrong or they put it at the wrong place but with this additional parameter, our setup is working again.

The sad thing about this is that this problem was already bug-reported in 2007 but the status was put to ‘wontfix’ and closed. Way to go 🙁

And I only wasted about 6 hours of working hours for that crap.

 

Pictures to movie: A simple example of avconv

Sunday, February 17th, 2013

In 2011, I travelled from 豊岡 (Toyooka) to 京都 (Kyoto) by train. Every couple of seconds I took a picture with my camera. Since then, I wanted to merge all these pictures into a movie, if possible with some music.

I tried the better known video editors such as OpenShot Video Editor, Pitivi Video Editor, Blender and such but never really got far. Either pictures couldn’t be mass imported or they couldn’t be distributed evenly etc. etc.

Finally, a friend mentioned ffmpeg last week so I tried it again. Unfortunately, ffmpeg seems to be deprecated but they recommend to use avconv instead which comes with “libav-tools”

Format converting is an art of its own, but if you just want the basics, all you need is the following command:

avconv -f image2 -r 3 -i ./%04d.JPG -i soundfile.wav -c copy -crf 20 output.avi

Run this from the folder where your *.JPG pictures are and you will get a movie “output.avi” including a soundtrack from soundfile.wav. “-r 3” means 3 frames per second, so play around with this value if you a ‘faster’ or ‘slower’ movie.
Something else to keep in mind: The pictures must be named in numerical order: 0001.JPG, 0002.JPG etc.
If you don’t have your pictures named like that, try this command (all on 1 line in the terminal):
i=0; for f in *.JPG; do ln -s $f $(printf “%04d.JPG” $i); i=$((i+1)); done
This will create symlinks for all .JPG in your folder in the order that `ln -s` would output them.

This is the result: The final video

As usual, a couple of simple examples in the man page would have been helpful.

Installing Galaxy on CentOS 6.3 with an mysql db and running it as a non-root user

Friday, February 8th, 2013

There’s a biomedical reaseach software called Galaxy. I didn’t know that either 😉
The installation is easy but it uses a sqlite ‘db’ and must be started by whoever wants to use it. In a production environment, this is not convenient and does not scale nicely. To be fair, the makers provide infos on how to run it in a production environment.

Here is one such installation in details. maybe this helps you.
-OS: CentOS 6.3
-DB: mysql
-Galaxy is run by a non-root user
-Galaxy starts at system boot

Lines starting with # must be run as root, some lines are comments so you can’t just paste line by line in your shell. Make sure you understand what you do (the line breaks make it a bit hard to read though, sorry)

After the installation, open firefox. To use galaxy, visit localhost:8080

**************************

===========================================================
= Installation of Galaxy with a local mysql DB on CentOS6 =
===========================================================

mysql
=====

# yum install mysql-server
# yum install mysql
# yum install mysql-devel

# service mysqld start

# /usr/bin/mysql_secure_installation

Set root password? [Y/n] Y
root pwd: <pwd>

Remove anonymous users? [Y/n] Y

Disallow root login remotely? [Y/n] Y

Remove test database and access to it? [Y/n] Y

Reload privilege tables now? [Y/n] Y

(http://wiki.galaxyproject.org/Admin/Get%20Galaxy)

(sets mysqld to start on reboot)
# chkconfig mysqld on

add another db user
——————-

/usr/bin/mysql -u root -p (enter pwd)

mysql> INSERT INTO mysql.user (User,Host,Password) VALUES(‘galaxy’,’localhost’,PASSWORD(‘<pwd>’));
mysql> FLUSH PRIVILEGES;

create a galaxy db
——————

mysql> CREATE DATABASE galadb;

grant user ‘galaxy’ all permissions on db ‘galadb’
————————————————–

mysql> GRANT ALL PRIVILEGES ON galadb.* to galaxy@localhost;
mysql> FLUSH PRIVILEGES;
mysql> quit

mercurial
=========

# yum install mercurial

galaxy installation
===================

# cd /usr/local
# mkdir galaxy
# cd galaxy/
# hg clone https://bitbucket.org/galaxy/galaxy-dist/

# sh galaxy-dist/run.sh

–> starts a local galaxy instance, can be opened in a browser with localhost:8080
^C –> quits

change settings for production server
=====================================

(http://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer)

disable developer settings
————————–

cd /usr/local/galaxy/galaxy-dist/
# cp universe_wsgi.ini universe_wsgi.ini.orig
# vim /usr/local/galaxy/galaxy-dist/universe_wsgi.ini
(line 370) debug = True –> debug = False
(line 383) use_interactive = True –>  use_interactive = False

use a local mysql db
——————–

set db connection in universe_wsgi.ini
(line 93) database_connection = mysql://galaxy:<pwd>@localhost/galadb?unix_socket=/var/lib/mysql/mysql.sock

securing the galaxy installation by running it as non-root
==========================================================

(create a local user “galaxy”)
# useradd -c “local user for galaxy installation” -d /home/galaxy -m -U galaxy
# passwd galaxy <pwd>

**********************
* running galaxy with the local user galaxy will throw an error
*
ssh galaxy@host
[galaxy@host ~]$ sh /usr/local/galaxy/galaxy-dist/run.sh
–>
OSError: [Errno 13] Permission denied: ‘./database/tmp/tmpeeJTbo’
*
* so we need to fix this by chowning the installation folder to galaxy
**********************

# cd /usr/local/galaxy/
# chown -R galaxy:galaxy galaxy-dist/

**********************
* now it should run
ssh galaxy@host
[galaxy@host ~]$ sh /usr/local/galaxy/galaxy-dist/run.sh
Starting server in PID <PID>.
serving on http://127.0.0.1:8080
* yes, it does
**********************

crontab fuer user galaxy:
SHELL=/bin/sh
@reboot $SHELL /usr/local/galaxy/galaxy-dist/run.sh >>/tmp/galaxy.log
**********************
* –> galaxy will run after the next reboot
* as the log file is in /tmp, it delete disappear after a reboot
* put it into /var/log and chown it to make it more persistent
**********************
* after reboot, you can check if galaxy was really run at system boot: * [user@host ~]$ ps -ef | grep gala
* galaxy    2864  2862  0 15:44 ?        00:00:00 /bin/sh -c $SHELL /usr/local/galaxy/galaxy-dist/run.sh >>/tmp/galaxy.log
* galaxy    2865  2864  0 15:44 ?        00:00:00 /bin/sh /usr/local/galaxy/galaxy-dist/run.sh
* galaxy    3148  2865  2 15:44 ?        00:00:07 python ./scripts/paster.py serve universe_wsgi.ini
* galaxy    3180  2862  0 15:44 ?        00:00:00 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root

=================================================

**************************

Using Powershell to keep your Windows server/ws from cluttering

Friday, February 8th, 2013

This blog entry and the script will give you an example of how to use Powershell to:

-specify a ‘model’ file
-specify a directory search and fill an array with filenames you filtered out
-work on each entry of the array and to something with it e.g. delete the file
-writing every into a log file

I use the script to delete huge dp dumps and delete them if they are the same size as the original db. Yes, the logic is flawed but for the moment, it does its job. The script is run by task scheduler on a daily basis.

Basically, the script does what it says above. All the required variable can be, no, must be set within the script. The script does not accept any parameters. For you convenience, you can also download it from here:
dbbakcleanup.ps1

here goes:
*************************************************************

# automated bkp cleanup:
# scans a directory, deletes files of the same size as a provided model size
# and writes a summary in a specified log file
#
# ideas for improvements:
# 1) –dry parameter to prevent any action
# 2) write log to event viewer
# 3) ?
#
###########################
#
# variables
#
###########################
#
# General error code, used to quit (exit) the script if not 0
$GenErrCode = 0

# array keeping all log entries which will be dumped into $logfile
$log_lines = @()
# `n <– inserts a newline in the logfile
$log_lines += “`n”
$log_lines += “**********************************”
$log_lines += “`nINFO: Date: ”
$log_lines += Get-Date
$log_lines += “`nInitializing…”

# where to look for files
$mydir = “%windir%:\path\to\my\dir”
if( Test-Path $mydir ) {
$log_lines += “`nINFO: Dir $mydir found, continuing”
} else {
$log_lines += “`nERR: Dir $mydir not found, stopping…”
$GenErrCode = 1
}

# reference file for size
$modelFile = Get-Item ‘%windir%:\path\to\my\file’
if( Test-Path $modelFile ) {
$log_lines += “`nINFO: File $modelFile found, continuing”
} else {
$log_lines += “`nERR: File $modelFile not found, stopping…”
$GenErrCode = 1
}

# filter the files by part of their file name when filling $my_array
# as an example, i specify the file extension. it could also be part of the file name.
# the filter gets applied in 52. if the filter is not a file extension, try setting the * before $pattern
$pattern = “.mdf”

# using $pattern to filter files for filling the array…
$my_array = @(Get-ChildItem -path $mydir\$pattern* | Sort-Object Name -Descending)
if ( $my_array.length -gt 0 ) {
$log_lines += “`nINFO: Array my_array not empty, continuing…”
} else {
# array empty, should not happen so just in case…
# in the line below, the name of the array is hardcoded (my_array), improved could use a variable instead
$log_lines += “`nERR: Array my_arry empty, stopping…”
$GenErrCode = 1
}

# array containing names of deleted files
#$del_files = New-Object System.Collection.ArrayList
$del_files = @()

# specify the name and location of the log file
# could be in a different directory than $mydir
$logfile = “$mydir\my_log.txt”
if( Test-Path $logfile ) {
$log_lines += “`nINFO: Logfile $logfile found…”
} else {
$log_lines += “`nINFO: Logfile $logfile not found, creating new logfile…”
}

# exiting script if an error occurred during initialization…
if ($GenErrCode -eq 1) {
$log_lines += “`nERR: GenErrCode not 0, writing logs and exiting…”
# redirect @log_lines to $logfile
Add-Content $logfile $log_lines
exit
}

#########################
#
# end of initialization
#
#########################
#
# functions
#
#########################

function fileProcess($filejob)
{
# during testing, use the whatif line, a dry-run function within powershell
#Remove-Item $filejob -whatif
Remove-Item $filejob -whatif
}

#########################
#
# end of functions
#
#########################
#
# script
#
#########################

$log_lines += “`nINFO: Starting mainloop…”

$filecount = 0
ForEach ($objElement in $my_array)
{
# compare file size, if equal, call fileProcess to delete file
if ( $objElement.Length -eq $modelFile.Length ) {
fileProcess $objElement
$del_files += $objElement.Name
}
$filecount += 1
}

$log_lines += “`nINFO: Counted ” + $filecount + ” instance(s) to process…”

# all the lines starting with Write-Host were used for troubleshooting
# feel free to uncomment them to get some additional information during runtime
#Write-Host “filecount is” $filecount
#Write-Host “del_files is” $del_files

if ( $del_files.length -lt 1) {
$log_lines += “`nINFO: No files deleted!”
} else {
if ( $del_files.length -eq 1 ) {
$log_lines += “`nINFO: Deleted this file: $del_files”
} else {
$log_lines += “`nINFO: Deleted these file: $del_files”
}
}

#Write-Host $del_files
#Write-Host $log_lines

#
$log_lines += “`nScript finished, writing logs…”
Add-Content $logfile $log_lines

#########################
#
# end of script
#
#########################

*************************************************************

I’m sure there are suboptimal things in the script. Please take it as it is, improve on it, use it as you like.
I wanted the function to do more than just deleting the file – I also wanted it to write to the log file but I couldn’t get that to work.

Cross-stitching a Chinese character (hanzi)

Wednesday, February 6th, 2013

How to cross stitch a hanzi from Zero to Completion:

1) Have a friend give you a cross stitch set for X-mas 2011
2) Get started on it
3) Persevere (or not)
4) Finish it in February 2013
5) Brag about it in your blog

 

The completed cross-stitching. The keyboard behind it should give you an idea about its dimension.

The completed cross-stitching. The keyboard behind it should give you an idea about its dimension.

This is very good picture because it shows how exquisite the handy work is without revealing the flawed spots

This is a very good picture because it shows how exquisite the handy work is without revealing the flawed spots

For more detailed instructions read on…

So maybe you started learning Chinese (or Japanese) and you joined a learning group or managed to get some Chinese friends. Then, for X-mas they give you this cross stitch set which will let you cross stitch a hanzi (or kanji) and there you are.
The package sits there, unopened, glaring at you and your consicence is nagging you. Finally you open the package and things don’t get better. Alas, it’s a complete set including needles, yarn, the fabric and a sheet with incomplete instructions, maybe even written in Chinese (or Japanese) which you can’t read.
Intertubes to the rescue! Google “how to cross stitch” and all the results seriously impede on your masculinity as technocrat. Middle-aged+ ladies explain how to hold the needles, where to start a pattern and what yarn to use. You can already see yourself ending up with a bent back, shaking fingers and bad eyesight after years and years of labour and being laughed at by your friends who meanwhile hack out c0de.

Well, it’s not all that bad. Life is about broaden your horizon and if you should be glad to get such an opportunity. It’ll be an experience! Admittedly, if you’re spending every single minute outside frolicking then you shouldn’t start this li’l project when the beach volleyball season starts. Maybe pick a time where you can spend more time in the house like winter… and consider this: once you’re done, literally everyone will a) unable to refrain him/herself from making a comment on your hanzi (kanji) cross stitch and b) you get the pleasure of seeing their jaws fall to the floor when you tell them you did it all by yourself!

Alright, let’s get on with this… the hardest thing is to get started. But how do you get started on something like this? The instructions I found on the intertubes told me to get started in the middle of pattern. So, find the middle on the pattern sheet and the middle on the fabric. On the fabric you can get the exact middle by folding the fabric diagonally. Then, look at the pattern sheet. How many units to the left of the middle do you have to start? In general, once you find the starting point, work yourself horizontally to the right and back until the line is done, then move on the line below. Until you have completed the whole pattern!

For some good infos on how to cross stitch for beginners, check out this page, which I found very helpful: http://yarntree.com/007begin.htm

In the beginning, I meticulously kept track of how much time I spent on this:

research prior to starting: 5 hours
1.1.: 1 hour
2.1.: 1 hour
3.1.: 1 hour

Quite a long way to go...

Quite a long way to go…

4.1.: 2.5 hours
5.1.: 1.6 hours

Still not there...

Still not there…

6.1.: 1.2 hours
7.1.: 2.2 hours
8.1.: 2.1 hours

I think I can already read this... or not?

I think I can already read this… or not?

9.1.: 0.7 hour
10.1.: 1 hour
11.1.: 0.7 hour
15.1.: 2.3 hours
16.1.: 1.3 hours
17.1.: 0.7 hour
18.1.: 1.8 hours
19.1.: 2.5 hours
21.1.: 4.8 hours
22.1.: 0.9 hours
23.1.: 0.5 hours
24.1.: 1.0 hour
25.1.: 0.9 hours
28.1.: 1.2 hours
30.1.: 2.6 hours
31.1.: 2.0 hours
1.2.: 1.0 hour
3.2.: 1.1 hours
4.2.: 1.8 hours
5.2.: 3.9 hours
6.2.: 1.0 hour

Maybe I gave up writing down the times after that. Also, other things became more important again and I didn’t seriously continue working on this until November 2012… and finally finishing it in February 2013.