qsub. Parameters I use often

It is faster if I keep track of the parameters I use when submitting job to arc1.

# require 8 hours (required)
#$ -l h_rt=08:00:00
# Execute the job from the current working directory (optional, recommended)
#$ -cwd
# Export all current environment variables to all spawned processes (optional, recommended)
#$ -V
# require 4 GB of memory (optional, default = 1 GB)
#$ -l h_vmem=4G
# require 2 processors (optional, default = 1)
#$ -pe smp 2
# put all ST errors and ST output in directory logs
#$ -e ./logs
#$ -o ./logs

rsync

rsync has a lot of options I tend to forget.
Here a command that works

rsync --verbose --progress --stats --times --rsh="/usr/bin/ssh -c arcfour" --recursive --bwlimit=40000 <origin> <destination>

<origin> is the path to one or several files/directory. You can use wildcard characters

<destination> is the path to one direcotry or file (if you want to change file name)

Either  <origin> or <destination> can be on a remote location. It this case should look like user@server.example.com:/path/to/destination

--rsh="/usr/bin/ssh -c arcfour" Transmits over ssh using a very fast cypher (arcfour) to reduce CPU usage at both ends (sometimes CPU is the bottleneck!)
--bwlimit=40000 Limit I/O bandwidth; KBytes per second. Limits bandwith at 40MB/s (link LIMM -> UNI approx 100MB/s) to avoid saturation of the link.
--times preserve modification times. rsync, by default, checks file size and modification times to decide if a file was changed. If modification time differs, performs a checksum of both files. This might be slower than the transfer itself! To always force checksum use -c, to only compare file size use --size-only. Be careful with this last option.
--perms keep file permissions. Destination files will be set with same permissions as origin.
--recursive recurse into directories.
--progress and --stats give you something to look at if bored (and want to monitor the connection) and a final report to be impressed by 😉

Interactive decripting with gpg

To interactively unencript files (such as those provided by EGA) which have been encripted using a single password with gpg, here how to do it

files=`ls *.gpg`
for f in $files
do
    qsub -b y -cwd -V -l h_rt=4:00:00 \ 
    gpg -d --no-tty --batch --passphrase-file passFile.txt -o ${f%.gpg} $f
done

In red the command to use the queing system, in blue the gpg options

-d = decompress

–passphrase-file = name of the one line file with the password to use. Not the most secure way, but better than providing the password to the command line

–batch = use batch mode

–no-tty = do not send to tty. to use together with –batch and the queing system

find and xargs

Some working example on how to use find together with xargs

find syntax:

find [options] <where> <what>

find -L . -name *17a69*

find in local directory (.) and subdirectories (also follows symbolic links due to option -L) all files that contains 17a69 in the name

find . -cmin -30

find all files in current directory that changed status in the last 30 minutes. Use -amin or -mmin to find those that where accessed or modified in the last 30 minutes. Use -ctime -atime -mtime to specify days.

find . -cmin +30

same as above, but looks for files created/accessed/modified MORE than 30 minutes ago

xargs

xargs let you apply a comand to a list of arguments found by, for instance, find

find . -name interm -type d | xargs rm -r

find all directory that match “interm” and delete them

find . -mmin +100 -type f | xargs -I xxx mv xxx ~/tmp1/

finds all files (-type f) modified more than 100 minutes ago and then move them to ~/tmp1
-I xxx tell xargs to put the arguments where the next occurrence of xxx is

Some links

Unix Find Command Tutorial
Some examples of using UNIX find command

MySQL

Administration

First connection as root from localhost

mysql -u root

Set root password

SET PASSWORD FOR 'root'@'localhost' = PASSWORD('newpwd')

Now you can connect from localhost using

mysql -u root -p

For security reason, it is better not to allow connection as root from everywhere. Password are stored and communicated encrypted, however SSH is much more robust protocol for encryption.
Add a “normal” user that can connect from localhost as well as from any other host
And provide all privileges on a user database. The user can read, alter, create and delete the database and its content. Do not “GRANT ALL” to the database called “mysql”, or that user will become like root.

Continue reading ‘MySQL’ »

bash shortcut

Some bash shortcut for Mac OsX found in different webpages

Very useful to move around and do little magic with bash

Ctrl-a | Move to the start of the line.

Ctrl-e | Move to the end of the line.

Ctrl-] x | Where x is any character, moves the cursor forward to the next occurance of x.

Ctrl-u | Delete from the cursor to the beginning of the line. Continue reading ‘bash shortcut’ »