Each read in a SAM/BAM/CRAM file has a binary flag. I find it very difficult to remember what they mean and build up a flag that does what I need.
This site (http://broadinstitute.github.io/picard/explain-flags.html) is very hard to find when I need it, but very useful. It both explain each bit and allow to build the flag you need.
to use with samtools view -f or samtools view -F
It is faster if I keep track of the parameters I use when submitting job to arc1.
# require 8 hours (required)
#$ -l h_rt=08:00:00
# Execute the job from the current working directory (optional, recommended)
#$ -cwd
# Export all current environment variables to all spawned processes (optional, recommended)
#$ -V
# require 4 GB of memory (optional, default = 1 GB)
#$ -l h_vmem=4G
# require 2 processors (optional, default = 1)
#$ -pe smp 2
# put all ST errors and ST output in directory logs
#$ -e ./logs
#$ -o ./logs
rsync has a lot of options I tend to forget.
Here a command that works
rsync --verbose --progress --stats --times --rsh="/usr/bin/ssh -c arcfour" --recursive --bwlimit=40000 <origin> <destination>
<origin> is the path to one or several files/directory. You can use wildcard characters
<destination> is the path to one direcotry or file (if you want to change file name)
Either <origin> or <destination> can be on a remote location. It this case should look like user@server.example.com:/path/to/destination
--rsh="/usr/bin/ssh -c arcfour" Transmits over ssh using a very fast cypher (arcfour) to reduce CPU usage at both ends (sometimes CPU is the bottleneck!)
--bwlimit=40000 Limit I/O bandwidth; KBytes per second. Limits bandwith at 40MB/s (link LIMM -> UNI approx 100MB/s) to avoid saturation of the link.
--times preserve modification times. rsync, by default, checks file size and modification times to decide if a file was changed. If modification time differs, performs a checksum of both files. This might be slower than the transfer itself! To always force checksum use -c, to only compare file size use --size-only. Be careful with this last option.
--perms keep file permissions. Destination files will be set with same permissions as origin.
--recursive recurse into directories.
--progress and --stats give you something to look at if bored (and want to monitor the connection) and a final report to be impressed by 😉
To interactively unencript files (such as those provided by EGA) which have been encripted using a single password with gpg, here how to do it
files=`ls *.gpg` for f in $files do qsub -b y -cwd -V -l h_rt=4:00:00 \ gpg -d --no-tty --batch --passphrase-file passFile.txt -o ${f%.gpg} $f done
In red the command to use the queing system, in blue the gpg options
-d = decompress
–passphrase-file = name of the one line file with the password to use. Not the most secure way, but better than providing the password to the command line
–batch = use batch mode
–no-tty = do not send to tty. to use together with –batch and the queing system
Some working example on how to use find together with xargs
find [options] <where> <what>
find -L . -name *17a69*
find in local directory (.) and subdirectories (also follows symbolic links due to option -L) all files that contains 17a69 in the name
find . -cmin -30
find all files in current directory that changed status in the last 30 minutes. Use -amin or -mmin to find those that where accessed or modified in the last 30 minutes. Use -ctime -atime -mtime to specify days.
find . -cmin +30
same as above, but looks for files created/accessed/modified MORE than 30 minutes ago
xargs let you apply a comand to a list of arguments found by, for instance, find
find . -name interm -type d | xargs rm -r
find all directory that match “interm” and delete them
find . -mmin +100 -type f | xargs -I xxx mv xxx ~/tmp1/
finds all files (-type f) modified more than 100 minutes ago and then move them to ~/tmp1
-I xxx tell xargs to put the arguments where the next occurrence of xxx is
Unix Find Command Tutorial
Some examples of using UNIX find command
First connection as root from localhost
mysql -u root
Set root password
SET PASSWORD FOR 'root'@'localhost' = PASSWORD('newpwd')
Now you can connect from localhost using
mysql -u root -p
For security reason, it is better not to allow connection as root from everywhere. Password are stored and communicated encrypted, however SSH is much more robust protocol for encryption.
Add a “normal” user that can connect from localhost as well as from any other host
And provide all privileges on a user database. The user can read, alter, create and delete the database and its content. Do not “GRANT ALL” to the database called “mysql”, or that user will become like root.
Some bash shortcut for Mac OsX found in different webpages
Very useful to move around and do little magic with bash
Ctrl-a | Move to the start of the line.
Ctrl-e | Move to the end of the line.
Ctrl-] x | Where x is any character, moves the cursor forward to the next occurance of x.
Ctrl-u | Delete from the cursor to the beginning of the line. Continue reading ‘bash shortcut’ »