egrep to filter out lines in a text file NOT cointaining certain strings

this worked for me:

egrep '^str2|str1' file.txt

winscp delete after successful transfer (put/get)

I've been using WinScp and a script to compress files at maximum compressión, name the file as name_YYMMDD-HHMM.rar, and send them to a remote server; then I wanted to delete local files after a successful transfer. Here's my batch file (it works on WinXP). :

;FOR /F "tokens=1-4 delims=/ " %%I IN ('DATE /t') DO SET mydate=%%K%%J%%I
;
;FOR /F "eol=; tokens=1-2 delims=: " %%a IN ('TIME /t') DO SET mytime=%%a%%b
;
;ECHO The value is "%mydate% %mytime%"
;
;set varname=myname-%mydate%-%mytime%.rar
;
;rar m -m5 %varname% "FolderWithFilesToCompress\*.jpg"
;echo %varname%

start /min winscp.exe /console /command "option batch on" "open lab@64.150.188.62" "option confirm off" "put -delete %varname%" "exit"


To call the batch file automatically in a minimized mode, I call from window's Task Scheduler this vbs script (the last 0 means minimized):

Set WinScriptHost = CreateObject("WScript.Shell")
WinScriptHost.Run Chr(34) & "C:\path\to\batch\mybatch.bat" & Chr(34), 0
Set WinScriptHost = Nothing

And here the documentarion on WinScp

WinSCP Free SFTP, FTP and SCP client for Windows

• News
• About
• Download
• Plugins to File Managers
• Donations
• Forum
• Tracker
• Documentation

• Guides
• F.A.Q.
• Support
• Scripting
• Translations
• Protocols
• Requirements
• Awards
• Screenshots
• History

[[script_commands]]

WinSCP

Contents » Features » Scripting »

Script Commands

Table of Contents +-

• General Syntax
• call
• cd
• chmod
• close
• exit
• get
• help
• keepuptodate
• lcd
• lls
• ln
• lpwd
• ls
• mkdir
• mv
• open
• option
• put
• pwd
• rm
• rmdir
• session
• synchronize

In its scripting functionality, WinSCP supports set of commands described below.

You can see the very same help for the commands as shown here, if you type command help directly in console.

General Syntax

Command parameters that include space(s) have to be enclosed in double-quotes. To use double-quote literally, double it:

put "file with spaces and ""quotes"".html"

You can use environment variables in the commands, with syntax %NAME%1):

put "%FILE_TO_UPLOAD%"

You can reference script arguments (passed on command-line using parameter /parameter) using syntax %N%, where N is ordinal number of argument1):

put "%1%"

Note that WinSCP treats filenames in case sensitive manner. So even if your server treats filenames in case insensitive manner, make sure you specify case properly2).

Advertisements:

call

With SFTP and SCP protocols, executes arbitrary remote shell command. With FTP protocol, executes a protocol command.

call 

If current session does not allow execution of arbitrary remote command separate shell session will be automatically opened.

The command must not require user input.

Alias: !

XML Log Element: call

Examples:

call mysqldump --opt -u USERNAME --password=PASSWORD --all-databases > all_databases.sql
call gzip -c all_databases.sql > all_databases.gz

cd

Changes remote working directory for active session.

cd [  ]

If directory is not specified, changes to home directory.

Examples:

cd /home/martin
cd

chmod

Changes permissions of one or more remote files.

chmod   [  ... ]

mode can be specified as three or four-digit octal number.

Filename can be replaced with wildcard to select multiple files.

XML Log Element: chmod

Examples:

chmod 644 index.html about.html
chmod 1700 /home/martin/public_html
chmod 644 *.html

close

Closes session.

close [  ]

Closes session specified by its number. When session is not specified, closes currently selected session.

Examples:

close 1
close

exit

Closes all sessions and terminates the program.

Advertisements:

exit

Alias: bye

get

Downloads one or more files from remote directory to local directory.

get  [ [  ... ] \[  ] ]

Downloads one or more files from remote directory to local directory. If only one parameter is specified downloads the file to local working directory. If more parameters are specified, all except the last one specify set of files to download. The last parameter specifies target local directory and optionally operation mask to store file(s) under different name. Destination directory must end with backslash. Filename can be replaced with wildcard to select multiple files. To download more files to current working directory use .\ as the last parameter.

Use option command to set transfer options.

Alias: recv

Switches:

Switch	Description
-delete	Delete source remote file(s) after transfer.
-resume	Automatically resume transfer if possible (SFTP and FTP protocols only). Cannot be combined with -append.
-append	Append source file to the end of target file (SFTP protocol only). Cannot be combined with -resume.
-preservetime	Preserve timestamp
-nopreservetime	Do not preserve timestamp
-speed=	Limit transfer speed

Effective options: transfer, confirm, exclude, include, reconnecttime

XML Log Elements: download, rm (with -delete)

Examples:

get index.html
get -delete index.html about.html .\
get index.html about.html d:\www\
get public_html/index.html d:\www\about.*
get *.html *.png d:\www\*.bak

See also synchronize if you need to transfer modified or non-existing files only.

help

Displays help for script commands.

help [  [  ... ] ]

Displays list of commands when no parameters are specified. Displays help for each command when some are specified.

Alias: man

Examples:

help ls
help

keepuptodate

Watches for changes in local directory and reflects them on remote one.

keepuptodate [  [  ] ]

When directories are not specified, current working directories are synchronized. To stop watching for changes press Ctrl-C.

Note: Overwrite confirmations are always off for the command.

Switches:

Switch	Description
-delete	Delete obsolete files
-permissions=	Set permissions (SFTP and SCP protocols only)
-nopermissions	Keep default permissions
-speed=	Limit transfer speed

Effective options: transfer, exclude, include, reconnecttime

XML Log Elements: upload, touch, chmod (with -permissions), rm (with -delete)

Examples:

keepuptodate -delete
keepuptodate d:\www /home/martin/public_html

lcd

Changes local working directory for all sessions.

lcd 

Example:

lcd d:\

lls

Lists the contents of local directory.

Advertisements:

lls [  ]\[  ]

If directory is not specified, lists working directory. When wildcard is specified, it is treated as set of files to list. Otherwise, all files are listed.

Examples:

lls *.html
lls d:\
lls

ln

Creates remote symlink.

ln  

Alias: symlink

Example:

ln /home/martin/public_html www

lpwd

Prints current local working directory (valid for all sessions).

lpwd

ls

Lists the contents of specified directory.

ls [  ]/[  ]

Lists the contents of specified remote directory. If directory is not specified, lists working directory. When wildcard3) is specified, it is treated as set of files to list. Otherwise, all files are listed.

Alias: dir

XML Log Element: ls

Examples:

ls *.html
ls /home/martin
ls

mkdir

Creates remote directory.

mkdir 

XML Log Element: mkdir

Example:

mkdir public_html

mv

Moves or renames one or more remote files.

mv  [  ... ] [ / ][  ]

Destination directory or newname or both must be specified. Destination directory must end with slash. Operation mask can be used instead of new name. Filename can be replaced with wildcard to select multiple files.

Alias: rename

XML Log Element: mv

Examples:

mv index.html public_html/
mv index.html about.*
mv index.html public_html/about.*
mv public_html/index.html public_html/about.html /home/martin/*.bak
mv *.html /home/backup/*.bak

open

Establishes new connection.

open 
open [ sftp|ftp|scp:// ][  [ :password ] @ ]  [ : ]

Establishes connection to given host. Use name of the stored session (to open session, stored in folder, use path syntax “folder/session”). You can also specify host, username, port and protocol directly.

Switches:

Switch	Description
-privatekey=	Private key path
-timeout=	Server response timeout
-hostkey=“”	Specifies fingerprint of expected SSH host key (or several alternative fingerprints separated by semicolon). It makes WinSCP automatically accept hostkey with the fingerprint. As the hostkey fingerprint contains spaces you need to enclose it in quotes. SFTP and SCP protocols only.
-certificate=“”	Specifies fingerprint of expected SSL/TLS sertificate (or several fingerprints separated by semicolon). It makes WinSCP automatically accept certificate with the fingerprint. FTPS protocol only.
-passive	Passive mode (FTP protocol only)
-implicit	Implicit TLS/SSL (FTPS protocol only)
-explicitssl	Explicit SSL (FTPS protocol only)
-explicittls	Explicit TLS (FTPS protocol only)

XML Log Element: session

Examples:

open sftp://martin@example.com -hostkey="ssh-rsa 1024 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
open scp://test@example.com:2222 -privatekey=mykey.ppk
open ftps://martin@example.com -implicit -certificate="xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
open martin@example.com
open example.com
open

option

Shows or sets value of script options.

option [  [  ] ]

If no parameters are specified, lists all script options and their values. When one parameter is specified only, shows value of the option. When two parameters are specified sets value of the option. Initial values of some options are taken from application configuration, however modifing the options does not change the application configuration.

Options are:

Option	Values and description
echo	on|off Toggles echoing of command being executed. Commands affected: all
batch	on|off|abort|continue Toggles batch mode (all prompts are automatically replied negatively). When on, it is recommended to set confirm to off to allow overwrites. With abort script is aborted when any error occurs. With continue all errors are ignored. Commands affected: nearly all
confirm	on|off Toggles confirmations (overwrite, etc.). Commands affected: get, put
transfer	binary|ascii|automatic Transfer mode: binary, ascii (text), automatic (by extension). Commands affected: get, put, synchronize, keepuptodate
exclude include	clear | [;...] Sets exclusion or inclusion masks (only one can be set at time). Commands affected: get, put, synchronize, keepuptodate
reconnecttime	off | Sets time limit in seconds to try reconnecting broken sessions. Commands affected: get, put, synchronize, keepuptodate

Aliases: ascii (for option transfer ascii), binary (for option transfer binary)

Examples:

option transfer
option confirm off
option include "*.html; */"
option exclude "*.tpl.php"
option exclude "*.mp3; *.mp4; *.lnk; *.exe; *.msi; My Pictures; My Music; My Videos;"
option

Note that resetting the same option overwrites previous values, it does not append.

put

Uploads one or more files from local directory to remote directory.

put  [ [  ... ] /[  ] ]

If only one parameter is specified uploads the file to remote working directory. If more parameters are specified, all except the last one specify set of files to upload. The last parameter specifies target remote directory and optionally operation mask to store file(s) under different name. Destination directory must end with slash. Filename can be replaced with Windows wildcard3) to select multiple files. To upload more files to current working directory use ./ as the last parameter.

Use option command to set transfer options.

Switches:

Switch	Description
-delete	Delete source local file(s) after transfer.
-resume	Automatically resume transfer if possible (SFTP and FTP protocols only). Cannot be combined with -append.
-append	Append source file to the end of target file (SFTP protocol only). Cannot be combined with -resume.
-preservetime	Preserve timestamp
-nopreservetime	Do not preserve timestamp
-permissions=	Set permissions (SFTP and SCP protocols only)
-nopermissions	Keep default permissions
-speed=	Limit transfer speed

Alias: send

Effective options: transfer, confirm, exclude, include, reconnecttime

XML Log Elements: upload, chmod (with -permissions), touch (with -preservetime)

Examples:

put index.html
put -delete index.html about.html ./
put -permissions=644 index.html about.html /home/martin/public_html/
put d:\www\index.html about.*
put *.html *.png /home/martin/backup/*.bak

See also synchronize if you need to transfer modified files only.

pwd

Prints current remote working directory for active session.

pwd

rm

Removes one or more remote files.

rm  [  ... ]

If remote recycle bin is configured, moves file to the bin instead of deleting it. Filename can be replaced with wildcard to select multiple files.

XML Log Element: rm

Examples:

rm index.html
rm index.html about.html
rm *.html

rmdir

Removes one or more remote directories.

rmdir  [  ... ]

If remote recycle bin is configured, moves directory to the bin instead of deleting it.

XML Log Element: rm

Example:

rmdir public_html

session

Manages opened sessions.

session [  ]

Makes session specified by its number active. When session number is not specified, lists connected sessions.

Examples:

session 1
session

synchronize

Synchronizes directories.

synchronize local|remote|both [  [  ] ]

When the first parameter is local, changes from remote directory are applied to local directory. When the first parameter is remote, changes from the local directory are applied to the remote directory. When the first parameter is both, both local and remote directories can be modified.

When directories are not specified, current working directories are synchronized.

Note: Overwrite confirmations are always off for the command.

Switches:

Switch	Description
-delete	Delete obsolete files. Ignored for both.
-mirror	Mirror mode (synchronize also older files). Ignored for both.
-criteria=	Comparison criteria. Possible values are time, size, both and none. Ignored for both mode.
-permissions=	Set permissions (SFTP and SCP protocols only)
-nopermissions	Keep default permissions
-speed=	Limit transfer speed

Effective options: transfer, exclude, include, reconnecttime

XML Log Elements: download (with local or both), upload (with remote or both), touch (with remote or both), chmod (with remote or both and -permissions), rm (with remote and -delete)

Examples:

synchronize remote -delete
synchronize both d:\www /home/martin/public_html

1) Generally do enclose reference to double-quotes to cope properly with spaces in its value.

2) This is important particularly for FTP sessions.

3) Windows wildcard supports * and ? only. It does not support all the features of file masks.

script_commands.txt · Last modified: 7 Nov 2010 by prikryl

Search

Documentation
Whole site

powered by

This page

Donate

Associations

• Hosted by SourceForge
• Webhosting
• RSS Feed with Project News
• Driven by DokuWiki
• phpBB
• Bookmark Manager

Site design by Black Gate

sed script to strip ^M characters

this may also be handy, to strip ^M characters:

(from http://www.computing.net/answers/unix/remove-m-and-line-feeds/6934.html)
anupam May 23, 2005 at 21:11:55 Pacific

---

Hi..to remove ^M, give the command :
sed -e 's/^M//g' filename
you will have to write the control-M as follows : first press control-V,then control-M,this will show the correct control-M on the command line.
For removing the linefeeds,try this :
sed -e 's/.$//g' filename
this will remove the last character from the line,which happens to be the line feed.

installink XEphem on Ubuntu

had I found this link earlier, i wouldn't have wasted so much time and effort:

http://www.tc.umn.edu/~brams006/xephem_ubuntu.html

How to Compile XEphem on Ubuntu Linux

---

Following is a (hopefully) painless guide to compiling the excellent astronomy program named XEphem (http://www.clearskyinstitute.com/xephem/) on the Ubuntu linux distribution. Though we now have Stellarium and KStars, XEphem has some unique features and the source code is free for personal/educational use (check the URL for precise terms). In addition, it doesn't require a higher-end OpenGL graphics card. It'll run well on a fairly modest machine. I've tested these steps on Ubuntu 10.04 Lucid Lynx and they worked for me without any trouble.

(1) Preliminaries & Packages.

Use either "sudo apt-get install" or Synaptic to pull down (or verify that you already have) the following packages installed:

gcc
lesstif2-dev
libc6-dev
libxmu-dev
make

Since I embarked on this document (perhaps in 2005), I've noticed a couple changes with the Ubuntu packages. When you install the gcc compiler it sets up a symlink automatically from /usr/bin/gcc to the /usr/bin/gcc-{version number} executable. In the past, you may have needed to create this symlink yourself. For the newbies: 'cd /usr/bin', then 'ln -s gcc-4.4 gcc' or whatever was the gcc version.

If you'd rather compile with libmotif3 and libmotif3-dev (instead of the lesstif2 and lesstif2-dev respectively), you'll need to go into Synaptic (or edit /etc/apt/source.list manually), enable the Non-free Multiverse repository, and install those packages instead. There are some licensing politics surrounding motif.

In any case, I've successfully compiled XEphem 3.7.4 with gcc-4.x and the LGPL lesstif2/lesstif2-dev Ubuntu maintained packages, as well as the libmotif3/libmotif3-dev alternative.

(2) Get the XEphem source code.

Go here: http://www.clearskyinstitute.com/xephem/. Download it to a directory, for instance your user directory, and ungzip/untar it. It'll be in a directory named "xephem-3.7.4" (depending on version number).

(3) Make a change to the xephem-3.7.4/libz/Makefile

This file may initially be read-only, so be sure to chmod +w it if necessary. I noticed this issue with the source for XEphem 3.7.1 and 3.7.2 as well. There's a couple things missing from the makefile in the xephem-3.7.4/libz/Makefile file. Make sure the top of that file includes this:

CC = gcc
CLDFLAGS = -g
CFLAGS = $(CLDFLAGS) -Wall -O2
LDFLAGS =

And the bottom of the file needs to have "gcc" specified rather than cc.:

testzlib: testzlib.o libz.a
gcc $(LDFLAGS) -o testzlib testzlib.o libz.a

(4) Compile XEphem.

Whether you're using motif or lesstif, the XEphem makefiles expect some value for "MOTIF=" in order to compile. Go to the xephem-3.7.4/GUI/xephem directory. Issue this:

make MOTIF=/usr/lib/

XEphem should now successfully compile after a minute or so, depending on the speed of your hardware.

(5) Run and test.

Go to /xephem-3.7.4/GUI/xephem and type "./xephem" to execute. The first things you might want to do is to set the observer location to your own and confirm that your time/zone information is correctly indicated.

I like to create a KDE desktop icon for XEphem. From the KDE desktop, right-click and create a new link to an application. Go to the directory /xephem-3.7.4/GUI/xephem and select the xephem executable, change the icon to the XEphem.png file there. Then make sure the working directory is set to /xephem-3.7.4/GUI/xephem (or wherever you actually installed it).

As of Ubuntu 10.04 I decided to switch to Gnome. I see that the Gnome desktop application launcher doesn't include a "working directory" setting. Unless XEphem is executed from the path it resides in, some components may be improperly pathed. Being less experienced with Gnome, I'm not sure if this is the best approach, but it was awfully simple:

In the ~/xephem3.7.4/ directory I created a simple script called xephem.sh, and chmod +x to it. Here's the contents of the script:

cd /home/{my user directory}/xephem-3.7.4/GUI/xephem
./xephem

I then aimed the Gnome desktop launcher at this script, and used the icon here: /xephem-3.7.4/GUI/xephem/XEphem.png.

Enjoy!

Paul Bramscher
E-mail: brams006@umn.edu

Updated: July 11, 2010

The views and opinions expressed in this page are strictly those of the page author.
The contents of this page have not been reviewed or approved by the University of Minnesota.

missing files

when installing from source file, using make, and it says error "file not found" check apt-file utility for which package you need to install


FROM: http://superuser.com/questions/10997/find-what-package-a-file-belongs-to-in-ubuntu-debian

6 down vote favorite 1

I frequently find myself missing a program, man page, or other file when working on my Ubuntu 8.04 system. Is there any simple way to look up what package contains a given file (whether it is installed already or not)? Maybe some obscure option for apt or dpkg? ubuntu debian package-management dpkg apt link|flag edited Jul 24 '09 at 5:25 John T 65.4k64996 asked Jul 22 '09 at 19:06 Jay Conrod 7021410

3 Answers

active newest votes

up vote 7 down vote accepted

apt-file search filename or apt-file search /path/to/file if you need apt-file, use: apt-get install apt-file you may also want to update its database: sudo apt-file update

running a windows scheduled task minimized

also you can do start /min \path\to\program.exe inside the batch, to run programs from the batch file in minimized mode

You could run it silently using a Windows Script file instead. The Run Method allows you running a script in invisible mode. Create a .vbs file like this one

Set WinScriptHost = CreateObject("WScript.Shell")
WinScriptHost.Run Chr(34) & "C:\Scheduled Jobs\mybat.bat" & Chr(34), 0
Set WinScriptHost = Nothing

and schedule it. The second argument in this example sets the window style. 0 means "hide the window."

Complete syntax of the Run method:

 object.Run(strCommand, [intWindowStyle], [bWaitOnReturn])

Arguments:

• object: WshShell object.
• strCommand: String value indicating the command line you want to run. You must include any parameters you want to pass to the executable file.
• intWindowStyle: Optional. Integer value indicating the appearance of the program's window. Note that not all programs make use of this information.
• bWaitOnReturn: Optional. Boolean value indicating whether the script should wait for the program to finish executing before continuing to the next statement in your script. If set to true, script execution halts until the program finishes, and Run returns any error code returned by the program. If set to false (the default), the Run method returns immediately after starting the program, automatically returning 0 (not to be interpreted as an error code).

link|flag

edited Jun 29 '09 at 10:48

answered May 17 '09 at 9:08 splattne♦ 16.1k63974

taken from: http://serverfault.com/questions/9038/run-a-bat-file-in-a-scheduled-task-without-a-window

beautifying my code

these changes give more legibility to code:

sed script to change:

• if( -> if (
• if....){ _> if...) {
• }else -> } else
• else{ -> else {
• for( -> for (
• for...){ -> for...) {

find . -type f -name "*.php" -exec sed -i -e 's/if(/if (/g' -e 's/if\(.*\)){/if\1) {/g' -e 's/for(/for (/g' -e 's/for\(.*\)){/for\1) {/g' -e 's/else{/else {/g' -e 's/}else/} else/g' {} \;

I had forgotten "foreach":

find . -type f -name "*.php" -exec sed -i -e 's/foreach(/foreach (/g' -e 's/foreach\(.*\)){/\1) {/g' {} \;

sed script to trim trailing spaces:

find . -type f -name "*.php" -exec sed -i 's/[ \t]*$//' {} \;

this may also be handy, to strip ^M characters:

(from http://www.computing.net/answers/unix/remove-m-and-line-feeds/6934.html)
anupam May 23, 2005 at 21:11:55 Pacific

---

Hi..to remove ^M, give the command :
sed -e 's/^M//g' filename
you will have to write the control-M as follows : first press control-V,then control-M,this will show the correct control-M on the command line.
For removing the linefeeds,try this :
sed -e 's/.$//g' filename
this will remove the last character from the line,which happens to be the line feed.

turn on query timing in postgres

11. How to turn on timing, and checking how much time a query takes to execute ?

# \timing — After this if you execute a query it will show how much time it took for doing it.

# \timingTiming is on.

# SELECT * from pg_catalog.pg_attribute ;
Time: 9.583 ms

more stuff on: http://www.thegeekstuff.com/2009/04/15-practical-postgresql-database-adminstration-commands/ 

list all cron jobs for all users

quite good post in StackOverflow: http://stackoverflow.com/questions/134906/how-do-i-list-all-cron-jobs-for-all-users



as root:

i modified the original post to show the user's username:

for user in $(cut -f1 -d: /etc/passwd); do echo "crontab for $user:";crontab -u $user -l;  done

dont forget to also:

cat /etc/crontab

original: 

for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done

git: prune to remove old remote tracking branches

http://stay-calm.blogspot.com/2009/02/git-prune-to-remove-old-remote-tracking.html

Wednesday, February 4, 2009

git: prune to remove old remote tracking branches

I was trying to delete "somebranch" from a git repository today but got this error message, which baffled me for a bit:

$ git push origin :somebranch
error: unable to push to unqualified destination: somebranch
The destination refspec neither matches an existing ref on the remote nor
begins with refs/, and we are unable to guess a prefix based on the source ref.
fatal: The remote end hung up unexpectedly
error: failed to push some refs to 'git@my_server:my_repo.git'

Doing a 'git branch -a' showed a "origin/somebranch" remote branch. Why can't I delete it? Then I realized that the branch might have been deleted on the remote repository and I haven't updated my remote tracking branches yet. Doing a "git pull" won't remove remote tracking branches for branches that have been deleted. To do that for a remote named "origin", you'll need to use this command:

$ git remote prune origin

linux job control - kill a topped job

it is well known that when a process is running it can be stopped using ctrl-z

you can have in this way many stopped jobs, which you can see using the 'jobs' command

you can kill anyone of this jobs by issuing "kill %n", where n is the number of the job as listed by jobs...

fg put the job to foreground and activate the process again
jobs display your stopped/backgrounded jobs
kill %1 send signal TERM to the job numbered %1 in the job list

Janis

taken from: http://www.issociate.de/board/goto/842834/what_is_the_command_to_kill_suspended_job.html

interesting command in linux, nohup a posteriori

i found today this interesting command: i think it works like "nohup" if it's used with the option -h

taken from an excellent linux guide:

disown
disown [options] [jobIDs]
Let job run, but disassociate it from the shell. By default, do not even list the job as an active job; commands like jobs and fg will no longer recognize it. When -h is specified, the job is recognized but simply is kept from being killed when the shell dies.

Options

-a

Act on all jobs.

-h

Do not pass a SIGHUP signal received by the shell on to the job.

debugging and monitoring database usage on postgresql

this is a query that tells you the running queries on a database, the time they have been running, the process id (pid; in case you want to kill it).... it uses the table pg_stat_activity

UPDATE FOR POSTGRES 9.4: some fields have changed name : / procpid is now pid, and current_query is now query... if it still throughs error check the fields with \d
pg_stat_activity

here it is:

select pid, now() - query_start, trim(current_query), datname, usename, waiting from pg_stat_activity where query not ilike '%IDLE%' order by query_start;

this was told to me by a friend (thanks José Zap), i don't know where did he get it from...


if you see a query having trouble, use EXPLAIN <query> to check and see what is postgresql actually doing, try to avoid SEQuential SCANS as these are much slower... create indexes as necessary... watch out don't overcreate them ; )

How to Find All Unread Messages in Gmail

taken from: http://email.about.com/od/gmailtips/qt/et_find_unread.htm 

Type "label:unread"

 

How to Find All Unread Messages in Gmail

 An Inbox, a huge Archive, starred mail, chats, Spam and all kinds of labels: ways to view mail abound in Gmail.

Where, though, is the unread mail — all the unread mail, be it in the archive or the Inbox or anywhere else, and only the unread mail? Fortunately, "unread" is an attribute that messages in Gmail carry like a label. This means you can search — and find — these unread emails easily.

Find All Unread Messages in Gmail

To view all (and only) unread messages in your Gmail account:

• Type "label:unread" (not including the quotation marks) in the Gmail search field.
• Click Search Mail.

If you do this search repeatedly or generally appreciate brevity, you can use "l:unread" or even "l:^u" instead of "label:unread".
Of course, you can combine the quest for unread messages with others:

• "l:^u from:tim" finds all unread messages from "tim",
• "l:^u l:^t" finds all unread starred mail, and
• "l:^u l:^k subject:hi" finds all unread messages with "hi" in the subject that are in the Trash.

 

converting K, M and G to numbers in excel

i'm not sure if the command 'du' in linux to see Disk Usage has an option to show it in Bytes.

I normally use 'du -h' which shows it in "human readable format", but that show it with K for Kilobytes, "M"egabytes and "G"igabytes...

I needed to compare sizes of directories in Excel... so I did this formula for converting the letter to their equivalente numeric. it's in spanish, because my excel is in spanish, and excel has that annoying feature which translates function names !

=SI(ESNUMERO(ENCONTRAR("K";A4692));SUSTITUIR(A4692;"K";"")*1000;SI(ESNUMERO(ENCONTRAR("M";A4692));SUSTITUIR(A4692;"M";"")*1000000;SI(ESNUMERO(ENCONTRAR("G";A4692));SUSTITUIR(A4692;"G";"")*1000000000;"")))

exim exim4 configuration smtp_accept_queue_per_connection

I was getting the warning:  

no immediate delivery: more than 10 messages received in one connection

It was driving me crazy because I needed to send more mails, faster... ; )

I suffered to find this one out, and it was a stupid stupid detail....

I did:

vim /etc/exim4/update-exim4.conf.conf

and added the line at the end of the file:

smtp_accept_queue_per_connection=100

then reloaded exim's configuration

 $ sudo /etc/init.d/exim4 reload
 * Reloading exim4 configuration files   [ OK ]

the tricky part: WATCH OUT !! NO SPACES BETWEEN after or before the "=", they will cause an error

spf ptr reverse dns lookup

we have an application that sends mails to users.

emails sent to hotmail were being automatically redirected to the SPAM folder.

after some little research, we found out that we had to configure an SPF records in out DNS manager...

we did this, and apparently didn't work.

we noticed then, that making a reverse dns lookup to out domain returned a somewhat strange name. we included this name on the PTR part of the record, und alles ok.

we then noticed that /etc/hosts file had this strange name written into it. so we changed it to what it must say: our domain name.


there are many tools to help you write an SPF record.... just google it

enforce secure password to linux users

i'm trying libpam-cracklib....

it's kind of straight forward... takes just a few seconds

kind of good... taken from: http://www.deer-run.com/~hal/sysadmin/pam_cracklib.html

Linux Password Security with pam_cracklib

Hal Pomeranz, Deer Run Associates

Standard Unix reusable passwords are not really a good authentication system. However, the costs associated with migrating to an alternate authentication system such as two-factor token authentication or smartcard-based systems are too high for most enterprises. So sites are generally left with the "lowest common denominator" option provided by their vendors.

In order to improve the security of standard reusable passwords, "best practices" tell us to require users to change their passwords on a regular basis, enforce minimum lengths and good "rules" for new passwords (such as requiring mixed case and non-alphanumeric characters), and even keep a "history" of previous user passwords so that users don't "repeat". Interestingly, Unix systems have typically lagged behind other operating systems in providing this functionality--particularly when it comes to rule-based systems for requiring strong passwords, as well as functionality for keeping password history. In an effort to address this shortcoming, the PAM module pam_cracklib was developed for Linux systems.

I admit that for the longest time I thought pam_cracklib was just about useless. But it turns out that it's not useless, it's just really poorly documented. In an effort to correct this problem, I present the following article based on my research with the existing documentation, the pam_cracklib source code (when in doubt, read the source), and my trusty Knoppix (Debian) GNU/Linux system.

Enabling pam_cracklib

The pam_cracklib module is enabled via the system's standard PAM configuration interface. On Debian systems, this is the /etc/pam.d/common-password file (but it's /etc/pam.d/system-auth on RedHat-derived systems--can't we all just get along?). The typical configuration looks something like this:

password required pam_cracklib.so retry=3 minlen=6 difok=3

password required pam_unix.so md5 use_authtok

The first line enables the pam_cracklib module and sets several module parameters. "retry=3" means that users get three chances to pick a good password before the passwd program aborts. Users can always re-run the passwd program and start over again, however. "minlen=6" sets the minimum number of characters in the password. Actually, since Linux systems generally use MD5 password hashes, which are not limited to 8 character passwords like the old DES56 hashes, you probably want to think about increasing the "minlen" parameter to something a bit longer. We'll come back to this notion a bit later in the article. "difok=3" sets the minimum number of characters that must be different from the previous password. If you increase "minlen", you may also want to increase this value as well.

The second line invokes the standard pam_unix module. The "md5" argument here is what enables standard Linux MD5 password hashes, though you have the option of using old-style DES56 hashes for backwards compatibility with legacy Unix systems. "use_authtok" tells pam_unix to not bother doing any of its own internal password checks, which duplicate many of the checks in pam_cracklib, but instead accept the password that the user inputs after it's been thoroughly checked by pam_cracklib.

Simple Checks

By default, pam_cracklib performs a number of basic checks on the new password:

• Is the new password just the old password with the letters reversed ("password" vs. "drowssap") or rotated ("password" vs. "asswordp")?
• Does the new password only differ from the old one due to change of case ("password" vs. "Password")?
• Are at least some minimum number of characters in the new password not present in the old password? This is where the "difok" parameter comes into play.

These are the same checks you get in the pam_unix module if you turn on the "obscure" flag, but since we're already using pam_cracklib we don't need to do this.

Length and Strength

While the "minlen" parameter controls the minimum password length, things are not as simple as they might appear. This is because pam_cracklib combines the notion of password length with password "strength" (the use of mixed-case and non-letter characters).

"minlen" is actually the minimum required length for a password consisting of all lower-case letters. But users get "length credits" for using upper- and lower-case letters, numbers, and non-alphanumeric characters. The default is normally that you can only get a maximum of "1 credit" for each type of character. So if the administrator sets "minlen=12", a user could still have an 8 character password if they used all four types of characters. Actually, since using a lower-case letter gets you a credit, the real minimum length for an all lower-case password is minlen-1.

The maximum credit for any particular class of characters is actually customizable. The four parameters "lcredit", "ucredit", "dcredit", and "ocredit" are used to set the maximum credit for lower-case, upper-case, numeric (digit), and non-alphanumeric (other) characters, respectively. For example, you could add the following parameters on the pam_cracklib line in the /etc/pam.d/common-password file:

lcredit=0 ucredit=1 dcredit=1 ocredit=2

In other words, lower-case characters aren't special at all, so you get no credit there. On the other hand we give extra credit if the user puts two or more non-alphanumeric characters in their password. One point is still the max credit for upper-case characters and numbers. Note that no matter what you set "minlen" to and no matter how many "credits" you give to your users, pam_cracklib will never let users pick passwords with less than six characters--this is a hard-coded internal minimum.

Play around with these values and find something that makes sense for your site, but as a starting point I might recommend "minlen=12 difok=4" for machines using MD5 password hashes. This means that the smallest password a user could have is 8 characters, and that's only if they use all four character sets.

Dictionary Checks

pam_cracklib also checks the user's password against it's own internal dictionaries of easily guessed passwords. On Debian systems, pam_cracklib's dictionaries live in /var/cache/cracklib and are rebuilt nightly by the
/usr/sbin/update-cracklib script. Other Linux distros may have other mechanisms for updating the dictionaries (as far as I can tell, RedHat doesn't provide any tools for doing this).

The update-cracklib script searches a number of directories for input files, including /usr/local/dict and /usr/local/share/dict. So adding your own words is as easy as putting them in a file in one of these directories and running update-cracklib or waiting for cron to do it for you. Note that if you want to add other directories to update-cracklib's search path, you can do this by modifying the /etc/cracklib/cracklib.conf file (at least on Debian systems).

Password "History"

pam_cracklib is capable of consulting a user's password "history" and not allowing them to re-use old passwords. However, the functionality for actually storing the user's old passwords is enabled via the pam_unix module.

The first step is to make sure to create an empty /etc/security/opasswd file for storing old user passwords. If you forget to do this before enabling the history feature in the PAM configuration file, then all user password updates will fail because the pam_unix module will constantly be returning errors from the password history code due to the file being missing.

Treat your opasswd file like your /etc/shadow file because it will end up containing user password hashes (albeit for old user passwords that are no longer in use):

touch /etc/security/opasswd

chown root:root /etc/security/opasswd

chmod 600 /etc/security/opasswd

Once you've got the opasswd file set up, enable password history checking by adding the option "remember=<x>" to the pam_unix configuration line in the /etc/pam.d/common-password file. Here's how I have things set up on my Knoppix machine:

password required pam_cracklib.so retry=3 minlen=12 difok=4

password required pam_unix.so md5 remember=12 use_authtok

The value of the "remember" parameter is the number of old passwords you want to store for a user. It turns out that there's an internal maximum of 400 previous passwords, so values higher than 400 are all equivalent to 400. Before you complain about this limit, consider that even if your site forces users to change passwords every 30 days, 400 previous passwords represents over 30 years of password history. This is probably sufficient for even the oldest of legacy systems.

Once you've enabled password history, the opasswd file starts filling up with user entries that look like this:

hal:1000:<n>:<hash1>,<hash2>,...,<hashn>

The first two fields are the username and user ID. The <n> in the third field represents the number of old passwords currently being stored for the user--this value is incremented by one every time a new hash is added to the user's password history until <n> ultimately equals the value of the "remember" parameter set on the pam_unix configuration line. <hash1>,<hash2>,...,<hashn> are actually the MD5 password hashes for the user's old passwords.

Password Expiration

At this point you may be wondering how to get the system to automatically force users to change their password after some period of time. This is not actually the job of pam_cracklib. Instead, these parameters are set in the /etc/login.defs file on most Linux systems. PASS_MAX_DAYS is how often users have to change their passwords. PASS_MIN_DAYS is how long a user is forced to live with their new password before their allowed to change it again. PASS_WARN_AGE is the number of days before the password expiration date that the user is warned that their password is about to expire. The choice of values for these parameters is entirely dependent on site policy.

Note that these parameters are only applied to new accounts created with the default system useradd program. If you use some other mechanism for creating accounts on the system, then you'll have to use the chage command (this is not a typo) to manually set these parameters on your user accounts. And if you use a naming service such as LDAP or NIS for account management, then you're completely on your own.

By the way, if you've ever wondered what all those extra fields in the /etc/shadow file were for, the answer is that they store the password expiration/aging information for the user.

Conclusion

Hopefully this article sheds light on some of the more mysterious corners of pam_cracklib's functionality and the Linux password system in general. If pam_cracklib appears limited compared to the password enforcement routines on your particular operating system variant, consider that some Unix-like operating systems (such as Solaris--at least through Solaris 9) don't have any functionality of this type, other than the standard password expiration/aging routines.

About the Author

Hal Pomeranz (hal@deer-run.com) realizes that the wonderful thing about Linux is that you can always refer back to the source code, but would prefer to read a Unix manual page whenever possible.