Tuesday, February 26, 2013

windows registry files

As I wrote in a previous post, I had to change computers.

One of the things is I had a lot of configuration in the registry file of the old computer (from which I only have the hard drive).

So I needed to extract this info...

The registry files are located in

Windows\System32\Config

So I copied those files to a folder in the new computer.

Now I needed a program to read those files, so I used: Windows Registry Recovery

Lastly this didn't work as expected. Apparently the actual registry isn't located (or accesible) in Windows\System32\Config. I say this because I inspected the files found in that dir with the tool afore mentioned, and I didn't find what I was looking (the entries for WinSCP's configuration). So I had to mount the HDD in the old computer, cross my fingers and use REGEDIT to export them.

I was lucky enough in that the computer worked for the time I needed : )

apache user

The linux user that uses Apache webserver can be seen under User directive in apache's configuration

Fedora: /etc/httpd/conf/httpd.conf
Ubuntu: /etc/apache2/apache2.conf

Who in the world started changing the paths of files depending on the flavor of the linux distrubution ?

Sunday, February 24, 2013

query for table definition on mysql

in the console, issue:

show create table my_tables_name;

and it will show the table definition...

for example:

mysql> show create table client_users;
+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| client_users | CREATE TABLE `client_users` (
`id` varchar(36) NOT NULL,
`client_id` varchar(36) NOT NULL,
`user_id` varchar(36) NOT NULL,
UNIQUE KEY `id` (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 |

(for cakePHP guys: I wanted not to use a HABTM)

Apps I can't work without

I recently had to changed computer, and reinstall all the software... : s
So I guess it's a good time to write the (almost all freeware) apps I've been installing, without any particular order. Thanks you guys for that great development, I hope some day to contribute to you guys back.

Notepad++

This I need to configure: install some plugins (minimum compare, fileswitcher)
Then configure "find volatile" because I like doing it with ctrl+alt+arrow[right/left]; this sometimes requires to disable shortcut keys for the graphic card

Virtual Box (Ubuntu)
WinAmp
WinRar
Firefox, extensions:

Chrome, extensions:

almost the same

Bing Desktop (I love the wallpapers)
Gadwin PrintScreen Free
GoogleTalk desktop
TeamViewer
Dropbox
PuTty
WinSCP
PrimoPDF
Skype
LibreOffice (Though I strongly prefer MS Office)
uTorrent (tired of it, trying Tixati)
Foxit PDF Reader
KMPlayer
Resoph Notes
NuSphere PHPed: PHP IDE
poEdit
Paint.Net
window on top
cd burner xp

And that gets me on working...

Friday, January 25, 2013

maintaining session after redirect from payment (credit card) post

I was facing this exact problem in CakePHP, the answer was to use Security.low in core.php !:

(http://cakephp.1045679.n5.nabble.com/After-auto-redirect-from-another-domain-site-the-session-was-lost-td1324708.html)

Hey Guys,
Let me write down the steps to reproduce my problem:
Reproduce steps:
1. Customer login my site(https://www.mysite.com);
2. Choose a product and redirect to Paypal sandbox site(https://developer.paypal.com/cgi-bin/devscr) to complete the payment;
3. After the payment, the customer will be auto redirected to my site;
4. And then the customer session was lost.

I find the solution in Paypal forums, said that :

You could pass your "session variable" Through PayPal variable "custom" than read it back in when the buyer is returned to your site after completing the PayPal payment or through the IPN POST.

You could also use an authentication cookie which would stick around while the customer made a payment on PayPal's website and when they returned to your site they would still be authorized.

I try to store the session in my post form and get it after the redirect from paypal site.
And then I use $this->Session->id($lastSessionID) to restore it. But fail as before.

Is there anybody has the idea how to implement it in Cake App?
Appreciate for any reply from you.
--
Thanks
Joshua

thanks indeed

Saturday, December 8, 2012

Working with Amazon's Elastic Beanstalk

I decided to start using Amazon cloud services.

There is a pretty instresting solution Amazon Beastalk, which you should consider.

There is a lot of documentation, but as normal, there are always pseudo dark steps not thoroughly documented.

For intance, when you arrive to "Develop, Test, and Deploy" section of the "AWS Elastic Beanstalk" documentation, there is a step, name #1 quite obscure:

1.- From your Git repository directory, type the following command.

git aws.config

Nowhere it is told that you must previously 'set up your gir repository' as described here.

I also encountered what I think is a bug regarding timezones on Amazon's service, which was easily fixed. Following the README of the "AWS Command line tools". I got this error:

$ elastic-beanstalk-describe-applications
Service returned an error.
Type: Sender
Code: RequestExpired
Message: Request has expired. Timestamp date: 2012-06-02T14:44:13-04:30

Strangely, this happened because I had my ubuntu virtual box configured to show time as in my timezone (-04:30), while the EC2 instance (at least that's we're I checked) showed in UTC format.

So I decided to change my locale to UTC,

doing
$ sudo dpkg-reconfigure tzdata

I had to select "ETC", and then "UTC"...

Done that, the command "elastic-beanstalk-describe-applications" ran without problems. showing my applications.

Then I continued following the beanstalk documentation and did
git aws.config
git aws.push

The only problem I encountered after doing this is that I realized that Amazon Beanstalk doesn't support Git Submodules !! So I had to make a new directory, copy all my application there, do a new complete repository with all the code and push it to amazon. At first I was kind of reluctant to do this, but then I started to think that it was not that bad ... (I also didn't have other options : )

I realized that my app was configured to always redirect to https protocol (secure http) (this gave me a lot of headaches, because if Beanstalk is not configured to work with SSL then the Health Check URL didn't work at all because of the difference http/https... ). So I decided to buy a $13 SSL certificate from godaddy (they cost this if you click on their ad after a google search, not if you enter directy to godaddy).

I wanted to configure mydomain.com instead of www.mydomain.com, and also to use Amazon Route 53. This was not a problem with the current documentation at Amazon.

Friday, December 7, 2012

rsync over ssh with a private key

RSYNC is my favorite tool to sync two directories, either local or remotely.

Now I learned how to do it over ssh using a private key:

Thanks to Troy and this link: http://troy.jdmz.net/rsync/index.html

This is the command:

rsync -e "ssh -i /path/to/my/key.pem -l myuser" myremotehost:/remote/path/ localpath -iva

Easy peasy

Thursday, December 6, 2012

joomla white page in template after server migration/update // comparing packages

I changed servers mounted all the stuff and some of the Joomla sites I had opened correctly, others just showed a white page.

I didn't know why.

After many many failed attemps to fix it, god's hand enlightened me and I decided to compare the installed php packages between servers.

This can be done using

dpkg --get-selections | grep php

Comparing these packages between servers, it showed evidently I had missed installation of the package

php5-mcrypt

An apt-get install put my joomla sites running again.

Tuesday, December 4, 2012

install LAMP - postgres - change data dir - ssh without password... all donde minuted in AWS Amazon Web Services

I just wanted to comment on how good Amazon Web Services are.

I had my server hosted with another provider, whose name I'm not going to say...

I was with them a lot of time, maybe 10 years. Upgraded my server a couple of times and all. One day my server went nuts and they wanted to charge me $99 dollars AN HOUR just to take a look and help me, no warranties.

So I decided it was time to migrate to a better service: Amazon AWS.

I was getting used to it while setting up the a webapp on Elastic Beanstalk, which is also an excellent service you should try if you have a big webapp you want to deploy. It has and all integration with git, so its all really easy. Using it was that I started understading the relationship between the different services:

EC2: the app servers
RDS: database servers
Route52: domain record management
CloudFront: asset delivery
etc...

Well, as I was saying I was urged to migrate from my old server.

I could set up a running new server in minutes. Maybe the most complicated stuff would be opening a new account and passing through the telephone validation.

So I wrote all this because I wanted to documents the command I did to install the LAMP+Postgres environment I needed. Also to change the datadir of mysql:

I decided to use an instance running Ubuntu.

commands (with sudo or as root):

apt-get update
apt-get install apache2 libapache2-mod-php5 mysql-server libapache2-mod-auth-mysql php5-mysql postgresql-client php5-pgsql

that get all I needed installed.

I had a snapshot of a volume I wanted to mount in the new server. So in Amazon's interface I select attach volume to xxxx instance. After that I had to modify mount the drive that appeared in /dev/xvdf with the command

mount /dev/xvdf /mnt

To make the change permanent I edited the file

/etc/fstab

and added the line

/dev/xvdf /mnt ext4 defaults 0 0

where /mnt is the mount point... the other options are googlable.

I also needed to change MySql's data directory. This was kind of tricky the first time I had to do it.

One needs to change /etc/mysql/my.cnf file find datadir options and change it to what one needs.

The restart mysql with

service mysql restart

But that't not all, it is necessaty to update ubuntu's apparmor file with

vim /etc/apparmor.d/usr.sbin.mysqld

There you must find the old datadir path and change for the new one. Afterwards, reconfigure mysql with

dpkg-reconfigure mysql-server-5.5

And then restart again mysql (I did this, I don't know if its absolutely necessary).

I also needed to log in to the server using classic user/password combination, so its necessary to edit sshd_config with

vim /etc/ssh/sshd_config

There it is necessary to change the directive to

PasswordAuthentication yes

Finally I installed exim4 as an MTA

apt-get install exim4

and configured it with

dpkg-reconfigure exim4-config

I then chose a user to forward me root and postmaster mails. It is necessary to create a file in the user home

~/.forward

and the contents are the email address to which you wish to forward the emails.

that's it !

Monday, May 28, 2012

Database migration from PostgreSQL to MySql

Since I need to use Amazon RDS for my new application, I needed to migrate from my current PostgreSQL database to MySql.

Luckily I didn't realy in anything specific of postgresql, so migration of schema and data was a breeze. I used a free downloadable online console tool pg2mysql

Well, in the schema generated my pg2mysql the TIMESTAMP fields in Postgres were converted to TIMESTAMP datatype in MySql (which has a DEFAULT CURRENT_TIMESTAMP by default), which is not what I wanted. So I manually edited my schema dump replacing "timestamp" to "datetime".

I also changed the default MyISAM to InnoDB...

In particular, in my code I had to be careful because certain syntax changes:

DISTINCT ON (field1, field2) field2 as f1, field2 as f2, field3 -> DISTINCT(CONCAT(field1, field2)), field3

Wednesday, May 16, 2012

error running shared postrotate script for mysql

I was receiving this emails on a daily basis:

/etc/cron.daily/logrotate:
error: error running shared postrotate script for /var/log/mysql.log /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log
run-parts: /etc/cron.daily/logrotate exited with return code 1

I decided to do something about it. After checking this link I determined it was because of a password error in the user debian-sys-maint.

I had deleted this user without noticing it's importance.

So in this link I found how to recover the user, and also I read Ubuntu: Reset debian-sys-maint’s mysql password ...

I hope I don't receive the email tomorrow.

Just for documentation sake, I copy the relevant contents of the previous link I mentioned.

Recently I upgraded Linux on my home server and every day I would get this email:

Subject: Anacron job 'cron.daily' on server.local

/etc/cron.daily/logrotate:
error: error running shared postrotate script for '/var/log/mysql.log /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

I first examined the /etc/cron.daily/logrotate script.
There was only one executable line: /usr/sbin/logrotate /etc/logrotate.conf

I next examined /etc/logrotate.conf and found this: include /etc/logrotate.d

logrotate.d is a directory of scripts to run.

SInce my error message was for MySQL, I examined the /etc/logrorate.d/mysql-server script.
One line in this script is
MYADMIN="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf"

I examined /etc/mysql/debian.cnf and found the cause of the error message:

user = debian-sys-maint
password = oMhAfEiEiO

It was the PASSWORD! There was a mismatch between the password in debian.cnf and the password for thedebian-sys-maint user in MySQL.

Finally, I changed the password for debian-sys-maint in MySQL to the password listed in debian.cnf and the daily error message email stopped.

Log into MySQL as the root user and run the following SQL query, substituting PASSWORD-HERE for the actual plain text password which is the same as the password in the /etc/mysql/debian.conf file:

INSERT INTO `user` (
 `Host`,
 `User`,
 `Password`,
 `Select_priv`,
 `Insert_priv`,
 `Update_priv`,
 `Delete_priv`,
 `Create_priv`,
 `Drop_priv`,
 `Reload_priv`,
 `Shutdown_priv`,
 `Process_priv`,
 `File_priv`,
 `Grant_priv`,
 `References_priv`,
 `Index_priv`,
 `Alter_priv`,
 `Show_db_priv`,
 `Super_priv`,
 `Create_tmp_table_priv`,
 `Lock_tables_priv`,
 `Execute_priv`,
 `Repl_slave_priv`,
 `Repl_client_priv`,
 `Create_view_priv`,
 `Show_view_priv`,
 `Create_routine_priv`,
 `Alter_routine_priv`,
 `Create_user_priv`,
 `ssl_type`,
 `ssl_cipher`,
 `x509_issuer`,
 `x509_subject`,
 `max_questions`,
 `max_updates`,
 `max_connections`,
 `max_user_connections`
)
VALUES (
 'localhost',
 'debian-sys-maint',
 password('PASSWORD-HERE'),
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'Y',
 'N',
 'N',
 'N',
 'N',
 'N',
 '',
 '',
 '',
 '',
 0,
 0,
 0,
 0
);
FLUSH PRIVILEGES;

3)


On Ubuntu systems there is a (system) mysql user debian-sys-maint that is used by the system’s init scripts to control the mysql database, e.g. to start or stop the mysql server. The password of this user is stored (in clear text) in /etc/mysql/debian.cnf. If this password does not match the the actual password in the mysql server the mysql init scripts will fail:


# /etc/init.d/mysql restart
 * Stopping MySQL database server mysqld     [fail]
 * Starting MySQL database server mysqld     [ OK ]
# /etc/init.d/mysql status
/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'debian-sys-maint'@'localhost' (using password: YES)'



Moreover trying to update the mysql server will fail with an error like:


Fehler traten auf beim Bearbeiten von:
 /var/cache/apt/archives/mysql-server-5.1_5.1.37-1ubuntu5.1_i386.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)



To fix the problem you have to update the mysql password for the user debian-sys-maint:

Get the password from /etc/mysql/debian.cnf. The clear text password is stored twice in the file (the lines starting with “password =”:

# Automatically generated for Debian scripts. DO NOT TOUCH!
[client]
host = localhost
user = debian-sys-maint
password = your-secret-password
socket = /var/run/mysqld/mysqld.sock
[mysql_upgrade]
host = localhost
user = debian-sys-maint
password = your-secret-password
socket = /var/run/mysqld/mysqld.sock
basedir = /usr



Update the password in the mysql server (you need mysql root access):

mysql --user root --password
mysql> SET PASSWORD FOR 'debian-sys-maint'@'localhost' = PASSWORD('your-secret-password');



If an previous mysql-server system upgrade failed, just restart the upgrade.


The (debian) documentation can be found in /usr/share/doc/mysql-server-5.1/README.Debian:


[...] You may never ever delete the special mysql user “debian-sys-maint”. This
user together with the credentials in /etc/mysql/debian.cnf are used by the
init scripts to stop the server as they would require knowledge of the mysql
root users password else.
So in most of the times you can fix the situation by making sure that the
debian.cnf file contains the right password, e.g. by setting a new one
(remember to do a “flush privileges” then). [...]

Friday, April 13, 2012

configuring MX records

i realized i didn't know how to configure correctly my MX record

i found out here:

http://www.petri.co.il/configure_mx_records_for_incoming_smtp_email_traffic.htm

Record FQDN	Record Type	Record Value	MX Pref
mail.dpetri.net	A	192.90.1.17
dpetri.net	MX	mail.dpetri.net	10

linux swap memory

interesting article from http://www.linux.com/feature/121916

these lines made want to include it on the blog:

The Linux 2.6 kernel added a new kernel parameter called swappiness to let administrators tweak the way Linux swaps. It is a number from 0 to 100. In essence, higher values lead to more pages being swapped, and lower values lead to more applications being kept in memory, even if they are idle. Kernel maintainer Andrew Morton has said that he runs his desktop machines with a swappiness of 100, stating that "My point is that decreasing the tendency of the kernel to swap stuff out is wrong. You really don't want hundreds of megabytes of BloatyApp's untouched memory floating about in the machine. Get it out on the disk, use the memory for something useful."
One downside to Morton's idea is that if memory is swapped out too quickly then application response time drops, because when the application's window is clicked the system has to swap the application back into memory, which will make it feel slow.
The default value for swappiness is 60. You can alter it temporarily (until you next reboot) by typing as root:

echo 50 > /proc/sys/vm/swappiness

CakePHP SELECT ... FOR UPDATE hack

Sometimes one needs to do a SELECT ... FOR UPDATE in order to preserve the integrity of our transactions. Here is a use-at-you-own-discretion 'hack', which saves coding time if you are in a hurry (like me at the moment) or don't mind writing a useful behaviour for all of us.

The problem can be seen easily with the following example:

Suppose we have a user, an `account table` with the `account.available_money` field on it. Suppose the user has initially $100.

The problem arrives if the hacker manages to concurrently run two processes:

One process does:
    BEGIN
    SELECT available_money from account where user_id = '1' ; // $money = $100
    UPDATE account SET available_money = 0 where user_id = '1';
    COMMIT

While another *concurrent* process does
    -- php process _also_ finds $100 money and uses it all _again_
    BEGIN
    SELECT available_money from account where user_id = '1'; // $money = $100
    UPDATE account SET available_money = 0 where user_id = '1'; //the user buy $100 and he depletes his account
    COMMIT

If this happens, then the user could use more money than he is entitled to. SELECT FOR UPDATE comes to the rescue. Following the previous example

One process does:
    BEGIN
    SELECT available_money from account where user_id = '1' FOR UPDATE; // $money = $100
    UPDATE account SET available_money = 0 where user_id = '1';
    COMMIT

While another *concurrent* process does
    BEGIN
    SELECT available_money from account where user_id = '1' FOR UPDATE; // now this second process must wait the first process to finish, so $money = $0
    -- the user doest not have money to buy anything
    COMMIT


Now, how to achieve this using CakePHP ?

CakePHP's dbo doesn't currently support FOR UPDATE syntax. BUT there is a litlle quick and dirty (maybe too dirty for some) **hack** you can do without doing any modifications to the core ...

Since in the dbo_source, in the `SELECT`statement, the `LIMIT` clause gets parsed at last (see line 1497 from cake/libs/models/dbo_source.php):

    switch (strtolower($type)) {
        case 'select':
            return "SELECT {$fields} FROM {$table} {$alias} {$joins} {$conditions} {$group} {$order} {$limit}";
        break;
        //...

If in a need, **and if using PostgreSQL** (mysql later), you *could* do
        $transac = $this->AccountModel->find('all', array(
            'conditions' => array(
                'user_id' =>$id
            ),
            'fields' => array('available_money'),
            'limit' => 'ALL FOR UPDATE'
        ));`

... and this would generate the `SELECT available_money from account where user_id = '1' FOR UPDATE` clause we are looking for... The trick is in the `'limit' => 'ALL FOR UPDATE'`. This selects ALL records that match the conditions FOR UPDATE.

For MySql you would have to do something a lot uglier. Since MySql doesn't support LIMIT ALL, you would have to use LIMIT 0,18446744073709551615. This is **pretty horrible**, I know... The other options are to code a behavior for all of us : )

Just remember to put your `find` inside a transaction !

**Watch out**, there is a little (but important) catch... You can't do `find('first)`, since the LIMIT clause would be overwritten by Cake as `LIMIT 1`, instead you can do find('all', ... 'limit' => '1 FOR UPDATE' ...)

Monday, December 5, 2011

exim cheat sheet

I just need to document this great resource which always helps me deal with exim basics ... I fear one day it won't be there : )

taken from: http://bradthemad.org/tech/notes/exim_cheatsheet.php

Exim Cheatsheet

Home Pics Journal Personal Guitar Technical Contact

Here are some useful things to know for managing an Exim 4 server. This assumes a prior working knowledge of SMTP, MTAs, and a UNIX shell prompt.

Message-IDs and spool files

The message-IDs that Exim uses to refer to messages in its queue are mixed-case alpha-numeric, and take the form of: XXXXXX-YYYYYY-ZZ. Most commands related to managing the queue and logging use these message-ids.

There are three -- count 'em, THREE -- files for each message in the spool directory. If you're dealing with these files by hand, instead of using the appropriate exim commands as detailed below, make sure you get them all, and don't leave Exim with remnants of messages in the queue. I used to mess directly with these files when I first started running Exim machines, but thanks to the utilities described below, I haven't needed to do that in many months.

Files in /var/spool/exim/msglog contain logging information for each message and are named the same as the message-id.

Files in /var/spool/exim/input are named after the message-id, plus a suffix denoting whether it is the envelope header (-H) or message data (-D).

These directories may contain further hashed subdirectories to deal with larger mail queues, so don't expect everything to always appear directly in the top /var/spool/exim/input or /var/spool/exim/msglog directories; any searches or greps will need to be recursive. See if there is a proper way to do what you're doing before working directly on the spool files.

Basic information

Print a count of the messages in the queue:

root@localhost# exim -bpc

Print a listing of the messages in the queue (time queued, size, message-id, sender, recipient):

root@localhost# exim -bp

Print a summary of messages in the queue (count, volume, oldest, newest, domain, and totals):

root@localhost# exim -bp | exiqsumm

Print what Exim is doing right now:

root@localhost# exiwhat

Test how exim will route a given address:

root@localhost# exim -bt alias@localdomain.com user@thishost.com     <-- alias@localdomain.com   router = localuser, transport = local_delivery root@localhost# exim -bt user@thishost.com user@thishost.com   router = localuser, transport = local_delivery root@localhost# exim -bt user@remotehost.com   router = lookuphost, transport = remote_smtp   host mail.remotehost.com [1.2.3.4] MX=0

Run a pretend SMTP transaction from the command line, as if it were coming from the given IP address. This will display Exim's checks, ACLs, and filters as they are applied. The message will NOT actually be delivered.

root@localhost# exim -bh 192.168.11.22

Display all of Exim's configuration settings:

root@localhost# exim -bP

Searching the queue with exiqgrep

Exim includes a utility that is quite nice for grepping through the queue, called exiqgrep. Learn it. Know it. Live it. If you're not using this, and if you're not familiar with the various flags it uses, you're probably doing things the hard way, like piping `exim -bp` into awk, grep, cut, or `wc -l`. Don't make life harder than it already is.

First, various flags that control what messages are matched. These can be combined to come up with a very particular search.

Use -f to search the queue for messages from a specific sender:

root@localhost# exiqgrep -f [luser]@domain

Use -r to search the queue for messages for a specific recipient/domain:

root@localhost# exiqgrep -r [luser]@domain

Use -o to print messages older than the specified number of seconds. For example, messages older than 1 day:

root@localhost# exiqgrep -o 86400 [...]

Use -y to print messages that are younger than the specified number of seconds. For example, messages less than an hour old:

root@localhost# exiqgrep -y 3600 [...]

Use -s to match the size of a message with a regex. For example, 700-799 bytes:

root@localhost# exiqgrep -s '^7..$' [...]

Use -z to match only frozen messages, or -x to match only unfrozen messages.

There are also a few flags that control the display of the output.

Use -i to print just the message-id as a result of one of the above two searches:

root@localhost# exiqgrep -i [ -r | -f ] ...

Use -c to print a count of messages matching one of the above searches:

root@localhost# exiqgrep -c ...

Print just the message-id of the entire queue:

root@localhost# exiqgrep -i

Managing the queue

The main exim binary (/usr/sbin/exim) is used with various flags to make things happen to messages in the queue. Most of these require one or more message-IDs to be specified in the command line, which is where `exiqgrep -i` as described above really comes in handy.

Start a queue run:

root@localhost# exim -q -v

Start a queue run for just local deliveries:

root@localhost# exim -ql -v

Remove a message from the queue:

root@localhost# exim -Mrm  [  ... ]

Freeze a message:

root@localhost# exim -Mf  [  ... ]

Thaw a message:

root@localhost# exim -Mt  [  ... ]

Deliver a message, whether it's frozen or not, whether the retry time has been reached or not:

root@localhost# exim -M  [  ... ]

Deliver a message, but only if the retry time has been reached:

root@localhost# exim -Mc  [  ... ]

Force a message to fail and bounce as "cancelled by administrator":

root@localhost# exim -Mg  [  ... ]

Remove all frozen messages:

root@localhost# exiqgrep -z -i | xargs exim -Mrm

Remove all messages older than five days (86400 * 5 = 432000 seconds):

root@localhost# exiqgrep -o 432000 -i | xargs exim -Mrm

Freeze all queued mail from a given sender:

root@localhost# exiqgrep -i -f luser@example.tld | xargs exim -Mf

View a message's headers:

root@localhost# exim -Mvh

View a message's body:

root@localhost# exim -Mvb

View a message's logs:

root@localhost# exim -Mvl

Add a recipient to a message:

root@localhost# exim -Mar   [  ... ]

Edit the sender of a message:

root@localhost# exim -Mes

Access control

Exim allows you to apply access control lists at various points of the SMTP transaction by specifying an ACL to use and defining its conditions in exim.conf. You could start with the HELO string.

# Specify the ACL to use after HELO acl_smtp_helo = check_helo  # Conditions for the check_helo ACL: check_helo:      deny message = Gave HELO/EHLO as "friend"     log_message = HELO/EHLO friend     condition = ${if eq {$sender_helo_name}{friend} {yes}{no}}      deny message = Gave HELO/EHLO as our IP address     log_message = HELO/EHLO our IP address     condition = ${if eq {$sender_helo_name}{$interface_address} {yes}{no}}      accept

NOTE: Pursue HELO checking at your own peril. The HELO is fairly unimportant in the grand scheme of SMTP these days, so don't put too much faith in whatever it contains. Some spam might seem to use a telltale HELO string, but you might be surprised at how many legitimate messages start off with a questionable HELO as well. Anyway, it's just as easy for a spammer to send a proper HELO than it is to send HELO im.a.spammer, so consider yourself lucky if you're able to stop much spam this way.

Next, you can perform a check on the sender address or remote host. This shows how to do that after the RCPT TO command; if you reject here, as opposed to rejecting after the MAIL FROM, you'll have better data to log, such as who the message was intended for.

# Specify the ACL to use after RCPT TO acl_smtp_rcpt = check_recipient  # Conditions for the check_recipient ACL check_recipient:      # [...]      drop hosts = /etc/exim_reject_hosts     drop senders = /etc/exim_reject_senders      # [ Probably a whole lot more... ]

This example uses two plain text files as blacklists. Add appropriate entries to these files - hostnames/IP addresses to /etc/exim_reject_hosts, addresses to /etc/exim_reject_senders, one entry per line.

It is also possible to perform content scanning using a regex against the body of a message, though obviously this can cause Exim to use more CPU than it otherwise would need to, especially on large messages.

# Specify the ACL to use after DATA acl_smtp_data = check_message  # Conditions for the check_messages ACL check_message:      deny message = "Sorry, Charlie: $regex_match_string"     regex = ^Subject:: .*Lower your self-esteem by becoming a sysadmin      accept

Fix SMTP-Auth for Pine

If pine can't use SMTP authentication on an Exim host and just returns an "unable to authenticate" message without even asking for a password, add the following line to exim.conf:

  begin authenticators    fixed_plain:   driver = plaintext   public_name = PLAIN   server_condition = "${perl{checkuserpass}{$1}{$2}{$3}}"   server_set_id = $2 >  server_prompts = :

This was a problem on CPanel Exim builds awhile ago, but they seem to have added this line to their current stock configuration.

Log the subject line

This is one of the most useful configuration tweaks I've ever found for Exim. Add this to exim.conf, and you can log the subject lines of messages that pass through your server. This is great for troubleshooting, and for getting a very rough idea of what messages may be spam.

log_selector = +subject

Reducing or increasing what is logged.

Disable identd lookups

Frankly, I don't think identd has been useful for a long time, if ever. Identd relies on the connecting host to confirm the identity (system UID) of the remote user who owns the process that is making the network connection. This may be of some use in the world of shell accounts and IRC users, but it really has no place on a high-volume SMTP server, where the UID is often simply "mail" or whatever the remote MTA runs as, which is useless to know. It's overhead, and results in nothing but delays while the identd query is refused or times out. You can stop your Exim server from making these queries by setting the timeout to zero seconds in exim.conf:

rfc1413_query_timeout = 0s

Disable Attachment Blocking

To disable the executable-attachment blocking that many Cpanel servers do by default but don't provide any controls for on a per-domain basis, add the following block to the beginning of the /etc/antivirus.exim file:

if $header_to: matches "example\.com|example2\.com" then   finish endif

It is probably possible to use a separate file to list these domains, but I haven't had to do this enough times to warrant setting such a thing up.

Searching the logs with exigrep

The exigrep utility (not to be confused with exiqgrep) is used to search an exim log for a string or pattern. It will print all log entries with the same internal message-id as those that matched the pattern, which is very handy since any message will take up at least three lines in the log. exigrep will search the entire content of a log entry, not just particular fields.

One can search for messages sent from a particular IP address:

root@localhost# exigrep '<= .* \[12.34.56.78\] ' /path/to/exim_log

Search for messages sent to a particular IP address:

root@localhost# exigrep '=> .* \[12.34.56.78\]' /path/to/exim_log

This example searches for outgoing messages, which have the "=>" symbol, sent to "user@domain.tld". The pipe to grep for the "<=" symbol will match only the lines with information on the sender - the From address, the sender's IP address, the message size, the message ID, and the subject line if you have enabled logging the subject. The purpose of doing such a search is that the desired information is not on the same log line as the string being searched for.

root@localhost# exigrep '=> .*user@domain.tld' /path/to/exim_log | fgrep '<='

Generate and display Exim stats from a logfile:

root@localhost# eximstats /path/to/exim_mainlog

Same as above, with less verbose output:

root@localhost# eximstats -ne -nr -nt /path/to/exim_mainlog

Same as above, for one particular day:

root@localhost# fgrep YYYY-MM-DD /path/to/exim_mainlog | eximstats

Bonus!

To delete all queued messages containing a certain string in the body:

root@localhost# grep -lr 'a certain string' /var/spool/exim/input/ | \                 sed -e 's/^.*\/\([a-zA-Z0-9-]*\)-[DH]$/\1/g' | xargs exim -Mrm

Note that the above only delves into /var/spool/exim in order to grep for queue files with the given string, and that's just because exiqgrep doesn't have a feature to grep the actual bodies of messages. If you are deleting these files directly, YOU ARE DOING IT WRONG! Use the appropriate exim command to properly deal with the queue.

If you have to feed many, many message-ids (such as the output of an `exiqgrep -i` command that returns a lot of matches) to an exim command, you may exhaust the limit of your shell's command line arguments. In that case, pipe the listing of message-ids into xargs to run only a limited number of them at once. For example, to remove thousands of messages sent from joe@example.com:

root@localhost# exiqgrep -i -f '' | xargs exim -Mrm

Speaking of "DOING IT WRONG" -- Attention, CPanel forum readers

I get a number of hits to this page from a link in this post at the CPanel forums. The question is:

Due to spamming, spoofing from fields, etc., etc., etc., I am finding it necessary to spend more time to clear the exim queue from time to time. [...] what command would I use to delete the queue

The answer is: Just turn exim off, because your customers are better off knowing that email simply isn't running on your server, than having their queued messages deleted without notice.

Or, figure out what is happening. The examples given in that post pay no regard to the legitimacy of any message, they simply delete everything, making the presumption that if a message is in the queue, it's junk. That is total fallacy. There are a number of reasons legitimate mail can end up in the queue. Maybe your backups or CPanel's "upcp" process are running, and your load average is high -- exim goes into a queue-only mode at a certain threshold, where it stops trying to deliver messages as they come in and just queues them until the load goes back down. Or, maybe it's an outgoing message, and the DNS lookup failed, or the connection to the domain's MX failed, or maybe the remote MX is busy or greylisting you with a 4xx deferral. These are all temporary failures, not permanent ones, and the whole point of having temporary failures in SMTP and a mail queue in your MTA is to be able to try again after awhile.

Exim already purges messages from the queue after the period of time specified in exim.conf. If you have this value set appropriately, there is absolutely no point in removing everything from your queue every day with a cron job. You will lose legitimate mail, and the sender and recipient will never know if or why it happened. Do not do this!

If you regularly have a large number of messages in your queue, find out why they are there. If they are outbound messages, see who is sending them, where they're addressed to, and why they aren't getting there. If they are inbound messages, find out why they aren't getting delivered to your user's account. If you need to delete some, use exiqgrep to pick out just the ones that should be deleted.

Reload the configuration

After making changes to exim.conf, you need to give the main exim pid a SIGHUP to re-exec it and have the configuration re-read. Sure, you could stop and start the service, but that's overkill and causes a few seconds of unnecessary downtime. Just do this:

root@localhost# kill -HUP `cat /var/spool/exim/exim-daemon.pid`

You should then see something resembling the following in exim_mainlog:

pid 1079: SIGHUP received: re-exec daemon exim 4.52 daemon started: pid=1079, -q1h, listening for SMTP on port 25 (IPv4)

Read The Fucking Manual

The Exim Home Page

Documentation For Exim

The Exim Specification - Version 4.5x

Exim command line arguments

Any questions?

Well, don't ask me! I'm one guy, with just enough time and Exim skills to keep my own stuff running okay. There are several (perhaps even dozens) of people on the Internet who know way more than me, and who are willing to help out random strangers. Check into the Exim users mailing list, or one of the many web-based gateways to that list. And good luck.

More notes

Home Pics Journal Personal Guitar Technical Contact