NAME

Mystery Girl - A Bounce Handler For Dada Mail


DESCRIPTION

Mystery Girl intelligently handles bounces from Dada Mail list messages.

Mystery Girl hooks into your Dada Mail mailing lists indirectly. You'll first need to create a new POP3 email address which will be used to send all bounces from the Dada Mail lists. This address is known as your Bounce Email Address

The login information for this account will be set in Mystery Girl.

This same address will also be set in the Return-Path of messages sent by Dada Mail. Thus, when a message is bounced, it gets sent to this address, which is monitored by Mystery Girl. Hazzah.

Once Mystery Girl connects to this POP3 acccount, awaiting messages are first read, then, the message is parsed, in an attempt to understand why the message has bounced.

The parsed email will then be examined and an action will be taken. The examination and action are set in a collection of rules. These rules can be tweaked, added, removed and generally mucked about with.

The usual action that is taken is to apply a, score to the offending email address, everytime the address bounces back a message. Once the, Threshold is reached, the email address is unsubscribed from the list.

This usually means that it takes a few bounces from a particular email address to get it removed from a list. This gives a bit of wiggle room and makes sure an email address that is bouncing is bouncing for a fairly good reason, for example: it no longer exists.


OBTAINING A COPY OF THIS PROGRAM

Mystery Girl is located in the, dada/plugins directory of the main Dada Mail distribution, under the name, dada_bounce_handler.pl


REQUIREMENTS

These points are absolutely necessary. Please make sure you have them before you try to install this plugin:


RECOMMENDED

These points are not required, but recommended to have to use Mystery Girl:


Lightning Configuration/Installation Instructions

To get to the point:

Below is the detailed version of the above:


CONFIGURATION

There's a few things you need to configure in this script, they're all at the top.

As far as required changes, we are done.


INSTALLATION

Mystery Girl acts like a Dada Mail plugin.

Usually, you'll set up Dada Mail in your cgi-bin: in your cgi-bin, there's a directory called, ``dada''. Inside the, ``dada'' directory, there are at least two things, one called, ``DADA'' (uppercase) and the mail.cgi script.

In the, ``dada'' directory, create a new directory called, ``plugins''. Upload the dada_bounce_handler.pl script, already configured, into this directory. Change its permissions to, ``755''. Visit the script in your web browser - the URL will look something like this:

        http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl

To run the bounce handler on your bounced messages, click the, Parse Bounces... button.

If you would like have a link on the left hand side of the list control panel, find the following line in the Config.pm:

 #                                      {-Title      => 'Bounce Handler',
 #                                       -Title_URL  => $PLUGIN_URL."/dada_bounce_handler.pl",
 #                                       -Function   => 'dada_bounce_handler',
 #                                       -Activated  => 1,
 #                                      },

And uncomment it (Take off the, ``#'' on each line).

Mystery Girl is now installed.

The last thing you will have to configure is your Dada Mail list administration email address.

Telling Dada Mail to use the Bounce Handler.

You're going to have to tell Dada Mail explicitly that you want bounces to go to the bounce handler. The first step is to set the Dada List Administrator to your bounce email address. You'll set this per list in the each list's control panel, under Manage List - Change List Information

After that, you'll need to configure outgoing email messages to set the Dada List Administrator address in the Return-Path header. Sounds scary, but it's easy enough.

If you're using th sendmail command:

In the list control panel, go to Sending Options - Advanced and check: Add the Sendmail '-f' flag when sending messages ...

This should set the sending to the admin email, and in turn, set the Return-Path header. Dada Mail 3.0 is shipped to have this option set by default.

If you're using SMTP sending:

In the list control panel, go to: Sending Options - SMTP settings> and check the box labeled: Set the Sender of SMTP mailings to the list administration email address Dada Mail 3.0 is shipped to have this option set by default.

Testing

To test out any of these configurations, Send yourself a test message and view the source of the message itself, in your mail reader. In the mail headers, you should see the Return-Path header:

 Return-Path: <dadabounce@myhost.com>
 Delivered-To: justin@myhost.com
 Received: (qmail 75721 invoked from network); 12 May 2003 04:50:01 -0000
 Received: from myhost.com (208.10.44.140)
   by hedwig.myhost.com with SMTP; 12 May 2003 04:50:01 -0000
 Date:Sun, 11 May 2003 23:50:01 -0500
 From:justin <justin@myhost.com>
 Subject:Test, Test, Test
 To:justin@myhost.com
 Sender:dadabounce@myhost.com
 Reply-To:justin <justin@myhost.com>
 Precedence:list
 Content-type:text/plain; charset=iso-8859-1

Notice that the first line has the Return-Path header, correctly putting my bounce email address. My List Owner address, justin@myhost.com still occupies the To: and Reply-To headers, so whoever replies to my message will reply to me, not the bounce handler.

Once you've dialed in your list to use the bounce handler, you should be all set.


Configurating the Cronjob to Automatically Run Mystery Girl

We're going to assume that you already know how to set up the actual cronjob, but we'll be explaining in depth on what the cronjob you need to set is.

Setting the cronjob

Generally, setting the cronjob to have Mystery Girl run automatically, just means that you have to have a cronjob access a specific URL. The URL looks something like this:

 http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl?run=1&verbose=1

Where, http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl is the URL to your copy of dada_bounce_handler.pl

You'll see the specific URL used for your installation of Dada Mail in the web-based control panel for Mystery Girl, under the fieldset legend, Manually Run Mystery Girl, under the heading, Manual Run URL:

This will have Mystery Girl check any awaiting messages.

You may have to look through your hosting account's own FAQ, Knowledgebase and/or other docs to see exactly how you invoke a URL via a cronjob.

A Pretty Good Guess of what the entire cronjob should be set to is located in the web-based crontrol panel for Mystery Girl, under the fieldset legend, Manually Run Mystery Girl, under the heading, curl command example (for a cronjob):

From my testing, this should work for most Cpanel-based hosting accounts.

Here's the entire thing explained:

In all these examples, I'll be running the script every 5 minutes ( */5 * * * * ) - tailor to your taste.

$Plugin_Config->{Allow_Manual_Run}

If you DO NOT want to use this way of invoking the program to check awaiting messages and send them out, make sure to change the variable, $Plugin_Config-{Allow_Manual_Run}> to, 0:

 $Plugin_Config->{Allow_Manual_Run}    = 0;

at the top of the dada_bounce_handler.pl script. If this variable is not set to, 1 this method will not work.

Security Concerns and $Plugin_Config->{Manual_Run_Passcode}

Running the plugin like this is somewhat risky, as you're allowing an anonymous web browser to run the script in a way that was originally designed to only be run either after successfully logging into the list control panel, or, when invoking this script via the command line.

If you'd like, you can set up a simple Passcode, to have some semblence of security over who runs the program. Do this by setting the, $Plugin_Config-{Manual_Run_Passcode}> variable in the dada_bounce_handler.pl source itself.

If you set the variable like so:

    $Plugin_Config->{Manual_Run_Passcode} = 'sneaky';

You'll then have to change the URL in these examples to:

 http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl?run=1&passcode=sneaky

Other options you may pass

You can control quite a few things by setting variables right in the query string:

Notes on Setting the Cronjob for curl

You may want to check your version of curl and see if there's a speific way to pass a query string. For example, this:

 */5 * * * * /usr/local/bin/curl -s http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl?run=1&passcode=sneaky

Doesn't work for me.

I have to use the --get and --data flags, like this:

 */5 * * * * /usr/local/bin/curl -s --get --data run=1\;passcode=sneaky --url http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl

my query string is this part:

 run=1\;passcode=sneaky

And also note I had to escape the, ; character. You'll probably have to do the same for the & character.

Finally, I also had to pass the actual URL of the plugin using the --url flag.


Command Line Interface

There's a slew of optional arguments you can give to this script. To use Mystery Girl via the command line, first change into the directory that Mystery Girl resides in, and issue the command:

 ./dada_bounce_handler.pl --help

For a full list of paramaters.

One of the reasons why you may want to run Mystery Girl via the command line is to set the cronjob via the command line interface, rather than the web-based way. Fair enough!

Command Line Interface for Cronjobs:

One reason that the web-based way of running the cronjob is better, is that it doesn't involve reconfiguring the plugin, every time you upgrade. This makes the web-based invoking a bit more convenient.

#1 Change the lib path

You'll need to explicitly state where both the:

I'm going to rush through this, since if you want to run Mystery Girl this way you probably know the terminology, but:

This script will be running in a different environment and from a different location than what you'd run it as, when you visit it in a web-browser. It's annoying, but one of the things you have to do when running a command line script via a cronjob.

As an example: use lib qw() lines probably look like:

 use lib qw(
 
 ../ 
 ../DADA/perllib 
 ../../../../perl 
 ../../../../perllib 
 
 );

To this list, you'll want to append your site-wide Perl Libraries and the path to the Dada Mail libraries.

If you don't know where your site-wide Perl libraries are, try running this via the command line:

 perl -e 'print $_ ."\n" foreach @INC'; 

If you do not know how to run the above command, visit your Dada Mail in a web browser, log into your list and on the left hand menu and: click, About Dada Mail

Under Script Information, click the, +/- More Information link and under the, Perl Library Locations, select each point that begins with a, ``/'' and use those as your site-wide path to your perl libraries.

#2 Set the cron job

Cron Jobs are scheduled tasks. We need something to check your POP3 email account quite a bit. We're going to set a cron job to test for new messages every 5 minutes. Here's an example cron tab:

  */5  *  *  *  * /usr/bin/perl /home/myaccount/cgi-bin/dada/plugins/dada_bounce_handler.pl >/dev/null 2>&1

Where, /home/myaccount/cgi-bin/dada/plugins/dada_bounce_handler.pl is the full path to the script we just configured.


How Mystery Girl Works

Once you've set up and installed Mystery Girl correctly, bounced email messages now been set to be delivered to the email address that Mystery Girl checks - that is, the, Return-Path header has been set to the Bounce Email Address, which is also the address set for your List Admin Email Address.

When Mystery Girl checks each bounced email message, it'll attempt to figure out the severity of the bounce and score the email address belonging to the subscriber accordingly.

Mystery Girl's Web-Based Control Panel

There's a few things you may also do in Mystery Girl's own control panel:

Bounce Email Scorecard

You can view the bounce email scorecard - the scores that Mystery Girl gives to email addresses that are currently bouncing and haven't been unsubscribed yet.

Selecting an email address (and clicking it) will give you a rundown of the bounce report for each bounce the email address creates.

Manually Run Mystery Girl

If you like, you can run Mystery Girl whenever you like. Running Mystery Girl this way is a good way to see if everything is working correctly and give you an insight of how it all works.

Mystery Girl Configuration

View how Mystery Girl is configured.


Advanced Configuration: Rules, Rule!

dada_bounce_handler.pl figures out what to do with the bounce messages receives by consulting a group of rules. These rules are highly configurable, so if you need to change the behavior of this script, you don't have to change the code.

These rules are stored in the $Plugin_Config-\{Rules}> hashref. An example rule:

     {
        exim_user_unknown => { 
            Examine => { 
                Message_Fields => { 
                    Status      => [qw(5.x.y)], 
                    Guessed_MTA => [qw(Exim)],  
                }, 
                Data => { 
                    Email       => 'is_valid',
                    List        => 'is_valid', 
                }
            },
                Action => { 
                     add_to_score => $Plugin_Config->{Default_Hard_Bounce_Score},
                }, 
            }
    }, 

exim_user_unknown is the title of the rule - just a label, nothing else.

Examine holds a set of parameters that the handler looks at when trying to figure out what to do with a bounced message. This example has a Message_Fields entry and inside that, a Status entry. The Status entry holds a list of status codes. The ones in shown there all correspond to hard bounces; the mailbox probably doesn't exist. Message_Fields also hold a, Guessed_MTA entry - it's explicitly looking for a bounce back from the, Exim mail server.

Examine also holds a Data entry, which holds the Email or List entries, or both. Their values are either 'is_valid', or 'is_invalid'.

So, to sum this all up, this rule will match a message that has Status: Message Field contaning a user unknown error code, (5.1.1, etc) and also a Guessed_MTA Message Field containing, Exim. The message also has to be parsed to have found a valid email and list name.

Pretty Slick, eh?

If this all matches, the Action is... acted upon. In this case, the offending email address will be appended a, Bounce Score of, whatever, $Plugin_Config-{Default_Hard_Bounce_Score}>, which is by default, 4.

If you would like to have the bounced address automatically removed, without any sort of scoring happening, change the action from,

    add_to_score => $Plugin_Config->{Default_Hard_Bounce_Score}

to:

    unsubscribe_bounced_email => 'from_list'

Also, changing from_list, to from_all_lists will do the trick.

I could change the line:

 unsubscribe_bounced_email => 'from_list', 

to:

 mail_list_owner => 'user_unknown_message'

This will, instead of deleting the email automatically, send a message to the list owner, stating that, ``Hey, the message bounced, what do you want to do?''

Another example:

 {
 over_quota => {
         Examine => {
                Message_Fields => {
                        Status => [qw(5.2.2)]
                },
                Data => { 
                        Email => 'is_valid', 
                        List  => 'is_valid',
                }
        },
        Action => { 
                mail_list_owner => 'over_quota_message', 
        },
 }                    

This time, I created a list for messages that get bounced because the mailbox is full. This is still considered a hard bounce, but I don't want the subscriber removed because they haven't check their inbox during the week. In this case, the Action has been set to:

 mail_list_owner => 'over_quota_message', 

Which will do what it sounds like, it'll mail the list owner a message explaining the circumstances.

Here's a schematic of all the different things you can do:

 {
 rule_name => {
         Examine => {
                Message_Fields => {
                        Status               => qw([    ]), 
                        Last-Attempt-Date    => qw([    ]), 
                        Action               => qw([    ]), 
                        Status               => qw([    ]), 
                        Diagnostic-Code      => qw([    ]), 
                        Final-Recipient      => qw([    ]), 
                        Remote-MTA           => qw([    ]), 
                        # etc, etc, etc
                        
                },
                Data => { 
                        Email => 'is_valid' | 'is_invalid' 
                        List  => 'is_valid' | 'is_invalid' 
                }
        },
        Action => { 
                   add_to_score             =>  $x, # where, "$x" is a number
                           mail_list_owner           => 'user_unknown_message', 
                           mail_list_owner           => 'email_not_found_message', 
                           mail_list_owner           => 'over_quota_message', 
                           unsubscribe_bounced_email => 'from_list' | 'from_all_lists',
        },
 },     

Mystery Girl also supports the use of regular expressions for matching any of the Message_Fields. To tell the parser that you're using a regular expression, make the Message_Field key end in '_regex':

 'Final-Recipient_regex' => [(qr/RFC822/)], 

Setting rules is sort of the super advanced part of the configuration, but it may come in handy.


More on Scores, Thresholds, etc

We talked about scoring, but not in great detail, so let's do that:

By default, The Bounce Handler assigns a particular score to each email address that bounces back a message. These scores are tallied each time an email address bounces a message.

Since Dada Mail understands the differences between Hard Bounces and Soft Bounces, it'll append a smaller score for soft bounces, and a larger score for hard bounces.

Once the email address's Bounce Score reaches the Threshold, the email address is then removed from the list.

You can manipulate the Soft and Hard Bounce Scores and Threshold pretty easily. On the top of this script, you'll see the necessary variables to tweak,

Fairly self-explanitory.

If you want even greater control over what kind of bounces give what scores, you can manipulate the Bounce Rules themselves, as described above.

Some things to understand:

If you would like to periodically erase the saved scores, you may do so, by running this script via the command line, like so:

    ./dada_bounce_handler.pl --erase_score_card


FAQs


Thanks

Thanks to: Jake Ortman Henry Hughes for some prelim bounce examples.

Thanks to Eryq ( http://www.zeegee.com ) for the amazing MIME-tools collection. It's a gnarly group of modules.


COPYRIGHT

Copyright (c) 1999-2008 Justin Simoni http://justinsimoni.com All rights reserved.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

Parts of this script were swiped from Mail::Bounce::Qmail module, fetched from here:

http://mikoto.sapporo.iij.ad.jp/cgi-bin/cvsweb.cgi/fmlsrc/fml/lib/Mail/Bounce/Qmail.pm

The copyright of that code stated:

Copyright (C) 2001,2002,2003 Ken'ichi Fukamachi All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

Thanks Ken'ichi