Mystery Girl - A Bounce Handler For Dada Mail
Mystery Girl intelligently handles bounces from Dada Mail list messages.
Mystery Girl hooks into your Dada Mail mailing lists indirectly. You'll first need to create a new POP3 email address which will be used to send all bounces from the Dada Mail lists. This address is known as your Bounce Email Address
The login information for this account will be set in Mystery Girl.
This same address will also be set in the Return-Path of messages sent by Dada Mail. Thus, when a message is bounced, it gets sent to this address, which is monitored by Mystery Girl. Hazzah.
Once Mystery Girl connects to this POP3 acccount, awaiting messages are first read, then, the message is parsed, in an attempt to understand why the message has bounced.
The parsed email will then be examined and an action will be taken. The examination and action are set in a collection of rules. These rules can be tweaked, added, removed and generally mucked about with.
The usual action that is taken is to apply a, score to the offending email address, everytime the address bounces back a message. Once the, Threshold is reached, the email address is unsubscribed from the list.
This usually means that it takes a few bounces from a particular email address to get it removed from a list. This gives a bit of wiggle room and makes sure an email address that is bouncing is bouncing for a fairly good reason, for example: it no longer exists.
Mystery Girl is located in the, dada/plugins directory of the main Dada Mail distribution, under the name, dada_bounce_handler.pl
These points are absolutely necessary. Please make sure you have them before you try to install this plugin:
Basically, use the version of Mystery Girl that comes with the version of Dada Mail you're running.
Mystery Girl works by checking a bounce email address via the POP3 protocol.
You will need to setup a new email address for Mystery Girl to check. I usually set up an account named, ``bounces@yourdomain.com'', where, ``yourdomain.com'' is the name of the domain Dada Mail is installed on.
Some things to consider:
Meaning: don't periodically check it yourself via a mail reader. Doing so will not break Dada Mail, but it will stop Mystery Girl from working correctly. Why? Because sometimes checking a POP3 address will download the messages awaiting in the POP3 Inbox and remove them from this inbox. If you need to periodically check this inbox, make sure to have your mail reader set to not automatically remove the mssages.
Meaning, if your domain is, ``yourdomain.com'', the bounce email address should be something like, ``bounces@yourdomain.com''. In other words, do not use a Yahoo! Gmail, or Hotmail account for your bounce address. This will most likely disrupt all regular mail sending in Dada Mail.
Sometimes, it can't - there may be a block for connections to port 110 from connections that come from your hosting account server.
These points are not required, but recommended to have to use Mystery Girl:
Mystery Girl can be configured to run automatically by using a cron tab - In Other Words: a scheduled task.
If you do not know how to set up a cron job, attempting to set one up for Dada Mail will result in much aggravation. Please read up on the topic before attempting!
Shell Access is sometimes required to set up a cronjob, using the:
crontab -e
command. You may also be able to set up a cron tab using a web-based control panel tool, like Cpanel.
Shell access also facilitates testing of the program.
To get to the point:
Below is the detailed version of the above:
There's a few things you need to configure in this script, they're all at the top.
Your bounce email address login information is saved in the, dada_bounce_handler.pl script itself -
Create a new POP3 email account. This email account will be the address that bounced messages will be directed towards.
Change the following variables:
to reflect the permissions for the email address you're going to use for the bounce handler.
As far as required changes, we are done.
Mystery Girl acts like a Dada Mail plugin.
Usually, you'll set up Dada Mail in your cgi-bin: in your cgi-bin, there's a directory called, ``dada''. Inside the, ``dada'' directory, there are at least two things, one called, ``DADA'' (uppercase) and the mail.cgi script.
In the, ``dada'' directory, create a new directory called, ``plugins''. Upload the dada_bounce_handler.pl script, already configured, into this directory. Change its permissions to, ``755''. Visit the script in your web browser - the URL will look something like this:
http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl
To run the bounce handler on your bounced messages, click the, Parse Bounces... button.
If you would like have a link on the left hand side of the list control panel, find the following line in the Config.pm:
# {-Title => 'Bounce Handler', # -Title_URL => $PLUGIN_URL."/dada_bounce_handler.pl", # -Function => 'dada_bounce_handler', # -Activated => 1, # },
And uncomment it (Take off the, ``#'' on each line).
Mystery Girl is now installed.
The last thing you will have to configure is your Dada Mail list administration email address.
You're going to have to tell Dada Mail explicitly that you want bounces to go to the bounce handler. The first step is to set the Dada List Administrator to your bounce email address. You'll set this per list in the each list's control panel, under Manage List - Change List Information
After that, you'll need to configure outgoing email messages to set the Dada List Administrator address in the Return-Path
header. Sounds scary, but it's easy enough.
In the list control panel, go to Sending Options - Advanced and check: Add the Sendmail '-f' flag when sending messages ...
This should set the sending to the admin email, and in turn, set the Return-Path header. Dada Mail 3.0 is shipped to have this option set by default.
In the list control panel, go to: Sending Options - SMTP settings> and check the box labeled: Set the Sender of SMTP mailings to the list administration email address Dada Mail 3.0 is shipped to have this option set by default.
To test out any of these configurations, Send yourself a test message and view the source of the message itself, in your mail reader. In the mail headers, you should see the Return-Path header:
Return-Path: <dadabounce@myhost.com> Delivered-To: justin@myhost.com Received: (qmail 75721 invoked from network); 12 May 2003 04:50:01 -0000 Received: from myhost.com (208.10.44.140) by hedwig.myhost.com with SMTP; 12 May 2003 04:50:01 -0000 Date:Sun, 11 May 2003 23:50:01 -0500 From:justin <justin@myhost.com> Subject:Test, Test, Test To:justin@myhost.com Sender:dadabounce@myhost.com Reply-To:justin <justin@myhost.com> Precedence:list Content-type:text/plain; charset=iso-8859-1
Notice that the first line has the Return-Path header, correctly putting my bounce email address. My List Owner address, justin@myhost.com still occupies the To: and Reply-To headers, so whoever replies to my message will reply to me, not the bounce handler.
Once you've dialed in your list to use the bounce handler, you should be all set.
We're going to assume that you already know how to set up the actual cronjob, but we'll be explaining in depth on what the cronjob you need to set is.
Generally, setting the cronjob to have Mystery Girl run automatically, just means that you have to have a cronjob access a specific URL. The URL looks something like this:
http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl?run=1&verbose=1
Where, http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl is the URL to your copy of dada_bounce_handler.pl
You'll see the specific URL used for your installation of Dada Mail in the web-based control panel for Mystery Girl, under the fieldset legend, Manually Run Mystery Girl, under the heading, Manual Run URL:
This will have Mystery Girl check any awaiting messages.
You may have to look through your hosting account's own FAQ, Knowledgebase and/or other docs to see exactly how you invoke a URL via a cronjob.
A Pretty Good Guess of what the entire cronjob should be set to is located in the web-based crontrol panel for Mystery Girl, under the fieldset legend, Manually Run Mystery Girl, under the heading, curl command example (for a cronjob):
From my testing, this should work for most Cpanel-based hosting accounts.
Here's the entire thing explained:
In all these examples, I'll be running the script every 5 minutes ( */5 * * * * ) - tailor to your taste.
*/5 * * * * /usr/local/bin/curl -s --get --data run=1 --url http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl
*/5 * * * * /usr/local/bin/curl -s --get --data run=1\;verbose=0\;test=0\;messages=100 --url http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl
If you DO NOT want to use this way of invoking the program to check awaiting messages and send them out, make sure to change the variable, $Plugin_Config-{Allow_Manual_Run}> to, 0:
$Plugin_Config->{Allow_Manual_Run} = 0;
at the top of the dada_bounce_handler.pl script. If this variable is not set to, 1 this method will not work.
Running the plugin like this is somewhat risky, as you're allowing an anonymous web browser to run the script in a way that was originally designed to only be run either after successfully logging into the list control panel, or, when invoking this script via the command line.
If you'd like, you can set up a simple Passcode, to have some semblence of security over who runs the program. Do this by setting the, $Plugin_Config-{Manual_Run_Passcode}> variable in the dada_bounce_handler.pl source itself.
If you set the variable like so:
$Plugin_Config->{Manual_Run_Passcode} = 'sneaky';
You'll then have to change the URL in these examples to:
http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl?run=1&passcode=sneaky
You can control quite a few things by setting variables right in the query string:
As mentioned above, the $Plugin_Config-{Manual_Run_Passcode}> allows you to set some sort of security while running in this mode. Passing the actual password is done in the query string:
http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl?run=1&passcode=sneaky
Overrides $Plugin_Config-{MessagesAtOnce}>. States how many messages should be checked and parsed in one execution of the program. Example:
http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl?run=1&messages=10
By default, you'll receive the a report of how Mystery Girl is doing parsing and adding scores (and what not). This is sometimes not so desired, especially in a cron environment, since all this informaiton will be emailed to you (or someone) everytime the script is run. You can run Mystery Girl with a cron that looks like this:
*/5 * * * * /usr/local/bin/curl -s --get --data run=1 --url http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl >/dev/null 2>&1
The, >/dev/null 2>&1
line throws away any values returned.
Since all the information being returned from the program is done sort of indirectly, this also means that any problems actually running the program will also be thrown away.
If you set verbose to, ``0'', under normal operation, Mystery Girl won't show any output, but if there's a server error, you'll receive an email about it. This is probably a good thing. Example:
* * * * * /usr/local/bin/curl -s --get --data run=1\;verbose=0 --url http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl
Runs Mystery Girl in test mode by checking the bounces and parsing them, but not actually carrying out the Rules.
You may want to check your version of curl
and see if there's a speific way to pass a query string. For example, this:
*/5 * * * * /usr/local/bin/curl -s http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl?run=1&passcode=sneaky
Doesn't work for me.
I have to use the --get
and --data
flags, like this:
*/5 * * * * /usr/local/bin/curl -s --get --data run=1\;passcode=sneaky --url http://example.com/cgi-bin/dada/plugins/dada_bounce_handler.pl
my query string is this part:
run=1\;passcode=sneaky
And also note I had to escape the, ; character. You'll probably have to do the same for the & character.
Finally, I also had to pass the actual URL of the plugin using the --url flag.
There's a slew of optional arguments you can give to this script. To use Mystery Girl via the command line, first change into the directory that Mystery Girl resides in, and issue the command:
./dada_bounce_handler.pl --help
For a full list of paramaters.
One of the reasons why you may want to run Mystery Girl via the command line is to set the cronjob via the command line interface, rather than the web-based way. Fair enough!
One reason that the web-based way of running the cronjob is better, is that it doesn't involve reconfiguring the plugin, every time you upgrade. This makes the web-based invoking a bit more convenient.
You'll need to explicitly state where both the:
I'm going to rush through this, since if you want to run Mystery Girl this way you probably know the terminology, but:
This script will be running in a different environment and from a different location than what you'd run it as, when you visit it in a web-browser. It's annoying, but one of the things you have to do when running a command line script via a cronjob.
As an example: use lib qw()
lines probably look like:
use lib qw( ../ ../DADA/perllib ../../../../perl ../../../../perllib );
To this list, you'll want to append your site-wide Perl Libraries and the path to the Dada Mail libraries.
If you don't know where your site-wide Perl libraries are, try running this via the command line:
perl -e 'print $_ ."\n" foreach @INC';
If you do not know how to run the above command, visit your Dada Mail in a web browser, log into your list and on the left hand menu and: click, About Dada Mail
Under Script Information, click the, +/- More Information link and under the, Perl Library Locations, select each point that begins with a, ``/'' and use those as your site-wide path to your perl libraries.
Cron Jobs are scheduled tasks. We need something to check your POP3 email account quite a bit. We're going to set a cron job to test for new messages every 5 minutes. Here's an example cron tab:
*/5 * * * * /usr/bin/perl /home/myaccount/cgi-bin/dada/plugins/dada_bounce_handler.pl >/dev/null 2>&1
Where, /home/myaccount/cgi-bin/dada/plugins/dada_bounce_handler.pl is the full path to the script we just configured.
Once you've set up and installed Mystery Girl correctly, bounced email messages
now been set to be delivered to the email address that Mystery Girl checks -
that is, the, Return-Path
header has been set to the Bounce Email Address,
which is also the address set for your List Admin Email Address.
When Mystery Girl checks each bounced email message, it'll attempt to figure out the severity of the bounce and score the email address belonging to the subscriber accordingly.
There's a few things you may also do in Mystery Girl's own control panel:
You can view the bounce email scorecard - the scores that Mystery Girl gives to email addresses that are currently bouncing and haven't been unsubscribed yet.
Selecting an email address (and clicking it) will give you a rundown of the bounce report for each bounce the email address creates.
If you like, you can run Mystery Girl whenever you like. Running Mystery Girl this way is a good way to see if everything is working correctly and give you an insight of how it all works.
View how Mystery Girl is configured.
dada_bounce_handler.pl figures out what to do with the bounce messages receives by consulting a group of rules. These rules are highly configurable, so if you need to change the behavior of this script, you don't have to change the code.
These rules are stored in the $Plugin_Config-\{Rules}> hashref. An example rule:
{ exim_user_unknown => { Examine => { Message_Fields => { Status => [qw(5.x.y)], Guessed_MTA => [qw(Exim)], }, Data => { Email => 'is_valid', List => 'is_valid', } }, Action => { add_to_score => $Plugin_Config->{Default_Hard_Bounce_Score}, }, } },
exim_user_unknown is the title of the rule - just a label, nothing else.
Examine holds a set of parameters that the handler looks at when trying to figure out what to do with a bounced message. This example has a Message_Fields entry and inside that, a Status entry. The Status entry holds a list of status codes. The ones in shown there all correspond to hard bounces; the mailbox probably doesn't exist. Message_Fields also hold a, Guessed_MTA entry - it's explicitly looking for a bounce back from the, Exim mail server.
Examine also holds a Data entry, which holds the Email or List entries, or both. Their values are either 'is_valid', or 'is_invalid'.
So, to sum this all up, this rule will match a message that has Status: Message Field contaning a user unknown error code, (5.1.1, etc) and also a Guessed_MTA Message Field containing, Exim. The message also has to be parsed to have found a valid email and list name.
Pretty Slick, eh?
If this all matches, the Action is... acted upon. In this case, the offending email address will be appended a, Bounce Score of, whatever, $Plugin_Config-{Default_Hard_Bounce_Score}>, which is by default, 4.
If you would like to have the bounced address automatically removed, without any sort of scoring happening, change the action from,
add_to_score => $Plugin_Config->{Default_Hard_Bounce_Score}
to:
unsubscribe_bounced_email => 'from_list'
Also, changing from_list, to from_all_lists will do the trick.
I could change the line:
unsubscribe_bounced_email => 'from_list',
to:
mail_list_owner => 'user_unknown_message'
This will, instead of deleting the email automatically, send a message to the list owner, stating that, ``Hey, the message bounced, what do you want to do?''
Another example:
{ over_quota => { Examine => { Message_Fields => { Status => [qw(5.2.2)] }, Data => { Email => 'is_valid', List => 'is_valid', } }, Action => { mail_list_owner => 'over_quota_message', }, }
This time, I created a list for messages that get bounced because the mailbox is full. This is still considered a hard bounce, but I don't want the subscriber removed because they haven't check their inbox during the week. In this case, the Action has been set to:
mail_list_owner => 'over_quota_message',
Which will do what it sounds like, it'll mail the list owner a message explaining the circumstances.
Here's a schematic of all the different things you can do:
{ rule_name => { Examine => { Message_Fields => { Status => qw([ ]), Last-Attempt-Date => qw([ ]), Action => qw([ ]), Status => qw([ ]), Diagnostic-Code => qw([ ]), Final-Recipient => qw([ ]), Remote-MTA => qw([ ]), # etc, etc, etc }, Data => { Email => 'is_valid' | 'is_invalid' List => 'is_valid' | 'is_invalid' } }, Action => { add_to_score => $x, # where, "$x" is a number mail_list_owner => 'user_unknown_message', mail_list_owner => 'email_not_found_message', mail_list_owner => 'over_quota_message', unsubscribe_bounced_email => 'from_list' | 'from_all_lists', }, },
Mystery Girl also supports the use of regular expressions for matching any of the Message_Fields. To tell the parser that you're using a regular expression, make the Message_Field key end in '_regex':
'Final-Recipient_regex' => [(qr/RFC822/)],
Setting rules is sort of the super advanced part of the configuration, but it may come in handy.
We talked about scoring, but not in great detail, so let's do that:
By default, The Bounce Handler assigns a particular score to each email address that bounces back a message. These scores are tallied each time an email address bounces a message.
Since Dada Mail understands the differences between Hard Bounces and Soft Bounces, it'll append a smaller score for soft bounces, and a larger score for hard bounces.
Once the email address's Bounce Score reaches the Threshold, the email address is then removed from the list.
You can manipulate the Soft and Hard Bounce Scores and Threshold pretty easily. On the top of this script, you'll see the necessary variables to tweak,
Fairly self-explanitory.
If you want even greater control over what kind of bounces give what scores, you can manipulate the Bounce Rules themselves, as described above.
Some things to understand:
If you would like to periodically erase the saved scores, you may do so, by running this script via the command line, like so:
./dada_bounce_handler.pl --erase_score_card
Yes. Because of the Rules, you can set what happens, depending on what type of bounce you receive. By default, the bounce handler is set up to think, ``hard bounces'' are email addresses that are invalid because they simply don't exist, and soft bounces as email addresses that because the email box is full, or there was some sort of problem actually sending the message to the subscriber.
Dada Mail basically works by saying, After x amount of bounces, just remove from the list.
It's very possible that Mystery Girl can't read your subscription database or the list settings database. This is because Dada Mail may be running under the webserver's username, usually, nobody, and not what Mystery Girl is running under, usually your account username.
You'll need to do a few things:
You'll most likely need to change the permissions of these files to, '777'. PlainText subscription databases have the format of listshortname.list and are usually located where you set the $FILES Config file variable. .List settings Databases have the format of mj-listshortname and are usually located in the same location.
So you don't need to change the permissions of the list files for every new list you create, set the $FILE_CMOD Config variable to 0777: $FILE_CHMOD = 0777;
Notice there are no quotes around 0777.
Make sure that you have checked, Print list-specific headers in all list emails in Sending Options -> Advanced. Mystery Girl uses the List header to figure out what list the bounce is coming from.
Report it to the bug tracker:
You'll most likely have to make a new rule for it. If you want, attach a copy of the bounced message to the bug tracker:
http://sourceforge.net/tracker/?group_id=13002&atid=113002
And we'll see if we can't get that kind of bounce in a new version.
It's from a Yeah Yeah Yeahs song: Mystery Girl. A bounce handler is sort of a mysterious tool, making decisions for you and a mysterious girl just seems to be one full of power and allusion. The song itself is about rejecting a guy that just doesn't make it anymore, so that gives a good metaphor to a bounced mail, in a slightly weird, nerdy, nerdy, nerdy... artsy way.
When the bounce handler emails a list owner, you can do nothing but answer back to it. Yeah Yeah Yeah.
(colophon)
Actually, the lyrics I'm thinking of aren't from the song, Mystery Girl, but from the song, ``Bang!'' off of the YYY's self titled release. Mystery Girl is the next song on that album. The song after that is one called, ``Art Star'', which is what I am in the daytime! The next song is called, ``Miles Away'', which is where you probably are to me. All this in, ``Our Time'' (the last song) See? it's like this was all written in the stars.
Here's a small clip of the YYY's performing ``Mystery Girl'' at the Gothic on 11.20.03 that I took:
http://dadamailproject.com/media/YYYs_Mystery_Girl_Clip.mov
hot!
Thanks to: Jake Ortman Henry Hughes for some prelim bounce examples.
Thanks to Eryq ( http://www.zeegee.com ) for the amazing MIME-tools collection. It's a gnarly group of modules.
Copyright (c) 1999-2008 Justin Simoni http://justinsimoni.com All rights reserved.
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
Parts of this script were swiped from Mail::Bounce::Qmail module, fetched from here:
http://mikoto.sapporo.iij.ad.jp/cgi-bin/cvsweb.cgi/fmlsrc/fml/lib/Mail/Bounce/Qmail.pm
The copyright of that code stated:
Copyright (C) 2001,2002,2003 Ken'ichi Fukamachi All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Thanks Ken'ichi