View Issue Details

IDProjectCategoryView StatusLast Update
0006015phplist applicationMessage Managementpublic18-02-08 14:16
Reporterhola 
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
Product Version 
Target Version2.10.7Fixed in Version2.10.3 
Summary0006015: invalid email user marked unconfirmed
Descriptionhello - I just configured my PHPList to use Domain Throttling using the values below:

define('USE_DOMAIN_THROTTLE',1);
define('DOMAIN_BATCH_SIZE',2);
define('DOMAIN_BATCH_PERIOD',60);

From what I can see it does not seem to work well and for some reason has marked a huge number of my subscribers unconfirmed!!

I have no idea what is going on but I am sure this is a major bug - is there any immediate fix available as this is a great idea.
TagsNo tags attached.

Relationships

related to 0003721 closed phplist 2.10.x 
related to 0005406 closed Users marked 'unconfirmed' upon send 
has duplicate 0006509 resolvedmichiel Users being marked 'unconfirmed' during prossesqueue 

Activities

michiel

07-04-06 01:45

manager   ~0012340

ah, yes, well the way it works is like this.

if an email is not sent for whatever reason, eg in this case, the throttle value disallows it, phplist will check on "validity and deliverability" of the email to make sure that if the reason is that the email is incorrect, it's not included in the next run. So basically that check fails, is either incorrect or there is a network failure. I'm not sure whether it's a bug, or a network setup problem.

hola

07-04-06 02:06

reporter   ~0012342

Hello,

Okay I suppose what is confusing to people using this awesome product is when to use batch processing and what would the values be for the parameters for example here is what I will start testing now:

define("MAILQUEUE_BATCH_SIZE",500);
define("MAILQUEUE_BATCH_PERIOD",60);
define('MAILQUEUE_THROTTLE',1);

Then you also have throttling features as well that adds to the confusion.

I have currently turned off domain throttling as I am still confused how it can unconfirm valid emails address even if the email cannot be sent - how it works out that a valid email cannot be sent is unclear to me.

michiel

07-04-06 02:10

manager   ~0012344

well, the choice in using batches or throttling is a personal preference. But on the whole I was looking for a bit more feedback, which you are now doing, which is great.
the email is validated in all kinds of steps, but one of them being a live lookup of deliverability of the domain, ie does it have an MX record.

hola

07-04-06 02:47

reporter   ~0012346

Okay Michiel - thats really interesting so what you are saying is infact the email address could be invalid for the fact that here is no MX for the domain. Thats cool though there were huge amounts of subscribers unconfirmed that have hotmail and yahoo and xtra.co.nz accounts.

If I try the domain throttling again to test what parameters do you suggest for a mailing list of say 2000 on a server that is not required to control mailout numbers - meaning I have a server that only I use. Below was the settings I used

define('USE_DOMAIN_THROTTLE',1);
define('DOMAIN_BATCH_SIZE',2);
define('DOMAIN_BATCH_PERIOD',60);
define('DOMAIN_AUTO_THROTTLE',0);

michiel

07-04-06 03:00

manager   ~0012348

sorry, I have no idea, and I was hoping others would experiment and find good values. It's very dependent on your hosting situation.
By the way, I used domainthrottling on a 60k list and 5 days later it was still sending out the hotmail emails. So for the time being I switched it off again, but it's still a nice idea.

hola

07-04-06 03:04

reporter   ~0012349

okay no problem - I will have a play on a less important mailing list - what would be helpful is if you could give any understanding why large amounts of valid hotmail and yahoo accounts would be unconfirmed - that could give me some data for my research.

michiel

07-04-06 03:24

manager   ~0012350

sorry, I have no idea, and I'll investigate it, it may just as well be a bug, when I have time. I'll need to work my way through quite a bit though 0003721

hola

07-04-06 03:41

reporter   ~0012351

Understood - I will review the processqueue code and see if I can spot whats going on

hola

10-04-06 07:49

reporter   ~0012439

Hello - here is an update on my research with the domain throttle issue.

I have noticed that when the errors appear within the event log they do not mention another event log message that would have been generated if the mail send failed - file :: sendemaillib.php line 674

if (!$mail->send("", $destinationemail, $fromname, $fromemail, $subject)) {
      logEvent("Error sending message $messageid to $email ($destinationemail)");
      return 0;
    } else {
      return 1;
    }

This does not appear on my event log only the unconfirmed email messages does.

In the sendEmail function within sendemailib.php there are 3 return calls with false as the return value. All bar one have also got a logevent call to document the failure. There offending the return value is at the top of the sendEmail function which is:

if ($email == "")
      return 0;
 

I will be runnign tests where I have added a logevent call to that email null check to see if that is the offending cause.

hola

10-04-06 10:05

reporter   ~0012440

Okay - we have a good update on the throttle issue!

Within the processqueue.php around line 583 there is the following code
if ($cansend) {
          $success = 0;
          if (!TEST) {
            if (!$throttled) {....


Now if the throttle has not been reached then code is executed which attempts to send the email - if not in test mode of course. Note that success var is set to 0.

After that code block the code tests to see the success of sending the email - however if the throttle has been reached the attempt to send an email code block is not even executed so the success var will send be 0.

 if ($success) {
            if (USE_DOMAIN_THROTTLE) {
              list($mailbox,$domainname) = explode('@',$useremail);...
        

Since the success var is set to 0 the email failure code is executed and since the current email did not even have a chance to execute since the throttle was reached there is a very good chance that the validateEmail returns true and thus the code that sets the current subscriber to confirmed=0 is executed.

What I did was add a simple check to see if the throttle had been reached - and if so there is no point in checking the email for send failure since it did not get a chance to send !

     if(!$throttled){
             $failed_sent++;
             if (VERBOSE) {
               output($GLOBALS['I18N']->get('Failed sending to').' '. $useremail);
               logEvent("Failed sending message $messageid to $useremail");
             }
             # make sure it's not because it's an invalid email
             # unconfirm this user, so they're not included next time
             if (!validateEmail($useremail)) {
               logEvent("invalid email $useremail user marked unconfirmed");
               Sql_Query(sprintf('update %s set confirmed = 0 where email = "%s"',
                 $GLOBALS['tables']['user'],$useremail));
             }
             
               }else{
                   
                   logEvent("we have reached the throttle mark so not checking for invalid emails");
                   
               }//end if(!$throttled){

I hope this has fixed the issue! I have been running a test with 1000 dummy emals sent to my domain using thecode change above and throttling on wih auto throttling on... No unconfirmed emails yet have been done - before I added the fix every email after the throttle was reached was marked unconfirmed.

hola

10-04-06 10:20

reporter   ~0012441

Here are my throttling values - not sure if I need the auto throttle though with the processqueue cron running every 15 minutes...Please advise

define('USE_DOMAIN_THROTTLE',1);
define('DOMAIN_BATCH_SIZE',10);
//1 minute space
define('DOMAIN_BATCH_PERIOD',60);
define('DOMAIN_AUTO_THROTTLE',1);

hola

10-04-06 10:37

reporter   ~0012442

I am not sure if this is a bug but the eventlog call for the code below

 logEvent("The $domainname has had the following throttle reaches so far :: $domainthrottle[$domainname]['attempted'] ");
              if (DOMAIN_AUTO_THROTTLE
                && $domainthrottle[$domainname]['attempted'] > 25 # skip a few before auto throttling
                && $num_messages <= 1 # only do this when there's only one message to process otherwise the other ones don't get a change
                && $num_users < 1000 # and also when there's not too many left, because then it's likely they're all being throttled
              ) {

Comes out like
"The <my domain> has had the following throttle reaches so far :: Array['attempted']"

Can the $domainthrottle[$domainname]['attempted'] be accessed by the code?

hola

10-04-06 10:47

reporter   ~0012443

Ignore last question about $domainthrottle[$domainname]['attempted']

hola

11-04-06 01:25

reporter   ~0012476

Further testing has proven that the values for the throttle values in the config file must have a high DOMAIN_BATCH_PERIOD value compared to DOMAIN_BATCH_SIZE.

For example I had set the following

DOMAIN_BATCH_SIZE = 10
DOMAIN_BATCH_PERIOD = 60

What I found is that more than 10 emails were being sent because the interval calculated in the code seemed to cause the $domainthrottle[$domainname]['sent']=0; to be set to zero in a period that caused more emails to be sent than the required DOMAIN_BATCH_SIZE.

When I set the following for a second test

DOMAIN_BATCH_SIZE = 1
DOMAIN_BATCH_PERIOD = 120

There seemed to be more control over the amount of emails sent....

There needs to be more documentation on how the two values control each other as I still do not fully understand why my first test failed to have the email throttling controled.



if ($success) {
            if (USE_DOMAIN_THROTTLE) {
              list($mailbox,$domainname) = explode('@',$useremail);
              if ($domainthrottle[$domainname]['interval'] != $interval) {
                $domainthrottle[$domainname]['interval'] = $interval;
                $domainthrottle[$domainname]['sent']=0;
                logEvent("we have not reached the interval for $domainname");
              } else {
                  logEvent("we have reached the interval for $domainname");
                $domainthrottle[$domainname]['sent']++;
              }
            }

ConteZero

14-04-06 12:58

reporter   ~0012672

The problem is in userlib.php in validateEmail() function:

function validateEmail($email) {
  if ($GLOBALS["config"]["dont_require_validemail"])
    return 1;
  if (isset($email) && (!isset($GLOBALS["check_for_host"]) || $GLOBALS["check_for_host"])) {
    list($username,$domaincheck) = split('@',$email);
    # checking for an MX is not sufficient
# $mxhosts = array();
# $validhost = getmxrr ($domaincheck,$mxhosts);
    $validhost = checkdnsrr($domaincheck, "MX") || checkdnsrr($domaincheck, "A");
  } else {
    $validhost = 0;
  }
  return $validhost && is_email($email);
}

You have to change in config.php:

$check_for_host = 1;

Also commenting the variable should work.
Default is 0 and that cause the function validateEmail() to return false also if the email address is correct.

hola

14-04-06 13:03

reporter   ~0012673

Hello ConteZero,

You are right! A simple solution!

michiel

14-04-06 13:12

manager   ~0012674

Thats brilliant, thanks for finding that.

ConteZero

14-04-06 15:41

reporter   ~0012676

I suggest to change the following code in processqueue.php

     if (!validateEmail($useremail)) {
       logEvent("invalid email $useremail user marked unconfirmed");
       Sql_Query(sprintf('update %s set confirmed = 0 where email = "%s"',
         $GLOBALS['tables']['user'],$useremail));
     }

With:

     if (!$throttled && !validateEmail($useremail)) {
       logEvent("invalid email $useremail user marked unconfirmed");
       Sql_Query(sprintf('update %s set confirmed = 0 where email = "%s"',
         $GLOBALS['tables']['user'],$useremail));
     }

It "should" (I've not yet tested) not control if the email is valid when email sending is trottled for Domain Throttling.

hola

15-04-06 02:27

reporter   ~0012677

Hello ConteZero,

Good solution - I for on have left the if conditional for the throttled value as I don't see the point in wasting CPU cycles to test an email address that didn't even get a chance to be sent.

user1177

17-04-06 05:02

  ~0012682

related forum thread: http://www.phplist.com/forums/viewtopic.php?t=4916

shumisha

17-04-06 13:01

reporter   ~0012686

Hello,

I'd like to point that this also happens if there is an error while sending emails. For example, I'm using smtp server. If I try to send a message while the server is down, or there is a typo in the configuration file, then all users get unconfirmed regardless of their address being correct or not. I imagine this can be due to the same reason. So far, I had check_for_host=0, so I cannot say at the moment if turning that to 1 will fix it.
Just to let you know as you select a solution to fix this issue.

michiel

27-04-06 22:47

manager   ~0013154

I have updated CVS to fix two issues:

- the validateEmail routine did not correctly return whether an email was valid. Basically the "$validhost = 0;" should be "$validhost = 1;" because if host checking is not enabled in the config, it should consider the host "valid" and not "invalid".

- also, as suggested, just to ensure that throttled users are not unconfirmed, they will not be checked on validity.