View Issue Details

IDProjectCategoryView StatusLast Update
0015502phpList 3 applicationMessage Send Processpublic06-06-13 02:56
Reportermartinjoiner 
PrioritynormalSeverityblockReproducibilityalways
Status resolvedResolutionno change required 
Product Version2.10.10 
Target VersionFuture developmentsFixed in Version 
Summary0015502: On a list of 500 users only the first 50 receive messages yet the log suggests complete success
DescriptionI have a list of just over 500 users which I entered in bulk through PHPMyAdmin. The bug manifests itself in 2 ways, when I send to that list only the first 50 (alphabetically) receive messages. Secondly it manifests itself in the inability to view any of the users past 50 - you click next page and it simply reloads the current page (1 - 50). Although they are there and visible if you view them by clicking "Users" and scroll through the list of all users. Help!
TagsNo tags attached.

Relationships

related to 0008905 resolveduser1822 Database error 2006 while doing query MySQL server has gone 
related to 0010058 resolvedmichiel Many users not receiving email because of too-large SQL query in processqueue.php (with solution) 
related to 0015402 resolvedmichiel Process Message Send Hangs In Broswer 

Activities

h2b2

24-06-10 00:16

manager   ~0051046

v2.10.10 is an older version. You might try upgrading.

gazouteast

08-10-10 12:42

reporter   ~0051119

.
@H2B2 - upgrading won't cure it - I'm a week into deep investigation of this as I'm getting it too on a cloud server setup.

It looks like th eissue is coming out of Apache and the mod_fcgid handler. Fastcgi by default only allows 500 open processes, and any phpList queue with a couple of hundred recipients or more, is going to chew that in the first few seconds of processing, then hang / fail - usually with a server error log pointing to mod_fcgid and either a process terminated or failed type message - this usually gets accompanied by the 500 Server error white screen of death on the processqueue window in the browser.

Upon clearing the messagequeue window (closing it) the event log shows anywhere from a few to a few dozen emails were sent, but not a full batch or list.

The fix lies in the server -
- if using Plesk as a cpanel, it will be in the vhost.conf in the individual domain's conf folder (accessible only to server admins, but visible to domain owners (not readable, just visible) in file manager).
- if using cPanel, it will be in the vhost.conf master file for the server - again only accessible to server admins.

The mod_fcgid settings in the vhosts.conf or httpd.conf need updated - Apache.org deprecated the old command config names from Apache 2.x and almost no hosts have updated to the new ones. Additionally, most (especially shared) hosts are far too miserly on the max processes setting - usually with something like 4 or 8 (single digit) when big phplist runs need 10,000+, they also usually have the process life settings far too short - e.g. 20 seconds or 40 seconds (default) when phplist processqueue needs hours or days before it life expires.

Two solutions are open - get onto a VPS or dedicated server, or rewrite phpLIST processqueue to pre-assemble in batches the outgoing messages, then send the batch, then preassemble the next batch, then send it, etc.

This would also work well with all forms of phplist batch processing - e.g. use half the hour for pre-assembly, and the other half for sending - it would avoid constant load on the mail queue handler, and could use a file level temporary caching for the assembled messaged, deleting them after each successful send. This would reduce the issues related to 500 server errors and fcgid process limits and timeouts - might not catch all of them, but would certainly reduce their incidence level.

All sorts of wizardry could go into splitting the list up for batching - prioritising by user attributes - e.g. location, customer status, age, gender, duration of subscription (I like that one as a loyalty reward for announcing special offers and time sensitive news) and so forth, with each sub-session handling one or more user attributes as the "batch of the hour".

Sorry - didn't mean to drift into the realm of wish lists.

Gaz

.

michiel

09-08-11 12:56

manager   ~0051395


Sounds like what would be needed is a mix between batch processing and throttling to tweak it to handle a specific server environment. If you have any suggestions on this, it would be great to add it to the documentation to explain the specific requirements, so that other people might benefit from that knowledge.

On the whole, I tend to advise to simply use commandline processing for sending, and forget about the web interface, as it requires an apache connection, which is not necessary to actually send the mails out.

michiel

06-06-13 02:56

manager   ~0052105

just use commandline processing. I send 4 million emails a month :-)