View Issue Details

IDProjectCategoryView StatusLast Update
0017997phplist applicationHTML Email Supportpublic23-05-16 15:08
Reporterbertpoort 
PrioritylowSeverityfeatureReproducibilityN/A
Status resolvedResolutionfixed 
PlatformAnyOSAnyOS VersionAny
Product Version3.2.4 
Target Versionnext minorFixed in Version3.2.5 
Summary0017997: [Feature request] Option to embed images from external domains in HTML emails (patch included)
DescriptionI would like to suggest adding an option to embed images from external domains in HTML emails. I wrote a patch which downloads and caches external images and makes them available to phpList using the existing functions filesystem_image_exists and get_filesystem_image. It also cleans up old cached files.
Steps To ReproducePatch:

*** class.phplistmailer.php Mon Dec 07 20:59:00 2015
--- class.phplistmailer.patched.php Fri Jan 29 15:41:04 2016
***************
*** 440,445 ****
--- 440,546 ----
  
      public function filesystem_image_exists($filename)
      {
+ if (defined('EMBEDEXTERNALIMAGES') && EMBEDEXTERNALIMAGES) {
+ // Check for a http(s) address excluding this host
+ if ((strpos($filename, 'http') === 0) && (strpos($filename, '://'.$_SERVER['SERVER_NAME'].'/') === FALSE)) {
+ $extCacheDir = $GLOBALS['tmpdir'].'/external_cache';
+
+ // Create cache directory
+ if (!file_exists($extCacheDir))
+ @mkdir($extCacheDir);
+
+ if (file_exists($extCacheDir) && is_writable($extCacheDir)) {
+ // Remove old files in cache directory
+ if ($extCacheDirHandle = @opendir($extCacheDir)) {
+ while (FALSE !== ($cacheFile = @readdir($extCacheDirHandle))) {
+ if (($cacheFile != '.') && ($cacheFile != '..')) {
+ $cacheFileMTime = @filemtime($extCacheDir.'/'.$cacheFile);
+
+ if (is_numeric($cacheFileMTime) && ($cacheFileMTime > 0) && ((time() - $cacheFileMTime) >= 86400)) // 1 day
+ @unlink($extCacheDir.'/'.$cacheFile);
+ }
+ }
+
+ @closedir($extCacheDirHandle);
+ }
+
+ // Generate local filename
+ //$cacheFile = $extCacheDir.'/'.$this->messageid.'_'.hash('sha256', $filename);
+ $cacheFile = $extCacheDir.'/'.$this->messageid.'_'.preg_replace(array('~[\.][\.]+~i', '~[^\w\.]~i'), array('', '_'), $filename);
+
+ // Download and cache file
+ if (!file_exists($cacheFile)) {
+ $cacheFileContent = '';
+
+ // Try downloading using cURL
+ if (function_exists('curl_init')) {
+ $cURLHandle = curl_init($filename);
+
+ if ($cURLHandle !== FALSE) {
+ //curl_setopt($cURLHandle, CURLOPT_URL, $filename);
+ curl_setopt($cURLHandle, CURLOPT_HTTPGET, TRUE);
+ curl_setopt($cURLHandle, CURLOPT_HEADER, 0);
+ curl_setopt($cURLHandle, CURLOPT_BINARYTRANSFER, TRUE);
+ curl_setopt($cURLHandle, CURLOPT_RETURNTRANSFER, TRUE);
+ //curl_setopt($cURLHandle, CURLOPT_FILE, $cacheFileHandle);
+ curl_setopt($cURLHandle, CURLOPT_TIMEOUT, 30);
+ curl_setopt($cURLHandle, CURLOPT_FOLLOWLOCATION, TRUE);
+ curl_setopt($cURLHandle, CURLOPT_MAXREDIRS, 10);
+ curl_setopt($cURLHandle, CURLOPT_SSL_VERIFYPEER, FALSE);
+ curl_setopt($cURLHandle, CURLOPT_FAILONERROR, TRUE);
+
+ $cacheFileContent = curl_exec($cURLHandle);
+
+ $cURLErrNo = curl_errno($cURLHandle);
+ $cURLInfo = curl_getinfo($cURLHandle);
+
+ curl_close($cURLHandle);
+
+ if ($cURLErrNo != 0)
+ $cacheFileContent = 'CURL_ERROR_'.$cURLErrNo;
+ if ($cURLInfo['http_code'] >= 400)
+ $cacheFileContent = 'HTTP_CODE_'.$cURLInfo['http_code'];
+ }
+ }
+
+ // Try downloading using file_get_contents
+ if ($cacheFileContent == '') {
+ $remoteURLContext = stream_context_create(array(
+ 'http' =>
+ array(
+ 'method' => 'GET',
+ 'timeout' => '30',
+ 'max_redirects' => '10'
+ )));
+
+ $cacheFileContent = file_get_contents($filename, FALSE, $remoteURLContext);
+ if ($cacheFileContent === FALSE)
+ $cacheFileContent = 'FGC_ERROR';
+ }
+
+ // Limit size
+ if (strlen($cacheFileContent) > 1048576) // 1 MB
+ $cacheFileContent = 'MAX_SIZE';
+
+ // Write cache file
+ //file_put_contents($cacheFile, $cacheFileContent, LOCK_EX);
+ $cacheFileHandle = @fopen($cacheFile, 'wb');
+ if ($cacheFileHandle !== FALSE) {
+ if (flock($cacheFileHandle, LOCK_EX)) {
+ fwrite($cacheFileHandle, $cacheFileContent);
+ fflush($cacheFileHandle);
+ flock($cacheFileHandle, LOCK_UN);
+ }
+ fclose($cacheFileHandle);
+ }
+ }
+
+ if (file_exists($cacheFile) && (@filesize($cacheFile) > 64))
+ return TRUE;
+ }
+ }
+ }
+
          ## find the image referenced and see if it's on the server
        $imageroot = getConfig('uploadimageroot');
  # cl_output('filesystem_image_exists '.$docroot.' '.$filename);
***************
*** 469,474 ****
--- 570,583 ----
  
      public function get_filesystem_image($filename)
      {
+ if (defined('EMBEDEXTERNALIMAGES') && EMBEDEXTERNALIMAGES) {
+ $extCacheDir = $GLOBALS['tmpdir'].'/external_cache';
+ $cacheFile = $extCacheDir.'/'.$this->messageid.'_'.preg_replace(array('~[\.][\.]+~i', '~[^\w\.]~i'), array('', '_'), $filename);
+
+ if (file_exists($cacheFile) && (@filesize($cacheFile) > 64))
+ return base64_encode(file_get_contents($cacheFile));
+ }
+
          ## get the image contents
        $localfile = basename(urldecode($filename));
  # cl_output('get file system image'.$filename.' '.$localfile);
TagsNo tags attached.

Relationships

parent of 0018123 resolvedsupport document define(EMBEDEXTERNALIMAGES, 1); in config extended 

Activities

bertpoort

29-01-16 14:43

reporter  

class.phplistmailer.patched.php (31,794 bytes)

bertpoort

29-01-16 14:44

reporter  

class.phplistmailer.php.patch (6,750 bytes)
*** class.phplistmailer.org.php	Mon Dec 07 20:59:00 2015
--- class.phplistmailer.patched.php	Fri Jan 29 15:41:04 2016
***************
*** 440,445 ****
--- 440,546 ----
  
      public function filesystem_image_exists($filename)
      {
+         if (defined('EMBEDEXTERNALIMAGES') && EMBEDEXTERNALIMAGES) {
+             // Check for a http(s) address excluding this host
+             if ((strpos($filename, 'http') === 0) && (strpos($filename, '://'.$_SERVER['SERVER_NAME'].'/') === FALSE)) {
+                 $extCacheDir = $GLOBALS['tmpdir'].'/external_cache';
+ 
+                 // Create cache directory
+                 if (!file_exists($extCacheDir))
+                     @mkdir($extCacheDir);
+ 
+                 if (file_exists($extCacheDir) && is_writable($extCacheDir)) {
+                     // Remove old files in cache directory
+                     if ($extCacheDirHandle = @opendir($extCacheDir)) {
+                         while (FALSE !== ($cacheFile = @readdir($extCacheDirHandle))) {
+                             if (($cacheFile != '.') && ($cacheFile != '..')) {
+                                 $cacheFileMTime = @filemtime($extCacheDir.'/'.$cacheFile);
+ 
+                                 if (is_numeric($cacheFileMTime) && ($cacheFileMTime > 0) && ((time() - $cacheFileMTime) >= 86400)) // 1 day
+                                     @unlink($extCacheDir.'/'.$cacheFile);
+                             }
+                         }
+ 
+                         @closedir($extCacheDirHandle);
+                     }
+ 
+                     // Generate local filename
+                     //$cacheFile = $extCacheDir.'/'.$this->messageid.'_'.hash('sha256', $filename);
+                     $cacheFile = $extCacheDir.'/'.$this->messageid.'_'.preg_replace(array('~[\.][\.]+~i', '~[^\w\.]~i'), array('', '_'), $filename);
+ 
+                     // Download and cache file
+                     if (!file_exists($cacheFile)) {
+                         $cacheFileContent = '';
+ 
+                         // Try downloading using cURL
+                         if (function_exists('curl_init')) {
+                             $cURLHandle = curl_init($filename);
+ 
+                             if ($cURLHandle !== FALSE) {
+                                 //curl_setopt($cURLHandle, CURLOPT_URL, $filename);
+                                 curl_setopt($cURLHandle, CURLOPT_HTTPGET, TRUE);
+                                 curl_setopt($cURLHandle, CURLOPT_HEADER, 0);
+                                 curl_setopt($cURLHandle, CURLOPT_BINARYTRANSFER, TRUE);
+                                 curl_setopt($cURLHandle, CURLOPT_RETURNTRANSFER, TRUE);
+                                 //curl_setopt($cURLHandle, CURLOPT_FILE, $cacheFileHandle);
+                                 curl_setopt($cURLHandle, CURLOPT_TIMEOUT, 30);
+                                 curl_setopt($cURLHandle, CURLOPT_FOLLOWLOCATION, TRUE);
+                                 curl_setopt($cURLHandle, CURLOPT_MAXREDIRS, 10);
+                                 curl_setopt($cURLHandle, CURLOPT_SSL_VERIFYPEER, FALSE);
+                                 curl_setopt($cURLHandle, CURLOPT_FAILONERROR, TRUE);
+ 
+                                 $cacheFileContent = curl_exec($cURLHandle);
+ 
+                                 $cURLErrNo = curl_errno($cURLHandle);
+                                 $cURLInfo = curl_getinfo($cURLHandle);
+ 
+                                 curl_close($cURLHandle);
+ 
+                                 if ($cURLErrNo != 0)
+                                     $cacheFileContent = 'CURL_ERROR_'.$cURLErrNo;
+                                 if ($cURLInfo['http_code'] >= 400)
+                                     $cacheFileContent = 'HTTP_CODE_'.$cURLInfo['http_code'];
+                             }
+                         }
+ 
+                         // Try downloading using file_get_contents 
+                         if ($cacheFileContent == '') {
+                             $remoteURLContext = stream_context_create(array(
+                                 'http' =>
+                                     array(
+                                         'method' => 'GET',
+                                         'timeout' => '30',
+                                         'max_redirects' => '10'
+                                     )));
+ 
+                             $cacheFileContent = file_get_contents($filename, FALSE, $remoteURLContext);
+                             if ($cacheFileContent === FALSE)
+                                 $cacheFileContent = 'FGC_ERROR';
+                         }
+ 
+                         // Limit size
+                         if (strlen($cacheFileContent) > 1048576) // 1 MB
+                             $cacheFileContent = 'MAX_SIZE';
+ 
+                         // Write cache file
+                         //file_put_contents($cacheFile, $cacheFileContent, LOCK_EX);
+                         $cacheFileHandle = @fopen($cacheFile, 'wb');
+                         if ($cacheFileHandle !== FALSE) {
+                             if (flock($cacheFileHandle, LOCK_EX)) {
+                                 fwrite($cacheFileHandle, $cacheFileContent);
+                                 fflush($cacheFileHandle);
+                                 flock($cacheFileHandle, LOCK_UN);
+                             }
+                             fclose($cacheFileHandle);
+                         }
+                     }
+ 
+                     if (file_exists($cacheFile) && (@filesize($cacheFile) > 64))
+                         return TRUE;
+                 }
+             }
+         }
+ 
          ##  find the image referenced and see if it's on the server
        $imageroot = getConfig('uploadimageroot');
  #      cl_output('filesystem_image_exists '.$docroot.' '.$filename);
***************
*** 469,474 ****
--- 570,583 ----
  
      public function get_filesystem_image($filename)
      {
+         if (defined('EMBEDEXTERNALIMAGES') && EMBEDEXTERNALIMAGES) {
+             $extCacheDir = $GLOBALS['tmpdir'].'/external_cache';
+             $cacheFile = $extCacheDir.'/'.$this->messageid.'_'.preg_replace(array('~[\.][\.]+~i', '~[^\w\.]~i'), array('', '_'), $filename);
+ 
+             if (file_exists($cacheFile) && (@filesize($cacheFile) > 64))
+                 return base64_encode(file_get_contents($cacheFile));
+         }
+ 
          ## get the image contents
        $localfile = basename(urldecode($filename));
  #      cl_output('get file system image'.$filename.' '.$localfile);

bertpoort

30-01-16 10:50

reporter   ~0057476

If you'd like I can make a pull request on GitHub for this patch.

gingerling

30-01-16 10:54

manager   ~0057477

Hi, that would be amazing - I was just about to do it for you as a courtesy, but I am at a conference and a bit distracted :) Thanks, Ax

bertpoort

30-01-16 10:58

reporter   ~0057478

Done :)

gingerling

30-01-16 12:34

manager   ~0057479

great, code review next I guess, but a few people are away till Tuesday for the conference so might not be till then :)

michiel

01-02-16 12:46

manager   ~0057481

Nice one, thanks. As you put a flag around, I've just merged it, and we can safely run some tests. It will not affect the general public and for now only be known to people who are aware.

It might be useful to write to the developers list to explain and ask for some feedback.

michiel

01-02-16 20:23

manager   ~0057483

https://github.com/phpList/phplist3/pull/44/files

michiel

20-02-16 19:30

manager   ~0057528

PR update https://github.com/phpList/phplist3/pull/46

Some new configs were added, needs documenting.

gingerling

22-05-16 15:37

manager   ~0057729

Am not sure how this works from a users perspective right now, is there a way for someone to use this through the normal UI or is it more of a back-end only change?

gingerling

23-05-16 15:08

manager   ~0057730

ah, got it, needs config. Works great!