Problem retrieving data using HTTP request with socket
Posted by Oliver Riesen-Mallmann
Problem retrieving data using HTTP request with socket May 15, 2008 09:18AM |
Admin Registered: 18 years ago Posts: 744 |
I tried to use Embed Images Module from Maurice on my Phorum but I got a lot of error messages. First I thought it's a problem of Maurice's module but it seems, that I have a strange behavior in http_get.php related to invalid(?) HTTP responses.
So since it's not a problem of the module, I'll explain the problem here in a new thread. Please read before the history starting here.
The Embed Images Module gets an image by a HTTP request, stores that image on a local storage and gets image properties. HTTP request is made by phorum_api_http_get in http_get.php. On my server it uses the extension "sockets".
The function phorum_api_http_get returns the body of the HTTP request in this case the requested image. But it seems that in same cases the body not only contains the image but also some extra data like file length.
To show the problem I worked with these two images:
1. [images.spritmonitor.de] (original image causing the problem)
2. [www.riesen.org] (same image stored on my own web server)
To compare the result of the HTTP request I putted the following code just after closing the socket connection (line 289 in http_get.php):
Here is the result:
You can see, that after the end of the header lines (first occurrence of "\r\n\r\n") there is some extra data ("131") in the first images just before starting the content of the png image.
For the moment I don't know if it is a problem of file_get_contents, or lighttpd, or the script from spritmonitor with produces the image, or my socket connection... Perhaps you have same idea.
To avoid the problem I changed phorum_api_http_get to use file_get_contents which is working well.
Regards
Oliver
Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
So since it's not a problem of the module, I'll explain the problem here in a new thread. Please read before the history starting here.
The Embed Images Module gets an image by a HTTP request, stores that image on a local storage and gets image properties. HTTP request is made by phorum_api_http_get in http_get.php. On my server it uses the extension "sockets".
The function phorum_api_http_get returns the body of the HTTP request in this case the requested image. But it seems that in same cases the body not only contains the image but also some extra data like file length.
To show the problem I worked with these two images:
1. [images.spritmonitor.de] (original image causing the problem)
2. [www.riesen.org] (same image stored on my own web server)
To compare the result of the HTTP request I putted the following code just after closing the socket connection (line 289 in http_get.php):
return phorum_api_error_set (PHORUM_ERRNO_ERROR, 'response='.str_replace("\n", '\n', str_replace("\r", '\r', $response)).'EOF');
Here is the result:
You can see, that after the end of the header lines (first occurrence of "\r\n\r\n") there is some extra data ("131") in the first images just before starting the content of the png image.
For the moment I don't know if it is a problem of file_get_contents, or lighttpd, or the script from spritmonitor with produces the image, or my socket connection... Perhaps you have same idea.
To avoid the problem I changed phorum_api_http_get to use file_get_contents which is working well.
Regards
Oliver
Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Re: Problem retrieving data using HTTP request with socket May 15, 2008 09:53AM |
Admin Registered: 19 years ago Posts: 8,532 |
I found the problem. I didn't notice the Transfer-Encoding: chunked before. Chunked encoding is an HTTP 1.1 invention to send arbitrary sized data to the client. I didn't expect http_get to ever need that, but apparently spritmonitor is sending their tiny image as chunked already. I'll have to implement the chunked protocol in the http_get API code.
Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
Re: Problem retrieving data using HTTP request with socket May 15, 2008 10:11AM |
Admin Registered: 18 years ago Posts: 744 |
Hi Maurice,
I'm happy that the problem is found!
I took the image from spritmonitor like example but I have I lot of other pictures in my Phorum with same problem...
Regards
Oliver
Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Quote
Maurice Makaay
I didn't expect http_get to ever need that, but apparently spritmonitor is sending their tiny image as chunked already.
I'm happy that the problem is found!
I took the image from spritmonitor like example but I have I lot of other pictures in my Phorum with same problem...
Regards
Oliver
Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Re: Problem retrieving data using HTTP request with socket May 15, 2008 10:13AM |
Admin Registered: 19 years ago Posts: 8,532 |
A lot of script generated images then? Static images normally aren't delivered using chunked encoding, because the server knows their size when sending thim. Working on a fix now. Chunked encoding is ugly, but not too difficult to handle.
Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
Re: Problem retrieving data using HTTP request with socket May 15, 2008 10:33AM |
Admin Registered: 19 years ago Posts: 8,532 |
Please check if my crude chunked transfer decoding works for you.
See changeset 3297 for the updated code.
Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
See changeset 3297 for the updated code.
Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
Re: Problem retrieving data using HTTP request with socket May 15, 2008 10:42AM |
Admin Registered: 18 years ago Posts: 744 |
Hi Maurice,
Yes, it's working. Thanks!
Regards
Oliver
Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Quote
Maurice Makaay
Please check if my crude chunked transfer decoding works for you.
Yes, it's working. Thanks!
Regards
Oliver
Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Re: Problem retrieving data using HTTP request with socket May 15, 2008 05:37PM |
Admin Registered: 23 years ago Posts: 4,495 |
Yes, whatever is sending these images is doing so badly. The Phorum code does not claim to accept chunked encoding when making the request. Therefore, the server in question should not be returning chunked data. It was nice of Maurice to work around an error in your image delivery system. =)
Brian - Cowboy Ninja Coder - Personal Blog - Twitter
Re: Problem retrieving data using HTTP request with socket May 15, 2008 05:58PM |
Admin Registered: 23 years ago Posts: 4,495 |
Maurice pointed out that he was already using HTTP 1.1 in his code. So, I guess this was his bad all along. =)
Brian - Cowboy Ninja Coder - Personal Blog - Twitter
Re: Problem retrieving data using HTTP request with socket May 16, 2008 02:25AM |
Admin Registered: 18 years ago Posts: 744 |
Hi Brian,
But it seems that it is some settings in PHP or Apache. Because Phorum itself returns chunked data on my server.
Example: My avatar file on phorum.org:
An attachment in my test forum:
Not my image delivery system ;-)
regards
Oliver
Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Quote
Brian Moon
Therefore, the server in question should not be returning chunked data.
But it seems that it is some settings in PHP or Apache. Because Phorum itself returns chunked data on my server.
Example: My avatar file on phorum.org:
[www.phorum.org] Date: Fri, 16 May 2008 06:18:43 GMT Server: Apache X-Powered-By: PHP/5.2.6RC4-pl0-gentoo Set-Cookie: (removed) content-disposition: filename="volkswagen-microbus.jpg" Content-Length: 3741 Content-Type: image/jpeg
An attachment in my test forum:
[phorum.langzeittest.de] Date: Fri, 16 May 2008 06:17:35 GMT Server: Apache/1.3.27 (Unix) PHP/4.4.2 FrontPage/5.0.2.2510 mod_ssl/2.8.11 OpenSSL/0.9.6c X-Powered-By: PHP/4.4.2 Set-Cookie: (removed) content-disposition: filename="190058_3.png" Transfer-Encoding: chunked Content-Type: image/png
Quote
Brian Moon
It was nice of Maurice to work around an error in your image delivery system. =)
Not my image delivery system ;-)
regards
Oliver
Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Re: Problem retrieving data using HTTP request with socket May 16, 2008 04:07AM |
Admin Registered: 19 years ago Posts: 8,532 |
One difference might be that you run apache 1.3.27 and phorum.org runs apache 2.
I don't know the 1.3.27 code that well anymore, but in apache 2, a lot of automatic handling is done for chunked or unchunked data sending. If you do not send a Content-Length header from the script, then apache 1.3.27 might be sending chunked at all times. In Apache 2, a Content-Length header is added when apache sees the end of the data to send while not having sent any data to the client yet. If data is to be sent to the client when the end of the data has not yet been reached, then chunked encoding will be used.
Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
I don't know the 1.3.27 code that well anymore, but in apache 2, a lot of automatic handling is done for chunked or unchunked data sending. If you do not send a Content-Length header from the script, then apache 1.3.27 might be sending chunked at all times. In Apache 2, a Content-Length header is added when apache sees the end of the data to send while not having sent any data to the client yet. If data is to be sent to the client when the end of the data has not yet been reached, then chunked encoding will be used.
Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
Sorry, only registered users may post in this forum.