Firefox PHP

Problem retrieving data using HTTP request with socket

Problem retrieving data using HTTP request with socket
May 15, 2008 09:18AM
I tried to use Embed Images Module from Maurice on my Phorum but I got a lot of error messages. First I thought it's a problem of Maurice's module but it seems, that I have a strange behavior in http_get.php related to invalid(?) HTTP responses.

So since it's not a problem of the module, I'll explain the problem here in a new thread. Please read before the history starting here.

The Embed Images Module gets an image by a HTTP request, stores that image on a local storage and gets image properties. HTTP request is made by phorum_api_http_get in http_get.php. On my server it uses the extension "sockets".

The function phorum_api_http_get returns the body of the HTTP request in this case the requested image. But it seems that in same cases the body not only contains the image but also some extra data like file length.

To show the problem I worked with these two images:

1. [images.spritmonitor.de] (original image causing the problem)
2. [www.riesen.org] (same image stored on my own web server)

To compare the result of the HTTP request I putted the following code just after closing the socket connection (line 289 in http_get.php):

return phorum_api_error_set
  (PHORUM_ERRNO_ERROR,
    'response='.str_replace("\n", '\n', str_replace("\r", '\r', $response)).'EOF');

Here is the result:



You can see, that after the end of the header lines (first occurrence of "\r\n\r\n") there is some extra data ("131") in the first images just before starting the content of the png image.

For the moment I don't know if it is a problem of file_get_contents, or lighttpd, or the script from spritmonitor with produces the image, or my socket connection... Perhaps you have same idea.

To avoid the problem I changed phorum_api_http_get to use file_get_contents which is working well.

Regards
Oliver


Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Re: Problem retrieving data using HTTP request with socket
May 15, 2008 09:53AM
I found the problem. I didn't notice the Transfer-Encoding: chunked before. Chunked encoding is an HTTP 1.1 invention to send arbitrary sized data to the client. I didn't expect http_get to ever need that, but apparently spritmonitor is sending their tiny image as chunked already. I'll have to implement the chunked protocol in the http_get API code.


Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
Re: Problem retrieving data using HTTP request with socket
May 15, 2008 10:11AM
Hi Maurice,

Quote
Maurice Makaay
I didn't expect http_get to ever need that, but apparently spritmonitor is sending their tiny image as chunked already.

I'm happy that the problem is found!

I took the image from spritmonitor like example but I have I lot of other pictures in my Phorum with same problem...

Regards
Oliver


Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Re: Problem retrieving data using HTTP request with socket
May 15, 2008 10:13AM
A lot of script generated images then? Static images normally aren't delivered using chunked encoding, because the server knows their size when sending thim. Working on a fix now. Chunked encoding is ugly, but not too difficult to handle.


Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
Re: Problem retrieving data using HTTP request with socket
May 15, 2008 10:33AM
Please check if my crude chunked transfer decoding works for you.
See changeset 3297 for the updated code.


Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
Re: Problem retrieving data using HTTP request with socket
May 15, 2008 10:42AM
Hi Maurice,

Quote
Maurice Makaay
Please check if my crude chunked transfer decoding works for you.

Yes, it's working. Thanks!

Regards
Oliver


Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Re: Problem retrieving data using HTTP request with socket
May 15, 2008 05:37PM
Yes, whatever is sending these images is doing so badly. The Phorum code does not claim to accept chunked encoding when making the request. Therefore, the server in question should not be returning chunked data. It was nice of Maurice to work around an error in your image delivery system. =)

Brian - Cowboy Ninja Coder - Personal Blog - Twitter
Re: Problem retrieving data using HTTP request with socket
May 15, 2008 05:58PM
Maurice pointed out that he was already using HTTP 1.1 in his code. So, I guess this was his bad all along. =)

Brian - Cowboy Ninja Coder - Personal Blog - Twitter
Re: Problem retrieving data using HTTP request with socket
May 16, 2008 02:25AM
Hi Brian,

Quote
Brian Moon
Therefore, the server in question should not be returning chunked data.

But it seems that it is some settings in PHP or Apache. Because Phorum itself returns chunked data on my server.

Example: My avatar file on phorum.org:

[www.phorum.org]

Date: Fri, 16 May 2008 06:18:43 GMT
Server: Apache
X-Powered-By: PHP/5.2.6RC4-pl0-gentoo
Set-Cookie: (removed)
content-disposition: filename="volkswagen-microbus.jpg"
Content-Length: 3741
Content-Type: image/jpeg

An attachment in my test forum:

[phorum.langzeittest.de]

Date: Fri, 16 May 2008 06:17:35 GMT
Server: Apache/1.3.27 (Unix) PHP/4.4.2 FrontPage/5.0.2.2510 mod_ssl/2.8.11 OpenSSL/0.9.6c
X-Powered-By: PHP/4.4.2
Set-Cookie: (removed)
content-disposition: filename="190058_3.png"
Transfer-Encoding: chunked
Content-Type: image/png

Quote
Brian Moon
It was nice of Maurice to work around an error in your image delivery system. =)

Not my image delivery system ;-)

regards
Oliver


Using Phorum since 7/2000: forum.langzeittest.de (actual version 5.2.23)
Modules "Made in Germany" for version 5.2: Author_as_Sender, CarCost, Close_Topic, Conceal_Message_Timestamp,
Format_Email, Index_Structure, Mailing_List, Pervasive_Forum, Spritmonitor, Terms_of_Service and German_Language_Files_Package.
Re: Problem retrieving data using HTTP request with socket
May 16, 2008 04:07AM
One difference might be that you run apache 1.3.27 and phorum.org runs apache 2.

I don't know the 1.3.27 code that well anymore, but in apache 2, a lot of automatic handling is done for chunked or unchunked data sending. If you do not send a Content-Length header from the script, then apache 1.3.27 might be sending chunked at all times. In Apache 2, a Content-Length header is added when apache sees the end of the data to send while not having sent any data to the client yet. If data is to be sent to the client when the end of the data has not yet been reached, then chunked encoding will be used.


Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce
Sorry, only registered users may post in this forum.

Click here to login