Module: Store Files on Disk

All files from this thread

File Name	File Size		Posted by	Date
store_files_on_disk-1.1.1.tar.gz	5.8 KB	open | download	Maurice Makaay	04/23/2010	Read message
store_files_on_disk-1.1.1.zip	7.7 KB	open | download	Maurice Makaay	04/23/2010	Read message

This module takes care of storing file data (attachments, personal user files) on disk instead of using the database as the file storage. This is an advanced module for users that have a problem with storing files in the database. Don't just do this because you read on a blog somewhere that putting files in a database is bad.

This module contains an on-the-fly conversion mechanism, which will convert files that are stored in the database to disk storage, right at the moment that they are requested. This mechanism will gradually convert the available files to the disk storage.

If you want to forcibly convert all available files to file storage, then you can use the conversion script console_convert.php that you can find in the module directory. This script's intended use it to run it from the system command prompt, using the PHP interpreter. See the module's README file for more info.

Changelog:
----------

2010-04-23 1.1.1

    - Changed the file path generation code to use gmstrftime() instead
      of strftime(). Because strftime() dependens on the server timezone,
      changing the server timezone resulted in changed file storage path
      and broken file downloads as result.

      Files will be automatically converted to the new storage path on
      access. It is possible to use the console_convert.php script
      (which is a command line script) to speed up this process.

      Note:
      The automatic conversion from the 1.1.0 release will also
      be performed, so you can upgrade directly to this release when
      coming from a pre-1.1.0 version.

2009-07-10 1.1.0

    - Modified the path that is used for storing files on disk. Before this
      version, we used a deep hashing tree, based on the MD5 sum of the
      file_id. This provided perfect distribution of the files, however it
      has made several hosting providers crazy, because their backup systems
      could not cope with the file tree traversal.
      Additionally, some Phorum owners would like to purge old stored files
      once in a while. The hashing mechanism is not ideal for doing this,
      because old and new files will be intermixed in the directories.

      The old storage path looked like:
      /path/to/store/XXX/XXX/XXX/XXX/XXX/XXX/XXX/XXX/XXX/XXX/<file_id>
      (where the X-es are determined by the file_id's MD5 sum)

      The new storage path looks like:
      /path/to/store/YYYY/MMDD/HH/<file_id>

      Files will be automatically converted to the new storage path on
      access. It is possible to use the console_convert.php script
      (which is a command line script) to speed up this process.

      Note that this change required some changes in the Phorum core
      as well. This means that Phorum needs to be upgraded to at least
      version 5.2.12 to be able to use version 1.1.0 of this module.

2007-09-24 1.0.1

    - Implemented more robust file writing to prevent file data loss
      on storage problems.

2007-09-15 1.0.0

    - Initial release.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce



Edited 3 time(s). Last edit at 04/23/2010 04:40PM by Maurice Makaay.

Quote

Please, enter the filesystem path (either relative or absolute) of the directory that you want to use for storing the Phorum files. This path does not need to be below the webserver's document root (it is even preferable to have it outside the document root). The webserver must be allowed to write to the directory.

I'm not getting this. I am on a shared hosting server and thought it would be to the tune of something like this...

/below_public_view/web_view_dot_com/forum_subdirectory/files_stored

But I am getting a 'directory doesn't exist' (but it does) error.

- Bob

The directory probably does not exist, because you are referring to the root of the filesystem by starting with a "/" and I am sure that they didn't setup your data right at the root dir. Try simply "files_stored" and make that directory writable for the webserver. The module should be able to run with this relative path.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce

Thank you, it worked. Simply typing the name of a directory (without any slashes) located in my forum directory worked great.

Although, I was hoping to see a linkable copy of an image from one of my test posts, but instead saw a massive series of subdirectories with information at the end. But although I cannot link to the file uploads, I suppose this will be useful should I change servers and cannot get my sql database from my host.

- Bob

could you elaborate on

Quote

but instead saw a massive series of subdirectories with information at the end.

I don't get what you mean. Maybe post an example of what you got.

---

Thomas Seifert

He's talking about the hashing system that is used for storing the files on disk. This file storage module was not designed to upload the files with the same name as they were uploaded to disk. It was designed to store the data anywhere on the filesystem (so possibly outside the website's document root even) and still retrieve it through the Phorum file API. Because a lot of files in a single directory makes the system work very slowly and even might exceed the maximum number of allowed files in a direcxtory, we use the hashing system where we distribute the uploaded files over a deep file tree.

One very important reason is a security reason. If you would store the files using their real names somewhere inside the document root and have not put good restrictions on the type of files that can be uploaded, malicious users could try to upload *.cgi, *.php, *.asp or other interpreted script files. When accessing those directly through the server, they would not be downloaded but executed instead. That would be a huge security hole, since that would provide a really easy way to run arbitrary code on the server. The solution would be to not allow *.php files from being uploaded, but then forums like the modules forum could no longer be used to upload *.php files anymore, which would be a real loss.

Next to that, for making the file downloads honor Phorum's security settings, the best thing is to use a directory outside the document root for storing the files. That way, there is no way to access them directly through the webserver, but instead downloads are always handled by the file.php script. That script will only let the user download the file if he/she has read rights for the file. This can for example be important in case there is a forum which not all users can read. You wouldn't want restricted users to be able to read the attachment files for that forum directly.

Because of the reasons above, only for forums that are fully open for reading and which are strictly configured to only allow uploads of safe file types, it could work to setup a download link methodology like you were looking for. I see that as a special situation though, for which I did not intend the module and a situation that I do not want to burn my fingers on security-wise. The module's goal is to take the file data out of the database, so very large forums won't get a Phorum files table that exceeds many gigabytes of data. It does a great job at that.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce

To sum up what Maurice said in a couple of sentences: All the reasons he has to jump through hoops to store files on disk is why Phorum does not do this by default. Putting stuff on disk from a forum is not easy and can cause lots of problems for the casual user. This is an advanced module for users that have a problem with storing files in the database. Don't just do this because you read on a blog somewhere that putting files in a database is bad.

Brian - Cowboy Ninja Coder - Personal Blog - Twitter

hi,

I have a virtual hosting plan which does not give much for database allowance. And the main activity on the forum I am setting up will be picture sharing. Therefore with ~500K per file, I believe I will get complaints soon. Therefore I need this module badly. Thank you for writing this.

Though one problem, I loaded this module, activated it, setup the directory. Then nothing happens. Both the old and new picture files are still in the database. Refreshed all the old posts to get the "on-the-fly conversion mechanism" to work. No luck.

I wonder if you can give a bit more guidance for how to get this to work? Did I miss anything here? Thanks.

Nonmem

There is not much to miss here. Just make sure that the path for storing files is correct and that the webserver is allowed to write to that directory. The module implements the writing of the module data to disk very paranoia and if anything goes wrong, it will keep the data in the database, so I asume that something is going wrong now. Maybe you could enable the event logging module and retrieve some attachments to fire the on-the-fly conversion. Then check the event log to see if there are errors in there which explain the behavior.

Also, if you look at the storage directory, do you see any subdirectories created at all? This module will setup a directory tree and store the files in that tree. If there aren't subdirectories in there, then things go wrong at a very early stage. The server might not be able to write to the directory or some paranoia PHP configuration by your hosting provider might prevent directories from being created in there.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce

Thank you for the reply.

The directory is, as you guessed, empty.

It has been chmod to 777, therefore I guess it is the PHP configuration. Will enable the logging module. Hopefully I can get this figured out.

Best regards.

[well, I do need more help. can not find the logging module you mentioned in the module index. What is the full name? thanks. ]



Edited 1 time(s). Last edit at 10/21/2007 06:02PM by Nonmem.