Module: Store Files on Disk

By installing the pecl imagick extension, the imagemagick sytsem will be used instead of GD. That can be arranged for without hacking thhe code. Would that be an option for you?

I've seen occasional errors like these too with GD. It seems to be dependent on the contents of the image and something (possibly a bug) in GD is blowing up.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce

*self-deleted*



Edited 1 time(s). Last edit at 06/24/2018 05:49PM by Voltius.

the file entry will still be in the database but the file_data (or whatever this column was called) will be empty. so the meta-data about the file needs to be in the db.
the error is about the embed_images module and I don't know what you expect in the event_logging about the store_files_on_disk module, it doesn't log anything there.

---

Thomas Seifert

*self-deleted*



Edited 1 time(s). Last edit at 06/24/2018 05:49PM by Voltius.

*self-deleted*



Edited 1 time(s). Last edit at 06/24/2018 05:49PM by Voltius.

The table might have been large, but it doesn't fully correlate to the exact amount of data in there. The only way to really check what's in a database table, it looking at the table contents. I bet that last night the data field was already empty in there.

What does happen with database systems (each in their own way), is that table files become sparse when doing record deletes and/or updates. This means that when deleting a record, it might leave an empty spot in the middle of the table file. When updating a record, the system might add a new record (making the table bigger) and deleting the old record, also leaving a gap.

To prevent table from growing endlessly, database systems can do various things. One thing could be to fill up the gaps with new records, to reuse the available space. Another thing that can be done (this is probably what happened in your case) is optimization of the table.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce

I just released version 1.1.0 of this module. Note that I have 1.0.1 available as well. For now, I recommend using version 1.0.1. I recommend this because that one is compatible with the current stable version of Phorum 5.2.11, while 1.1.0 requires still-to-be-released Phorum version 5.2.12. Also, 1.1.0 has not been tested extensively (although I think that it should be stable).

The big change for this release is a modified file storage path. We have had some complaints that the hashing algorithm that we used for distributing uploaded attachments over multiple directories. This provided perfect distribution of the files, however it has made several hosting providers crazy, because their backup systems could not cope with the file tree traversal.

The old storage path looked like:
/path/to/store/XXX/XXX/XXX/XXX/XXX/XXX/XXX/XXX/XXX/XXX/<file_id>
(where the X-es are determined by the file_id's MD5 sum)
This distribution guarantees that there will never be more than 4096 entries in a single directory (which is a hard limit on some OS-es).

The new storage path looks like:
/path/to/store/YYYY/MMDD/HH/<file_id>
This distribution gives us a directory per year, with at most 366 subdirectories for the days in the year. Each day directory gets at most 24 hour subdirectories. Now, as long as the users are not posting more than 4096 attachments per hour, this distribution should work very well.

Files will be automatically converted to the new storage path on access. It is possible to use the console_convert.php script (which is a command line script) to speed up this process.

Note that this change required some changes in the Phorum core as well. This means that Phorum needs to be upgraded to at least version 5.2.12 to be able to use version 1.1.0 of this module. If you want to give it a try before 5.2.12 is released, then download an svn snapshot of the stable tree or use an svn checkout directly.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce

Quote
Maurice Makaay
I just released version 1.1.0 of this module. Note that I have 1.0.1 available as well. For now, I recommend using version 1.0.1. I recommend this because that one is compatible with the current stable version of Phorum 5.2.11, while 1.1.0 requires still-to-be-released Phorum version 5.2.12. Also, 1.1.0 has not been tested extensively (although I think that it should be stable).

The big change for this release is a modified file storage path. We have had some complaints that the hashing algorithm that we used for distributing uploaded attachments over multiple directories. This provided perfect distribution of the files, however it has made several hosting providers crazy, because their backup systems could not cope with the file tree traversal.

The old storage path looked like:
/path/to/store/XXX/XXX/XXX/XXX/XXX/XXX/XXX/XXX/XXX/XXX/<file_id>
(where the X-es are determined by the file_id's MD5 sum)
This distribution guarantees that there will never be more than 4096 entries in a single directory (which is a hard limit on some OS-es).

The new storage path looks like:
/path/to/store/YYYY/MMDD/HH/<file_id>
This distribution gives us a directory per year, with at most 366 subdirectories for the days in the year. Each day directory gets at most 24 hour subdirectories. Now, as long as the users are not posting more than 4096 attachments per hour, this distribution should work very well.

Files will be automatically converted to the new storage path on access. It is possible to use the console_convert.php script (which is a command line script) to speed up this process.

Note that this change required some changes in the Phorum core as well. This means that Phorum needs to be upgraded to at least version 5.2.12 to be able to use version 1.1.0 of this module. If you want to give it a try before 5.2.12 is released, then download an svn snapshot of the stable tree or use an svn checkout directly.

Awesome!

My backbone provider will be kissing your feets! Ha!

I will give it a try on Sunday night when there no traffic and I will put phorum in maintance mode so I could back up my existing file storage before trying the new module.

Thank for making it possible!

Bill

I tried it on my smaller phorum site since it easier to back up and do some testing with the new module.

So far it test out fine and working but I did get a bunch of warning when using the convert console.

53 similar warning below but I am copying and pasting the last warning in the post

Warning: unlink(./attachments/f4b/9ec/30a/d9f/68f/89b/296/397/86c/b62/94): Permission denied in /home/stlaware/public_html/mods/store_files_on_disk/store_files_on_disk.php on line 137
converting 53 / 53 [====================] (100%)

Then I went and deleted all the folders with old file path but left the ones with /YYYY/ alone .. I have 4 of them /2005/ /2006/ /2007/ /2009/

So far it all seem working, so then it ok to ignore those warnings and delete the old path folder manually?


Bill


Edited to add .....

I just tried this on my second phorum site and using root this time to run the script and no warning. So it seem you need to be root to run this convert script.



Edited 1 time(s). Last edit at 07/20/2009 02:46AM by CBiLL.

Well, you need file moving and directory deleting permissions, so when you are root, it is likely to work.
It is best to either to a chmod 777 on dirs and 666 on files in the file data tree or (better IMO) 770 on dirs and 660 on files and make sure that all the files and directory are chown-ed to the web server user and/or group. That way you can make sure that the webserver can always access the files.

Good to see that the change seems to work out for you. Those YYYY directories look really good (my biggest fear would be that a new directory would pop up with 1970/0101/... for the files).

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce