[Suggestion] Keep read status flag when moving messages

Hello,

I have a suggestion for future versions. I hope this is the right place to put it.
I move threads everyday due to lazy users posting on wrong subforum. But when I move the thread, i think it changes the thread id so it looses read status and appears unread again, confusing the users.

I know all this philosophy of "make a module as desired", but I really thing this is an usability issue, as a message should keep the read/unread flag regardless if it is moved or not.

What do you think?

No, they are not getting a new id but the newflags code in moving messages has its flaws.

Comment from Maurice about that code:

Language: PHP
         /**
         * @todo In the move thread code, there are some flaws. The
         *       newflags for the user that is moving the message
         *       are used as the source for deciding what flags
         *       to delete or move for all other users. This results
         *       in strange newflag problems.
         *
         *       This main issue here is that the newflags should be
         *       handled separately for each user; no updates should be
         *       based on the newflags for the active user. The current
         *       algorithm will only make sure that the newflags will look
         *       correct for that specific user. The problem is that we
         *       do not yet have an idea on how to handle this with
         *       enough performance.
         */

---

Thomas Seifert



Edited 2 time(s). Last edit at 03/23/2009 06:28PM by Thomas Seifert.

oh thanks for reply

i see it has no easy solution, and i appreciate the reply, but in a forum like mine (moving many threads a day) users get really frustrated.
even considering disabling completely the read flag just because of this....

thank you anyway for details. I appreciate the support

Getting this to perform will is the big issue here. Keeping tabs on who has read what messages for what forum is quite a daunting task for big forums and we have not yet come up with a solution for that.

I know that this behavior can confuse users. The only advice that I can give on that is to educate users to post in the correct forums and to move messages as quickly as possible. This needs a rather active approach to moderation, which might not be possible for you. But until we or somebody else comes up with a smart solution for the read message tracking, it's the only option available.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce

Quote
Diego
but in a forum like mine (moving many threads a day) users get really frustrated.

Hehe, use it as a peer pressure for the lazy users.

---
-=[ Panu ]=-

thanks for suggestions
I always try to keep active moderations, but as you mentioned, is not always possible. And when you have an active forum (around 700 new messages a day) and have fast read times and move so many threads, at the end of the day it is an annoying issue you can't keep the read flag.

But i understand the technical issues, so no prob

Thanks

I'm just working through this one in my own mind so please feel free to correct me at any point in my logic.

When moving a thread to a new forum, we need to change the following in the user_newflags table, repeated for each message in the thread:

Language: SQL
UPDATE user_newflags SET forum_id = {new_forum_id} WHERE message_id = {moved_message_id}

I take it there can be a performance hit when this is done for 5000 users who have read the thread and even more so when those 5000 users have read all 150 messages in the thread. This would lead to updating 750,000 rows of the table.

So, to avoid this performance hit, could we have two tables:
user_newflags:
user_id, message_id

and
forum_newflags:
forum_id, message_id

When moving messages, only one row would need to be updated, though again repeated for each message in the thread:

Language: SQL
UPDATE forum_newflags SET forum_id = {new_forum_id} WHERE message_id = {moved_message_id}

When checking for new flags, a join would be used to find the proper newflags for the forum:

Language: SQL
SELECT user_newflags.user_id, user_newflags.message_id, forum_newflags.forum_id
FROM user_newflags
LEFT JOIN forum_newflags ON user_newflags.message_id = forum_newflags.message_id
WHERE user_newflags.user_id ={current_user_id}
AND forum_newflags.forum_id ={current_forum_id}

This structure would allow the thread to be moved and the newflags to follow for every user without any extra database work.

This could be simplified further by adding a parent_message_id column to the user_newflags table and only linking that id to the message_id column in the forum_newflags table. This would add extra data to the user_newflags table but would actually reduce data in the forum_newflags table as only one row would be necessary per thread as opposed to one row per message. This would also speed up the moving process as only the parent_message_id would need to be updated in the forum_newflags table.

I could easily be missing something here, but I hope this might help.

---

Joe Curia (aka Azumandias)
Modules: l0Admin Mass Email00000000l000000Automatic Time Zones000ll.l00000Enhanced Custom Profiles0.00Google Calendar0000l.l000000Post Previews
000000000Admin Security Suite000000000000Check Modules for Upgrades0000External Authentication000000Group Auto-Email00000.00000Private Message Alerts
000000000Attachment Download Counter0000Custom Attachment Icons000ll.ll00Favorite Forums000000.00000Highlighted Search Terms0000Self-Delete Posts Option
000000000Attachment Watermarks0l00000000Custom Language Database00l.l.0Forum Lockdown00000.00000Ignore Forums0000000000000Threaded Tree View
000000000Automatic Message Pruning00.llll.00Easy Color Scheme Manager0l.l00Forum Subscriptions0000lll000Moderated User Group
Templates:lGeneric Integration000000000 0000Simple Rounded000000 00000000Tabbed Emerald

The tiny thing that you are missing here, is the fact that the oldest message that is in the table for a forum implicitly flags all messages that are older in that forum as read. Therefore, you cannot simply move the message to a different forum alone.

Example messages:

     forum 1      forum 2
       msg1(X)
                    msg2
       msg3(X)       
       msg4(*)
       msg5
                    msg6
       msg7(*)

(*) = message that is read and has an entry in the newflag table
(X) = message that is implicitly marked read, because it's older than the oldest read message for the forum (msg4).

Now if msg4 is moved to forum2, we get this:

     forum 1      forum 2
       msg1(X)
                    msg2
       msg3(X)       
                    msg4(*)
       msg5(X)
                    msg6
       msg7(*)

The message msg5 is now implicitly marked read too, because msg7 is now the oldest read message in the newflags table for forum1.

So in the current architecture, making this work correctly would not only involve the work of moving the readflags for a message to a different forum, but also checking for each user if it's the oldest read message and inserting an appropriate newflag to make things right if this is the case.

One solution direction that I have been thinking about, is not letting the newflags table determine the min_id (that's how it's called in the code), but adding an extra table that is dedicated to storing min_ids for the users. That way, we'd have a quick lookup table to see what we're dealing with and a safer system for moving the newflags to the new forum.

We'd have to do some live tests to see if your idea to store the forum_id outside the newsflags would work, performance-wise. For moving purposes it would definitely work. For the standard case of keeping track of unread messages, I'm not sure if the extra join wouldn't slow down the queries too much.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce

i don't know if i am saying something crazy, but even if this a demanding operation, isnt in this case "less" important as you only slow down the performance when you moving the thread? I mean, everyday operation (read, reply, register) wouldnt suffer from this. After all, only really busy forums would need to move more than few threads everyday.

In my opinion I wouldn't mind paying this price if that solves the read issue.

Am I wrong here?

I agree that it's better to bother moderators than to bother end users.

Note however that heavy db work will always affect end users as well. So if a move thread process is hacking away on the database server, the users could experience performance impact.

But more important than this: we should be aware that we could be dealing with systems with a lot of users. When moving threads on such system, other system limits might be kicking in. Think about PHP execution time for example. The only correct way to implement a system that can handle an arbitrary amount of users is to implement an interface that can handle the updates in batches, to circumvent execution time issues (visually presented as a progress bar to the person that is moving a message). With our new API layer code, this kind of functionality is now in reach, so it might be feasible to handle an interface like this. Not a high priority todo item though.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce

Quote
Maurice Makaay
The tiny thing that you are missing here, is the fact that the oldest message that is in the table for a forum implicitly flags all messages that are older in that forum as read. Therefore, you cannot simply move the message to a different forum alone.

Example messages:

     forum 1      forum 2
       msg1(X)
                    msg2
       msg3(X)       
       msg4(*)
       msg5
                    msg6
       msg7(*)

(*) = message that is read and has an entry in the newflag table
(X) = message that is implicitly marked read, because it's older than the oldest read message for the forum (msg4).

Now if msg4 is moved to forum2, we get this:

     forum 1      forum 2
       msg1(X)
                    msg2
       msg3(X)       
                    msg4(*)
       msg5(X)
                    msg6
       msg7(*)

The message msg5 is now implicitly marked read too, because msg7 is now the oldest read message in the newflags table for forum1.

I knew I must be missing something :-P I think the issue you have pointed out could be solved by simply duplicating the new flag in the forum_newflags table but with the new forum_id. Thus:

     forum 1      forum 2
       msg1(X)
                    msg2
       msg3(X)       
       msg4(*)      msg4(*)
       msg5( )
                    msg6
       msg7(*)

However, this would then cause a problem for forum 2. If the users had never read messages in that forum, they would suddenly see new flags for msg6 and beyond.

So, still brainstorming. What if we also add a user profile field to record each forum a user has read (ie $user["read_forums"][{forum_id}]) which is set to the id of the first message user reads a in a particular forum. Then, if msg4 is moved to the new forum, it is ignored unless the user has already read another message in the forum because the new flags will check for the earliest message id provided there is a $user["read_forums"] id for that forum and the earliest message id is not less than the $user["read_forums"] id for that forum. Or is this what you meant by:

Quote

One solution direction that I have been thinking about, is not letting the newflags table determine the min_id (that's how it's called in the code), but adding an extra table that is dedicated to storing min_ids for the users. That way, we'd have a quick lookup table to see what we're dealing with and a safer system for moving the newflags to the new forum.

---

Joe Curia (aka Azumandias)
Modules: l0Admin Mass Email00000000l000000Automatic Time Zones000ll.l00000Enhanced Custom Profiles0.00Google Calendar0000l.l000000Post Previews
000000000Admin Security Suite000000000000Check Modules for Upgrades0000External Authentication000000Group Auto-Email00000.00000Private Message Alerts
000000000Attachment Download Counter0000Custom Attachment Icons000ll.ll00Favorite Forums000000.00000Highlighted Search Terms0000Self-Delete Posts Option
000000000Attachment Watermarks0l00000000Custom Language Database00l.l.0Forum Lockdown00000.00000Ignore Forums0000000000000Threaded Tree View
000000000Automatic Message Pruning00.llll.00Easy Color Scheme Manager0l.l00Forum Subscriptions0000lll000Moderated User Group
Templates:lGeneric Integration000000000 0000Simple Rounded000000 00000000Tabbed Emerald



Edited 1 time(s). Last edit at 03/26/2009 12:30PM by Joe Curia.

Yes, that is indeed what I meant. Implement a separate storage for the min_id. A database table would be the best solution for that IMO. Especially because then it would be possible to execute queries on the data, which can be used as input for beating the newflags back into shape once in a while.

---

Maurice Makaay
Phorum Development Team
my blog linkedin profile secret sauce