It might be helpful to have a continuous version of the seclog without having to rely on archived pages.

I might suggest dumping the post history of individual users/accounts. This could be restricted by rank and otherwise be rate limited. I think this would be difficult to recreate any meaningful mirror site with this information.

As others have mentioned, the security log would be beneficial. The mod log, not so much because of its limited information.

It would be helpful if users outboxes (and other folders) can be downloaded since they cannot be easily searched. Obviously downloading this information would be restricted to users who are logged into their own account.

I can think of a few:
1. Add "Activity" (not just "potential")
2. Add a banned-status to this list (ignore temporary bans)
3. Add either "merit earned" or "merit received for free at introduction"

Side note: there are more than 200 usernames with a comma, this will make processing a CSV difficult. Can you make this a file with just UID and name?

It all comes down really to what needs to be found out. That is, building a set of questions that need to be answered and derive the raw data information that enables an aggregated or derived dataset to be queried for the answers.

Some questions are answerable by a snapshot of the data, whilst others require the inclusion on a timeframe and datestamps to resolve.

For example, in order to see how long it takes to rank up for members, we would need the whole history per UserId  of rank changes <UserId, Rank, Activity, Date>, where the registry would only be necessary to be created when there is a user creation or a change in the Rank, being Date the associated timestamp.
If we wanted to see this in relation to Merit, we would need to build a registry in the shape of <UserId, Rank, Activity, Merit, InitialMerit, Date > .

The other key factor is related to the current way in which data is stored. The raw data layout and capture process is part of the process to reach our solution goal.
For example, if there is a trigger in the database that currently logs  changes on the User Table for the <UserId, Rank, Activity, Date> record structure, the underlying table is direct and all that has to be done, once exported, is to select records that relate to a change in user’s rank (and ignore those that are a mere activity change).

If alas the underlying user table does not hold a historical record of changes (i.e. no logged timestamp historical), then the question of how long it takes to rank up would not be answerable or need to be crossed with other raw data from another table.

Questions that I would boldly put on the list due to sMerit introduction would be such as:

- What is the average time per Rank to rank-up?: before and after the introduction of the Merit system (this is not entirely comparable yet, since merit system is only a few months old so top Ranks are not comparable yet).

- How much sMerit is assigned per rank (from/to), per forum section, per forum subsection, in relation to number of posts in topic, in relation to topic heatness, in relation to post position in topic (quartiles for example), in relation to size of merited post, etc.

- How much sMerit is being withheld and for how long (averages).

- Round merit assignment candidate (from User A to User B and back -> That is derivable from current Merit.txt file as I’ve posted previously – it is not necessarily a cheat, but a source of study for such cases).

The match between a closed set of key questions to answer, and potential raw data structure should give us what additional files are required in my opinion.

i think the UID -> name, merit, potential activity, post is useful. in this you can easily compile the post contents of a user and create an outbox for each user to be compiled into it and easy to look or search the users activity and recent post, also some useful ideas are suggested like this you can also monitor the give and take of merits by each user. this is what i understand by this thread please feel free to correct me if im wrong.

I was asking for such information here today and just see this thread now.

I think all dumps related to forum architecture will be great to compute local boards stats.


This kind of requests would be easier to implement.

And what about some automatic dump archiving to avoid several people to do the same?

How big of a operation is to auto-update the data on a daily basis ? I was thinking I could set-up end points which downloads the file daily and keep my source for the charts (whatever I choose to represent) updated every day.

It would be great if you can send a status flag along with the account details like "active/inactive/banned".

Is there a possibility of including the Rank in the merit.txt file or having another file to complement it so as to perform rank analysis tied to data in the merit.txt file?
I've seen Zentdex managed to cross this information, but it's not in the public raw data files for general usage as far as I can see.

It's True that Rank will vary for some user's within the timeframe of data within the merit.txt file, but is would be a helpful source to breakdown data and comprehend it better.

Any follow up on this?