Does it have anything to do with the recount theymos did? Not sure how that would affect the data LoyceV uses.


Edit: I guess not, just behind the scenes

There was a bug that overwrote files a few times. Some of the scraped posts were as old as a minute or more.
I fixed the bug, let's see how long it takes to post scrape this.

Update: http://loyce.club/archive/posts/5224/52243599.html was scraped 1 second after posting. Let me know if anything else fails.

The scraping should be okay now, but I didn't fix the upload yet. I've uploaded an update for just this file.

I didn't notice before, but it shows many duplicate lines. The links to "scraped" don't work either, you'll have to manually change "/posts/posts/" in the URL to "/posts/".

Sorry for this, I haven't had the time to fix this yet.

Last night, "recent" was broken for 8 hours.  It's the first time I've seen that, theymos didn't post yet what caused it.

As a result, I couldn't scrape anything between post 52295210 and post 52297852. That means I'm missing 2641 posts.

A great tool. But I have some thoughts about the possible use and misuse of this.

Let's talk about the positive first.

I am amidst manually collecting data from a few accounts that is operated by one individual. Now, I have an X number of accounts linked with this person but I am sure this person has 2X the number of accounts in total.

I am just being lucky to collect the ones where he has made some errors.

So if I post this half report of these linked accounts, then the person would be alerted and would, in theory, try to delete all his errors from all of his accounts' post history.

Your tool, could stop this. This basically enables you to get the posts made by anyone at the first instance they hit post. Which is great in this kind of example.


But now let's talk about the bad side.

Your tool is freely available for anyone to use. The very same person I am trying to bust for multi-accounting, is also using information scattered about me on this forum to track my identity and send me threatening messages on my telegram, phone etc.

This kind of tool would essentially be a weapon in the hands of such blackmailers and extortioners.

Furthermore,

In our country, our government is banning crypto, so if I posted my information here and if tomorrow some government agency wanted to track me down - then your tool is going to enable them to do that. And possibly land a person in jail for 10 years - just for being associated with crypto.

---

Now, I'd like to hear your thoughts about these 2 situations. What do you intend to do to avoid solve such issues?

You're glowing..

Anything you put on the internet, should be considered public information for the rest of eternity.

If that's the case, you shouldn't have put the information online in the first place.

Several phishing sites clone all posts on Bitcointalk too. I just do it without trying to steal passwords. Archive.org and Archive.is store many pages too.

As much as I hate government oppression, it's not my responsibility to follow local laws from any country on earth. If it's illegal to use crypto, and you insist on doing it anyway, all I can say is you should be very careful what you post and ensure your real identity stays hidden. Use encryption to hide yourself

Agreed!
I don't scrape Investigations, which is the only place on Bitcointalk that allows DOXing. If someone gets DOXed anywhere else, posts can be reported and the user gets banned. If that happens, feel free to contact me to edit one of the archived posts. When I do that, I'll also edit the filename to make it very obvious it was edited by me, and I'll probably create a log here. Until now, I haven't done this.
Another thing I've been thinking about is if someone posts something that's not allowed by my webhost. If that would happen, I'll have to edit it too.
I'm not sure how Archive.org and the likes handle illegal stuff on their servers.

I'm not sure what that means

If someone posts something on Facebook or Twitter, with privacy settings allowing anyone to see the post with the hashtag #crypto, anyone specifically looking at your profile can see your post, but it is not trivial for someone to obtain all posts containing a hashtag.

There are limitations as to how many tweets can be distributed to entity user per day and per month. The daily number of tweets that can be sent to a third party is large (50,000), but is a small percentage of the total tweets posted every day (500 million). If someone like DPR was posting on twitter instead of bitcointalk, they probably would still have gotten caught, while a HK protestor would probably be safe on twitter, while the HK government (a sockpuppet of the Chinese government) might be investigated similar to how legendster describes if they are posting on bitcointalk.

LoyceV is not the one who invented scraping forum posts, nor is he the only one to be actively scraping posts. There are many ways to download forum posts via automated means, and it is not difficult to get posts into a DataFrame that can later be analyzed.  

The the root cause problem is the administration allows too much access to posts. A straightforward solution is to have an hourly rate limit as to how many page views an individual IP address/range can access on an hourly and daily basis that is something above what a *person* would see in the normal course of reading, but well below the necessary amount of page views required to view all posts. The scraping of posts for non-academic use should also be explicitly prohibited by the administration.

The 1 page download per second is a standard generalized limit when no commercial relationship exists. Although this limit has been communicated by the administration, it is also what should be the assumed limit when scraping information from a website.

I believe merit data is actually published by the administration, along with trust data. This information is less intrusive than information contained in posts. If my above proposal were to be changed to 'thread page view' I understand BPIP would be entirely unaffected.

I would encourage you to review the Twitter developer terms, and the Instagram API TOS. These policy documents prohibit many of the things that are done with bitcointalk information. I would presume the 'average' bitcointalk user to care more about privacy than the 'average' Instagram or Twitter user.

From a technical perspective, there is nothing to prevent a government from collecting post information on bitcointalk. However if bitcointalk policies explicitly disallow government law enforcement from collecting information in mass via automation, in general, law enforcement will have trouble using information gained via these means as the basis for a warrant, or as admissible evidence in court.