Psience Quest

Yes, Consciousness & Science is on the chopping block as well.

(2017-08-16, 08:55 PM)Max_B Wrote: [ -> ]OK, tried getting a new Skeptiko password and logging in, but even when logged in FUPS hits the same error... so it doesn't work if you're banned, even if your logged in.

Oh, the login has to be via the FUPS app - I'm guessing you logged in via a browser? That has no effect on FUPS.

(2017-08-16, 08:55 PM)Max_B Wrote: [ -> ]Just so I understand clearly... you think after making changes to FUPS it will be possible for FUPS to scrape my posts from Skeptiko despite being banned? And further you expect to have those changes in place before the 21st Aug?

Yes. I have just finished updating the script to scrape entire forums, and am currently scraping the CD forum - it seems to be progressing just fine. So now I can get onto the changes I promised you so that you can scrape your posts.

(2017-08-16, 08:55 PM)Max_B Wrote: [ -> ]There is no pressure from me, but I'd like to understand clearly, so that I don't get to the 21st August only to discover that I've misunderstood you, and I should have just manually copied and pasted my posts by hand, because FUPS won't be able to scrape them automatically as I'm banned.

I estimate I'll have something ready for you at least within 24 hours, probably much sooner.

(2017-08-17, 02:03 AM)Laird Wrote: [ -> ]I estimate I'll have something ready for you at least within 24 hours, probably much sooner.

As in much, much sooner, lol.

Should be working for you now, Max. All you need to do is enter all details as before, but this time for the new setting "Extract User Username", enter "Max_B". Any problems, let me know!

Oh, but a warning: don't enter anything in the new field, "Forum IDs" - that will result in FUPS downloading not only your posts, but also the entire set of threads/posts of any forums that you specify here. I am already downloading these myself, and will make them publicly available, so there's no need for anybody else to do that too.

(And yes, I know the documentation of that new field isn't very good).

(2017-08-17, 11:40 AM)Max_B Wrote: [ -> ]Thanks so much Laird, you've saved all those many many hours of my research, and interesting ideas that could have been lost forever. Honestly I'm indebted to you.

Happy to have helped, Max. You've helped us too, so it's a circle of giving.

Also, just to let folks know, I'm uploading my scrapes of the Skeptiko Forums here: http://psiencequest.net/skeptiko-forum-scrapes/. So far the only one is the Critical Discussions forum. Obviously, the format (JSON) is a bit unwieldy, and I'll need to work on converting it into a bunch of HTML pages, but all the data is there. The file is huge, and is probably best opened either in Google Chrome or a text editor - Firefox seems to struggle with it, but if you have lots of memory, Firefox might manage it.

An update: the scrape of the "Other" forum has now been uploaded too. Am currently downloading threads from the Consciousness & Science forum.

(2017-08-18, 03:45 AM)Laird Wrote: [ -> ]Am currently downloading threads from the Consciousness & Science forum.

Now finished, and uploaded to http://psiencequest.net/skeptiko-forum-scrapes/

Extended Consciousness scrape now underway, last one, woohoo.

(2017-08-18, 08:20 AM)Laird Wrote: [ -> ]Extended Consciousness scrape now underway, last one, woohoo.

Done and uploaded. We have all we need.

If anybody wants to work on converting this JSON data into HTML pages for easy browsing, then please speak up, otherwise I'll get onto it myself in due course.

Thanks for all your hard work in saving that material from being lost altogether. And thanks also to Alex for his cooperation.

Laird - what you are doing is wonderful, and I hope will work successfully - will the end result be read-only, or will it be possible for people to continue a discussion in place?

After several conversations with Alex, I must say that I still don't understand why he wants to do this. Alex's actions seem terribly destructive of what he himself has built up over many years.

David