SEO and site structure [split from: Is this forum dying?]

[Laird]

I'm planning to add a new forum, "General parapsychology", under the "Parapsychology forums: psi and survival (life after death) research" category.

This would be for topics that don't fit strictly into any of the existing forums under that category, for example, general parapsychology journals, societies, (book) awards, conferences, and other events, etc.

I'm also wondering whether it's worth adding a "General survival research" forum under that category, for survival research (and related arguments, theories, ideas, etc) that, similarly, doesn't fit neatly into any of our existing (survival-related) forums: NDEs, afterlife contact, reincarnation, ghost/poltergeist/spirit.

In the absence of objections, I'll go ahead at least with the first idea, and potentially the second - I just wanted to float the ideas here first in case there are any sound objections or better proposed alternatives.

[Laird]

Responding here to this from another thread:

Interestingly, as time has gone on, finding the Loderunner thread via a normal search engine has goten increasingly hard. I often find threads that mention it, but the thread itself often doesn't show up at all even when doing a direct site search using google, presearch, and a couple others. I haven't tried that for awhile and I know they change that stuff occassionally so who knows maybe it works again. But yeah it was almost like it was being censored when I saw it not even come up in a site search. Probably nothign though.

It's definitely real and not nothing.

Google Search Console reports that your Loderunner thread is not indexed. I submitted it for indexing not long after you posted the post to which I'm responding, but it's still not indexed.

This reflects a broader problem as you've noticed: over time, the number (and proportion) of our pages being indexed has dropped, to the point that, currently, Google Search Console reports these numbers for our pages:

Code:

Indexed                           :  2,180.
Crawled but currently not indexed : 10,295.
Discovered - currently not indexed:  1,935.

In other words, while Google is aware of 14,410 of our pages that could be indexed, it has actually indexed only about 15.1% of them.

A little tailored database query indicates that the number of publicly-visible thread pages on our forum is 7,899, so it's plausible that all of them are already included in all the pages of which Google is aware.

I had been hoping that the recent tweaks I've made in the past few months would have turned things around, but they're still pretty bad.

The good news is that this situation has improved (a little) over the last few months. The same figures on the 10th of June were:

Code:

Indexed                           : 1,214.
Crawled but currently not indexed : 8,775.
Discovered - currently not indexed: 2,940.

In other words, of the 12,929 pages of which Google was aware back then, it had actually indexed only about 9.4% of them.

From 9.4% to 15.1% is a 61.1% increase in proportion of pages being indexed over about three months. It's better than nothing, but way short of what we need.

I have no idea why Google is indexing so few of our pages. Browsing through the list of those not being indexed doesn't give me any clues, nor does the "Learn more" link. There's nothing obviously "wrong" with any of them, including, in many cases, lack of original content: your own Loderunner thread is a case in point; it is full of original content.

If anybody has any ideas, then please let us know.

I might otherwise try to find an SEO forum and seek advice.

I have no idea why Google is indexing so few of our pages. Browsing through the list of those not being indexed doesn't give me any clues, nor does the "Learn more" link. There's nothing obviously "wrong" with any of them, including, in many cases, lack of original content: your own Loderunner thread is a case in point; it is full of original content.

Firstly, I had no idea you could check Google's indexing like that. If I had to guess it's because Google and all mainstream search engines (but especially them) are more accurately "hiding" engines. They openly admit to manipuating search results to manipulate public percpetion of particular narratives and have been blatantly caught doing so. And as a result will delist or downrank things that don't go along with whatever narrative they're helping to push. Regardless of it's factual accuracy. To that end I suspect it's shadowbanning (greylisting?) parapsychology evidence in general to the best of it's ability. But more specifically, Loderunner references several real, provable things in it's links that would be consider conspiracy theory by the hard of thinking, or inconvenient truths to the ruling class. For example, the link to the article about biometric survellance in China from 2012. However, I've since learned about more legit things about that whole adventure as the years have gone on. Since 2021 alone I've lerned that the usage of the American foster care network for child trafficking is indeed a real thing that some people are trying to call attention to. As well as that there have been legitimate programs in the past to attempt to induce psionic abilities in children via drugs and trauma. Apparently this was done under MKUltra and some of it's sub projects. Most notably Project Monarch. I was not aware of any of that at the time Loderunner happened but I can't deny that it's all a little too similar for me to ignore.  Likewise learning of child cases of oddly strong psi abilities (at least compared to the norm) that cause troubles for their life, such as spontaneous and periodic electromagnetic disturbances.

Programs which I'm assuming failed just as spectacularly as they did in Nadia's reality (or whateve that place was), but without creating an exception like her. At least as far as I can tell. It's possible Google is downranking that thread, but not threads mentioning it, simply because of some of the links shared which contain inconvenient truths. But it's also possible that the content itself might be deemed something it wants to hide, even if the simple explanation is that it was all a very vivid dream and the result of somneone who'd heard about all these programs and things somewhere else, or otherwised imagined them.

I can only imagine how it would react if I finished polishing up Aheadjro and posted it.

[Laird]

Firstly, I had no idea you could check Google's indexing like that.

It's been possible for many years, decades even, via Google Search Console (previously it was called Google Webmaster Tools). You sign up at that link, prove that you are in control of the website on a given domain, and then you have access to a variety of data, including which of its pages are indexed (or not) and any indexing errors, which search terms are leading to pages on your site and how many clicks they get, etc.

Bing has something similar: Bing Webmaster Tools.

If I had to guess it's because Google and all mainstream search engines (but especially them) are more accurately "hiding" engines. They openly admit to manipuating search results to manipulate public percpetion of particular narratives and have been blatantly caught doing so. And as a result will delist or downrank things that don't go along with whatever narrative they're helping to push. Regardless of it's factual accuracy. To that end I suspect it's shadowbanning (greylisting?) parapsychology evidence in general to the best of it's ability. But more specifically, Loderunner references several real, provable things in it's links that would be consider conspiracy theory by the hard of thinking, or inconvenient truths to the ruling class.

I'm not so sure that that's really what's going on, for a couple of reasons:

Firstly, plenty of our threads that are explicitly about evidential parapsychology are indexed.

Secondly, plenty of other pages on the web relating to evidential parapsychology are indexed too: just search Google for "parapsychology evidence".

Maybe it's part of the problem though. If it is, then there's not much we can do about it short of activism.

[Laird]

I'm planning to add a new forum, "General parapsychology", under the "Parapsychology forums: psi and survival (life after death) research" category.

This would be for topics that don't fit strictly into any of the existing forums under that category, for example, general parapsychology journals, societies, (book) awards, conferences, and other events, etc.

I'm also wondering whether it's worth adding a "General survival research" forum under that category, for survival research (and related arguments, theories, ideas, etc) that, similarly, doesn't fit neatly into any of our existing (survival-related) forums: NDEs, afterlife contact, reincarnation, ghost/poltergeist/spirit.

In the absence of objections, I'll go ahead at least with the first idea, and potentially the second - I just wanted to float the ideas here first in case there are any sound objections or better proposed alternatives.

Now done. I've created both new forums, seeded with appropriate threads from existing forums.

[Laird]

One last time Skeptiko links on wiki are broken... and may look like keyword stuffing to google... for example any links under
Consciousness & Science forum threads: headings are dead.

Alright, Max: even though I'm not sure this is a problem, to satisfy your concerns I've blocked that page from indexing by adding an `X-Robots-Tag: noindex` HTTP header to it as recommended by Google. I've confirmed with the URL Inspection Tool in Google's Search Console that it's recognising the header. That page should no longer be indexed by Google (and most if not all other search engines).

Given that that page was unlikely ever to be edited again, and that it used the most PHP memory of all requests to our web server, I've migrated it to a static page: https://psiencequest.net/skeptiko-forums...-page.html

That decreases the maximum memory we need to account for PHP using, allowing us to increase the maximum number of php-fpm processes that will be dynamically spawned as needed.

@Max_B, FYI, that new static page has the same `X-Robots-Tag: noindex` HTTP header as the migrated wiki page, so it shouldn't be crawled by Google and thus still shouldn't be seen as keyword stuffing.