“Physical stuff” can’t do this and that - a false Dichotomy

This is why I think even an Idealist can still speak of the "physical", which is something Physicalists tell us lacks mental character but is also the stuff that makes up this universe.

So when someone who isn't a Physicalist notes the "physical" cannot do something, it is that part of reality that is claimed to lack this ability.

I agree that an idealist can speak of the "physical", but the "physical" for an idealist is vastly different than the "physical" of a physicalist or a dualist.
For the latter two there exists an objective reality independent of an observer. For the idealist "objective reality of physical properties" does not exist.
This is actually a hotly contested question. I recommend reading the following blog post by Bernardo Kastrup which is quite interesting in relation to this discussion:  

My unfortunate attempt at debating Tim Maudlin ~ Bernardo Kastrup, PhD, PhD