I'm somewhat with jkmac on this, and that's because I tend towards Idealism.
Those who use the apparent size of the universe as a put-down are accepting, unquestioned, the proposition that the universe is as perceived. Unwittingly, they're relying on perception as the measure of all things -- which to some extent it is, though not perhaps in the way they think.
What if distance is an illusion? What if how "distant" things like galaxies are presented to our senses isn't actually an expression of how far they are away, or how big they are, but of their degree of knowability to us? We can get to know them a bit better by looking at them in telescopes or through other scientific means, but we can never know them as intimately as things that appear closer to us, like the earth or other human beings.
It could be that the more we know about a thing, the closer it appears. I'm not talking about the so-called knowingness that arises through acceptance of scientific theories, which may merely give an impression of knowing: about black holes, dark energy and matter, the Big Bang, etc. To my mind, this isn't really knowing, so much as acceptance of theories about the nature of the universe, which essentially are theories about how to interpret our perceptions.
For all we know, galaxies may be no "distance" away, but because we actually know little of them, they appear to be unimaginably distant. Our sun, on the other hand, we know a bit more about, and so it appears nearer. The usual way of thinking about things is that according to distance, we tend to know less about certain "objects", but we could turn that on its head and say that because certain "objects" are less knowable, they appear to be farther away.
This also works to some extent when we look at things we deem much smaller than ourselves: the smaller they seem to get, the less possible it seems to truly know them.
What is the key thing is that we're talking about perception, not necessarily reality. Perception, in a very real sense, determines everything we know about the world, however much or little that might be. The field of our perception is ****ing ginormous, encompassing particles, paramecia, people, planets, stars, galaxies, galaxy clusters, the whole universe in fact. All we have to do is turn our attention towards "things" to be able to formulate some degree of knowing about them, always limited by what can be known about them.
The materialist accepts our perceptions in a very literal sense: the galaxies are as huge and far away as our senses (or their extensions in, e.g. scientific instruments) seem to tell us they are. When taken literally, that makes us seem small and insignificant. However, without us, without perceivers, what ontological status could be attributed to them? Could they be said to even exist? It's a moot point, since without perception they couldn't be observed at all. The capacity to observe is what enables materialists to claim the reality of an external universe that can exist independently of us.
To perceive, to ratiocinate about the apparent objects we perceive, as well as to formulate hypotheses about what's "really out there" is what each of us bases our world view on. Lying at the bottom of all interpretations of the world is consciousness: to me, as an Idealist, it's obviously primal, the sine qua non of every"thing" else. But even if I were a monistic materialist, I'd have to admit that consciousness is an indispensible prerequisite to making observations and formulating theories about the universe.
Personally, I view materialism as a kind of naive realism. It has a touching faith in apparent reality.