The U.S. military takes UFOs seriously. Why doesn't Silicon Valley or academia?

Sure, but there remain holes in our understanding of physics that might allow effective means of breaking the light speed barrier.  (e.g., folding space)  Could be even more exotic ideas we haven't even dreamed up as yet.

Obviously, this is hardly evidential but just my personal view on eliminating the potential of advance life forms having figured out how to deal with these challenges.

Yes, I agree. There may be "methods" (waiting to be discovered) that might enable the traversing of the universe quickly so that here and there might not be so important anymore. To be honest, I don't like to think about the size of the universe too much, it's enough to induce insanity.