You already said it, air cooled in zero atmosphere doesn't work. Heat is actually already a big problem on space stations as heat can only be shed via thermal radiation, which is much less effective than convection. You need a very large surface area to radiate all the heat of a nuclear reactor away, and for maximum efficiency that surface needs to be as hot as possible. There are several advanced cooling designs out there as people already thought long and hard about different kinds of nuclear drives for spaceships, but in the end it boils down to "radiate heat away", because vacuum. Some (open core) drives like the Zubrin Saltwater rocket fix that by using the fission products as propellant and fuel (like a chemical rocket) and thus chucking out the heat of the drive as a side effect of how it works, but most designs don't do that.