Forget all the exotic cartridges for the AR-15 platform. Stick with the original 5.56 NATO/.223 chamber, with a fast twist barrel of 1:8 or faster.
Instead of changing calibers, change bullets. For hunting or self defense, switch over to Barnes Bullets 62 grain TSX. It is a no lead bullet and is known as a barrier buster bullet. Barnes Bullets are marketing 62 grain .223 TSX as a medium game cartridge. Barnes Bullets also makes a 70 grain TSX .223.
Hornady makes a somewhat similar no lead, all copper/zinc bullet in .223 called GMX. Again, its a barrier buster/terminal ballistics bullet instead of a match bullet or a full metal jacket bullet.
Again though, you do need velocity for these Barnes Bullets, be sure to use a 1:8 twist or faster even for the 62 grain TSX.