"I'm not gonna have that conversation," Walton Goggins said when asked about the speculated rift between him and Aimee Lou ...