Steve Harvey won't have Dave Portnoy back after facing Cooper Flagg's relatives on Family Feud