Lifetime refugee visa ban not on PM agenda