Once an invitation has been sent it cannot be withdrawn. However, you can delete the member once they join. Go to Delete a Member.