You have permanent residence? Or do you have a visa entitling you to permanent residence?
This is a Canadian immigration forum, not an Australian one, so I do not know anything about Australian immigration policies, but I have trouble imagining they would attempt to restrict where you can travel from.