One is permited to Daven before the Zman. Making Havdalah before the Zman (without a candle) is a bit more complicated and IIRC it can only be done when the person cannot do it after the Zman.
Regarding Havdalah after the Zman but before the time your Minhag is to end Shabbas (unless you really follow Rabbenu Tam) you could be Yotzei if you wanted to but probably wouldn’t want to.