Land convenant

We recently purchased a property in Frankston area, a 3 bed house with 650sqm of land.

Before we looked at the property it was confirmed by the Real estate agent that potentially we can build a unit at the back of the house and rent it out seperately(have this in writing).

Being able to build a second dwelling on the land was a major criteria in the property we aspired, therefore we agreed to buy this property. At the time of signing the contract they confirmed(the lady spoke to her boss/legal adviser) who saw the section 32 and confirmed that building a second dwelling at the back was possible STCA.

However recently while looking for a builder/surveyor for this project, we found that the land has a convenant where only one dwelling is allowed. This was later confirmed by our conveyancer.

As inexperienced first home buyers we didn't confirm with our convenyacer about a possible restriction/convenants before singing the contract because we blindly trusted what the real estate agents said and throughout we were under the belief that building a second dwelling was not an issue at all.

Further inquiries with our conveyancer revealed that removing this covenant is not only time consuming but can be a very expensive exercise.

My question to all the experienced posters here is do we have any grounds to take any action against the real estate agent for giving false information and misleading us or is it entirely our responsibility as buyers to check these things prior to signing the contract and we just have to regret not doing proper checks for the rest of our lives.?