Better off buying a square port of the right size if the round port is bigger. If not, mark out the square cut lines along the box and use a jigsaw to cut it out, but be careful and cut slowly both for safety and accuracy.
How To Change A Round Port To A Square Port?
Depending on if you are talking about the wire plug ins or the speaker.or if you are trying to enlarge the hole or decrease the size of the hole. You could add a square framework to the round hole it will step the hole back as far as the thickness of material used but you won't have to destroy your box. Just pre-drill and screw it to the inside of the box. Sort of like a spacer.