Only one set of wires brings in the power. Find those and then you can go from there. The other wires may be for other switches.
What Is The Wiring Diagram For A Harbor Breeze Ceiling Fan W/lights To 1974 Electrical Wiring?