When there is no one around, a construction site might be a place for kids to explore, for vandals to target or for opportunists a chance to pick up any loose items. A construction site, irrespective of the stage of the works, can be a dangerous place. As the builder, you are also deemed to be the PCBU and the person responsible for ensuring that the site is as safe and secure as reasonably practicable to prevent any loss or injury to any person whilst there are no works being undertaken.
Simple tips such as ensuring your site fencing is secure or locking loose items in the garage or off-site might be simple yet effective measures to mitigate any risk to you. As mentioned above, also remind your client that there is to be no unauthorised access to site during this period either in accordance with your building contract. Whilst you cannot entirely remove risks, you should do what you can to mitigate them.