When no one is around, a building site might be an enticing place for kids to explore, vandals to target or provide opportunists a chance to pick up any loose items. As the builder, you are also deemed to be the person conducting a business or undertaking (PCBU) and therefore the person responsible for ensuring that the site is as safe and secure as reasonably practicable. A site, regardless of the stage of construction, is a dangerous place and the purpose of securing your site is to prevent loss or injury to any person while there are no works being undertaken.
Simple tips such as ensuring your site fencing is secure or locking loose items in the garage or off-site might be simple yet effective measures to mitigate any risk to you. Also remind your client that there is to be no unauthorised access to site during this period, in accordance with your building contract. While you cannot entirely remove risks, you should do what you can to mitigate them.