The big data managed by the Australian government as well as many other governments is of great importance to develop and deliver services to citizens. The purpose of this issues paper is to discuss the opportunities given to different agencies due to the use of big data, and the political concerns that this might raise. The greatest challenge that agencies face regarding this topic is to leverage the value of big data while protecting the citizen’s rights to privacy.

Big data is commonly defined as high-volume, high-velocity, and/or high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight, decision making, and process optimization. Today data is being produced at an ever increasing rate due to individuals and their use of media, organizations, the switch from analogue to digital technologies, and the proliferation of internet connected devices and systems.

A big data strategy was initiated due to the need for a strategy to enhance cross-agency data analytic capability for improved policy and service delivery. It is designed to highlight key opportunities and challenges that big data will bring to government agencies. This strategy will also consider a coordinated approach across governments, for the adoption of big data analytics technologies.


The value of big data is limited by the agencies ability to effectively manage the data. They must overcome the challenges that might arise in order to enable the benefits of big data. Big data analysis may provide insights into a number of key areas of society such as:

A successful big data strategy is expected to assist in realizing each of the following priority areas. The delivery of better services- will allow agencies to deliver more personalized services. Improved efficiency of government operations - will allow government agencies to better assess risk and feasibility. Open engagement – where agencies will be able to engage with interested parties locally and internationally.


There are many challenges that present to the use of big data. The volume of the data is already enormous and it keeps increasing. Furthermore, the variety of data being generated keeps expanding, and the capability to capture and process the data is limited. In order to enhance protection and set clear boundaries for usage of personal information, the Australian government strengthened the privacy act. There is also a risk involved in terms of managing information. The transport and delivery systems within and across agencies will become targets of interest for hackers.

There are various legislative instruments designed to maintain public confidence in the government as an effective and secure repository and steward of citizen information. However, the public release of large machine readable data sets, as part of the open government policy, could potentially provide an opportunity for unfriendly actors to glean sensitive information. This threat needs to be considered and carefully managed.

Government agencies realize that for data to have any value it needs to be discoverable, accessible, and usable. They must achieve these requirements while still adhering to privacy laws. Data also needs to be accurate, complete, and timely if it is to be used to support complex analysis and decision making.

If big data analytics is to be adopted by agencies, a large amount of stress may be placed upon current ICT systems and solutions. Agencies will need to manage these new requirements with the adoption of new technologies. These technologies include low cost storage arrays, in-memory processing, and cloud based storage.


To realize the outcomes of the ICT Strategy in relation to big data, Australian Government agencies need to achieve the following objectives: leveraging the big data experience and skills that exist within a small number of government agencies, and across the ICT industry and the research community to address the skills deficit that exists in this area. Ensuring that privacy issues are addressed up front. Encouraging the release of public sector information (PSI) consistent with all privacy and security legislative instruments and guidance.