Big data. Big problem: security and reliability

Data fraud and hacking can be devastating for any business, and they are two issues that are very much a reality with no organisation one hundred per cent immune.

big data

The floppy disc has long been a symbol for data and storage. Despite traditionally only storing 1.44 megabytes it remains synonymous with data thanks to its use as the “save” icon in many software packages, not least of all MS Office.

Their consequences can be extremely damaging, tainting corporate reputations irredeemably, and can even lead to costly court cases. The result of which can be a hefty fine or even  potentially a custodial sentence for directors in serious cases. The recent breach at Tesco Bank springs to mind as an example that is only too real for its customers. So how do companies ensure that their big data is secure and reliable?

How Do You Know Your Big Data Is Reliable?

In instances where missing data points are vital to performance  the inclusion of unreliable information can sway results, disrupt user experiences or, in extreme cases can mean big problems for business. In today’s world all aspects of our life are changing faster and more often. This influences reliability by increasing the speed at which data expires – that is to go out of date and no longer be relevant or accurate. There are hundreds of micro factors involved nowadays these range from staff retention. INC.com highlighted loss of knowledge/information as one the hidden costs of staff turnover1.

More residential and business property transactions are taking place each year since 2011, this figure is lower than ten years ago, before the economic downturn2. Individuals moving house has implications for staff retention too. So many of these issues are interlinked. Less than 50% of FTSE business disclose staff turnover it is surely increasing each decade in part due to our shortening attention span3 as humans. Could our attention span also be affecting our likelihood to change or stick with a supplier as we get bored, look for change or attempt to follow the latest trend? Could it also be affecting accuracy of data input and checking. Businesses must therefore continually innovate to stay on top of data that is going out of date much faster.

Data quality is an issue all around the world. With a large emphasis on the speed of capture the collection process still tends to neglect the true value of validation. Figures from Royal Mail Data Service released last year estimated that in the UK 63.3% of businesses have missing, incomplete or out-of-date customer data. Validation was an issue cited there too with basic checks reportedly not taking place.

With simple automated verification and validation techniques you can reduce the amount of unreliable data entering an organisation and with proper systems in place to internally sense check data the reliability can be greatly increased.

How Do You Improve the Security of Your Big Data?

The recent Tesco security breach was a hostile one which resulted in a severe impact to individuals and whilst big data is becoming a more visible issue, as businesses we must take steps to understand the potential risks of the data we collect. The two main points for businesses to consider are the same factors as calculating risk – probability and consequence. In this content therefore we must ask:

  • How important is this data?
  • How likely is a data breach – in light of the robustness of security?

The old adage of prevention is better than the cure applies here – and therefore the more important the data the more steps must be taken to prevent any intrusions or breaches.

With the likelihood of a data breach you are dealing in terms of risk. Ease of access is probably the greatest factor here – how many people, internally and externally access your system and how easily can they do so. Is the data limited to only the people who need to have access? Is it limited to secure connections? Is it encrypted? Is it only accessible via certain IPs or offices?

In these cases, where the sensors themselves are actuators and in charge of controlling the security and safety, data collection can affect the outcome of whether a machine shuts itself down or causes potentially dangerous failures.

In the case of information request sensors, data can be tampered with or the sensors themselves can be hijacked, causing an abundance of new issues. Here, deploying big data fraud detection schemes with careful planning and management log systems can work wonders to reduce foul play.

Ultimately, data security is unique to each business and so the ways in which we protect data need to be unique. Whilst there is a need to reassure clients about security, it is beneficial to continually improve and update your security systems keeping specific details of updates under wraps. To do otherwise can act as a way to point hackers in the right direction and therefore increase the possibility of exploitation.

The key then is to regularly assess your data collection needs, pinpoint potential places for data breaches and invest in robust systems and software to combat data loss and intrusions.


1. Employee Turnover Hidden Costs [INC.com]
2. UK Residential and Non-Residential Property Transaction Section 1, Chart 1 [HM Revenue and Customs Annual UK Property Transaction Statistics 2015]
3. Attention Span Statistics 2000 vs 2015 [Statistic Brain]