Scottish cops dangle £6m for help understanding 160TB treasure trove of structured and unstructured data

The Scottish Police Authority is on the hunt for virtual data warehouse and data lake providers as part of a £6m tender intended to help the public body derive some value from the huge volumes of data it is sat on.

The authority said it stores around 100TB of unstructured data alongside a further 60TB of structured data that it wants to make better use of in order to catch baddies and work more efficiently.

In a tender notice published this week, the authority requests bids from software and services suppliers to create a virtual analytics platform, acquire a metadata management product and buy a data discovery solution for compliance purposes.

The tender notice asks for a "suitable data virtualisation software solution into the Police Scotland in-house ICT environment to allow a single view of all available data to all authorised users."

Once installed in-house, the plan is to migrate the system to the cloud in the fullness of time. The tender requests a "discretionary transition to cloud model during contract term."

"The solution shall be deployed, initially, on-premise, but deployable, in part or in whole, on a private or public hosted cloud environment at a later date," the tender says.

The authority will also address metadata in the contract. It is requesting an "intelligent metadata layer, constructed, inferred, enriched" to provide "deep insight into the service's rapidly growing data sets as well as maximise the use and value of its data."

The notice adds that given the scale of the metadata problem, "automation and collaboration tools are imperative." Metadata should cover governance, risk, and compliance.

"Metadata will provide a means of understanding the breadth and type of information available across the service, its quality, and relevance thereby enabling self-service by providing the right data context for users," it said.

Lastly, the Scottish Police Authority is looking for a compliance solution for GDPR and "other key legislation."

The tender is divided into three lots related to each strand of activity. The initial contract will be let for four years with the option to extend the contract for a further three periods of up to a year each.

The authority may have grand plans for its data strategy but the precedent is not good. According to its 2018/2019 audit report [PDF], the business case for better IT and data support up to 2026 estimated that the necessary investment was £298m over nine years. But funding for the strategy has not been agreed with the Scottish government.

Meanwhile, the audit notes the authority's IT projects have been dogged by delays. "Limited funding is not the only factor contributing to the delays, as the timescales associated with procurement and the governance of business cases have also had an impact on project plans." ®