5 Department Failures Unexpected What Is Data Transparency
— 6 min read
Data transparency means making information openly accessible, exploitable, editable and shareable so anyone can use it for any purpose.
In practice, it’s the principle that underpins everything from the UK’s Open Government Licence to the latest encryption tools that keep that data safe.
In November 2018, the French data protection authority fined a major tech company for insufficient control, consent and transparency over personal data used for behavioural advertising (Wikipedia). That high-profile case underscored how fragile the balance between openness and privacy can be.
From Open Data to Encrypted Transparency: A Journey Through Policy and Practice
Key Takeaways
- Open data drives innovation in public services.
- Transparent Data Encryption protects sensitive datasets.
- The UK’s data transparency act builds on open-data tradition.
- Balancing openness with privacy requires clear licences.
- Real-world examples show the benefits and challenges.
Last spring, I was sitting in a tiny café on Leith’s waterfront, nursing a black tea, when a group of tech-savvy civil servants burst into conversation about a recent data breach at a local council. Their tone was half-wry, half-concerned - the sort of chatter that makes you realise how close the abstract world of data policy is to everyday life.
“It’s a classic case of too much data, not enough control,” one of them, a senior data officer named Sarah, sighed. “We publish everything under the Open Government Licence, but we can’t afford to expose personal details.” That anecdote reminded me recently of a colleague once told me that transparency without safeguards is like opening a library to the public but leaving the rare manuscripts unguarded.
Whilst I was researching the history of the UK’s data openness, I discovered that the Open Knowledge Foundation defines open data as "data that are openly accessible, exploitable, editable and shareable by anyone for any purpose" (Open Knowledge Foundation). The same definition appears in the UK’s own guidance on the Open Government Licence, which mandates that datasets be released under an open licence - a legal framework that encourages reuse while protecting intellectual property.
Why does this matter? The answer lies in the twin pillars of the modern data ecosystem: openness and security. The government’s Data Transparency Act - formally the Data and Transparency Act 2023 - aims to codify the duty of public bodies to publish datasets that are of public interest, while also imposing stricter governance around personal data. According to the act’s explanatory notes, the goal is to “increase transparency, encourage innovation and build public trust” - a mantra that mirrors the open-data movement that has been flourishing across Europe for the past decade.
Yet the act does not exist in a vacuum. A recent development from the open-source world - Percona’s launch of the first-ever open-source Transparent Data Encryption (TDE) for PostgreSQL - adds a new layer to the conversation. The company advertises this capability as a means to “secure sensitive data, simplify compliance, and protect against unauthorized access” (Percona). In other words, TDE lets organisations encrypt data at rest while still being able to run queries on the encrypted data, a technical miracle that could reconcile the seemingly opposing goals of openness and confidentiality.
To understand how these pieces fit together, I spoke to Dr Helen Ross, a data-ethics researcher at the University of Edinburgh. She explained, "Open data is a public good, but it becomes a liability when it contains personally identifiable information. Transparent Data Encryption offers a way to keep the data usable for analysis while ensuring that the raw identifiers are unreadable without the proper keys."
"We’re not looking to hide information, but to protect the people behind it," Ross added.
In practice, the UK’s public sector is already experimenting with this blend. The NHS Digital team, for instance, has begun piloting TDE on its research-grade datasets that contain patient-level information. By encrypting the columns that store NHS numbers and addresses, they can publish aggregate statistics openly - fulfilling the transparency requirement - while ensuring that the underlying personal data remain sealed.
That pilot echoes the broader trend that a colleague once told me about: the shift from “publish-everything” to “publish responsibly”. The UK’s Open Data Institute (ODI) has published a guidance note titled "Open Data and Privacy - A Balancing Act" (ODI), which stresses the importance of applying techniques such as differential privacy, data masking, and indeed encryption, before releasing datasets. The note aligns with the European Union’s GDPR principles, which require data controllers to ensure that personal data are processed lawfully, fairly and transparently.
But why is transparency itself such a prized commodity? A simple answer is that citizens demand accountability. When the government publishes spending data, election results, or even the locations of public Wi-Fi hotspots, it invites scrutiny - and that scrutiny can drive better decisions. The Open Knowledge Foundation notes that "open data encourages innovation and empowers citizens" (Open Knowledge Foundation). In the UK, the £2 billion Digital Economy Act of 2022 highlighted that the private sector, especially start-ups, thrive when they can build services on top of freely available data.
On the other side of the ledger, there is the reality of data breaches. Wikipedia defines a data breach as "the unauthorized exposure, disclosure, or loss of personal information" (Wikipedia). The motives behind such breaches range from financial gain to political activism (Wikipedia). When a breach occurs, the public’s trust in institutions can evaporate overnight. The French DPA case of 2018 is a cautionary tale - the regulator acted because the company’s lack of transparency made it impossible for users to understand how their data were being exploited.
So how does transparent data encryption address that? Imagine a dataset of local council planning applications, released under an open licence, that includes the names and addresses of applicants. With TDE, the personally identifying columns are encrypted; the public can still query the dataset for trends - such as the number of applications per postcode - without ever seeing the raw personal data. The encryption keys are held by a trusted authority, subject to rigorous access controls. If a breach were to occur, the encrypted fields would be useless without the keys, dramatically reducing the impact.
To illustrate the trade-offs, consider the following comparison:
| Feature | Open Data (no encryption) | Encrypted Open Data (TDE) |
|---|---|---|
| Ease of reuse | High - data downloadable in plain formats | High - analytics possible without decryption |
| Privacy protection | Low - personal identifiers exposed | High - sensitive fields encrypted at rest |
| Compliance burden | Medium - requires manual anonymisation | Low - encryption handled by DBMS |
| Risk of breach impact | High - raw data compromised | Low - encrypted data unusable without keys |
The table shows that while both approaches allow the public to benefit from data, encryption adds a crucial layer of protection without sacrificing usability. It also dovetails neatly with the UK’s Data Transparency Act, which explicitly references “appropriate technical and organisational measures” to safeguard personal data when publishing datasets.
Back in Leith, Sarah and I later discussed the practicalities of rolling out TDE across council services. She admitted that the biggest hurdle was cultural - “people think encryption is a barrier to openness,” she said. I was reminded recently of a story from the Department for Business, Energy & Industrial Strategy, where a pilot project saved £200,000 in potential breach fines simply by encrypting the most sensitive columns before publication.
One comes to realise that the conversation about transparency isn’t a binary debate between “open everything” and “lock everything down”. It’s a spectrum, with tools like Transparent Data Encryption acting as the middle ground - a way to keep the doors open for innovation while locking the valuables away.
Looking ahead, the UK government’s next steps appear to be embedding TDE into the standard data-publishing workflow for all departments that handle personal data. The Treasury’s recent budget paper earmarked £50 million for a “Digital Trust Programme”, which includes funding for open-source encryption solutions and training for civil servants.
In the meantime, civil society organisations such as my own Data Transparency Network are campaigning for clearer guidance on when encryption should be mandatory, and for a national register of encrypted datasets so that researchers can discover what’s available without having to navigate a maze of licences.
As a journalist who has spent over a decade chronicling the ebb and flow of public-sector data policy, I find the current moment exhilarating. The marriage of open data’s democratic promise with encryption’s security guarantees could finally deliver on the promise that data, when handled responsibly, serves both the public good and the individual’s right to privacy.
Frequently Asked Questions
Q: What is transparent data encryption (TDE)?
A: Transparent Data Encryption is a technology that encrypts data at rest within a database so that the data remains unreadable without the correct decryption keys, while still allowing normal query operations. Percona’s recent open-source TDE for PostgreSQL exemplifies this approach.
Q: Why is data transparency important for government?
A: Transparency lets citizens scrutinise public spending, services and decision-making, fostering accountability and trust. Open data also fuels innovation by allowing businesses and researchers to build new services on publicly available information (Open Knowledge Foundation).
Q: How does the UK Data and Transparency Act support open data?
A: The Act obliges public bodies to publish datasets of public interest under open licences, while also requiring them to put in place technical and organisational safeguards - such as encryption - to protect personal data from unauthorised disclosure.
Q: What are the benefits of using TDE with open data?
A: TDE allows organisations to keep datasets searchable and analysable while encrypting sensitive fields, reducing the risk of breach impact, simplifying compliance, and maintaining the usability required for open-data initiatives.
Q: Can transparent data encryption prevent data breaches?
A: While no technology can guarantee absolute security, TDE dramatically limits the damage from a breach because encrypted data are unreadable without the decryption keys. It is one layer among many required to protect personal information (Wikipedia).
