Pulse Mix

Anna Collard

Beneath the surface of GenAI’s outputs lies a massive, mostly unregulated engine powered by data – your data. And whether it’s through innocent prompts or habitual oversharing, users are feeding these machines with information that, in the wrong hands, becomes a security time bomb.

A recent Harmonic report found that 8.5% of employee prompts to generative AI tools like ChatGPT and Copilot included sensitive data – most notably customer billing and authentication information – raising serious security, compliance, and privacy risks.

Since ChatGPT’s 2022 debut, generative AI has exploded in popularity and value – surpassing $25 billion in 2024 – but its rapid rise brings risks many users and organisations still overlook.

“One of the privacy risks when using AI platforms is unintentional data leakage,” warns Anna Collard, SVP Content Strategy &amp; Evangelist at KnowBe4 Africa. “Many people don’t realise just how much sensitive information they’re inputting.”

It’s not just names or email addresses that get hoovered up. When an employee asks a GenAI assistant to “rewrite this proposal for client X” or “suggest improvements to our internal performance plan,” they may be sharing proprietary data, customer records, or even internal forecasts.

<a href="https://www.pulselive.co.ke/articles/business/what-ai-enabled-ad-campaigns-mean-for-kenya-2025021213300007334" target="_blank">READ: What AI-enabled ad campaigns mean for Kenya</a>

READ: What AI-enabled ad campaigns mean for Kenya

If done via platforms with vague privacy policies or poor security controls, that data may be stored, processed, or exposed. And the risk doesn’t end there.

<blockquote>Because GenAI feels casual and friendly, people let their guard down, says Collard. They might reveal far more than they would in a traditional work setting – interests, frustrations, company tools, even team dynamics.</blockquote>

In aggregate, these seemingly benign details can be stitched into detailed profiles by cybercriminals or data brokers – fuelling targeted phishing, identity theft, and sophisticated social engineering.

<h3>A surge of niche platforms, a bunch of new risks</h3>

Adding fuel to the fire is the rapid proliferation of niche AI platforms. Tools for generating product mock-ups, social posts, songs, resumes, or legalese are sprouting up at speed – many of them developed by small teams using open-source foundation models.

While these platforms may be brilliant at what they do, they may not offer the hardened security architecture of enterprise-grade tools.

<blockquote>Smaller apps are less likely to have been tested for edge-case privacy violations or undergone rigorous penetration tests and security audits. And many have opaque or permissive data usage policies.</blockquote>

Even if an app’s creators have no malicious intent, weak oversight can lead to major leaks. Collard warns that user data could end up in:-<ul><li>Third-party data broker databases</li><li>AI training sets without consent</li><li>Cybercriminal marketplaces following a breach</li></ul>

In some cases, the apps might themselves be fronts for data-harvesting operations.

<h3>From individual oversights to corporate exposure</h3>

The consequences of oversharing aren’t limited to the person typing the prompt.

<blockquote>When employees feed confidential information into public GenAI tools, they can inadvertently expose their entire company. That includes client data, internal operations, product strategies – things that competitors, attackers, or regulators would care deeply about.</blockquote>

While unauthorised shadow AI remains a major concern, the rise of semi-shadow AI – paid tools adopted by business units without IT oversight – is increasingly risky, with free-tier generative AI apps like ChatGPT responsible for 54% of sensitive data leaks due to permissive licensing and lack of controls, <a target="_blank" class="editor-link" href="https://www.harmonic.security/resources/from-payrolls-to-patents-the-spectrum-of-data-leaked-into-genai#prId=343092">according to the Harmonic report</a>.

Responsible adoption starts with understanding the risk – and reining in the hype.

<blockquote>Businesses must train their employees on which tools are okay to use, and what’s safe to input and what isn't. And they should implement real safeguards – not just policies on paper. Cyber hygiene now includes AI hygiene. This should include restricting access to generative AI tools without oversight or only allowing those approved by the company. Organisations need to adopt a privacy-by-design approach when it comes to AI adoption. This includes only using AI platforms with enterprise-level data controls and deploying browser extensions that detect and block sensitive data from being entered.</blockquote>

As a further safeguard, she believes internal compliance programmes should align AI use with both data protection laws and ethical standards.

“I would strongly recommend companies adopt ISO/IEC 42001, an international standard that specifies requirements for establishing, implementing, maintaining and continually improving an Artificial Intelligence Management System (AIMS),” she urges.

Ultimately, by balancing productivity gains with the need for data privacy and maintaining customer trust, companies can succeed in adopting AI responsibly.

As businesses race to adopt these tools to drive productivity, that balance – between ‘wow’ and ‘whoa’ – has never been more crucial.

An AI generated image of two men in a workplace setting

Seemingly benign details can be stitched into detailed profiles by cybercriminals or data brokers – fuelling targeted phishing, identity theft, and sophisticated social engineering

How generative AI is leaking companies' secrets [Report]

ChatGPT has had more airtime than the world’s most renowned celebrity.

Its arrival has sparked questions and concerns that some did not even think to ask, including if it is a threat to our critical thinking skills.

Write this essay. Complete this analysis. Unpack the meaning behind this excerpt. These are all sentences that bring horror to those who do homework – lengthy essays about things that students are not sure they care about, and that many do not see the point in doing in the first place.

This homework is not designed to be terrifying; it is designed to ignite the mind and critical thinking; to challenge students to form their own opinions and then using research and cognitive thinking to back them up.

Currently, this is being threatened by tools like ChatGPT.

<h3>In 1956, the average human could retain 7 items in memory, in 2023 this number is 4</h3>

Yet, artificial intelligence (AI) tools are here to stay so instead of seeing it as a threat, it should be seen as an asset, but one that supports cognitive growth and memory function rather than replacing it, says Anna Collard, SVP Content Strategy &amp; Evangelist at KnowBe4 AFRICA.

“Research by George Millar in 1956 found that the average person can only keep around seven items in their working memory,” she explains.

Collard adds: “Miller’s Law of seven plus-minus two means up to nine or as little as five items are the limit of a human’s processing abilities. Today, that number has gone down to four. This research, along with other academic papers and analyses, points to reduced memory due to an overreliance on technology also called the Google effect.”

This is a concern, one that has grown increasingly vocal over the past year as educators and researchers have pondered the impact of technology, and now AI, on cognitive behaviour and memory retention.

<h3>ChatGPT, a tool that supports students in research &amp; studies</h3>

However, it is also balanced by research that has pointed out that actually, human beings have been outsourcing their memory to various materials and solutions for centuries. Paper, parchment, papyrus and wood are some prime examples.

Modern technology is no different. It can be a tool to bolster memory and make it far easier for humans to manage lives that are deluged by information, noise and digital clutter.

“The research goes in both directions, suggesting that technology is both an enabler and an inhibitor of human memory,” says Collard. “This points to the fact that actually, the impact does not lie in using it, but in how it is used or, in the case of ChatGPT, abused. The latter can be an immensely useful tool that supports students in their research and studies, but if it becomes the sole source of information and does all the writing for them, that is where the problems start.”

Nowadays, remembering where you found a fact has become more important than remembering the fact itself. This translates directly into the biggest challenge with ChatGPT – it is vague on its sources and often out of date, which means that it rarely is as accurate as it needs to be.

“ChatGPT uses machine learning to infer information and this introduces inaccuracies.

“If you ask ChatGPT what happens if you break a mirror, it replies with ‘You will have seven years of bad luck’. This is not a fact; it is based on superstition. If users do not constantly check the factual accuracy of ChatGPT, they run the risk of sharing fake news, inaccurate information and even conspiracy theories,” says Collard

<h3>Most generative AI models are only truthful 25% of the time</h3>

The TruthfulQA benchmark test has found that most generative models are only truthful 25% of the time, according to the&nbsp;2022 Stanford University <a href="https://apo-opa.info/3o0ZCGJ">Artificial Intelligence Index Report</a>.

This means over-reliance on technology such as ChatGPT is simply going to make the student’s work look stupid. However, that is not enough to stop people from turning to tech when they are tired, overwhelmed or lazy, which means that the best approach is to embrace it.

“Do not ban it, rather teach students how to use it within practical guidelines and policies that help them to enhance their understanding of AI and this type of tool.

“This will enhance their own critical thinking skills by asking them to question the sources, content, truthfulness and accuracy of the content that the platform serves up to them, and it will turn the threat into an opportunity,” concludes Collard.

The views and opinions expressed in this article are those of the author and do not necessarily reflect the position of Pulse as its publisher.

Student writing on a greenboard alongside a ChatGPT logo

Remembering where you found a fact has become more important than remembering the fact itself. This translates directly into the biggest challenge with ChatGPT – it is vague on its sources and often out of date.

Will ChatGPT impact critical thinking skills for the youth? [Opinion]

It is now possible to use a publicly available artificial chatbot to generate a complete infection chain, possibly beginning with a spear phishing email written in entirely convincing, human-like language and eventually causing a complete takeover of a company’s computer systems.

Researchers at Checkpoint recently created such a plausible phishing email as a test.

They only used ChatGPT, a chatbot that uses deep learning techniques to generate text and conversations that can convince basically anyone that it was written by a real person.

In reality, there are many potential cybersecurity dangers wrapped up in this impressive technology developed by OpenAI and currently available online for free.

<ol><li>Social engineering: ChatGPT’s powerful language model can be used to generate realistic and convincing phishing messages, making it easier for attackers to trick victims into providing sensitive information or downloading malware.</li><li>Scamming: The generation of text through ChatGPT’s language models allows attackers to create fake ads, listings and many other forms of scamming material.</li><li>Impersonation: ChatGPT can be used to create a convincing digital copy of an individual’s writing style, allowing attackers to impersonate their target in a text-based setting, such as in an email or text message.</li><li>Automation of attacks: ChatGPT can also be used to automate the creation of malicious messages and phishing emails making it possible for attackers to launch large-scale attacks more efficiently.</li><li>Spamming: The language model can be fine-tuned to produce large amounts of low-quality content, which can be used in a variety of contexts, including as spam comments on social media or in spam email campaigns.</li></ol>

All five points above are legit threats to companies and all internet users that will only become more prevalent as OpenAI continues to train its model.

If the list managed to convince you, the technology succeeded in its purpose, although in this instance not with malicious intent.

All the text from points one to five was actually written by ChatGPT with minimal tweaks for clarity. The tool is so powerful it can convincingly identify and word its own inherent dangers to cybersecurity.

However, there are mitigating steps individuals and companies can take, including new-school security awareness training. Cybercrime is moving at light speed.

A few years ago, cybercriminals used to specialise in identity theft, but now they take over your organisation’s network, hack into your bank accounts, and steal.

An intelligent platform like ChatGPT may have been created with the best intentions, but it only adds to the burden on internet users to always stay vigilant, trust their instincts and always know the risks involved in clicking on any link or opening an attachment.

An updated version of OpenAI's chatbot, ChatGPT, launched on November 30.CFOTO/Future Publishing via Getty Images

ChatGPT, is a chatbot that uses deep learning techniques to generate text and conversations that can convince basically anyone that it was written by a real person.

ChatGPT: What cybersecurity dangers lurk behind impressive new technology? [Opinion]

Politics

Local

World

News

Celebrities

Music

Movies

Entertainment

Sports 

Fashion

Beauty & Health

Relationships & Weddings

Food & Travel

Lifestyle

Domestic

International 

Business

About

Highlights

Selection Criteria

Jury

Terms And Conditions

FAQs

Pulse Influencer Awards

Courses

Careers

Explore categories

Pulse Kenya is the most followed news and entertainment platform in Kenya. It informs and engages young Kenyans around the clock and provides updates by the hour.

Pulselive Kenya

Anna Collard is the Senior Vice President of Content Strategy and Evangelist at KnowBe4 Africa.

Anna Collard

Articles written by the author

How generative AI is leaking companies' secrets [Report]

Will ChatGPT impact critical thinking skills for the youth? [Opinion]

ChatGPT: What cybersecurity dangers lurk behind impressive new technology? [Opinion]