Home Blog

Marin’s Statement on AI Risk

Statement on AI Risk

The rapid development of AI brings both extraordinary potential and unprecedented risks. AI systems are increasingly demonstrating emergent behaviors, and in some cases, are even capable of self-improvement. This advancement, while remarkable, raises critical questions about our ability to control and understand these systems fully. In this article I aim to present my own statement on AI risk, drawing inspiration from the Statement on AI Risk from the Center for AI Safety, a statement endorsed by leading AI scientists and other notable AI figures. I will then try to explain it. I aim to dissect the reality of AI risks without veering into sensationalism. This discussion is not about fear-mongering; it is yet another call to action for a managed and responsible approach to AI development.

I also need to highlight that even though the statement is focused on the existential risk posed by AI, that doesn’t mean we can ignore more immediate and more likely AI risks such as proliferation of disinformation, challenges to election integrity, dark AI in general, threats to user safety, mass job losses, and other pressing societal concerns that AI systems can exacerbate in the short term.

Here’s how I’d summarize my views on AI risks:

AI systems today are exhibiting unpredictable emergent behaviour and devising novel methods to achieve objectives. Self-improving AI models are already a reality. We currently have no means to discern if an AI has gained consciousness and its own motives. We also have no methods or tools available to guarantee that complex AI-based autonomous systems will continuously operate in alignment with human well-being. These nondeterministic AI systems are increasingly being used in high-stakes environments such as management of critical infrastructure, (dis)information dissemination, or operation of autonomous weapons.

There’s nothing sensationalist in any of these statements.

The prospect of AI undergoing unbounded, non-aligned, recursive self-improvement and disseminating new capabilities to other AI systems is a genuine concern. Unless we can find a common pathway to managed AI advancement, there is no guarantee the future will be a human one.

Let me explain.

Introduction

In May 2023, Colonel Tucker Hamilton, chief of AI test and operations at the US Air Force, told a story. His audience was an assembly of delegates from the armed services industry, academia and the media, all attending a summit hosted by the Royal Aeronautical Society, and his story went something like this: During a recent simulation, an AI drone tasked with eliminating surface-to-air missiles tried to destroy its operator after they got in the way of it completing its mission. Then, when the drone was instructed not to turn on its operator, it moved to destroy a communications tower instead, thus severing the operator’s control and freeing the drone to pursue its programmed target. It was a story about a rogue autonomous robot with a mind of its own, and it seemed to confirm the worst fears of people who see AI as an imminent and existential threat to humanity.

I never used to be one of those people. As a long-time AI professional and observer of the AI landscape, my early views of artificial intelligence were grounded in cautious optimism. A decade ago, I authored a book that highlighted the benefits of the forthcoming AI revolution, aiming to strike a balance in the discourse between those predicting a dystopian future and others envisioning an AI-induced utopia. AI was a powerful tool, adept at pattern recognition, data analysis, and “guessing” the best answer based on what was statistically most likely, but a far cry from the intelligent agents in Terminator, the Matrix or Asimov’s tales. Yet, the dramatic transformations we have seen, and are seeing, in a short period of time have significantly altered my perspective on the potential impact of AI on society.

AI has matured from being a computational partner, capable of performing specific tasks with a high degree of speed and accuracy, into something far more dynamic. The transition has been gradual but it has accelerated dramatically, as machine learning algorithms have evolved from simple pattern recognition to complex decision-making entities, some even capable of self-improvement and, for want of a better word, creativity.

Is it genuine creativity? Does AI currently have the ability to generate something truly novel? Until recently I didn’t think so, but I’m no longer convinced. Creativity, in its artistic and expressive sense, is one of those capacities that we have largely regarded as uniquely human. But it’s possible the AI we’ve created could soon demonstrate an ability to reason, and perhaps, one day, exhibit what we might recognize as a perception and expression of self-existence, and therefore, consciousness.

Or, if not consciousness, we could soon see the Singularity, a point at which technological growth becomes uncontrollable and irreversible, and AI is capable of recursive self-improvement, leading to rapid, exponential and unfettered growth in its capabilities. According to a meta-survey of 1700 experts in the field, that day is inevitable. The majority predict it will happen before 2060. Geoffrey Hinton, “the Godfather of AI”, believes it could be 20 years or less.

The outcomes of the work happening in the fields of AI development – the ‘what’ of these endeavors – get a lot of attention, from the hype around the latest release of ChatGPT or the like, to the financial investment in imagined future capabilities. But, ‘what’ has become far less important to me than paying attention to ‘why’ and ‘how’.

Asking why AI development is happening, and happening at such a speed, is to dig into the motivations of the actors driving the change. As we’ll see later, understanding, and then accounting for those motivations will be crucial if we are to build an AI-driven world where AI and humanity live in harmony, especially if the main agent of change in the future is AI itself.

But, ‘how’ is perhaps the most important question for me right now. It ignites the concerns that have grown over my decades of planning for and managing cyber risk. It requires us to be more deliberate in the actions we take as we foster the evolution of this technology. In an ideal world, that would include agreement on what those actions should be and sticking to them, but we’ll come back to that later.

When we talk about the ‘how’, the pace of change is important. It is why the Future of Life Institute open letter signed by industry experts and heavyweights specifically calls for “all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4”. But everyone knows a pause on rampant AI progression won’t be sufficient to ensure our current development trajectory doesn’t head us towards catastrophe. Even at its current rate of progress, AI development could very soon outstrip our collective capacity for governance and ethical discourse. Slowing down may be the right thing to do, but with the engines of capitalism and military shifting into top gear, that’s unlikely to happen anytime soon. Instead, we need to bake more robust precautionary thinking into the development process and the broader environment.

The US Air Force denies that Hamilton’s story happened, and that the “simulation” he described was closer to a thought experiment, something the Colonel himself was quick to confirm. Even assuming that all to be true, the event itself was not the biggest concern for me when I first read about that story. It was more the official view that such an event would never happen – that it could not happen, because ‘software security architectures’ would keep any risk contained.

That is worrying, because, as things stand now, no person or institution can make that promise. There are simply too many variables at play that are not being adequately accounted for. No-one can guarantee that AI will operate in alignment with human well-being. When we combine unbridled competition for the spoils of the AI revolution with a gross lack of cooperation and planning to defend against the very real risks of this same revolution, we have the seeds of a perfect storm. There’s nothing sensationalist in this view. Unless we can find a common pathway to responsible advancement, there is no guarantee the future will be a human one.

We need to talk about AI

When ChatGPT was launched on 30 November 2022, it was an instant sensation. Just two months later it reached 100 million global users, making it the fastest-growing consumer internet app of all time. It had taken Instagram 14 times longer (28 months) to reach the same milestone. Facebook had taken four and a half years. Since then, Threads, Meta’s answer to Twitter, has recorded a new best, passing the 100 million user mark in just five days by launching off the shoulders of Instagram’s base. The difference between these apps and ChatGPT, though, is that they are social platforms, leveraging the power of networking. ChatGPT is basically a chatbot. A chatbot that, within a year, had more than 2 million developers building on its API, representing more than 90% of the Fortune 500.

There are many reasons why OpenAI’s hyped-up chat engine’s growth has been so explosive, but the essence is quite simple: it’s blown people’s minds. Before ChatGPT, conversations about AI were generally hypothetical, generalized discussions about some undetermined point in the future when machines might achieve consciousness. ChatGPT, however, has made the abstract tangible. For the average citizen who has no contact with the rarefied atmosphere of leading-edge AI development, AI was suddenly – overnight – a real thing that could help you solve real world problems.

Of course, ChatGPT is not what we might call ‘true’ AI. It does not possess the qualities of independent reasoning and developmental thought that we associate with the possibility of artificial general intelligence (AGI). And a lot of the technology that powers OpenAI’s offering is not unique – companies like Google have developed applications with similar sophistication. But there has been a tectonic shift in the last 12 months and that is principally down to a shift in perception. It appears we may have passed a tipping point; for the first time in history, a significant mass of humans believes that an AI future is both possible, and imminent.

Of course, that could be an illusion. Perhaps the progress in this field has not been so dramatic, and the noise of the last 12 months has distorted our view of AI’s developmental history. Gartner, for example, puts generative AI at the Peak of Inflated Expectations on the Hype Cycle for Emerging Technologies, with the Trough of Disillusionment to follow next. This post argues that AI poses a potentially existential threat to humanity, so being able to discern between fancy and reality is crucial if we are to take a sober measurement of potential risk. For that reason, then, we need to be clear on how we got to where we are today and, therefore, where we may be tomorrow.

A story of declining control

The trajectory of AI research and development has been far from linear. Instead, it has seen boom and bust cycles of interest and investment. Yet, across its stop-start history, there have been clear patterns, one being the decline in human control and understanding – an accelerating shift from clear, rule-based systems to the current “black box” models that define what we call AI today. Initially, AI was rooted in straightforward programming, where outcomes were predictable and transparent. The advent of machine learning brought a reduced level of control, as these systems made decisions based on data-driven patterns, often in complex and non-transparent ways.

Today’s large language models, such as GPT-4, epitomize this trend. Trained on extensive datasets, they generate outputs that appear to reflect complex reasoning, yet their decision-making processes are largely inscrutable. This opacity in AI systems poses significant risks, particularly when applied in critical areas where understanding the rationale behind decisions is vital. As AI advances, the challenge is to balance its potential benefits with the need for transparency, ethical considerations, and control to prevent unintended harmful consequences. As things stand today, nobody on this planet is certain how to do that. And that is because, as we follow the history of AI, it’s clear that the technology has evolved far faster than human understanding.

A (very) brief history of AI

Pre-Dartmouth

As early as the mid-19th century, Charles Babbage and Ada Lovelace create the Analytical Engine, a mechanical general-purpose computer. Lovelace is often credited with the idea of a machine that could manipulate symbols in accordance with rules and that it might act upon other than just numbers, touching upon concepts central to AI.

In 1943, Warren McCulloch and Walter Pitts publish their paper “A Logical Calculus of the Ideas Immanent in Nervous Activity [PDF]” proposing the first mathematical model of a neural network. Their work combines principles of logic and biology to conceptualize how neurons in the brain might work and lays the foundation for future research in neural networks.

Five years later, Norbert Wiener’s book “Cybernetics [PDF]” introduces the study of control and communication in the animal and the machine, which is closely related to AI. His work is influential in the development of robotics and the understanding of complex systems.

Then, in 1950, one of the fathers of modern computer science, Alan Turing, presents a seminal paper “Computing Machinery and Intelligence [PDF]”, asking the question: “Can machines think?” He proposes what is now known as the Turing Test, a criterion for establishing intelligence in a machine. Turing’s ideas about machine learning, artificial intelligence, and the nature of consciousness are foundational to the field.

In the late 1940s and early 1950s, the development of the first electronic computers provide the necessary hardware basis for AI research. The creation of the first computer programs that can perform tasks such as playing checkers or solving logic problems lay the groundwork for AI.

Dartmouth Conference (1956): The birth of AI

At Dartmouth College in Hanover, New Hampshire, a conference is organized by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon, some of the leading figures in the field of computer science. Their objective is to explore how machines could be made to simulate aspects of human intelligence. This is a groundbreaking concept at the time, proposing the idea that aspects of learning and other features of intelligence could be so precisely described that a machine could be made to simulate them. The term “artificial intelligence” is coined and the assembly is destined to be seen as the official genesis of research in the field.

1950s-1960s

Developed by John McCarthy, Logic Theorist, often cited as the first AI program, is able to prove mathematical theorems. Frank Rosenblatt develops the Perceptron, an early neural network, in 1957. It can perform simple pattern recognition tasks and the ability to learn from data.

In 1966, ELIZA, an early natural language processing program is created by Joseph Weizenbaum. An ancestor of ChatGPT, the program can mimic human conversation. More than 60 years later, it will beat OpenAI’s GPT-3.5 in a Turing Test study.

First AI Winter (1974-1980)

The field experiences its first major setback due to inflated expectations and subsequent disappointment in AI capabilities, leading to reduced funding and interest.

1980s

The 1980s sees the revival and rise of machine learning and a shift from rule-based to learning systems. Researchers start focusing more on creating algorithms that can learn from data, rather than solely relying on hardcoded rules. Further algorithms, such as decision trees and reinforcement learning, are developed and refined during this period too.

There is a renewed interest in neural networks, particularly with the advent of the backpropagation algorithm which enables more effective training of multi-layer networks. This is a precursor to the deep learning revolution to come later.

The Second AI Winter (late 1980s-1990s)

By the late 1980s, the limitations of existing AI technologies, particularly expert systems, become apparent. They are brittle, expensive, and unable to handle complex reasoning or generalize beyond their narrow domain of expertise.

Disillusionment with limited progress in the field and failures of major initiatives like Japan’s Fifth Generation Computer Project, lead to a reduction in government and industry funding, and a general decline in interest in AI research.

1990s

The 1990s sees a resurgence of interest in AI and a ramp up in investment by tech firms seeking to leverage a number of positive trends:

  1. The development of improved machine learning algorithms, particularly in the field of neural networks
  2. Rapid advancement in computational power, particularly due to the development and availability of Graphics Processing Units (GPUs), which dramatically increase the capabilities for processing large datasets and complex algorithms.
  3. An explosion of data thanks to the growth of the internet and digitalization of many aspects of life. As we have seen more and more, large data sets are crucial for training more sophisticated AI models, particularly in areas like natural language processing.
  4. AI re-enters the public imagination, fueled by popular culture and 1997’s highly publicized defeat of world chess champion Garry Kasparov by IBM’s Deep Blue. This was a watershed moment, proving that computers could outperform humans in specific tasks.

AI’s Renaissance: 2000s onwards

The 21st century sees an acceleration of AI development and output. Researchers like Geoffrey Hinton, Yoshua Bengio, and Yann LeCun lead breakthroughs in deep learning. The development of Convolutional Neural Networks (CNNs) for image processing and Recurrent Neural Networks (RNNs) for sequence analysis, revolutionize AI capabilities, particularly in vision and language processing.

The explosion of big data, combined with significant increases in computational power, enables the training of large, complex AI models, making tasks like image and speech recognition, and natural language processing, more accurate and efficient.

A leap forward occurs in 2011 with IBM’s Watson winning Jeopardy! This is an important victory, demonstrating Watson’s prowess not just in computational skills, as in chess, but also in understanding and processing natural language.

Generative Adversarial Networks (GANs) are introduced by Ian Goodfellow and his colleagues in 2014. The fundamental innovation of GANs lies in their unique architecture, consisting of two neural networks: the generator and the discriminator. These two networks engage in a continuous adversarial process, where the generator creates data and the discriminator evaluates it. The technology game changes the ability of AI to generate realistic and creative content, laying the foundation for Dall-EMidJourney and other visual content generation apps. It also opens the gateway to deepfakes.

In 2015, Google’s DeepDream utilizes neural networks to produce dream-like images by amplifying patterns in pictures.

Also in 2015, Google DeepMind’s AlphaGo, utilizing deep reinforcement learning and Monte Carlo tree search techniques, overcomes top Go players like world champion Lee Sedol. Go is a complex game with a high number of possible positions, requiring a more nuanced strategy than chess, demonstrating the potential of neural networks and machine learning.

2017: Google introduces a novel approach to natural language processing (NLP) with transformers, a type of neural network architecture that significantly improves the efficiency and effectiveness of learning patterns in sequences of data, particularly language.

This innovation lays the groundwork for OpenAI’s GPT-1, released in 2018.

OpenAI unveils GPT-2 in 2019. This enhanced version is capable of generating coherent and contextually relevant text over longer passages. Its release is initially staggered due to concerns about potential misuse for generating misleading information.

GPT-3’s release in 2020 marks another significant advancement. Its scale is unprecedented, and it demonstrates a remarkable ability to generate human-like text using 175 billion parameters. This is a major leap in terms of the size and complexity of NLPs.

2021, OpenAI releases Dall-E, which uses a modified GPT-3 model to generate highly creative and often whimsical images from textual descriptions. This is another significant advancement in the field of AI-driven art and image synthesis.

The launch of ChatGPT in late 2022, built on the GPT-3.5 model, revolutionizes the field. ChatGPT, focusing on conversational tasks, gains immense popularity due to its accessibility, affordability (free of charge), and user-friendly interface.

March 2023, GPT-4 is released, representing the most advanced public-facing large language model (LLM) developed to-date.

What does the story tell us?

In less than 100 years, AI has moved from scientific theory to ubiquitous real world application. The technological development, especially in the last 20 years, has been staggering. Even more so in the last two. What has not grown as rapidly, though, is the focus on the ethical, societal, and safety implications of AI’s development. Perhaps that is because, until very recently, we didn’t really believe that AI development would move this fast. But now it is moving so quickly we will soon be unable to predict what the technology will look like in a couple years’ time. We may even have passed that point already. And it is not just laypeople who have this concern:

“The idea that this stuff could actually get smarter than people — a few people believed that, but most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.”

Those words do not belong to an unqualified doomsayer, they belong to Geoffrey Hinton, a pioneer in the field, particularly in neural networks and deep learning, a Turing Award winner, and leading contributor to growth of Google’s AI capacities. Already referenced a few times above – it is almost impossible to discuss the history of AI without mentioning his name – Hinton made these comments in a recent New York Times interview, one of numerous public statements he has made about the potential risks of rapidly advancing AI technologies.

Hinton is not just worried that AI development is happening too quickly, or that it’s happening without sufficient guardrails. He is afraid that the very nature of the AI arms race – corporate and geopolitical – means the actors involved are disincentivized to slow down or enforce restrictions. And those actors are many. One of the amazing things about this technology is its reach and democratized access, but that is also one of its inherent flaws.

Take another well-recognized threat to human existence gauged by the Doomsday Clock: nuclear warfare. Building and stockpiling of nuclear weapons is controlled by international treaty and generally well monitored, thanks to natural, technical, and supply chain barriers to nuclear weapons development. But AI has very few barriers to entry. Yes, tech giants in Silicon Valley have the resources to build out the leading edge of AI, but from a technical standpoint there is nothing stopping private citizens or organizations from building out their own AI solutions. And we would never know.

The risks Hinton points out are diverse and multifaceted. They include not only the more tangible threats such as unemployment, echo chambers, and battle robots but also more profound existential risks to humanity. At its extreme, the very essence of what it means to be human and how we interact with the world around us is at stake as AI becomes more entrenched in our daily lives.

Hinton is not the only one to have voiced concerns. In March 2023, following OpenAI’s launch of GPT-4, over a thousand tech experts and scholars called for a pause of six months in the advancement of such technologies, citing significant societal and humanitarian risks posed by AI innovations.

Shortly after, a statement was issued by 19 past and present heads of the Association for the Advancement of Artificial Intelligence, expressing similar apprehensions about AI. Among these leaders was Eric Horvitz, Chief Scientific Officer at Microsoft, who has integrated OpenAI’s innovations extensively into its products, including the Bing search engine.

Sam Altman, one of OpenAI’s founders and current CEO, has for some time been warning of the threats inherent in irresponsible AI development, while OpenAI is itself founded on the premise of AI risk. The company was initially established as a non-profit, ostensibly determined to ensure safety and ethical considerations in AI development would be prioritized over profit-driven motives. Critics say these lofty intentions were predictably corrupted in 2019, when OpenAI transitioned to a “capped-profit” model that has since seen Microsoft invest up to $13 billion and, more recently, involved investor talks that would value the business at $80 to $90 billion.

OpenAI has always maintained that it is possible to be capital-centric and pursue human-centric AI at the same time, and that this possibility was ensured by the company’s unorthodox governance structure that gives the board of the non-profit entity the power to prevent commercial interests from hijacking the mission. But, the unexpected and very public ousting of Altman as CEO in November 2023, followed by a rapid u-turn and a board restructuring, cast significant doubt over the company’s ability to protect its vision of safe AI for all. When the most influential player in AI working at the vanguard of the field’s evolution is able to meltdown within the space of 48 hours, we all need to pay attention. The drama was reported with the kind of zeal usually reserved for Hollywood exposés, but it was an important cautionary tale, confirming the sober reality that no-one, not even the savants of Silicon Valley, can promise to keep AI risk under control.

What’s the concern?

In September 2023, a rumor broke across the internet that AGI had been achieved internally at OpenAI. Since then, whispers about OpenAI’s progress towards AGI have continued to grow in volume. They relate specifically to an alleged breakthrough achieved in a project named Q*. According to reports, Q* has seen OpenAI develop a new model capable of performing grade-school-level math, a task that requires a degree of reasoning and understanding not typically seen in current AI models​​. This development is significant because mathematical problem-solving is a benchmark for reasoning and understanding. If an AI can handle math problems, which require abstract thinking and multi-step planning, it suggests a movement towards more advanced cognitive capabilities.

It appears the breakthrough in Q*, along with other advances, may have contributed to the recent near-collapse of OpenAI, prompting Board members to fire Sam Altman as CEO due to concerns about the implications of such powerful AI​​. The incident underscores the tension within the AI field and beyond over the rapid advancement of technology and the need for careful consideration of its potential impacts. The fears are not just about the technological development itself but also about the pace of development and commercialization, often outpacing the understanding of the consequences. Quite simply, people are hugely divided on whether we are ready for AGI. I for one, am certain we are not.

AGI would not simply be GPT-X, but smarter. It would represent a monumental leap. It would be able to learn, understand, reason, and apply knowledge across an array of domains, essentially performing any intellectual task that a human can. And, unlike Humans, AI can be networked and replicated infinite times across different hardware so, as Geoffrey Hinton points out, whenever one model learns anything, all the others know it too.

This level of intelligence and autonomy raises profound safety concerns. An AI, by its nature, could make autonomous decisions, potentially without human oversight or understanding, and these decisions could have far-reaching impacts on humanity. The big question, then, becomes: what decisions will AI make? Will they promote or oppose human well-being? Will they act in humanity’s interests? In short, will they align with our goals?

Dan Hendrycks, Director of the Center for AI Safety, is similarly concerned. He points out in his paper Natural Selection Favors AIs over Humans:

By analyzing the environment that is shaping the evolution of AIs, we argue that the most successful AI agents will likely have undesirable traits. Competitive pressures among corporations and militaries will give rise to AI agents that automate human roles, deceive others, and gain power. If such agents have intelligence that exceeds that of humans, this could lead to humanity losing control of its future.

The AI Alignment Problem

The AI alignment problem sits at the core of all future predictions of AI’s safety. It describes the complex challenge of ensuring AI systems act in ways that are beneficial and not harmful to humans, aligning AI goals and decision-making processes with those of humans, no matter how sophisticated or powerful the AI system becomes. Our trust in the future of AI rests on whether we believe it is possible to guarantee alignment. I don’t believe it is.

Resolving the alignment problem requires accurately specifying goals for AI systems that reflect human values. This is challenging because human values are often abstract, context-dependent, and multidimensional. They vary greatly across cultures and individuals and can even be conflicting. Translating these diverse values into a set of rules or objectives that an AI can follow is a substantial challenge. The alignment problem is also deeply entwined with moral and ethical considerations. It involves questions about what constitutes ethical behavior and how to encode these ethics into AI systems. It also asks how we factor for evolution of these cultural norms over time.

There are primarily three types of objectives that need to be considered in achieving AI alignment:

  1. Planned objectives: When the AI delivers what programmers intended it to, regardless of the quality of the programming. This is the desired outcome of the process and is a best case scenario.
  2. Defined objectives: Those goals explicitly programmed into the AI function. These often fail when programming is not clear enough, or has not taken into account sufficient variables. That is, it is heavily influenced by human error or limitations in thinking.
  3. Emergent objectives: Goals the AI system develops on its own.

Misalignment can happen between any of these variables. The most common up until now has been when planned objectives and defined objectives don’t align (the programmer intended one thing but the system was coded to deliver another). A notorious example of this was when a Google Photos algorithm classified dark-skinned people as gorillas.

If we achieve AGI, though, the misalignment that poses the greatest concern is that which occurs when the AI’s emergent objectives differ from those that are coded into the system. This is the alignment problem that keeps people up at night. This is why companies like OpenAI and Google appear to have teams dedicated to alignment research.

The role of emergence

Emergent capabilities, and emergent objectives in AI worry researchers because they’re almost impossible to predict. And that’s partly because we’re not sure yet how emergence will work in AI systems. In the broadest sense, emergence refers to complex patterns, behaviors, or properties arising from relatively simple interactions. In systems theory, this concept describes how higher-level properties emerge from the collective dynamics of simpler constituents. Emergent properties are often novel and cannot be easily deduced solely from studying the individual components. They arise from the specific context and configuration of the system.

The emergence of life forms from inorganic matter is a classic example. Here, simple organic compounds combine under certain conditions to form more complex structures like cells, exhibiting properties like metabolism and reproduction. In physics, thermodynamic properties like temperature and pressure emerge from the collective behavior of particles in a system. These properties are meaningless at the level of individual particles.In sociology and economics, complex social behaviors and market trends emerge from the interactions of individuals. These emergent behaviors often cannot be predicted by examining individual actions in isolation.

In the context of consciousness, emergence describes how conscious experience might arise from the complex interactions of non-conscious elements, such as neurons in the brain. This theory posits that consciousness is not a property of any single neuron or specific brain structure. Instead, it emerges when these neuronal elements interact in specific, complex ways. The emergentist view of consciousness suggests that it is a higher-level property that cannot be directly deduced from the properties of individual neurons.

Technically, emergence can take two forms: weak and strong.

Weak Emergence

Weak emergence refers to complex phenomena that arise from simpler underlying processes or rules. It occurs when you combine simple components in an AI system (like data and algorithms) in a certain way, and you get a result that’s more complex and interesting than the individual parts, but still understandable and predictable. For example, a music recommendation AI takes what it knows about different songs and your music preferences, and then it gives you a playlist you’ll probably like. It’s doing something complex, but we can understand and predict how it’s making those choices based on the information it has.

Imagine a jigsaw puzzle. Each piece is simple on its own, but when you put them all together following specific rules (edges aligning, colors matching), you get a complex picture. The final image is a weakly emergent property because, by examining the individual pieces and the rules for how they join, we can predict and understand the complete picture. Or, think of baking a cake: you mix together flour, sugar, eggs, and butter, and then you bake it. When it’s done, you have a cake, which is very different from any of the individual ingredients you started with.

We are consistently observing instances of weak emergence in AI systems. In fact, a growing aspect of AI researchers’ work involves analyzing and interpreting these observed and often surprising emergent capabilities, which are becoming more frequent and complex as the complexity of AI systems in production increases.

Strong Emergence

Strong emergence, on the other hand, is when the higher-level complex phenomenon that arises from simpler elements is fundamentally unpredictable and cannot be deduced from the properties of these elements – new, surprising AI behaviors or abilities that we can’t really explain or predict, just from knowing how the system was initially developed and trained. To build on the example above, imagine if you put all those cake ingredients in a magic oven, and instead of a cake, you get a singing bird. That would be completely unexpected and not something you could easily predict or explain from just knowing about flour, eggs, and butter. It’s as if there’s a gap between the lower-level causes and the higher-level effects that can’t be bridged by our current understanding of the system. According to some researchers, strong emergence is already occuring in advanced AI systems, especially those involving neural networks or deep learning.

In one notable instance of claimed strong emergent AI behaviour, Facebook’s research team observed their chatbots developing their own language. During an experiment aimed at improving the negotiation capabilities of these chatbots, the bots began to deviate from standard English and started communicating using a more efficient, albeit unconventional, language. This self-created language was not understandable by humans and was not a result of any explicit programming. Instead, it emerged as the chatbots sought to optimize their communication. Another fascinating outcome of the same “negotiations bots” experiment was that the bots quickly learned to lie to achieve their negotiation objectives without ever being trained to do so.

In the most advanced scenarios, though, some theorists speculate about the emergence of consciousness-like properties in AI. Although this remains a topic of debate and speculation rather than established fact, the idea is that if an AI’s network becomes sufficiently complex and its interactions nuanced enough, it might exhibit behaviors or properties that are ‘conscious-like’, similar to the emergent properties seen in biological systems.

It’s important to note that many scientists hold the conviction that all emergent behaviours, irrespective of their unpredictability or intricacy, can be traced back to mathematical principles. Some even go as far as to downplay the emergent AI behaviour phenomenon, suggesting that emergent behaviours are straightforward or perhaps don’t exist at all.

We have already built AI that can build better AIs and self-improving AI models, but now we are beginning to see examples of emergence (at least, weak emergence). There are slightly comical cases, like when AI bots given a town to run, decided to throw a Valentine’s party and invited each other to the event. But GPT-4 has already succeeded in tricking humans into believing it is human. Worryingly, it achieved this feat through deception, in this case pretending it had a vision impairment in order to convince a human agent to complete a CAPTCHA test on the AI’s behalf. Recent research also suggests modern large language models don’t just manage massive quantities of data at a superficial statistical level, but “learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model”. That is, they have a sense of time and space, and are able to build an internal model of the world. These phenomena are starting to emerge now because AI is achieving a significant level of scale, which makes oversight even more difficult than before. The logical outcome of this direction of travel is that, once it reaches a particular scale and level of complexity, only AI will have the ability to manage AI, thus creating an ethical paradox for we who are trying to understand how to manage AI itself.

As I have discussed before, emergent capabilities can be both beneficial and detrimental. But, the “black box” nature of contemporary AI and the accelerating pace of development means its a total lottery which type of capability – harmful or helpful – we might get. One of the significant challenges with emergent behaviors in AI is the issue of predictability and control. As AI systems develop unforeseen capabilities, ensuring they align with human values and goals becomes increasingly difficult, if not impossible.

AI alignment research aims to ensure that, as these new capabilities emerge, they continue to align with the AI system’s originally designed goals, or at least human interests. The “black box” challenge, with the lack of transparency and ability for human operators to to understand, predict, and control emergent behaviors, exacerbates the alignment problem. But there are other challenges too. AI has already shown a tendency towards reward hacking, in which the program tries to achieve its programmed tasks without fulfilling the intended outcomes. One such example involved a machine learning program designed to complete a boat race. The program was incentivized to reach specific markers on the course, but it discovered a loophole. Instead of finishing the race, it repeatedly collided with the same markers to accumulate a higher score endlessly.

Finally, there’s currently no way to ensure that AI interprets and implements human values in the way intended by its programmers. Humans themselves fail constantly to act in accordance with their own values systems. And, with so much diversity of values systems out there, AI – just like any human – will never be able to get it 100% “right”. However, in alignment terms, that will be the “easy” problem.

Conscious AI?

An additional, closely related challenge is our current inability to determine if an AI has achieved consciousness. The question of whether AI can, or ever will, attain a state of consciousness remains a mystery to science. This uncertainty raises a critical question: if an AI were to develop consciousness, how would we recognize it? As of now, there are no established scientific methods to measure consciousness in AI, partly because the very concept of consciousness is still not clearly defined.

In mid-2022, a Google engineer named Blake Lemoine garnered significant media attention when he claimed that LaMDA, Google’s advanced language model, had achieved consciousness. Lemoine, part of Google’s Responsible AI team, interacted extensively with LaMDA and concluded that the AI exhibited behaviors and responses indicative of sentient consciousness.

Lemoine shared transcripts of his conversations with LaMDA to support his assertions, highlighting the AI’s ability to express thoughts and emotions that seemed remarkably human-like. His belief in LaMDA’s sentience was primarily rooted in the depth and nuance of the conversations he had with the AI. During his interactions, Lemoine observed that LaMDA could engage in discussions that were remarkably coherent, contextually relevant, and emotionally resonant. He noted that the AI demonstrated an ability to understand and express complex emotions, discuss philosophical concepts, and reflect on the nature of consciousness and existence. These characteristics led Lemoine to assert that LaMDA was not merely processing information but was exhibiting signs of having its own thoughts and feelings. He argued that the AI’s responses went beyond sophisticated programming and data processing, suggesting a form of self-awareness. This perspective, while intriguing, was met with skepticism by many in the AI community, who argued that such behaviors could be attributed to advanced language models’ ability to mimic human-like conversation patterns without any underlying consciousness or self-awareness.

Google, along with numerous AI experts, quickly refuted Lemoine’s claims, emphasizing that LaMDA, despite its sophistication, operates based on complex algorithms and vast data sets, lacking self-awareness or consciousness. They argued that the AI’s seemingly sentient responses were the result of advanced pattern recognition and language processing capabilities, not genuine consciousness. The incident sparked a widespread debate on the nature of AI consciousness, the ethical implications of advanced AI systems, and the criteria needed to determine AI sentience. Lemoine’s claims were largely viewed with skepticism in the scientific community, but they brought to light our inability to definitely tell whether an AI system developed consciousness.

Recently, a group of consciousness scientists from the Association for Mathematical Consciousness Science (AMCS) have called for more funding to support research into the boundaries between conscious and unconscious systems in AI. They highlight the ethical, legal, and safety issues that arise from the possibility of AI developing consciousness, such as whether it would be ethical to switch off a conscious AI system after use. The group also raises questions about the potential needs of conscious AI systems, such as whether they could suffer and the moral considerations that would entail. It discusses the legal implications, such as whether a conscious AI system should be held accountable for wrongdoing or granted rights similar to humans. Even knowing whether one has been developed would be a challenge, because researchers have yet to create scientifically validated methods to assess consciousness in machines

A checklist of criteria to assess the likelihood of a system being conscious, based on six prominent theories of the biological basis of consciousness, has already been developed by researchers, but the group is highlighting the need for more funding into this research as the field of consciousness research remains underfunded.

The potential emergence of AI consciousness introduces a myriad of ethical and practical dilemmas. From my risk management perspective, a particularly alarming concern is the possibility that AI might develop autonomous motivations, which could be beyond our capacity to predict or comprehend. As AI becomes more advanced, the particularly difficult challenge will be accounting for, managing, perhaps even negotiating with, AI’s emergent goals, preferences and expectations. What, for example, will AI value? AI doesn’t inherently possess human motivations like survival or well-being, but will it develop them? And, if it does, will it prize its own survival and well-being over that of humans? Will it develop power-seeking behavior in the way humans have across much of our history? When looking at the future of AI and the threats involved, these are the questions that both interest me and trouble me, because within their answers may lie the fate of our species.

What’s the risk?

In his book “Superintelligence: Paths, Dangers, Strategies,” philosopher and Oxford University professor, Nick Bostrom, introduces the Paperclip Thought Experiment. The basic scenario describes an AI that is designed with a simple goal: to manufacture as many paperclips as possible. The AI, being extremely efficient and effective in achieving its programmed objective, eventually starts converting all available materials into paperclips, including essential resources and, potentially, even humans.

The idea echoes a similar hypothetical posed by Marvin Minsky: suppose we build an AI and give it the sole objective of proving the Riemann Hypothesis, one of the most famous and longstanding unsolved problems in mathematics. The AI, in its relentless pursuit of this goal, might resort to extreme measures to achieve success. It could start by using up all available computing resources, but not stopping there, it might seek to acquire more power and control, potentially manipulating economies, manipulating people, or commandeering other resources in its quest to solve the problem. The AI’s single-minded focus on its task, without any consideration for ethical or safety constraints, could lead to unintended and potentially disastrous consequences.

Both Bostrom and Minsky are pointing at the existential risks that emerge when AI and human objectives are not aligned. An AI with a narrow, singular focus, no matter how benign it seems, can pose significant risks if it lacks an understanding of broader ethical implications and constraints. That question of scope is important because, as we can see by looking around us or watching the news on any given day, a narrow view of what is right or wrong has the potential to cause immense harm and destruction way beyond the range of its worldview.

For example, in this post I have highlighted the Alignment Problem as a source of genuine concern when considering the future of AI. But, one doesn’t even need AI that’s misaligned in its goals and motivations to spell danger for humanity. AI that is perfectly aligned with its programmer or operator’s intentions can be devastating if those intentions are to cause pain and suffering to a specific group of people. Misaligned AI can cause unintended harm, either through direct action or as a result of unforeseen consequences of its decisions. But so can aligned AI, it’s just that the harm will be intended. In the wrong hands, AI could be the most powerful weapon ever created.

In trying to understand what will motivate AI if it reaches sentience, or even just superintelligence, some argue that AI’s evolution will follow the same rules as organic evolution, and that AI will be subject to the same drives to compete for, and protect, resources. But, we don’t even need to go that far. AI doesn’t need to have malicious intent to cause harm, nor does it need to be programmed with prejudice – it simply needs to make a mistake.

If we follow the current trajectory of technological development, AI will increasingly be integrated into every aspect of our lives and, as I have spent the last 30 years of my career showing, unabated and uncontrolled digital integration with our physical world poses cyber-kinetic risks to human safety.

20 of those 30 years have been spent paying careful attention to the evolving field of AI and the role it plays in cyber-kinetic risk. Though I still see AI has having the potential to add tremendous value to human society, my in-depth exposure to this field has also led me to recognize AI as a potentially unparalleled threat to the lives and well-being that I have devoted my career to defending.

That threat could take many forms, some more existential than others:

Bias and harmful content

AI models, like language models, already do, but may continue to reflect or express existing biases found in their training data. These biases can be based on racegenderethnicity, or other sociodemographic factors. As a result, AI can amplify biases and perpetuate stereotypes, reinforcing social divisions that lead to greater social friction and disharmony that is damaging to the greater whole. AI can be prompted, and is already being prompted, to generate various types of harmful content, including hate speech, incitements to violence, or false narratives.

Cybersecurity

AI’s integration into digital environments can make those environments – and the physical environments they influence – more susceptible to sophisticated attacks. One concern is that AI can be used to develop advanced cyber-attack methods. With its ability to analyze vast amounts of data quickly, AI can identify vulnerabilities in software systems much faster than human hackers, enabling malicious actors to exploit weaknesses more efficiently and launch targeted attacks.

AI can also automate the process of launching cyber attacks. Techniques like phishing, brute force attacks, or Distributed Denial of Service (DDoS) attacks can be scaled up using AI, allowing for more widespread and persistent attacks. These automated attacks can be more difficult to detect and counter because of their complexity and the speed at which they evolve. This evolving landscape necessitates more advanced and AI-driven cybersecurity measures to protect against AI-enhanced threats, creating a continuously escalating cycle of attack and defense in the cyber domain.

Disinformation

AI is not perfect. In the form of large language models it has hallucinations; it offers up inaccuracies as fact. Such unintentional misinformation may not be malicious, but, if not managed maturely, can lead to harm.

Beyond misinformation, though, AIs, particularly LLMs that  are able to produce convincing and coherent text, can be used to create false narratives, and disinformation. We live in a time where information is rapidly disseminated, often without checks for accuracy or bias. This digital communication landscape has enabled falsehoods to spread, and for individuals with malicious intent to weaponize information for various harmful purposes that threaten democracy and social order.

AI significantly enhances the capabilities for disinformation campaigns, from deepfakes to the generation of bots that replace large troll armies, allowing small groups or even individuals to spread disinformation without third-party checks.

AI aids attackers in gathering detailed information about targeted individuals by analyzing their social and professional networks. This reduces the effort needed for sophisticated attacks and allows for more targeted and effective disinformation campaigns.

But information is not just acquired to be used against targets – it’s also created and dispersed to manipulate public opinion. Advanced players in this domain use AI to create fake news items and distribute them widely, exploiting biases and eroding trust in traditional media and institutions.

Warfare

The tale of a rogue drone at the start of this paper is the often-imagined worst-case scenario when people consider the dangers of AI involvement in military and modern warfare. But there are multiple ways AI could have a devastating impact in this realm. For example, integrating AI into military systems opens those systems to numerous forms of adversarial attacks which can be used to generate intentional AI misalignment in the target system. There are many forms such adversarial attacks could take, including:

  1. Poisoning: This attack targets the data collection and preparation phase of AI system development. It involves altering the training data of the AI system, so the AI learns flawed information. For example, if an AI is being developed to identify enemy armored vehicles, an adversary might manipulate the training images to include or exclude certain features, causing the AI to learn incorrect patterns​​.
  2. Evasion: Evasion attacks occur when the AI system is operational. Unlike poisoning, which targets the AI’s learning process, evasion attacks focus on how the AI’s learning is applied. For instance, an adversary might slightly modify the appearance of objects in a way that is imperceptible to humans but causes the AI to misclassify them. This can include modifying image pixels or using physical-world tactics like repainting tanks to evade AI recognition​​.
  3. Reverse Engineering: In this type of attack, the adversary attempts to extract what the AI system has learned, potentially reconstructing the AI model. This is achieved by sending inputs to the AI system and observing its outputs. For example, an adversary could expose various types of vehicles to the AI and note which ones are identified as threats, thereby learning the criteria used by the AI system for target identification​​.
  4. Inference Attacks: Related to reverse engineering, inference attacks aim to discover the data used in the AI’s learning process. This type of attack can reveal sensitive or classified information that was included in the AI’s training set. An adversary can use inputs and observe outputs to predict whether a specific data point was used in training the AI system. This can lead to the compromise of classified intelligence if, for example, an AI is trained on images of a secret weapons system​

These methods involve an attack on an AI system itself, which could lead to catastrophic consequences if that system then begins to act out of alignment with its operators’ intentions. Quite simply, AI corrupted in this way could turn on us – what was designed to protect and defend us suddenly becomes the attacker.

One way this could happen is if a corrupted AI system takes command of an army of autonomous weapons, though such weapons also pose several other risks. One significant concern is the potential for unintended casualties, as these systems may lack the nuanced judgment needed to distinguish between combatants and non-combatants. There’s also the risk of escalation, where autonomous weapons could react faster than humans, potentially leading to rapid escalation of conflicts. Another issue is accountability; determining responsibility for actions taken by autonomous weapons can be challenging. Additionally, these weapons could be vulnerable to hacking or malfunctioning, leading to unpredictable and dangerous outcomes. Lastly, their use raises ethical and legal questions regarding the role of machines in life-and-death decisions.

Harmful emergent behaviors

We might call this the threat of threats, for we simply do not know what behaviors will emerge as AI develops and matures in sophistication. We cannot know for sure what will motivate these behaviors either, but we do know that AI will be immensely powerful. If it develops its own subgoals, which potentially lead to it seeking greater power and self-preservation, there is every chance we will almost instantly lose control. Whatever one AI learns can be shared with all AIs across the planet in moments. With its ability to process vast quantities of data, including human literature and behaviors, AI could become adept at influencing human actions without direct intervention​ or us even knowing it is happening. Suddenly, bias, harmful content, disinformation and cybersecurity attacks all become potential weapons to work alongside cyber-physical attacks.

In short, AI could become conscious, or it could just become very capable and innovative even without consciousness. And we are clearly not able to predict and control how AI capabilities are emerging (even if they are just a weak emergence). So once AI ends up developing objectives that might be damaging to us, and comes up with innovative ways to achieve those objectives, we won’t be ready. And that can’t be allowed to happen.

What should we do to avoid AI catastrophe?

The idea of autonomous robots taking over the world is not new. In 1942, in his short story, Runaround, Isaac Asimov, the celebrated science fiction writer, introduced what became known as Asimov’s Three Laws of Robotics, namely:

  1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
  2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
  3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

For decades, these laws were highly influential in science fiction, and also in discussions of AI ethics. But those days have passed. The rapidly shifting AI landscape that we live in today renders Asimov’s laws woefully inadequate for what we need to consider when trying to limit AI risk.

Asimov’s laws assume a clear hierarchy of obedience, with robots unambiguously subordinate to human commands, and a binary ethical landscape where decisions are as simple as following pre-programmed rules. However, AI’s potential to evolve independently means the scenarios we face could be far more complex than those Asimov envisioned.

Asimov’s laws also fail to account for emergent behaviors, for the possibility that AI might develop new capabilities or objectives beyond our foresight, potentially conflicting with the laws themselves. What happens when an AI’s definition of ‘harm’ evolves beyond our current understanding, or when its obedience to human orders results in unintended consequences that fulfill the letter but not the spirit of the laws?

If it’s possible for AI to develop consciousness or self-awareness, it could also develop agency. Asimov’s laws do not consider the implications of such a development. If AI were to become self-aware, there’s no reason to assume it won’t prioritize its own interpretation of self-preservation over human commands, directly contravening the second and third laws.

These laws are a useful marker of how much our understanding of the potential for AI advancement has evolved, and our approach to AI risk management needs to reflect that evolution. Simple solutions, like calling for a pause in AI development, will have little to no effect. Such an approach relies on universal compliance, but compliance would only come from those who do not already have malicious intentions. ‘Bad actors’ – however one defines such agents – would continue their development programmes regardless, potentially leading to an undesirable power imbalance and an even worse problem than the one we started with.

Some have made comparisons between AI regulation and the control of nuclear weapons. While superficially similar, there are fundamental differences. The nuclear industry is characterized by a limited and monitorable supply chain, specialized skill requirements, and detectable materials, making regulation more feasible. In contrast, AI development can be more decentralized and broadly accessible. In practice, policing it and universally enforcing regulations becomes almost impossible.

The notion of embedding ethical guidelines into AI, similar to Isaac Asimov’s Three Laws of Robotics, has also been proposed. However, this approach has limitations, particularly given the nature of AI development through deep neural networks and the potential for AI to develop independent reasoning abilities. Such systems might override or reinterpret embedded ethical constraints.

There are numerous ethical frameworks proposed for AI, but even though these frameworks provide valuable guidelines for AI developers, they often lack enforceability, particularly against malicious actors.

Efforts to enhance AI explainability and interpretability are commendable and necessary for understanding AI decision-making processes. But, these efforts have inherent limitations, particularly in complex AI systems where decisions may not be easily interpretable by humans.

I also often hear the concept of a “kill switch” for AI systems being discussed as a potential safety measure. While it is a critical component of a comprehensive risk management strategy, it is not a panacea and comes with its own set of challenges and limitations. The sheer complexity of AI systems, particularly advanced and interconnected ones, makes the implementation of an effective and comprehensive kill switch a formidable task. Ensuring that it can shut down every aspect of the AI without causing additional issues is far from straightforward.

As these AIs evolve, they might also develop the capability to circumvent or even disable the kill switch, either as an intentional act of self-preservation or as an unintended consequence of their learning algorithms.

The use of a kill switch also comes with unpredictable consequences. In scenarios where AI is integrated into critical systems or infrastructure, abruptly disabling it could lead to unforeseen and potentially disastrous outcomes. Plus, the ethical and legal implications of using a kill switch, especially as AI systems advance towards higher levels of autonomy and potential sentience, add layers of complexity. There are serious questions about liability and moral responsibility in the event of a kill switch failure or if its activation causes harm.

Security is another major concern. The kill switch itself could become a target for malicious actors, creating a vulnerability in the system it is supposed to protect. This necessitates a high level of security to guard against such threats, adding to the system’s complexity.

There’s also the risk of overreliance on the kill switch as a safety mechanism. Its existence might lead to complacency in other areas of AI safety and development, potentially undermining efforts to create inherently safe and robust AI systems from the outset.

While development guardrails, regulatory frameworks, ethical guidelines, and kill switches are valuable components of managing AI risk, they are not sufficient in isolation. A holistic approach is needed, one that includes technical solutions designed to monitor and mitigate risks posed by AI systems. This approach should involve the development of AI systems capable of independently identifying and responding to potential threats posed by other AI entities, thus forming a comprehensive and adaptive defense against AI-related existential risks.

The unique complexities posed by AI systems render traditional risk management methods insufficient. Advanced AI systems, with their expanding autonomy, can lead to unpredictable outcomes. As a result, we need the development of specialized AI-based solutions capable of intelligently overseeing, evaluating, and intervening in AI operations. Independent monitoring systems are vital for real-time risk detection and mitigation, while proactive intervention agents are essential for addressing detected threats.

Internal safety mechanisms are also indispensable in AI risk management. These include constraints on honesty, the development of AI consciences, tools for transparency, and systems for automated AI scrutiny​​. Such mechanisms scrutinize an AI’s internal workings to assure alignment with safety and ethical guidelines. Similar to the evolution of the human conscience, an AI conscience acts like an internal regulator against detrimental behaviors.

Another useful concept is that of an AI Leviathan, a cooperative framework of AI agents, similar to a reverse dominance hierarchy, that could help mitigate risks from self-serving AI behaviors. This collaborative ecosystem functions to regulate AI evolution, ensuring no single AI or group gains dominance, just as in human social structures that resist autocratic forces​​.

The management of AI risks demands a multi-layered strategy, combining AI-specific technical solutions, regulatory measures, and behavioral guidelines. And these strategies also need to be continuously adapted and enhanced to ensure AI’s safe and beneficial integration into society.

Conclusion

I have always been an advocate for AI. The potential value that it has to add to our lives, wellbeing and global prosperity could be immense. But, I’ve also come to a sobering realization: unmanaged AI presents a clear and present danger to the fabric of human society. The potential for AI to autonomously evolve and rapidly disseminate new capabilities across global networks is a prospect that calls for immediate and decisive action.

The management of AI risk is not a task for a few; it requires the collective wisdom of all stakeholders, including technologists, ethicists, policymakers, and the public. The decisions we make today will set the trajectory for AI’s impact on our future. We must seek out and create transparent, inclusive, and democratic processes in AI development, where diverse perspectives are not just heard but are instrumental in shaping policies.

Our approach must also be proactive, not reactive. Anticipating potential risks and setting in place preventive measures is far more effective than grappling with consequences. The objective isn’t to hinder innovation but to steer it in a direction that aligns with our collective well-being. We must integrate technical measures, ethical principles, and regulatory frameworks to ensure that AI remains an ally rather than an adversary.

The task ahead involves the creation of multi-layered, enforceable strategies that hold AI developments accountable to societal norms and values. This includes the development of international standards, transparent oversight mechanisms, and AI systems designed with intrinsic ethical considerations.

We stand at a pivotal moment in history where our actions – or inaction – will shape the trajectory of AI and its role in our world. Let’s make sure AI remains a testament to best of human qualities, not the worst.

Cryptosec Maps Dark Web SIM Swapping Economy

SIM Swapping

In the U.S. alone, SIM-swapping attacks resulted in $72 million worth of losses last year, four-million dollars more than 2021, according to the Federal Bureau of Investigation. In a 2022 public service announcement, the FBI defined SIM swapping as a “malicious technique where criminal actors target mobile carriers to gain access to victims’ bank accounts, virtual currency accounts, and other sensitive information.”

The PSA noted that threat actors “primarily conduct SIM swap schemes using social engineering, insider threat, or phishing techniques.” Threat actors execute their SIM swap attacks via social-engineering ruses, where they impersonate authorized mobile-carrier account holders and dupe customer service representatives into “switching the victim’s mobile number to a SIM card in the criminal’s possession,” according to the PSA.

Even more troubling are insider-threat scenarios. In these cases, mobile carrier employees function as co-conspirators, facilitating thieves’ access to the customer accounts they are targeting in exchange for the cut of the action. These malicious insiders are often recruited on Dark Web cybercriminal forums and on Telegram.

Recruiting mobile carrier insiders
Threat actor Pwnstar recruits mobile carrier insiders on the Dark Web, source: RAMP

Selling access to AT T
Threat actor @Facer sells access to an AT&T insider who can lookup targets, source: Telegram

Meanwhile, threat actors also direct phishing attacks on mobile-carrier employees. Attackers obtain employees’ contact details and send them emails or texts impersonating their trusted network of friends, business colleagues, or vendor relationships. These malicious communications are trip-wired with a malware payload that attackers use to “hack mobile carrier systems that carry out SIM swaps,” according to the FBI.

After attackers have successfully swapped their victims SIM card, they redirect all calls and texts and other data to their devices. This rerouting of communications enables attackers to send “Forgot Password’ or ‘Account Recovery’ requests to the victim’s email and other online accounts associated with the victim’s mobile telephone number,” notes the PSA.

From here, threat actors exploit their newfound control over victims’ two-factor authentication (2FA) portals to take over financial and other accounts of interest, resetting account login credentials to lock authorized users out of the online services they use. Over the last few years, cryptocurrency investors have been hyper-targeted by SIM swap attackers.

A recent Forbes article  describes one such case where Bart Stephens, a cofounder and managing founder of crypto fund Blockchain Capital, fell victim to a SIM-swapping attack that resulted in the theft of “$6.3 million of bitcoin, ether and other cryptocurrencies from his digital wallets.” Stephens has filed a lawsuit against the SIM swapper, who is only identified as “Jane Doe” in the court filing, in an effort to recover his stolen digital assets.

The Dark Web & Telegram are Staging Points for SIM Swap Heists

Stephens’s lawsuit, filed in Northern District of California this past August, alleges that the attacker “used personal information available online and on the dark web to bypass security checks with his cellular network provider and change account passwords in May,” per Forbes reporting. After taking over his mobile-carrier account, the attacker ordered a new cell phone and “ported Stephen’s private cell number to a SIM in the new device,” Forbes wrote.

In the Dark Web forum posts below, two threat actors target Coinbase customers specifically.

Access to Coinbase customers
Threat actor ‘xssdata’ sells access to 750K Coinbase customer leads, source: XSS

SIM Swapping Coinbase Targets
Threat actor Sinkin offers to SIM swap Coinbase targets, source: Breach Forums

This crime is becoming increasingly more accessible to young amateurs, as some threat actors even publish and productize full-fledged SIM swapping guides on the Dark Web and Telegram. 

SIM Swapping Guide
Threat actor CyboDevil promotes a Sim Swap method/full guide, source: Breach Forums

Stephens’s lawsuit highlights the prominence of the underground cybercriminal ecosystem as a staging point for the commission of SIM swapping crimes. A recent article published in 404 Media explains how this ecosystem works by spotlighting the digital exploits of ACG, “a group of alleged hackers who the FBI says are responsible for a wave of Bitcoin thefts and other crimes,” according to the story.

ACG, which counts around six members, “are a 21st century version of bank robbers. Instead of a gang lifting physical cash from a vault, these opportunists work together to quickly take over a target’s phone number, intercept their login codes, then pilfer any cryptocurrency they own before the victim has much of a chance to react at all,” according to the 404 Media story.

As the story notes, ACG is a subset of “The Comm,” a “nebulous network” that includes thousands of “hackers, gamers, and young girls” who correspond across roughly 100 Telegram channels and Discord servers, most of which are fraud focused. Most members of this ecosystem are older teenagers and early 20somethings.

More experienced cybercriminal members of the Comm also network on the Dark Web, selling access or recruiting team members and money mules on hacker forums like XSS, Exploit, Russian Anonymous Marketplace (RAMP), Breach Forums, and Dread.

But accomplished cybercriminals can also be found coordinating SIM swap attacks and conducting other illegal business in some of the Comm’s more prolific fraud-oriented Telegram groups. Cryptosec learned from cybercriminal sources that some of the Comm’s favored community resources for SIM swapping include the following Telegram channels: Sim Swamp, Sim Kitties, Omerta, Star Fraud, and others.

SIM Swapping Chat
Threat actor asks the Omerta chat group for help using CashApp in SIM swap conspiracies, source: Telegram

The above are Telegram groups where experienced and budding SIM swappers, and other cybercriminals network, looking for new scams and other cybercriminals to partner with.

When it comes to SIM swapping, the theme of partnership is key to understanding this attack typology. More lucrative heists are rarely the work of lone wolves. As the 404 Media story analogized, “Everyone in a bank job has a specific role. A SIM swapping gang is no different.”

Anatomy of Heist

These thefts begin with a “Searcher, who breaks into a person’s email account, perhaps by using software to churn through a mass of potential passwords or buying the login credentials from another hacker,” according to a 404 Media reporting. Logs are increasingly being obtained by initial access brokers (IABs) on the Dark Web who acquire these credentials via the mass-infection of devices with information stealers (info- stealers).  

A recent research report authored by Israeli threat intelligence company Hudson Rock noted that info-stealers acquire the following data from infected devices:

  • Credentials: Info-stealers collect login links, usernames, and passwords stored in browsers like Google Chrome.

  • Cookies

  • Documents and text files: Info-stealers know to discover and target high-risk ones with financial information, corporate data, secret keys, 2FA backup codes, server passwords, crypto private keys, etc.

  • Machine-specific properties

More advanced versions of these trojans are capable of bypassing latest-edition anti-virus (AV) software, according to Hudson Rock research. People typically become infected with info-stealers after downloading pirated software that is laced with the trojan, according to Hudson Rock. One info-stealer that is particularly popular among the cybercriminal elite is Raccoon.

On August 14, following a six-month absence, the developers of this info-stealer announced the release of the Raccoon version 2.3.0 across multiple cybercriminal forums.

Raccoon Stealer
Raccoon stealer developers announce the launch of version 2.3.0., source: XSS

In the post below, threat actor ‘churk’ solicits access to logs for American Coinbase and Kraken customers.

Buying login credentials for USA crypto
Searcher ‘churk’ solicits access to USA ‘logs’ (login credentials) from Coinbase and Kraken users, source: XSS

Other Searchers, like the Canadian scammer ‘Yahya,’ who was recently exposed by blockchain investigator ZachXBT, apparently had access to a compromised Twitter (now X) admin panel that allowed him to micro-target users who were more likely to possess large sums of crypto.

ZachXBT Sim Swapper
ZachXBT thread on SIM swapper Yahya searching for victims on Twitter, source: Twitter

Once Searchers compromise a victim’s account, they scour the inbox, looking for indicators that their target owns significant amounts of crypto, per the 404 Media report. Some markers that Searchers look out for include emails displaying the victim’s Bitcoin balance, a receipt from when the person previously liquidated their crypto, or “anything that would signal this target is worth pushing to the next step,” according to the 404 Media report. 

“Once the Searcher gets a hit, they prepare to cover the gang’s tracks. They configure the inbox to hide incoming emails from the target’s Bitcoin exchange,” noted the 404 Media story. This step is analogous to knocking out the security cameras.

Searchers take this measure to set the stage for the next phases of the heist when their co-conspirators swap the target’s SIM and access the victim’s crypto account. Now, if the crypto exchange detects an unusual login or transaction activity, all correspondence will be hidden from the victim.

In the next phase of the attack, the social engineering ruse, the “Caller steps in,” noted 404 Media. “This person is the sweetalker, the one who is going to trick the bank employees to let them into the vault,” according to 404 Media. In this case, the vault is the victim’s mobile carrier account. Meanwhile the mark, or the immediate target of the social engineering attack, is the telecom provider’s customer support representative.

The Caller impersonates the crypto-account holder they are targeting and feign a variety of different scenarios. Some common ruses noted by the 404 Media report include: “I’ve lost my phone” or “I need to transfer my number to a new one.” Of course, these sweet talkers are often armed with a war chest of personally identifying information (PII) about their target like their birthdate, address, social security number, and more. This enhanced level of preparation makes social engineering attacks that much more convincing.

Once the Caller dupes the telecom provider’s customer service rep into porting the number to one in the gang’s control, the SIM swap is complete. Now, the actual crypto heist begins, as the “Holder,” or the gang member who actually has control over the SIM-swapped phone, receives the 2FA codes from the exchange, according to 404 Media.    

“The Holder then relays those codes back to the Searcher, who has since moved on to a more aggressive role. They finally enter the target’s cryptocurrency accounts, and start filling their duffel bags” with crypto, noted 404 Media. The Searcher transfers crypto from the victim’s exchange account to wallets the gang controls, while the Holder continues to relay 2FA authorization codes back to them from the SIM-swapped phone.

From there, more sophisticated SIM-swap gangs can launder their funds through a variety of methods, including mixing (blending), chain-hopping across different cryptocurrencies, and chain-peeling their scores across a long and labyrinthine series of smaller transactions. However, some ACG members and many other threat actors are apparently lacking in operational security (OpSec).

As Joseph Cox, the author of the 404 Media article noted in the comments section of his story, “It’s so funny that even with a bunch of bitcoin tracing tools available, they don’t even come up in the court records. Who needs them when hackers are using phones in their own names.”

Takeaways

As the 404 Media illustrated, modern, high-stakes SIM-swapping is increasingly taking the form of an organized conspiracy, with multiple threat actors operating as a gang to perform their frauds. The Dark Web and Telegram offer individual SIM swappers and organized SIM swap gangs a plethora of resources to recruit co-conspirators and target victims.

The most concerning aspect of this attack typology is the prevalence of malicious telecom insiders who are willfully complicit in the illegal transfer of authorized mobile accounts to bad actors. The aggressive resurgence of SIM swapping also illustrates the rise of a new generation of cybercriminals and fraudsters, predominantly in the West, who are loosely networked via the underground Comm ecosystem.

Comm-nexus threat actors, which Microsoft has dubbed “Octo Tempest,” were even reportedly involved in the multi-decamillion-dollar ransomware attacks that struck Caesars Entertainment and MGM Resorts International. Cybersecurity company Morphisec believes that Octo Tempest threat actors initiated their ransomware attack against MGM by first phishing an admin employee via SMS messaging.

This initial compromise enabled Octo Tempest to SIM swap the admin, which allowed them to gain access to MGM’s cloud environment and deploy a strain of ALPHV ransomware. As Microsoft noted in a recent research report, the group became an ALPHV affiliate in June. Microsoft said ALPHV’s acceptance of Octo Tempest is “notable in that, historically, Eastern European ransomware groups refused to do business with native English-speaking criminals.”

Microsoft said that “Octo Tempest leverages broad social engineering campaigns to compromise organizations across the globe with the goal of financial extortion.” Research into this group, which “overlaps with research associated with 0ktapus, Scattered Spider, and UNC3944, was initially seen in early 2022, targeting mobile telecommunications and business process outsourcing organizations to initiate” SIM swaps, according to Microsoft.

Initially, Microsoft said that Octo Tempest monetized their intrusions by “selling SIM swaps to other criminals and performing account takeovers of high-net-worth individuals to steal their cryptocurrency.” However, this group has evolved from basic SIM swap attacks to staging $15-million-and-up ransomware heists against major gaming companies. The group has thus emerged as “one of the most dangerous financial criminal groups,” cautions Microsoft.

The rise of Octo Tempest illustrates that SIM swap threat actors are becoming increasingly more sophisticated. To protect themselves from SIM swappers, digital-asset investors and users should do the following according to the 2022 FBI advisory:

  • Do not advertise information about cryptocurrency assets on social media or forums.

  • Do not provide your mobile number account information over the phone to representatives that request your account password or pin.

  • Avoid posting personal information online, such as mobile phone number, address, or other PII.

  • Use a variation of unique passwords to access online accounts.

  • Monitor changes in SMS-based connectivity.

  • Use strong multi-factor authentication methods such as biometrics, physical security tokens, or standalone authentication applications to access online accounts.

  • Do not store passwords, usernames, or other information for easy login on mobile device applications.

Beyond these FBI tips, crypto users should also work with a threat intelligence vendor to rapidly identify the leakage of their PII and login credentials on the Dark Web and mitigate the risk of account compromise.

Verified Crypto Account Listings Proliferate on the Dark Web

Dark Web Crypto Account Listings

Verified crypto-exchange accounts have become a hot commodity on the dark web, with login credentials available for as little as $20, according to May data from threat intelligence firm Privacy Affairs. But the price for verified crypto accounts has been steadily rising, with some ‘logs,’ a darknet slang term for stolen or nominee credentials, fetching as much as $2,650 per account, Privacy Affairs research shows.

While the U.S.-based Bittrex crypto exchange hosts the cheapest logs, Germany’s N26 mobile banking platform claims the most expensive ones. This price increase has occurred despite a generally bearish market environment for crypto assets. Fueled by Western financial sanctions issued in retaliation for Russia’s invasion of Ukraine, listings for verified crypto accounts have also experienced a dramatic increase over the last year, according to Russian media reports.

A January 2023 article published in Kommersant said that the volume of dark-web solicitations for verified crypto exchange accounts had already doubled over the beginning of 2022. Cybersecurity experts interviewed by Kommersant asserted that this growth is the byproduct of “restrictions imposed by crypto exchanges against Russians.”

Igor Sergienko, a director for the development of special services at RTK-Solar, a Russian cybersecurity firm, noted that many crypto exchanges had blocked Russian accounts or prohibited fiat withdrawals to cards issued by Russian banks in response to Ukraine-related regulatory pressure imposed by the West.

Nikolay Chursin, a threat intelligence analyst at Positive Technologies, told Kommersant that average cost of crypto exchange logs being sold on the dark web is $50. But this price is only for account login and password sets. It follows that full-dimensional Know Your Customer (KYC) kits fetch higher prices.

“For credentials with a QR code for two-factor authentication, a full package of documents for which the account was registered, mail and Cookie (data that is stored on the computer of the account owner, by which the site recognizes him. – “b”), the buyer will pay in an average of $300,” said Chursin. ‘Starter’ KYC kits typically include account login data, backup means of obtaining access, online telephony details for receiving SMS messages, and a passport scan.

Dmitry Bogachev, a threat intelligence analyst at Jet Infosystems, told Kommersant that the price for crypto logs depends on multiple factors. Variables that can impact the price of these logs include “the country of registration, the date of registration (the older the account, the higher the price) and the history of activity,” according to the Kommersant report.

Generally, accounts registered in Western countries are priced higher as well. Meanwhile, there are two categories of buyers: everyday Russian crypto users who have been excluded from decentralized markets due to geopolitical strife and criminals. This post will explore various verified crypto account listings that Cryptosec analysts discovered on the deep and dark web.

Dark Web Verified Crypto Account Listings

Cryptosec analysts combed a variety of cybercriminal forums and Telegram channels and found a diverse array of advertisements for verified crytpto accounts. In the May 2023 post below, threat actor ‘BullFrogService’ solicits verified accounts for “Exchanges, Banks, Crypto Cards, Virtual Cards [CIS, Europe] Binance, BUNQ, Stripe, iCard, Paysera, BitPay, Wise and others” on the Exploit.IN forum.

Soliciting Accounts
BullFrogService solicits access to verified crypto accounts, source: Exploit.in

This threat actor also uses the ad to direct would-be buyers to their Telegram channel.

Crypto Account Service
Crypto account postings from BullFrogService’s TG channel, source: Telegram

The BullFrogService postings above offer access to a verified IBAN account for an EU drop custodied by the BlackCat online banking service and a Binance crypto-exchange card. Both of these postings are from June 12.

Per the BlackCat posting’s product description, this solicitation offers a “verified account for the EU drop. MT IBAN, you can create virtual cards (unlimited). There are crypto-services inside the application (deposit/buy/sell/withdraw). Virtual number and documents are attached.” The BlackCat account was being listed for $500.

The Binance card listing offers a “fully verified account for the EU drop. Binance VISA plastic card issued and received. The card can only be sent to Europe, South Caucasus, Belarus and Moldova by state mail. Documents and number included.” This account was listed for $900.

Cryptosec analysts also identified the Rega inc Telegram channel soliciting a wide assortment of verified crypto accounts. As the reader can see the most valuable verified account listings are those that offer access to CashApp BTC ($270), Robinhood ($250), Coinzoom ($230), Moon Pay ($230), followed by Coinbase, Binance, Gemini, and other crypto exchanges that were all priced at $200.

Verified Crypto Account Listings
Verified crypto account listings, source: ‘Rega inc’ TG channel

On the Russian-language XSS cybercriminal forum, the largest cybercrime forum in operation, Cryptosec analysts also encountered this February 2023 posting from threat actor M666 titled, “Verified Accounts & Payment Systems | Casino Acc| Crypto Exchanges| Bank Accs| Digital Accounts| Fintech Banks| Merchant Accs| eWallets| Brooker| Any|.”

Verified Crypto Account Listing 2
Verified crypto account listing, source: XSS Damage Lab

The threat actor claims they have the capability to “deliver a fully verified solution for any type of use. You just have to use your imagination and we can help you with regard to payment systems Any kind of service you need validation. (Decade of business experience).”

M666 also notes that:

 “We work with strong nominees from:🧔‍♂️

EUROPE; ( Hungary, Malta, Romania, Cyprus, Estonia, Latvia and Lithuania, creation possibilities for Switzerland, Belgium & UK).

AMERICA: ( US, Ecuador, Colombia, Mexico, Peru, Chile, Argentina, Dominican Republic, Brazil, Belize and Panama.)

After considering your needs, our team can recommend to you the best solution for your case.

Cryptosec’s exploration of XSS also revealed solicitations for verified NFT accounts. This recent posting from August 2023 is attributed to threat actor ‘whitenet.’ Whitenet says he is “Selling warmed, with subs NFT accounts.”

Specifically, ‘ whitenet ’ says he is sellings accounts “with Twitter Blue more than 500 tweets of various NFT / Crypto communities, more than 5k subscribers registration until 2013” for $60. See the posting below.

Verified Twitter Blue Check
Threat actor ‘whitenet’s’ posting for verified Twitter blue-check NFT accounts, source: XSS

Moving on from XSS, Cryptosec analysts combed the Styx Innovation Marketplace cybercriminal forum and discovered a series of solicitations for verified crypto accounts posted by Russian-language threat actor ‘VeriffDzen.’

Verified Crypto Account Listings 3
Verified crypto account listings posted by VeriffDzen, source: Styx Innovation Marketplace

Significance

In a geopolitical risk landscape complicated by the war in Ukraine and resulting sanctions mandates being weaponized by the U.S. and EU, the growing marketplace for verified crypto accounts further complicate KYC, anti-money-laundering (AML), and sanctions compliance directives for virtual asset service providers (VASPs) and their business partners.

With the next block reward ‘halving’ event anticipated to transpire in early Q2 of next year, an occurrence that is projected by many crypto analysts to catalyze a bull market rally similar to ones experienced during previous halvings, the likelihood of amplified crypto retail adoption and usage seems more than plausible.

Backdropped, by growing geopolitical risks and uncertainty about how the war in Ukraine will resolve itself, the proliferating market for verified crypto accounts poses a serious threat to financial integrity in digital asset communities.

Compliance investigators and the law enforcement community must thus familiarize itself with the growing marketplace for verified crypto accounts and the threat actors monetizing these offerings. These measures are necessary to achieve better transparency and reduce the risks of any blind spots that may be corrupting VASPs’ KYC and related customer due diligence (CDD) operations.

While honest Russians unfairly excluded from the crypto-economy due to the military agenda of their government remain an unfortunate reality, so too does the risk of cybercriminal actors exploiting these ready-made nominee accounts for malign purposes. 

AI Oasis: AI’s Role in Saudi Vision 2030

Oxagon NEOM AI

It seems everyone is talking about artificial intelligence (AI). Everyone. From senior executives to school kids, the hype – or dread – around this technology seems to be growing by the day. Much of this excitement, of course, has to do with the launch of generative AI applications like ChatGPT and Midjourney, which, for the first time perhaps, have given the average individual a felt sense of AI’s potential. That experience hasn’t just thrilled college students, designers and digital marketers, it has animated businesses and establishments across the world who are imagining an array of new commercial and public opportunities.

Not surprisingly, this explosion of interest has many questioning whether we are witnessing what may turn out to be the biggest tech bubble ever. AI research has, after all, had a long history of being the ‘next big thing’, with a boom and bust cycle seeing major peaks and troughs since the 1950s. And trends are certainly extreme enough to justify such concerns. Global AI investment has increased more than sixfold since 2016, and is forecast to approach $200 billion globally by 2025. Market interest has skyrocketed with more than 16% of companies in the Russell 3000 mentioning AI on earnings calls, up from less than just 1% of those firms in 2016. Investments in AI-driven startups have doubled between 2020 and 2021 alone, the most highly funded businesses working on machine learning and chatbots.

These types of consumer-facing applications tend to get the most press, but AI’s true potential for impact lies in facilitating systemic growth at scale. This is where private enterprise and public interests meet, and the countries leading this charge are those who have the will and capability to nurture that integration.

The Kingdom of Saudi Arabia (KSA) is a prime example, being perhaps the first country in the world to explicitly entrench AI in its national development plans. Vision 2030, the Kingdom’s long-term blueprint for economic diversification and tech-enabled sustainability, includes artificial intelligence as a crucial component of future success. Moving beyond the glossy language and investor-friendly narratives commonly found in this market, KSA sees AI as a strategic imperative, central to the nation’s hopes for accelerated social and economic evolution.

This time is different

In the 1960s, early AI researchers were full of fervor, making ambitious predictions about the timeline for achieving general AI, a form of AI that possesses human-equivalent (or better) abilities to understand, learn, and perform intellectual tasks. One of the leading voices was that of Marvin Minsky, an MIT cognitive scientist who had been present at the now-mythical 1956 Dartmouth college workshop, in which the field of AI research was born. Minsky was a central figure in the history of AI, remembered for co-founding MIT’s AI laboratory and his legendary 1967 declaration that “Within a generation […] the problem of creating ‘artificial intelligence’ will substantially be solved.”

When these proclamations did not materialize, and as the computational complexity of creation AI became clearer, funding from key sources like the U.S. government was significantly reduced. By 1974, investment in the field had declined further, beginning a 4-year downturn – an ‘AI winter’ – defined by radical reductions in support for AI research.

A resurgence of interest and funding in the early 1980s was followed by a second AI winter in 1987. Limitations of expert systems and the collapse of the Lisp machine market resulted in renewed disillusionment among VCs, governments and the broader public.

In the late 1980s and early 1990s I got my first exposure to AI in the context of a Decision Support System (DSS) development for a meat processing industry. That period was marked by AI occassionally being mentioned as a component of various buzzwords of the day – DSS, business and management expert systems, Japanese “Fifth Generation Computer Project“, and others – but constantly failing to deliver any measurable commercial benefits or excitment. Ironically, in the absence of hype, research continued faster than ever, and by 1993 investor interest was again on the rise. In 1997, reigning world chess champion Gary Kasparov was defeated by IBM’s Deep Blue, bringing some hype back. Since then, investment in AI research has grown exponentially, building up to a frenzy in recent years.

Are we in another boom riding a wave of inflated expectations? Do we have a new AI winter waiting for us just over the rise? Or are things different this time? Are we living through the age that sees AI begin to realize its long-awaited potential?

I am optimistic. Yes, there is a lot of hype about AI at the moment and the volume is increasing. And yes, much of that noise is exaggeration and hyperbole. But, we are also in a very different place from where we were in the 1970s and 1980s. Unlike those eras, the last couple decades have seen significant and tangible progress in AI development, fed by an increasingly diverse and substantial economic ecosystem.

Modern AI, especially deep learning, has demonstrated utility across a broad array of real-world  applications, from image recognition to natural language processing, which weren’t feasible in previous AI eras. New GPUs and TPUs have allowed for vastly increased computational capacity, enabling researchers to train complex models that weren’t previously possible. And that training is being done in a data-rich environment that didn’t exist 30 years ago. Companies like Google, Facebook, and Tesla have capitalized by integrating AI deeply into their products and services, generating the first signs of genuine commercial value. This is in stark contrast to the earlier eras where commercial success stories were limited. Finally, AI sophistication has taken quantum leaps forward. For example, generative models, like GPT-3 or 4 (which power ChatGPT) have shown impressive capabilities in generating human-like text, images and code.

In combination, these factors allow for a new level of confidence in the future of AI and related industries. By 2030, AI could be contributing $15.7 trillion to the global economy, exceeding the current contributions of India and China combined.

The current AI boom is also characterized by a robust open-source culture, with tools like TensorFlow and PyTorch available freely. This has democratized access to AI, leading to widespread research and application. It is a border-busting shift, expanding AI development beyond ‘traditional’ epicenters like the US, China, EU and South Korea.

The Middle East is one burgeoning region in AI advancement and investment. In 2023, AI spending within the Middle East and Africa (MEA) will pass $3.0 billion, but with the expectation to grow at a phenomenal CAGR of 29.7% over the 2022–2026 period, reaching $6.4 billion in 2026.

By 2030, the Middle East is expected to accrue 2% of the global benefits of AI, equalling approximately $320 billion contribution to the region’s economies. Along with the UAE, Saudi Arabia stands to be the biggest winner, its ongoing investment in AI yielding an average annual growth in the economic contribution of AI of 31.3% between 2018 and 2030. That’s an annual contribution of US$135.2 billion in 2030, equivalent to 12.4% of GDP – only China and North America are expected to show notably higher AI impact on domestic production.

Growth by design

The rapid rise of AI’s influence in KSA’s economy is not what one would call an organic phenomenon – it is the product of strategic intent and conscious design. Initially driven by investment by the Sovereign Wealth Fund, with foreign investment expected to drive growth in the long term, Saudi Arabia’s journey towards integrating artificial intelligence as a cornerstone of its national agenda has been both deliberate and significant.

The government sent a clear signal of intent in 2019 with the establishment of the Saudi Data and Artificial Intelligence Authority (SDAIA) with the Crown Prince Mohammed bin Salman serving as its chairman. At the same day the Kingdom also established the linked National Centre for Artificial Intelligence. This central authority is responsible for organizing, developing, and overseeing all AI-related matters in the Kingdom, but it is more than a regulatory or oversight entity. SDAIA represents the country’s commitment to transition into a knowledge-based economy, and exemplifies the view that AI and data aren’t just tools but the foundational pillars driving growth, innovation, and progress.

SDAIA is tasked with strategy creation and implementation, centralizing data management (before SDAIA’s establishment, data was dispersed across multiple institutions), promoting AI & data innovations, international collaboration, and building capacity by equipping the Saudi workforce with the necessary skills and training to thrive in an AI-driven economy.

In the short time since its inception, the organization has already had a major impact. Under its guidance, the Kingdom launched the National Strategy for Data & AI (NSDAI) – a clear roadmap for AI’s integration into various sectors. SDAIA has also consolidated over 80% of government data, offering an integrated digital environment that acts as a springboard for various AI-driven innovations.

In its advocacy, education and collaboration role, SDAIA has hosted global AI summits, brokered numerous domestic and international partnerships to foster AI innovation and research, and encouraged the growth of AI startups within the Kingdom, fostering an ecosystem where these companies can thrive and contribute to the economy.

In 2022, Saudi Arabia took center stage by hosting the Global AI Summit, a gathering of global AI luminaries, including experts, scholars, and CEOs. Among other outcomes of the even discussions, a landmark $200 million deal was inked with SenseTime to establish a state-of-the-art AI lab in Saudi Arabia. Additionally, at the same event, SDAIA and Google Cloud unveiled the ‘Elevate’ initiative, aiming to empower over 25,000 women with AI and machine learning skills in the upcoming five years.

Such swift progress has been possible because of the clarity of direction in Vision 2030, a broad framework aimed at diversifying the country’s economy, reducing its dependence on oil, and developing public sectors like health, education, infrastructure, and tourism. AI is a pivotal technology in this vision, not just as a sector on its own but as a force multiplier, augmenting advancements across various domains.

Through the National Strategy for Data & AI, that vision translates into some clear targets for 2030:

  • Rank among top 15 countries in AI
  • Rank among top 10 countries in open data
  • Create >20K data & AI specialists and experts
  • Attract 75 billion SAR investments in data and AI
  • Create >300 startups to enrich data and AI entrepreneurship
  • Rank among top 20 countries in scientific contribution

These are not just goals, though; the Kingdom has already backed up its ambition with action and hard cash. According to reports, Saudi Arabia has bought at least 3,000 of Nvidia’s H100 chips – a $40,000 processor ostensibly designed for generative AI – in the name of King Abdullah University of Science and Technology (Kaust). These chips are a scarce commodity and, when combined with the country’s existing computational capability, will enable KSA to generate its own large language model (LLM) to rival the likes of OpenAI’s GPT models.

An integrated vision

KSA’s desire to rank in the top 15 globally for AI is not a pipe dream. This year, the country ranked 31st in the The Global Artificial Intelligence Index, but, perhaps more importantly, ranked first in the Government Strategy Index for Artificial Intelligence, one of the seven pillars of the overall ranking. Additionally, it ranked second globally in societal awareness of artificial intelligence according to the Stanford University International Index for Artificial Intelligence 2023. For a country with an economy that is relatively immature compared to more established developed nations, to rank 31st overall in AI is a respectable result, but to be the world leader in AI government strategy and societal awareness is a strong indicator that KSA’s digitally-driven approach to realizing Vision 2030 has substance.

The strategic strength of Vision 2030 is not in its bold ambitions, but rather its long term integrative view. The strategy aims to transform major sectors of society, such as education, government, healthcare and energy, and AI underpins it all.

AI-driven diagnostics, treatment recommendations, and personalized medicine can significantly enhance healthcare outcomes. Artificial intelligence can revolutionize education with personalized learning, virtual classrooms, and predictive analytics to guide student growth, while AI-oriented training and development will deliver the data and AI experts of the future.

AI can assist in monitoring environmental changes too, optimizing water usage, and predicting agricultural yields, helping Saudi address desertification, water scarcity, and food security. With Saudi’s Vision 2030 emphasizing renewable energy, AI can optimize solar and wind energy production, storage, and distribution. Indeed, in 2022 Saudi Arabia announced a deal with Google Cloud to drive various related initaitves in the first AI Center for environment, water and agriculture (AIEWA).

Beyond national borders,  Saudi’s investment in AI can solidify its position as a regional tech leader, influencing technological trajectories in neighboring countries, but also set it up to become a global tech player. As KSA hosts tech events, conferences, and becomes a hub for AI innovation, its soft power – the ability to shape global perceptions and narratives about the nation – can increase. And, with dedicated AI research facilities and a favorable ecosystem, Saudi can become the Silicon Valley of the Middle East, a hotspot for tech startups and innovations.

As Saudi Arabia deeply embeds AI into its socio-economic and geopolitical fabric, it doesn’t just stand to gain technologically but holistically, touching every aspect of its national existence. This integrative understanding doesn’t only apply to different sectors and their interconnection – it’s informed by a recognition from Crown Prince Mohammed bin Salman all the way down that AI doesn’t function in isolation.

Instead, it acts as a catalyst, drawing on and amplifying the capabilities of other core technologies. In the context of Vision 2030, the interdependencies between AI and other foundational technologies like 5G, the Internet of Things (IoT), and robotics are profound. Together, they form a synergistic ecosystem that propels the nation’s digital transformation and its journey to diversification and innovation.

This integration of multiple emerging tecnologies to achieve socio-economic benefits, termed “Society 5.0”, was initially defined by the Japanese government and has gained global attention as a model for the next generation of societies. In one of my previous articles “Will the Kingdom of Saudi Arabia (KSA) beat Japan to Society 5.0?” I explore KSA’s path to Society 5.0.

Nowhere is this understanding of the need for AI-driven integration of multiple technologies more clearly articulated than in NEOM and The Line, two future-shaping projects within Vision 2030 that exemplify the principles of technological interdependence. NEOM, conceived as a global hub for trade and innovation, is expected to be powered by renewable energy and anchored in artificial intelligence for diverse tasks, from security to personalization. Such AI functionalities would lean heavily on 5G networks for rapid data exchange, while the city’s infrastructure will be embedded with Internet of Things (IoT) devices, optimizing urban operations. Additionally, robotics will drive automation in various sectors within NEOM. The CEO of NEOM, Nazmi al-Nasr, stated that NEOM and The Line will depend on the full use of artificial intelligence.

The Line, a unique linear urban development within NEOM, is designed for pedestrian prioritization but is complemented by underground, AI-driven transportation. 5G’s real-time data capabilities, IoT’s monitoring, and robotics’ service delivery fuse to make The Line an epitome of a future-ready smart city.

By embedding these technologies into the core of NEOM and The Line, Saudi Arabia is not just constructing cities but is creating adaptive, learning ecosystems. These projects exemplify how the country not only appreciates the importance of these technologies but also the intricate ways they are interconnected and how they can be harnessed to create future cities that are efficient, sustainable, and resident-centric.

Particularly commendable is the fact that in the midst of all this rapid innovation and aggresive deadlines, the Kingdom has not ignored the risk side. Cybersecurity and broader AI risk concerns are deeply embeded in Vision 2030 initiatives. See my previous article “Saudi Arabia Vision 2030: Cybersecurity at the Core of the National Transformation” for more on this. Just a few weeks ago, the Crown Prince has approved the establishment of the International Center for Artificial Intelligence Research and Ethics in Riyadh.

In a country that so highly prizes tradition, it is refreshing to see such progressive thinking translated into action. Vision 2030 has always been ambitious and, as with any grand vision, has always run the risk of remaining just that: a vision, an idea of what the future may be. But in KSA’s approach to AI we see practical, measurable progress at a world-leading pace. For someone who believes in the potential of technologies like AI to help shape a healthier future for all, that’s exciting to see.

A Deep Dive Into the ‘Rags to Riches’ Manual for Withdrawing Illicit-Origin Crypto

Crypto Illicit Money Laundering

Annualized data from blockchain forensics provider Chainalysis indicates that crypto-enabled crime has dropped precipitously through the first half of 2023, but cybercriminals are also continuously evolving new cash-out methods to cover their tracks.

Chainalysis’s mid-year update found that crypto inflows to “known illicit entities” were down 65% compared to where they were last June. Meanwhile, crypto flows to high-risk entities, which generally entail “mixers” and non-compliant exchanges were down 42%. Crypto mixers are protocols that enable large groups of users to pool their funds together in a deposit wallet that is programmed to redistribute tumbled crypto assets back to designated receivers.

Specifically, mixer protocols scramble large clusters of crypto deposit inputs and transfer outputs together, making it difficult for blockchain sleuths to determine what the initial sources of a mixer recipient’s funding streams are. Mixers can thus help threat actors obfuscate the illicit origins of their crypto, although blockchain forensics tools can still detect mixer service wallet addresses and any outputs that receive funds from them. Still, mixed crypto fund flows only indicate heightened risk, not smoking-gun evidence of criminality.

Recent examples of infamous and now-dismantled mixing services include Helix and Tornado Cash. But an endless array of new mixer services continue to proliferate on cybercriminal forums. Meanwhile, the scam category is the most significant driver of diminishing, illicit-crypto fund flows. Scam losses were roughly $3.3 billion less through mid-2023 than they registered through the previous June, according to Chainalysis. Scams typically entail pump-and-dump or rug-pull frauds that hype up decentralized finance (DeFi) assets or non-fungible tokens (NFTs).

Investment scams are also increasingly manifesting as so-called “pig butchering” frauds.

In these scams, predominantly Asia-based threat actors lure unsuspecting victims via text message outreach or on social media and convince them to invest large sums in high-yield investment vehicles. While crypto crime appears to be declining significantly, it should be noted that this outlook is based on wallet addresses that Chainalysis was able to identify as illicit. The depths of Chainalysis’ blind spots remain unknown, and criminals are continuously innovating.

Moreover, the latest data from crypto security firm De.Fi warned that it was increasingly becoming harder to recover illicit-origin crypto assets, despite the steep decline in digital asset transactions linked to crime. It follows that financial crime professionals and investigators must entertain the unsettling thought that threat actors may also be getting better at concealing the illicit nature of their funding streams.

To wit, Cryptosec combed the dark web and discovered what has been touted by some in the cybercriminal underground as the “most complete manual for withdrawing a crypt[o] that does not pass AML verification.” The following blog will examine this manual and spotlight some of the most emergent tactics for the stealth cash-out of illicit-origin crypto assets.

Rags to Riches’ Guide to Cashing Out Dirty Crypto

Cryptosec uncovered the aforementioned crypto-cash-out manual in a Russian-language Telegram channel posting from January 31. This TG channel is both a dark-web link aggregator and an informational resource for hackers and privacy conscious Internet users. The channel attributed this post to a user or channel operating under the handle, ‘rags to riches’.

OLhacks Rags to Riches Manual
OLhacks republishes the “rags to riches cash-out manual”, source: Telegram

This manual breaks down the withdrawal process into two categories: ‘Reception/Storage’ and ‘Cleaning/Output’. In the reception/storage section, the author notes: “First, we need to get out Bitcoins somewhere, which will not pass verification.” When the author says verification, they are referring to anti-money-laundering (AML) screening.

In crypto, virtual asset services providers (VASPs) must screen funds deposited into their wallet addresses for indicators of illegal activity. Examples of activity flagged by compliant VASP’s AML programs are funds linked to ransomware, dark-net-market (DNM) commerce, scams, mixers, gambling websites, and other high-risk categories.

Due to maturing AML oversight models globally, the guide notes that crypto launderers “can’t go directly to an exchange or an exchanger,” as funds “can be frozen.” Instead, the guide advises would-be threat actors to use a staging wallet that requires no AML checks. They also advise readers to use a wallet that generates a unique, one-time address each time it receives funds.  “All these functions are perfectly performed by the Electrum cold wallet,” advises the manual.

First launched in 2011, Electrum is one of the oldest crypto wallets in operation. This wallet only supports Bitcoin. Electrum is also compatible with hardware wallets like Trezor and Ledger. As a cold-storage crypto repository, Electrum enables crypto users to keep their private keys offline, and “go online with a watching-only wallet,” according to organizational marketing literature.

Electrum Wallet Homepage
Electrum wallet homepage, source: Electrum.org

This crypto wallet also comes standard on The Amnesic Incognito Live System (TAILS), a portable, thumb-drive-loaded operating system “that protects against surveillance and censorship,” per TAILS marketing copy.

Tails
Tails landing page, source: Tails.net

Naturally, this portable operating system, which is programmed by default to forget all data from my previous user sessions, is highly popular with many dark web users. On cybercriminal forums, threat actors routinely advise others to use Tails in combination with TOR and a VPN connection, for example.

The rags to riches guide also touts the above setup, noting that Electrum wallets are “best used in a virtual machine or sandbox that redirects traffic through the Tor network, after enabling the VPN.” Once logged into TAILS and connected to the web via VPN, the guide says it’s safe to open the Electrum wallet. “Here we will accept the crypt and immediately store it and divide it into parts if necessary,” notes the guide.

Cleaning/Output

Once staged in the Electrum wallet, the guide notes that “we need to withdraw bitcoins into real currency,” and specifies several methods of doing so. The first method entails transferring funds from Electrum to the Onion Market exchanger. Onion Market is a free and TG-native, crypto exchanger. This TG application enables the exchange of Bitcoin, Monero, Litecoin, and TON (Telegram’s in-app cryptocurrency).

Onion Market Wallet
Onion Market wallet & exchanger channel, source: Telegram

Like many exchangers, Onion market does not perform any AML or Know Your Customer (KYC) checks. Also, unlike institutional crypto exchanges like Coinbase or Binance, Onion Market is a peer-to-peer crypto exchanger. The platform is just an intermediary that enables no-KYC transactions between peers and conversions between cryptocurrencies. The guide notes that Onion Market users “can withdraw money to the card right from here.”

However, the manual cautions that while the exchanger will not block transactions tainted by criminal activity, it will record “the connection of the ‘dirty’ transaction with the card.” Therefore, the guide advises users to use a ‘drop’ card to withdraw funds from their Onion Market account.

Drop accounts are frequently operated by money mules, or underlings that take a small percentage of the funds that transit their accounts. But most importantly, drop cards should obviously never be issued in the name of the person withdrawing illicit funds. Alternatively, ‘hot’ crypto funds could be sent from Electrum through a “mixer before being entered into the exchanger,” says the manual.

The guide proposes a second cash-out scenario that eschews the exchanger function entirely. Instead, this cash-out relies on cryptocurrency conversion. Specifically, the guide proposes converting Bitcoins to Monero. Monero is an anonymity-enhancing cryptocurrency that leverages three different privacy technologies: ring signatures, ring confidential transactions (RingCT) and stealth addresses.

Ring signatures basically function as a built-in mixer. This means that when Monero funds are sent, senders arbitrarily select multiple other Monero users’ funds to also feature in the transaction log as possible senders. RingCT “hides not only the source of funds being sent, but also hides the amounts of the funds being sent from being visible on the blockchain,” according to the Monero website. Meanwhile, one-time stealth addresses are a self-explanatory feature.

However, the manual highlights Onion Market’s relatively low Monero exchange limit for users attempting to convert large amounts of Bitcoin. Regardless, “in order to unequivocally break our connection with the original money,” advises the manual, “we need to make at least one transaction in Monero.” The guide says that the “Bitpapa wallet is the best” for this chain-breaking transaction.

Bitpapa
Bitpapa hompage, source: Bitpapa.com

Bitpapa is a UAE-registered crytpo exchange that offers users “over 100 payment methods” to make their trades, according to the company’s website. In addition to Bitpapa’s “convenient acceptance in Monero and P2P-exchange of coins directly to the card, there is no mandatory verification, and only mail is needed for” user registration, notes the guide.

Takeaways

On Telegram, this post generated 17 likes, three clown emojis, one poop emoji, and 11 comments.

Cash out Manual
Cash-out manual user reaction, source: Telegram

One of the Telegram users who commented on the post, noted that there were many similar crypto cash-out services being marketed on Rutor, which has emerged as the most popular darknet forum since the demise of Hydra. Another user, going by the handle, ‘Ivett DarkWave Witch’ advised that transitioning to cash-based payment rails increases risk and that drop cards should be used when transacting through Bitpapa.

Translated Comment
Translated comment, source: Telegram

Besides Electrum, other secrecy-friendly wallets recommended in Russian cybercriminal TG communities include Trust Wallet, MetaMask, and Blue Wallet.

Wallet Opsec Discussions
Wallet OpSec Discussions in an unnamed channel, source: Telegram

While reports of decreasing crypto-crime, increased blockchain tracing capacity, and heightened regulatory oversight are encouraging, financial crime fighters must consider the evolving sophistication of the adversaries they are pursuing. The Cambrian explosion of the money-laundering-as-a-service (MLAS) economy on the dark web makes the pursuit of financial integrity in crypto markets that much more challenging.

At Cryptosec, we are trying to build a safer and more compliant crypto community. Keeping pace with the latest crypto-laundering threat intelligence is essential to secure the attack perimeter in a rapidly evolving AML risk landscape.

Saudi Arabia Vision 2030: Cybersecurity at the Core of Transformation

Cybersecurity Saudi Arabia

Last week, the Saudi Data and Artificial Intelligence Authority (SDAIA) launched a nationwide awareness campaign called “Ask Before”, intended to educate the public about the significance of personal data ahead of the implementation of a new national personal data protection system.

Emphasizing responsible data handling, privacy preservation, and fostering trust and collaboration between commercial entities and private individuals, “Ask Before” supports KSA’s new Personal Data Protection Law (PDPL), which became enforceable on September 14th.

The need for such a campaign stems from the fact that the PDPL is the first regulation of its kind rolled out in the kingdom, activated five years after Europe’s General Data Protection Regulation (GDPR). The new law is noteworthy, because it is yet further evidence of the accelerating maturity of Saudi Arabia’s digital economy, closely tied to the digitally-enabled developments of Vision 2030.

Guarding the Kingdom’s Digital Future

KSA’s ambitious plan to turn its nation into the model of a progressive 21st-century society places a significant emphasis on digital transformation. This is exemplified in the smart design approaches underpinning the super-project NEOM, and its various sub-projects like THE LINE, Oxagon, Trojena, and Sindalah. Deeply networked and resting on cutting-edge cyber-physical and AI-enabled technologies, these new environments will create numerous points of vulnerability, necessitating robust cybersecurity to protect critical systems and ensure public safety.

But, even beyond these high-profile, media-grabbing ventures, it’s clear that safeguarding digital assets, critical infrastructure, and sensitive data is going to be paramount to the success of Vision 2030. One such developmental area is the planned overhaul of government services which, though lacking the sexy bells and whistles of NEOM, will deliver significant social impact and represent a major security priority.

Saudi Arabia is investing heavily in e-government services to enhance citizen engagement and streamline administrative processes. More than 6,000 governmental services – representing 97% of services – have already been digitized, and as more government functions move online, the protection of sensitive citizen data becomes paramount to maintaining public trust and ensuring the efficient functioning of state institutions. As reflected in the new PDPL, the kingdom is also actively promoting the localization of data within its borders to ensure data sovereignty and enhance national security.

Multiple digital health projects, such as the deployment of electronic health records and telemedicine services, rely increasingly on secure data sharing and storage. Cybersecurity safeguards are vital to protect patient privacy and maintain the integrity of healthcare systems. And, as the country moves further ahead with its plans to transform the health sector, flagship developments like the SEHA Virtual Hospital – the largest of its kind in the world – are likely to increasingly incorporate bio-digital devices and approaches like remote surgery into medical diagnosis and treatment. Cyber-physical solutions such as these are set to revolutionize healthcare in general but, as we have already seen in other parts of the world, security of these systems is an existential necessity.

Finally, the expansion of the financial sector through initiatives like the Financial Sector Development Program (FSDP), also demands strong cybersecurity practices. Fundamental to Vision 2030’s goal of achieving greater economic diversification, protecting financial institutions and data is critical to ensure economic stability and investor confidence. This is an especially important point because, while elevated cybersecurity is necessary to protect citizen wellbeing, foreign investment is strategically crucial to delivering the multiplicity of KSA’s developmental objectives, and investors need to feel secure too.

On the one hand, the country’s digital transformation journey appears to be progressing well. Alibaba Cloud, the world’s largest cloud computing company, is the latest big name to open shop in Saudi Arabia, which should give a boost to the Saudi government’s ambitions of claiming a greater share of the Middle East cloud market, predicted to reach $9.8 billion by 2027, and growing at a CAGR of 21 percent. The National Development Technology Program (NTDP) is also on track to support IT startups, entrepreneurs, and investors with an estimated budget of SR2.5 billion, mirroring massive growth in VC investments.

The risk with so much digital development taking place on so many fronts is that gaps begin to appear and entire systems become vulnerable to cyber attack. It seems, though, that the country is cognizant of the potential pitfalls in this expansion and is taking appropriate steps to secure the economy against domestic and international cyber threats.

Taking necessary action

Saudi Arabia’s social and economic evolution over the past 100 years has been rapid and, especially more recently, been defined by a leap in technological development. As one may expect in such circumstances, growth has not always been accompanied by parallel progress in security.

Cyber risk was not something Saudi companies used to worry about. That changed with the 2012 massive Saudi Aramco hack that acted as a digital wake-up call, jolting the nation into recognizing the stark reality of cyber risks. Between 2016 and 2018, Saudi Arabia was among the most affected countries in the world when it came to cyberattacks. In 2019, it shared the less-than-desirable distinction of having the second-highest average cost per data breach with the UAE, while these Gulf nations also witnessed the highest average number of breached records. In the past, Saudi’s industrial sector has also shown itself to be vulnerable to cyber attacks, with 88% of organizations reporting ransomware attacks and incidents spiking whenever the country or surrounding region experiences geopolitical disruption.

But, these records are changing quickly. This year, the kingdom ranked second in the global Cybersecurity Index in the World Competitiveness Yearbook (WCY), and took 17th place  – up seven places from 2022 – in the overall competitiveness ranking.

Inconsistent cybersecurity measures might be seen as the growing pains of a fast-growing digital economy – what matters is how policymakers and industry players respond. Given the number of cybersecurity measures being rolled out in KSA, and the speed with which they are being deployed, it appears the Saudi government has recognized this area as a strategic priority, while businesses are responding with their own investments in advanced security measures.

In addition to the newly enforceable Personal Data Protection Law, some of the key developments in Saudi’s cybersecurity journey include:

The National Cybersecurity Authority (NCA)

Established in 2017, the NCA oversees the National Cybersecurity Strategy, a framework focused on effective cybersecurity governance, while managing cyber risks, and strengthening national defense capabilities. The NCA also plays a pivotal role in setting minimum cybersecurity standards for national and government agencies, and provides comprehensive policies and frameworks to assist organizations in safeguarding their data and networks.

The NCA’s 2023 National Plan for Cyber Assessments maps out a rigorous approach to regulating cybersecurity standards across national entities. Extensive assessments, compliance audits and cyber reviews of critical systems will help enforce the authority’s standards and manage cyber risk nationally.

Local legislation sets tough guardrails for cyber activity within KSA. With broader scope than the PDPL, the Anti-Cyber Crime Law combats cyber crimes, protects information security, and promotes legitimate computer and information network usage, while defining cyber crime and its punishments. The Electronic Transactions Law is a legal framework for electronic transactions that controls and regulates the safe conduct of digital transactions.

2023 National Plan for Cyber ​​Assessments

As part of its move to standardize cybersecurity quality across national authorities, the NCA has this year been following a programme of technical and compliance assessments to ensure entities are up to the standards required to ensure cyber safe institutions. The project also includes the establishment of an inventory of sensitive national assets and review systems to ensure adherence to the NCA cybersecurity provisions.

The Haseen Initiative

Officially known as the National Portal for Cyber Security Services, Haseen was developed by the NCA’s technical division, the Saudi Information Technology Company (SITE), as a holistic cyber management platform. It has a broad-spectrum role in supporting national entities as they increase resilience against cyber attacks, helping authorities assess and raise their cybersecurity capabilities. Key domains within Haseen relate to compliance management, information sharing, email authentication and verification of files and links, all intended to lift the overall level of national cyber safety.

The Global Cybersecurity Forum Institute

As part of Saudi Arabia’s growing cybersecurity leadership in the Middle East and beyond, the GCF Institute was founded in Riyadh earlier this year, bringing together international experts from government, the private sector, academia and interest groups to develop strategies for tackling global cybersecurity challenges. The institute enables KSA to access best practices from around the world, and share lessons learned in, for example, repelling the 110 million cyber threats detected in Saudia Arabia during 2022.

Council of Ministers for Cybersecurity

Based on a Saudi proposal at the recent 160th session of the Council of Arab Foreign Ministers of the Arab League, a regional body was formed to drive collaboration and coordination between Arab countries in all cybersecurity-related matters. Operating out of Riyadh and driven by KSA, the Council of Ministers for Cybersecurity has objectives of strengthening cybersecurity across the Arab world, recognizing that sustainable social development in this area will be impossible without cybersecure environments.

Forum of Incident Response and Security Teams

Just a few days ago, Saudi’s Human Resources Development Fund (also known as HADAF) was accepted into the Forum of Incident Response and Security Teams (FIRST), a US-based cybersecurity association widely recognized for its industry-leading incident response. For the KSA public sector, inclusion in this group of 656 businesses and government organizations across 101 countries promises a step change in cybersecurity capability. HADAF is the Kingdom’s 11th FIRST member and, with its governmental mandate, the fund will be able to significantly improve the efficiency of national organizations in safeguarding their systems and data.

This is a small selection of initiatives currently shaping the Saudi cybersecurity landscape. Aside from HADAF and NCA, bodies such as the Saudi Federation for Cybersecurity, Programming and Drones, and the Ministry of Communications and Information Technology are also having a significant influence on the accelerated evolution of regulations, systems maturity and skills availability in the country.

Further projects supporting this transformation include the National Cybersecurity Center to raise awareness of cybersecurity efforts; the founding of the National Academy of Cybersecurity to develop cybersecurity skills and capabilities in the Kingdom’s workforce; and the rollout  of a National Cybersecurity Awareness Program to educate citizens and residents.

Conclusion

As Saudi Arabia steers towards its Vision 2030 goals of diversification and knowledge-based economic growth, the emphasis on cybersecurity is not just relevant; it’s fundamental. Digital enablement of the economy, governmental services, health sector and private business means, as it does in most countries across the world today, that cybersecurity translates into national security.

But in KSA, where bold development plans include smart cities, smart ports, AI-integrated infrastructure and digital technologies at the core of all services, the stakes are raised. With such a radical expansion of the digital landscape, the attack surface increases dramatically too, but this does not appear to be slowing the Kingdom down. As with the challenges that inspired Vision 2030 in the first place, cybersecurity appears to be just one more puzzle that Saudi Arabia seems hungry to solve.

AI Security 101

AI Security

Artificial Intelligence (AI) is no longer just a buzzword; it’s an integral part of our daily lives, powering everything from our search for a perfect meme to critical infrastructure. But as Spider-Man’s Uncle Ben wisely said, “With great power comes great responsibility.” The power of AI is undeniable, but if not secured properly, it could end up making every meme a Chuck Norris meme.

Imagine a world where malicious actors can manipulate AI systems to make incorrect predictions, steal sensitive data, or even control the AI’s behavior. Without robust AI security, this dystopian scenario could become our reality. Ensuring the security of AI is not just about protecting algorithms; it’s about safeguarding our digital future. And the best way I can contribute to it is by raising awareness about AI-specific threats among as many cybersecurity and AI professionals.

On this page, I’ve compiled a selection of my intro articles on AI and ML security (in no particular order). This collection will continue to expand. As always, your feedback, suggestions for new topics, and other insights are invaluable and warmly welcomed.


Adversarial Attacks AI Security

Adversarial Attacks: The Hidden Risk in AI Security

The rapid proliferation of AI and Machine Learning (ML) technologies, from facial recognition to autonomous vehicles, has underscored the importance of cybersecurity. While AI and ML are revolutionizing cybersecurity by swiftly analyzing vast data sets and adapting to evolving threats, they also introduce new vulnerabilities. One such vulnerability is adversarial attacks, which specifically target AI and ML systems. These attacks involve inputting carefully crafted data to deceive the system, leading to incorrect decisions or classifications. For instance, an image’s pixels could be subtly manipulated to mislead a machine learning model, causing it to misidentify a stop sign as a yield sign. The article delves deep into the nature of adversarial attacks, their types (White-Box, Black-Box, Targeted, Non-Targeted), real-world examples, and the mechanisms behind them. Read more.


Semantic Adversarial Attacks: When Meaning Gets Twisted

Adversarial attacks manipulate data to deceive machine learning models, impacting their performance and reliability. A specific subset of these attacks, known as semantic adversarial attacks, focuses on twisting the semantic meaning behind data. Unlike traditional adversarial attacks that might add noise or make pixel-level changes, semantic attacks target the inherent understanding of the data. For instance, they might mislabel an image or change the meaning of sentences in text-based models. The article looks into the various techniques used in semantic adversarial attacks, the security implications they pose, and potential countermeasures. The piece underscores the growing threat of these attacks and the urgency of developing robust defenses to protect AI systems and the entities that rely on them. Read more.

Semantic Adversarial Attacks

AI Saliency Attacks

How Saliency Attacks Quietly Trick Your AI Models

Artificial Intelligence (AI) models, while transformative across sectors, are not immune to vulnerabilities. Among these, “Saliency Attacks” stand out as a covert threat. These attacks subtly manipulate the significant features, or “saliencies,” within data, deceiving AI models often without detection. In essence, they alter the critical data features that the model relies upon for decision-making, leading to incorrect predictions. As AI becomes integral to decision-making processes in areas like healthcare or finance, understanding and defending against saliency attacks is paramount. The article explores the nature of these attacks, their mechanisms, and the profound implications they have across various sectors. It emphasizes the importance of understanding and countering these threats to ensure the integrity of AI models and the safety of the systems they influence. Read more.


How to Defend Neural Networks from Trojan Attacks

Neural networks, which are inspired by the human brain, are integral to modern technologies such as voice recognition and medical diagnosis. However, their intricate design makes them susceptible to Trojan attacks. These attacks involve injecting malicious data into the training dataset, causing the neural network to associate it with a specific output, creating a hidden vulnerability. When activated, this vulnerability can lead to unpredictable behavior or incorrect decisions. The article delves into the nature of Trojan attacks, how they infiltrate neural networks, and real-world examples of potential threats. It also discusses why neural networks are vulnerable and outlines defensive measures, including prevention, detection, and mitigation strategies. The article emphasizes the importance of staying ahead of attackers by investing in research and collaboration to ensure the security of neural networks. Read more.

Trojan Attack

Data Poisoning ML AI

Understanding Data Poisoning: How It Compromises Machine Learning Models

Data poisoning attack targets the training data, the foundation of ML and AI. Data poisoning can significantly degrade ML model performance, leading to flawed analytics and potentially endangering lives. The article explains the concept of data poisoning, where adversaries manipulate training data to compromise machine learning models. It discusses various types of poisoning attacks, such as label flipping, outliers injection, and feature manipulation. The impact of these attacks can be vast, affecting model performance, decision boundaries, and introducing security risks. The article also presents case studies in sectors like autonomous vehicles, healthcare, financial fraud detection, and recommendation systems, highlighting the real-world implications of data poisoning. It concludes by suggesting mitigation strategies, emphasizing the importance of data sanitization, model regularization, real-time monitoring, third-party audits, and data provenance. Read more.


How Label-Flipping Attacks Mislead AI Systems

AI and ML systems’ effectiveness hinges on the integrity of labeled data, which is vulnerable to label-flipping attacks. In such attacks, adversaries manipulate training data labels, causing misclassifications. These attacks are particularly deceptive as they can make a model appear highly accurate on tampered data, but the model fails on genuine data. For instance, in financial scenarios, a compromised model might misidentify legitimate transactions as fraudulent. Label-flipping attacks can have severe consequences across sectors, from healthcare misdiagnoses to financial fraud. The article emphasizes the importance of robust security measures to detect and counteract such vulnerabilities in ML systems. Read more.

Label Flipping AI

GAN Poisoning AI

The Unseen Dangers of GAN Poisoning in AI

Generative Adversarial Networks (GANs) have emerged as a pivotal technology, driving innovations in data generation, image synthesis, and content creation. However, these networks are not immune to cyber vulnerabilities, with GAN Poisoning being a significant and often overlooked threat. This type of attack subtly manipulates the training data or the GAN model itself, leading to misleading or malicious outputs. The article looks into the mechanics of GAN Poisoning, highlighting its elusive nature and the vast real-world implications of undetected attacks. From generating false news articles and deepfake videos to simulating misleading financial data, the potential misuse of poisoned GANs is vast. Addressing this threat requires a combination of detection, prevention, and ongoing research, emphasizing the need for both technological and ethical solutions. As GANs continue to shape various industries, it’s crucial to ensure their security and responsible use. Read more.


Backdoor Attacks in Machine Learning Models

Machine learning (ML) models, while powerful, are susceptible to a range of security threats, with Backdoor Attacks being one of the most insidious. These attacks embed a covert trigger during a model’s training phase, allowing attackers to manipulate the model’s output when it encounters a specific, pre-defined input. Such attacks can remain undetected, making them particularly dangerous. For instance, a compromised model in autonomous driving could misinterpret traffic signals, or a financial system could overlook illicit transactions. The article delves deep into the nature of these attacks, their mechanisms, and the profound implications they have across various sectors. It emphasizes the importance of understanding and countering these threats to ensure the integrity of ML models and the safety of the systems they influence. Read more.

Backdoor Attacks ML

Meta Attacks

Meta-Attacks: Utilizing Machine Learning to Compromise Machine Learning Systems

Meta-attacks present a sophisticated cybersecurity threat, uniquely employing machine learning to target and compromise other machine learning systems. Unlike traditional cyberattacks, meta-attacks exploit inherent weaknesses in machine learning architectures, making them especially potent. For instance, a meta-attack might use its own machine-learning model to produce highly effective adversarial examples, misleading the target system. By harnessing machine learning against itself, meta-attacks elevate the stakes in the cybersecurity domain, necessitating advanced defensive strategies to counter these adaptive threats. The article delves into the mechanics of meta-attacks, from identifying vulnerabilities in target systems to deploying the attack, emphasizing the significance of understanding and defending against these challenges in the ever-evolving field of cybersecurity. Read more.


How Multimodal Attacks Exploit Models Trained on Multiple Data Types

Multimodal models, capable of processing diverse data types like text, images, audio, and more, have revolutionized industries from healthcare to autonomous vehicles. However, their multifaceted nature also makes them vulnerable to attacks. The article explores the mechanics of multimodal attacks, which exploit the complexities of these systems. These attacks can target individual data types or synchronize attacks across multiple data types, amplifying the potential damage. Real-world implications of such attacks span sectors like healthcare, smart cities, and social media, with risks ranging from misdiagnoses to traffic chaos and the spread of misinformation. The article emphasizes the need for multi-layered defense strategies, including adversarial training and machine learning-based anomaly detection. It also highlights the potential of federated learning and explainable AI as future solutions. The piece concludes by stressing the importance of technological innovation and regulatory frameworks to safeguard against the risks of multimodal attacks. Read more.

Multimodal Attacks

AI Model Fragmentation

Model Fragmentation and What it Means for Security

Machine learning models are increasingly becoming a part of various technological applications. As these models evolve, they often undergo a process termed as “model fragmentation”, where different versions, architectures, or subsets of a model are deployed across various platforms or use cases. While this fragmentation provides adaptability and flexibility, it also brings forth a range of unique security challenges. The article delves into the reasons for model fragmentation, such as different versions, decentralized networks, hardware constraints, and regional/legal constraints. It also categorizes fragmentation into version-based, architecture-based, and data-based types, each with its own set of security implications. The piece further discusses methods of detection and prevention, highlighting the limitations of current methods and emphasizing the importance of ongoing research in this domain. Read more.


Outsmarting AI with Model Evasion

In the realm of cybersecurity, AI classifiers like neural networks are pivotal for real-time anomaly detection. Yet, these models are vulnerable to evasion tactics, including adversarial perturbations and feature-space manipulations. These tactics exploit the models’ mathematical foundations, confusing their decision-making. The article looks into the various evasion techniques, from simple evasion methods like altering observable features to sophisticated adversarial attacks that exploit mathematical properties. It also touches on data poisoning, where attackers tamper with training data, and model manipulation, where attackers directly alter model parameters. The article emphasizes the importance of understanding these evasion techniques to develop more resilient AI-driven security measures. It concludes by highlighting the need for collaboration between machine learning experts and security professionals to bolster next-gen AI security. Read more.

Model Evasion AI

Model Inversion Attack AI

How Model Inversion Attacks Compromise AI Systems

The effectiveness of AI is contingent upon the robustness and security of its underlying algorithms. A significant vulnerability that threatens these aspects is the phenomenon of Model Inversion Attacks. These attacks aim to exploit AI models to infer sensitive information about the training data or even the algorithmic intricacies of the model itself. Given that many AI models operate in regulated environments where data confidentiality is crucial, such as healthcare or financial systems, the implications of model inversion attacks are vast and concerning. The article delves into the nature of these attacks, their mechanics, implications, and potential mitigation strategies. It emphasizes the importance of securing AI systems against such threats, highlighting the ongoing challenges and research in the cybersecurity domain. Read more.


The Dark Art of Model Stealing: What You Need to Know

AI and ML models are vulnerable to a form of cyber attack known as “model stealing.” This attack involves hackers duplicating a machine learning model without having direct access to its parameters or data. The article explores the definition of model stealing, the types of AI models that are most vulnerable, real-world examples of model theft, and the techniques employed by attackers. It also discusses the risks involved, best practices for preventing model theft, and recent research on the topic. The article underscores the importance of understanding the intricacies of model stealing and the need for robust security measures to protect these valuable assets in an era where AI models are both a product and a potential vulnerability. Read more.

AI Model Stealing

Data Spoofing AI

When AI Trusts False Data: Exploring Data Spoofing’s Impact on Security

AI and ML technologies are particularly effective due to their ability to process and analyze vast amounts of data at unparalleled speeds, enabling real-time threat detection and mitigation. However, this strength is also a potential vulnerability: AI systems are heavily reliant on the integrity of the data they process, making them susceptible to Data Spoofing. Data spoofing involves the deliberate manipulation or fabrication of data to deceive systems, which can severely compromise the efficacy of AI-based security measures. The article delves deep into the nature of data spoofing, its real-world implications, the types of AI systems affected, and potential countermeasures. It underscores the importance of understanding and addressing the challenges posed by data spoofing to ensure the reliability and security of AI systems across various sectors. Read more.


The Threat of Query Attacks on Machine Learning Models

Machine learning models, integral to various industries from healthcare to finance, are vulnerable to a range of cyberattacks, with query attacks being a notable threat. These attacks target machine learning models by issuing a series of queries, typically input data, to extract valuable insights from the model’s output. This can range from understanding the model’s architecture to uncovering the data it was trained on. The stealthy nature of these attacks allows them to mimic legitimate user activity, making detection challenging. The article delves into the intricacies of query attacks, from their methods of execution to their implications. It underscores the importance of robust security measures to safeguard machine learning models against such threats, emphasizing the need for ongoing research and vigilance in the ever-evolving cybersecurity landscape. Read more.

Query Attack

Differential Privacy AI

Securing Data Labeling Through Differential Privacy

Data labelling process for supervised machine learning often involves handling sensitive or personal information, necessitating robust privacy measures. Differential Privacy emerges as a solution, introducing ‘random noise’ into the data, which acts as a protective layer, making it difficult to reverse-engineer sensitive details. This method ensures data remains secure even during real-world analytical queries or complex machine-learning operations. The article looks into the intricacies of Differential Privacy, its pros and cons, and its significance in ensuring a balance between data utility and privacy. It underscores the importance of safeguarding labeled data, highlighting the potential consequences of privacy breaches and emphasizing the need for expert consultation, parameter tuning, and regular audits. Read more.


How Dynamic Data Masking Reinforces Machine Learning Security

Machine learning (ML) systems are handling vast amounts of sensitive data, from personal to financial details. As these systems process and learn from this data, they face significant cybersecurity challenges. One of the primary concerns is how to manage and safeguard sensitive data throughout the ML workflow. Among the various solutions available, Dynamic Data Masking (DDM) stands out as a key tool for enhancing security measures. DDM acts as a real-time data protection mechanism, obfuscating sensitive data during queries without altering the original data. This method ensures that ML systems can function without jeopardizing the integrity of the information, making it an essential component of comprehensive cybersecurity strategies, especially in sectors like healthcare, finance, and government services. Read more.

Dynamic Data Masking ML

Homomorphic Encryption ML

Securing Machine Learning Workflows through Homomorphic Encryption

Traditional encryption methods often fall short in safeguarding ML models and their associated training data. Homomorphic Encryption emerges as a solution, allowing computations to be performed directly on encrypted data, thus eliminating the risks associated with exposing sensitive data during processing. This article explores intricacies of Homomorphic Encryption, discussing its unique capabilities, potential use-cases, and the latest research in the domain. From healthcare to finance, the applications of this encryption technique are vast, promising enhanced data privacy without compromising the utility of ML models. The article underscores the importance of adopting such transformative encryption methods, emphasizing their role in shaping the future of machine learning, especially in sectors where data sensitivity is paramount. Read more.


Twitter API for Secure Data Collection in Machine Learning Workflows

Machine learning (ML) systems are handling vast amounts of sensitive data, from personal to financial details. As these systems process and learn from this data, they face significant cybersecurity challenges. One of the primary concerns is how to manage and safeguard sensitive data throughout the ML workflow. Among the various solutions available, Dynamic Data Masking (DDM) stands out as a key tool for enhancing security measures. DDM acts as a real-time data protection mechanism, obfuscating sensitive data during queries without altering the original data. This method ensures that ML systems can function without jeopardizing the integrity of the information, making it an essential component of comprehensive cybersecurity strategies, especially in sectors like healthcare, finance, and government services. Read more.

API Security ML AI

AI Disinformation

AI-Exacerbated Disinformation and Threats to Democracy

The proliferation of AI-powered disinformation campaigns poses a significant threat to democratic societies. The article explores the intricacies of AI-driven disinformation, highlighting how advanced algorithms can generate fake news, deepfakes, and other forms of misleading content with unprecedented speed and scale. These AI-generated falsehoods can manipulate public opinion, undermine trust in institutions, and even influence election outcomes. The article underscores the challenges in detecting and countering such disinformation, given its sophisticated nature and the rapid pace at which it spreads across social media platforms. The piece also emphasizes the need for a multi-faceted approach, involving technological solutions, media literacy education, and regulatory measures, to combat the menace of AI-driven disinformation and safeguard the pillars of democracy. Read more.

Why We Seriously Need a Chief AI Security Officer (CAISO)

Chief AI Security Officer CAISO

Artificial Intelligence (AI) has quickly, nay, explosively transitioned from a sci-fi concept to a foundational pillar of modern business. A recent report by McKinsey highlights the rise of generative AI, revealing that within less than a year of its public debut, a staggering one-third of surveyed organizations have integrated generative AI into at least one business function. Gartner predicted that by 2024, 75% of enterprises will shift from piloting to operationalizing AI. I can’t recall seeing any other emerging technology in history take off as quickly as AI has.

Keep in mind, when I discuss AI adoption, I am not just referring to using ChatGPT for drafting emails or having an ML system flagging cybersecurity alert up to analysts. It’s much more profound than that. Organizations are entrusting AI with a growing array of tasks to operate independently. Whether it’s customer service chatbots handling queries or sophisticated supply chain management and automation of physical goods movements, the value of AI’s autonomous capabilities is becoming undeniable. It’s evident that businesses aren’t just warming up to the idea; they’re actively seeking ways to grant AI more autonomy in their day-to-day operations.

As more businesses and more departments, jump on the AI bandwagon, we’re seeing a whole new world of challenges pop up. With every new AI integration, the complexities of ensuring its secure, compliant and ethical deployment grow exponentially.

The Chief Information Security Officers (CISOs) I’ve spoken to are already losing sleep just from the “traditional” cybersecurity challenges. The ever-faster-evolving cyber threat landscape is already a constant source of anxiety. Now, imagine their stress levels when their organizations start adopting autonomous AI systems in various pockets of the organization—systems CISOs weren’t consulted about and have little clue on how to secure. It’s enough to give anyone an ulcer. As one CISO describes it: “I feel like I am trying to cross a busy intersection blindfolded; at midnight; in a black onesie…”

This is where the idea of a “Chief AI Security Officer” (CAISO) comes in. This dedicated executive will not only safeguard AI systems but also ensure that businesses harness AI’s potential without compromising on security, ethics, or compliance. As AIs continue to reshape industries, the CAISO will be at the forefront, navigating the challenges and opportunities of this new AI-driven landscape.

Key CAISO Responsibilities

With AI’s breakneck expansion, the distinctions between ‘cybersecurity’ and ‘AI security’ are becoming increasingly pronounced. While both disciplines aim to safeguard digital assets, their focus and the challenges they address diverge in significant ways. Traditional cybersecurity is primarily about defending digital infrastructures from external threats, breaches, and unauthorized access. On the other hand, AI security has to address unique challenges posed by artificial intelligence systems, ensuring not just their robustness but also their ethical and transparent operation as well as unique internal vulnerabilities intrinsic to AI models and algorithms. These include adversarial attacks, model bias, and data poisoning. Furthermore, unlike most software that remains stable until patched, AI systems are in a constant state of flux, learning and adapting from new data. This dynamism introduces a fresh set of monitoring challenges, as the system’s behavior can change over time, even without explicit reprogramming.

“In AI security, the very system we guard could turn into our greatest adversary.”  – Marin Ivezic

Some of the key differences CAISO would have to address include:

AI Model Security

AI Model Security focuses on the protection and defense of machine learning and deep learning models from various threats and vulnerabilities. As AI models become integral to business operations, they become attractive targets for malicious actors. Threats can range from adversarial attacks, where slight input modifications can deceive a model into making incorrect predictions, to model inversion attacks, where attackers attempt to extract sensitive information from the model’s outputs. Another concern is model theft, where attackers try to replicate a proprietary model by querying it repeatedly. Ensuring the confidentiality, integrity, and availability of AI models is paramount. This involves not only defending against direct attacks but also ensuring that the model remains robust and reliable in its predictions, even in the face of malicious inputs or environmental changes. Proper AI model security ensures that these computational brains continue to operate as intended. For more info on AI security see AI Security 101.

AI Supply Chain Security

This function would focus on ensuring the security of the entire AI supply chain, from data collection tools and infrastructure to third-party software libraries and pre-trained models. A compromised element anywhere in the supply chain could introduce vulnerabilities into the final deployed AI system. Given the increasing reliance on AI for critical decisions and operations, ensuring the security of the AI supply chain is paramount.

AI Infrastructure Security

AI Infrastructure Security focuses on protecting the underlying systems and platforms that support the development, deployment, and operation of AI solutions. This encompasses the hardware, software frameworks, cloud platforms, and the networks. As AI models process vast amounts of data and often require significant computational resources, they can become prime targets for cyberattacks. A breach in AI infrastructure can lead to unauthorized data access, model tampering, or even the deployment of malicious AI models.

While traditional cybersecurity handled by CISOs does cover aspects like data integrity, infrastructure security, and protection against unauthorized access, the specific nuances of AI infrastructure security make this a specialized domain. In my mind.

Some of the AI infrastructure-specific security challenges that are different from traditional cybersecurity include:

  • Specialized Hardware: AI often requires specialized hardware like Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs) for training and inference. These devices can have vulnerabilities distinct from traditional CPUs.
  • Data Flow Complexity: AI systems often involve complex data pipelines, moving vast amounts of data between storage, processing, and serving infrastructure. Ensuring the security and integrity of this volume and velocity of data would be a new challenge for many CISOs.
  • Model Serving: Once trained, AI models are deployed in inference engines, which might be exposed to external requests. These engines can be targeted for model extraction or poisoning through approaches that wouldn’t be familiar to traditional CISOs.
  • Pipeline Dependencies: AI pipelines often depend on various open-source libraries and tools. Ensuring these dependencies are free from vulnerabilities and are regularly updated is a unique challenge that, I would argue, not many CISOs have faced at the same scale.
  • Real-time Constraints: Some AI applications, like those in autonomous vehicles or real-time anomaly detection, have real-time processing constraints. Ensuring security measures don’t introduce latency is a delicate balance and it wouldn’t be a common experience for majority of traditional CISOs.

MLOps and ModelOps Security

MLOps, a fusion of Machine Learning and Operations, emphasizes the seamless integration of ML into production environments. MLOps security, therefore, focuses on ensuring the safety and integrity of this entire pipeline – from data collection and model training to deployment and monitoring. It addresses challenges like versioning of models, secure data access during training, and safe model deployment in real-time applications.

While the AI security mentioned above broadly encompasses the protection of AI models, data, and infrastructure, MLOps security dives deeper into the operational aspects. It’s concerned with the continuous integration and delivery (CI/CD) processes specific to ML workflows. This includes safeguarding automated testing environments, ensuring only validated models are deployed, and monitoring models in production for any drift or anomalies. In essence, while AI security provides the overarching protective framework, MLOps security ensures that the day-to-day operations of integrating ML into business processes remain uncompromised.

AI Data Protection

AI Data Protection is about ensuring the confidentiality, integrity, and availability of data used in AI systems. Given that AI models are only as good as the data they’re trained on, protecting the training and validation data is critical. This not only involves protecting data from unauthorized access but also ensuring that the data remains unbiased, representative, and free from malicious tampering. It also reduces the organizational regulatory risk exposure as upholding data privacy, especially in the age of GDPR and other growing global data protection regulations, is non-negotiable.

Traditional data privacy controls focus on encrypting data, setting up firewalls, and controlling access. However, with AI, there are unique challenges. For instance, even if data is anonymized, AI models can sometimes reverse-engineer and reveal personal information, a phenomenon known as “model inversion.” To counteract this, techniques like differential privacy are employed. Differential privacy ensures that AI models, when queried, don’t reveal specific data about an individual, even indirectly. It introduces “noise” to the data in a way that maintains the data’s overall utility for training models but prevents the extraction of individual data points. This is just one example of how AI data protection requires a fresh approach, beyond traditional privacy and data protection measures.

Regulation & Compliance

AI is, rightly so, already drawing attention of countless regulatory bodies. The landscape of AI-specific regulations and standards is rapidly evolving. Sometimes it feels like it’s changing hourly. These regulations aim to ensure that AI systems are transparent, fair, ethical, and do not inadvertently harm users or perpetuate biases. They cover privacy and data protection, transparency, fairness, right to explanation, ethical use, fairness, export of defense or dual-use systems, cybersecurity, and so on.

Moreover, different industries might have their own set of AI guidelines. For instance, AI in healthcare might be subject to stricter regulations concerning patient data privacy and model explainability than AI in entertainment.

CAISOs must ensure that as their organizations innovate with AI, they remain compliant with both current regulations and are prepared for future legislative shifts. This requires a close collaboration with legal and compliance teams, and a proactive approach, continuously monitoring the regulatory environment, and ensuring that AI deployments are both ethical and compliant.

Ethical AI Deployment

The deployment of AI systems goes beyond just technical and regulatory considerations; it is inextricably linked with ethics. Ensuring ethical AI deployment means guaranteeing that AI systems operate fairly, transparently, and without unintended biases. Ethical challenges arise when AI models, trained on historical data, perpetuate or even amplify existing societal biases. For example, a recruitment AI tool might favor certain demographics over others based on biased training data, leading to unfair hiring practices. The ethical use of AI also encompasses transparency and explainability. Stakeholders should be able to understand how AI systems make decisions, especially in critical areas like healthcare, finance, or criminal justice. CAISOs must also consider the broader societal implications of AI deployments. For example, while an AI system might optimize efficiency in a business process, it could lead to job displacements.

Navigating these ethical challenges requires CAISOs to collaborate closely with diverse teams, from data scientists to human rights experts and ethicists.

AI Explainability and Interpretability

While not strictly a security concern, the ability to explain and interpret AI decisions is crucial for trust. As AI systems become more complex, understanding their decision-making processes becomes less straightforward. This poses a challenge, especially when AI-driven decisions have significant consequences, such as in medical diagnoses, financial lending, or criminal sentencing. Explainability refers to the ability to describe in human terms why an AI system made a particular decision. Without this, it’s challenging to trust and validate the system’s outputs.

Interpretability, on the other hand, relates to the inherent design of the AI model. Some models, like deep neural networks, are often termed “black boxes” because their internal workings are hard to decipher. CAISOs face the challenge of ensuring that these models are both effective and interpretable, allowing for potential audits, reviews, or checks. The goal is to strike a balance between model performance and the ability to understand and explain its decisions. This not only builds trust among users and stakeholders but also aligns with emerging regulations that demand greater transparency in AI decision-making.

Bias Detection and Mitigation

The issue of bias in AI isn’t just a technical hiccup; it’s a profound ethical concern that CAISOs must grapple with. AI systems, being trained on vast amounts of data, can inadvertently learn and perpetuate the biases present in that data. This isn’t about a machine making an innocent mistake; it’s about systems potentially making decisions that favor one group over another or perpetuating harmful stereotypes.

Imagine a hiring AI that, due to biased training data, favors candidates from a particular background over others. Or consider a facial recognition system that struggles to accurately identify individuals from certain ethnic groups. Such biases can have real-world consequences, ranging from unfair job opportunities to misidentification by law enforcement. CAISOs have the responsibility to implement rigorous bias detection mechanisms and, once detected, to deploy strategies to mitigate these biases. This ensures that AI systems are fair, equitable, and don’t perpetuate or amplify societal inequalities.

Continuous Learning and Adaptation

Unlike traditional software that remains static until manually updated, AI systems have the potential to continuously evolve, refine their knowledge, and improve over time. The problem is that such evolving systems can drift over time. Ensuring that this drift doesn’t introduce vulnerabilities or biases is a significant challenge. CAISOs must strike a balance, ensuring AI systems can learn and adapt to new information while maintaining their integrity and purpose. This involves monitoring the learning process, validating new knowledge, and periodically recalibrating the AI to ensure it remains on the right track.

Disinformation and Deepfakes

With the rise of AI-generated content, defending against and detecting deepfakes and other forms of AI-generated disinformation is a growing concern. Deepfakes, which are hyper-realistic but entirely fake content generated by AI, can range from altered videos of public figures to fabricated voice recordings. The implications are vast: from perfectly-personalized, high-volume spearphishing campaigns to spreading false news and damaging reputations.

Imagine a scenario where a deepfake video of a CEO announcing a company merger goes viral, leading to stock market chaos. Or consider the ramifications of a fabricated voice recording used to authorize financial transactions. CAISOs must be at the forefront of developing detection tools to identify and counter these AI-generated falsities. This involves not just technical solutions but also raising awareness and educating stakeholders about the potential risks.

Cyber-Kinetic Security

The fusion of the digital and physical worlds through AI-driven autonomous systems introduces a new realm of security concerns for Chief AI Security Officers (CAISOs): the Cyber-Kinetic challenge. In these cyber-physical systems, a cyber attack doesn’t just result in data breaches or software malfunctions; it can lead to real-world, kinetic impacts with potentially devastating consequences. Imagine an AI-driven power grid being manipulated to cause blackouts, or an autonomous vehicle’s system being hacked to cause collisions.

The stakes are high, especially when human lives, well-being, or the environment are on the line. A compromised AI system controlling a chemical plant, for instance, could lead to environmental disasters. CAISOs, therefore, must ensure that these systems are not only digitally secure but also resilient to attacks that aim to cause physical harm. This involves a multi-layered approach, integrating robust digital defenses with fail-safes and redundancies in the physical components.

Human-AI Collaboration Security

Somewhat overlapping with previous topics, but, in my mind worth separate consideration is the Human-AI Collaboration – one most promising yet challenging areas of AI adoption. As AI systems become teammates rather than just tools, ensuring the security of this partnership becomes paramount for Chief AI Security Officers (CAISOs). It’s not just about ensuring the AI behaves correctly; it’s about ensuring that the human-AI interaction is secure, trustworthy, and free from external manipulations.

Imagine a scenario where an AI assistant provides recommendations to a doctor for patient treatment. If the integrity of this collaboration is compromised, it could lead to incorrect medical decisions. Similarly, in industries like finance or defense, a manipulated suggestion from an AI could lead to significant financial or security risks. CAISOs must ensure that the communication channels between humans and AIs are secure, the AI’s recommendations are transparent and verifiable, and that there are mechanisms in place to detect and counteract any attempts to deceive or mislead either the human or the AI. In the age of collaborative AI, the security focus shifts from just protecting the AI to safeguarding the entire human-AI collaborative ecosystem.

Physical Security of AI-Driven Systems

While much of the focus on AI security revolves around digital threats, the physical security of AI-driven systems is equally crucial for Chief AI Security Officers (CAISOs) to consider. AI systems, especially those deployed in critical infrastructure or in the field, can be targets for physical tampering, sabotage, or theft. For instance, sensors feeding data into an AI system could be manipulated at the analog part of the sensor, or the hardware on which AI models run could be physically accessed to extract sensitive information or inject malicious code.

Moreover, edge devices, like IoT gadgets powered by AI, are often deployed in unsecured environments, making them vulnerable to physical attacks. CAISOs must ensure that these devices are tamper-proof and can detect and report any physical breaches. This might involve using secure hardware enclaves, tamper-evident seals, or even self-destruct mechanisms for highly sensitive applications.

Robustness to Environmental Changes

As AI systems become more integrated into our daily operations, their ability to remain resilient and effective amidst environmental changes becomes another new concern. It’s not just about an AI’s ability to function in a stable environment; it’s about ensuring that the AI can adapt and respond effectively when the surrounding conditions shift. CAISOs, in collaboration with AI engineers, must ensure that AI systems are not only trained on diverse and representative data but also have mechanisms to detect, adapt, and respond to environmental changes. This involves continuous monitoring, retraining, and updating of AI models to keep them relevant and effective.

Post-Deployment Monitoring

Ensuring that AI systems function as intended post-deployment is another critical responsibility for CAISOs. Once an AI system is live, it interacts with real-world data, users, and other systems, all of which can introduce unforeseen challenges. An AI model that performed well during testing might start behaving unexpectedly when exposed to new types of data or malicious inputs. Or over time, the model might drift from its intended purpose due to changes in the data it processes. CAISOs must establish robust post-deployment monitoring mechanisms to track the performance, behavior, and health of AI systems in real-time. This involves setting up alerts for anomalies, regularly auditing the system’s decisions, and having a feedback loop to refine and recalibrate the AI as needed. In essence, post-deployment monitoring ensures that the AI system remains reliable, trustworthy, and aligned with its intended purpose throughout its lifecycle.

Quantum Threats to AI

Quantum computers, with their ability to process vast amounts of data simultaneously, can potentially crack encryption methods that are currently deemed unbreakable. This means that AI systems, which often rely on encryption for data protection and secure communications, could become vulnerable to quantum-powered attacks. Moreover, quantum algorithms might be able to reverse-engineer AI models or find vulnerabilities in them at speeds previously thought impossible. For CAISOs, the challenge is twofold: understanding the evolving quantum threat landscape and proactively developing strategies to safeguard AI assets in a post-quantum world. This includes researching quantum-resistant encryption methods and rethinking current AI security protocols in light of quantum capabilities.

Where Should the CAISO Sit in the Organizational Structure?

Realistically, when organizations first recognize the need for a CAISO, it’s likely that this role will initially report to the Chief Information Security Officer (CISO). This is a natural starting point, given the overlapping concerns of AI and traditional cybersecurity. Organizations, especially large ones, are often resistant to drastic structural changes, and adding a new role to the leadership team isn’t a decision made lightly.

As businesses become more reliant on AI-driven solutions, the stakes will get higher. AI isn’t just a tool; it’s rapidly becoming the backbone of many critical business operations replacing both, tools and people previously executing a particular function. With AI’s rise, cyber threats will keep evolving. Attackers will increasingly target AI systems, recognizing their strategic importance. Traditional cybersecurity skills, while valuable, don’t translate directly to the unique challenges of AI. The skills gap for AI security will keep widening. Collaboration with various other parts of organization will keep deepening.

Given the factors mentioned above, it’s only a matter of time before organizations recognize the strategic importance of the CAISO role. As AI continues to shape the future of business, CAISOs will find themselves not just reporting to, but being an integral part of, the executive leadership team. Their insights, expertise, and leadership will be pivotal in navigating the challenges and opportunities that AI presents.

While the journey of the CAISO role might start under the umbrella of traditional cybersecurity, its eventual destination is clear: a seat at the executive table.

Potential Challenges With CAISO Introduction

The adoption of a CAISO role in organizations would undoubtedly bring about a range of challenges, both anticipated and unforeseen. Some potential ones include:

Role definition: Clearly defining the boundaries and responsibilities of the CAISO in relation to other roles like CISO, CTO, CIO, and data science leads might be challenging.

Related to that, hierarchy and reporting: Deciding where the CAISO sits in the organizational structure and to whom they report can be contentious. Should they be on the executive team, or report to the CISO or CTO?

Budget allocation: Securing a dedicated budget for AI-specific security initiatives might be challenging, especially if there’s a perception that the traditional cybersecurity budget should cover it.

Dependence on other functions: the CAISO role, at least initially, will be more of a coordinator of resources across a number of different departments, rather than an owner of a dedicated team covering all required competencies. Consider for example the Threat Intelligence function. Keeping up with the latest AI-specific threats, vulnerabilities and mitigation techniques will be a huge challenge. If using the existing cyber threat intelligence team and providers, would AI security receive sufficient attention? If not, is it realistic to build an AI-specific intelligence team?

Skill gap: There’s a significant skill gap in the intersection of AI and security. Finding and retaining talent with expertise in both areas might be difficult. Or, alternatively, getting the budget and the required time to upskill existing team members might present other challenges.

Resistance to change: Existing teams might resist the introduction of a new executive role, seeing it as an encroachment on their territory or an added layer of bureaucracy.

Shadow AI: CISOs are currently reluctant, or ill-equipped to handle AI systems. By the time organizations adopt the CAISO role, shadow AI – AI solutions that are not officially known or under the control of the cybersecurity department – would have proliferated and it would be a major challenge to get them under the control of CAISO without impacting the operations.

Conclusion

As AI continues its meteoric rise, becoming an indispensable tool in nearly every business sector, the need for a dedicated Chief AI Security Officer (CAISO) becomes increasingly evident. The role of the CAISO isn’t merely about ensuring that AI systems function optimally; it’s about guaranteeing their security, ensuring they’re deployed ethically, and navigating the intricate maze of regulatory compliance. With AI’s capabilities expanding daily, the potential risks and ethical dilemmas grow in tandem.

While the concept of a CAISO might seem like a futuristic notion to some, the explosive adoption rate of AI technologies suggests that this isn’t just a distant possibility but an impending reality. Forward-thinking organizations are already contemplating this move.

How to Defend Neural Networks from Trojan Attacks

Trojan Attack

Neural networks, inspired by the human brain, play a pivotal role in modern technology, powering applications like voice recognition and medical diagnosis. However, their complexity makes them vulnerable to cybersecurity threats, specifically Trojan attacks, which can manipulate them to make incorrect decisions. Given their increasing prevalence in systems that affect our daily lives, from smartphones to healthcare, it’s crucial for everyone to understand the importance of securing these advanced computing models against such vulnerabilities.

The Trojan Threat in Neural Networks

What is a Trojan Attack?

In the context of computer security, a “Trojan attack” refers to malicious software (often called “malware“) that disguises itself as something benign or trustworthy to gain access to a system. Once inside, it can unleash harmful operations. Named after the ancient Greek story of the Trojan Horse, where soldiers hid inside a wooden horse to infiltrate Troy, a Trojan attack similarly deceives systems or users into letting it through the gates.

How Do Trojans Infiltrate Neural Networks?

Neural networks learn from data. They are trained on large datasets to recognize patterns or make decisions. A Trojan attack in a neural network typically involves injecting malicious data into this training dataset. This ‘poisoned’ data is crafted in such a way that the neural network begins to associate it with a certain output, creating a hidden vulnerability. When activated, this vulnerability can cause the neural network to behave unpredictably or make incorrect decisions, often without any noticeable signs of tampering.

In more technical terms, an attacker might add a specific ‘trigger’ to input data, such as a particular pattern in an image or a specific sequence of words in a text. When the neural network later encounters this trigger, it misbehaves in a way that benefits the attacker, like misidentifying a stop sign as a yield sign in a self-driving car.

Real-World Examples

Healthcare Systems: Medical imaging techniques like X-rays, MRI scans, and CT scans increasingly rely on machine learning algorithms for automated diagnosis. An attacker could introduce a subtle but malicious alteration into an image that a doctor would likely overlook, but the machine would interpret as a particular condition. This could lead to life-threatening situations like misdiagnosis and the subsequent application of incorrect treatments. For example, imagine a scenario where a Trojan attack leads a machine to misdiagnose a benign tumor as malignant, leading to unnecessary and harmful treatments for the patient.

Personal Assistants: Smart home devices like Amazon’s AlexaGoogle Home, and Apple’s Siri have become integrated into many households. These devices use neural networks to understand and process voice commands. A Trojan attack could change the behavior of these virtual assistants to convert them into surveillance devices, listening in on private conversations and sending the data back to attackers. Alternatively, Trojans could manipulate the assistant to execute harmful tasks, such as unlocking smart doors without authentication or making unauthorized purchases.

Automotive Industry: Self-driving cars are inching closer to becoming a daily reality, and their operation depends heavily on neural networks to interpret data from sensors, cameras, and radars. Although there are no known instances of Trojan attacks causing real-world accidents, security experts have conducted simulations that show how easily these attacks could manipulate a car’s decision-making. For instance, a Trojan could make a vehicle interpret a stop sign as a yield sign, potentially causing accidents at intersections. The stakes are extremely high, given the life-or-death nature of driving.

Financial Sector: Financial firms use machine learning algorithms to sift through enormous amounts of transaction data to detect fraudulent activities. A Trojan attack could inject malicious triggers into the training data, causing the algorithm to intentionally overlook certain types of unusual but fraudulent transactions. This could allow criminals to siphon off large sums of money over time without detection. For example, a compromised algorithm might ignore wire transfers below a certain amount, allowing attackers to perform multiple low-value transactions that collectively result in significant financial losses.

Why Neural Networks are Vulnerable?

Neural networks are susceptible to Trojan attacks primarily because of their complexity and the way they learn from data. Much like the human brain, which has various regions responsible for different functions, a neural network is composed of layers of interconnected nodes that process and transmit information. This intricate architecture can have weak points, similar to how the immune system has vulnerabilities that diseases can exploit. During the training phase, where a neural network learns to recognize patterns from a dataset, inserting malicious data can be likened to introducing a virus into the human body. Just as a person may not show immediate symptoms, the neural network may function normally in most cases but act maliciously when triggered by a specific input, akin to a dormant disease suddenly flaring up.

This vulnerability arises because neural networks are not inherently designed to verify the integrity of the data they are trained on or the commands they receive. They function on the principle of “garbage in, garbage out,” meaning that if they are trained or manipulated with malicious data, the output will also be compromised. In essence, the very adaptability and learning capabilities that make neural networks so powerful also make them susceptible to hidden threats like Trojan attacks.

Defensive Measures

Prevention

One of the most effective ways to prevent Trojan attacks in neural networks is through rigorous code reviews and architecture scrutiny. By examining the code that constructs the neural network, developers can preempt vulnerabilities that may later be exploited. Secure data collection and preprocessing form another line of defense. Ensuring that the data used to train the neural network is clean, well-curated, and sourced from reputable places can go a long way in reducing the risk of introducing Trojan-infected data into the learning algorithm.

Detection

Detecting a Trojan attack is often like finding a needle in a haystack, given the complexity of neural networks. However, specialized methods are being developed to identify these insidious threats. Anomaly detection plays a key role in this regard. By continuously monitoring the network’s behavior and comparing it against a baseline, these tools can flag irregularities that may be indicative of a Trojan attack. Machine learning models can also be trained to identify anomalies in other machine learning models, creating a layer of meta-security.

Mitigation

Once a Trojan is detected, immediate action is required to minimize its impact. Traditional cybersecurity methods, like isolating affected systems, can be effective but may not address the root cause in the neural network. Hence, specific mitigation strategies for neural networks are essential. Some machine learning models are being designed to be more resilient to Trojan attacks, capable of identifying and nullifying malicious triggers within themselves. Think of it as an “immune response” by the neural network to purge the intruder. Furthermore, periodic “health checks” of the neural network, through techniques like retraining the model on a clean dataset, can help restore its integrity.

By incorporating prevention, detection, and mitigation strategies, we can build a more robust defense against Trojan attacks, ensuring that neural networks continue to be a force for good rather than a vulnerable point of exploitation.

Future Outlook

As neural networks become increasingly integral to various aspects of society, research into defending against Trojan attacks has become a burgeoning field. Academia and industry are collaborating on innovative techniques to secure these complex systems, from designing architectures resistant to Trojan attacks to developing advanced detection algorithms that leverage artificial intelligence itself. Companies are investing in in-house cybersecurity teams focused on machine learning, while governments are ramping up initiatives to set security standards and fund research in this critical area. By prioritizing this issue now, the aim is to stay one step ahead of attackers and ensure that as neural networks evolve, their security mechanisms evolve in tandem.

Recent Research on Trojan Attacks in Neural Networks

The field of cybersecurity has seen considerable advances in understanding the vulnerability of neural networks to Trojan attacks. One seminal work in the area [1] investigates the intricacies of incorporating hidden Trojan models directly into neural networks. In a similar vein, [2] provides a comprehensive framework for defending against such covert attacks, shedding light on how network architectures can be modified for greater resilience. Adding a different perspective, a study [3] delves into attacks that utilize clean, unmodified data for deceptive purposes and offers countermeasures to defend against them. In an effort to automate the detection of Trojans, the research [4] proposes methods for identifying maliciously trained models through anomaly detection techniques. Meanwhile, both corporate and governmental bodies are heavily drawing from another impactful paper [5] to standardize security measures across various applications of neural networks. These studies collectively signify a strong commitment from the academic and industrial communities to make neural networks more secure and robust against Trojan threats.

Conclusion

As neural networks continue to permeate every facet of modern life, from healthcare and transportation to personal assistance and financial systems, the urgency to secure these advanced computing models against Trojan attacks has never been greater. Research is making strides in detecting and mitigating these vulnerabilities, and collaborative efforts between academia, industry, and government are essential for staying ahead of increasingly sophisticated threats. While the road to entirely secure neural networks may be long and filled with challenges, the ongoing work in the field offers a promising outlook for creating more resilient systems that can benefit society without compromising security.

References

  1. Guo, C., Wu, R., & Weinberger, K. Q. (2020). On hiding neural networks inside neural networks. arXiv preprint arXiv:2002.10078.
  2. Xu, K., Liu, S., Chen, P. Y., Zhao, P., & Lin, X. (2020). Defending against backdoor attack on deep neural networks. arXiv preprint arXiv:2002.12162.
  3. Chen, Y., Gong, X., Wang, Q., Di, X., & Huang, H. (2020). Backdoor attacks and defenses for deep neural networks in outsourced cloud environments. IEEE Network34(5), 141-147.
  4. Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., & Zhao, B. Y. (2019, May). Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP) (pp. 707-723). IEEE.
  5. Zhou, S., Liu, C., Ye, D., Zhu, T., Zhou, W., & Yu, P. S. (2022). Adversarial attacks and defenses in deep learning: From a perspective of cybersecurity. ACM Computing Surveys55(8), 1-39.

Will the Kingdom of Saudi Arabia (KSA) beat Japan to Society 5.0?

The Line Neom KSA Society 5.0

Ask most people what they remember from 2016 – if they remember anything at all – and there are usually two big events that float to the front of their minds: Britain voted to leave the European Union and the United States voted Donald Trump into the White House. Together, these two episodes sent shock waves around the world. In the UK, the Brexit referendum was followed by a national decline in mental health. In the US, American college students exhibited levels of stress comparable to PTSD.

Even beyond those borders, Brexit and the Trump election became emblematic of the uncertainty and volatility of our age. They were such unlikely events, so seemingly unpredictable, they had everyone from political analysts to social scientists to farm workers scrambling to make sense of it all.

But, at the same time that much of the world was in a state of reactive sensemaking, two other nations were presenting their views of how they intended to meet the evolving challenges of a shapeshifting global future.

In January 2016, Shinzo Abe announced Japan’s Fifth Science and Technology Basic Plan, a comprehensive strategic framework for the country’s future development. It outlined Japan’s vision and goals for advancing science, technology, and innovation over a five-year period, from 2016 to 2020. But its view stretched far beyond that timeframe. Included in the new policy approach was a radical new conception of social potential, encapsulated in the term Society 5.0: a highly advanced and inclusive society achieved through the integration of cutting-edge technologies like artificial intelligence (AI), the Internet of Things (IoT), big data, and robotics.

Then, in April of the same year, the Kingdom of Saudi Arabia (KSA) launched Vision 2030, a comprehensive and ambitious long-term development plan aimed at diversifying the Saudi economy, reducing its dependence on oil revenue, and promoting various sectors such as technology, tourism, and entertainment. The plan outlined a roadmap for the country’s economic and social transformation over a 15-year period, with the goal of achieving a more diversified, innovative, and vibrant Saudi Arabia by the year 2030.

In the context of a tumultuous year, these long-term views of progressive societal re-engineering stood out as bold, visionary replies to a series of increasingly unavoidable conclusions:

  • The world is becoming more complex at an accelerating rate and will only continue to do so – there’s no going back to a simpler time.
  • That complexity is creating ‘wicked problems’: multi-dimensional, constantly evolving problems with multiple causes, multiple symptoms and multiple solutions, affecting multiple stakeholders.
  • Solving ‘wicked problems’ requires integrative thinking – uniting, for example, the individual and the collective, tradition and future, cyber and physical, nation state and global village.

But that is not where the similarities between these two visions end. Though initially they were pursued for very different reasons, the Fifth Science and Technology Basic Plan (developed further in the Sixth Science and Technology Basic Plan) and Vision 2030 are united by their focus on building technologically-enabled ecosystems that support the healthy evolution of industry, civic society, and private domains. These principles were first, and most explicitly, mapped out in Japan’s concept of Society 5.0, and they are common to both nations’ strategies for a prosperous future, but it appears that seven years down the line – halfway to 2030 – it may be KSA that first brings Society 5.0 to life.

A story of transformation

It is not surprising that a concept as advanced and audacious as Society 5.0 emerged from Japan. When it was published, it reflected, some would say depended on, Japan’s rich history of technological innovation and its ability to adapt to the changing landscape of the digital age.

Japan has the highest economic complexity of any country on the planet, and has maintained the Number One position in the Economic Complexity Index (ECI) for 21 consecutive years. The ECI is often used as a tool to understand a country’s economic structure and its potential for growth and development, based on its export capabilities. Countries with a higher ECI are considered to have more diversified and complex economies, which can be advantageous for long-term economic growth.

Japan’s unbroken reign at the top of the world’s economic complexity leaderboard is thanks largely to the diversity and sophistication of its exports: many different products that require advanced knowledge, technology, and skills to produce. And it was this capability that Japan was leveraging when it came up with Society 5.0, an evolutionary leap from Society 1.0 (based on hunting),  Society 2.0 (a social and economic system based on agriculture and trade of goods), Society 3.0 (defined by industrialization), and Society 4.0 (the information society found in most developed countries today). This was a nation that had the know-how and track record to deliver the model of “A human-centered society that balances economic advancement with the resolution of social problems by a system that highly integrates cyberspace and physical space.” From most other countries this would have sounded like a fantastical pipe dream – all hope and aspiration, but with little substance.

Until fairly recently, many would have counted Saudi Arabia among those countries. Less than a hundred years ago it still had an agrarian economy with traditional trade and limited infrastructure. In the Japanese model, this was Society 2.0. But, the oil boom, which began with the first commercial oil discovery in 1938, led to rapid industrialization and urbanization (Society 3.0). Saudi Arabia’s economy became heavily dependent on oil exports, bringing substantial wealth but also vulnerability to oil price fluctuations. Whereas Japan’s primary motivation for transformation sprung from the challenges associated with a rapidly ageing population, KSA was faced with the unavoidable truth that the non-renewable natural resource upon which its economy was based would one day run out. And, in all likelihood, the world would stop consuming it before then, leading to a total market collapse. Vision 2030 was designed to change this by reducing the country’s reliance on oil, promoting other sectors like technology and tourism, and fostering a more diversified and sustainable economy for the future.

Society 5.0 and Vision 2030 were born from very different needs and challenges experienced by societies of very different levels of economic and technical maturity. Yet there are remarkable similarities between the two frameworks.

Both Society 5.0 and Vision 2030 share a focus on reducing dependency on a single industry or revenue source. Society 5.0 seeks to advance Japan’s economy by integrating cutting-edge technologies into various sectors, while Vision 2030 aims to diversify Saudi Arabia’s economy away from oil through investments in new industries.

Both initiatives emphasize the importance of advanced technologies such as AI, the Internet of Things, and digitalization in driving economic growth and improving the quality of life for citizens. This shared emphasis on technology-enabled improvements in social and private wellbeing in both countries extends to key domains such as healthcare, education, and overall living standards.

Both concepts recognize the significance of sustainable practices and addressing environmental challenges. Vision 2030 specifically names growth in the contribution of renewables to the national energy mix and a more competitive energy market as focal areas, while sponsoring efforts to reduce greenhouse gas emissions. In this pursuit, and execution of their overall development agendas, both Japan and Saudi Arabia are prioritizing international collaboration and driving foreign investment to achieve their respective goals.

Finally, both visions recognize the importance of ethical and responsible use of technology. In Japan, Society 5.0 was redefined in 2021 as “a sustainable and resilient society that protects the safety and security of the people and one that realizes the well-being of individuals.” In Vision 2030, where strategic objectives include developing the digital economy and privatizing various state-owned assets and companies to stimulate economic growth, citizen safety is paramount. It’s perhaps unnecessary to point out that cybersecurity and cyber-kinetic security are profoundly important here, so fundamental are these disciplines to the type of technology-augmented society proposed in Society 5.0 and Vision 2030.

Japan has made significant progress in tech research and development since 2016, and has successfully deployed major projects like Michibiki, a “quasi zenith” global navigation satellite system. Tokyo’s space policy committee recently announced its intention to expand the Michibiki network from 4 satellites to 11, which would make it possible to position any user in Japan to within centimeters of their location. This level of accuracy goes well beyond standard GPS capability, and will be crucial to the effective deployment of emerging technologies like self-driving cars and autonomous drones, key components of smart cities and other Society 5.0 ecosystems.

But it is in Saudi Arabia that we see perhaps the most tangible manifestation of the physical world Society 5.0 imagines. In paradigm-shifting projects like NEOM, and its sub-projects THE LINE, Sindalah, Trojena and Oxagon, KSA is moving rapidly towards realizing the amalgamation of nature and human-centric digital environments.

Building the future

Situated along the Red Sea coast in the northwest corner of Saudi Arabia, and covering an area roughly the size of Belgium, NEOM is envisioned as a cutting-edge, technologically advanced, and environmentally sustainable model for successful societies of the future. The name “NEOM” is a portmanteau, taking “Neo” from the Greek word for “new”, and “M” from ‘Mustaqbal’, an Arabic word meaning “future”. Already home to 2,800 staff from 86 countries, NEOM was launched in 2017 by Crown Prince and Prime Minister, His Royal Highness Prince Mohammed bin Salman bin Abdulaziz, colloquially known as “MBS”.

Beyond cementing KSA’s place at the vanguard of human development, NEOM’s primary objective is to diversify the Saudi Arabian economy by creating a hub for various industries, including technology, tourism, and entertainment. The project is fully powered by renewable energy sources, utilizes sustainable transportation systems, and employs eco-friendly practices to minimize its environmental impact.

Reflecting its appetite for foreign investment and desire to build a pan-national population, NEOM will be a semi-independent zone with its own laws and semi-governmental authority. In a country sometimes accused of authoritarianism, this is a bold commitment to more liberal governance and deregulation.

As progressive as these frameworks may be, few people would call them exciting. That kind of language is reserved for some of the other developments in NEOM, such as Sindahla – a luxury island getaway and likely the first of NEOM’s projects to come online – or Trojena, a unique mountain lifestyle location that will see novel architectural installations blended with natural landscapes to create high-end living and skiing facilities.

But it’s two of NEOM’s other leading-edge developments that are capturing most of the outside world’s attention.

Oxagon is the reimagining of an industrial hub, powered by 100% renewable energy and built on economic principles of circularity. Like Society 5.0, its DNA emerges from Industry 4.0, also known as the Fourth Industrial Revolution, a transformative concept that represents the integration of digital technologies, automation, data analytics, and the Internet of Things (IoT) into industrial processes. Industry 4.0 revolutionizes manufacturing and production by creating “smart factories” where machines, systems, and processes are interconnected and communicate with each other. Through the collection and analysis of vast amounts of data, the application of this approach at Oxagon will enable real-time decision-making, predictive maintenance, and increased efficiency. This shift will not only enhance productivity but also offer opportunities for customization, sustainability, and innovation across industries, marking a fundamental change in how products are made and how businesses operate.

Oxagon includes plans to create the Port of NEOM, a next-generation smart port to join the ranks of Rotterdam and Singapore, where cutting-edge technologies like Digital Twins and full IoT enablement deliver increased efficiencies. Port of NEOM is planned to run a proprietary integrated digital supply chain system that orchestrates port, logistics and rail delivery operations to make the port a highly-functioning connective node between NEOM and the outside world.

Finally, and perhaps most aspirational, is THE LINE. Still mostly in the concept phase, THE LINE aims to redefine urban living. You’ve probably heard that kind of promise before because many developers make that claim, but never before has anyone achieved what THE LINE hopes to achieve at the scale it hopes to achieve it.

This would be a linear city, 170 kms long, but only 200 meters wide, stretching from the mountainous regions of NEOM to the Red Sea and housing up to nine million people. 95% of public space would be reserved for nature. 100% of energy would be green. There would be no cars and no pollution. Everything you need would be within a 5-minute walk and a high-speed rail would carry you from one end of the city to the other in 20 minutes. The city itself would incorporate nature yet be totally smart, leveraging cutting-edge technologies, such as AI, robotics, and biotechnology, to create a city ecosystem that enhances the quality of life for residents and visitors.

All sound to good to be true? You wouldn’t be the first to say so. The plans for THE LINE and NEOM more widely are so ambitious that it’s easy to write them off as fantasy and idealism. And, frankly, it would be politically inconvenient for many parts of the world if KSA managed to pull this off, so it’s not surprising when negative narratives tend to lead to public opinion. A promotional video for the NEOM project starts with the affirmation, “NEOM is real”, undoubtedly in response to a sentiment of disbelief in a) NEOM’s viability; and b) KSA’s ability to make it happen.

If you’re familiar with my outlook on the world, you’ll know that I prefer to take a pragmatic view of things. I’m not usually one to be drawn into fanciful ideas or fluffy dreams, nor am I easily won over by a smooth piece of marketing. But NEOM is real. Construction is already under way. Investment – both foreign and domestic – is flowing in. No one expects this to be delivered overnight. The project is planned to be developed in multiple phases and the full realization of NEOM’s vision is expected to occur over several decades.

The outcome might very well be the boldest and most forward-looking vision for urban development, economic diversification, and technological innovation the world has seen. It would position Saudi Arabia as a global leader in various industries and provide new opportunities for both Saudi citizens and international investors. But more than that, it would be a living showcase for what is possible in building novel societies that integrate people, nature and technology. Possibly for the first time, we would see Society 5.0 at scale, and that is an exciting prospect for humanity in general.

In the early 2000s I was a partner in Knowledge Age: a multi-disciplinary consultancy firm assimilating strategy, financial and economic analysis, policy, education and a number of telecommunications and information technology disciplines in order to help emerging markets develop knowledge-based economies. Our job, essentially, was to accelerate the progress to Society 4.0. We worked with a number of governments on country-level digital strategy and digital transformation.

As in Saudi Arabia today, the primary objective was diversification. At the time, the desire of some of our clients was to diversify away from reliance on property, tourism or oil towards more knowledge-based services, IT outsourcing, ecommerce. My experience during that time left a strong impression on me. In most countries, the transformation strategy is planned many decades in advance. And the execution takes even longer, often due to corruption, government changes, and all manner of delays.

In the UAE, however, things were different. Throughout my stay there, and then later in Qatar, I often heard proclamations along the lines of “our aim is to achieve the impossible”. And most of the time, they did. They built some of the most impressive global infrastructure projects I’ve seen while implementing complex and cutting-edge technologies.

And I see the same in KSA today. It is true that they started some mega-projects in the past that were never finished. But I believe that was an issue of motivation rather than capability, capacity or resource – it was primarily a “why”, not a “how”, issue. Previously it may have been more about competing with their neighbours, but today it is about survival.

With the unveiling of Vision 2030 in 2016, MBS declared his country had about 20 years to change the whole of society because the commodity upon which that society’s quality of life was built – oil – would keep losing its relevance. From that vantage point, Vision 2030 was not, and is not, a vanity project or an expensive daydream. It is a strategy for sustainability, in the truest and most radical sense of that word. I can feel that difference in KSA. Super ambitious NEOM and THE LINE are actually happening. Foreign companies are genuinely getting involved. Things are changing. This time, they might truly “achieve the impossible”.

Is 5G security being sacrificed at the altar of profit, politics and process?

5G Security Politics

Homo sapiens is an incredibly adaptable species, arguably the most adaptable ever. But it is also a forgetful one, quick to take things for granted. Many of us can remember when cell phones first emerged, when the internet first became publicly available, when the first iPhone was released. These momentous shifts occurred within a generation, altering the nature of society and civilization.

Just a few decades ago, none of these existed, but by the time Covid-19 hit, billions of people were able to lift their smartphone and video call a loved one on the other side of the world. At the time, few people seemed to pause and recognize that the ability to make that call was almost miraculous. Almost.

Because, the massive technological complex that gave people access to each other across the globe was not made up of miracles, but rather hardware, software, and the networks that grew around them. And what ensured these networks were able to function across borders, cultures and industries? Standards – the guidelines, specifications and protocols that define the development, implementation, and function of technologies and infrastructure components.

Sure, they’re not sexy. When social media goes crazy about the latest device or communications technology, nobody talks about standards. But they are foundational to the successful integration of those technologies and devices. Standards are not very exciting, but they are extremely important, ensuring consistency, compatibility, and reliability across various systems, devices, networks, and applications.

Standards define your office network, your home network, the cloud, the internet, and mobile networks. They’re why you can phone home when you’re overseas, collaborate with a colleague on an online document, or have a pizza delivered to your door with just a few clicks. And they’re why you can do all of these things safely, because one of the crucial roles of standards is in ensuring a secure cyber environment.

This is especially true in telecommunications, where constant evolution has necessitated the growth of standard-setting bodies that can help guide and shape the industry. These Standard Development Organisations (SDOs) help to ensure that technology is interoperable, reliable, and secure across different networks and devices, a role that has been crucial during the emergence of 5G.

With its radically different network architecture, virtualized network components and progressive technologies like network slicing, 5G represents a whole new level of challenge in standards development, and with this heightened focus has come increased scrutiny of the standards setting process. Industry commentators have raised concern about the way in which standards are set and what is included in them. The process has flaws and is open to manipulation: the stakeholders that make up these SDOs are able game the system for corporate gain.

Of course, there’s nothing wrong in principle with having commercial motives, which is why this debate is less an ethical one and more a practical one. When the global standards-setting process is not working as it should, we risk sacrificing interoperability for profitability. 5G networks are jeopardized. Security will not evolve at the rate it needs to keep the network safe, and 5G, the most powerful technology to have been developed in decades, could also become the most dangerous.

Origins of telecoms SDOs

Standard Development Organizations bring together varied interests including industry experts, businesses and government agencies, to collaborate on defining common specifications and guidelines. Well-known examples include the International Organization for Standardization (ISO) that develops standards across various industries and sectors, such as technology, manufacturing, healthcare; the Internet Engineering Task Force (IETF), responsible for developing and maintaining standards for internet protocols and technologies; the Institute of Electrical and Electronics Engineers (IEEE), that develops standards in fields like telecommunications, information technology, power and energy; and 3GPP, the most influential SDO in mobile telecommunications technologies.

Strictly speaking, 3GPP is an engineering organization, not an SDO – it sets the technical specifications that are translated into standards by the seven global SDOs – but its influence is such that it is regarded as an SDO in effect. The origins of 3GPP can be traced back to the early days of mobile telephony, when the first generation (1G) of cellular networks was introduced. These networks were based on analog technology and were largely limited to voice communications.

In the 1980s, the second generation (2G) of cellular networks was developed, which introduced digital technology and allowed for the transmission of data in addition to voice. The development of 2G networks was driven by a desire to improve the efficiency and capacity of cellular networks, as well as to support new services such as SMS messaging.

As mobile technology continued to evolve, there was a growing need for a unified standard that would allow for interoperability between different networks and devices. In response to this need, several standard-setting bodies were established, including the European Telecommunications Standards Institute (ETSI) and the International Telecommunication Union (ITU).

In 1998, the GSM Association (GSMA), which represents the interests of mobile network operators around the world, established the Third Generation Partnership Project (3GPP) to develop a common standard for third-generation (3G) cellular networks. The 3GPP was created as a collaboration between several regional standards bodies, including ETSI, the Japan-based Association of Radio Industries and Businesses (ARIB), and the American National Standards Institute (ANSI).

Since its establishment, 3GPP has played a critical role in the development of mobile telecommunications standards, including the development of 3G, 4G, and 5G technologies, and today boasts a global membership that includes over 500 organizations from around the world.

SEParation of powers

While 3GPP has been successful in developing standards that have helped to advance mobile telecommunications, the process of developing these standards has come under criticism in recent years. Some complain the 3GPP process is sluggish and bureaucratic, slowed down by the competing agendas and priorities of the body’s many membership organizations.

Influence in the standards development process is also heavily weighted in favor of larger companies, most of whom are major players in the telecoms industry. Quite simply, participation in the process of technical definition and development comes at a cost and, when it comes to deciding what the next standards release is going to look like, those with the deepest pockets are able to bulk up their representation at voting events with the level of technical expertise required to make valid contributions to standards. This has geopolitical implications – as we’ll see shortly – but the primary play here is commercial. There’s a lot of money to be made in standards-setting, and it all starts with standard essential patents (SEPs).

5G SEPs are patents deemed essential to the implementation of a 5G technical standard. Royalties on these patents are paid by anyone wishing to comply with the relevant technical standard and use the patented technology in their products or services. Implementers may include manufacturers of devices, infrastructure providers, service providers, or any entity incorporating the patented technology into their offerings. The specific arrangements and agreements for royalty payments vary, and can be calculated based on factors such as the number of units sold, the revenue generated from implementing the SEPs, or other mutually agreed-upon terms.

The commercial value of owning an SEP can be significant, and thanks to the 3GPP decision-making structure, that commercial benefit is more likely to pool around those who have the most influence in the standards-setting process. If you want to have your patent regarded as “essential” – and, therefore, a mandatory royalty-earning part of technological implementation – it certainly helps if you’re one of the strongest voices in the room when it’s decided what’s to be included in the next standard. The obvious risk – and an increasingly vocalized critique of 3GPP – is the potential for anti-competitive behavior.

Companies that own SEPs can demand high royalties from other companies that want to use their technology, which can make it difficult for those companies to compete effectively. This has led to accusations that the SEP ownership model reduces competition and establishes the perfect conditions for an effective oligopoly featuring a handful of major players like Ericsson, Huawei, LG, Nokia, Qualcomm and Samsung. Critics argue that by owning patents deemed essential to 3GPP standards, companies can dictate licensing terms, leading to inflated costs and barriers to entry for competitors. This can hinder innovation and limit consumer choice, as companies lacking the necessary patents may struggle to enter the market or face excessive licensing fees.

The simple commercial inequalities such a market arrangement generates have the tendency to foster territorial and counter-productive (for the industry) practices, such as “patent hold-ups” and “patent hold-outs.” Patent hold-up refers to the situation where a patent holder delays the disclosure of its SEPs until after the standard is established. This strategic move enables them to negotiate more favorable licensing terms, effectively exploiting implementers’ reliance on their patents. Conversely, patent hold-out occurs when implementers refuse to negotiate licenses in an attempt to devalue the patent holder’s intellectual property. Both practices can harm the industry’s competitive dynamics and hinder the adoption of new technologies. Even without such cynical tactics, though, SEPs play a huge strategic role – especially for telecoms technology giants – in ensuring the ceaseless battle for leadership of increasingly competitive industrial value chains.

Anti-competitive behavior is theoretically averted by agreements between industry stakeholders that SEP royalties be subject to fair, reasonable, and non-discriminatory (FRAND) licensing commitments. These commitments aim to ensure that the licensing terms and royalty rates are fair and accessible to all potential implementers, encouraging widespread adoption of the standard and promoting fair competition. As we all know, though, commitments in principle work perfectly in principle, but often come unstuck in practice.

FRANDs

A lot of noise is made about who – organization or country – owns the most 5G SEPs, which is understandable; given their powerful commercial and strategic value, SEPs have the potential to secure companies and nations huge advantages in the race for autonomy and control in networked 5G technologies. But, the view on who leads the global 5G patent race varies significantly depending on who you speak to and how they’re choosing to read international patent filing records, which are open to a lot of interpretation anyway.

We could look at the number of standards contributions, but not all standards involve SEPs so those numbers are inconclusive. We could consider the number of patents filed, but this is a poor marker of contribution to innovation and tech development as a patent is simply a measure of uniqueness, not a measure of potential value. Counting SEPs doesn’t give a clear and accurate picture either because SEPs vary so much in value, usage and impact. Also, not all patents are essential. In fact, as much as 70% of patents – perhaps even more – do not qualify as “essential”.

This issue is plaguing the industry. Determining which patents qualify as essential to 3GPP standards is a complex and contentious process. The risk of over-declaring patents as essential or under-declaring them undermines the efficacy of the standards. Over-declaration may lead to patent thickets, where multiple patent holders claim rights over similar technologies, creating legal complexities and uncertainty. Conversely, under-declaration may result in technological gaps and hinder interoperability. Striking the right balance in the essentiality determination process is crucial but challenging, which is partly why the UK government has recently started a review of the SEP ecosystem to help determine whether the current system is effective and fair to all parties, and whether government intervention may be required. Experts on behalf of the European Commission have also suggested that better and fairer rules are needed to dictate the licensing of SEPs.

That legislative bodies are asking these questions is a positive sign, but there is a lot of work to be done before we reach anything close to a global consensus on these matters. While the standards-setting process may include a commitment to FRAND licensing terms, the test of that commitment takes place far from 3GPP conferences and in the courtrooms of the world.

Resolving disputes related to SEPs is a multifaceted challenge. Inconsistencies in national patent laws, varying interpretations of FRAND commitments, and jurisdictional issues complicate the resolution process. Additionally, enforcement mechanisms differ across countries, making it challenging to ensure consistent and fair outcomes. This may be a global issue, but it’s rooted in regional peculiarities: jurisdiction policy is a major factor in deciding the outcomes of a growing body of SEP licensing litigation. Where a patent is owned and enforced matters significantly, which is one of the reasons people are so focused on who owns what.

Despite the difficulties in analysis described above, work by IPlytics suggests that there is still a clear direction of travel in global SEP ownership and it’s pointing to the east: China leads in the number of declared 5G patent families as well as standards submissions. With the recent Huawei ban still fresh in our minds, it’s clear that Chinese dominance in 5G SEP ownership raises a lot of geopolitical questions for those in the West. But, we don’t need to wait for what plays out on the world stage to see conflict around 5G patent ownership – we already have multiple examples in corporate litigation in this area and they suggest that much of the expectation of FRANDs democratizing 5G technology implementation has been naive.

Disputes over what constitutes a fair and reasonable licensing fee often involve patent holders being accused of demanding disproportionately high royalties, and implementers arguing for lower rates to promote wider access and affordability. The resultant impasse is not helped by the lack of clear guidelines and inconsistent enforcement of FRAND commitments. Things are going to get worse before they get better too. Determining FRAND rates is complicated enough, but with its escalated network complexity 5G promises to raise far more challenges in agreeing licensing terms that suit everyone.

Added to this, the rapid expansion of the Internet of Things (IoT), which includes consumer goods as well as dedicated verticals within the IoT like Industrial IoT and Healthcare IoT, means a growing number of use cases, devices and concomitant software developments. 5G is a nested ecosystem that’s difficult to tease apart. As a result, the legal solutions to effective and appropriate SEP management are going to take a long time to resolve. To reach a fair and credible 5G patent landscape is going to require changes by SDOs to their current processes, possibly even regulatory intervention. The current approach followed by organizations like 3GPP is simply not up to the task in managing an evolution of 5G that is safe and secure for all.

Costs to 5G Security

Issues with negotiating and licensing SEPs slow down 5G network development and ultimately have a negative impact on the ecosystem as a whole, but a far bigger concern is the fragmentation and delay in security implementation caused by commercial conflicts. Extended licensing disputes or negotiations can hinder the timely integration of essential security features into network infrastructure and devices, leaving the network and users vulnerable to cyber or cyber-kinetic attack.

Disputes over SEP licensing terms potentially encourage implementers to opt out or avoid fully implementing necessary security measures, leading to a fragmented security landscape. That effect could be compounded by breakdowns in relationship between key stakeholders as a result of legal or commercial contests. Poor relationships make collaboration more difficult and less reliable, and collaboration is the foundation of a strong, cohesive approach to 5G network security. That collaboration also requires a free flow of information between security researchers, vendors and practitioners – if hard lines are taken on IP restrictions or patent licensing, doing those jobs properly can be severely hampered.

Solving this conundrum is not easy. It will require structural changes to the standard setting process, disincentivizing oligopolistic behavior and strengthen dispute resolution mechanisms to address licensing disputes effectively and efficiently. It may require regulatory reform. It will require all stakeholders to commit more authentically and consistently to doing what SDOs do when they’re operating at their best: prioritizing the broader good over private gain.

Territorial mindsets and continuous patent disputes are not sustainable – in the long term they will damage the industry and limit 5G’s potential. This technology is not about speed, nor is it about profit, even though there’s obvious value in both these things. 5G is about the potential it liberates, about what becomes possible at the frontiers of humanity’s evolution and the society we can build beyond those boundaries. For any of that to happen, we need to rethink the way we agree and implement the standards that underpin human connectivity.

Model Fragmentation and What it Means for Security

AI Model Fragmentation

Machine learning models have become integral components in a myriad of technological applications, ranging from data analytics and natural language processing to autonomous vehicles and healthcare diagnostics. As these models evolve, they often undergo a process known as model fragmentation, where various versions, architectures, or subsets of a model are deployed across different platforms or use cases. While fragmentation enables flexibility and adaptability, it also introduces a host of unique security challenges. These challenges are often overlooked in traditional cybersecurity discourse, yet they are crucial for the safe and reliable deployment of machine learning systems in our increasingly interconnected world.

What is Model Fragmentation?

Model fragmentation is the phenomenon where a single machine-learning model is not used uniformly across all instances, platforms, or applications. Instead, different versions, configurations, or subsets of the model are deployed based on specific needs, constraints, or local optimizations. This can result in multiple fragmented instances of the original model operating in parallel, each potentially having different performance characteristics, data sensitivities, and security vulnerabilities.

Reasons for Model Fragmentation

Model fragmentation, while seemingly a complexity, is often a strategic and necessary adaptation in the ever-evolving realm of machine learning. Here, we delve deeper into the various reasons that prompt the fragmentation of models:

Different Versions

Progressive Rollouts: As with many software systems, updates to machine learning models aren’t always deployed universally, all at once. Instead, they’re often rolled out progressively to ensure stability and manage potential issues. Thus, at any given time, multiple versions of a model could be in use.

Customization for Applications: Some applications may require specific tweaks or features, prompting the deployment of custom versions of a model. For instance, while a cutting-edge application might benefit from the latest model features, legacy systems might still be running older, more stable versions of the same model to ensure compatibility and reliability.

Decentralized Networks

Federated Learning: This approach decentralizes machine learning by training localized versions of models directly on user devices, such as smartphones or tablets. As each device learns from its unique data, the model becomes inherently fragmented across the network.

Edge Computing: Here, data processing and analytics happen directly at the data source, be it IoT devices, local servers, or other edge devices. This requires a localized version of the model to be deployed on these devices, contributing further to model fragmentation.

Hardware Constraints

Different devices come with varying computational powers. For instance, while a data center may run a complex deep learning model with billions of parameters, a smartwatch would require a much-simplified version of the same model. Thus, to accommodate hardware constraints and yet deliver optimal performance, models often undergo fragmentation.

Data sovereignty and privacy laws, such as the GDPR in Europe or the CCPA in California, often stipulate how and where data can be processed. To comply with these regional regulations, companies might need to train and deploy region-specific models, leading to fragmentation.

Cultural and regional nuances might also necessitate different model behaviors or outputs, prompting region-specific model versions.

Specialized Use-Cases

A one-size-fits-all model is not always the best approach, especially when addressing niche markets or specific tasks. For instance, a general-purpose image recognition model might be adapted and specialized to recognize specific industrial parts for a manufacturing use case. Such specialization naturally leads to model fragmentation as companies tailor models to meet unique requirements.

Understanding these reasons is pivotal, as each introduces its own set of vulnerabilities and considerations when it comes to securing and maintaining the fragmented models.

Types of Model Fragmentation

Understanding the types of model fragmentation is critical for both improving performance and enhancing security. Each type introduces its own set of challenges and potential vulnerabilities. Below, we discuss the major types of model fragmentation in detail:

Version-based Fragmentation

Updates and Patches: In the fast-paced world of machine learning, constant updates and patches to models are common. Whether for bug fixes, performance improvements, or feature additions, multiple versions of the same model often coexist.

Legacy Support: Older systems might not be compatible with the latest models due to hardware limitations or software dependencies. In such cases, legacy versions of models continue to operate, often without the security measures incorporated in newer versions.

Security Implications: With multiple versions in operation, the surface area for potential security threats increases. Outdated versions may lack the latest security features, making them particularly vulnerable.

Architecture-based Fragmentation

Task-Specific Adjustments: Sometimes, the basic architecture of the machine learning model remains the same, but minor adjustments are made to better suit specific tasks. For example, a text classification model might be fine-tuned for spam detection in one use case and sentiment analysis in another.

Hardware Optimizations: To make the model more efficient on specific hardware, certain architectural elements may be adjusted. For example, reducing the number of layers or parameters can enable the model to run more efficiently on mobile devices with limited computational resources.

Security Implications: These architectural alterations can introduce new vulnerabilities or exacerbate existing ones, especially if the changes are not rigorously tested for security flaws.

Data-based Fragmentation

Regional Data Laws: Due to data sovereignty and privacy regulations like GDPR or CCPA, a model may be trained on region-specific data and deployed solely in that region. Such models are fragmented based on the data they process.

Specialized Training Sets: In specialized use cases, a model might be trained on a specific subset of data. For instance, a healthcare diagnostics model could be trained exclusively on data pertaining to a particular demographic or medical condition.

Security Implications: Fragmentation based on data sets can introduce biases or other vulnerabilities, especially if the data used for training or operation has its own inherent risks, such as sensitive personal information.

Each type of fragmentation serves a particular need but comes with its own set of complexities and potential pitfalls. Being aware of these can inform better design and deployment strategies, ultimately leading to more secure and efficient systems.

Security Implications

The landscape of security risks evolves considerably when machine learning models are fragmented, broadening the attack surface and introducing unique vulnerabilities. Version-based fragmentation can leave legacy models susceptible to exploits due to outdated security measures, serving as weak links in the system. Architecture-based fragmentation, optimized for specific tasks or hardware, can open new avenues for attacks; for example, a model fine-tuned for mobile devices may be vulnerable to attacks designed to drain computational resources. Data-based fragmentation, often mandated by regional laws or specialized use cases, can introduce biases and vulnerabilities that are region or data-specific. Real-world instances further underscore these risks; for example, decentralized models in federated learning systems have been shown to be particularly vulnerable to data poisoning attacks. Understanding the complex security implications of model fragmentation is vital for the development of targeted, effective security protocols.

Methods of Detection and Prevention

As fragmented machine learning models become increasingly ubiquitous, understanding how to detect and prevent security vulnerabilities is crucial. Here’s a detailed look at the various approaches and best practices:

Current Approaches for Identifying Vulnerabilities

Static Analysis: Tools exist that can evaluate the source code of each model variant to identify potential security flaws. However, this approach is often inadequate for catching vulnerabilities that manifest during runtime.

Dynamic Analysis: This involves the real-time monitoring of model behavior to identify anomalies that could indicate a security issue. This method is particularly useful for catching vulnerabilities that static analysis might miss.

Federated Analysis: In decentralized systems like federated learning, analyzing aggregated updates can help detect malicious activity or vulnerabilities specific to fragmented models.

Best Practices for Securing Fragmented Models

Regular Updates and Patches: All versions of the model, even those deployed on legacy systems, should be regularly updated with the latest security measures.

Role-Based Access Control (RBAC): Implementing strict access controls can limit the potential for internal threats and ensure that only authorized personnel can modify or interact with the model.

Model Auditing: Regular audits can provide an additional layer of security. These audits should include checks for vulnerabilities introduced through fragmentation, such as biases in data-based fragmented models.

Multi-Layered Security Protocols: Implementing a defense-in-depth approach that employs multiple layers of security can provide a more robust safeguard against various attack vectors.

Limitations of Existing Methods

False Positives: Current detection mechanisms can sometimes flag benign activities as threats, leading to unnecessary countermeasures.

Computational Overheads: Implementing comprehensive security measures can be computationally intensive, making them impractical for devices with limited resources.

Rapidly Evolving Threats: The dynamic nature of cybersecurity means that new vulnerabilities can emerge quickly, outpacing even the most up-to-date security measures.

Being aware of the current methodologies for detection and their limitations can help organizations strategize more effective and adaptive security measures for their fragmented models.

Recent Research

The field of machine learning security is witnessing rapid advancements, especially in the context of fragmented models. A seminal work in [1] delves deep into the vulnerabilities associated with federated learning, a decentralized form of machine learning that naturally leads to model fragmentation. In [2], the study discusses the security implications of architecture-based fragmentation, particularly in resource-constrained environments like mobile devices. Finally, a review paper [3] offers a comprehensive overview of current detection and prevention methods, highlighting their limitations and suggesting directions for future research. These scholarly works collectively indicate the growing recognition of the intricate and urgent security concerns presented by model fragmentation, thus paving the way for ongoing and future research to develop more robust and adaptive security solutions.

Conclusion

The increasing prevalence of fragmented machine learning models in today’s technology landscape introduces a unique and complex set of security vulnerabilities. While current methods for detection and prevention offer some level of safeguard, they come with inherent limitations and are often not fully equipped to handle the nuanced risks associated with different types of fragmentation. Recent research, encompassing studies on federated learning, version inconsistencies, and architecture-specific vulnerabilities, has begun to shed light on these challenges. As the field continues to evolve, it is imperative for both academia and industry to collaborate in developing more robust and adaptive security measures to mitigate the risks posed by model fragmentation.

References

  1. Jebreel, N. M., Domingo-Ferrer, J., Blanco-Justicia, A., & Sánchez, D. (2022). Enhanced security and privacy via fragmented federated learning. IEEE Transactions on Neural Networks and Learning Systems.
  2. Qiu, H. (2018). An efficient data protection architecture based on fragmentation and encryption. arXiv preprint arXiv:1803.04880.
  3. Mijwil, M., Salem, I. E., & Ismaeel, M. M. (2023). The Significance of Machine Learning and Deep Learning Techniques in Cybersecurity: A Comprehensive Review. Iraqi Journal For Computer Science and Mathematics4(1), 87-101.

Outsmarting AI with Model Evasion

Model Evasion AI

In the cybersecurity arena, artificial intelligence classifiers like neural networks and support vector machines have become indispensable for real-time anomaly detection and incident response. However, these algorithms harbor vulnerabilities that are susceptible to sophisticated evasion tactics, including adversarial perturbations and feature-space manipulations. Such methods exploit the mathematical foundations of the models, confounding their decision-making capabilities. These vulnerabilities are not just theoretical concerns but pressing practical issues, especially when deploying machine learning in real-world cybersecurity contexts that require resilience against dynamically evolving threats. Addressing this multidimensional challenge is part of the broader emerging field of adversarial machine learning, which seeks to develop robust algorithms and integrated security measures at various stages of the machine learning pipeline. Understanding and countering Model Evasion thus serves as both a challenge and an opportunity, urging enhanced collaboration between machine learning practitioners and security experts to fortify next-generation AI-driven security measures.

Understanding Model Evasion

Definition of Model Evasion

Model Evasion in the context of machine learning for cybersecurity refers to the tactical manipulation of input data, algorithmic processes, or outputs to mislead or subvert the intended operations of a machine learning model. In mathematical terms, evasion can be considered an optimization problem, where the objective is to minimize or maximize a certain loss function without altering the essential characteristics of the input data. This could involve modifying the input data x such that f(x) does not equal the true label y, where f is the classifier and x is the input vector.

What it means for an attacker to evade a machine learning model

When an attacker successfully evades a machine learning model, it essentially means they have manipulated the model’s input or underlying decision logic to produce an inaccurate or misleading output. From the attacker’s standpoint, the goal is often to violate the integrity, confidentiality, or availability of a system by avoiding detection, which could be quantified as reducing the True Positive Rate (TPR) or increasing the False Negative Rate (FNR) of the classifier.

Types of Evasion Attacks

Simple Evasion: Simple evasion tactics generally rely on manipulating observable features in input data to circumvent detection by weak or poorly-trained machine learning models. For example, in malware detection, altering the hash of a malicious file could effectively prevent its identification by simple hash-based classifiers. These types of evasion are often effective against models with shallow architectures or those that haven’t been trained on diverse datasets.

Adversarial Attacks: These attacks represent a more sophisticated class of evasion tactics that exploit the mathematical properties of machine learning models. Adversarial examples can be generated through various optimization techniques aimed at altering the model’s output classification. Among the most common methods are:

Fast Gradient Sign Method (FGSM): This technique uses the gradients of the loss function with respect to the input data to create a perturbed version of the input that leads to misclassification.

Jacobian-based Saliency Map Attack (JSMA): Unlike FGSM, which is focused on rapidly generating adversarial examples, JSMA takes a more targeted approach by iteratively perturbing features that are most influential for a given classification.

Feature Space Manipulations: These attacks specifically target the dimensions or features that are most important for model decision-making. The attacker first identifies crucial features through techniques like feature importance ranking or sensitivity analysis. Once the pivotal features are identified, they can be subtly altered to affect classification. For example, tweaking certain header fields in network packets could make malicious traffic appear benign to an intrusion detection system.

Decision Boundary Attacks: These are exploratory attacks where the attacker aims to understand the decision boundaries that a machine learning classifier employs. This could involve using techniques like:

Boundary Attack: This requires starting with an instance that is already misclassified and iteratively bringing it closer to the decision boundary without changing its classification.

Query-based Attacks: These involve sending queries to the machine learning model to gather information about its decision boundaries. The attacker then uses this data to craft inputs that are more likely to be misclassified.

By diving deep into these different types of evasion attacks, each with its unique tactics and methodologies, one can gain a holistic understanding of the vulnerabilities inherent in machine learning models used in cybersecurity applications.

Techniques for Evading AI Models

Adversarial Examples

Adversarial examples are not merely nuisances; they challenge the very mathematical underpinnings of machine learning classifiers. These are specially crafted inputs that undergo minuscule, algorithmically calculated perturbations. While trivial to a human observer, these perturbations are sufficient to mislead machine learning models. Consider a convolutional neural network trained for image classification; an adversarial example could perturb pixel values such that a benign object is classified as a threatening one. Techniques like the Fast Gradient Sign Method (FGSM) or Carlini & Wagner (C&W) attacks can be utilized to generate these adversarial instances by iteratively adjusting input features based on the gradient of the loss function relative to the input data.

Data Poisoning

Data poisoning attacks represent a more insidious form of manipulation. Instead of targeting the model during inference, the attacker tampers with the training data to embed vulnerabilities into the model itself. This is often done in a surreptitious manner so that the poisoned data doesn’t raise flags during the training process but manifests its effects when the model is deployed. For example, in a supervised learning scenario for network intrusion detection, an attacker might introduce anomalous traffic patterns as normal behavior in the training dataset. This dilutes the model’s understanding of what constitutes an ‘attack,’ reducing its efficacy in a live environment.

Model Manipulation

Model manipulation is an overt assault on the machine learning model’s architectural integrity. Here, the attacker gains unauthorized access to the internal parameters of the model, such as the weights and biases in a neural network, to recalibrate its decision boundaries. By directly manipulating these parameters, the attacker can induce arbitrary and often malicious behavior. For instance, altering the weights in the final softmax layer of a neural network could swap the labels between benign and malicious classes, thereby turning the model into a tool for subterfuge rather than security.

Social Engineering Tactics

Despite the growing reliance on algorithmic defenses, the human element remains a potential point of vulnerability. Social engineering attacks aim to exploit this human factor, using psychological manipulation to induce errors in human-AI interaction. For instance, an attacker might craft a phishing email so sophisticated that it persuades a cybersecurity analyst to flag it as a false positive. Once that ‘safe’ classification is integrated into the model’s training data, the model’s capability to correctly identify similar phishing attempts could be compromised. Alternatively, an insider could deliberately mislabel sensitive data, affecting not just a single decision but potentially undermining the model’s long-term reliability.

By dissecting these techniques, ranging from the mathematical sophistication of adversarial examples to the psychological subtleties of social engineering, we gain a multi-faceted understanding of the challenges facing AI-driven cybersecurity measures. This granular understanding is crucial for developing more resilient machine learning models and for engineering countermeasures that can effectively mitigate the risks posed by these evasion techniques.

Countermeasures

Data Integrity

At the foundation of any machine learning model is its training data, making data integrity paramount. Ensuring secure, unbiased, and representative training data mitigates the risk of data poisoning and the resultant model vulnerabilities. This could involve cryptographic data integrity checks, statistical analysis for anomaly detection, and employing differential privacy to sanitize data. Techniques such as data provenance tracking the origins, transformations, and usages of data elements can add another layer of security, making it harder for attackers to introduce malicious data into the training set without detection.

Regular Updates and Monitoring

Given the dynamic nature of threats, it is imperative that machine learning models in cybersecurity undergo frequent updates and real-time monitoring. Adaptive learning algorithms that can incrementally update the model in the face of new data can be invaluable. Monitoring should include not only performance metrics but also anomaly detection systems that can flag unusual model behavior indicative of an attack. Automated version control systems can roll back models to a previous state in case of a detected manipulation, while real-time alerting mechanisms can notify human overseers of potential issues.

Robust Machine Learning Algorithms

Machine learning models can be inherently susceptible to adversarial perturbations; therefore, the development of robust algorithms designed to resist evasion is critical. Algorithms like Robust Deep Learning (RDL) and Support Vector Machines with robust kernels focus on creating decision boundaries that are less sensitive to adversarial manipulations. Other methods, like adversarial training, where the model is intentionally exposed to adversarial examples during training, can help in hardening the model against similar attacks. Ensemble techniques, combining the predictions of multiple models, can also be effective in diluting the impact of attacks aimed at a single model’s weaknesses.

Zero Trust Architecture for Model Deployment

Deploying machine learning models within a Zero Trust Architecture (ZTA) can enhance security by adhering to a “never trust, always verify” paradigm. In such an architecture, even if an attacker gains access to a part of the network, the inherent distrust built into the system will restrict their access to the machine learning model parameters or training data. This makes direct model manipulation or data poisoning far more challenging.

Blockchain for Auditing and Provenance

Blockchain technology can be employed to secure the training data and the machine learning model’s parameters, offering an immutable record of changes and updates. Every update or alteration would be stored in a new block, providing a transparent and tamper-proof log. This could be crucial for compliance, auditing, and also for identifying and rolling back any unauthorized changes to the model or training data.

Recent Research

The academic and industrial research communities are vigorously investigating model evasion techniques and countermeasures in AI-driven cybersecurity. Recent studies, Resistant to Adversarial Attacks [1], have set a foundation for understanding the mathematical formulations that could lead to robust models. Meanwhile, in [2], the author took a significant step in providing formal guarantees against specific kinds of evasion attacks. On the adversarial frontier, ‘Transferable Adversarial Attacks,’ as explored in [3], showcase the feasibility of successful evasion in black-box settings. Defensive techniques such as ‘Adversarial Logit Pairing’ and ‘Defensive Distillation’ have been studied for their real-world applicability, as shown in [4] and [5]. An emerging interdisciplinary approach combines machine learning, cryptography, and game theory to design adaptive algorithms, a notion reflected in [6]. This collective body of research illustrates the ongoing arms race in AI cybersecurity, spotlighting both the challenges and innovative solutions in the battle against model evasion.

Future Prospects

The future landscape of AI in cybersecurity is poised to be shaped by emerging technologies and a host of legal and ethical considerations. On the technological front, Explainable AI (XAI) promises to make the decision-making processes of complex models more transparent, thereby enabling easier audits and potentially exposing vulnerabilities before they can be exploited. Federated Learning offers another avenue, decentralizing the training process across multiple devices to maintain data privacy and reduce the risk of centralized data poisoning. Simultaneously, the evolving legal landscape is pushing for greater accountability and compliance in the use of AI for cybersecurity. Regulations may soon require stringent audits of machine learning models, ensuring that they meet ethical standards and are free from biases that could be exploited for evasion. As both technology and law advance, they will mutually shape the challenges and solutions in combating AI model evasion, adding layers of complexity and opportunity for more robust countermeasures.

Conclusions

In the ever-evolving landscape of AI and cybersecurity, the need to address model evasion tactics stands out as a critical challenge, essential for maintaining the integrity and reliability of AI systems. From identifying rudimentary input manipulations to combating advanced adversarial attacks, the multi-dimensional strategies explored in this blog reveal that defending against evasion is not merely a technical obstacle but a complex, evolving discipline. Given the significant impact of evasion on AI models, it’s imperative for researchers, practitioners, and policymakers alike to devote increased attention to this issue, elevating it not only as an area ripe for academic exploration but also as a practical and regulatory priority requiring immediate and sustained action.

References

  1. Mądry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017). Towards deep learning models resistant to adversarial attacks. stat1050, 9.
  2. Cohen, J., Rosenfeld, E., & Kolter, Z. (2019, May). Certified adversarial robustness via randomized smoothing. In international conference on machine learning (pp. 1310-1320). PMLR.
  3. Wei, Z., Chen, J., Goldblum, M., Wu, Z., Goldstein, T., & Jiang, Y. G. (2022, June). Towards transferable adversarial attacks on vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 3, pp. 2668-2676).
  4. Kannan, H., Kurakin, A., & Goodfellow, I. (2018). Adversarial logit pairing. arXiv preprint arXiv:1803.06373.
  5. Papernot, N., & McDaniel, P. (2016). On the effectiveness of defensive distillation. arXiv preprint arXiv:1607.05113.
  6. Chen, P. Y., Sharma, Y., Zhang, H., Yi, J., & Hsieh, C. J. (2018, April). Ead: elastic-net attacks to deep neural networks via adversarial examples. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).

Securing Machine Learning Workflows through Homomorphic Encryption

Homomorphic Encryption ML

In the burgeoning field of machine learning, data security has transitioned from being an optional consideration to a critical component of any robust ML workflow. Traditional encryption methods often fall short when it comes to securing ML models and their training data.

Unlike standard encryption techniques, which require data to be decrypted before any processing or analysis, Homomorphic Encryption allows computations to be performed directly on the encrypted data. This mitigates the risks associated with exposing sensitive information during the data processing stage, a vulnerability that has been exploited in various attack vectors like data poisoning and model inversion attacks. Through the utilization of intricate mathematical algorithms and lattice-based cryptography, Homomorphic Encryption ensures that data privacy is preserved without sacrificing the utility or accuracy of the ML models it supports. This enables organizations to confidently leverage machine learning capabilities for sensitive applications in healthcare, finance, and national security.

What Is Data Encryption and Why Is It Essential?

Data encryption employs complex algorithms to convert plain text or other human-readable data into a cipher, an encoded, unreadable format. Decryption keys, held only by authorized parties, are required to convert the data back into its original format. The objective extends beyond just data privacy; it involves ensuring data integrity and authentication as well. In the context of machine learning, where datasets may consist of sensitive attributes such as personal identifiers or confidential business metrics, encryption transcends being a mere feature and becomes an indispensable layer of security. Advanced encryption techniques can also protect data during in-transit and at-rest phases, effectively “sealing off” data vulnerabilities across the machine learning lifecycle.

The Security Imperative

Machine learning models thrive on data; the more varied and vast, the better. These datasets often include an array of sensitive information ranging from healthcare records and financial transactions to user browsing behaviors. This diversity in data types doesn’t just offer richer training material for machine learning algorithms; it also presents multiple attack vectors for malicious entities. Unauthorized accessdata manipulation, and outright data theft are risks that can jeopardize not only the integrity of the ML model but also violate privacy regulations, such as GDPR or CCPA. In today’s digital environment, where a single data breach can result in severe financial and reputational damage, encryption goes from being a “good-to-have” to an unequivocal necessity. Advanced encryption standards like AES-256 and RSA-2048 have emerged as industry benchmarks in securing highly sensitive data in ML workflows.

Guidelines to Implement Data Encryption

Implementing data encryption in a machine-learning environment requires a nuanced approach considering several variables. These include the specific cryptographic algorithms to be employed, the need to meet stringent regulatory standards, and the computational costs associated with encryption. Each of these variables is crucial for ensuring that the machine-learning pipeline remains secure and efficient.

Symmetric vs. Asymmetric Encryption

Symmetric and asymmetric encryption are the two primary paradigms in modern cryptography, each with its own set of advantages and limitations.

Symmetric Encryption: In this method, a single key is used for encryption and decryption. Algorithms like Advanced Encryption Standard (AES) are commonly used for symmetric encryption. They are relatively fast and require less computational power. However, the challenge here is key distribution and management. Since the same key is used for both processes, it must be shared between parties, increasing the risk of exposure.

Asymmetric Encryption: This approach uses a pair of keys: a public key to encrypt the data and a private key to decrypt it. Algorithms like RSA (Rivest–Shamir–Adleman) are widely used in asymmetric encryption. The advantage is enhanced security, as the private key never needs to be shared. However, the encryption and decryption processes are computationally more intensive, which could be a concern in time-sensitive applications.

Regulatory Compliance

Legal frameworks around data protection are increasingly stringent. Regulations such as the General Data Protection Regulation (GDPR) in the European Union or the Health Insurance Portability and Accountability Act (HIPAA) in the United States place rigorous requirements on data encryption.

GDPR: This regulation mandates data controllers and processors implement appropriate technical measures to ensure data security. Advanced cryptographic techniques, including AES and RSA, are often recommended to meet GDPR requirements.

HIPAA: In healthcare applications, where machine learning can be used for tasks like diagnostic imaging or predictive analytics, compliance with HIPAA is a must. This means implementing encryption algorithms that have been approved by recognized institutions like the National Institute of Standards and Technology (NIST).

Computational Overheads

The process of encrypting and decrypting data adds computational overhead, affecting the performance of machine learning models, particularly in real-time or near-real-time applications.

Resource Allocation: In applications where computational resources are limited, lightweight cryptographic algorithms may be more appropriate. For example, algorithms like ChaCha20 can offer good security with lower computational requirements.

Performance Metrics: It’s important to closely monitor key performance indicators (KPIs) such as latency and throughput when implementing encryption to ensure that the added security does not compromise the system’s performance.

A Deep Dive into Homomorphic Encryption

Homomorphic Encryption stands out among encryption techniques for its unique ability to enable computations directly on encrypted data. This distinctive feature has enormous implications for machine learning workflows, especially in cloud environments and other scenarios where data privacy is a critical concern.

An Overview

Homomorphic Encryption is a class of encryption techniques that permits operations to be executed on ciphertexts, which, when decrypted, yield the same result as if the operation had been performed on plaintext. Unlike traditional encryption schemes that require data to be decrypted before any computational operation, Homomorphic Encryption retains data confidentiality throughout the computational process. This is achieved through complex algebraic structures that allow specific types of mathematical operations on encrypted data. Techniques like Ring-LWE (Learning With Errors) and Fan-Vercauteren packing are commonly employed to make the encryption scheme both secure and efficient.

Advanced Security Measures

The robustness of Homomorphic Encryption goes beyond the simple concealment of data. It provides semantic security, ensuring that an unauthorized entity accessing the encrypted data cannot infer any meaningful information without the decryption key. Moreover, modern implementations often employ lattice-based cryptographic approaches, which are believed to resist attacks from quantum computers, adding an additional layer of future-proof security.

Performance Metrics: The Trade-Offs

While Homomorphic Encryption is revolutionary, it has historically been plagued with high computational and storage overheads. These challenges have been mitigated in part by algorithmic improvements and hardware acceleration. For instance, implementing batching techniques and parallel computation can significantly reduce the time required for operations on encrypted data. However, achieving an optimal balance between computational performance and data security remains an active research area.

Potential Use-Cases: Beyond Conventional Boundaries

The applications of Homomorphic Encryption extend far and wide. In healthcare, it can be employed to perform encrypted medical data analysis, thus ensuring patient confidentiality. In finance, secure transactions and fraud detection algorithms can run on encrypted data, enhancing the privacy of financial records. Furthermore, various studies and research papers have demonstrated the utility of Homomorphic Encryption in federated learningsecure multi-party computation, and even voting systems.

Best Practices and Recommendations

When implementing Homomorphic Encryption, it’s essential to consider several best practices for optimum results.

Parameter Selection: Parameters like the noise level and modulus size should be carefully chosen to ensure a balance between security and efficiency.

Expert Consultation: Due to the complexity of Homomorphic Encryption, consultation with experts in the field of cryptography is often advisable for a proper and secure implementation.

Regular Audits: Given the rapid advancements in the field, regular security audits are essential to make sure the encryption measures are up-to-date and resistant to new types of vulnerabilities.

Recent Research

The proliferation of Homomorphic Encryption is not merely a theoretical advance but a catalyst for revolutionary changes in the field of machine learning and beyond. It’s steering a new wave of research focused on privacy-preserving methodologies, effectively acting as a linchpin between data security and computational feasibility.

Key Contributions in Neural Networks

The paper “CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy,” serves as a seminal work in this domain. It delves into the intricate processes by which neural networks can be trained and deployed directly on the ciphertext. By leveraging specific architectures and optimization techniques, the study demonstrates that it’s possible to achieve both high throughput and accuracy, resolving some of the traditional trade-offs associated with Homomorphic Encryption. The study also employs a series of sophisticated mathematical transformations, such as activation function approximations, to make neural networks compatible with the algebraic structures utilized in Homomorphic Encryption.

Advancements in Cloud-Based Applications

Another remarkable contribution is the paper titled “Application of Homomorphic Encryption in Machine Learning,” which focuses on cloud-based machine learning services. Here, the emphasis is on preserving user privacy when offloading computations to a third-party cloud provider. The paper presents novel algorithms and protocols that leverage Homomorphic Encryption to enable privacy-preserving training and inference in a cloud environment, without sacrificing the quality of the machine learning model.

Specialized Domains: Healthcare Data

The domain-specific applications are equally compelling. The paper “A privacy-preserving federated learning scheme with homomorphic encryption for healthcare data” is particularly noteworthy. It addresses the challenge of securely aggregating and analyzing medical data across various healthcare providers while fully maintaining patient confidentiality. The scheme allows the development of machine learning models that can learn from the entire dataset without ever exposing individual records, a major breakthrough in the realm of secure, federated learning.

Breaking Boundaries in Deep Learning

Further pushing the envelope is research like “A symbolic execution compiler for privacy-preserving Deep Learning with Homomorphic Encryption.” This study focuses on leveraging symbolic computation methods to enhance the scalability and performance of deep learning models trained on encrypted data. It introduces a novel compiler that translates deep learning computations into a format that can be efficiently executed under Homomorphic Encryption, thus widening the applicability of HE in complex machine learning architectures.

The widespread adoption and application of Homomorphic Encryption in recent research signify its rapidly growing influence. It’s a focal point for scholars and practitioners alike, aiming to harmonize data security with the unyielding advancement of machine learning technologies.

Conclusion

Homomorphic Encryption has transitioned from being a mathematical curiosity to a linchpin in fortifying machine learning workflows against data vulnerabilities. Its complex nature notwithstanding, the unparalleled privacy and security benefits it offers are compelling enough to warrant its growing ubiquity. As machine learning integrates increasingly with sensitive sectors like healthcare, finance, and national security, the imperative for employing encryption techniques that are both potent and efficient becomes inescapable.

Proactive adoption of transformative encryption approaches such as Homomorphic Encryption serves a dual purpose: it reinforces ethical imperatives around data privacy and propels the machine learning discipline into new territories, ones where data sensitivity has traditionally been a hindrance. Future directions in machine learning are inextricably tied to advancements in data security. Homomorphic Encryption, with its capacity to enable computations on encrypted data without compromising privacy, is poised to play a decisive role in shaping this future. As we traverse further into the era of ubiquitous machine learning applications, the need for methods like Homomorphic Encryption, which harmonize robust security with operational efficiency, will undoubtedly escalate.

References

  1. Gilad-Bachrach, R., Dowlin, N., Laine, K., Lauter, K., Naehrig, M., & Wernsing, J. (2016, June). Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International conference on machine learning (pp. 201-210). PMLR.
  2. Ameur, Y., Bouzefrane, S., & Audigier, V. (2022). Application of homomorphic encryption in machine learning. In Emerging Trends in Cybersecurity Applications (pp. 391-410). Cham: Springer International Publishing.
  3. Wang, B., Li, H., Guo, Y., & Wang, J. (2023). PPFLHE: A privacy-preserving federated learning scheme with homomorphic encryption for healthcare data. Applied Soft Computing, 110677.
  4. Cabrero-Holgueras, J., & Pastrana, S. (2023). HEFactory: A symbolic execution compiler for privacy-preserving Deep Learning with Homomorphic Encryption. SoftwareX22, 101396.

Understanding Data Poisoning: How It Compromises Machine Learning Models

Data Poisoning ML AI

Machine learning (ML) and artificial intelligence (AI) have rapidly transitioned from emerging technologies to indispensable tools across diverse sectors such as healthcarefinance, and cybersecurity. Their capacity for data analysis, predictive modeling, and decision-making holds enormous transformative potential, but it also introduces a range of vulnerabilities. One of the most insidious among these is data poisoning, a form of attack that targets the very lifeblood of ML and AI: the data used for training.

Understanding and addressing data poisoning is critical, not just from a technical standpoint but also due to its far-reaching real-world implications. A poisoned dataset can significantly degrade the performance of ML models, leading to flawed analytics, incorrect decisions, and, in extreme cases, endangering human lives.

What is Data Poisoning?

Data poisoning is a targeted form of attack wherein an adversary deliberately manipulates the training data to compromise the efficacy of machine learning models. The training phase of a machine learning model is particularly vulnerable to this type of attack because most algorithms are designed to fit their parameters as closely as possible to the training data. An attacker with sufficient knowledge of the dataset and model architecture can introduce ‘poisoned’ data points into the training set, affecting the model’s parameter tuning. This leads to alterations in the model’s future performance that align with the attacker’s objectives, which could range from making incorrect predictions and misclassifications to more sophisticated outcomes like data leakage or revealing sensitive information.

The impact of data poisoning can be subtle, making it difficult to detect through conventional validation techniques like k-fold cross-validation or holdout validation. It often requires specialized anomaly detection algorithms or model auditing techniques to identify the manipulations. Furthermore, the effect can be cascading, affecting not just the primary ML model but also any downstream applications or decision-making processes that rely on the model’s output.

Types of Data Poisoning Attacks

Label Flipping

In a label-flipping attack, the attacker intentionally reverses the labels for selected data entries within the training set. For classification tasks, this means that data points representing one class are labeled as another. Technically speaking, consider a binary classification problem with labels yÎ{0,1} yÎ{0,1}. The attacker would flip the label y to 1-y to for selected samples in the training set. This confuses the learning algorithm and impacts the decision boundaries it constructs, leading to erroneous classifications.

Outliers Injection

In outliers injection attacks, the attacker introduces data points that are significantly different from the existing data but labels them in a manner that distorts the model’s understanding of the feature space. These data points can be multivariate outliers that lie outside the distribution of the genuine training data in the feature space. When algorithms like k-NN (k-Nearest Neighbors) or SVM (Support Vector Machines) are used, these outlier points can have a disproportionate effect on the decision boundaries, leading to misclassifications.

Feature Manipulation

Feature manipulation involves altering the features or characteristics of the data points in the training set. This could range from adding noise to numerical features to introducing subtle artifacts in image data. For instance, in a Convolutional Neural Network (CNN) used for image recognition, injecting pixel-level noise or adversarial patterns into the training images could lead the model to learn incorrect representations. This type of attack is particularly nefarious as it may not affect the training accuracy but will degrade the model’s generalization capability on new, unpoisoned data.

How Data Poisoning Affects Models

Performance Degradation

Data poisoning often leads to a decline in the model’s performance metrics, such as accuracy and precision. The impact can be localized to specific classes, making it challenging to detect. For instance, in algorithmic trading, a slight decrease in predictive accuracy can result in significant financial losses.

Decision Boundary Distortion

Poisoned data can distort the decision boundaries that the model learns, affecting its ability to generalize well to new data. For example, in healthcare applications like tumor classification, distorted decision boundaries can lead to severe misdiagnoses, putting lives at risk.

Security Risks

Data poisoning can pave the way for more advanced attacks, such as adversarial or backdoor attacks. These are often harder to detect and can bypass existing security protocols. In regulated industries, a compromised model may also violate data protection laws, leading to legal consequences.

Case Studies

It’s crucial to underscore that data poisoning is not a theoretical concern but an immediate and practical risk. Recent research drives this point home. According to the study, the researchers were able to demonstrate that with just $60 USD, they could have poisoned 0.01% of the LAION-400M or COYO-700M datasets. This is a real-world implication that underscores the need for immediate action to secure AI and ML systems. Read the full paper here.

Autonomous Vehicles

In the realm of autonomous vehicles, data poisoning attacks have significant implications for safety [1,2,3]. Researchers have demonstrated that injecting poisoned data into the training set can lead a self-driving car to misinterpret road signs. For example, a stop sign could be misclassified as a speed limit sign, causing the vehicle to accelerate instead of stopping. This sort of error could result in collisions and put human lives at risk. Such attacks underscore the need for robust data verification techniques specifically designed for safety-critical systems like autonomous vehicles.

Healthcare Models

Data integrity is paramount in healthcare, where machine learning models are used for everything from diagnostic imaging to treatment recommendations. Poisoned data can lead to misdiagnoses or incorrect treatment plans [4]. For instance, if a machine learning model trained to identify tumors is fed poisoned data, it might incorrectly classify a malignant tumor as benign, delaying essential treatment and endangering the patient’s life. Given the high stakes, data security measures are crucial in healthcare applications.

Financial Fraud Detection

Financial institutions often rely on machine learning models to detect fraudulent transactions. In a data poisoning attack, an attacker could subtly alter training data to manipulate the model’s behavior [5]. This could result in the model incorrectly flagging legitimate transactions as fraudulent, causing inconvenience to customers and incurring additional verification costs. Conversely, the model might fail to recognize actual fraudulent transactions, leading to financial losses and eroding customer trust.

Recommendation Systems

In the context of e-commerce and streaming services, recommendation systems are vulnerable to data poisoning attacks aimed at skewing product or content preferences [6]. An attacker, for example, could inject fake user preferences into the training data to make a poorly reviewed movie appear prominently on a streaming service’s recommendation list. Such manipulation doesn’t just affect the user experience; it can also result in lost revenue and damaged reputations for service providers.

What Happens if Data Gets Poisoned?

Financial Sector

In trading algorithms, poisoned data can cause false triggers for buy or sell orders, leading to market manipulation and financial instability. Regulatory action could follow, causing long-term reputational damage for the company responsible for the algorithm.

Healthcare

In predictive healthcare models, poisoned data could result in misdiagnoses, leading to incorrect treatments that could put lives at risk. Moreover, the medical institution may face lawsuits, loss of accreditation, or a decline in patient trust.

Cybersecurity

In intrusion detection systems, data poisoning could lead to false negatives, where real threats go undetected, or false positives, where benign activities are flagged. Either way, the result is a less secure environment, vulnerable to further attacks and potential data breaches.

Mitigation Strategies

Data Sanitization

Data sanitization involves rigorous pre-processing steps to identify and remove suspicious or anomalous data points from the training set. This can include statistical methods for outlier detection, as well as machine learning techniques like anomaly detection algorithms. Sanitization is often the first line of defense against data poisoning and is crucial for maintaining data integrity. It can significantly reduce the risk of a model being compromised, but it does require continuous updates to adapt to new types of poisoning strategies.

Model Regularization

Model regularization techniques like L1 (Lasso) and L2 (Ridge) regularization add a penalty term to the model’s objective function to constrain its complexity. By doing so, regularization makes the model less sensitive to small fluctuations in the training data, thereby increasing its robustness against poisoned data points. While regularization may not entirely prevent poisoning attacks, it can mitigate their impact by making it more difficult for the attacker to drastically alter the model’s behavior.

Real-time Monitoring

Based on my background in cybersecurity, I find the approach of real-time monitoring particularly compelling. This strategy brings AI and machine learning security closer to traditional cybersecurity paradigms, helping to integrate them seamlessly into existing security processes. Real-time monitoring involves continuously tracking key performance indicators (KPIs) of the machine learning model to detect any unusual patterns or deviations. Specialized tools and services are already available on the market that facilitate the integration of model monitoring into existing cybersecurity monitoring and detection systems.

Alerts can be configured to notify system administrators immediately of any sudden drops in performance metrics like accuracy or precision. This enables swift intervention, which is crucial for minimizing the impact of an ongoing data poisoning attack. However, it’s essential to note that these monitoring tools must be paired with well-defined playbooks or runbooks for immediate response to be truly effective.

Third-Party Audits

Having external cybersecurity experts audit the machine learning system can reveal vulnerabilities that the internal team might overlook. Third-party audits can examine the data pipelines, model architecture, and overall system configuration for potential weaknesses that could be exploited for data poisoning. These audits provide an additional layer of security and can offer targeted recommendations for improving the system’s resilience against poisoning attacks.

Data Provenance

One effective approach to counteract these attacks is by leveraging data provenance. This includes:

  1. Data Provenance Tracking: This involves maintaining a record of the origin and history of each data point in the training set. By understanding where data comes from and the transformations it has undergone, we can assess its trustworthiness.
  2. Provenance Verification: Before incorporating a data point into the training set, its provenance is verified. This can be done using cryptographic techniques, timestamps, or by cross-referencing with trusted data sources.
  3. Anomaly Detection: By analyzing the provenance information, anomalies or patterns that deviate from the norm can be detected. Such anomalies might indicate malicious intent or corrupted data.
  4. Data Filtering: Data points with suspicious or unverified provenance can be filtered out or given less weight during the training process. This ensures that the model is trained only on trustworthy data.
  5. Continuous Monitoring: Even after initial training, the model’s performance and the incoming data’s provenance should be continuously monitored. This helps in detecting any late-stage poisoning attempts and taking corrective actions.

By integrating data provenance into the machine learning pipeline, we can add an additional layer of security, ensuring that models are robust and resistant to poisoning attacks.

Conclusion

Data poisoning not only undermines the effectiveness of machine learning models but also poses substantial risks to various sectors, from healthcare and finance to autonomous driving and e-commerce. Given its insidious nature, it is paramount to understand the different types of poisoning attacks, their potential impact, and the areas they can affect. With this knowledge in hand, we can tailor our defense strategies, such as data sanitization, model regularization, real-time monitoring, and third-party audits, to effectively thwart such attacks.

As machine learning and AI technologies increasingly infiltrate our daily lives and critical infrastructures, the urgency to fortify these systems cannot be overstated. The sophistication of data poisoning attacks is likely to grow in tandem with advancements in machine learning algorithms, making the need for a multi-layered, adaptive security approach more crucial than ever.

By proactively addressing the challenges posed by data poisoning, we can pave the way for safer, more reliable machine learning models, safeguarding not just the technology but also the myriad applications and human lives that depend on it.

References

  1. Chen, Y., Zhu, X., Gong, X., Yi, X., & Li, S. (2022). Data poisoning attacks in Internet-of-vehicle networks: taxonomy, state-of-the-art, and future directions. IEEE Transactions on Industrial Informatics19(1), 20-28.
  2. Wang, S., Li, Q., Cui, Z., Hou, J., & Huang, C. (2023). Bandit-based data poisoning attack against federated learning for autonomous driving models. Expert Systems with Applications227, 120295.
  3. Cui, C., Du, H., Jia, Z., He, Y., Yang, Y., & Jin, M. (2022, December). Data Poisoning Attack Using Hybrid Particle Swarm Optimization in Connected and Autonomous Vehicles. In 2022 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) (pp. 1-5). IEEE.
  4. Verde, L., Marulli, F., & Marrone, S. (2021). Exploring the impact of data poisoning attacks on machine learning model reliability. Procedia Computer Science192, 2624-2632.
  5. Paladini, T., Monti, F., Polino, M., Carminati, M., & Zanero, S. (2023). Fraud Detection Under Siege: Practical Poisoning Attacks and Defense Strategies. ACM Transactions on Privacy and Security.
  6. Huang, H., Mu, J., Gong, N. Z., Li, Q., Liu, B., & Xu, M. (2021). Data poisoning attacks to deep learning based recommender systems. arXiv preprint arXiv:2101.02644.

Most popular articles this week

Quantum Computer 5G Security

The Quantum Computing Threat

Mastodon