If you're evaluating AI-driven security tooling, you'll hit the term "large language model" in the first sentence of every vendor pitch. What's underneath it is a single trick, and the thing that makes an LLM impressive is exactly what makes it unreliable.
Feed it a partial sentence and it estimates probabilities over what comes next, picks one, and repeats. Scale that to billions of parameters and trillions of examples, and the next-token predictor starts to do things that look like reasoning, code generation, and natural language understanding. This is not necessarily because it "understands" in any deep sense. It lacks what we would usually point to as evidence of understanding: persistent goals, a model of truth it checks itself against, any experience outside the current context window. The patterns in human language apparently encode enough structure that prediction and competence start to overlap, whatever "understanding" ends up meaning.
Where did large language models come from?
The groundwork was laid long before neural networks became practical. Alan Turing's 1950 paper "Computing Machinery and Intelligence," which framed the question as "can machines think?" and proposed the Turing Test, was one of the first serious attempts to define machine intelligence. Five years later, John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon proposed a summer research project to be held at Dartmouth College, and that 1955 proposal is where the term "artificial intelligence" first appears in writing. Minsky went on to anchor the MIT AI Lab in Cambridge, a few miles from where this is being written.
Early approaches were symbolic: hand-coded rules that worked in narrow domains and fell apart everywhere else. Statistical machine learning shifted the frame in the 1980s and '90s. Instead of encoding rules, you gave the machine data and let it find patterns, though someone still had to decide in advance what features mattered. Deep learning broke that constraint, letting models learn their own features from raw data. The transformer architecture, introduced in 2017, was the turning point: rather than processing text sequentially, transformers let every token attend to every other token at once. The mechanism is literally called attention. That cracked the long-range memory problem and parallelized efficiently across hardware, enabling the scale that makes current LLMs possible. Every major model today descends from it. The through-line across all seventy years is a field repeatedly running into the limits of each approach and reaching for the next one, and the current wave is no exception.
How is an LLM trained?
An LLM is trained by adjusting billions of internal numbers, called weights, until it gets good at predicting text, then fine-tuning that raw ability into useful behavior. Understanding this explains both what these models can do and why they fail the way they do.
To start, those weights are random and the model guesses poorly. The loop that fixes that takes four steps: feed it text, measure how wrong it was (the loss), run backpropagation to figure out which weights contributed to the error, and nudge them in the right direction. Repeat trillions of times. No one hand-codes rules like "adjectives come before nouns." A model learns all of it from the statistical pressure of being wrong billions of times and getting corrected. By the end, the weights encode an enormous amount of structure about language, logic, and the world, not because anyone put it there but because the data forced it.
It happens in two phases. Pretraining is where the money goes: the model works through a massive, unlabeled corpus of code, books, web pages, and documentation, learning the statistical shape of language, at a cost that now runs into the hundreds of millions of dollars for frontier models. A pretrained model is fluent but not useful yet; it will only continue whatever you give it. The second phase, supervised fine-tuning and reinforcement learning from human feedback (RLHF), trains it on examples of desired behavior and nudges it toward responses that score well with human raters. This is where behavioral differences between models emerge: two models built on near-identical pretraining data can behave very differently afterward, which is why "it's just an LLM" undersells how much vendor-specific tuning shapes what you're deploying.
What can't an LLM do on its own?
On its own, an LLM has no persistent memory, no built-in internet access, and no way to verify its own outputs against ground truth. The most interesting thing about these models isn't what they can do. It's the edges of what they can't, and the fact that the model has no idea where those edges are.
Everything an LLM produces is generated, not retrieved, which means it can produce fluent, confident, entirely wrong answers with no internal alarm going off. This is the hallucination problem, and it's not a bug that gets patched out. It's structural. The model learned to predict plausible text, not true text, and it has no mechanism to tell the difference.
That matters more in security than almost anywhere else. A hallucinated CVE number, a fabricated function signature, or a confidently wrong claim about protocol behavior isn't a stylistic quirk. It's the kind of error that wastes an analyst's afternoon or gets baked into a client report. A human analyst who has made a mistake usually knows they're uncertain. An LLM that's wrong often sounds even more confident. Any serious deployment must assume the model will sometimes fail and design around that, through verification layers, sandboxed execution, and human review at decision points, not by hoping a bigger model eventually stops hallucinating.
The properties that matter more than the hype
- It's statistical, not deterministic. Ask the same question twice and you can get different phrasing, different reasoning, sometimes different conclusions. A feature for creative work, a liability for anything that needs to be repeatable.
- The context window is the model's working memory. What it "knows" for a task is either frozen into its training weights or supplied in the prompt. There's no memory between calls unless an application layer builds it in, and models tend to weight information unevenly across a long context, losing track of details buried in the middle.
- Tool use turns a text predictor into an agent. On its own, an LLM just produces text. Wire it to APIs, browsers, or code execution and it can take real actions. That's where "agentic" systems come from, and where the risk surface changes entirely. A wrong fact in a chat window is annoying. A wrong action taken with filesystem or network access is a problem.
- No two models are alike. Vendors train on different data, tune for different behaviors, and ship at different price, latency, and capability points. A model excellent at code generation might be mediocre at long-form reasoning; one tuned to be refusal-heavy might be a poor fit for red team tooling that legitimately needs to discuss offensive techniques. That's a design choice, not a constraint.
Where does the intelligence live?
If the model has no memory, no ground truth, and no sense of where its own knowledge ends, then the model can't be the whole of an AI system's intelligence. A lot of it lives in what gets built around it.
Like a computer with only a CPU, reasoning without context is raw capability with nowhere to stand. It is quick and fluent, but unable to hold onto anything, check itself, or reach beyond the prompt. The processing is real; on its own it doesn't add up to much. What makes it useful is everything around it: memory to carry state, a link to the outside world, and a loop to run in, generating, checking against reality, remembering, and going again.
That's how we think about building ThreatWell, though it isn't really about any one platform. It's the premise this piece opened with, seen from the other side: the trick that makes an LLM impressive is the same one that makes it unreliable. Building with that in mind, rather than around it, is the whole game.
ADCL publishes what we find in Field Notes, written for defenders. AI 101 is our explainer track: the concepts behind AI-driven security tooling, in plain language, from the people building it.